https://wiki.jcihosting.com/api.php?action=feedcontributions&feedformat=atom&user=SupportJCWiki - User contributions [en]2026-06-13T05:55:20ZUser contributionsMediaWiki 1.42.3https://wiki.jcihosting.com/index.php?title=Main_Page&diff=1120Main Page2013-03-15T22:54:49Z<p>Support: </p>
<hr />
<div>* [[Special:AllPages|All Pages]]<br />
* [[Payment_System|Payment System]]<br />
* [[Management_System_/_Public_Website_/_Signup_/_Account_Manager|Management System / Public Website / Signup / Account Manager]]<br />
* [[New_Signups|New Signups]]<br />
* [[Shutting_Down_Service|Shutting Down Service]]<br />
* [[Routine_Maintenance|Routine Maintenance]]<br />
* Reference<br />
** [[VPS_Management|VPS Management]]<br />
** [[Jail_Server_Install|Jail Server Install]]<br />
** [[Virtuozzo_Server_Install|Virtuozzo Server Install]]<br />
** [[Infrastructure_Machines|Infrastructure Machines]]<br />
** [[Colo_Server_Setup|Colo Server Setup]]<br />
** [[DNS]]<br />
** [[FreeBSD_Reference|FreeBSD Reference]]<br />
** [[Virtuozzo_/_Linux_Reference|Virtuozzo / Linux Reference]]<br />
** [[Cabinetmap]]<br />
** [[Switch_Control|Switch Control]]<br />
** [[Private_IP_Mapping|Private IP Mapping]]<br />
** [[Data_Center_Notes|Data Center Notes]]<br />
** [[Crash_Reference|Crash Reference]]<br />
** [[Hardware_Reference|Hardware Reference]]<br />
** [[ATS]]<br />
** [[Wiring_Reference|Wiring Reference]]<br />
** [[Screen]]<br />
** [[RAID_Cards|RAID Cards]]<br />
** [[BigBrother]]<br />
** [[Bandwidth_Management|Bandwidth Management]]<br />
** [[Network_Time_(ntp)|Network Time (ntp)]]<br />
** [[DRAC/RMM|DRAC/RMM]]<br />
** [[Customer_Backups|Customer Backups]]<br />
** [[DoS_Attacks|DoS Attacks]]<br />
** [[IPKVM|IPKVM]]<br />
* [[System-generated_Notifications|System-generated Notifications]]<br />
* [[Customer_Interaction|Customer Interaction]]<br />
* [[Backup_Accounts_(rsync.net)| Backup Accounts (rsync.net)]]<br />
* [[E-Monitoring_notes|E-Monitoring notes]]<br />
* [[Todo]]<br />
<br />
<br />
== Getting started ==<br />
* [//www.mediawiki.org/wiki/Manual:Configuration_settings Configuration settings list]<br />
* [//www.mediawiki.org/wiki/Manual:FAQ MediaWiki FAQ]<br />
* [https://lists.wikimedia.org/mailman/listinfo/mediawiki-announce MediaWiki release mailing list]<br />
* [https://meta.wikimedia.org/wiki/Help:Advanced_editing Advanced Editing]</div>Supporthttps://wiki.jcihosting.com/index.php?title=E-Monitoring_notes&diff=1119E-Monitoring notes2013-03-15T22:54:32Z<p>Support: </p>
<hr />
<div>= isys =<br />
<br />
runs mail for e-monitoring.net<br />
<br />
the key on root@mail should let you ssh w/o pass to isys.e-monitoring.net<br />
<br />
isys has a mail queue problem where it gets big and we run out of inodes. we setup a cronjob to clear that out.<br />
<br />
== isys - out of inodes ==<br />
<br />
<br />
# ssh to root@isys.e-monitoring.net<br />
# isys# cd /var/spool/mqueue<br />
# isys# sh<br />
# # for f in `ls`; do rm $f; done<br />
<br />
:(wait awhile)<br />
<br />
<pre> wrote a script:<br />
<br />
isys# cat > /root/clearqueue.sh<br />
#!/bin/sh<br />
for f in `ls /var/spool/mqueue`; do rm /var/spool/mqueue/$f; done<br />
isys# sh /root/clearqueue.sh</pre><br />
<br />
<br />
<span style="color:red">crontabed for 3am on the 1st of the month</span><br />
<br />
<br />
= polling / magrathea = <br />
== config new IP ==<br />
Edit:<br />
<pre>/etc/defaultrouter<br />
/export/home/groundwater/bin/getit.pl<br />
/etc/hosts<br />
/etc/hostname.le1</pre><br />
<br />
On production:<br />
<pre>/home/htdocs/gw/index.pl<br />
/home/htdocs/isys/pollscan.pl<br />
/home/gw/shipley/etc/config.pl<br />
/home/gw/owrd/etc/config.pl<br />
/home/dboodman/www/isys/pollscan.pl<br />
Apache conf</pre><br />
<br />
== stop it from trying to dial out ==<br />
<pre>mysql> select * from resources;<br />
+------------+------------+----------------+-------+---------+----------+<br />
| resourceID | resource | connectionType | inUse | inUseBy | inUseFor |<br />
+------------+------------+----------------+-------+---------+----------+<br />
| 1 | /dev/cuaa0 | 1 | NULL | NULL | NULL |<br />
| 2 | network | 2 | NULL | NULL | NULL |<br />
| 3 | network | 2 | NULL | NULL | NULL |<br />
| 4 | network | 2 | NULL | NULL | NULL |<br />
| 5 | network | 2 | NULL | NULL | NULL |<br />
| 6 | network | 2 | NULL | NULL | NULL |<br />
| 7 | network | 2 | NULL | NULL | NULL |<br />
| 8 | network | 2 | NULL | NULL | NULL |<br />
| 9 | network | 2 | NULL | NULL | NULL |<br />
| 10 | network | 2 | NULL | NULL | NULL |<br />
| 11 | network | 2 | NULL | NULL | NULL |<br />
+------------+------------+----------------+-------+---------+----------+<br />
11 rows in set (0.00 sec)<br />
delete from resources where resourceID =1;</pre></div>Supporthttps://wiki.jcihosting.com/index.php?title=DRAC/RMM&diff=1059DRAC/RMM2013-03-12T00:15:41Z<p>Support: /* RMM */</p>
<hr />
<div>= DRAC Notes =<br />
<br />
A list of all the DRAC cards and their IPs can be seen in the [[Private_IP_Mapping|ipmap]]<br />
<br />
To ssh in, login as root.<br />
<br />
Common commands:<br />
racadm serveraction hardreset<br />
performs a reset<br />
<br />
racadm serveraction powerdown<br />
power off server<br />
<br />
racadm serveraction powerup<br />
power on server<br />
<br />
racadm racreset<br />
reset the DRAC card (if kvm console becomes frozen)<br />
<br />
<br />
To get virtual media plugin to install, add site to trusted list and make security low<br />
<br />
Installing openmanage on CEOS4:<br />
<br />
Mount install cd from local drive<br />
<br />
<pre>mount /dev/scd0 /mnt<br />
cd /mnt/srvadmin/linux/RPMS/RHEL4<br />
cd /mnt/srvadmin/linux/supportscripts/<br />
./srvadmin-install.sh --express</pre><br />
<br />
Copy linux to disk<br />
Edit the script to make it think the os is RHEL4<br />
<br />
BIOS settings to enable serial:<br />
Serial comm.: <tt>on with cons. Redir via com1</tt><br />
Failsafe: <tt>115200</tt><br />
Redir after boot: <tt>enabled</tt><br />
<br />
To boot to usb, switch hd sequence in bios. <br />
<br />
After removing usb it will have to reorder boot seq to get virtual cd back<br />
<br />
Can’t get floppy/iso to boot- need to use thumb drive<br />
<br />
Latest f/w as of 2/16/09:<br />
<pre>Bios: 2.5.0<br />
Perc 6/i: 6.1.1-0047<br />
Drac: 1.4<br />
BMC: 2.37</pre><br />
<br />
= DRAC setup =<br />
<br />
Here's the settings we use when we initially setup a new DRAC.<br />
<br />
From within the DRAC web gui:<br />
<br />
Remote Access -> config -> network: enter DNS, DNS DRAC name=jailX, DNS domain=johncompanies.com<br><br />
Remote Access -> config -> services: disable SNMP<br><br />
System -> Alert mgmt -> platform events: enable<br><br />
System -> Alert mgmt -> email alert settings: smtp=69.55.230.2, add email alert 1 to support@jc, desc: jailX drac hardware alert<br><br />
<br />
= RMM =<br />
<br />
the user/ip/pass can be set via the BIOS. If you can't here's how you'd do that manually:<br />
<br />
boot to a usb drive that contains <tt>syscfg</tt> utility<br />
<br />
<pre>fs0:<br />
cd intel\mgmt\syscfg<br />
syscfg -le 3 static IP NM<br />
syscfg -lc 3 12 GW<br />
syscfg /u 2 "root" "pass"<br />
syscfg /ue 2 enable 3<br />
<br />
syscfg /d user 2 3<br />
syscfg /d lan 3</pre></div>Supporthttps://wiki.jcihosting.com/index.php?title=DRAC/RMM&diff=1058DRAC/RMM2013-03-12T00:14:21Z<p>Support: /* DRAC */</p>
<hr />
<div>= DRAC Notes =<br />
<br />
A list of all the DRAC cards and their IPs can be seen in the [[Private_IP_Mapping|ipmap]]<br />
<br />
To ssh in, login as root.<br />
<br />
Common commands:<br />
racadm serveraction hardreset<br />
performs a reset<br />
<br />
racadm serveraction powerdown<br />
power off server<br />
<br />
racadm serveraction powerup<br />
power on server<br />
<br />
racadm racreset<br />
reset the DRAC card (if kvm console becomes frozen)<br />
<br />
<br />
To get virtual media plugin to install, add site to trusted list and make security low<br />
<br />
Installing openmanage on CEOS4:<br />
<br />
Mount install cd from local drive<br />
<br />
<pre>mount /dev/scd0 /mnt<br />
cd /mnt/srvadmin/linux/RPMS/RHEL4<br />
cd /mnt/srvadmin/linux/supportscripts/<br />
./srvadmin-install.sh --express</pre><br />
<br />
Copy linux to disk<br />
Edit the script to make it think the os is RHEL4<br />
<br />
BIOS settings to enable serial:<br />
Serial comm.: <tt>on with cons. Redir via com1</tt><br />
Failsafe: <tt>115200</tt><br />
Redir after boot: <tt>enabled</tt><br />
<br />
To boot to usb, switch hd sequence in bios. <br />
<br />
After removing usb it will have to reorder boot seq to get virtual cd back<br />
<br />
Can’t get floppy/iso to boot- need to use thumb drive<br />
<br />
Latest f/w as of 2/16/09:<br />
<pre>Bios: 2.5.0<br />
Perc 6/i: 6.1.1-0047<br />
Drac: 1.4<br />
BMC: 2.37</pre><br />
<br />
= DRAC setup =<br />
<br />
Here's the settings we use when we initially setup a new DRAC.<br />
<br />
From within the DRAC web gui:<br />
<br />
Remote Access -> config -> network: enter DNS, DNS DRAC name=jailX, DNS domain=johncompanies.com<br><br />
Remote Access -> config -> services: disable SNMP<br><br />
System -> Alert mgmt -> platform events: enable<br><br />
System -> Alert mgmt -> email alert settings: smtp=69.55.230.2, add email alert 1 to support@jc, desc: jailX drac hardware alert<br><br />
<br />
= RMM =</div>Supporthttps://wiki.jcihosting.com/index.php?title=Virtuozzo_/_Linux_Reference&diff=1057Virtuozzo / Linux Reference2013-03-12T00:13:17Z<p>Support: /* Installing OS via net */</p>
<hr />
<div>= Updating VZ =<br />
<br />
Update Virtuozzo to version 4.0.0<br />
Apply minor updates for Virtuozzo 3.0.0<br />
<br />
...<br />
When you get to the last screen, you will be given an option to <br />
<br />
( ) Automatically reboot after update<br />
(*) Reboot later manually<br />
<br />
make sure you choose to reboot later.<br />
<br />
= Migrating Templates =<br />
One nice feature VZ offers is the ability to run a virtual environment (VE or CT as it's now called) based on a particular ditribution on a host which may or may not share the same distribution. For instance, you could run a CT running Ubuntu while the host machine (Hardware Node or HN) is running CentOS. This is accomplished via the use of OS templates. As long as the HN contains the OS template on which a particular CT relies, it will run there. As such, if you want to move a CT from one HN to another, you will need to make sure the OS template exists there.<br />
<br />
Modern versions of VZ (>= 4.x) will check for and automatically move the OS template over to the host as you're (vz)migrating the CT over to a new HN. However we like to make sure the template is in place prior to moving the CT. <br />
<br />
The first step is to see if the template exists on the host. To see OS templates installed:<br />
<br />
2.x (shows OS and application templates):<br />
<pre># vzpkgls<br />
mod_ssl-rh9 20040624<br />
mysql-rh9 20030814 20050412<br />
openwebmail-rh9 20050427<br />
php-rh9 20050506<br />
phpBB-rh9 20041229<br />
postgresql-rh9 20050719<br />
sl-webalizer-rh9 20040119<br />
spamassassin-rh9 20040127<br />
tomcat-rh9 20040524<br />
usermin-rh9 20040116</pre><br />
<br />
>= 3.x (shows just OS templates, run without -O to see application as well):<br />
<pre># vzpkg list -O<br />
ubuntu-10.04-x86 2010-06-01 11:39:05<br />
ubuntu-11.04-x86 2011-06-22 14:23:59<br />
ubuntu-9.10-x86 2010-03-11 17:39:51<br />
centos-5-x86 2010-03-11 17:12:27<br />
debian-5.0-x86 2010-03-11 17:33:34</pre><br />
<br />
<br />
All template files are located:<br />
<br />
<pre># ls /vz/template/<br />
ColdFusion-fc1 jre-fc1 openwebmail-fc1 sl-webalizer-fc1<br />
ColdFusion-fc2 jre-fc2 openwebmail-fc2 sl-webalizer-fc2<br />
ColdFusion-rh9 jre-rh9 openwebmail-rh9 sl-webalizer-rh9<br />
PostNuke-fc1 jre-suse92 openwebmail-suse92 spamassassin-fc1<br />
PostNuke-fc2 jrun php-deb31 spamassassin-fc2<br />
PostNuke-rh9 mailman-deb31 php-fc1 spamassassin-rh9<br />
analog-fc1 mailman-fc1 php-fc2 spamassassin-suse92<br />
analog-fc2 mailman-fc2 php-rh9 suse-9.2<br />
analog-rh9 mailman-rh9 php-suse92 tomcat-fc1<br />
awstats-fc1 mailman-suse92 phpBB-fc1 tomcat-fc2<br />
awstats-rh9 mod_frontpage-fc1 phpBB-fc2 tomcat-rh9<br />
bbClone-fc1 mod_frontpage-fc2 phpBB-rh9 tomcat-suse92<br />
bbClone-fc2 mod_frontpage-rh9 phpmyadmin-deb31 urchin-fc1<br />
bbClone-rh9 mod_frontpage-suse92 postgresql-deb31 usermin-fc1<br />
conf mod_perl-deb31 postgresql-fc1 usermin-fc2<br />
debian-3.1 mod_perl-fc1 postgresql-fc2 usermin-rh9<br />
devel-deb31 mod_perl-fc2 postgresql-rh9 uw-imap-fc1<br />
devel-fc2 mod_perl-rh9 postgresql-suse92 uw-imap-fc2<br />
devel-suse92 mod_perl-suse92 proftpd-deb31 uw-imap-rh9<br />
fedora-core-1 mod_ssl-fc1 proftpd-fc1 vzredhat-7.3<br />
fedora-core-2 mod_ssl-fc2 proftpd-fc2 vzredhat-8.0<br />
jdk-deb31 mod_ssl-rh9 proftpd-rh9 webalizer-suse92<br />
jdk-fc1 mysql-deb31 proftpd-suse92 webmin-fc1<br />
jdk-fc2 mysql-fc1 redhat-7.2 webmin-fc2<br />
jdk-rh9 mysql-fc2 redhat-9 webmin-rh9<br />
jdk-suse92 mysql-rh9 redhat-as3-minimal<br />
jre-deb31 mysql-suse92 redhat-devel-9<br />
</pre><br />
<br />
What we see here are the base OS templates: <tt>redhat-9, fedora-core-1</tt> and supporting application templates: <tt>postgresql-rh9, mailman-rh9, jdk-fc1, mod_ssl-fc1</tt>. So if you wanted to copy over all of redhat 9 you'd need to copy the contents of:<br />
<pre># ls -d /vz/template/*rh9*<br />
/vz/template/ColdFusion-rh9 /vz/template/mod_frontpage-rh9 /vz/template/proftpd-rh9<br />
/vz/template/PostNuke-rh9 /vz/template/mod_perl-rh9 /vz/template/sl-webalizer-rh9<br />
/vz/template/analog-rh9 /vz/template/mod_ssl-rh9 /vz/template/spamassassin-rh9<br />
/vz/template/awstats-rh9 /vz/template/mysql-rh9 /vz/template/tomcat-rh9<br />
/vz/template/bbClone-rh9 /vz/template/openwebmail-rh9 /vz/template/usermin-rh9<br />
/vz/template/jdk-rh9 /vz/template/php-rh9 /vz/template/uw-imap-rh9<br />
/vz/template/jre-rh9 /vz/template/phpBB-rh9 /vz/template/webmin-rh9<br />
/vz/template/mailman-rh9 /vz/template/postgresql-rh9<br />
<br />
# ls -d /vz/template/*redhat-9*<br />
/vz/template/redhat-9<br />
</pre><br />
<br />
To get a template from one HN to another, it simply needs to be rsync'd over to the target HN. It's rsync'd over in this fashion:<br />
<br />
rsync -av -e ssh /vz/template/redhat-9 root@10.1.4.65:/vz/template<br />
rsync -av -e ssh /vz/template/*rh9 root@10.1.4.65:/vz/template<br />
<br />
Newer (>= 3.x) VZ employs "EZ" templates which are organized a little differently:<br />
<br />
<pre># ls /vz/template/<br />
cache CmdTool.log debian redhat-as3-minimal-20080630.tar.gz vc<br />
centos conf redhat-as3-minimal ubuntu<br />
<br />
# ls /vz/template/ubuntu/<br />
10.04 11.04 9.10<br />
# ls /vz/template/ubuntu/10.04/<br />
x86<br />
# ls /vz/template/ubuntu/10.04/x86/<br />
aap_1.091-1_all<br />
aap-doc_1.091-1_all<br />
adduser_3.112ubuntu1_all<br />
anacron_2.3-13.1ubuntu11_i386<br />
apache2_2.2.14-5ubuntu8.2_i386<br />
apache2_2.2.14-5ubuntu8.4_i386<br />
apache2_2.2.14-5ubuntu8.6_i386<br />
apache2_2.2.14-5ubuntu8.7_i386<br />
apache2_2.2.14-5ubuntu8.8_i386<br />
apache2_2.2.14-5ubuntu8.9_i386<br />
<...snip><br />
<br />
# ls /vz/template/cache/<br />
centos-5-x86.tar.gz suse-11.3-x86.tar.gz ubuntu-9.10-x86.tar.gz<br />
debian-5.0-x86.tar.gz ubuntu-10.04-x86.tar.gz<br />
fedora-core-14-x86.tar.gz ubuntu-11.04-x86.tar.gz<br />
</pre><br />
<br />
So what we see is the entire OS and all applications are in 1 directory, and organized by distribution and release version. So to move Ubuntu 10.04:<br />
<br />
rsync -av -e ssh /vz/template/ubuntu/10.04 root@10.1.4.69:/vz/template/ubuntu<br />
<br />
and you also need to move the cache:<br />
<br />
rsync -av -e ssh /vz/template/cache/ubuntu-10.04-x86.tar.gz root@10.1.4.69:/vz/template/cache/<br />
<br />
Once the transfer is complete, you will be able to run <tt>vzpkgls</tt> or <tt>vzpkg list -O</tt> and see the template(s) listed on the target HN.<br />
<br />
So far, this all supposes template doesn't exist on the target- very simple, just copy it over. If the template exists already on the target HN, this requires closer inspection. With older templates, simply moving over the templates would be the end of it- you see the version listed when you do a <tt>vzplgls</tt>:<br />
<br />
mysql-rh9 20030814 20050412<br />
<br />
this means there are 2 versions of the mysql package for RH9: 20030814 and 20050412<br />
As an aside, you can't copy over just 1 of them, but if you rsync'd over the <tt>mysql-rh9</tt> directory, you'd see 20030814 and 20050412 on the target HN. As long as the versions match up, you're good to go.<br />
<br />
With EZ templates, the versions are not as easy to decipher. When you look at the <tt>vzpkg list</tt> output, that simply tells you when a cache was created. A cache is simply a snapshot of all the data needed for the latest versions of the OS and it's applications at the time the cache was created. It is a date, and thus tells you nothing about the versions within. So for instance, if you had a cache for Ubuntu 10.04 from 2007 and you ran <tt>vzpkg create cache ubuntu-10.04-x86</tt> it would re-download the latest version of apache, mysql and so on. And any ''subsequent'' ubuntu 10.04 CT's created thereafter would use those newer versions, whereas older CT's based on 10.04 would use older templates. You can see how this works by looking at the files under:<br />
<br />
<pre># ls /vz/template/ubuntu/10.04/x86/<br />
aap_1.091-1_all<br />
aap-doc_1.091-1_all<br />
adduser_3.112ubuntu1_all<br />
anacron_2.3-13.1ubuntu11_i386<br />
apache2_2.2.14-5ubuntu8.2_i386<br />
apache2_2.2.14-5ubuntu8.4_i386<br />
apache2_2.2.14-5ubuntu8.6_i386<br />
apache2_2.2.14-5ubuntu8.7_i386<br />
apache2_2.2.14-5ubuntu8.8_i386<br />
apache2_2.2.14-5ubuntu8.9_i386<br />
<...snip><br />
</pre><br />
<br />
You can see that this template has in it 6 different versions of apache2. This means that this template was merged with other 10.04 templates and/or it has been cached a few times, each time a new apache was available. The unfortunate thing is it's almost impossible to know which CT's are using which version of apache so it's hard to try to prune this down and remove unwanted/unneeded applications.<br />
<br />
So, if you see another HN has Ubuntu 10.04 on it, you need to see/confirm that it has all the exact files your CT needs. You can run a test rsync to see what ''would'' be transferred (what's missing):<br />
<br />
rsync -avn --stats -e ssh /vz/template/ubuntu/10.04 root@10.1.4.69:/vz/template/ubuntu<br />
<br />
Based on the amount of data you get back from the rsync, you will be able to decide whether or not to remove the -n and allow the full transfer to go through. The downside to doing the transfer is the situation described above- you've permanently joined these templates and now you may have 10 versions of apache on the target HN. There's one other potential pitfall to this kind of merging- it will also potentially copy over shared files, for which there is no particular version, most of which are in <tt>/vz/template/ubuntu/10.04/x86/config</tt>: <br />
<br />
cat /vz/template/ubuntu/10.04/x86/config/os/default/repositories<br />
/vz/template/ubuntu/10.04/x86/config/os/default/repositories<br />
<br />
Here are some specific things to look out for. Case: copying centos-5 on virt19 to virt12. We dumped the rsync output to a file so we could inspect:<br />
<br />
[root@virt19 /vz/template]# rsync -an -e ssh /vz/template/centos/ root@10.1.4.62:/vz/template/centos/ > v12_c5<br />
<br />
(note, that can be dangerous, better to add on full path <tt>/vz/template/centos/5</tt>)<br />
<br />
So in the output file we see things like:<br />
<br />
<pre><br />
x86/base0/cachecookie<br />
x86/base0/primary.xml.gz <br />
x86/base0/primary.xml.gz.sqlite <br />
x86/base0/repomd.xml <br />
x86/base1/cachecookie <br />
x86/base1/filelists.xml.gz <br />
x86/base1/filelists.xml.gz.sqlite <br />
x86/base1/primary.xml.gz <br />
x86/base1/primary.xml.gz.sqlite <br />
x86/base1/repomd.xml<br />
x86/base2/cachecookie<br />
</pre><br />
<br />
Which make us nervous cause those are not versioned files, i.e.:<br />
5/x86/MAKEDEV-3.23-1.2.i386/usr/sbin/mksock<br />
<br />
So those files will replace existing files on v12 rather than the case of the latter where it's likely being added. So caution should be given and deference to whichever version is newer (which rsync should take care of, but that will depend on the date each file was created and perhaps not the actual data within).<br />
<br />
If we look further in the file we see:<br />
<br />
x86/psa-appvault-moodle-1.8-8202920080409011254.noarch/usr/local/psa/var/cgitory/moodle-1.<br />
9/htdocs/lib/editor/htmlarea/plugins/TableOperations/img/cell-split.gif<br />
<br />
and other files looking like:<br />
5/x86/plesk-base-9.0.1-cos5.build90090127.18.i586/etc/sw/keys/info<br />
<br />
Which means plesk is here. We don't need plesk on v12 (the CT moving over doesn't use it) so we rerun the rsync and exclude it:<br />
<br />
[root@virt19 /vz/template]# rsync -an -e ssh --exclude="plesk*" --exclude="psa*" /vz/template/centos/ root@10.1.4.62:/vz/template/centos/ > v12_c5<br />
<br />
and now it's much smaller:<br />
<br />
[root@virt19 /vz/template]# cat v12_c5|wc -l<br />
97104<br />
<br />
Before running with excludes it was:<br />
[root@virt19 /vz/template]# cat v12_c5|wc -l <br />
238238<br />
<br />
So again, look at what you're doing. Generally, combining templates is fine however.<br />
<br />
If you're happy with the merge, run the rsync again without the "-n" to let it actually do the transfer.<br />
<br />
Once done, don't forget to add the new template to the HN in Management -> Reference -> Templates<br />
<br />
= getting help =<br />
<br />
= vzup2date =<br />
TODO <br />
= Adding new OS/application templates =<br />
<br />
Virtuozzo will lag a bit behind the initial release of an OS, perhaps by a month or more. Further, a newly-released OS may be available, but there may not be many/any application templates available initially. It pays to wait a bit.<br />
<br />
A template is easily installed via [[#vzup2date|vzup2date]]:<br />
<br />
vzup2date -z<br />
<br />
You will be given a list of OS's to install. Select an OS, then customize that selection by checking/unchecking the application templates we do/don't want to include/offer.<br />
<br />
Generally, here are the app templates we include/offer:<br />
<pre>cyrus-imap<br />
devel<br />
imp<br />
jre<br />
jsdk<br />
mailman<br />
mod_perl<br />
mod_ssl<br />
mysql<br />
php<br />
phpmyadmin<br />
phppgadmin<br />
postgresql<br />
proftpd<br />
spamassassin<br />
squirrelmail<br />
tomcat<br />
vsftpd<br />
</pre><br />
<br />
We do not (any longer) offer webalizer or analog.<br />
<br />
After selecting and installing the OS and apps, you should ensure it's installed:<br />
<br />
vzpkg list -O<br />
<br />
Your new OS (ex. fedora-core-13-x86_64) will be listed, without a corresponding cache date:<br />
<pre><br />
[root@virt13 /vzconf]# vzpkg list -O<br />
centos-6-x86 2011-07-22 13:24:41<br />
centos-6-x86_64 2012-03-09 20:42:22<br />
centos-5-x86 2009-07-27 16:58:24<br />
centos-5-x86_64 2012-03-09 22:38:53<br />
ubuntu-11.10-x86_64 2012-03-01 23:13:33<br />
ubuntu-9.04-x86 2009-06-02 11:32:39<br />
ubuntu-12.04-x86_64 2012-05-29 13:50:50<br />
ubuntu-8.04-x86 2008-09-17 21:27:20<br />
ubuntu-8.10-x86 2009-03-13 13:58:57<br />
ubuntu-11.04-x86 2011-12-05 11:15:46<br />
ubuntu-7.04-x86 2008-09-24 16:42:50<br />
redhat-el6-x86_64<br />
fedora-core-13-x86_64 <br />
fedora-core-12-x86 2010-02-06 13:24:32<br />
fedora-core-15-x86 2011-12-05 11:36:43<br />
fedora-core-11-x86 2009-10-01 16:30:59<br />
fedora-core-14-x86<br />
fedora-core-16-x86_64 2012-03-13 11:40:23<br />
fedora-core-9-x86 2008-11-29 10:52:18<br />
debian-6.0-x86 2011-07-29 16:00:35<br />
debian-6.0-x86_64 2012-03-12 19:53:33<br />
debian-5.0-x86 2009-03-12 12:39:11<br />
</pre><br />
<br />
You can also confirm the apps installed:<br />
<pre>[root@virt13 /vzconf]# vzpkg list fedora-core-13-x86_64<br />
fedora-core-13-x86_64 2013-03-08 11:39:44<br />
fedora-core-13-x86_64 devel<br />
fedora-core-13-x86_64 php<br />
fedora-core-13-x86_64 proftpd<br />
fedora-core-13-x86_64 postgresql<br />
fedora-core-13-x86_64 phpmyadmin<br />
fedora-core-13-x86_64 mysql<br />
</pre><br />
<br />
Before you can use the OS you must cache it. To create the cache run:<br />
vzpkg cache [OS]<br />
<br />
ex: <tt>vzpkg cache fedora-core-13-x86_64</tt><br />
<br />
Now it's ready to use. In order to be able to use it with the [[VPS_Management#vm|vm]] command, you must create a file:<br />
jctmpl-[OS]<br />
<br />
ex: <tt>jctmpl-fedora-core-13-x86_64</tt><br />
<br />
In it, on line 1 we place the OS, followed by the app names, ex:<br />
<pre>more jctmpl-fedora-core-13-x86_64<br />
fedora-core-13-x86_64<br />
postgresql<br />
jsdk<br />
mod_ssl<br />
phpmyadmin<br />
mod_perl<br />
tomcat<br />
devel<br />
php<br />
mailman<br />
cyrus-imap<br />
squirrelmail<br />
proftpd<br />
mysql<br />
phppgadmin<br />
spamassassin<br />
</pre><br />
<br />
The packages will be installed in that order, so any pre-req's should be handled there.<br />
<br />
Now, when you run [[VPS_Management#vm|vm]] it will show you the newly-added OS and you may create a CT using it.<br />
<br />
The last few tasks are related to mgmt:<br />
<br />
1. add to mgmt->reference->templates (use the existing templates as an example). Once added, the new template will be included with the list of OS's available for the given HN.<br />
2. if this is going to be a new OS offered for ordering, it needs to be added to the order form and webpage.<br />
<br />
a. to get a list of applications and versions in the new OS, you should create a test CT, enter it and run <tt>dpkg -l|sort</tt> (or <tt>rpm -aq|sort</tt> in centos/fedora) and enter that list in the following places:<br><br />
<tt>/usr/local/www/jc_pub/[osname].html</tt><br><br />
<tt>/usr/local/www/signup/html/[osname].html</tt><br><br />
Note, you will have to move the most recent OS data from <tt>/usr/local/www/signup/html/[osname].html</tt> into the corresponding <tt>/usr/local/www/signup/html/[osname]_old.html</tt> updating the version links in both.<br />
<br />
b. to add the new OS to the order form, you will need to update <tt>/usr/local/www/signup/html/step1.html</tt> in several places (for regular and OSS orders) as well as updating the templateID which you can get from mgmt->reference->templates (or [[Management_System_/_Public_Website_/_Signup_/_Account_Manager#jc:ref_templates|jc:ref_templates]])<br />
<br />
<br />
<br />
Old method to obtain the templates:<br />
<br />
Go to http://www.parallels.com/en/virtuozzo/templates/catalog/<br />
<br />
Download the RPM's and install them<br />
<br />
vzpkg list<br />
to see the new packages/os templates<br />
<br />
vzpkg create cache "OS_EZ_template_name"<br />
to create a cache<br />
<br />
<br />
= Update OS template =<br />
<br />
Installation and update of the new version of OS template will not affect the existing containers. It will affect only the new containers, created after this update operation. In terms of fixing "vzctl enter" error and SSH connectivity to Ubuntu based containers, it is necessary to recreate the template and not to update it, so the commands to run are:<br />
<br />
<pre>rpm -Uhv ubuntu-8.04-x86-ez-4.0.0-9.swsoft.noarch.rpm<br />
vzpkg clean ubuntu-8.04-x86<br />
vzpkg remove cache ubuntu-8.04-x86<br />
vzpkg create cache ubuntu-8.04-x86</pre><br />
<br />
<br />
= Installing new SSL cert on VZCP =<br />
<br />
<pre>openssl req -days 1999 -new -x509 -nodes -out server.crt -keyout server.key<br />
<br />
US<br />
CA<br />
San Diego<br />
johncompanies.com<br />
johncompanies.com<br />
virt15ohncompanies.com<br />
linux@johncompanies.com<br />
<br />
cp -p /etc/httpd/conf/ssl.crt/server.crt /etc/httpd/conf/ssl.crt/server.crt.bak<br />
cp -p /etc/httpd/conf/ssl.key/server.key /etc/httpd/conf/ssl.key/server.key.bak<br />
cp server.crt /etc/httpd/conf/ssl.crt/server.crt<br />
cp server.key /etc/httpd/conf/ssl.key/server.key<br />
/etc/init.d/vzcp restart</pre><br />
<br />
= Moving virt (drives) to new hardware =<br />
<br />
If you have a case where a virt is dead (due to hardware failure), you can move the drives to a new hardware setup.<br />
Here are the steps and things to look for:<br />
<br />
Move the drives to the new box. Most likely, the drives will be recognized and there will be no problems booting up. On Dell PERC 5/6 cards you will need to import the new "foreign" volume via the raid BIOS.<br />
<br />
If the hardware (motherboard) is different, it’s possible the nic’s won’t be recognized. Look at /etc/modules.conf the ethernet drivers are either:<br />
<br />
<pre>alias eth0 tg3<br />
alias eth1 tg3</pre><br />
(supermicro)<br />
<br />
Or<br />
<br />
<pre>alias eth0 e100<br />
alias eth1 e1000</pre><br />
(intel)<br />
<br />
Or<br />
<br />
<pre>alias eth0 bnx2<br />
alias eth1 bnx2</pre><br />
(dell 29500)<br />
<br />
Also, you may see eth0 and eth1 swapped so if after confirming the eth devices are present (may require a reboot), then you may still need to swap the green/yellow network cables in the back to get them on the right port.<br />
<br />
VZ will not run (for long) on the new hardware, so you will need to get the license reissued. You may create/download a temporary license via their website for the interim.<br />
<br />
= Updating kernel on virt =<br />
<br />
We now use [[#vzup2date|vzup2date]] to update systems and kernels. What follows is what we used to do when we had to download kernels and install manually on old systems:<br />
<br />
<br />
1. install kernel and modules<br />
<pre># rpm -ivh vzkernel-enterprise-2.4.20-021stab028.12.777.i686.rpm \<br />
vzmodules-enterprise-2.4.20-021stab028.12.swsoft.i686.rpm<br />
vzkernel-smp ##################################################<br />
vzmodules-smp ##################################################</pre><br />
<br />
DO NOT USE the "rpm -Uhv" command to install the kernel, otherwise all the previously installed kernels may be removed from your system. <br />
<br />
2. update lilo/grub<br />
<br />
Lilo:<br />
<br />
vi /etc/lilo.conf<br />
<br />
rename old Virtuozzo kernel label from "linux+virtuozzo" to "virtuozzo_old"<br />
and add a new entry pointing to the newly installed virtuozzo kernel image.<br />
For example:<br />
<br />
<pre>prompt<br />
timeout=50<br />
default=linux+virtuozzo<br />
append="debug panic=5 console=tty console=ttyS0,38400"<br />
boot=/dev/sda<br />
map=/boot/map<br />
install=/boot/boot.b<br />
message=/boot/message<br />
linear<br />
<br />
image=/boot/vmlinuz-2.4.18-3smp<br />
label=linux<br />
initrd=/boot/initrd-2.4.18-3smp.img<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.18-3<br />
label=linux-up<br />
initrd=/boot/initrd-2.4.18-3.img<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-020stab009.5.777-enterprise<br />
label=2.4.20-9.5.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-020stab009.8.777-enterprise<br />
label=2.4.20-9.8.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-020stab009.21.777-enterprise<br />
label=2.4.20-9.21.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-020stab009.24.777-enterprise<br />
label=2.4.20-9.24.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-021stab028.3.777-enterprise<br />
label=2.4.20-28.3.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-021stab028.5.777-enterprise<br />
label=old_linux+vz<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-021stab028.12.777-enterprise<br />
label=linux+virtuozzo<br />
read-only<br />
root=/dev/sda1</pre><br />
<br />
With this configuration, the new kernel is the default kernel to use at boot. The original kernel entry has been retained with the label "old_linux+vz ".<br />
<br />
Grub:<br />
<br />
vi /boot/grub /menu.lst<br />
<br />
Add new entry at the top.<br />
For example:<br />
<br />
<pre>serial --unit=0 --speed=38400<br />
terminal --timeout=10 serial console<br />
default=0<br />
timeout=10<br />
splashimage=(hd0,0)/boot/grub/splash.xpm.gz<br />
title Virtuozzo 2.6.1 (2.4.20-021stab028.12.777-enterprise)<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-22.4.20-021stab028.12.777-enterprise root=/dev/sda1 console=tty0 \ console=ttyS0,38400<br />
title Virtuozzo 2.6.1 (2.4.20-021stab028.3.777-enterprise)<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.20-021stab028.3.777-enterprise root=/dev/sda1 console=tty0 \ console=ttyS0,38400<br />
title Red Hat Linux (2.4.18-3smp)<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.18-3smp ro root=/dev/sda1<br />
initrd /boot/initrd-2.4.18-3smp.img<br />
title Red Hat Linux-up (2.4.18-3)<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.18-3 ro root=/dev/sda1<br />
initrd /boot/initrd-2.4.18-3.img<br />
title Linux with Virtuozzo 2.5<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.20-020stab009.3.777-smp ro root=/dev/sda1<br />
title vz-enterpriseo<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.20-020stab009.21.777-enterprise ro root=/dev/sda1<br />
title vz-enterprise<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.20-020stab009.24.777-enterprise ro root=/dev/sda1 \ console=tty0 console=ttyS0,38400</pre><br />
<br />
3. run the lilo (if using lilo)<br />
<br />
<pre># lilo<br />
Added linux<br />
Added linux-up<br />
Added virtuozzo_old<br />
Added linux+virtuozzo *</pre><br />
<br />
4. reboot the server<br />
shutdown -r 23:05<br />
<br />
when it comes time for the shutdown, run [[VPS_Management#stopvirt|stopvirt]] to get things shut down faster<br />
<br />
<br />
= Remaking service VE (<=3.x only) =<br />
<br />
Occasionally the control panel for VEs (aka VZPP) will stop working and you just need to reinstall the service VE (which runs the control panels for all VEs on the HN). Here's how that's done :<br />
<br />
<pre>vzctl stop 1<br />
ipdel 1 69.55.234.152<br />
vzmlocal 1:2<br />
vzctl set 2 --save --onboot no<br />
vzsveinstall -t redhat-as3-minimal -d /root/Rel261/HW/RPMS -s 69.55.234.152 --skip-client</pre><br />
<br />
on 3.0:<br />
<br />
<pre>vzsveinstall -t redhat-as3-minimal -d /vz/Rel300/HW/RPMS -s 69.55.229.151 --skip-client<br />
<br />
vzctl stop 1<br />
vzctl mount 1<br />
vzctl mount 2<br />
/bin/cp -aRf /vz/root/2/home/vzagent0 /vz/root/1/home<br />
/bin/cp -aRf /vz/root/2/var/lib/mysql/VZADB /vz/root/1/var/lib/mysql<br />
/bin/cp -af /vz/root/2/etc/shadow /vz/root/1/etc/<br />
/bin/cp -aRf /vz/root/2/var/vzcp/static/vz/skins/ /vz/root/1/var/vzcp/static/vz<br />
/bin/cp -af /vz/root/2/etc/vzcp/pp/menu.xml /vz/root/1/etc/vzcp/pp/<br />
/bin/cp -af /vz/root/2/etc/vzcp/pp/dashboard.xml /vz/root/1/etc/vzcp/pp/<br />
<br />
vzctl umount 2<br />
vzctl umount 1<br />
vzctl start 1</pre><br />
<br />
<br />
= GRUB: boot once to a new kernel =<br />
<br />
To use a different kernel for the next reboot only:<br />
<pre>grub --no-floppy<br />
savedefault --default=N --once<br />
(where N is the number of the kernel you want to boot once)<br />
quit</pre><br />
<br />
Reboot your machine. It will boot using kernel "N".<br />
<br />
grub-reboot N<br />
will do essentially the same thing and prompt to reboot the machine<br />
<br />
<br />
= Recovering from GRUB failure =<br />
<br />
(example from DPE 2950)<br />
<br />
Boot to CEOS4 server CD (original install cd). Did this example with iso mounted via drac<br />
<br />
Boot: linux rescue<br />
<br />
<br />
= Make/edit an ISO on Linux =<br />
<br />
<pre>mount file.iso /mnt/iso/ -t iso9660 -o ro,loop=/dev/loop0<br />
mkdir /tmp/iso<br />
cp -aR /mnt/iso/* /tmp/iso/<br />
umount /mnt/iso/<br />
mkisofs -iso-level 2 -o /tmp/new.iso /tmp/iso/<br />
rm -fr /tmp/iso</pre><br />
<br />
To mount it:<br />
<br />
mount -t vfat -o loop /tmp/3wfirmware.iso /mnt/usb/<br />
<br />
http://www.damnsmalllinux.org/wiki/index.php/Installing_to_a_USB_Flash_Drive<br />
<br />
= Mount/edit USB drive on Linux =<br />
<br />
mount -t vfat -o loop /dev/sdc1 /mnt/usb/<br />
<br />
This example is for our floating usb drive on virt16. You should edit the device as necessary on other hardware.<br />
<br />
<br />
= Installing OS via net =<br />
<br />
Centos 5.1:<br><br />
http site name: <tt>vault.centos.org</tt><br><br />
dir: <tt>/5.1/os/i386/</tt><br />
<br />
Centos 5.4:<br><br />
http site name: <tt>mirrors.kernel.org</tt><br><br />
dir: <tt>/centos/5.4/os/i386/</tt><br><br />
or<br><br />
<tt>/centos/5.4/os/x86_64/</tt><br><br />
or<br><br />
<tt>http://mirrors.usc.edu/pub/linux/distributions/centos/5.4/os/i386/</tt><br><br />
<br />
<br />
Centos 6.2:<br><br />
<tt>http://mirrors.usc.edu/pub/linux/distributions/centos/6.2/os/x86_64/</tt><br><br />
<br />
<br />
= Single user mode (Ubuntu) =<br />
<br />
add to end of kernel line: <tt>init=/bin/bash rw</tt><br />
<br />
<br />
FC9:<br><br />
<tt>http://linux.nssl.noaa.gov/fedora/linux/releases/9/Fedora/i386/os/</tt><br><br />
<br />
<br />
FC10:<br><br />
<tt>http://linux.nssl.noaa.gov/fedora/linux/releases/10/Fedora/i386/os/</tt><br></div>Supporthttps://wiki.jcihosting.com/index.php?title=Virtuozzo_/_Linux_Reference&diff=1056Virtuozzo / Linux Reference2013-03-12T00:12:41Z<p>Support: /* Installing OS via net */</p>
<hr />
<div>= Updating VZ =<br />
<br />
Update Virtuozzo to version 4.0.0<br />
Apply minor updates for Virtuozzo 3.0.0<br />
<br />
...<br />
When you get to the last screen, you will be given an option to <br />
<br />
( ) Automatically reboot after update<br />
(*) Reboot later manually<br />
<br />
make sure you choose to reboot later.<br />
<br />
= Migrating Templates =<br />
One nice feature VZ offers is the ability to run a virtual environment (VE or CT as it's now called) based on a particular ditribution on a host which may or may not share the same distribution. For instance, you could run a CT running Ubuntu while the host machine (Hardware Node or HN) is running CentOS. This is accomplished via the use of OS templates. As long as the HN contains the OS template on which a particular CT relies, it will run there. As such, if you want to move a CT from one HN to another, you will need to make sure the OS template exists there.<br />
<br />
Modern versions of VZ (>= 4.x) will check for and automatically move the OS template over to the host as you're (vz)migrating the CT over to a new HN. However we like to make sure the template is in place prior to moving the CT. <br />
<br />
The first step is to see if the template exists on the host. To see OS templates installed:<br />
<br />
2.x (shows OS and application templates):<br />
<pre># vzpkgls<br />
mod_ssl-rh9 20040624<br />
mysql-rh9 20030814 20050412<br />
openwebmail-rh9 20050427<br />
php-rh9 20050506<br />
phpBB-rh9 20041229<br />
postgresql-rh9 20050719<br />
sl-webalizer-rh9 20040119<br />
spamassassin-rh9 20040127<br />
tomcat-rh9 20040524<br />
usermin-rh9 20040116</pre><br />
<br />
>= 3.x (shows just OS templates, run without -O to see application as well):<br />
<pre># vzpkg list -O<br />
ubuntu-10.04-x86 2010-06-01 11:39:05<br />
ubuntu-11.04-x86 2011-06-22 14:23:59<br />
ubuntu-9.10-x86 2010-03-11 17:39:51<br />
centos-5-x86 2010-03-11 17:12:27<br />
debian-5.0-x86 2010-03-11 17:33:34</pre><br />
<br />
<br />
All template files are located:<br />
<br />
<pre># ls /vz/template/<br />
ColdFusion-fc1 jre-fc1 openwebmail-fc1 sl-webalizer-fc1<br />
ColdFusion-fc2 jre-fc2 openwebmail-fc2 sl-webalizer-fc2<br />
ColdFusion-rh9 jre-rh9 openwebmail-rh9 sl-webalizer-rh9<br />
PostNuke-fc1 jre-suse92 openwebmail-suse92 spamassassin-fc1<br />
PostNuke-fc2 jrun php-deb31 spamassassin-fc2<br />
PostNuke-rh9 mailman-deb31 php-fc1 spamassassin-rh9<br />
analog-fc1 mailman-fc1 php-fc2 spamassassin-suse92<br />
analog-fc2 mailman-fc2 php-rh9 suse-9.2<br />
analog-rh9 mailman-rh9 php-suse92 tomcat-fc1<br />
awstats-fc1 mailman-suse92 phpBB-fc1 tomcat-fc2<br />
awstats-rh9 mod_frontpage-fc1 phpBB-fc2 tomcat-rh9<br />
bbClone-fc1 mod_frontpage-fc2 phpBB-rh9 tomcat-suse92<br />
bbClone-fc2 mod_frontpage-rh9 phpmyadmin-deb31 urchin-fc1<br />
bbClone-rh9 mod_frontpage-suse92 postgresql-deb31 usermin-fc1<br />
conf mod_perl-deb31 postgresql-fc1 usermin-fc2<br />
debian-3.1 mod_perl-fc1 postgresql-fc2 usermin-rh9<br />
devel-deb31 mod_perl-fc2 postgresql-rh9 uw-imap-fc1<br />
devel-fc2 mod_perl-rh9 postgresql-suse92 uw-imap-fc2<br />
devel-suse92 mod_perl-suse92 proftpd-deb31 uw-imap-rh9<br />
fedora-core-1 mod_ssl-fc1 proftpd-fc1 vzredhat-7.3<br />
fedora-core-2 mod_ssl-fc2 proftpd-fc2 vzredhat-8.0<br />
jdk-deb31 mod_ssl-rh9 proftpd-rh9 webalizer-suse92<br />
jdk-fc1 mysql-deb31 proftpd-suse92 webmin-fc1<br />
jdk-fc2 mysql-fc1 redhat-7.2 webmin-fc2<br />
jdk-rh9 mysql-fc2 redhat-9 webmin-rh9<br />
jdk-suse92 mysql-rh9 redhat-as3-minimal<br />
jre-deb31 mysql-suse92 redhat-devel-9<br />
</pre><br />
<br />
What we see here are the base OS templates: <tt>redhat-9, fedora-core-1</tt> and supporting application templates: <tt>postgresql-rh9, mailman-rh9, jdk-fc1, mod_ssl-fc1</tt>. So if you wanted to copy over all of redhat 9 you'd need to copy the contents of:<br />
<pre># ls -d /vz/template/*rh9*<br />
/vz/template/ColdFusion-rh9 /vz/template/mod_frontpage-rh9 /vz/template/proftpd-rh9<br />
/vz/template/PostNuke-rh9 /vz/template/mod_perl-rh9 /vz/template/sl-webalizer-rh9<br />
/vz/template/analog-rh9 /vz/template/mod_ssl-rh9 /vz/template/spamassassin-rh9<br />
/vz/template/awstats-rh9 /vz/template/mysql-rh9 /vz/template/tomcat-rh9<br />
/vz/template/bbClone-rh9 /vz/template/openwebmail-rh9 /vz/template/usermin-rh9<br />
/vz/template/jdk-rh9 /vz/template/php-rh9 /vz/template/uw-imap-rh9<br />
/vz/template/jre-rh9 /vz/template/phpBB-rh9 /vz/template/webmin-rh9<br />
/vz/template/mailman-rh9 /vz/template/postgresql-rh9<br />
<br />
# ls -d /vz/template/*redhat-9*<br />
/vz/template/redhat-9<br />
</pre><br />
<br />
To get a template from one HN to another, it simply needs to be rsync'd over to the target HN. It's rsync'd over in this fashion:<br />
<br />
rsync -av -e ssh /vz/template/redhat-9 root@10.1.4.65:/vz/template<br />
rsync -av -e ssh /vz/template/*rh9 root@10.1.4.65:/vz/template<br />
<br />
Newer (>= 3.x) VZ employs "EZ" templates which are organized a little differently:<br />
<br />
<pre># ls /vz/template/<br />
cache CmdTool.log debian redhat-as3-minimal-20080630.tar.gz vc<br />
centos conf redhat-as3-minimal ubuntu<br />
<br />
# ls /vz/template/ubuntu/<br />
10.04 11.04 9.10<br />
# ls /vz/template/ubuntu/10.04/<br />
x86<br />
# ls /vz/template/ubuntu/10.04/x86/<br />
aap_1.091-1_all<br />
aap-doc_1.091-1_all<br />
adduser_3.112ubuntu1_all<br />
anacron_2.3-13.1ubuntu11_i386<br />
apache2_2.2.14-5ubuntu8.2_i386<br />
apache2_2.2.14-5ubuntu8.4_i386<br />
apache2_2.2.14-5ubuntu8.6_i386<br />
apache2_2.2.14-5ubuntu8.7_i386<br />
apache2_2.2.14-5ubuntu8.8_i386<br />
apache2_2.2.14-5ubuntu8.9_i386<br />
<...snip><br />
<br />
# ls /vz/template/cache/<br />
centos-5-x86.tar.gz suse-11.3-x86.tar.gz ubuntu-9.10-x86.tar.gz<br />
debian-5.0-x86.tar.gz ubuntu-10.04-x86.tar.gz<br />
fedora-core-14-x86.tar.gz ubuntu-11.04-x86.tar.gz<br />
</pre><br />
<br />
So what we see is the entire OS and all applications are in 1 directory, and organized by distribution and release version. So to move Ubuntu 10.04:<br />
<br />
rsync -av -e ssh /vz/template/ubuntu/10.04 root@10.1.4.69:/vz/template/ubuntu<br />
<br />
and you also need to move the cache:<br />
<br />
rsync -av -e ssh /vz/template/cache/ubuntu-10.04-x86.tar.gz root@10.1.4.69:/vz/template/cache/<br />
<br />
Once the transfer is complete, you will be able to run <tt>vzpkgls</tt> or <tt>vzpkg list -O</tt> and see the template(s) listed on the target HN.<br />
<br />
So far, this all supposes template doesn't exist on the target- very simple, just copy it over. If the template exists already on the target HN, this requires closer inspection. With older templates, simply moving over the templates would be the end of it- you see the version listed when you do a <tt>vzplgls</tt>:<br />
<br />
mysql-rh9 20030814 20050412<br />
<br />
this means there are 2 versions of the mysql package for RH9: 20030814 and 20050412<br />
As an aside, you can't copy over just 1 of them, but if you rsync'd over the <tt>mysql-rh9</tt> directory, you'd see 20030814 and 20050412 on the target HN. As long as the versions match up, you're good to go.<br />
<br />
With EZ templates, the versions are not as easy to decipher. When you look at the <tt>vzpkg list</tt> output, that simply tells you when a cache was created. A cache is simply a snapshot of all the data needed for the latest versions of the OS and it's applications at the time the cache was created. It is a date, and thus tells you nothing about the versions within. So for instance, if you had a cache for Ubuntu 10.04 from 2007 and you ran <tt>vzpkg create cache ubuntu-10.04-x86</tt> it would re-download the latest version of apache, mysql and so on. And any ''subsequent'' ubuntu 10.04 CT's created thereafter would use those newer versions, whereas older CT's based on 10.04 would use older templates. You can see how this works by looking at the files under:<br />
<br />
<pre># ls /vz/template/ubuntu/10.04/x86/<br />
aap_1.091-1_all<br />
aap-doc_1.091-1_all<br />
adduser_3.112ubuntu1_all<br />
anacron_2.3-13.1ubuntu11_i386<br />
apache2_2.2.14-5ubuntu8.2_i386<br />
apache2_2.2.14-5ubuntu8.4_i386<br />
apache2_2.2.14-5ubuntu8.6_i386<br />
apache2_2.2.14-5ubuntu8.7_i386<br />
apache2_2.2.14-5ubuntu8.8_i386<br />
apache2_2.2.14-5ubuntu8.9_i386<br />
<...snip><br />
</pre><br />
<br />
You can see that this template has in it 6 different versions of apache2. This means that this template was merged with other 10.04 templates and/or it has been cached a few times, each time a new apache was available. The unfortunate thing is it's almost impossible to know which CT's are using which version of apache so it's hard to try to prune this down and remove unwanted/unneeded applications.<br />
<br />
So, if you see another HN has Ubuntu 10.04 on it, you need to see/confirm that it has all the exact files your CT needs. You can run a test rsync to see what ''would'' be transferred (what's missing):<br />
<br />
rsync -avn --stats -e ssh /vz/template/ubuntu/10.04 root@10.1.4.69:/vz/template/ubuntu<br />
<br />
Based on the amount of data you get back from the rsync, you will be able to decide whether or not to remove the -n and allow the full transfer to go through. The downside to doing the transfer is the situation described above- you've permanently joined these templates and now you may have 10 versions of apache on the target HN. There's one other potential pitfall to this kind of merging- it will also potentially copy over shared files, for which there is no particular version, most of which are in <tt>/vz/template/ubuntu/10.04/x86/config</tt>: <br />
<br />
cat /vz/template/ubuntu/10.04/x86/config/os/default/repositories<br />
/vz/template/ubuntu/10.04/x86/config/os/default/repositories<br />
<br />
Here are some specific things to look out for. Case: copying centos-5 on virt19 to virt12. We dumped the rsync output to a file so we could inspect:<br />
<br />
[root@virt19 /vz/template]# rsync -an -e ssh /vz/template/centos/ root@10.1.4.62:/vz/template/centos/ > v12_c5<br />
<br />
(note, that can be dangerous, better to add on full path <tt>/vz/template/centos/5</tt>)<br />
<br />
So in the output file we see things like:<br />
<br />
<pre><br />
x86/base0/cachecookie<br />
x86/base0/primary.xml.gz <br />
x86/base0/primary.xml.gz.sqlite <br />
x86/base0/repomd.xml <br />
x86/base1/cachecookie <br />
x86/base1/filelists.xml.gz <br />
x86/base1/filelists.xml.gz.sqlite <br />
x86/base1/primary.xml.gz <br />
x86/base1/primary.xml.gz.sqlite <br />
x86/base1/repomd.xml<br />
x86/base2/cachecookie<br />
</pre><br />
<br />
Which make us nervous cause those are not versioned files, i.e.:<br />
5/x86/MAKEDEV-3.23-1.2.i386/usr/sbin/mksock<br />
<br />
So those files will replace existing files on v12 rather than the case of the latter where it's likely being added. So caution should be given and deference to whichever version is newer (which rsync should take care of, but that will depend on the date each file was created and perhaps not the actual data within).<br />
<br />
If we look further in the file we see:<br />
<br />
x86/psa-appvault-moodle-1.8-8202920080409011254.noarch/usr/local/psa/var/cgitory/moodle-1.<br />
9/htdocs/lib/editor/htmlarea/plugins/TableOperations/img/cell-split.gif<br />
<br />
and other files looking like:<br />
5/x86/plesk-base-9.0.1-cos5.build90090127.18.i586/etc/sw/keys/info<br />
<br />
Which means plesk is here. We don't need plesk on v12 (the CT moving over doesn't use it) so we rerun the rsync and exclude it:<br />
<br />
[root@virt19 /vz/template]# rsync -an -e ssh --exclude="plesk*" --exclude="psa*" /vz/template/centos/ root@10.1.4.62:/vz/template/centos/ > v12_c5<br />
<br />
and now it's much smaller:<br />
<br />
[root@virt19 /vz/template]# cat v12_c5|wc -l<br />
97104<br />
<br />
Before running with excludes it was:<br />
[root@virt19 /vz/template]# cat v12_c5|wc -l <br />
238238<br />
<br />
So again, look at what you're doing. Generally, combining templates is fine however.<br />
<br />
If you're happy with the merge, run the rsync again without the "-n" to let it actually do the transfer.<br />
<br />
Once done, don't forget to add the new template to the HN in Management -> Reference -> Templates<br />
<br />
= getting help =<br />
<br />
= vzup2date =<br />
TODO <br />
= Adding new OS/application templates =<br />
<br />
Virtuozzo will lag a bit behind the initial release of an OS, perhaps by a month or more. Further, a newly-released OS may be available, but there may not be many/any application templates available initially. It pays to wait a bit.<br />
<br />
A template is easily installed via [[#vzup2date|vzup2date]]:<br />
<br />
vzup2date -z<br />
<br />
You will be given a list of OS's to install. Select an OS, then customize that selection by checking/unchecking the application templates we do/don't want to include/offer.<br />
<br />
Generally, here are the app templates we include/offer:<br />
<pre>cyrus-imap<br />
devel<br />
imp<br />
jre<br />
jsdk<br />
mailman<br />
mod_perl<br />
mod_ssl<br />
mysql<br />
php<br />
phpmyadmin<br />
phppgadmin<br />
postgresql<br />
proftpd<br />
spamassassin<br />
squirrelmail<br />
tomcat<br />
vsftpd<br />
</pre><br />
<br />
We do not (any longer) offer webalizer or analog.<br />
<br />
After selecting and installing the OS and apps, you should ensure it's installed:<br />
<br />
vzpkg list -O<br />
<br />
Your new OS (ex. fedora-core-13-x86_64) will be listed, without a corresponding cache date:<br />
<pre><br />
[root@virt13 /vzconf]# vzpkg list -O<br />
centos-6-x86 2011-07-22 13:24:41<br />
centos-6-x86_64 2012-03-09 20:42:22<br />
centos-5-x86 2009-07-27 16:58:24<br />
centos-5-x86_64 2012-03-09 22:38:53<br />
ubuntu-11.10-x86_64 2012-03-01 23:13:33<br />
ubuntu-9.04-x86 2009-06-02 11:32:39<br />
ubuntu-12.04-x86_64 2012-05-29 13:50:50<br />
ubuntu-8.04-x86 2008-09-17 21:27:20<br />
ubuntu-8.10-x86 2009-03-13 13:58:57<br />
ubuntu-11.04-x86 2011-12-05 11:15:46<br />
ubuntu-7.04-x86 2008-09-24 16:42:50<br />
redhat-el6-x86_64<br />
fedora-core-13-x86_64 <br />
fedora-core-12-x86 2010-02-06 13:24:32<br />
fedora-core-15-x86 2011-12-05 11:36:43<br />
fedora-core-11-x86 2009-10-01 16:30:59<br />
fedora-core-14-x86<br />
fedora-core-16-x86_64 2012-03-13 11:40:23<br />
fedora-core-9-x86 2008-11-29 10:52:18<br />
debian-6.0-x86 2011-07-29 16:00:35<br />
debian-6.0-x86_64 2012-03-12 19:53:33<br />
debian-5.0-x86 2009-03-12 12:39:11<br />
</pre><br />
<br />
You can also confirm the apps installed:<br />
<pre>[root@virt13 /vzconf]# vzpkg list fedora-core-13-x86_64<br />
fedora-core-13-x86_64 2013-03-08 11:39:44<br />
fedora-core-13-x86_64 devel<br />
fedora-core-13-x86_64 php<br />
fedora-core-13-x86_64 proftpd<br />
fedora-core-13-x86_64 postgresql<br />
fedora-core-13-x86_64 phpmyadmin<br />
fedora-core-13-x86_64 mysql<br />
</pre><br />
<br />
Before you can use the OS you must cache it. To create the cache run:<br />
vzpkg cache [OS]<br />
<br />
ex: <tt>vzpkg cache fedora-core-13-x86_64</tt><br />
<br />
Now it's ready to use. In order to be able to use it with the [[VPS_Management#vm|vm]] command, you must create a file:<br />
jctmpl-[OS]<br />
<br />
ex: <tt>jctmpl-fedora-core-13-x86_64</tt><br />
<br />
In it, on line 1 we place the OS, followed by the app names, ex:<br />
<pre>more jctmpl-fedora-core-13-x86_64<br />
fedora-core-13-x86_64<br />
postgresql<br />
jsdk<br />
mod_ssl<br />
phpmyadmin<br />
mod_perl<br />
tomcat<br />
devel<br />
php<br />
mailman<br />
cyrus-imap<br />
squirrelmail<br />
proftpd<br />
mysql<br />
phppgadmin<br />
spamassassin<br />
</pre><br />
<br />
The packages will be installed in that order, so any pre-req's should be handled there.<br />
<br />
Now, when you run [[VPS_Management#vm|vm]] it will show you the newly-added OS and you may create a CT using it.<br />
<br />
The last few tasks are related to mgmt:<br />
<br />
1. add to mgmt->reference->templates (use the existing templates as an example). Once added, the new template will be included with the list of OS's available for the given HN.<br />
2. if this is going to be a new OS offered for ordering, it needs to be added to the order form and webpage.<br />
<br />
a. to get a list of applications and versions in the new OS, you should create a test CT, enter it and run <tt>dpkg -l|sort</tt> (or <tt>rpm -aq|sort</tt> in centos/fedora) and enter that list in the following places:<br><br />
<tt>/usr/local/www/jc_pub/[osname].html</tt><br><br />
<tt>/usr/local/www/signup/html/[osname].html</tt><br><br />
Note, you will have to move the most recent OS data from <tt>/usr/local/www/signup/html/[osname].html</tt> into the corresponding <tt>/usr/local/www/signup/html/[osname]_old.html</tt> updating the version links in both.<br />
<br />
b. to add the new OS to the order form, you will need to update <tt>/usr/local/www/signup/html/step1.html</tt> in several places (for regular and OSS orders) as well as updating the templateID which you can get from mgmt->reference->templates (or [[Management_System_/_Public_Website_/_Signup_/_Account_Manager#jc:ref_templates|jc:ref_templates]])<br />
<br />
<br />
<br />
Old method to obtain the templates:<br />
<br />
Go to http://www.parallels.com/en/virtuozzo/templates/catalog/<br />
<br />
Download the RPM's and install them<br />
<br />
vzpkg list<br />
to see the new packages/os templates<br />
<br />
vzpkg create cache "OS_EZ_template_name"<br />
to create a cache<br />
<br />
<br />
= Update OS template =<br />
<br />
Installation and update of the new version of OS template will not affect the existing containers. It will affect only the new containers, created after this update operation. In terms of fixing "vzctl enter" error and SSH connectivity to Ubuntu based containers, it is necessary to recreate the template and not to update it, so the commands to run are:<br />
<br />
<pre>rpm -Uhv ubuntu-8.04-x86-ez-4.0.0-9.swsoft.noarch.rpm<br />
vzpkg clean ubuntu-8.04-x86<br />
vzpkg remove cache ubuntu-8.04-x86<br />
vzpkg create cache ubuntu-8.04-x86</pre><br />
<br />
<br />
= Installing new SSL cert on VZCP =<br />
<br />
<pre>openssl req -days 1999 -new -x509 -nodes -out server.crt -keyout server.key<br />
<br />
US<br />
CA<br />
San Diego<br />
johncompanies.com<br />
johncompanies.com<br />
virt15ohncompanies.com<br />
linux@johncompanies.com<br />
<br />
cp -p /etc/httpd/conf/ssl.crt/server.crt /etc/httpd/conf/ssl.crt/server.crt.bak<br />
cp -p /etc/httpd/conf/ssl.key/server.key /etc/httpd/conf/ssl.key/server.key.bak<br />
cp server.crt /etc/httpd/conf/ssl.crt/server.crt<br />
cp server.key /etc/httpd/conf/ssl.key/server.key<br />
/etc/init.d/vzcp restart</pre><br />
<br />
= Moving virt (drives) to new hardware =<br />
<br />
If you have a case where a virt is dead (due to hardware failure), you can move the drives to a new hardware setup.<br />
Here are the steps and things to look for:<br />
<br />
Move the drives to the new box. Most likely, the drives will be recognized and there will be no problems booting up. On Dell PERC 5/6 cards you will need to import the new "foreign" volume via the raid BIOS.<br />
<br />
If the hardware (motherboard) is different, it’s possible the nic’s won’t be recognized. Look at /etc/modules.conf the ethernet drivers are either:<br />
<br />
<pre>alias eth0 tg3<br />
alias eth1 tg3</pre><br />
(supermicro)<br />
<br />
Or<br />
<br />
<pre>alias eth0 e100<br />
alias eth1 e1000</pre><br />
(intel)<br />
<br />
Or<br />
<br />
<pre>alias eth0 bnx2<br />
alias eth1 bnx2</pre><br />
(dell 29500)<br />
<br />
Also, you may see eth0 and eth1 swapped so if after confirming the eth devices are present (may require a reboot), then you may still need to swap the green/yellow network cables in the back to get them on the right port.<br />
<br />
VZ will not run (for long) on the new hardware, so you will need to get the license reissued. You may create/download a temporary license via their website for the interim.<br />
<br />
= Updating kernel on virt =<br />
<br />
We now use [[#vzup2date|vzup2date]] to update systems and kernels. What follows is what we used to do when we had to download kernels and install manually on old systems:<br />
<br />
<br />
1. install kernel and modules<br />
<pre># rpm -ivh vzkernel-enterprise-2.4.20-021stab028.12.777.i686.rpm \<br />
vzmodules-enterprise-2.4.20-021stab028.12.swsoft.i686.rpm<br />
vzkernel-smp ##################################################<br />
vzmodules-smp ##################################################</pre><br />
<br />
DO NOT USE the "rpm -Uhv" command to install the kernel, otherwise all the previously installed kernels may be removed from your system. <br />
<br />
2. update lilo/grub<br />
<br />
Lilo:<br />
<br />
vi /etc/lilo.conf<br />
<br />
rename old Virtuozzo kernel label from "linux+virtuozzo" to "virtuozzo_old"<br />
and add a new entry pointing to the newly installed virtuozzo kernel image.<br />
For example:<br />
<br />
<pre>prompt<br />
timeout=50<br />
default=linux+virtuozzo<br />
append="debug panic=5 console=tty console=ttyS0,38400"<br />
boot=/dev/sda<br />
map=/boot/map<br />
install=/boot/boot.b<br />
message=/boot/message<br />
linear<br />
<br />
image=/boot/vmlinuz-2.4.18-3smp<br />
label=linux<br />
initrd=/boot/initrd-2.4.18-3smp.img<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.18-3<br />
label=linux-up<br />
initrd=/boot/initrd-2.4.18-3.img<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-020stab009.5.777-enterprise<br />
label=2.4.20-9.5.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-020stab009.8.777-enterprise<br />
label=2.4.20-9.8.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-020stab009.21.777-enterprise<br />
label=2.4.20-9.21.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-020stab009.24.777-enterprise<br />
label=2.4.20-9.24.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-021stab028.3.777-enterprise<br />
label=2.4.20-28.3.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-021stab028.5.777-enterprise<br />
label=old_linux+vz<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-021stab028.12.777-enterprise<br />
label=linux+virtuozzo<br />
read-only<br />
root=/dev/sda1</pre><br />
<br />
With this configuration, the new kernel is the default kernel to use at boot. The original kernel entry has been retained with the label "old_linux+vz ".<br />
<br />
Grub:<br />
<br />
vi /boot/grub /menu.lst<br />
<br />
Add new entry at the top.<br />
For example:<br />
<br />
<pre>serial --unit=0 --speed=38400<br />
terminal --timeout=10 serial console<br />
default=0<br />
timeout=10<br />
splashimage=(hd0,0)/boot/grub/splash.xpm.gz<br />
title Virtuozzo 2.6.1 (2.4.20-021stab028.12.777-enterprise)<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-22.4.20-021stab028.12.777-enterprise root=/dev/sda1 console=tty0 \ console=ttyS0,38400<br />
title Virtuozzo 2.6.1 (2.4.20-021stab028.3.777-enterprise)<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.20-021stab028.3.777-enterprise root=/dev/sda1 console=tty0 \ console=ttyS0,38400<br />
title Red Hat Linux (2.4.18-3smp)<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.18-3smp ro root=/dev/sda1<br />
initrd /boot/initrd-2.4.18-3smp.img<br />
title Red Hat Linux-up (2.4.18-3)<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.18-3 ro root=/dev/sda1<br />
initrd /boot/initrd-2.4.18-3.img<br />
title Linux with Virtuozzo 2.5<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.20-020stab009.3.777-smp ro root=/dev/sda1<br />
title vz-enterpriseo<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.20-020stab009.21.777-enterprise ro root=/dev/sda1<br />
title vz-enterprise<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.20-020stab009.24.777-enterprise ro root=/dev/sda1 \ console=tty0 console=ttyS0,38400</pre><br />
<br />
3. run the lilo (if using lilo)<br />
<br />
<pre># lilo<br />
Added linux<br />
Added linux-up<br />
Added virtuozzo_old<br />
Added linux+virtuozzo *</pre><br />
<br />
4. reboot the server<br />
shutdown -r 23:05<br />
<br />
when it comes time for the shutdown, run [[VPS_Management#stopvirt|stopvirt]] to get things shut down faster<br />
<br />
<br />
= Remaking service VE (<=3.x only) =<br />
<br />
Occasionally the control panel for VEs (aka VZPP) will stop working and you just need to reinstall the service VE (which runs the control panels for all VEs on the HN). Here's how that's done :<br />
<br />
<pre>vzctl stop 1<br />
ipdel 1 69.55.234.152<br />
vzmlocal 1:2<br />
vzctl set 2 --save --onboot no<br />
vzsveinstall -t redhat-as3-minimal -d /root/Rel261/HW/RPMS -s 69.55.234.152 --skip-client</pre><br />
<br />
on 3.0:<br />
<br />
<pre>vzsveinstall -t redhat-as3-minimal -d /vz/Rel300/HW/RPMS -s 69.55.229.151 --skip-client<br />
<br />
vzctl stop 1<br />
vzctl mount 1<br />
vzctl mount 2<br />
/bin/cp -aRf /vz/root/2/home/vzagent0 /vz/root/1/home<br />
/bin/cp -aRf /vz/root/2/var/lib/mysql/VZADB /vz/root/1/var/lib/mysql<br />
/bin/cp -af /vz/root/2/etc/shadow /vz/root/1/etc/<br />
/bin/cp -aRf /vz/root/2/var/vzcp/static/vz/skins/ /vz/root/1/var/vzcp/static/vz<br />
/bin/cp -af /vz/root/2/etc/vzcp/pp/menu.xml /vz/root/1/etc/vzcp/pp/<br />
/bin/cp -af /vz/root/2/etc/vzcp/pp/dashboard.xml /vz/root/1/etc/vzcp/pp/<br />
<br />
vzctl umount 2<br />
vzctl umount 1<br />
vzctl start 1</pre><br />
<br />
<br />
= GRUB: boot once to a new kernel =<br />
<br />
To use a different kernel for the next reboot only:<br />
<pre>grub --no-floppy<br />
savedefault --default=N --once<br />
(where N is the number of the kernel you want to boot once)<br />
quit</pre><br />
<br />
Reboot your machine. It will boot using kernel "N".<br />
<br />
grub-reboot N<br />
will do essentially the same thing and prompt to reboot the machine<br />
<br />
<br />
= Recovering from GRUB failure =<br />
<br />
(example from DPE 2950)<br />
<br />
Boot to CEOS4 server CD (original install cd). Did this example with iso mounted via drac<br />
<br />
Boot: linux rescue<br />
<br />
<br />
= Make/edit an ISO on Linux =<br />
<br />
<pre>mount file.iso /mnt/iso/ -t iso9660 -o ro,loop=/dev/loop0<br />
mkdir /tmp/iso<br />
cp -aR /mnt/iso/* /tmp/iso/<br />
umount /mnt/iso/<br />
mkisofs -iso-level 2 -o /tmp/new.iso /tmp/iso/<br />
rm -fr /tmp/iso</pre><br />
<br />
To mount it:<br />
<br />
mount -t vfat -o loop /tmp/3wfirmware.iso /mnt/usb/<br />
<br />
http://www.damnsmalllinux.org/wiki/index.php/Installing_to_a_USB_Flash_Drive<br />
<br />
= Mount/edit USB drive on Linux =<br />
<br />
mount -t vfat -o loop /dev/sdc1 /mnt/usb/<br />
<br />
This example is for our floating usb drive on virt16. You should edit the device as necessary on other hardware.<br />
<br />
<br />
= Installing OS via net =<br />
<br />
Centos 5.1:<br><br />
http site name: <tt>vault.centos.org</tt><br><br />
dir: <tt>/5.1/os/i386/</tt><br />
<br />
Centos 5.4:<br><br />
http site name: <tt>mirrors.kernel.org</tt><br><br />
dir: <tt>/centos/5.4/os/i386/</tt><br><br />
or<br><br />
<tt>/centos/5.4/os/x86_64/</tt><br><br />
or<br><br />
<tt>http://mirrors.usc.edu/pub/linux/distributions/centos/5.4/os/i386/</tt><br><br />
<br />
<br />
Centos 6.2:<br><br />
<tt>http://mirrors.usc.edu/pub/linux/distributions/centos/6.2/os/x86_64/</tt><br><br />
<br />
<br />
FC9:<br><br />
<tt>http://linux.nssl.noaa.gov/fedora/linux/releases/9/Fedora/i386/os/</tt><br><br />
<br />
<br />
FC10:<br><br />
<tt>http://linux.nssl.noaa.gov/fedora/linux/releases/10/Fedora/i386/os/</tt><br></div>Supporthttps://wiki.jcihosting.com/index.php?title=Virtuozzo_/_Linux_Reference&diff=1055Virtuozzo / Linux Reference2013-03-12T00:12:20Z<p>Support: /* Mount/edit USB drive on Linux */</p>
<hr />
<div>= Updating VZ =<br />
<br />
Update Virtuozzo to version 4.0.0<br />
Apply minor updates for Virtuozzo 3.0.0<br />
<br />
...<br />
When you get to the last screen, you will be given an option to <br />
<br />
( ) Automatically reboot after update<br />
(*) Reboot later manually<br />
<br />
make sure you choose to reboot later.<br />
<br />
= Migrating Templates =<br />
One nice feature VZ offers is the ability to run a virtual environment (VE or CT as it's now called) based on a particular ditribution on a host which may or may not share the same distribution. For instance, you could run a CT running Ubuntu while the host machine (Hardware Node or HN) is running CentOS. This is accomplished via the use of OS templates. As long as the HN contains the OS template on which a particular CT relies, it will run there. As such, if you want to move a CT from one HN to another, you will need to make sure the OS template exists there.<br />
<br />
Modern versions of VZ (>= 4.x) will check for and automatically move the OS template over to the host as you're (vz)migrating the CT over to a new HN. However we like to make sure the template is in place prior to moving the CT. <br />
<br />
The first step is to see if the template exists on the host. To see OS templates installed:<br />
<br />
2.x (shows OS and application templates):<br />
<pre># vzpkgls<br />
mod_ssl-rh9 20040624<br />
mysql-rh9 20030814 20050412<br />
openwebmail-rh9 20050427<br />
php-rh9 20050506<br />
phpBB-rh9 20041229<br />
postgresql-rh9 20050719<br />
sl-webalizer-rh9 20040119<br />
spamassassin-rh9 20040127<br />
tomcat-rh9 20040524<br />
usermin-rh9 20040116</pre><br />
<br />
>= 3.x (shows just OS templates, run without -O to see application as well):<br />
<pre># vzpkg list -O<br />
ubuntu-10.04-x86 2010-06-01 11:39:05<br />
ubuntu-11.04-x86 2011-06-22 14:23:59<br />
ubuntu-9.10-x86 2010-03-11 17:39:51<br />
centos-5-x86 2010-03-11 17:12:27<br />
debian-5.0-x86 2010-03-11 17:33:34</pre><br />
<br />
<br />
All template files are located:<br />
<br />
<pre># ls /vz/template/<br />
ColdFusion-fc1 jre-fc1 openwebmail-fc1 sl-webalizer-fc1<br />
ColdFusion-fc2 jre-fc2 openwebmail-fc2 sl-webalizer-fc2<br />
ColdFusion-rh9 jre-rh9 openwebmail-rh9 sl-webalizer-rh9<br />
PostNuke-fc1 jre-suse92 openwebmail-suse92 spamassassin-fc1<br />
PostNuke-fc2 jrun php-deb31 spamassassin-fc2<br />
PostNuke-rh9 mailman-deb31 php-fc1 spamassassin-rh9<br />
analog-fc1 mailman-fc1 php-fc2 spamassassin-suse92<br />
analog-fc2 mailman-fc2 php-rh9 suse-9.2<br />
analog-rh9 mailman-rh9 php-suse92 tomcat-fc1<br />
awstats-fc1 mailman-suse92 phpBB-fc1 tomcat-fc2<br />
awstats-rh9 mod_frontpage-fc1 phpBB-fc2 tomcat-rh9<br />
bbClone-fc1 mod_frontpage-fc2 phpBB-rh9 tomcat-suse92<br />
bbClone-fc2 mod_frontpage-rh9 phpmyadmin-deb31 urchin-fc1<br />
bbClone-rh9 mod_frontpage-suse92 postgresql-deb31 usermin-fc1<br />
conf mod_perl-deb31 postgresql-fc1 usermin-fc2<br />
debian-3.1 mod_perl-fc1 postgresql-fc2 usermin-rh9<br />
devel-deb31 mod_perl-fc2 postgresql-rh9 uw-imap-fc1<br />
devel-fc2 mod_perl-rh9 postgresql-suse92 uw-imap-fc2<br />
devel-suse92 mod_perl-suse92 proftpd-deb31 uw-imap-rh9<br />
fedora-core-1 mod_ssl-fc1 proftpd-fc1 vzredhat-7.3<br />
fedora-core-2 mod_ssl-fc2 proftpd-fc2 vzredhat-8.0<br />
jdk-deb31 mod_ssl-rh9 proftpd-rh9 webalizer-suse92<br />
jdk-fc1 mysql-deb31 proftpd-suse92 webmin-fc1<br />
jdk-fc2 mysql-fc1 redhat-7.2 webmin-fc2<br />
jdk-rh9 mysql-fc2 redhat-9 webmin-rh9<br />
jdk-suse92 mysql-rh9 redhat-as3-minimal<br />
jre-deb31 mysql-suse92 redhat-devel-9<br />
</pre><br />
<br />
What we see here are the base OS templates: <tt>redhat-9, fedora-core-1</tt> and supporting application templates: <tt>postgresql-rh9, mailman-rh9, jdk-fc1, mod_ssl-fc1</tt>. So if you wanted to copy over all of redhat 9 you'd need to copy the contents of:<br />
<pre># ls -d /vz/template/*rh9*<br />
/vz/template/ColdFusion-rh9 /vz/template/mod_frontpage-rh9 /vz/template/proftpd-rh9<br />
/vz/template/PostNuke-rh9 /vz/template/mod_perl-rh9 /vz/template/sl-webalizer-rh9<br />
/vz/template/analog-rh9 /vz/template/mod_ssl-rh9 /vz/template/spamassassin-rh9<br />
/vz/template/awstats-rh9 /vz/template/mysql-rh9 /vz/template/tomcat-rh9<br />
/vz/template/bbClone-rh9 /vz/template/openwebmail-rh9 /vz/template/usermin-rh9<br />
/vz/template/jdk-rh9 /vz/template/php-rh9 /vz/template/uw-imap-rh9<br />
/vz/template/jre-rh9 /vz/template/phpBB-rh9 /vz/template/webmin-rh9<br />
/vz/template/mailman-rh9 /vz/template/postgresql-rh9<br />
<br />
# ls -d /vz/template/*redhat-9*<br />
/vz/template/redhat-9<br />
</pre><br />
<br />
To get a template from one HN to another, it simply needs to be rsync'd over to the target HN. It's rsync'd over in this fashion:<br />
<br />
rsync -av -e ssh /vz/template/redhat-9 root@10.1.4.65:/vz/template<br />
rsync -av -e ssh /vz/template/*rh9 root@10.1.4.65:/vz/template<br />
<br />
Newer (>= 3.x) VZ employs "EZ" templates which are organized a little differently:<br />
<br />
<pre># ls /vz/template/<br />
cache CmdTool.log debian redhat-as3-minimal-20080630.tar.gz vc<br />
centos conf redhat-as3-minimal ubuntu<br />
<br />
# ls /vz/template/ubuntu/<br />
10.04 11.04 9.10<br />
# ls /vz/template/ubuntu/10.04/<br />
x86<br />
# ls /vz/template/ubuntu/10.04/x86/<br />
aap_1.091-1_all<br />
aap-doc_1.091-1_all<br />
adduser_3.112ubuntu1_all<br />
anacron_2.3-13.1ubuntu11_i386<br />
apache2_2.2.14-5ubuntu8.2_i386<br />
apache2_2.2.14-5ubuntu8.4_i386<br />
apache2_2.2.14-5ubuntu8.6_i386<br />
apache2_2.2.14-5ubuntu8.7_i386<br />
apache2_2.2.14-5ubuntu8.8_i386<br />
apache2_2.2.14-5ubuntu8.9_i386<br />
<...snip><br />
<br />
# ls /vz/template/cache/<br />
centos-5-x86.tar.gz suse-11.3-x86.tar.gz ubuntu-9.10-x86.tar.gz<br />
debian-5.0-x86.tar.gz ubuntu-10.04-x86.tar.gz<br />
fedora-core-14-x86.tar.gz ubuntu-11.04-x86.tar.gz<br />
</pre><br />
<br />
So what we see is the entire OS and all applications are in 1 directory, and organized by distribution and release version. So to move Ubuntu 10.04:<br />
<br />
rsync -av -e ssh /vz/template/ubuntu/10.04 root@10.1.4.69:/vz/template/ubuntu<br />
<br />
and you also need to move the cache:<br />
<br />
rsync -av -e ssh /vz/template/cache/ubuntu-10.04-x86.tar.gz root@10.1.4.69:/vz/template/cache/<br />
<br />
Once the transfer is complete, you will be able to run <tt>vzpkgls</tt> or <tt>vzpkg list -O</tt> and see the template(s) listed on the target HN.<br />
<br />
So far, this all supposes template doesn't exist on the target- very simple, just copy it over. If the template exists already on the target HN, this requires closer inspection. With older templates, simply moving over the templates would be the end of it- you see the version listed when you do a <tt>vzplgls</tt>:<br />
<br />
mysql-rh9 20030814 20050412<br />
<br />
this means there are 2 versions of the mysql package for RH9: 20030814 and 20050412<br />
As an aside, you can't copy over just 1 of them, but if you rsync'd over the <tt>mysql-rh9</tt> directory, you'd see 20030814 and 20050412 on the target HN. As long as the versions match up, you're good to go.<br />
<br />
With EZ templates, the versions are not as easy to decipher. When you look at the <tt>vzpkg list</tt> output, that simply tells you when a cache was created. A cache is simply a snapshot of all the data needed for the latest versions of the OS and it's applications at the time the cache was created. It is a date, and thus tells you nothing about the versions within. So for instance, if you had a cache for Ubuntu 10.04 from 2007 and you ran <tt>vzpkg create cache ubuntu-10.04-x86</tt> it would re-download the latest version of apache, mysql and so on. And any ''subsequent'' ubuntu 10.04 CT's created thereafter would use those newer versions, whereas older CT's based on 10.04 would use older templates. You can see how this works by looking at the files under:<br />
<br />
<pre># ls /vz/template/ubuntu/10.04/x86/<br />
aap_1.091-1_all<br />
aap-doc_1.091-1_all<br />
adduser_3.112ubuntu1_all<br />
anacron_2.3-13.1ubuntu11_i386<br />
apache2_2.2.14-5ubuntu8.2_i386<br />
apache2_2.2.14-5ubuntu8.4_i386<br />
apache2_2.2.14-5ubuntu8.6_i386<br />
apache2_2.2.14-5ubuntu8.7_i386<br />
apache2_2.2.14-5ubuntu8.8_i386<br />
apache2_2.2.14-5ubuntu8.9_i386<br />
<...snip><br />
</pre><br />
<br />
You can see that this template has in it 6 different versions of apache2. This means that this template was merged with other 10.04 templates and/or it has been cached a few times, each time a new apache was available. The unfortunate thing is it's almost impossible to know which CT's are using which version of apache so it's hard to try to prune this down and remove unwanted/unneeded applications.<br />
<br />
So, if you see another HN has Ubuntu 10.04 on it, you need to see/confirm that it has all the exact files your CT needs. You can run a test rsync to see what ''would'' be transferred (what's missing):<br />
<br />
rsync -avn --stats -e ssh /vz/template/ubuntu/10.04 root@10.1.4.69:/vz/template/ubuntu<br />
<br />
Based on the amount of data you get back from the rsync, you will be able to decide whether or not to remove the -n and allow the full transfer to go through. The downside to doing the transfer is the situation described above- you've permanently joined these templates and now you may have 10 versions of apache on the target HN. There's one other potential pitfall to this kind of merging- it will also potentially copy over shared files, for which there is no particular version, most of which are in <tt>/vz/template/ubuntu/10.04/x86/config</tt>: <br />
<br />
cat /vz/template/ubuntu/10.04/x86/config/os/default/repositories<br />
/vz/template/ubuntu/10.04/x86/config/os/default/repositories<br />
<br />
Here are some specific things to look out for. Case: copying centos-5 on virt19 to virt12. We dumped the rsync output to a file so we could inspect:<br />
<br />
[root@virt19 /vz/template]# rsync -an -e ssh /vz/template/centos/ root@10.1.4.62:/vz/template/centos/ > v12_c5<br />
<br />
(note, that can be dangerous, better to add on full path <tt>/vz/template/centos/5</tt>)<br />
<br />
So in the output file we see things like:<br />
<br />
<pre><br />
x86/base0/cachecookie<br />
x86/base0/primary.xml.gz <br />
x86/base0/primary.xml.gz.sqlite <br />
x86/base0/repomd.xml <br />
x86/base1/cachecookie <br />
x86/base1/filelists.xml.gz <br />
x86/base1/filelists.xml.gz.sqlite <br />
x86/base1/primary.xml.gz <br />
x86/base1/primary.xml.gz.sqlite <br />
x86/base1/repomd.xml<br />
x86/base2/cachecookie<br />
</pre><br />
<br />
Which make us nervous cause those are not versioned files, i.e.:<br />
5/x86/MAKEDEV-3.23-1.2.i386/usr/sbin/mksock<br />
<br />
So those files will replace existing files on v12 rather than the case of the latter where it's likely being added. So caution should be given and deference to whichever version is newer (which rsync should take care of, but that will depend on the date each file was created and perhaps not the actual data within).<br />
<br />
If we look further in the file we see:<br />
<br />
x86/psa-appvault-moodle-1.8-8202920080409011254.noarch/usr/local/psa/var/cgitory/moodle-1.<br />
9/htdocs/lib/editor/htmlarea/plugins/TableOperations/img/cell-split.gif<br />
<br />
and other files looking like:<br />
5/x86/plesk-base-9.0.1-cos5.build90090127.18.i586/etc/sw/keys/info<br />
<br />
Which means plesk is here. We don't need plesk on v12 (the CT moving over doesn't use it) so we rerun the rsync and exclude it:<br />
<br />
[root@virt19 /vz/template]# rsync -an -e ssh --exclude="plesk*" --exclude="psa*" /vz/template/centos/ root@10.1.4.62:/vz/template/centos/ > v12_c5<br />
<br />
and now it's much smaller:<br />
<br />
[root@virt19 /vz/template]# cat v12_c5|wc -l<br />
97104<br />
<br />
Before running with excludes it was:<br />
[root@virt19 /vz/template]# cat v12_c5|wc -l <br />
238238<br />
<br />
So again, look at what you're doing. Generally, combining templates is fine however.<br />
<br />
If you're happy with the merge, run the rsync again without the "-n" to let it actually do the transfer.<br />
<br />
Once done, don't forget to add the new template to the HN in Management -> Reference -> Templates<br />
<br />
= getting help =<br />
<br />
= vzup2date =<br />
TODO <br />
= Adding new OS/application templates =<br />
<br />
Virtuozzo will lag a bit behind the initial release of an OS, perhaps by a month or more. Further, a newly-released OS may be available, but there may not be many/any application templates available initially. It pays to wait a bit.<br />
<br />
A template is easily installed via [[#vzup2date|vzup2date]]:<br />
<br />
vzup2date -z<br />
<br />
You will be given a list of OS's to install. Select an OS, then customize that selection by checking/unchecking the application templates we do/don't want to include/offer.<br />
<br />
Generally, here are the app templates we include/offer:<br />
<pre>cyrus-imap<br />
devel<br />
imp<br />
jre<br />
jsdk<br />
mailman<br />
mod_perl<br />
mod_ssl<br />
mysql<br />
php<br />
phpmyadmin<br />
phppgadmin<br />
postgresql<br />
proftpd<br />
spamassassin<br />
squirrelmail<br />
tomcat<br />
vsftpd<br />
</pre><br />
<br />
We do not (any longer) offer webalizer or analog.<br />
<br />
After selecting and installing the OS and apps, you should ensure it's installed:<br />
<br />
vzpkg list -O<br />
<br />
Your new OS (ex. fedora-core-13-x86_64) will be listed, without a corresponding cache date:<br />
<pre><br />
[root@virt13 /vzconf]# vzpkg list -O<br />
centos-6-x86 2011-07-22 13:24:41<br />
centos-6-x86_64 2012-03-09 20:42:22<br />
centos-5-x86 2009-07-27 16:58:24<br />
centos-5-x86_64 2012-03-09 22:38:53<br />
ubuntu-11.10-x86_64 2012-03-01 23:13:33<br />
ubuntu-9.04-x86 2009-06-02 11:32:39<br />
ubuntu-12.04-x86_64 2012-05-29 13:50:50<br />
ubuntu-8.04-x86 2008-09-17 21:27:20<br />
ubuntu-8.10-x86 2009-03-13 13:58:57<br />
ubuntu-11.04-x86 2011-12-05 11:15:46<br />
ubuntu-7.04-x86 2008-09-24 16:42:50<br />
redhat-el6-x86_64<br />
fedora-core-13-x86_64 <br />
fedora-core-12-x86 2010-02-06 13:24:32<br />
fedora-core-15-x86 2011-12-05 11:36:43<br />
fedora-core-11-x86 2009-10-01 16:30:59<br />
fedora-core-14-x86<br />
fedora-core-16-x86_64 2012-03-13 11:40:23<br />
fedora-core-9-x86 2008-11-29 10:52:18<br />
debian-6.0-x86 2011-07-29 16:00:35<br />
debian-6.0-x86_64 2012-03-12 19:53:33<br />
debian-5.0-x86 2009-03-12 12:39:11<br />
</pre><br />
<br />
You can also confirm the apps installed:<br />
<pre>[root@virt13 /vzconf]# vzpkg list fedora-core-13-x86_64<br />
fedora-core-13-x86_64 2013-03-08 11:39:44<br />
fedora-core-13-x86_64 devel<br />
fedora-core-13-x86_64 php<br />
fedora-core-13-x86_64 proftpd<br />
fedora-core-13-x86_64 postgresql<br />
fedora-core-13-x86_64 phpmyadmin<br />
fedora-core-13-x86_64 mysql<br />
</pre><br />
<br />
Before you can use the OS you must cache it. To create the cache run:<br />
vzpkg cache [OS]<br />
<br />
ex: <tt>vzpkg cache fedora-core-13-x86_64</tt><br />
<br />
Now it's ready to use. In order to be able to use it with the [[VPS_Management#vm|vm]] command, you must create a file:<br />
jctmpl-[OS]<br />
<br />
ex: <tt>jctmpl-fedora-core-13-x86_64</tt><br />
<br />
In it, on line 1 we place the OS, followed by the app names, ex:<br />
<pre>more jctmpl-fedora-core-13-x86_64<br />
fedora-core-13-x86_64<br />
postgresql<br />
jsdk<br />
mod_ssl<br />
phpmyadmin<br />
mod_perl<br />
tomcat<br />
devel<br />
php<br />
mailman<br />
cyrus-imap<br />
squirrelmail<br />
proftpd<br />
mysql<br />
phppgadmin<br />
spamassassin<br />
</pre><br />
<br />
The packages will be installed in that order, so any pre-req's should be handled there.<br />
<br />
Now, when you run [[VPS_Management#vm|vm]] it will show you the newly-added OS and you may create a CT using it.<br />
<br />
The last few tasks are related to mgmt:<br />
<br />
1. add to mgmt->reference->templates (use the existing templates as an example). Once added, the new template will be included with the list of OS's available for the given HN.<br />
2. if this is going to be a new OS offered for ordering, it needs to be added to the order form and webpage.<br />
<br />
a. to get a list of applications and versions in the new OS, you should create a test CT, enter it and run <tt>dpkg -l|sort</tt> (or <tt>rpm -aq|sort</tt> in centos/fedora) and enter that list in the following places:<br><br />
<tt>/usr/local/www/jc_pub/[osname].html</tt><br><br />
<tt>/usr/local/www/signup/html/[osname].html</tt><br><br />
Note, you will have to move the most recent OS data from <tt>/usr/local/www/signup/html/[osname].html</tt> into the corresponding <tt>/usr/local/www/signup/html/[osname]_old.html</tt> updating the version links in both.<br />
<br />
b. to add the new OS to the order form, you will need to update <tt>/usr/local/www/signup/html/step1.html</tt> in several places (for regular and OSS orders) as well as updating the templateID which you can get from mgmt->reference->templates (or [[Management_System_/_Public_Website_/_Signup_/_Account_Manager#jc:ref_templates|jc:ref_templates]])<br />
<br />
<br />
<br />
Old method to obtain the templates:<br />
<br />
Go to http://www.parallels.com/en/virtuozzo/templates/catalog/<br />
<br />
Download the RPM's and install them<br />
<br />
vzpkg list<br />
to see the new packages/os templates<br />
<br />
vzpkg create cache "OS_EZ_template_name"<br />
to create a cache<br />
<br />
<br />
= Update OS template =<br />
<br />
Installation and update of the new version of OS template will not affect the existing containers. It will affect only the new containers, created after this update operation. In terms of fixing "vzctl enter" error and SSH connectivity to Ubuntu based containers, it is necessary to recreate the template and not to update it, so the commands to run are:<br />
<br />
<pre>rpm -Uhv ubuntu-8.04-x86-ez-4.0.0-9.swsoft.noarch.rpm<br />
vzpkg clean ubuntu-8.04-x86<br />
vzpkg remove cache ubuntu-8.04-x86<br />
vzpkg create cache ubuntu-8.04-x86</pre><br />
<br />
<br />
= Installing new SSL cert on VZCP =<br />
<br />
<pre>openssl req -days 1999 -new -x509 -nodes -out server.crt -keyout server.key<br />
<br />
US<br />
CA<br />
San Diego<br />
johncompanies.com<br />
johncompanies.com<br />
virt15ohncompanies.com<br />
linux@johncompanies.com<br />
<br />
cp -p /etc/httpd/conf/ssl.crt/server.crt /etc/httpd/conf/ssl.crt/server.crt.bak<br />
cp -p /etc/httpd/conf/ssl.key/server.key /etc/httpd/conf/ssl.key/server.key.bak<br />
cp server.crt /etc/httpd/conf/ssl.crt/server.crt<br />
cp server.key /etc/httpd/conf/ssl.key/server.key<br />
/etc/init.d/vzcp restart</pre><br />
<br />
= Moving virt (drives) to new hardware =<br />
<br />
If you have a case where a virt is dead (due to hardware failure), you can move the drives to a new hardware setup.<br />
Here are the steps and things to look for:<br />
<br />
Move the drives to the new box. Most likely, the drives will be recognized and there will be no problems booting up. On Dell PERC 5/6 cards you will need to import the new "foreign" volume via the raid BIOS.<br />
<br />
If the hardware (motherboard) is different, it’s possible the nic’s won’t be recognized. Look at /etc/modules.conf the ethernet drivers are either:<br />
<br />
<pre>alias eth0 tg3<br />
alias eth1 tg3</pre><br />
(supermicro)<br />
<br />
Or<br />
<br />
<pre>alias eth0 e100<br />
alias eth1 e1000</pre><br />
(intel)<br />
<br />
Or<br />
<br />
<pre>alias eth0 bnx2<br />
alias eth1 bnx2</pre><br />
(dell 29500)<br />
<br />
Also, you may see eth0 and eth1 swapped so if after confirming the eth devices are present (may require a reboot), then you may still need to swap the green/yellow network cables in the back to get them on the right port.<br />
<br />
VZ will not run (for long) on the new hardware, so you will need to get the license reissued. You may create/download a temporary license via their website for the interim.<br />
<br />
= Updating kernel on virt =<br />
<br />
We now use [[#vzup2date|vzup2date]] to update systems and kernels. What follows is what we used to do when we had to download kernels and install manually on old systems:<br />
<br />
<br />
1. install kernel and modules<br />
<pre># rpm -ivh vzkernel-enterprise-2.4.20-021stab028.12.777.i686.rpm \<br />
vzmodules-enterprise-2.4.20-021stab028.12.swsoft.i686.rpm<br />
vzkernel-smp ##################################################<br />
vzmodules-smp ##################################################</pre><br />
<br />
DO NOT USE the "rpm -Uhv" command to install the kernel, otherwise all the previously installed kernels may be removed from your system. <br />
<br />
2. update lilo/grub<br />
<br />
Lilo:<br />
<br />
vi /etc/lilo.conf<br />
<br />
rename old Virtuozzo kernel label from "linux+virtuozzo" to "virtuozzo_old"<br />
and add a new entry pointing to the newly installed virtuozzo kernel image.<br />
For example:<br />
<br />
<pre>prompt<br />
timeout=50<br />
default=linux+virtuozzo<br />
append="debug panic=5 console=tty console=ttyS0,38400"<br />
boot=/dev/sda<br />
map=/boot/map<br />
install=/boot/boot.b<br />
message=/boot/message<br />
linear<br />
<br />
image=/boot/vmlinuz-2.4.18-3smp<br />
label=linux<br />
initrd=/boot/initrd-2.4.18-3smp.img<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.18-3<br />
label=linux-up<br />
initrd=/boot/initrd-2.4.18-3.img<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-020stab009.5.777-enterprise<br />
label=2.4.20-9.5.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-020stab009.8.777-enterprise<br />
label=2.4.20-9.8.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-020stab009.21.777-enterprise<br />
label=2.4.20-9.21.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-020stab009.24.777-enterprise<br />
label=2.4.20-9.24.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-021stab028.3.777-enterprise<br />
label=2.4.20-28.3.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-021stab028.5.777-enterprise<br />
label=old_linux+vz<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-021stab028.12.777-enterprise<br />
label=linux+virtuozzo<br />
read-only<br />
root=/dev/sda1</pre><br />
<br />
With this configuration, the new kernel is the default kernel to use at boot. The original kernel entry has been retained with the label "old_linux+vz ".<br />
<br />
Grub:<br />
<br />
vi /boot/grub /menu.lst<br />
<br />
Add new entry at the top.<br />
For example:<br />
<br />
<pre>serial --unit=0 --speed=38400<br />
terminal --timeout=10 serial console<br />
default=0<br />
timeout=10<br />
splashimage=(hd0,0)/boot/grub/splash.xpm.gz<br />
title Virtuozzo 2.6.1 (2.4.20-021stab028.12.777-enterprise)<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-22.4.20-021stab028.12.777-enterprise root=/dev/sda1 console=tty0 \ console=ttyS0,38400<br />
title Virtuozzo 2.6.1 (2.4.20-021stab028.3.777-enterprise)<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.20-021stab028.3.777-enterprise root=/dev/sda1 console=tty0 \ console=ttyS0,38400<br />
title Red Hat Linux (2.4.18-3smp)<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.18-3smp ro root=/dev/sda1<br />
initrd /boot/initrd-2.4.18-3smp.img<br />
title Red Hat Linux-up (2.4.18-3)<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.18-3 ro root=/dev/sda1<br />
initrd /boot/initrd-2.4.18-3.img<br />
title Linux with Virtuozzo 2.5<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.20-020stab009.3.777-smp ro root=/dev/sda1<br />
title vz-enterpriseo<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.20-020stab009.21.777-enterprise ro root=/dev/sda1<br />
title vz-enterprise<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.20-020stab009.24.777-enterprise ro root=/dev/sda1 \ console=tty0 console=ttyS0,38400</pre><br />
<br />
3. run the lilo (if using lilo)<br />
<br />
<pre># lilo<br />
Added linux<br />
Added linux-up<br />
Added virtuozzo_old<br />
Added linux+virtuozzo *</pre><br />
<br />
4. reboot the server<br />
shutdown -r 23:05<br />
<br />
when it comes time for the shutdown, run [[VPS_Management#stopvirt|stopvirt]] to get things shut down faster<br />
<br />
<br />
= Remaking service VE (<=3.x only) =<br />
<br />
Occasionally the control panel for VEs (aka VZPP) will stop working and you just need to reinstall the service VE (which runs the control panels for all VEs on the HN). Here's how that's done :<br />
<br />
<pre>vzctl stop 1<br />
ipdel 1 69.55.234.152<br />
vzmlocal 1:2<br />
vzctl set 2 --save --onboot no<br />
vzsveinstall -t redhat-as3-minimal -d /root/Rel261/HW/RPMS -s 69.55.234.152 --skip-client</pre><br />
<br />
on 3.0:<br />
<br />
<pre>vzsveinstall -t redhat-as3-minimal -d /vz/Rel300/HW/RPMS -s 69.55.229.151 --skip-client<br />
<br />
vzctl stop 1<br />
vzctl mount 1<br />
vzctl mount 2<br />
/bin/cp -aRf /vz/root/2/home/vzagent0 /vz/root/1/home<br />
/bin/cp -aRf /vz/root/2/var/lib/mysql/VZADB /vz/root/1/var/lib/mysql<br />
/bin/cp -af /vz/root/2/etc/shadow /vz/root/1/etc/<br />
/bin/cp -aRf /vz/root/2/var/vzcp/static/vz/skins/ /vz/root/1/var/vzcp/static/vz<br />
/bin/cp -af /vz/root/2/etc/vzcp/pp/menu.xml /vz/root/1/etc/vzcp/pp/<br />
/bin/cp -af /vz/root/2/etc/vzcp/pp/dashboard.xml /vz/root/1/etc/vzcp/pp/<br />
<br />
vzctl umount 2<br />
vzctl umount 1<br />
vzctl start 1</pre><br />
<br />
<br />
= GRUB: boot once to a new kernel =<br />
<br />
To use a different kernel for the next reboot only:<br />
<pre>grub --no-floppy<br />
savedefault --default=N --once<br />
(where N is the number of the kernel you want to boot once)<br />
quit</pre><br />
<br />
Reboot your machine. It will boot using kernel "N".<br />
<br />
grub-reboot N<br />
will do essentially the same thing and prompt to reboot the machine<br />
<br />
<br />
= Recovering from GRUB failure =<br />
<br />
(example from DPE 2950)<br />
<br />
Boot to CEOS4 server CD (original install cd). Did this example with iso mounted via drac<br />
<br />
Boot: linux rescue<br />
<br />
<br />
= Make/edit an ISO on Linux =<br />
<br />
<pre>mount file.iso /mnt/iso/ -t iso9660 -o ro,loop=/dev/loop0<br />
mkdir /tmp/iso<br />
cp -aR /mnt/iso/* /tmp/iso/<br />
umount /mnt/iso/<br />
mkisofs -iso-level 2 -o /tmp/new.iso /tmp/iso/<br />
rm -fr /tmp/iso</pre><br />
<br />
To mount it:<br />
<br />
mount -t vfat -o loop /tmp/3wfirmware.iso /mnt/usb/<br />
<br />
http://www.damnsmalllinux.org/wiki/index.php/Installing_to_a_USB_Flash_Drive<br />
<br />
= Mount/edit USB drive on Linux =<br />
<br />
mount -t vfat -o loop /dev/sdc1 /mnt/usb/<br />
<br />
This example is for our floating usb drive on virt16. You should edit the device as necessary on other hardware.<br />
<br />
<br />
= Installing OS via net =<br />
<br />
Centos 5.1:<br><br />
http site name: <tt>vault.centos.org</tt><br><br />
dir: <tt>/5.1/os/i386/</tt><br />
<br />
Centos 5.4:<br><br />
http site name: <tt>mirrors.kernel.org</tt><br><br />
dir: <tt>/centos/5.4/os/i386/</tt><br><br />
or<br><br />
<tt>/centos/5.4/os/x86_64/</tt><br><br />
or<br><br />
<tt>http://mirrors.usc.edu/pub/linux/distributions/centos/5.4/os/i386/</tt><br><br />
<br />
Centos 6.2:<br><br />
<tt>http://mirrors.usc.edu/pub/linux/distributions/centos/6.2/os/x86_64/</tt><br><br />
<br />
FC9:<br><br />
<tt>http://linux.nssl.noaa.gov/fedora/linux/releases/9/Fedora/i386/os/</tt><br><br />
<br />
FC10:<br><br />
<tt>http://linux.nssl.noaa.gov/fedora/linux/releases/10/Fedora/i386/os/</tt><br></div>Supporthttps://wiki.jcihosting.com/index.php?title=E-Monitoring_notes&diff=1054E-Monitoring notes2013-03-12T00:09:37Z<p>Support: Created page with "= isys = runs mail for e-monitoring.net the key on root@mail should let you ssh w/o pass to isys.e-monitoring.net isys has a mail queue problem where it gets big and we run..."</p>
<hr />
<div>= isys =<br />
<br />
runs mail for e-monitoring.net<br />
<br />
the key on root@mail should let you ssh w/o pass to isys.e-monitoring.net<br />
<br />
isys has a mail queue problem where it gets big and we run out of inodes. we setup a cronjob to clear that out.<br />
<br />
= polling / magrathea = <br />
== config new IP ==<br />
Edit:<br />
<pre>/etc/defaultrouter<br />
/export/home/groundwater/bin/getit.pl<br />
/etc/hosts<br />
/etc/hostname.le1</pre><br />
<br />
On production:<br />
<pre>/home/htdocs/gw/index.pl<br />
/home/htdocs/isys/pollscan.pl<br />
/home/gw/shipley/etc/config.pl<br />
/home/gw/owrd/etc/config.pl<br />
/home/dboodman/www/isys/pollscan.pl<br />
Apache conf</pre><br />
<br />
== stop it from trying to dial out ==<br />
<pre>mysql> select * from resources;<br />
+------------+------------+----------------+-------+---------+----------+<br />
| resourceID | resource | connectionType | inUse | inUseBy | inUseFor |<br />
+------------+------------+----------------+-------+---------+----------+<br />
| 1 | /dev/cuaa0 | 1 | NULL | NULL | NULL |<br />
| 2 | network | 2 | NULL | NULL | NULL |<br />
| 3 | network | 2 | NULL | NULL | NULL |<br />
| 4 | network | 2 | NULL | NULL | NULL |<br />
| 5 | network | 2 | NULL | NULL | NULL |<br />
| 6 | network | 2 | NULL | NULL | NULL |<br />
| 7 | network | 2 | NULL | NULL | NULL |<br />
| 8 | network | 2 | NULL | NULL | NULL |<br />
| 9 | network | 2 | NULL | NULL | NULL |<br />
| 10 | network | 2 | NULL | NULL | NULL |<br />
| 11 | network | 2 | NULL | NULL | NULL |<br />
+------------+------------+----------------+-------+---------+----------+<br />
11 rows in set (0.00 sec)<br />
delete from resources where resourceID =1;</pre></div>Supporthttps://wiki.jcihosting.com/index.php?title=Main_Page&diff=1053Main Page2013-03-12T00:07:38Z<p>Support: </p>
<hr />
<div>* [[Special:AllPages|All Pages]]<br />
* [[Payment_System|Payment System]]<br />
* [[Management_System_/_Public_Website_/_Signup_/_Account_Manager|Management System / Public Website / Signup / Account Manager]]<br />
* [[New_Signups|New Signups]]<br />
* [[Shutting_Down_Service|Shutting Down Service]]<br />
* [[Routine_Maintenance|Routine Maintenance]]<br />
* Reference<br />
** [[VPS_Management|VPS Management]]<br />
** [[Jail_Server_Install|Jail Server Install]]<br />
** [[Infrastructure_Machines|Infrastructure Machines]]<br />
** [[Colo_Server_Setup|Colo Server Setup]]<br />
** [[DNS]]<br />
** [[FreeBSD_Reference|FreeBSD Reference]]<br />
** [[Virtuozzo_/_Linux_Reference|Virtuozzo / Linux Reference]]<br />
** [[Cabinetmap]]<br />
** [[Switch_Control|Switch Control]]<br />
** [[Private_IP_Mapping|Private IP Mapping]]<br />
** [[Data_Center_Notes|Data Center Notes]]<br />
** [[Crash_Reference|Crash Reference]]<br />
** [[Hardware_Reference|Hardware Reference]]<br />
** [[ATS]]<br />
** [[Wiring_Reference|Wiring Reference]]<br />
** [[Screen]]<br />
** [[RAID_Cards|RAID Cards]]<br />
** [[BigBrother]]<br />
** [[Bandwidth_Management|Bandwidth Management]]<br />
** [[Network_Time_(ntp)|Network Time (ntp)]]<br />
** [[Isys_-_out_of_inodes|Isys - out of inodes]]<br />
** [[DRAC/RMM|DRAC/RMM]]<br />
** [[Customer_Backups|Customer Backups]]<br />
** [[DoS_Attacks|DoS Attacks]]<br />
** [[IPKVM|IPKVM]]<br />
* [[System-generated_Notifications|System-generated Notifications]]<br />
* [[Customer_Interaction|Customer Interaction]]<br />
* [[Backup_Accounts_(rsync.net)| Backup Accounts (rsync.net)]]<br />
* [[E-Monitoring_notes|E-Monitoring notes]]<br />
* [[Todo]]<br />
<br />
<br />
== Getting started ==<br />
* [//www.mediawiki.org/wiki/Manual:Configuration_settings Configuration settings list]<br />
* [//www.mediawiki.org/wiki/Manual:FAQ MediaWiki FAQ]<br />
* [https://lists.wikimedia.org/mailman/listinfo/mediawiki-announce MediaWiki release mailing list]<br />
* [https://meta.wikimedia.org/wiki/Help:Advanced_editing Advanced Editing]</div>Supporthttps://wiki.jcihosting.com/index.php?title=DRAC/RMM&diff=1052DRAC/RMM2013-03-12T00:06:17Z<p>Support: /* DRAC */</p>
<hr />
<div>= DRAC =<br />
<br />
A list of all the DRAC cards and their IPs can be seen in the [[Private_IP_Mapping|ipmap]]<br />
<br />
To ssh in, login as root.<br />
<br />
Common commands:<br />
racadm serveraction hardreset<br />
performs a reset<br />
<br />
racadm serveraction powerdown<br />
power off server<br />
<br />
racadm serveraction powerup<br />
power on server<br />
<br />
racadm racreset<br />
reset the DRAC card (if kvm console becomes frozen)<br />
<br />
<br />
To get virtual media plugin to install, add site to trusted list and make security low<br />
<br />
Installing openmanage on CEOS4:<br />
<br />
Mount install cd from local drive<br />
<br />
<pre>mount /dev/scd0 /mnt<br />
cd /mnt/srvadmin/linux/RPMS/RHEL4<br />
cd /mnt/srvadmin/linux/supportscripts/<br />
./srvadmin-install.sh --express</pre><br />
<br />
Copy linux to disk<br />
Edit the script to make it think the os is RHEL4<br />
<br />
BIOS settings to enable serial:<br />
Serial comm.: <tt>on with cons. Redir via com1</tt><br />
Failsafe: <tt>115200</tt><br />
Redir after boot: <tt>enabled</tt><br />
<br />
To boot to usb, switch hd sequence in bios. <br />
<br />
After removing usb it will have to reorder boot seq to get virtual cd back<br />
<br />
Can’t get floppy/iso to boot- need to use thumb drive<br />
<br />
Latest f/w as of 2/16/09:<br />
<pre>Bios: 2.5.0<br />
Perc 6/i: 6.1.1-0047<br />
Drac: 1.4<br />
BMC: 2.37</pre><br />
<br />
= RMM =</div>Supporthttps://wiki.jcihosting.com/index.php?title=DRAC/RMM&diff=1051DRAC/RMM2013-03-12T00:02:36Z<p>Support: /* DRAC */</p>
<hr />
<div>= DRAC =<br />
<br />
A list of all the DRAC cards and their IPs can be seen in the [[IPmap|ipmap]]<br />
To ssh in, login as root.<br />
<br />
<br />
When DRAC console kvm gets frozen, ssh in and<br />
<br />
$ racadm racreset<br />
<br />
To fix login bug, disable roboform<br />
<br />
To get virtual media plugin to install, add site to trusted list and make security low<br />
-----------------<br />
<br />
FC4 won’t find the drives<br />
<br />
Installing openmanage on CEOS4:<br />
<br />
Mount install cd from local drive<br />
<br />
mount /dev/scd0 /mnt<br />
<br />
cd /mnt/srvadmin/linux/RPMS/RHEL4<br />
cd /mnt/srvadmin/linux/supportscripts/<br />
./srvadmin-install.sh --express<br />
<br />
Copy linux to disk<br />
Edit the script to make it think the os is RHEL4<br />
<br />
racadm serveraction hardreset<br />
racadm serveraction powerdown<br />
racadm serveraction powerup<br />
racadm racreset<br />
<br />
BIOS settings:<br />
Serial comm.: on with cons. Redir via com1<br />
Failsafe: 115200<br />
Redir after boot: enabled<br />
<br />
To boot to usb, switch hd sequence in bios. <br />
After removing usb it will have to reorder boot seq to get virtual cd back<br />
<br />
Can’t get floppy/iso to boot- need to use thumb drive<br />
<br />
Latest f/w as of 2/16/09:<br />
Bios: 2.5.0<br />
Perc 6/i: 6.1.1-0047<br />
Drac: 1.4<br />
BMC: 2.37<br />
<br />
= RMM =</div>Supporthttps://wiki.jcihosting.com/index.php?title=DRAC/RMM&diff=1050DRAC/RMM2013-03-12T00:02:22Z<p>Support: /* DRAC */</p>
<hr />
<div>= DRAC =<br />
<br />
A list of all the DRAC cards and their IPs can be seen in the [[ipmap|ipmap]]<br />
To ssh in, login as root.<br />
<br />
<br />
When DRAC console kvm gets frozen, ssh in and<br />
<br />
$ racadm racreset<br />
<br />
To fix login bug, disable roboform<br />
<br />
To get virtual media plugin to install, add site to trusted list and make security low<br />
-----------------<br />
<br />
FC4 won’t find the drives<br />
<br />
Installing openmanage on CEOS4:<br />
<br />
Mount install cd from local drive<br />
<br />
mount /dev/scd0 /mnt<br />
<br />
cd /mnt/srvadmin/linux/RPMS/RHEL4<br />
cd /mnt/srvadmin/linux/supportscripts/<br />
./srvadmin-install.sh --express<br />
<br />
Copy linux to disk<br />
Edit the script to make it think the os is RHEL4<br />
<br />
racadm serveraction hardreset<br />
racadm serveraction powerdown<br />
racadm serveraction powerup<br />
racadm racreset<br />
<br />
BIOS settings:<br />
Serial comm.: on with cons. Redir via com1<br />
Failsafe: 115200<br />
Redir after boot: enabled<br />
<br />
To boot to usb, switch hd sequence in bios. <br />
After removing usb it will have to reorder boot seq to get virtual cd back<br />
<br />
Can’t get floppy/iso to boot- need to use thumb drive<br />
<br />
Latest f/w as of 2/16/09:<br />
Bios: 2.5.0<br />
Perc 6/i: 6.1.1-0047<br />
Drac: 1.4<br />
BMC: 2.37<br />
<br />
= RMM =</div>Supporthttps://wiki.jcihosting.com/index.php?title=Virtuozzo_/_Linux_Reference&diff=1049Virtuozzo / Linux Reference2013-03-12T00:01:24Z<p>Support: /* Mount/edit USB drive on Linux */</p>
<hr />
<div>= Updating VZ =<br />
<br />
Update Virtuozzo to version 4.0.0<br />
Apply minor updates for Virtuozzo 3.0.0<br />
<br />
...<br />
When you get to the last screen, you will be given an option to <br />
<br />
( ) Automatically reboot after update<br />
(*) Reboot later manually<br />
<br />
make sure you choose to reboot later.<br />
<br />
= Migrating Templates =<br />
One nice feature VZ offers is the ability to run a virtual environment (VE or CT as it's now called) based on a particular ditribution on a host which may or may not share the same distribution. For instance, you could run a CT running Ubuntu while the host machine (Hardware Node or HN) is running CentOS. This is accomplished via the use of OS templates. As long as the HN contains the OS template on which a particular CT relies, it will run there. As such, if you want to move a CT from one HN to another, you will need to make sure the OS template exists there.<br />
<br />
Modern versions of VZ (>= 4.x) will check for and automatically move the OS template over to the host as you're (vz)migrating the CT over to a new HN. However we like to make sure the template is in place prior to moving the CT. <br />
<br />
The first step is to see if the template exists on the host. To see OS templates installed:<br />
<br />
2.x (shows OS and application templates):<br />
<pre># vzpkgls<br />
mod_ssl-rh9 20040624<br />
mysql-rh9 20030814 20050412<br />
openwebmail-rh9 20050427<br />
php-rh9 20050506<br />
phpBB-rh9 20041229<br />
postgresql-rh9 20050719<br />
sl-webalizer-rh9 20040119<br />
spamassassin-rh9 20040127<br />
tomcat-rh9 20040524<br />
usermin-rh9 20040116</pre><br />
<br />
>= 3.x (shows just OS templates, run without -O to see application as well):<br />
<pre># vzpkg list -O<br />
ubuntu-10.04-x86 2010-06-01 11:39:05<br />
ubuntu-11.04-x86 2011-06-22 14:23:59<br />
ubuntu-9.10-x86 2010-03-11 17:39:51<br />
centos-5-x86 2010-03-11 17:12:27<br />
debian-5.0-x86 2010-03-11 17:33:34</pre><br />
<br />
<br />
All template files are located:<br />
<br />
<pre># ls /vz/template/<br />
ColdFusion-fc1 jre-fc1 openwebmail-fc1 sl-webalizer-fc1<br />
ColdFusion-fc2 jre-fc2 openwebmail-fc2 sl-webalizer-fc2<br />
ColdFusion-rh9 jre-rh9 openwebmail-rh9 sl-webalizer-rh9<br />
PostNuke-fc1 jre-suse92 openwebmail-suse92 spamassassin-fc1<br />
PostNuke-fc2 jrun php-deb31 spamassassin-fc2<br />
PostNuke-rh9 mailman-deb31 php-fc1 spamassassin-rh9<br />
analog-fc1 mailman-fc1 php-fc2 spamassassin-suse92<br />
analog-fc2 mailman-fc2 php-rh9 suse-9.2<br />
analog-rh9 mailman-rh9 php-suse92 tomcat-fc1<br />
awstats-fc1 mailman-suse92 phpBB-fc1 tomcat-fc2<br />
awstats-rh9 mod_frontpage-fc1 phpBB-fc2 tomcat-rh9<br />
bbClone-fc1 mod_frontpage-fc2 phpBB-rh9 tomcat-suse92<br />
bbClone-fc2 mod_frontpage-rh9 phpmyadmin-deb31 urchin-fc1<br />
bbClone-rh9 mod_frontpage-suse92 postgresql-deb31 usermin-fc1<br />
conf mod_perl-deb31 postgresql-fc1 usermin-fc2<br />
debian-3.1 mod_perl-fc1 postgresql-fc2 usermin-rh9<br />
devel-deb31 mod_perl-fc2 postgresql-rh9 uw-imap-fc1<br />
devel-fc2 mod_perl-rh9 postgresql-suse92 uw-imap-fc2<br />
devel-suse92 mod_perl-suse92 proftpd-deb31 uw-imap-rh9<br />
fedora-core-1 mod_ssl-fc1 proftpd-fc1 vzredhat-7.3<br />
fedora-core-2 mod_ssl-fc2 proftpd-fc2 vzredhat-8.0<br />
jdk-deb31 mod_ssl-rh9 proftpd-rh9 webalizer-suse92<br />
jdk-fc1 mysql-deb31 proftpd-suse92 webmin-fc1<br />
jdk-fc2 mysql-fc1 redhat-7.2 webmin-fc2<br />
jdk-rh9 mysql-fc2 redhat-9 webmin-rh9<br />
jdk-suse92 mysql-rh9 redhat-as3-minimal<br />
jre-deb31 mysql-suse92 redhat-devel-9<br />
</pre><br />
<br />
What we see here are the base OS templates: <tt>redhat-9, fedora-core-1</tt> and supporting application templates: <tt>postgresql-rh9, mailman-rh9, jdk-fc1, mod_ssl-fc1</tt>. So if you wanted to copy over all of redhat 9 you'd need to copy the contents of:<br />
<pre># ls -d /vz/template/*rh9*<br />
/vz/template/ColdFusion-rh9 /vz/template/mod_frontpage-rh9 /vz/template/proftpd-rh9<br />
/vz/template/PostNuke-rh9 /vz/template/mod_perl-rh9 /vz/template/sl-webalizer-rh9<br />
/vz/template/analog-rh9 /vz/template/mod_ssl-rh9 /vz/template/spamassassin-rh9<br />
/vz/template/awstats-rh9 /vz/template/mysql-rh9 /vz/template/tomcat-rh9<br />
/vz/template/bbClone-rh9 /vz/template/openwebmail-rh9 /vz/template/usermin-rh9<br />
/vz/template/jdk-rh9 /vz/template/php-rh9 /vz/template/uw-imap-rh9<br />
/vz/template/jre-rh9 /vz/template/phpBB-rh9 /vz/template/webmin-rh9<br />
/vz/template/mailman-rh9 /vz/template/postgresql-rh9<br />
<br />
# ls -d /vz/template/*redhat-9*<br />
/vz/template/redhat-9<br />
</pre><br />
<br />
To get a template from one HN to another, it simply needs to be rsync'd over to the target HN. It's rsync'd over in this fashion:<br />
<br />
rsync -av -e ssh /vz/template/redhat-9 root@10.1.4.65:/vz/template<br />
rsync -av -e ssh /vz/template/*rh9 root@10.1.4.65:/vz/template<br />
<br />
Newer (>= 3.x) VZ employs "EZ" templates which are organized a little differently:<br />
<br />
<pre># ls /vz/template/<br />
cache CmdTool.log debian redhat-as3-minimal-20080630.tar.gz vc<br />
centos conf redhat-as3-minimal ubuntu<br />
<br />
# ls /vz/template/ubuntu/<br />
10.04 11.04 9.10<br />
# ls /vz/template/ubuntu/10.04/<br />
x86<br />
# ls /vz/template/ubuntu/10.04/x86/<br />
aap_1.091-1_all<br />
aap-doc_1.091-1_all<br />
adduser_3.112ubuntu1_all<br />
anacron_2.3-13.1ubuntu11_i386<br />
apache2_2.2.14-5ubuntu8.2_i386<br />
apache2_2.2.14-5ubuntu8.4_i386<br />
apache2_2.2.14-5ubuntu8.6_i386<br />
apache2_2.2.14-5ubuntu8.7_i386<br />
apache2_2.2.14-5ubuntu8.8_i386<br />
apache2_2.2.14-5ubuntu8.9_i386<br />
<...snip><br />
<br />
# ls /vz/template/cache/<br />
centos-5-x86.tar.gz suse-11.3-x86.tar.gz ubuntu-9.10-x86.tar.gz<br />
debian-5.0-x86.tar.gz ubuntu-10.04-x86.tar.gz<br />
fedora-core-14-x86.tar.gz ubuntu-11.04-x86.tar.gz<br />
</pre><br />
<br />
So what we see is the entire OS and all applications are in 1 directory, and organized by distribution and release version. So to move Ubuntu 10.04:<br />
<br />
rsync -av -e ssh /vz/template/ubuntu/10.04 root@10.1.4.69:/vz/template/ubuntu<br />
<br />
and you also need to move the cache:<br />
<br />
rsync -av -e ssh /vz/template/cache/ubuntu-10.04-x86.tar.gz root@10.1.4.69:/vz/template/cache/<br />
<br />
Once the transfer is complete, you will be able to run <tt>vzpkgls</tt> or <tt>vzpkg list -O</tt> and see the template(s) listed on the target HN.<br />
<br />
So far, this all supposes template doesn't exist on the target- very simple, just copy it over. If the template exists already on the target HN, this requires closer inspection. With older templates, simply moving over the templates would be the end of it- you see the version listed when you do a <tt>vzplgls</tt>:<br />
<br />
mysql-rh9 20030814 20050412<br />
<br />
this means there are 2 versions of the mysql package for RH9: 20030814 and 20050412<br />
As an aside, you can't copy over just 1 of them, but if you rsync'd over the <tt>mysql-rh9</tt> directory, you'd see 20030814 and 20050412 on the target HN. As long as the versions match up, you're good to go.<br />
<br />
With EZ templates, the versions are not as easy to decipher. When you look at the <tt>vzpkg list</tt> output, that simply tells you when a cache was created. A cache is simply a snapshot of all the data needed for the latest versions of the OS and it's applications at the time the cache was created. It is a date, and thus tells you nothing about the versions within. So for instance, if you had a cache for Ubuntu 10.04 from 2007 and you ran <tt>vzpkg create cache ubuntu-10.04-x86</tt> it would re-download the latest version of apache, mysql and so on. And any ''subsequent'' ubuntu 10.04 CT's created thereafter would use those newer versions, whereas older CT's based on 10.04 would use older templates. You can see how this works by looking at the files under:<br />
<br />
<pre># ls /vz/template/ubuntu/10.04/x86/<br />
aap_1.091-1_all<br />
aap-doc_1.091-1_all<br />
adduser_3.112ubuntu1_all<br />
anacron_2.3-13.1ubuntu11_i386<br />
apache2_2.2.14-5ubuntu8.2_i386<br />
apache2_2.2.14-5ubuntu8.4_i386<br />
apache2_2.2.14-5ubuntu8.6_i386<br />
apache2_2.2.14-5ubuntu8.7_i386<br />
apache2_2.2.14-5ubuntu8.8_i386<br />
apache2_2.2.14-5ubuntu8.9_i386<br />
<...snip><br />
</pre><br />
<br />
You can see that this template has in it 6 different versions of apache2. This means that this template was merged with other 10.04 templates and/or it has been cached a few times, each time a new apache was available. The unfortunate thing is it's almost impossible to know which CT's are using which version of apache so it's hard to try to prune this down and remove unwanted/unneeded applications.<br />
<br />
So, if you see another HN has Ubuntu 10.04 on it, you need to see/confirm that it has all the exact files your CT needs. You can run a test rsync to see what ''would'' be transferred (what's missing):<br />
<br />
rsync -avn --stats -e ssh /vz/template/ubuntu/10.04 root@10.1.4.69:/vz/template/ubuntu<br />
<br />
Based on the amount of data you get back from the rsync, you will be able to decide whether or not to remove the -n and allow the full transfer to go through. The downside to doing the transfer is the situation described above- you've permanently joined these templates and now you may have 10 versions of apache on the target HN. There's one other potential pitfall to this kind of merging- it will also potentially copy over shared files, for which there is no particular version, most of which are in <tt>/vz/template/ubuntu/10.04/x86/config</tt>: <br />
<br />
cat /vz/template/ubuntu/10.04/x86/config/os/default/repositories<br />
/vz/template/ubuntu/10.04/x86/config/os/default/repositories<br />
<br />
Here are some specific things to look out for. Case: copying centos-5 on virt19 to virt12. We dumped the rsync output to a file so we could inspect:<br />
<br />
[root@virt19 /vz/template]# rsync -an -e ssh /vz/template/centos/ root@10.1.4.62:/vz/template/centos/ > v12_c5<br />
<br />
(note, that can be dangerous, better to add on full path <tt>/vz/template/centos/5</tt>)<br />
<br />
So in the output file we see things like:<br />
<br />
<pre><br />
x86/base0/cachecookie<br />
x86/base0/primary.xml.gz <br />
x86/base0/primary.xml.gz.sqlite <br />
x86/base0/repomd.xml <br />
x86/base1/cachecookie <br />
x86/base1/filelists.xml.gz <br />
x86/base1/filelists.xml.gz.sqlite <br />
x86/base1/primary.xml.gz <br />
x86/base1/primary.xml.gz.sqlite <br />
x86/base1/repomd.xml<br />
x86/base2/cachecookie<br />
</pre><br />
<br />
Which make us nervous cause those are not versioned files, i.e.:<br />
5/x86/MAKEDEV-3.23-1.2.i386/usr/sbin/mksock<br />
<br />
So those files will replace existing files on v12 rather than the case of the latter where it's likely being added. So caution should be given and deference to whichever version is newer (which rsync should take care of, but that will depend on the date each file was created and perhaps not the actual data within).<br />
<br />
If we look further in the file we see:<br />
<br />
x86/psa-appvault-moodle-1.8-8202920080409011254.noarch/usr/local/psa/var/cgitory/moodle-1.<br />
9/htdocs/lib/editor/htmlarea/plugins/TableOperations/img/cell-split.gif<br />
<br />
and other files looking like:<br />
5/x86/plesk-base-9.0.1-cos5.build90090127.18.i586/etc/sw/keys/info<br />
<br />
Which means plesk is here. We don't need plesk on v12 (the CT moving over doesn't use it) so we rerun the rsync and exclude it:<br />
<br />
[root@virt19 /vz/template]# rsync -an -e ssh --exclude="plesk*" --exclude="psa*" /vz/template/centos/ root@10.1.4.62:/vz/template/centos/ > v12_c5<br />
<br />
and now it's much smaller:<br />
<br />
[root@virt19 /vz/template]# cat v12_c5|wc -l<br />
97104<br />
<br />
Before running with excludes it was:<br />
[root@virt19 /vz/template]# cat v12_c5|wc -l <br />
238238<br />
<br />
So again, look at what you're doing. Generally, combining templates is fine however.<br />
<br />
If you're happy with the merge, run the rsync again without the "-n" to let it actually do the transfer.<br />
<br />
Once done, don't forget to add the new template to the HN in Management -> Reference -> Templates<br />
<br />
= getting help =<br />
<br />
= vzup2date =<br />
TODO <br />
= Adding new OS/application templates =<br />
<br />
Virtuozzo will lag a bit behind the initial release of an OS, perhaps by a month or more. Further, a newly-released OS may be available, but there may not be many/any application templates available initially. It pays to wait a bit.<br />
<br />
A template is easily installed via [[#vzup2date|vzup2date]]:<br />
<br />
vzup2date -z<br />
<br />
You will be given a list of OS's to install. Select an OS, then customize that selection by checking/unchecking the application templates we do/don't want to include/offer.<br />
<br />
Generally, here are the app templates we include/offer:<br />
<pre>cyrus-imap<br />
devel<br />
imp<br />
jre<br />
jsdk<br />
mailman<br />
mod_perl<br />
mod_ssl<br />
mysql<br />
php<br />
phpmyadmin<br />
phppgadmin<br />
postgresql<br />
proftpd<br />
spamassassin<br />
squirrelmail<br />
tomcat<br />
vsftpd<br />
</pre><br />
<br />
We do not (any longer) offer webalizer or analog.<br />
<br />
After selecting and installing the OS and apps, you should ensure it's installed:<br />
<br />
vzpkg list -O<br />
<br />
Your new OS (ex. fedora-core-13-x86_64) will be listed, without a corresponding cache date:<br />
<pre><br />
[root@virt13 /vzconf]# vzpkg list -O<br />
centos-6-x86 2011-07-22 13:24:41<br />
centos-6-x86_64 2012-03-09 20:42:22<br />
centos-5-x86 2009-07-27 16:58:24<br />
centos-5-x86_64 2012-03-09 22:38:53<br />
ubuntu-11.10-x86_64 2012-03-01 23:13:33<br />
ubuntu-9.04-x86 2009-06-02 11:32:39<br />
ubuntu-12.04-x86_64 2012-05-29 13:50:50<br />
ubuntu-8.04-x86 2008-09-17 21:27:20<br />
ubuntu-8.10-x86 2009-03-13 13:58:57<br />
ubuntu-11.04-x86 2011-12-05 11:15:46<br />
ubuntu-7.04-x86 2008-09-24 16:42:50<br />
redhat-el6-x86_64<br />
fedora-core-13-x86_64 <br />
fedora-core-12-x86 2010-02-06 13:24:32<br />
fedora-core-15-x86 2011-12-05 11:36:43<br />
fedora-core-11-x86 2009-10-01 16:30:59<br />
fedora-core-14-x86<br />
fedora-core-16-x86_64 2012-03-13 11:40:23<br />
fedora-core-9-x86 2008-11-29 10:52:18<br />
debian-6.0-x86 2011-07-29 16:00:35<br />
debian-6.0-x86_64 2012-03-12 19:53:33<br />
debian-5.0-x86 2009-03-12 12:39:11<br />
</pre><br />
<br />
You can also confirm the apps installed:<br />
<pre>[root@virt13 /vzconf]# vzpkg list fedora-core-13-x86_64<br />
fedora-core-13-x86_64 2013-03-08 11:39:44<br />
fedora-core-13-x86_64 devel<br />
fedora-core-13-x86_64 php<br />
fedora-core-13-x86_64 proftpd<br />
fedora-core-13-x86_64 postgresql<br />
fedora-core-13-x86_64 phpmyadmin<br />
fedora-core-13-x86_64 mysql<br />
</pre><br />
<br />
Before you can use the OS you must cache it. To create the cache run:<br />
vzpkg cache [OS]<br />
<br />
ex: <tt>vzpkg cache fedora-core-13-x86_64</tt><br />
<br />
Now it's ready to use. In order to be able to use it with the [[VPS_Management#vm|vm]] command, you must create a file:<br />
jctmpl-[OS]<br />
<br />
ex: <tt>jctmpl-fedora-core-13-x86_64</tt><br />
<br />
In it, on line 1 we place the OS, followed by the app names, ex:<br />
<pre>more jctmpl-fedora-core-13-x86_64<br />
fedora-core-13-x86_64<br />
postgresql<br />
jsdk<br />
mod_ssl<br />
phpmyadmin<br />
mod_perl<br />
tomcat<br />
devel<br />
php<br />
mailman<br />
cyrus-imap<br />
squirrelmail<br />
proftpd<br />
mysql<br />
phppgadmin<br />
spamassassin<br />
</pre><br />
<br />
The packages will be installed in that order, so any pre-req's should be handled there.<br />
<br />
Now, when you run [[VPS_Management#vm|vm]] it will show you the newly-added OS and you may create a CT using it.<br />
<br />
The last few tasks are related to mgmt:<br />
<br />
1. add to mgmt->reference->templates (use the existing templates as an example). Once added, the new template will be included with the list of OS's available for the given HN.<br />
2. if this is going to be a new OS offered for ordering, it needs to be added to the order form and webpage.<br />
<br />
a. to get a list of applications and versions in the new OS, you should create a test CT, enter it and run <tt>dpkg -l|sort</tt> (or <tt>rpm -aq|sort</tt> in centos/fedora) and enter that list in the following places:<br><br />
<tt>/usr/local/www/jc_pub/[osname].html</tt><br><br />
<tt>/usr/local/www/signup/html/[osname].html</tt><br><br />
Note, you will have to move the most recent OS data from <tt>/usr/local/www/signup/html/[osname].html</tt> into the corresponding <tt>/usr/local/www/signup/html/[osname]_old.html</tt> updating the version links in both.<br />
<br />
b. to add the new OS to the order form, you will need to update <tt>/usr/local/www/signup/html/step1.html</tt> in several places (for regular and OSS orders) as well as updating the templateID which you can get from mgmt->reference->templates (or [[Management_System_/_Public_Website_/_Signup_/_Account_Manager#jc:ref_templates|jc:ref_templates]])<br />
<br />
<br />
<br />
Old method to obtain the templates:<br />
<br />
Go to http://www.parallels.com/en/virtuozzo/templates/catalog/<br />
<br />
Download the RPM's and install them<br />
<br />
vzpkg list<br />
to see the new packages/os templates<br />
<br />
vzpkg create cache "OS_EZ_template_name"<br />
to create a cache<br />
<br />
<br />
= Update OS template =<br />
<br />
Installation and update of the new version of OS template will not affect the existing containers. It will affect only the new containers, created after this update operation. In terms of fixing "vzctl enter" error and SSH connectivity to Ubuntu based containers, it is necessary to recreate the template and not to update it, so the commands to run are:<br />
<br />
<pre>rpm -Uhv ubuntu-8.04-x86-ez-4.0.0-9.swsoft.noarch.rpm<br />
vzpkg clean ubuntu-8.04-x86<br />
vzpkg remove cache ubuntu-8.04-x86<br />
vzpkg create cache ubuntu-8.04-x86</pre><br />
<br />
<br />
= Installing new SSL cert on VZCP =<br />
<br />
<pre>openssl req -days 1999 -new -x509 -nodes -out server.crt -keyout server.key<br />
<br />
US<br />
CA<br />
San Diego<br />
johncompanies.com<br />
johncompanies.com<br />
virt15ohncompanies.com<br />
linux@johncompanies.com<br />
<br />
cp -p /etc/httpd/conf/ssl.crt/server.crt /etc/httpd/conf/ssl.crt/server.crt.bak<br />
cp -p /etc/httpd/conf/ssl.key/server.key /etc/httpd/conf/ssl.key/server.key.bak<br />
cp server.crt /etc/httpd/conf/ssl.crt/server.crt<br />
cp server.key /etc/httpd/conf/ssl.key/server.key<br />
/etc/init.d/vzcp restart</pre><br />
<br />
= Moving virt (drives) to new hardware =<br />
<br />
If you have a case where a virt is dead (due to hardware failure), you can move the drives to a new hardware setup.<br />
Here are the steps and things to look for:<br />
<br />
Move the drives to the new box. Most likely, the drives will be recognized and there will be no problems booting up. On Dell PERC 5/6 cards you will need to import the new "foreign" volume via the raid BIOS.<br />
<br />
If the hardware (motherboard) is different, it’s possible the nic’s won’t be recognized. Look at /etc/modules.conf the ethernet drivers are either:<br />
<br />
<pre>alias eth0 tg3<br />
alias eth1 tg3</pre><br />
(supermicro)<br />
<br />
Or<br />
<br />
<pre>alias eth0 e100<br />
alias eth1 e1000</pre><br />
(intel)<br />
<br />
Or<br />
<br />
<pre>alias eth0 bnx2<br />
alias eth1 bnx2</pre><br />
(dell 29500)<br />
<br />
Also, you may see eth0 and eth1 swapped so if after confirming the eth devices are present (may require a reboot), then you may still need to swap the green/yellow network cables in the back to get them on the right port.<br />
<br />
VZ will not run (for long) on the new hardware, so you will need to get the license reissued. You may create/download a temporary license via their website for the interim.<br />
<br />
= Updating kernel on virt =<br />
<br />
We now use [[#vzup2date|vzup2date]] to update systems and kernels. What follows is what we used to do when we had to download kernels and install manually on old systems:<br />
<br />
<br />
1. install kernel and modules<br />
<pre># rpm -ivh vzkernel-enterprise-2.4.20-021stab028.12.777.i686.rpm \<br />
vzmodules-enterprise-2.4.20-021stab028.12.swsoft.i686.rpm<br />
vzkernel-smp ##################################################<br />
vzmodules-smp ##################################################</pre><br />
<br />
DO NOT USE the "rpm -Uhv" command to install the kernel, otherwise all the previously installed kernels may be removed from your system. <br />
<br />
2. update lilo/grub<br />
<br />
Lilo:<br />
<br />
vi /etc/lilo.conf<br />
<br />
rename old Virtuozzo kernel label from "linux+virtuozzo" to "virtuozzo_old"<br />
and add a new entry pointing to the newly installed virtuozzo kernel image.<br />
For example:<br />
<br />
<pre>prompt<br />
timeout=50<br />
default=linux+virtuozzo<br />
append="debug panic=5 console=tty console=ttyS0,38400"<br />
boot=/dev/sda<br />
map=/boot/map<br />
install=/boot/boot.b<br />
message=/boot/message<br />
linear<br />
<br />
image=/boot/vmlinuz-2.4.18-3smp<br />
label=linux<br />
initrd=/boot/initrd-2.4.18-3smp.img<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.18-3<br />
label=linux-up<br />
initrd=/boot/initrd-2.4.18-3.img<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-020stab009.5.777-enterprise<br />
label=2.4.20-9.5.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-020stab009.8.777-enterprise<br />
label=2.4.20-9.8.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-020stab009.21.777-enterprise<br />
label=2.4.20-9.21.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-020stab009.24.777-enterprise<br />
label=2.4.20-9.24.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-021stab028.3.777-enterprise<br />
label=2.4.20-28.3.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-021stab028.5.777-enterprise<br />
label=old_linux+vz<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-021stab028.12.777-enterprise<br />
label=linux+virtuozzo<br />
read-only<br />
root=/dev/sda1</pre><br />
<br />
With this configuration, the new kernel is the default kernel to use at boot. The original kernel entry has been retained with the label "old_linux+vz ".<br />
<br />
Grub:<br />
<br />
vi /boot/grub /menu.lst<br />
<br />
Add new entry at the top.<br />
For example:<br />
<br />
<pre>serial --unit=0 --speed=38400<br />
terminal --timeout=10 serial console<br />
default=0<br />
timeout=10<br />
splashimage=(hd0,0)/boot/grub/splash.xpm.gz<br />
title Virtuozzo 2.6.1 (2.4.20-021stab028.12.777-enterprise)<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-22.4.20-021stab028.12.777-enterprise root=/dev/sda1 console=tty0 \ console=ttyS0,38400<br />
title Virtuozzo 2.6.1 (2.4.20-021stab028.3.777-enterprise)<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.20-021stab028.3.777-enterprise root=/dev/sda1 console=tty0 \ console=ttyS0,38400<br />
title Red Hat Linux (2.4.18-3smp)<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.18-3smp ro root=/dev/sda1<br />
initrd /boot/initrd-2.4.18-3smp.img<br />
title Red Hat Linux-up (2.4.18-3)<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.18-3 ro root=/dev/sda1<br />
initrd /boot/initrd-2.4.18-3.img<br />
title Linux with Virtuozzo 2.5<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.20-020stab009.3.777-smp ro root=/dev/sda1<br />
title vz-enterpriseo<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.20-020stab009.21.777-enterprise ro root=/dev/sda1<br />
title vz-enterprise<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.20-020stab009.24.777-enterprise ro root=/dev/sda1 \ console=tty0 console=ttyS0,38400</pre><br />
<br />
3. run the lilo (if using lilo)<br />
<br />
<pre># lilo<br />
Added linux<br />
Added linux-up<br />
Added virtuozzo_old<br />
Added linux+virtuozzo *</pre><br />
<br />
4. reboot the server<br />
shutdown -r 23:05<br />
<br />
when it comes time for the shutdown, run [[VPS_Management#stopvirt|stopvirt]] to get things shut down faster<br />
<br />
<br />
= Remaking service VE (<=3.x only) =<br />
<br />
Occasionally the control panel for VEs (aka VZPP) will stop working and you just need to reinstall the service VE (which runs the control panels for all VEs on the HN). Here's how that's done :<br />
<br />
<pre>vzctl stop 1<br />
ipdel 1 69.55.234.152<br />
vzmlocal 1:2<br />
vzctl set 2 --save --onboot no<br />
vzsveinstall -t redhat-as3-minimal -d /root/Rel261/HW/RPMS -s 69.55.234.152 --skip-client</pre><br />
<br />
on 3.0:<br />
<br />
<pre>vzsveinstall -t redhat-as3-minimal -d /vz/Rel300/HW/RPMS -s 69.55.229.151 --skip-client<br />
<br />
vzctl stop 1<br />
vzctl mount 1<br />
vzctl mount 2<br />
/bin/cp -aRf /vz/root/2/home/vzagent0 /vz/root/1/home<br />
/bin/cp -aRf /vz/root/2/var/lib/mysql/VZADB /vz/root/1/var/lib/mysql<br />
/bin/cp -af /vz/root/2/etc/shadow /vz/root/1/etc/<br />
/bin/cp -aRf /vz/root/2/var/vzcp/static/vz/skins/ /vz/root/1/var/vzcp/static/vz<br />
/bin/cp -af /vz/root/2/etc/vzcp/pp/menu.xml /vz/root/1/etc/vzcp/pp/<br />
/bin/cp -af /vz/root/2/etc/vzcp/pp/dashboard.xml /vz/root/1/etc/vzcp/pp/<br />
<br />
vzctl umount 2<br />
vzctl umount 1<br />
vzctl start 1</pre><br />
<br />
<br />
= GRUB: boot once to a new kernel =<br />
<br />
To use a different kernel for the next reboot only:<br />
<pre>grub --no-floppy<br />
savedefault --default=N --once<br />
(where N is the number of the kernel you want to boot once)<br />
quit</pre><br />
<br />
Reboot your machine. It will boot using kernel "N".<br />
<br />
grub-reboot N<br />
will do essentially the same thing and prompt to reboot the machine<br />
<br />
<br />
= Recovering from GRUB failure =<br />
<br />
(example from DPE 2950)<br />
<br />
Boot to CEOS4 server CD (original install cd). Did this example with iso mounted via drac<br />
<br />
Boot: linux rescue<br />
<br />
<br />
= Make/edit an ISO on Linux =<br />
<br />
<pre>mount file.iso /mnt/iso/ -t iso9660 -o ro,loop=/dev/loop0<br />
mkdir /tmp/iso<br />
cp -aR /mnt/iso/* /tmp/iso/<br />
umount /mnt/iso/<br />
mkisofs -iso-level 2 -o /tmp/new.iso /tmp/iso/<br />
rm -fr /tmp/iso</pre><br />
<br />
To mount it:<br />
<br />
mount -t vfat -o loop /tmp/3wfirmware.iso /mnt/usb/<br />
<br />
http://www.damnsmalllinux.org/wiki/index.php/Installing_to_a_USB_Flash_Drive<br />
<br />
= Mount/edit USB drive on Linux =<br />
<br />
mount -t vfat -o loop /dev/sdc1 /mnt/usb/<br />
<br />
This example is for our floating usb drive on virt16. You should edit the device as necessary on other hardware.</div>Supporthttps://wiki.jcihosting.com/index.php?title=Virtuozzo_/_Linux_Reference&diff=1048Virtuozzo / Linux Reference2013-03-12T00:00:35Z<p>Support: /* Mount/edit USB drive on Linux */</p>
<hr />
<div>= Updating VZ =<br />
<br />
Update Virtuozzo to version 4.0.0<br />
Apply minor updates for Virtuozzo 3.0.0<br />
<br />
...<br />
When you get to the last screen, you will be given an option to <br />
<br />
( ) Automatically reboot after update<br />
(*) Reboot later manually<br />
<br />
make sure you choose to reboot later.<br />
<br />
= Migrating Templates =<br />
One nice feature VZ offers is the ability to run a virtual environment (VE or CT as it's now called) based on a particular ditribution on a host which may or may not share the same distribution. For instance, you could run a CT running Ubuntu while the host machine (Hardware Node or HN) is running CentOS. This is accomplished via the use of OS templates. As long as the HN contains the OS template on which a particular CT relies, it will run there. As such, if you want to move a CT from one HN to another, you will need to make sure the OS template exists there.<br />
<br />
Modern versions of VZ (>= 4.x) will check for and automatically move the OS template over to the host as you're (vz)migrating the CT over to a new HN. However we like to make sure the template is in place prior to moving the CT. <br />
<br />
The first step is to see if the template exists on the host. To see OS templates installed:<br />
<br />
2.x (shows OS and application templates):<br />
<pre># vzpkgls<br />
mod_ssl-rh9 20040624<br />
mysql-rh9 20030814 20050412<br />
openwebmail-rh9 20050427<br />
php-rh9 20050506<br />
phpBB-rh9 20041229<br />
postgresql-rh9 20050719<br />
sl-webalizer-rh9 20040119<br />
spamassassin-rh9 20040127<br />
tomcat-rh9 20040524<br />
usermin-rh9 20040116</pre><br />
<br />
>= 3.x (shows just OS templates, run without -O to see application as well):<br />
<pre># vzpkg list -O<br />
ubuntu-10.04-x86 2010-06-01 11:39:05<br />
ubuntu-11.04-x86 2011-06-22 14:23:59<br />
ubuntu-9.10-x86 2010-03-11 17:39:51<br />
centos-5-x86 2010-03-11 17:12:27<br />
debian-5.0-x86 2010-03-11 17:33:34</pre><br />
<br />
<br />
All template files are located:<br />
<br />
<pre># ls /vz/template/<br />
ColdFusion-fc1 jre-fc1 openwebmail-fc1 sl-webalizer-fc1<br />
ColdFusion-fc2 jre-fc2 openwebmail-fc2 sl-webalizer-fc2<br />
ColdFusion-rh9 jre-rh9 openwebmail-rh9 sl-webalizer-rh9<br />
PostNuke-fc1 jre-suse92 openwebmail-suse92 spamassassin-fc1<br />
PostNuke-fc2 jrun php-deb31 spamassassin-fc2<br />
PostNuke-rh9 mailman-deb31 php-fc1 spamassassin-rh9<br />
analog-fc1 mailman-fc1 php-fc2 spamassassin-suse92<br />
analog-fc2 mailman-fc2 php-rh9 suse-9.2<br />
analog-rh9 mailman-rh9 php-suse92 tomcat-fc1<br />
awstats-fc1 mailman-suse92 phpBB-fc1 tomcat-fc2<br />
awstats-rh9 mod_frontpage-fc1 phpBB-fc2 tomcat-rh9<br />
bbClone-fc1 mod_frontpage-fc2 phpBB-rh9 tomcat-suse92<br />
bbClone-fc2 mod_frontpage-rh9 phpmyadmin-deb31 urchin-fc1<br />
bbClone-rh9 mod_frontpage-suse92 postgresql-deb31 usermin-fc1<br />
conf mod_perl-deb31 postgresql-fc1 usermin-fc2<br />
debian-3.1 mod_perl-fc1 postgresql-fc2 usermin-rh9<br />
devel-deb31 mod_perl-fc2 postgresql-rh9 uw-imap-fc1<br />
devel-fc2 mod_perl-rh9 postgresql-suse92 uw-imap-fc2<br />
devel-suse92 mod_perl-suse92 proftpd-deb31 uw-imap-rh9<br />
fedora-core-1 mod_ssl-fc1 proftpd-fc1 vzredhat-7.3<br />
fedora-core-2 mod_ssl-fc2 proftpd-fc2 vzredhat-8.0<br />
jdk-deb31 mod_ssl-rh9 proftpd-rh9 webalizer-suse92<br />
jdk-fc1 mysql-deb31 proftpd-suse92 webmin-fc1<br />
jdk-fc2 mysql-fc1 redhat-7.2 webmin-fc2<br />
jdk-rh9 mysql-fc2 redhat-9 webmin-rh9<br />
jdk-suse92 mysql-rh9 redhat-as3-minimal<br />
jre-deb31 mysql-suse92 redhat-devel-9<br />
</pre><br />
<br />
What we see here are the base OS templates: <tt>redhat-9, fedora-core-1</tt> and supporting application templates: <tt>postgresql-rh9, mailman-rh9, jdk-fc1, mod_ssl-fc1</tt>. So if you wanted to copy over all of redhat 9 you'd need to copy the contents of:<br />
<pre># ls -d /vz/template/*rh9*<br />
/vz/template/ColdFusion-rh9 /vz/template/mod_frontpage-rh9 /vz/template/proftpd-rh9<br />
/vz/template/PostNuke-rh9 /vz/template/mod_perl-rh9 /vz/template/sl-webalizer-rh9<br />
/vz/template/analog-rh9 /vz/template/mod_ssl-rh9 /vz/template/spamassassin-rh9<br />
/vz/template/awstats-rh9 /vz/template/mysql-rh9 /vz/template/tomcat-rh9<br />
/vz/template/bbClone-rh9 /vz/template/openwebmail-rh9 /vz/template/usermin-rh9<br />
/vz/template/jdk-rh9 /vz/template/php-rh9 /vz/template/uw-imap-rh9<br />
/vz/template/jre-rh9 /vz/template/phpBB-rh9 /vz/template/webmin-rh9<br />
/vz/template/mailman-rh9 /vz/template/postgresql-rh9<br />
<br />
# ls -d /vz/template/*redhat-9*<br />
/vz/template/redhat-9<br />
</pre><br />
<br />
To get a template from one HN to another, it simply needs to be rsync'd over to the target HN. It's rsync'd over in this fashion:<br />
<br />
rsync -av -e ssh /vz/template/redhat-9 root@10.1.4.65:/vz/template<br />
rsync -av -e ssh /vz/template/*rh9 root@10.1.4.65:/vz/template<br />
<br />
Newer (>= 3.x) VZ employs "EZ" templates which are organized a little differently:<br />
<br />
<pre># ls /vz/template/<br />
cache CmdTool.log debian redhat-as3-minimal-20080630.tar.gz vc<br />
centos conf redhat-as3-minimal ubuntu<br />
<br />
# ls /vz/template/ubuntu/<br />
10.04 11.04 9.10<br />
# ls /vz/template/ubuntu/10.04/<br />
x86<br />
# ls /vz/template/ubuntu/10.04/x86/<br />
aap_1.091-1_all<br />
aap-doc_1.091-1_all<br />
adduser_3.112ubuntu1_all<br />
anacron_2.3-13.1ubuntu11_i386<br />
apache2_2.2.14-5ubuntu8.2_i386<br />
apache2_2.2.14-5ubuntu8.4_i386<br />
apache2_2.2.14-5ubuntu8.6_i386<br />
apache2_2.2.14-5ubuntu8.7_i386<br />
apache2_2.2.14-5ubuntu8.8_i386<br />
apache2_2.2.14-5ubuntu8.9_i386<br />
<...snip><br />
<br />
# ls /vz/template/cache/<br />
centos-5-x86.tar.gz suse-11.3-x86.tar.gz ubuntu-9.10-x86.tar.gz<br />
debian-5.0-x86.tar.gz ubuntu-10.04-x86.tar.gz<br />
fedora-core-14-x86.tar.gz ubuntu-11.04-x86.tar.gz<br />
</pre><br />
<br />
So what we see is the entire OS and all applications are in 1 directory, and organized by distribution and release version. So to move Ubuntu 10.04:<br />
<br />
rsync -av -e ssh /vz/template/ubuntu/10.04 root@10.1.4.69:/vz/template/ubuntu<br />
<br />
and you also need to move the cache:<br />
<br />
rsync -av -e ssh /vz/template/cache/ubuntu-10.04-x86.tar.gz root@10.1.4.69:/vz/template/cache/<br />
<br />
Once the transfer is complete, you will be able to run <tt>vzpkgls</tt> or <tt>vzpkg list -O</tt> and see the template(s) listed on the target HN.<br />
<br />
So far, this all supposes template doesn't exist on the target- very simple, just copy it over. If the template exists already on the target HN, this requires closer inspection. With older templates, simply moving over the templates would be the end of it- you see the version listed when you do a <tt>vzplgls</tt>:<br />
<br />
mysql-rh9 20030814 20050412<br />
<br />
this means there are 2 versions of the mysql package for RH9: 20030814 and 20050412<br />
As an aside, you can't copy over just 1 of them, but if you rsync'd over the <tt>mysql-rh9</tt> directory, you'd see 20030814 and 20050412 on the target HN. As long as the versions match up, you're good to go.<br />
<br />
With EZ templates, the versions are not as easy to decipher. When you look at the <tt>vzpkg list</tt> output, that simply tells you when a cache was created. A cache is simply a snapshot of all the data needed for the latest versions of the OS and it's applications at the time the cache was created. It is a date, and thus tells you nothing about the versions within. So for instance, if you had a cache for Ubuntu 10.04 from 2007 and you ran <tt>vzpkg create cache ubuntu-10.04-x86</tt> it would re-download the latest version of apache, mysql and so on. And any ''subsequent'' ubuntu 10.04 CT's created thereafter would use those newer versions, whereas older CT's based on 10.04 would use older templates. You can see how this works by looking at the files under:<br />
<br />
<pre># ls /vz/template/ubuntu/10.04/x86/<br />
aap_1.091-1_all<br />
aap-doc_1.091-1_all<br />
adduser_3.112ubuntu1_all<br />
anacron_2.3-13.1ubuntu11_i386<br />
apache2_2.2.14-5ubuntu8.2_i386<br />
apache2_2.2.14-5ubuntu8.4_i386<br />
apache2_2.2.14-5ubuntu8.6_i386<br />
apache2_2.2.14-5ubuntu8.7_i386<br />
apache2_2.2.14-5ubuntu8.8_i386<br />
apache2_2.2.14-5ubuntu8.9_i386<br />
<...snip><br />
</pre><br />
<br />
You can see that this template has in it 6 different versions of apache2. This means that this template was merged with other 10.04 templates and/or it has been cached a few times, each time a new apache was available. The unfortunate thing is it's almost impossible to know which CT's are using which version of apache so it's hard to try to prune this down and remove unwanted/unneeded applications.<br />
<br />
So, if you see another HN has Ubuntu 10.04 on it, you need to see/confirm that it has all the exact files your CT needs. You can run a test rsync to see what ''would'' be transferred (what's missing):<br />
<br />
rsync -avn --stats -e ssh /vz/template/ubuntu/10.04 root@10.1.4.69:/vz/template/ubuntu<br />
<br />
Based on the amount of data you get back from the rsync, you will be able to decide whether or not to remove the -n and allow the full transfer to go through. The downside to doing the transfer is the situation described above- you've permanently joined these templates and now you may have 10 versions of apache on the target HN. There's one other potential pitfall to this kind of merging- it will also potentially copy over shared files, for which there is no particular version, most of which are in <tt>/vz/template/ubuntu/10.04/x86/config</tt>: <br />
<br />
cat /vz/template/ubuntu/10.04/x86/config/os/default/repositories<br />
/vz/template/ubuntu/10.04/x86/config/os/default/repositories<br />
<br />
Here are some specific things to look out for. Case: copying centos-5 on virt19 to virt12. We dumped the rsync output to a file so we could inspect:<br />
<br />
[root@virt19 /vz/template]# rsync -an -e ssh /vz/template/centos/ root@10.1.4.62:/vz/template/centos/ > v12_c5<br />
<br />
(note, that can be dangerous, better to add on full path <tt>/vz/template/centos/5</tt>)<br />
<br />
So in the output file we see things like:<br />
<br />
<pre><br />
x86/base0/cachecookie<br />
x86/base0/primary.xml.gz <br />
x86/base0/primary.xml.gz.sqlite <br />
x86/base0/repomd.xml <br />
x86/base1/cachecookie <br />
x86/base1/filelists.xml.gz <br />
x86/base1/filelists.xml.gz.sqlite <br />
x86/base1/primary.xml.gz <br />
x86/base1/primary.xml.gz.sqlite <br />
x86/base1/repomd.xml<br />
x86/base2/cachecookie<br />
</pre><br />
<br />
Which make us nervous cause those are not versioned files, i.e.:<br />
5/x86/MAKEDEV-3.23-1.2.i386/usr/sbin/mksock<br />
<br />
So those files will replace existing files on v12 rather than the case of the latter where it's likely being added. So caution should be given and deference to whichever version is newer (which rsync should take care of, but that will depend on the date each file was created and perhaps not the actual data within).<br />
<br />
If we look further in the file we see:<br />
<br />
x86/psa-appvault-moodle-1.8-8202920080409011254.noarch/usr/local/psa/var/cgitory/moodle-1.<br />
9/htdocs/lib/editor/htmlarea/plugins/TableOperations/img/cell-split.gif<br />
<br />
and other files looking like:<br />
5/x86/plesk-base-9.0.1-cos5.build90090127.18.i586/etc/sw/keys/info<br />
<br />
Which means plesk is here. We don't need plesk on v12 (the CT moving over doesn't use it) so we rerun the rsync and exclude it:<br />
<br />
[root@virt19 /vz/template]# rsync -an -e ssh --exclude="plesk*" --exclude="psa*" /vz/template/centos/ root@10.1.4.62:/vz/template/centos/ > v12_c5<br />
<br />
and now it's much smaller:<br />
<br />
[root@virt19 /vz/template]# cat v12_c5|wc -l<br />
97104<br />
<br />
Before running with excludes it was:<br />
[root@virt19 /vz/template]# cat v12_c5|wc -l <br />
238238<br />
<br />
So again, look at what you're doing. Generally, combining templates is fine however.<br />
<br />
If you're happy with the merge, run the rsync again without the "-n" to let it actually do the transfer.<br />
<br />
Once done, don't forget to add the new template to the HN in Management -> Reference -> Templates<br />
<br />
= getting help =<br />
<br />
= vzup2date =<br />
TODO <br />
= Adding new OS/application templates =<br />
<br />
Virtuozzo will lag a bit behind the initial release of an OS, perhaps by a month or more. Further, a newly-released OS may be available, but there may not be many/any application templates available initially. It pays to wait a bit.<br />
<br />
A template is easily installed via [[#vzup2date|vzup2date]]:<br />
<br />
vzup2date -z<br />
<br />
You will be given a list of OS's to install. Select an OS, then customize that selection by checking/unchecking the application templates we do/don't want to include/offer.<br />
<br />
Generally, here are the app templates we include/offer:<br />
<pre>cyrus-imap<br />
devel<br />
imp<br />
jre<br />
jsdk<br />
mailman<br />
mod_perl<br />
mod_ssl<br />
mysql<br />
php<br />
phpmyadmin<br />
phppgadmin<br />
postgresql<br />
proftpd<br />
spamassassin<br />
squirrelmail<br />
tomcat<br />
vsftpd<br />
</pre><br />
<br />
We do not (any longer) offer webalizer or analog.<br />
<br />
After selecting and installing the OS and apps, you should ensure it's installed:<br />
<br />
vzpkg list -O<br />
<br />
Your new OS (ex. fedora-core-13-x86_64) will be listed, without a corresponding cache date:<br />
<pre><br />
[root@virt13 /vzconf]# vzpkg list -O<br />
centos-6-x86 2011-07-22 13:24:41<br />
centos-6-x86_64 2012-03-09 20:42:22<br />
centos-5-x86 2009-07-27 16:58:24<br />
centos-5-x86_64 2012-03-09 22:38:53<br />
ubuntu-11.10-x86_64 2012-03-01 23:13:33<br />
ubuntu-9.04-x86 2009-06-02 11:32:39<br />
ubuntu-12.04-x86_64 2012-05-29 13:50:50<br />
ubuntu-8.04-x86 2008-09-17 21:27:20<br />
ubuntu-8.10-x86 2009-03-13 13:58:57<br />
ubuntu-11.04-x86 2011-12-05 11:15:46<br />
ubuntu-7.04-x86 2008-09-24 16:42:50<br />
redhat-el6-x86_64<br />
fedora-core-13-x86_64 <br />
fedora-core-12-x86 2010-02-06 13:24:32<br />
fedora-core-15-x86 2011-12-05 11:36:43<br />
fedora-core-11-x86 2009-10-01 16:30:59<br />
fedora-core-14-x86<br />
fedora-core-16-x86_64 2012-03-13 11:40:23<br />
fedora-core-9-x86 2008-11-29 10:52:18<br />
debian-6.0-x86 2011-07-29 16:00:35<br />
debian-6.0-x86_64 2012-03-12 19:53:33<br />
debian-5.0-x86 2009-03-12 12:39:11<br />
</pre><br />
<br />
You can also confirm the apps installed:<br />
<pre>[root@virt13 /vzconf]# vzpkg list fedora-core-13-x86_64<br />
fedora-core-13-x86_64 2013-03-08 11:39:44<br />
fedora-core-13-x86_64 devel<br />
fedora-core-13-x86_64 php<br />
fedora-core-13-x86_64 proftpd<br />
fedora-core-13-x86_64 postgresql<br />
fedora-core-13-x86_64 phpmyadmin<br />
fedora-core-13-x86_64 mysql<br />
</pre><br />
<br />
Before you can use the OS you must cache it. To create the cache run:<br />
vzpkg cache [OS]<br />
<br />
ex: <tt>vzpkg cache fedora-core-13-x86_64</tt><br />
<br />
Now it's ready to use. In order to be able to use it with the [[VPS_Management#vm|vm]] command, you must create a file:<br />
jctmpl-[OS]<br />
<br />
ex: <tt>jctmpl-fedora-core-13-x86_64</tt><br />
<br />
In it, on line 1 we place the OS, followed by the app names, ex:<br />
<pre>more jctmpl-fedora-core-13-x86_64<br />
fedora-core-13-x86_64<br />
postgresql<br />
jsdk<br />
mod_ssl<br />
phpmyadmin<br />
mod_perl<br />
tomcat<br />
devel<br />
php<br />
mailman<br />
cyrus-imap<br />
squirrelmail<br />
proftpd<br />
mysql<br />
phppgadmin<br />
spamassassin<br />
</pre><br />
<br />
The packages will be installed in that order, so any pre-req's should be handled there.<br />
<br />
Now, when you run [[VPS_Management#vm|vm]] it will show you the newly-added OS and you may create a CT using it.<br />
<br />
The last few tasks are related to mgmt:<br />
<br />
1. add to mgmt->reference->templates (use the existing templates as an example). Once added, the new template will be included with the list of OS's available for the given HN.<br />
2. if this is going to be a new OS offered for ordering, it needs to be added to the order form and webpage.<br />
<br />
a. to get a list of applications and versions in the new OS, you should create a test CT, enter it and run <tt>dpkg -l|sort</tt> (or <tt>rpm -aq|sort</tt> in centos/fedora) and enter that list in the following places:<br><br />
<tt>/usr/local/www/jc_pub/[osname].html</tt><br><br />
<tt>/usr/local/www/signup/html/[osname].html</tt><br><br />
Note, you will have to move the most recent OS data from <tt>/usr/local/www/signup/html/[osname].html</tt> into the corresponding <tt>/usr/local/www/signup/html/[osname]_old.html</tt> updating the version links in both.<br />
<br />
b. to add the new OS to the order form, you will need to update <tt>/usr/local/www/signup/html/step1.html</tt> in several places (for regular and OSS orders) as well as updating the templateID which you can get from mgmt->reference->templates (or [[Management_System_/_Public_Website_/_Signup_/_Account_Manager#jc:ref_templates|jc:ref_templates]])<br />
<br />
<br />
<br />
Old method to obtain the templates:<br />
<br />
Go to http://www.parallels.com/en/virtuozzo/templates/catalog/<br />
<br />
Download the RPM's and install them<br />
<br />
vzpkg list<br />
to see the new packages/os templates<br />
<br />
vzpkg create cache "OS_EZ_template_name"<br />
to create a cache<br />
<br />
<br />
= Update OS template =<br />
<br />
Installation and update of the new version of OS template will not affect the existing containers. It will affect only the new containers, created after this update operation. In terms of fixing "vzctl enter" error and SSH connectivity to Ubuntu based containers, it is necessary to recreate the template and not to update it, so the commands to run are:<br />
<br />
<pre>rpm -Uhv ubuntu-8.04-x86-ez-4.0.0-9.swsoft.noarch.rpm<br />
vzpkg clean ubuntu-8.04-x86<br />
vzpkg remove cache ubuntu-8.04-x86<br />
vzpkg create cache ubuntu-8.04-x86</pre><br />
<br />
<br />
= Installing new SSL cert on VZCP =<br />
<br />
<pre>openssl req -days 1999 -new -x509 -nodes -out server.crt -keyout server.key<br />
<br />
US<br />
CA<br />
San Diego<br />
johncompanies.com<br />
johncompanies.com<br />
virt15ohncompanies.com<br />
linux@johncompanies.com<br />
<br />
cp -p /etc/httpd/conf/ssl.crt/server.crt /etc/httpd/conf/ssl.crt/server.crt.bak<br />
cp -p /etc/httpd/conf/ssl.key/server.key /etc/httpd/conf/ssl.key/server.key.bak<br />
cp server.crt /etc/httpd/conf/ssl.crt/server.crt<br />
cp server.key /etc/httpd/conf/ssl.key/server.key<br />
/etc/init.d/vzcp restart</pre><br />
<br />
= Moving virt (drives) to new hardware =<br />
<br />
If you have a case where a virt is dead (due to hardware failure), you can move the drives to a new hardware setup.<br />
Here are the steps and things to look for:<br />
<br />
Move the drives to the new box. Most likely, the drives will be recognized and there will be no problems booting up. On Dell PERC 5/6 cards you will need to import the new "foreign" volume via the raid BIOS.<br />
<br />
If the hardware (motherboard) is different, it’s possible the nic’s won’t be recognized. Look at /etc/modules.conf the ethernet drivers are either:<br />
<br />
<pre>alias eth0 tg3<br />
alias eth1 tg3</pre><br />
(supermicro)<br />
<br />
Or<br />
<br />
<pre>alias eth0 e100<br />
alias eth1 e1000</pre><br />
(intel)<br />
<br />
Or<br />
<br />
<pre>alias eth0 bnx2<br />
alias eth1 bnx2</pre><br />
(dell 29500)<br />
<br />
Also, you may see eth0 and eth1 swapped so if after confirming the eth devices are present (may require a reboot), then you may still need to swap the green/yellow network cables in the back to get them on the right port.<br />
<br />
VZ will not run (for long) on the new hardware, so you will need to get the license reissued. You may create/download a temporary license via their website for the interim.<br />
<br />
= Updating kernel on virt =<br />
<br />
We now use [[#vzup2date|vzup2date]] to update systems and kernels. What follows is what we used to do when we had to download kernels and install manually on old systems:<br />
<br />
<br />
1. install kernel and modules<br />
<pre># rpm -ivh vzkernel-enterprise-2.4.20-021stab028.12.777.i686.rpm \<br />
vzmodules-enterprise-2.4.20-021stab028.12.swsoft.i686.rpm<br />
vzkernel-smp ##################################################<br />
vzmodules-smp ##################################################</pre><br />
<br />
DO NOT USE the "rpm -Uhv" command to install the kernel, otherwise all the previously installed kernels may be removed from your system. <br />
<br />
2. update lilo/grub<br />
<br />
Lilo:<br />
<br />
vi /etc/lilo.conf<br />
<br />
rename old Virtuozzo kernel label from "linux+virtuozzo" to "virtuozzo_old"<br />
and add a new entry pointing to the newly installed virtuozzo kernel image.<br />
For example:<br />
<br />
<pre>prompt<br />
timeout=50<br />
default=linux+virtuozzo<br />
append="debug panic=5 console=tty console=ttyS0,38400"<br />
boot=/dev/sda<br />
map=/boot/map<br />
install=/boot/boot.b<br />
message=/boot/message<br />
linear<br />
<br />
image=/boot/vmlinuz-2.4.18-3smp<br />
label=linux<br />
initrd=/boot/initrd-2.4.18-3smp.img<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.18-3<br />
label=linux-up<br />
initrd=/boot/initrd-2.4.18-3.img<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-020stab009.5.777-enterprise<br />
label=2.4.20-9.5.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-020stab009.8.777-enterprise<br />
label=2.4.20-9.8.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-020stab009.21.777-enterprise<br />
label=2.4.20-9.21.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-020stab009.24.777-enterprise<br />
label=2.4.20-9.24.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-021stab028.3.777-enterprise<br />
label=2.4.20-28.3.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-021stab028.5.777-enterprise<br />
label=old_linux+vz<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-021stab028.12.777-enterprise<br />
label=linux+virtuozzo<br />
read-only<br />
root=/dev/sda1</pre><br />
<br />
With this configuration, the new kernel is the default kernel to use at boot. The original kernel entry has been retained with the label "old_linux+vz ".<br />
<br />
Grub:<br />
<br />
vi /boot/grub /menu.lst<br />
<br />
Add new entry at the top.<br />
For example:<br />
<br />
<pre>serial --unit=0 --speed=38400<br />
terminal --timeout=10 serial console<br />
default=0<br />
timeout=10<br />
splashimage=(hd0,0)/boot/grub/splash.xpm.gz<br />
title Virtuozzo 2.6.1 (2.4.20-021stab028.12.777-enterprise)<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-22.4.20-021stab028.12.777-enterprise root=/dev/sda1 console=tty0 \ console=ttyS0,38400<br />
title Virtuozzo 2.6.1 (2.4.20-021stab028.3.777-enterprise)<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.20-021stab028.3.777-enterprise root=/dev/sda1 console=tty0 \ console=ttyS0,38400<br />
title Red Hat Linux (2.4.18-3smp)<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.18-3smp ro root=/dev/sda1<br />
initrd /boot/initrd-2.4.18-3smp.img<br />
title Red Hat Linux-up (2.4.18-3)<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.18-3 ro root=/dev/sda1<br />
initrd /boot/initrd-2.4.18-3.img<br />
title Linux with Virtuozzo 2.5<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.20-020stab009.3.777-smp ro root=/dev/sda1<br />
title vz-enterpriseo<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.20-020stab009.21.777-enterprise ro root=/dev/sda1<br />
title vz-enterprise<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.20-020stab009.24.777-enterprise ro root=/dev/sda1 \ console=tty0 console=ttyS0,38400</pre><br />
<br />
3. run the lilo (if using lilo)<br />
<br />
<pre># lilo<br />
Added linux<br />
Added linux-up<br />
Added virtuozzo_old<br />
Added linux+virtuozzo *</pre><br />
<br />
4. reboot the server<br />
shutdown -r 23:05<br />
<br />
when it comes time for the shutdown, run [[VPS_Management#stopvirt|stopvirt]] to get things shut down faster<br />
<br />
<br />
= Remaking service VE (<=3.x only) =<br />
<br />
Occasionally the control panel for VEs (aka VZPP) will stop working and you just need to reinstall the service VE (which runs the control panels for all VEs on the HN). Here's how that's done :<br />
<br />
<pre>vzctl stop 1<br />
ipdel 1 69.55.234.152<br />
vzmlocal 1:2<br />
vzctl set 2 --save --onboot no<br />
vzsveinstall -t redhat-as3-minimal -d /root/Rel261/HW/RPMS -s 69.55.234.152 --skip-client</pre><br />
<br />
on 3.0:<br />
<br />
<pre>vzsveinstall -t redhat-as3-minimal -d /vz/Rel300/HW/RPMS -s 69.55.229.151 --skip-client<br />
<br />
vzctl stop 1<br />
vzctl mount 1<br />
vzctl mount 2<br />
/bin/cp -aRf /vz/root/2/home/vzagent0 /vz/root/1/home<br />
/bin/cp -aRf /vz/root/2/var/lib/mysql/VZADB /vz/root/1/var/lib/mysql<br />
/bin/cp -af /vz/root/2/etc/shadow /vz/root/1/etc/<br />
/bin/cp -aRf /vz/root/2/var/vzcp/static/vz/skins/ /vz/root/1/var/vzcp/static/vz<br />
/bin/cp -af /vz/root/2/etc/vzcp/pp/menu.xml /vz/root/1/etc/vzcp/pp/<br />
/bin/cp -af /vz/root/2/etc/vzcp/pp/dashboard.xml /vz/root/1/etc/vzcp/pp/<br />
<br />
vzctl umount 2<br />
vzctl umount 1<br />
vzctl start 1</pre><br />
<br />
<br />
= GRUB: boot once to a new kernel =<br />
<br />
To use a different kernel for the next reboot only:<br />
<pre>grub --no-floppy<br />
savedefault --default=N --once<br />
(where N is the number of the kernel you want to boot once)<br />
quit</pre><br />
<br />
Reboot your machine. It will boot using kernel "N".<br />
<br />
grub-reboot N<br />
will do essentially the same thing and prompt to reboot the machine<br />
<br />
<br />
= Recovering from GRUB failure =<br />
<br />
(example from DPE 2950)<br />
<br />
Boot to CEOS4 server CD (original install cd). Did this example with iso mounted via drac<br />
<br />
Boot: linux rescue<br />
<br />
<br />
= Make/edit an ISO on Linux =<br />
<br />
<pre>mount file.iso /mnt/iso/ -t iso9660 -o ro,loop=/dev/loop0<br />
mkdir /tmp/iso<br />
cp -aR /mnt/iso/* /tmp/iso/<br />
umount /mnt/iso/<br />
mkisofs -iso-level 2 -o /tmp/new.iso /tmp/iso/<br />
rm -fr /tmp/iso</pre><br />
<br />
To mount it:<br />
<br />
mount -t vfat -o loop /tmp/3wfirmware.iso /mnt/usb/<br />
<br />
http://www.damnsmalllinux.org/wiki/index.php/Installing_to_a_USB_Flash_Drive<br />
<br />
= Mount/edit USB drive on Linux =<br />
<br />
mount -t vfat -o loop /dev/sdc1 /mnt/usb/</div>Supporthttps://wiki.jcihosting.com/index.php?title=Virtuozzo_/_Linux_Reference&diff=1047Virtuozzo / Linux Reference2013-03-11T23:59:28Z<p>Support: /* Make/edit an ISO on Linux */</p>
<hr />
<div>= Updating VZ =<br />
<br />
Update Virtuozzo to version 4.0.0<br />
Apply minor updates for Virtuozzo 3.0.0<br />
<br />
...<br />
When you get to the last screen, you will be given an option to <br />
<br />
( ) Automatically reboot after update<br />
(*) Reboot later manually<br />
<br />
make sure you choose to reboot later.<br />
<br />
= Migrating Templates =<br />
One nice feature VZ offers is the ability to run a virtual environment (VE or CT as it's now called) based on a particular ditribution on a host which may or may not share the same distribution. For instance, you could run a CT running Ubuntu while the host machine (Hardware Node or HN) is running CentOS. This is accomplished via the use of OS templates. As long as the HN contains the OS template on which a particular CT relies, it will run there. As such, if you want to move a CT from one HN to another, you will need to make sure the OS template exists there.<br />
<br />
Modern versions of VZ (>= 4.x) will check for and automatically move the OS template over to the host as you're (vz)migrating the CT over to a new HN. However we like to make sure the template is in place prior to moving the CT. <br />
<br />
The first step is to see if the template exists on the host. To see OS templates installed:<br />
<br />
2.x (shows OS and application templates):<br />
<pre># vzpkgls<br />
mod_ssl-rh9 20040624<br />
mysql-rh9 20030814 20050412<br />
openwebmail-rh9 20050427<br />
php-rh9 20050506<br />
phpBB-rh9 20041229<br />
postgresql-rh9 20050719<br />
sl-webalizer-rh9 20040119<br />
spamassassin-rh9 20040127<br />
tomcat-rh9 20040524<br />
usermin-rh9 20040116</pre><br />
<br />
>= 3.x (shows just OS templates, run without -O to see application as well):<br />
<pre># vzpkg list -O<br />
ubuntu-10.04-x86 2010-06-01 11:39:05<br />
ubuntu-11.04-x86 2011-06-22 14:23:59<br />
ubuntu-9.10-x86 2010-03-11 17:39:51<br />
centos-5-x86 2010-03-11 17:12:27<br />
debian-5.0-x86 2010-03-11 17:33:34</pre><br />
<br />
<br />
All template files are located:<br />
<br />
<pre># ls /vz/template/<br />
ColdFusion-fc1 jre-fc1 openwebmail-fc1 sl-webalizer-fc1<br />
ColdFusion-fc2 jre-fc2 openwebmail-fc2 sl-webalizer-fc2<br />
ColdFusion-rh9 jre-rh9 openwebmail-rh9 sl-webalizer-rh9<br />
PostNuke-fc1 jre-suse92 openwebmail-suse92 spamassassin-fc1<br />
PostNuke-fc2 jrun php-deb31 spamassassin-fc2<br />
PostNuke-rh9 mailman-deb31 php-fc1 spamassassin-rh9<br />
analog-fc1 mailman-fc1 php-fc2 spamassassin-suse92<br />
analog-fc2 mailman-fc2 php-rh9 suse-9.2<br />
analog-rh9 mailman-rh9 php-suse92 tomcat-fc1<br />
awstats-fc1 mailman-suse92 phpBB-fc1 tomcat-fc2<br />
awstats-rh9 mod_frontpage-fc1 phpBB-fc2 tomcat-rh9<br />
bbClone-fc1 mod_frontpage-fc2 phpBB-rh9 tomcat-suse92<br />
bbClone-fc2 mod_frontpage-rh9 phpmyadmin-deb31 urchin-fc1<br />
bbClone-rh9 mod_frontpage-suse92 postgresql-deb31 usermin-fc1<br />
conf mod_perl-deb31 postgresql-fc1 usermin-fc2<br />
debian-3.1 mod_perl-fc1 postgresql-fc2 usermin-rh9<br />
devel-deb31 mod_perl-fc2 postgresql-rh9 uw-imap-fc1<br />
devel-fc2 mod_perl-rh9 postgresql-suse92 uw-imap-fc2<br />
devel-suse92 mod_perl-suse92 proftpd-deb31 uw-imap-rh9<br />
fedora-core-1 mod_ssl-fc1 proftpd-fc1 vzredhat-7.3<br />
fedora-core-2 mod_ssl-fc2 proftpd-fc2 vzredhat-8.0<br />
jdk-deb31 mod_ssl-rh9 proftpd-rh9 webalizer-suse92<br />
jdk-fc1 mysql-deb31 proftpd-suse92 webmin-fc1<br />
jdk-fc2 mysql-fc1 redhat-7.2 webmin-fc2<br />
jdk-rh9 mysql-fc2 redhat-9 webmin-rh9<br />
jdk-suse92 mysql-rh9 redhat-as3-minimal<br />
jre-deb31 mysql-suse92 redhat-devel-9<br />
</pre><br />
<br />
What we see here are the base OS templates: <tt>redhat-9, fedora-core-1</tt> and supporting application templates: <tt>postgresql-rh9, mailman-rh9, jdk-fc1, mod_ssl-fc1</tt>. So if you wanted to copy over all of redhat 9 you'd need to copy the contents of:<br />
<pre># ls -d /vz/template/*rh9*<br />
/vz/template/ColdFusion-rh9 /vz/template/mod_frontpage-rh9 /vz/template/proftpd-rh9<br />
/vz/template/PostNuke-rh9 /vz/template/mod_perl-rh9 /vz/template/sl-webalizer-rh9<br />
/vz/template/analog-rh9 /vz/template/mod_ssl-rh9 /vz/template/spamassassin-rh9<br />
/vz/template/awstats-rh9 /vz/template/mysql-rh9 /vz/template/tomcat-rh9<br />
/vz/template/bbClone-rh9 /vz/template/openwebmail-rh9 /vz/template/usermin-rh9<br />
/vz/template/jdk-rh9 /vz/template/php-rh9 /vz/template/uw-imap-rh9<br />
/vz/template/jre-rh9 /vz/template/phpBB-rh9 /vz/template/webmin-rh9<br />
/vz/template/mailman-rh9 /vz/template/postgresql-rh9<br />
<br />
# ls -d /vz/template/*redhat-9*<br />
/vz/template/redhat-9<br />
</pre><br />
<br />
To get a template from one HN to another, it simply needs to be rsync'd over to the target HN. It's rsync'd over in this fashion:<br />
<br />
rsync -av -e ssh /vz/template/redhat-9 root@10.1.4.65:/vz/template<br />
rsync -av -e ssh /vz/template/*rh9 root@10.1.4.65:/vz/template<br />
<br />
Newer (>= 3.x) VZ employs "EZ" templates which are organized a little differently:<br />
<br />
<pre># ls /vz/template/<br />
cache CmdTool.log debian redhat-as3-minimal-20080630.tar.gz vc<br />
centos conf redhat-as3-minimal ubuntu<br />
<br />
# ls /vz/template/ubuntu/<br />
10.04 11.04 9.10<br />
# ls /vz/template/ubuntu/10.04/<br />
x86<br />
# ls /vz/template/ubuntu/10.04/x86/<br />
aap_1.091-1_all<br />
aap-doc_1.091-1_all<br />
adduser_3.112ubuntu1_all<br />
anacron_2.3-13.1ubuntu11_i386<br />
apache2_2.2.14-5ubuntu8.2_i386<br />
apache2_2.2.14-5ubuntu8.4_i386<br />
apache2_2.2.14-5ubuntu8.6_i386<br />
apache2_2.2.14-5ubuntu8.7_i386<br />
apache2_2.2.14-5ubuntu8.8_i386<br />
apache2_2.2.14-5ubuntu8.9_i386<br />
<...snip><br />
<br />
# ls /vz/template/cache/<br />
centos-5-x86.tar.gz suse-11.3-x86.tar.gz ubuntu-9.10-x86.tar.gz<br />
debian-5.0-x86.tar.gz ubuntu-10.04-x86.tar.gz<br />
fedora-core-14-x86.tar.gz ubuntu-11.04-x86.tar.gz<br />
</pre><br />
<br />
So what we see is the entire OS and all applications are in 1 directory, and organized by distribution and release version. So to move Ubuntu 10.04:<br />
<br />
rsync -av -e ssh /vz/template/ubuntu/10.04 root@10.1.4.69:/vz/template/ubuntu<br />
<br />
and you also need to move the cache:<br />
<br />
rsync -av -e ssh /vz/template/cache/ubuntu-10.04-x86.tar.gz root@10.1.4.69:/vz/template/cache/<br />
<br />
Once the transfer is complete, you will be able to run <tt>vzpkgls</tt> or <tt>vzpkg list -O</tt> and see the template(s) listed on the target HN.<br />
<br />
So far, this all supposes template doesn't exist on the target- very simple, just copy it over. If the template exists already on the target HN, this requires closer inspection. With older templates, simply moving over the templates would be the end of it- you see the version listed when you do a <tt>vzplgls</tt>:<br />
<br />
mysql-rh9 20030814 20050412<br />
<br />
this means there are 2 versions of the mysql package for RH9: 20030814 and 20050412<br />
As an aside, you can't copy over just 1 of them, but if you rsync'd over the <tt>mysql-rh9</tt> directory, you'd see 20030814 and 20050412 on the target HN. As long as the versions match up, you're good to go.<br />
<br />
With EZ templates, the versions are not as easy to decipher. When you look at the <tt>vzpkg list</tt> output, that simply tells you when a cache was created. A cache is simply a snapshot of all the data needed for the latest versions of the OS and it's applications at the time the cache was created. It is a date, and thus tells you nothing about the versions within. So for instance, if you had a cache for Ubuntu 10.04 from 2007 and you ran <tt>vzpkg create cache ubuntu-10.04-x86</tt> it would re-download the latest version of apache, mysql and so on. And any ''subsequent'' ubuntu 10.04 CT's created thereafter would use those newer versions, whereas older CT's based on 10.04 would use older templates. You can see how this works by looking at the files under:<br />
<br />
<pre># ls /vz/template/ubuntu/10.04/x86/<br />
aap_1.091-1_all<br />
aap-doc_1.091-1_all<br />
adduser_3.112ubuntu1_all<br />
anacron_2.3-13.1ubuntu11_i386<br />
apache2_2.2.14-5ubuntu8.2_i386<br />
apache2_2.2.14-5ubuntu8.4_i386<br />
apache2_2.2.14-5ubuntu8.6_i386<br />
apache2_2.2.14-5ubuntu8.7_i386<br />
apache2_2.2.14-5ubuntu8.8_i386<br />
apache2_2.2.14-5ubuntu8.9_i386<br />
<...snip><br />
</pre><br />
<br />
You can see that this template has in it 6 different versions of apache2. This means that this template was merged with other 10.04 templates and/or it has been cached a few times, each time a new apache was available. The unfortunate thing is it's almost impossible to know which CT's are using which version of apache so it's hard to try to prune this down and remove unwanted/unneeded applications.<br />
<br />
So, if you see another HN has Ubuntu 10.04 on it, you need to see/confirm that it has all the exact files your CT needs. You can run a test rsync to see what ''would'' be transferred (what's missing):<br />
<br />
rsync -avn --stats -e ssh /vz/template/ubuntu/10.04 root@10.1.4.69:/vz/template/ubuntu<br />
<br />
Based on the amount of data you get back from the rsync, you will be able to decide whether or not to remove the -n and allow the full transfer to go through. The downside to doing the transfer is the situation described above- you've permanently joined these templates and now you may have 10 versions of apache on the target HN. There's one other potential pitfall to this kind of merging- it will also potentially copy over shared files, for which there is no particular version, most of which are in <tt>/vz/template/ubuntu/10.04/x86/config</tt>: <br />
<br />
cat /vz/template/ubuntu/10.04/x86/config/os/default/repositories<br />
/vz/template/ubuntu/10.04/x86/config/os/default/repositories<br />
<br />
Here are some specific things to look out for. Case: copying centos-5 on virt19 to virt12. We dumped the rsync output to a file so we could inspect:<br />
<br />
[root@virt19 /vz/template]# rsync -an -e ssh /vz/template/centos/ root@10.1.4.62:/vz/template/centos/ > v12_c5<br />
<br />
(note, that can be dangerous, better to add on full path <tt>/vz/template/centos/5</tt>)<br />
<br />
So in the output file we see things like:<br />
<br />
<pre><br />
x86/base0/cachecookie<br />
x86/base0/primary.xml.gz <br />
x86/base0/primary.xml.gz.sqlite <br />
x86/base0/repomd.xml <br />
x86/base1/cachecookie <br />
x86/base1/filelists.xml.gz <br />
x86/base1/filelists.xml.gz.sqlite <br />
x86/base1/primary.xml.gz <br />
x86/base1/primary.xml.gz.sqlite <br />
x86/base1/repomd.xml<br />
x86/base2/cachecookie<br />
</pre><br />
<br />
Which make us nervous cause those are not versioned files, i.e.:<br />
5/x86/MAKEDEV-3.23-1.2.i386/usr/sbin/mksock<br />
<br />
So those files will replace existing files on v12 rather than the case of the latter where it's likely being added. So caution should be given and deference to whichever version is newer (which rsync should take care of, but that will depend on the date each file was created and perhaps not the actual data within).<br />
<br />
If we look further in the file we see:<br />
<br />
x86/psa-appvault-moodle-1.8-8202920080409011254.noarch/usr/local/psa/var/cgitory/moodle-1.<br />
9/htdocs/lib/editor/htmlarea/plugins/TableOperations/img/cell-split.gif<br />
<br />
and other files looking like:<br />
5/x86/plesk-base-9.0.1-cos5.build90090127.18.i586/etc/sw/keys/info<br />
<br />
Which means plesk is here. We don't need plesk on v12 (the CT moving over doesn't use it) so we rerun the rsync and exclude it:<br />
<br />
[root@virt19 /vz/template]# rsync -an -e ssh --exclude="plesk*" --exclude="psa*" /vz/template/centos/ root@10.1.4.62:/vz/template/centos/ > v12_c5<br />
<br />
and now it's much smaller:<br />
<br />
[root@virt19 /vz/template]# cat v12_c5|wc -l<br />
97104<br />
<br />
Before running with excludes it was:<br />
[root@virt19 /vz/template]# cat v12_c5|wc -l <br />
238238<br />
<br />
So again, look at what you're doing. Generally, combining templates is fine however.<br />
<br />
If you're happy with the merge, run the rsync again without the "-n" to let it actually do the transfer.<br />
<br />
Once done, don't forget to add the new template to the HN in Management -> Reference -> Templates<br />
<br />
= getting help =<br />
<br />
= vzup2date =<br />
TODO <br />
= Adding new OS/application templates =<br />
<br />
Virtuozzo will lag a bit behind the initial release of an OS, perhaps by a month or more. Further, a newly-released OS may be available, but there may not be many/any application templates available initially. It pays to wait a bit.<br />
<br />
A template is easily installed via [[#vzup2date|vzup2date]]:<br />
<br />
vzup2date -z<br />
<br />
You will be given a list of OS's to install. Select an OS, then customize that selection by checking/unchecking the application templates we do/don't want to include/offer.<br />
<br />
Generally, here are the app templates we include/offer:<br />
<pre>cyrus-imap<br />
devel<br />
imp<br />
jre<br />
jsdk<br />
mailman<br />
mod_perl<br />
mod_ssl<br />
mysql<br />
php<br />
phpmyadmin<br />
phppgadmin<br />
postgresql<br />
proftpd<br />
spamassassin<br />
squirrelmail<br />
tomcat<br />
vsftpd<br />
</pre><br />
<br />
We do not (any longer) offer webalizer or analog.<br />
<br />
After selecting and installing the OS and apps, you should ensure it's installed:<br />
<br />
vzpkg list -O<br />
<br />
Your new OS (ex. fedora-core-13-x86_64) will be listed, without a corresponding cache date:<br />
<pre><br />
[root@virt13 /vzconf]# vzpkg list -O<br />
centos-6-x86 2011-07-22 13:24:41<br />
centos-6-x86_64 2012-03-09 20:42:22<br />
centos-5-x86 2009-07-27 16:58:24<br />
centos-5-x86_64 2012-03-09 22:38:53<br />
ubuntu-11.10-x86_64 2012-03-01 23:13:33<br />
ubuntu-9.04-x86 2009-06-02 11:32:39<br />
ubuntu-12.04-x86_64 2012-05-29 13:50:50<br />
ubuntu-8.04-x86 2008-09-17 21:27:20<br />
ubuntu-8.10-x86 2009-03-13 13:58:57<br />
ubuntu-11.04-x86 2011-12-05 11:15:46<br />
ubuntu-7.04-x86 2008-09-24 16:42:50<br />
redhat-el6-x86_64<br />
fedora-core-13-x86_64 <br />
fedora-core-12-x86 2010-02-06 13:24:32<br />
fedora-core-15-x86 2011-12-05 11:36:43<br />
fedora-core-11-x86 2009-10-01 16:30:59<br />
fedora-core-14-x86<br />
fedora-core-16-x86_64 2012-03-13 11:40:23<br />
fedora-core-9-x86 2008-11-29 10:52:18<br />
debian-6.0-x86 2011-07-29 16:00:35<br />
debian-6.0-x86_64 2012-03-12 19:53:33<br />
debian-5.0-x86 2009-03-12 12:39:11<br />
</pre><br />
<br />
You can also confirm the apps installed:<br />
<pre>[root@virt13 /vzconf]# vzpkg list fedora-core-13-x86_64<br />
fedora-core-13-x86_64 2013-03-08 11:39:44<br />
fedora-core-13-x86_64 devel<br />
fedora-core-13-x86_64 php<br />
fedora-core-13-x86_64 proftpd<br />
fedora-core-13-x86_64 postgresql<br />
fedora-core-13-x86_64 phpmyadmin<br />
fedora-core-13-x86_64 mysql<br />
</pre><br />
<br />
Before you can use the OS you must cache it. To create the cache run:<br />
vzpkg cache [OS]<br />
<br />
ex: <tt>vzpkg cache fedora-core-13-x86_64</tt><br />
<br />
Now it's ready to use. In order to be able to use it with the [[VPS_Management#vm|vm]] command, you must create a file:<br />
jctmpl-[OS]<br />
<br />
ex: <tt>jctmpl-fedora-core-13-x86_64</tt><br />
<br />
In it, on line 1 we place the OS, followed by the app names, ex:<br />
<pre>more jctmpl-fedora-core-13-x86_64<br />
fedora-core-13-x86_64<br />
postgresql<br />
jsdk<br />
mod_ssl<br />
phpmyadmin<br />
mod_perl<br />
tomcat<br />
devel<br />
php<br />
mailman<br />
cyrus-imap<br />
squirrelmail<br />
proftpd<br />
mysql<br />
phppgadmin<br />
spamassassin<br />
</pre><br />
<br />
The packages will be installed in that order, so any pre-req's should be handled there.<br />
<br />
Now, when you run [[VPS_Management#vm|vm]] it will show you the newly-added OS and you may create a CT using it.<br />
<br />
The last few tasks are related to mgmt:<br />
<br />
1. add to mgmt->reference->templates (use the existing templates as an example). Once added, the new template will be included with the list of OS's available for the given HN.<br />
2. if this is going to be a new OS offered for ordering, it needs to be added to the order form and webpage.<br />
<br />
a. to get a list of applications and versions in the new OS, you should create a test CT, enter it and run <tt>dpkg -l|sort</tt> (or <tt>rpm -aq|sort</tt> in centos/fedora) and enter that list in the following places:<br><br />
<tt>/usr/local/www/jc_pub/[osname].html</tt><br><br />
<tt>/usr/local/www/signup/html/[osname].html</tt><br><br />
Note, you will have to move the most recent OS data from <tt>/usr/local/www/signup/html/[osname].html</tt> into the corresponding <tt>/usr/local/www/signup/html/[osname]_old.html</tt> updating the version links in both.<br />
<br />
b. to add the new OS to the order form, you will need to update <tt>/usr/local/www/signup/html/step1.html</tt> in several places (for regular and OSS orders) as well as updating the templateID which you can get from mgmt->reference->templates (or [[Management_System_/_Public_Website_/_Signup_/_Account_Manager#jc:ref_templates|jc:ref_templates]])<br />
<br />
<br />
<br />
Old method to obtain the templates:<br />
<br />
Go to http://www.parallels.com/en/virtuozzo/templates/catalog/<br />
<br />
Download the RPM's and install them<br />
<br />
vzpkg list<br />
to see the new packages/os templates<br />
<br />
vzpkg create cache "OS_EZ_template_name"<br />
to create a cache<br />
<br />
<br />
= Update OS template =<br />
<br />
Installation and update of the new version of OS template will not affect the existing containers. It will affect only the new containers, created after this update operation. In terms of fixing "vzctl enter" error and SSH connectivity to Ubuntu based containers, it is necessary to recreate the template and not to update it, so the commands to run are:<br />
<br />
<pre>rpm -Uhv ubuntu-8.04-x86-ez-4.0.0-9.swsoft.noarch.rpm<br />
vzpkg clean ubuntu-8.04-x86<br />
vzpkg remove cache ubuntu-8.04-x86<br />
vzpkg create cache ubuntu-8.04-x86</pre><br />
<br />
<br />
= Installing new SSL cert on VZCP =<br />
<br />
<pre>openssl req -days 1999 -new -x509 -nodes -out server.crt -keyout server.key<br />
<br />
US<br />
CA<br />
San Diego<br />
johncompanies.com<br />
johncompanies.com<br />
virt15ohncompanies.com<br />
linux@johncompanies.com<br />
<br />
cp -p /etc/httpd/conf/ssl.crt/server.crt /etc/httpd/conf/ssl.crt/server.crt.bak<br />
cp -p /etc/httpd/conf/ssl.key/server.key /etc/httpd/conf/ssl.key/server.key.bak<br />
cp server.crt /etc/httpd/conf/ssl.crt/server.crt<br />
cp server.key /etc/httpd/conf/ssl.key/server.key<br />
/etc/init.d/vzcp restart</pre><br />
<br />
= Moving virt (drives) to new hardware =<br />
<br />
If you have a case where a virt is dead (due to hardware failure), you can move the drives to a new hardware setup.<br />
Here are the steps and things to look for:<br />
<br />
Move the drives to the new box. Most likely, the drives will be recognized and there will be no problems booting up. On Dell PERC 5/6 cards you will need to import the new "foreign" volume via the raid BIOS.<br />
<br />
If the hardware (motherboard) is different, it’s possible the nic’s won’t be recognized. Look at /etc/modules.conf the ethernet drivers are either:<br />
<br />
<pre>alias eth0 tg3<br />
alias eth1 tg3</pre><br />
(supermicro)<br />
<br />
Or<br />
<br />
<pre>alias eth0 e100<br />
alias eth1 e1000</pre><br />
(intel)<br />
<br />
Or<br />
<br />
<pre>alias eth0 bnx2<br />
alias eth1 bnx2</pre><br />
(dell 29500)<br />
<br />
Also, you may see eth0 and eth1 swapped so if after confirming the eth devices are present (may require a reboot), then you may still need to swap the green/yellow network cables in the back to get them on the right port.<br />
<br />
VZ will not run (for long) on the new hardware, so you will need to get the license reissued. You may create/download a temporary license via their website for the interim.<br />
<br />
= Updating kernel on virt =<br />
<br />
We now use [[#vzup2date|vzup2date]] to update systems and kernels. What follows is what we used to do when we had to download kernels and install manually on old systems:<br />
<br />
<br />
1. install kernel and modules<br />
<pre># rpm -ivh vzkernel-enterprise-2.4.20-021stab028.12.777.i686.rpm \<br />
vzmodules-enterprise-2.4.20-021stab028.12.swsoft.i686.rpm<br />
vzkernel-smp ##################################################<br />
vzmodules-smp ##################################################</pre><br />
<br />
DO NOT USE the "rpm -Uhv" command to install the kernel, otherwise all the previously installed kernels may be removed from your system. <br />
<br />
2. update lilo/grub<br />
<br />
Lilo:<br />
<br />
vi /etc/lilo.conf<br />
<br />
rename old Virtuozzo kernel label from "linux+virtuozzo" to "virtuozzo_old"<br />
and add a new entry pointing to the newly installed virtuozzo kernel image.<br />
For example:<br />
<br />
<pre>prompt<br />
timeout=50<br />
default=linux+virtuozzo<br />
append="debug panic=5 console=tty console=ttyS0,38400"<br />
boot=/dev/sda<br />
map=/boot/map<br />
install=/boot/boot.b<br />
message=/boot/message<br />
linear<br />
<br />
image=/boot/vmlinuz-2.4.18-3smp<br />
label=linux<br />
initrd=/boot/initrd-2.4.18-3smp.img<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.18-3<br />
label=linux-up<br />
initrd=/boot/initrd-2.4.18-3.img<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-020stab009.5.777-enterprise<br />
label=2.4.20-9.5.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-020stab009.8.777-enterprise<br />
label=2.4.20-9.8.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-020stab009.21.777-enterprise<br />
label=2.4.20-9.21.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-020stab009.24.777-enterprise<br />
label=2.4.20-9.24.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-021stab028.3.777-enterprise<br />
label=2.4.20-28.3.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-021stab028.5.777-enterprise<br />
label=old_linux+vz<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-021stab028.12.777-enterprise<br />
label=linux+virtuozzo<br />
read-only<br />
root=/dev/sda1</pre><br />
<br />
With this configuration, the new kernel is the default kernel to use at boot. The original kernel entry has been retained with the label "old_linux+vz ".<br />
<br />
Grub:<br />
<br />
vi /boot/grub /menu.lst<br />
<br />
Add new entry at the top.<br />
For example:<br />
<br />
<pre>serial --unit=0 --speed=38400<br />
terminal --timeout=10 serial console<br />
default=0<br />
timeout=10<br />
splashimage=(hd0,0)/boot/grub/splash.xpm.gz<br />
title Virtuozzo 2.6.1 (2.4.20-021stab028.12.777-enterprise)<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-22.4.20-021stab028.12.777-enterprise root=/dev/sda1 console=tty0 \ console=ttyS0,38400<br />
title Virtuozzo 2.6.1 (2.4.20-021stab028.3.777-enterprise)<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.20-021stab028.3.777-enterprise root=/dev/sda1 console=tty0 \ console=ttyS0,38400<br />
title Red Hat Linux (2.4.18-3smp)<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.18-3smp ro root=/dev/sda1<br />
initrd /boot/initrd-2.4.18-3smp.img<br />
title Red Hat Linux-up (2.4.18-3)<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.18-3 ro root=/dev/sda1<br />
initrd /boot/initrd-2.4.18-3.img<br />
title Linux with Virtuozzo 2.5<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.20-020stab009.3.777-smp ro root=/dev/sda1<br />
title vz-enterpriseo<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.20-020stab009.21.777-enterprise ro root=/dev/sda1<br />
title vz-enterprise<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.20-020stab009.24.777-enterprise ro root=/dev/sda1 \ console=tty0 console=ttyS0,38400</pre><br />
<br />
3. run the lilo (if using lilo)<br />
<br />
<pre># lilo<br />
Added linux<br />
Added linux-up<br />
Added virtuozzo_old<br />
Added linux+virtuozzo *</pre><br />
<br />
4. reboot the server<br />
shutdown -r 23:05<br />
<br />
when it comes time for the shutdown, run [[VPS_Management#stopvirt|stopvirt]] to get things shut down faster<br />
<br />
<br />
= Remaking service VE (<=3.x only) =<br />
<br />
Occasionally the control panel for VEs (aka VZPP) will stop working and you just need to reinstall the service VE (which runs the control panels for all VEs on the HN). Here's how that's done :<br />
<br />
<pre>vzctl stop 1<br />
ipdel 1 69.55.234.152<br />
vzmlocal 1:2<br />
vzctl set 2 --save --onboot no<br />
vzsveinstall -t redhat-as3-minimal -d /root/Rel261/HW/RPMS -s 69.55.234.152 --skip-client</pre><br />
<br />
on 3.0:<br />
<br />
<pre>vzsveinstall -t redhat-as3-minimal -d /vz/Rel300/HW/RPMS -s 69.55.229.151 --skip-client<br />
<br />
vzctl stop 1<br />
vzctl mount 1<br />
vzctl mount 2<br />
/bin/cp -aRf /vz/root/2/home/vzagent0 /vz/root/1/home<br />
/bin/cp -aRf /vz/root/2/var/lib/mysql/VZADB /vz/root/1/var/lib/mysql<br />
/bin/cp -af /vz/root/2/etc/shadow /vz/root/1/etc/<br />
/bin/cp -aRf /vz/root/2/var/vzcp/static/vz/skins/ /vz/root/1/var/vzcp/static/vz<br />
/bin/cp -af /vz/root/2/etc/vzcp/pp/menu.xml /vz/root/1/etc/vzcp/pp/<br />
/bin/cp -af /vz/root/2/etc/vzcp/pp/dashboard.xml /vz/root/1/etc/vzcp/pp/<br />
<br />
vzctl umount 2<br />
vzctl umount 1<br />
vzctl start 1</pre><br />
<br />
<br />
= GRUB: boot once to a new kernel =<br />
<br />
To use a different kernel for the next reboot only:<br />
<pre>grub --no-floppy<br />
savedefault --default=N --once<br />
(where N is the number of the kernel you want to boot once)<br />
quit</pre><br />
<br />
Reboot your machine. It will boot using kernel "N".<br />
<br />
grub-reboot N<br />
will do essentially the same thing and prompt to reboot the machine<br />
<br />
<br />
= Recovering from GRUB failure =<br />
<br />
(example from DPE 2950)<br />
<br />
Boot to CEOS4 server CD (original install cd). Did this example with iso mounted via drac<br />
<br />
Boot: linux rescue<br />
<br />
<br />
= Make/edit an ISO on Linux =<br />
<br />
<pre>mount file.iso /mnt/iso/ -t iso9660 -o ro,loop=/dev/loop0<br />
mkdir /tmp/iso<br />
cp -aR /mnt/iso/* /tmp/iso/<br />
umount /mnt/iso/<br />
mkisofs -iso-level 2 -o /tmp/new.iso /tmp/iso/<br />
rm -fr /tmp/iso</pre><br />
<br />
To mount it:<br />
<br />
mount -t vfat -o loop /tmp/3wfirmware.iso /mnt/usb/<br />
<br />
http://www.damnsmalllinux.org/wiki/index.php/Installing_to_a_USB_Flash_Drive<br />
<br />
= Mount/edit USB drive on Linux =<br />
<br />
mount -t vfat -o loop /tmp/3wfirmware.iso /mnt/usb/</div>Supporthttps://wiki.jcihosting.com/index.php?title=Data_Center_Notes&diff=1046Data Center Notes2013-03-11T23:59:03Z<p>Support: /* USB drive */</p>
<hr />
<div>= Castle =<br />
<br />
== Spare serial console ==<br />
There is a blue spare serial console cable (with adapter) hanging in cabinet 3-5. Connect to it with <tt>tip spare</tt> on the console server. You may need to adjust the speed in the <tt>/etc/remote</tt> file first. The cable is long enough to reach a few cabinets away.<br />
<br />
== cable color coding ==<br />
* blue & white pair: trunk lines from remote switches to p1a/b, A/B drops from castle<br />
* white: cross connects<br />
* yellow: public network<br />
* green: private network<br />
* orange: serial console<br />
* red: special<br />
<br />
== cross connect mapping ==<br />
(this is out of date, many probably aren't there any longer)<br />
<br />
230809 <-> 220809<br /><br />
130903 <-> 220808<br /><br />
130902 <-> 220807<br /><br />
120302 <-> 110502<br /><br />
120303 <-> 120803?<br /><br />
130703 <-> <br /><br />
230709 <-> <br /><br />
220308 <-> 231008<br /><br />
220309 <-> 231009<br /><br />
230708 <-> 240908 (switch P10)<br /><br />
131002 <-> 140902 (serial switch P10)<br /><br />
131003 <-> 140703 (serial console)<br /><br />
230607 <-> 260407 (serial switch P11)<br /><br />
230608 <-> 260508 (serial switch P12)<br /><br />
230609 <-> 260409 (switch P1 cross connect)<br /><br />
231007 <-> 261707 (serial switch P14)<br /><br />
130702 <-> 161603 (switch P13)<br /><br />
230707 <-> 261607 (serial switch P13)<br /><br />
130603 <-> 140803 (serial switch P15)<br /><br />
161602 <-> 130602<br /><br />
Cabinet 06-17 port 1 to cabinet 04-08 port 1 ??<br /><br />
<br />
Open ports<br />
<br />
3-6<br /><br />
130601 2<br />
<br />
3-5<br /><br />
all open<br />
<br />
== spare drives ==<br />
(in 3-6)<br />
<br />
st373307LC white sled #5<br /><br />
st373307LC white sled #4<br /><br />
ST3146807LC no sled<br /><br />
ST373455SS dell sled<br /><br />
ST373307LC white sled #0<br /><br />
ST373307LC white sled #1<br /><br />
146G SAS<br /><br />
<br />
== IP allocation ==<br />
<br />
We own the following IP block: 69.55.224.0/20<br />
Which consists of:<br />
<br />
<pre>69.55.224.0/24<br />
69.55.225.0/24<br />
69.55.226.0/24<br />
69.55.227.0/24<br />
69.55.228.0/24<br />
69.55.229.0/24 (announced at i2b)<br />
69.55.230.0/24 (internal use and “temp IPs”)<br />
69.55.231.0/24 (announced at i2b)<br />
69.55.232.0/24<br />
69.55.233.0/24 (dedicated customers, internal use)<br />
69.55.234.0/24<br />
69.55.235.0/24<br />
69.55.236.0/24<br />
69.55.237.0/24<br />
69.55.238.0/24<br />
69.55.239.0/24 (“bad boy” block)</pre><br />
<br />
All class C blocks above are subnetted as 255.255.255.0 and have their gateway at .1 (i.e. 69.55.224.12’s gateway is 69.55.224.1)<br />
<br />
The exception to this the 233 block which is subnetted up for use by dedicated customers and for some of our internal (routing) equipment. Here’s how that block is chopped up:<br />
<br />
<pre>Range Mask Slash Hosts Usage<br />
0-127 .128 25 126 dedicated’s (so any customer on this block has a DG of .1 and NM of 255.255.255.128)<br />
128-143 .240 28 14 <br />
144-151 .248 29 6 <br />
152-159 .248 29 6 fw external<br />
160-167 .248 29 6 fw internal<br />
168-183 .240 28 14 <br />
184-191 .248 29 6 <br />
192-223 .224 27 30 nat<br />
224-255 .224 27 30 dedicated’s DG: 69.55.233.225 (unverified)</pre><br />
<br />
== D800 laptop ==<br />
<br />
There is a Dell D800 laptop in cabinet 3-6 for use when at Castle (and to access via RDP when needed).<br />
<br />
It is nat'd as follows:<br />
<br />
69.55.233.197 <=> 10.1.4.240<br />
<br />
(it's on the private net)<br />
<br />
== USB drive ==<br />
<br />
We have a floating USB drive at castle, used for booting machines up in order to do BIOS updates or whatever.<br />
<br />
It lives on virt16<br />
<br />
See [[Virtuozzo_/_Linux_Reference#Mount.2Fedit_USB_drive_on_Linux|edit USB drive on Linux]] for more info.<br />
<br />
= i2b =<br />
<br />
== cable color coding ==<br />
* white: drops from i2b, i2b network<br />
* yellow: public JC network<br />
* green: private network<br />
* orange: serial console</div>Supporthttps://wiki.jcihosting.com/index.php?title=Data_Center_Notes&diff=1045Data Center Notes2013-03-11T23:58:51Z<p>Support: /* Castle */</p>
<hr />
<div>= Castle =<br />
<br />
== Spare serial console ==<br />
There is a blue spare serial console cable (with adapter) hanging in cabinet 3-5. Connect to it with <tt>tip spare</tt> on the console server. You may need to adjust the speed in the <tt>/etc/remote</tt> file first. The cable is long enough to reach a few cabinets away.<br />
<br />
== cable color coding ==<br />
* blue & white pair: trunk lines from remote switches to p1a/b, A/B drops from castle<br />
* white: cross connects<br />
* yellow: public network<br />
* green: private network<br />
* orange: serial console<br />
* red: special<br />
<br />
== cross connect mapping ==<br />
(this is out of date, many probably aren't there any longer)<br />
<br />
230809 <-> 220809<br /><br />
130903 <-> 220808<br /><br />
130902 <-> 220807<br /><br />
120302 <-> 110502<br /><br />
120303 <-> 120803?<br /><br />
130703 <-> <br /><br />
230709 <-> <br /><br />
220308 <-> 231008<br /><br />
220309 <-> 231009<br /><br />
230708 <-> 240908 (switch P10)<br /><br />
131002 <-> 140902 (serial switch P10)<br /><br />
131003 <-> 140703 (serial console)<br /><br />
230607 <-> 260407 (serial switch P11)<br /><br />
230608 <-> 260508 (serial switch P12)<br /><br />
230609 <-> 260409 (switch P1 cross connect)<br /><br />
231007 <-> 261707 (serial switch P14)<br /><br />
130702 <-> 161603 (switch P13)<br /><br />
230707 <-> 261607 (serial switch P13)<br /><br />
130603 <-> 140803 (serial switch P15)<br /><br />
161602 <-> 130602<br /><br />
Cabinet 06-17 port 1 to cabinet 04-08 port 1 ??<br /><br />
<br />
Open ports<br />
<br />
3-6<br /><br />
130601 2<br />
<br />
3-5<br /><br />
all open<br />
<br />
== spare drives ==<br />
(in 3-6)<br />
<br />
st373307LC white sled #5<br /><br />
st373307LC white sled #4<br /><br />
ST3146807LC no sled<br /><br />
ST373455SS dell sled<br /><br />
ST373307LC white sled #0<br /><br />
ST373307LC white sled #1<br /><br />
146G SAS<br /><br />
<br />
== IP allocation ==<br />
<br />
We own the following IP block: 69.55.224.0/20<br />
Which consists of:<br />
<br />
<pre>69.55.224.0/24<br />
69.55.225.0/24<br />
69.55.226.0/24<br />
69.55.227.0/24<br />
69.55.228.0/24<br />
69.55.229.0/24 (announced at i2b)<br />
69.55.230.0/24 (internal use and “temp IPs”)<br />
69.55.231.0/24 (announced at i2b)<br />
69.55.232.0/24<br />
69.55.233.0/24 (dedicated customers, internal use)<br />
69.55.234.0/24<br />
69.55.235.0/24<br />
69.55.236.0/24<br />
69.55.237.0/24<br />
69.55.238.0/24<br />
69.55.239.0/24 (“bad boy” block)</pre><br />
<br />
All class C blocks above are subnetted as 255.255.255.0 and have their gateway at .1 (i.e. 69.55.224.12’s gateway is 69.55.224.1)<br />
<br />
The exception to this the 233 block which is subnetted up for use by dedicated customers and for some of our internal (routing) equipment. Here’s how that block is chopped up:<br />
<br />
<pre>Range Mask Slash Hosts Usage<br />
0-127 .128 25 126 dedicated’s (so any customer on this block has a DG of .1 and NM of 255.255.255.128)<br />
128-143 .240 28 14 <br />
144-151 .248 29 6 <br />
152-159 .248 29 6 fw external<br />
160-167 .248 29 6 fw internal<br />
168-183 .240 28 14 <br />
184-191 .248 29 6 <br />
192-223 .224 27 30 nat<br />
224-255 .224 27 30 dedicated’s DG: 69.55.233.225 (unverified)</pre><br />
<br />
== D800 laptop ==<br />
<br />
There is a Dell D800 laptop in cabinet 3-6 for use when at Castle (and to access via RDP when needed).<br />
<br />
It is nat'd as follows:<br />
<br />
69.55.233.197 <=> 10.1.4.240<br />
<br />
(it's on the private net)<br />
<br />
== USB drive ==<br />
<br />
We have a floating USB drive at castle, used for booting machines up in order to do BIOS updates or whatever.<br />
<br />
It lives on virt16<br />
<br />
See [[Virtuozzo_/_Linux_Reference#Mount.2Fedit_USB_drive_on_Linux|edit_USB_drive_on_Linux]] for more info.<br />
<br />
= i2b =<br />
<br />
== cable color coding ==<br />
* white: drops from i2b, i2b network<br />
* yellow: public JC network<br />
* green: private network<br />
* orange: serial console</div>Supporthttps://wiki.jcihosting.com/index.php?title=Virtuozzo_/_Linux_Reference&diff=1044Virtuozzo / Linux Reference2013-03-11T23:56:30Z<p>Support: /* Recovering from GRUB failure */</p>
<hr />
<div>= Updating VZ =<br />
<br />
Update Virtuozzo to version 4.0.0<br />
Apply minor updates for Virtuozzo 3.0.0<br />
<br />
...<br />
When you get to the last screen, you will be given an option to <br />
<br />
( ) Automatically reboot after update<br />
(*) Reboot later manually<br />
<br />
make sure you choose to reboot later.<br />
<br />
= Migrating Templates =<br />
One nice feature VZ offers is the ability to run a virtual environment (VE or CT as it's now called) based on a particular ditribution on a host which may or may not share the same distribution. For instance, you could run a CT running Ubuntu while the host machine (Hardware Node or HN) is running CentOS. This is accomplished via the use of OS templates. As long as the HN contains the OS template on which a particular CT relies, it will run there. As such, if you want to move a CT from one HN to another, you will need to make sure the OS template exists there.<br />
<br />
Modern versions of VZ (>= 4.x) will check for and automatically move the OS template over to the host as you're (vz)migrating the CT over to a new HN. However we like to make sure the template is in place prior to moving the CT. <br />
<br />
The first step is to see if the template exists on the host. To see OS templates installed:<br />
<br />
2.x (shows OS and application templates):<br />
<pre># vzpkgls<br />
mod_ssl-rh9 20040624<br />
mysql-rh9 20030814 20050412<br />
openwebmail-rh9 20050427<br />
php-rh9 20050506<br />
phpBB-rh9 20041229<br />
postgresql-rh9 20050719<br />
sl-webalizer-rh9 20040119<br />
spamassassin-rh9 20040127<br />
tomcat-rh9 20040524<br />
usermin-rh9 20040116</pre><br />
<br />
>= 3.x (shows just OS templates, run without -O to see application as well):<br />
<pre># vzpkg list -O<br />
ubuntu-10.04-x86 2010-06-01 11:39:05<br />
ubuntu-11.04-x86 2011-06-22 14:23:59<br />
ubuntu-9.10-x86 2010-03-11 17:39:51<br />
centos-5-x86 2010-03-11 17:12:27<br />
debian-5.0-x86 2010-03-11 17:33:34</pre><br />
<br />
<br />
All template files are located:<br />
<br />
<pre># ls /vz/template/<br />
ColdFusion-fc1 jre-fc1 openwebmail-fc1 sl-webalizer-fc1<br />
ColdFusion-fc2 jre-fc2 openwebmail-fc2 sl-webalizer-fc2<br />
ColdFusion-rh9 jre-rh9 openwebmail-rh9 sl-webalizer-rh9<br />
PostNuke-fc1 jre-suse92 openwebmail-suse92 spamassassin-fc1<br />
PostNuke-fc2 jrun php-deb31 spamassassin-fc2<br />
PostNuke-rh9 mailman-deb31 php-fc1 spamassassin-rh9<br />
analog-fc1 mailman-fc1 php-fc2 spamassassin-suse92<br />
analog-fc2 mailman-fc2 php-rh9 suse-9.2<br />
analog-rh9 mailman-rh9 php-suse92 tomcat-fc1<br />
awstats-fc1 mailman-suse92 phpBB-fc1 tomcat-fc2<br />
awstats-rh9 mod_frontpage-fc1 phpBB-fc2 tomcat-rh9<br />
bbClone-fc1 mod_frontpage-fc2 phpBB-rh9 tomcat-suse92<br />
bbClone-fc2 mod_frontpage-rh9 phpmyadmin-deb31 urchin-fc1<br />
bbClone-rh9 mod_frontpage-suse92 postgresql-deb31 usermin-fc1<br />
conf mod_perl-deb31 postgresql-fc1 usermin-fc2<br />
debian-3.1 mod_perl-fc1 postgresql-fc2 usermin-rh9<br />
devel-deb31 mod_perl-fc2 postgresql-rh9 uw-imap-fc1<br />
devel-fc2 mod_perl-rh9 postgresql-suse92 uw-imap-fc2<br />
devel-suse92 mod_perl-suse92 proftpd-deb31 uw-imap-rh9<br />
fedora-core-1 mod_ssl-fc1 proftpd-fc1 vzredhat-7.3<br />
fedora-core-2 mod_ssl-fc2 proftpd-fc2 vzredhat-8.0<br />
jdk-deb31 mod_ssl-rh9 proftpd-rh9 webalizer-suse92<br />
jdk-fc1 mysql-deb31 proftpd-suse92 webmin-fc1<br />
jdk-fc2 mysql-fc1 redhat-7.2 webmin-fc2<br />
jdk-rh9 mysql-fc2 redhat-9 webmin-rh9<br />
jdk-suse92 mysql-rh9 redhat-as3-minimal<br />
jre-deb31 mysql-suse92 redhat-devel-9<br />
</pre><br />
<br />
What we see here are the base OS templates: <tt>redhat-9, fedora-core-1</tt> and supporting application templates: <tt>postgresql-rh9, mailman-rh9, jdk-fc1, mod_ssl-fc1</tt>. So if you wanted to copy over all of redhat 9 you'd need to copy the contents of:<br />
<pre># ls -d /vz/template/*rh9*<br />
/vz/template/ColdFusion-rh9 /vz/template/mod_frontpage-rh9 /vz/template/proftpd-rh9<br />
/vz/template/PostNuke-rh9 /vz/template/mod_perl-rh9 /vz/template/sl-webalizer-rh9<br />
/vz/template/analog-rh9 /vz/template/mod_ssl-rh9 /vz/template/spamassassin-rh9<br />
/vz/template/awstats-rh9 /vz/template/mysql-rh9 /vz/template/tomcat-rh9<br />
/vz/template/bbClone-rh9 /vz/template/openwebmail-rh9 /vz/template/usermin-rh9<br />
/vz/template/jdk-rh9 /vz/template/php-rh9 /vz/template/uw-imap-rh9<br />
/vz/template/jre-rh9 /vz/template/phpBB-rh9 /vz/template/webmin-rh9<br />
/vz/template/mailman-rh9 /vz/template/postgresql-rh9<br />
<br />
# ls -d /vz/template/*redhat-9*<br />
/vz/template/redhat-9<br />
</pre><br />
<br />
To get a template from one HN to another, it simply needs to be rsync'd over to the target HN. It's rsync'd over in this fashion:<br />
<br />
rsync -av -e ssh /vz/template/redhat-9 root@10.1.4.65:/vz/template<br />
rsync -av -e ssh /vz/template/*rh9 root@10.1.4.65:/vz/template<br />
<br />
Newer (>= 3.x) VZ employs "EZ" templates which are organized a little differently:<br />
<br />
<pre># ls /vz/template/<br />
cache CmdTool.log debian redhat-as3-minimal-20080630.tar.gz vc<br />
centos conf redhat-as3-minimal ubuntu<br />
<br />
# ls /vz/template/ubuntu/<br />
10.04 11.04 9.10<br />
# ls /vz/template/ubuntu/10.04/<br />
x86<br />
# ls /vz/template/ubuntu/10.04/x86/<br />
aap_1.091-1_all<br />
aap-doc_1.091-1_all<br />
adduser_3.112ubuntu1_all<br />
anacron_2.3-13.1ubuntu11_i386<br />
apache2_2.2.14-5ubuntu8.2_i386<br />
apache2_2.2.14-5ubuntu8.4_i386<br />
apache2_2.2.14-5ubuntu8.6_i386<br />
apache2_2.2.14-5ubuntu8.7_i386<br />
apache2_2.2.14-5ubuntu8.8_i386<br />
apache2_2.2.14-5ubuntu8.9_i386<br />
<...snip><br />
<br />
# ls /vz/template/cache/<br />
centos-5-x86.tar.gz suse-11.3-x86.tar.gz ubuntu-9.10-x86.tar.gz<br />
debian-5.0-x86.tar.gz ubuntu-10.04-x86.tar.gz<br />
fedora-core-14-x86.tar.gz ubuntu-11.04-x86.tar.gz<br />
</pre><br />
<br />
So what we see is the entire OS and all applications are in 1 directory, and organized by distribution and release version. So to move Ubuntu 10.04:<br />
<br />
rsync -av -e ssh /vz/template/ubuntu/10.04 root@10.1.4.69:/vz/template/ubuntu<br />
<br />
and you also need to move the cache:<br />
<br />
rsync -av -e ssh /vz/template/cache/ubuntu-10.04-x86.tar.gz root@10.1.4.69:/vz/template/cache/<br />
<br />
Once the transfer is complete, you will be able to run <tt>vzpkgls</tt> or <tt>vzpkg list -O</tt> and see the template(s) listed on the target HN.<br />
<br />
So far, this all supposes template doesn't exist on the target- very simple, just copy it over. If the template exists already on the target HN, this requires closer inspection. With older templates, simply moving over the templates would be the end of it- you see the version listed when you do a <tt>vzplgls</tt>:<br />
<br />
mysql-rh9 20030814 20050412<br />
<br />
this means there are 2 versions of the mysql package for RH9: 20030814 and 20050412<br />
As an aside, you can't copy over just 1 of them, but if you rsync'd over the <tt>mysql-rh9</tt> directory, you'd see 20030814 and 20050412 on the target HN. As long as the versions match up, you're good to go.<br />
<br />
With EZ templates, the versions are not as easy to decipher. When you look at the <tt>vzpkg list</tt> output, that simply tells you when a cache was created. A cache is simply a snapshot of all the data needed for the latest versions of the OS and it's applications at the time the cache was created. It is a date, and thus tells you nothing about the versions within. So for instance, if you had a cache for Ubuntu 10.04 from 2007 and you ran <tt>vzpkg create cache ubuntu-10.04-x86</tt> it would re-download the latest version of apache, mysql and so on. And any ''subsequent'' ubuntu 10.04 CT's created thereafter would use those newer versions, whereas older CT's based on 10.04 would use older templates. You can see how this works by looking at the files under:<br />
<br />
<pre># ls /vz/template/ubuntu/10.04/x86/<br />
aap_1.091-1_all<br />
aap-doc_1.091-1_all<br />
adduser_3.112ubuntu1_all<br />
anacron_2.3-13.1ubuntu11_i386<br />
apache2_2.2.14-5ubuntu8.2_i386<br />
apache2_2.2.14-5ubuntu8.4_i386<br />
apache2_2.2.14-5ubuntu8.6_i386<br />
apache2_2.2.14-5ubuntu8.7_i386<br />
apache2_2.2.14-5ubuntu8.8_i386<br />
apache2_2.2.14-5ubuntu8.9_i386<br />
<...snip><br />
</pre><br />
<br />
You can see that this template has in it 6 different versions of apache2. This means that this template was merged with other 10.04 templates and/or it has been cached a few times, each time a new apache was available. The unfortunate thing is it's almost impossible to know which CT's are using which version of apache so it's hard to try to prune this down and remove unwanted/unneeded applications.<br />
<br />
So, if you see another HN has Ubuntu 10.04 on it, you need to see/confirm that it has all the exact files your CT needs. You can run a test rsync to see what ''would'' be transferred (what's missing):<br />
<br />
rsync -avn --stats -e ssh /vz/template/ubuntu/10.04 root@10.1.4.69:/vz/template/ubuntu<br />
<br />
Based on the amount of data you get back from the rsync, you will be able to decide whether or not to remove the -n and allow the full transfer to go through. The downside to doing the transfer is the situation described above- you've permanently joined these templates and now you may have 10 versions of apache on the target HN. There's one other potential pitfall to this kind of merging- it will also potentially copy over shared files, for which there is no particular version, most of which are in <tt>/vz/template/ubuntu/10.04/x86/config</tt>: <br />
<br />
cat /vz/template/ubuntu/10.04/x86/config/os/default/repositories<br />
/vz/template/ubuntu/10.04/x86/config/os/default/repositories<br />
<br />
Here are some specific things to look out for. Case: copying centos-5 on virt19 to virt12. We dumped the rsync output to a file so we could inspect:<br />
<br />
[root@virt19 /vz/template]# rsync -an -e ssh /vz/template/centos/ root@10.1.4.62:/vz/template/centos/ > v12_c5<br />
<br />
(note, that can be dangerous, better to add on full path <tt>/vz/template/centos/5</tt>)<br />
<br />
So in the output file we see things like:<br />
<br />
<pre><br />
x86/base0/cachecookie<br />
x86/base0/primary.xml.gz <br />
x86/base0/primary.xml.gz.sqlite <br />
x86/base0/repomd.xml <br />
x86/base1/cachecookie <br />
x86/base1/filelists.xml.gz <br />
x86/base1/filelists.xml.gz.sqlite <br />
x86/base1/primary.xml.gz <br />
x86/base1/primary.xml.gz.sqlite <br />
x86/base1/repomd.xml<br />
x86/base2/cachecookie<br />
</pre><br />
<br />
Which make us nervous cause those are not versioned files, i.e.:<br />
5/x86/MAKEDEV-3.23-1.2.i386/usr/sbin/mksock<br />
<br />
So those files will replace existing files on v12 rather than the case of the latter where it's likely being added. So caution should be given and deference to whichever version is newer (which rsync should take care of, but that will depend on the date each file was created and perhaps not the actual data within).<br />
<br />
If we look further in the file we see:<br />
<br />
x86/psa-appvault-moodle-1.8-8202920080409011254.noarch/usr/local/psa/var/cgitory/moodle-1.<br />
9/htdocs/lib/editor/htmlarea/plugins/TableOperations/img/cell-split.gif<br />
<br />
and other files looking like:<br />
5/x86/plesk-base-9.0.1-cos5.build90090127.18.i586/etc/sw/keys/info<br />
<br />
Which means plesk is here. We don't need plesk on v12 (the CT moving over doesn't use it) so we rerun the rsync and exclude it:<br />
<br />
[root@virt19 /vz/template]# rsync -an -e ssh --exclude="plesk*" --exclude="psa*" /vz/template/centos/ root@10.1.4.62:/vz/template/centos/ > v12_c5<br />
<br />
and now it's much smaller:<br />
<br />
[root@virt19 /vz/template]# cat v12_c5|wc -l<br />
97104<br />
<br />
Before running with excludes it was:<br />
[root@virt19 /vz/template]# cat v12_c5|wc -l <br />
238238<br />
<br />
So again, look at what you're doing. Generally, combining templates is fine however.<br />
<br />
If you're happy with the merge, run the rsync again without the "-n" to let it actually do the transfer.<br />
<br />
Once done, don't forget to add the new template to the HN in Management -> Reference -> Templates<br />
<br />
= getting help =<br />
<br />
= vzup2date =<br />
TODO <br />
= Adding new OS/application templates =<br />
<br />
Virtuozzo will lag a bit behind the initial release of an OS, perhaps by a month or more. Further, a newly-released OS may be available, but there may not be many/any application templates available initially. It pays to wait a bit.<br />
<br />
A template is easily installed via [[#vzup2date|vzup2date]]:<br />
<br />
vzup2date -z<br />
<br />
You will be given a list of OS's to install. Select an OS, then customize that selection by checking/unchecking the application templates we do/don't want to include/offer.<br />
<br />
Generally, here are the app templates we include/offer:<br />
<pre>cyrus-imap<br />
devel<br />
imp<br />
jre<br />
jsdk<br />
mailman<br />
mod_perl<br />
mod_ssl<br />
mysql<br />
php<br />
phpmyadmin<br />
phppgadmin<br />
postgresql<br />
proftpd<br />
spamassassin<br />
squirrelmail<br />
tomcat<br />
vsftpd<br />
</pre><br />
<br />
We do not (any longer) offer webalizer or analog.<br />
<br />
After selecting and installing the OS and apps, you should ensure it's installed:<br />
<br />
vzpkg list -O<br />
<br />
Your new OS (ex. fedora-core-13-x86_64) will be listed, without a corresponding cache date:<br />
<pre><br />
[root@virt13 /vzconf]# vzpkg list -O<br />
centos-6-x86 2011-07-22 13:24:41<br />
centos-6-x86_64 2012-03-09 20:42:22<br />
centos-5-x86 2009-07-27 16:58:24<br />
centos-5-x86_64 2012-03-09 22:38:53<br />
ubuntu-11.10-x86_64 2012-03-01 23:13:33<br />
ubuntu-9.04-x86 2009-06-02 11:32:39<br />
ubuntu-12.04-x86_64 2012-05-29 13:50:50<br />
ubuntu-8.04-x86 2008-09-17 21:27:20<br />
ubuntu-8.10-x86 2009-03-13 13:58:57<br />
ubuntu-11.04-x86 2011-12-05 11:15:46<br />
ubuntu-7.04-x86 2008-09-24 16:42:50<br />
redhat-el6-x86_64<br />
fedora-core-13-x86_64 <br />
fedora-core-12-x86 2010-02-06 13:24:32<br />
fedora-core-15-x86 2011-12-05 11:36:43<br />
fedora-core-11-x86 2009-10-01 16:30:59<br />
fedora-core-14-x86<br />
fedora-core-16-x86_64 2012-03-13 11:40:23<br />
fedora-core-9-x86 2008-11-29 10:52:18<br />
debian-6.0-x86 2011-07-29 16:00:35<br />
debian-6.0-x86_64 2012-03-12 19:53:33<br />
debian-5.0-x86 2009-03-12 12:39:11<br />
</pre><br />
<br />
You can also confirm the apps installed:<br />
<pre>[root@virt13 /vzconf]# vzpkg list fedora-core-13-x86_64<br />
fedora-core-13-x86_64 2013-03-08 11:39:44<br />
fedora-core-13-x86_64 devel<br />
fedora-core-13-x86_64 php<br />
fedora-core-13-x86_64 proftpd<br />
fedora-core-13-x86_64 postgresql<br />
fedora-core-13-x86_64 phpmyadmin<br />
fedora-core-13-x86_64 mysql<br />
</pre><br />
<br />
Before you can use the OS you must cache it. To create the cache run:<br />
vzpkg cache [OS]<br />
<br />
ex: <tt>vzpkg cache fedora-core-13-x86_64</tt><br />
<br />
Now it's ready to use. In order to be able to use it with the [[VPS_Management#vm|vm]] command, you must create a file:<br />
jctmpl-[OS]<br />
<br />
ex: <tt>jctmpl-fedora-core-13-x86_64</tt><br />
<br />
In it, on line 1 we place the OS, followed by the app names, ex:<br />
<pre>more jctmpl-fedora-core-13-x86_64<br />
fedora-core-13-x86_64<br />
postgresql<br />
jsdk<br />
mod_ssl<br />
phpmyadmin<br />
mod_perl<br />
tomcat<br />
devel<br />
php<br />
mailman<br />
cyrus-imap<br />
squirrelmail<br />
proftpd<br />
mysql<br />
phppgadmin<br />
spamassassin<br />
</pre><br />
<br />
The packages will be installed in that order, so any pre-req's should be handled there.<br />
<br />
Now, when you run [[VPS_Management#vm|vm]] it will show you the newly-added OS and you may create a CT using it.<br />
<br />
The last few tasks are related to mgmt:<br />
<br />
1. add to mgmt->reference->templates (use the existing templates as an example). Once added, the new template will be included with the list of OS's available for the given HN.<br />
2. if this is going to be a new OS offered for ordering, it needs to be added to the order form and webpage.<br />
<br />
a. to get a list of applications and versions in the new OS, you should create a test CT, enter it and run <tt>dpkg -l|sort</tt> (or <tt>rpm -aq|sort</tt> in centos/fedora) and enter that list in the following places:<br><br />
<tt>/usr/local/www/jc_pub/[osname].html</tt><br><br />
<tt>/usr/local/www/signup/html/[osname].html</tt><br><br />
Note, you will have to move the most recent OS data from <tt>/usr/local/www/signup/html/[osname].html</tt> into the corresponding <tt>/usr/local/www/signup/html/[osname]_old.html</tt> updating the version links in both.<br />
<br />
b. to add the new OS to the order form, you will need to update <tt>/usr/local/www/signup/html/step1.html</tt> in several places (for regular and OSS orders) as well as updating the templateID which you can get from mgmt->reference->templates (or [[Management_System_/_Public_Website_/_Signup_/_Account_Manager#jc:ref_templates|jc:ref_templates]])<br />
<br />
<br />
<br />
Old method to obtain the templates:<br />
<br />
Go to http://www.parallels.com/en/virtuozzo/templates/catalog/<br />
<br />
Download the RPM's and install them<br />
<br />
vzpkg list<br />
to see the new packages/os templates<br />
<br />
vzpkg create cache "OS_EZ_template_name"<br />
to create a cache<br />
<br />
<br />
= Update OS template =<br />
<br />
Installation and update of the new version of OS template will not affect the existing containers. It will affect only the new containers, created after this update operation. In terms of fixing "vzctl enter" error and SSH connectivity to Ubuntu based containers, it is necessary to recreate the template and not to update it, so the commands to run are:<br />
<br />
<pre>rpm -Uhv ubuntu-8.04-x86-ez-4.0.0-9.swsoft.noarch.rpm<br />
vzpkg clean ubuntu-8.04-x86<br />
vzpkg remove cache ubuntu-8.04-x86<br />
vzpkg create cache ubuntu-8.04-x86</pre><br />
<br />
<br />
= Installing new SSL cert on VZCP =<br />
<br />
<pre>openssl req -days 1999 -new -x509 -nodes -out server.crt -keyout server.key<br />
<br />
US<br />
CA<br />
San Diego<br />
johncompanies.com<br />
johncompanies.com<br />
virt15ohncompanies.com<br />
linux@johncompanies.com<br />
<br />
cp -p /etc/httpd/conf/ssl.crt/server.crt /etc/httpd/conf/ssl.crt/server.crt.bak<br />
cp -p /etc/httpd/conf/ssl.key/server.key /etc/httpd/conf/ssl.key/server.key.bak<br />
cp server.crt /etc/httpd/conf/ssl.crt/server.crt<br />
cp server.key /etc/httpd/conf/ssl.key/server.key<br />
/etc/init.d/vzcp restart</pre><br />
<br />
= Moving virt (drives) to new hardware =<br />
<br />
If you have a case where a virt is dead (due to hardware failure), you can move the drives to a new hardware setup.<br />
Here are the steps and things to look for:<br />
<br />
Move the drives to the new box. Most likely, the drives will be recognized and there will be no problems booting up. On Dell PERC 5/6 cards you will need to import the new "foreign" volume via the raid BIOS.<br />
<br />
If the hardware (motherboard) is different, it’s possible the nic’s won’t be recognized. Look at /etc/modules.conf the ethernet drivers are either:<br />
<br />
<pre>alias eth0 tg3<br />
alias eth1 tg3</pre><br />
(supermicro)<br />
<br />
Or<br />
<br />
<pre>alias eth0 e100<br />
alias eth1 e1000</pre><br />
(intel)<br />
<br />
Or<br />
<br />
<pre>alias eth0 bnx2<br />
alias eth1 bnx2</pre><br />
(dell 29500)<br />
<br />
Also, you may see eth0 and eth1 swapped so if after confirming the eth devices are present (may require a reboot), then you may still need to swap the green/yellow network cables in the back to get them on the right port.<br />
<br />
VZ will not run (for long) on the new hardware, so you will need to get the license reissued. You may create/download a temporary license via their website for the interim.<br />
<br />
= Updating kernel on virt =<br />
<br />
We now use [[#vzup2date|vzup2date]] to update systems and kernels. What follows is what we used to do when we had to download kernels and install manually on old systems:<br />
<br />
<br />
1. install kernel and modules<br />
<pre># rpm -ivh vzkernel-enterprise-2.4.20-021stab028.12.777.i686.rpm \<br />
vzmodules-enterprise-2.4.20-021stab028.12.swsoft.i686.rpm<br />
vzkernel-smp ##################################################<br />
vzmodules-smp ##################################################</pre><br />
<br />
DO NOT USE the "rpm -Uhv" command to install the kernel, otherwise all the previously installed kernels may be removed from your system. <br />
<br />
2. update lilo/grub<br />
<br />
Lilo:<br />
<br />
vi /etc/lilo.conf<br />
<br />
rename old Virtuozzo kernel label from "linux+virtuozzo" to "virtuozzo_old"<br />
and add a new entry pointing to the newly installed virtuozzo kernel image.<br />
For example:<br />
<br />
<pre>prompt<br />
timeout=50<br />
default=linux+virtuozzo<br />
append="debug panic=5 console=tty console=ttyS0,38400"<br />
boot=/dev/sda<br />
map=/boot/map<br />
install=/boot/boot.b<br />
message=/boot/message<br />
linear<br />
<br />
image=/boot/vmlinuz-2.4.18-3smp<br />
label=linux<br />
initrd=/boot/initrd-2.4.18-3smp.img<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.18-3<br />
label=linux-up<br />
initrd=/boot/initrd-2.4.18-3.img<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-020stab009.5.777-enterprise<br />
label=2.4.20-9.5.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-020stab009.8.777-enterprise<br />
label=2.4.20-9.8.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-020stab009.21.777-enterprise<br />
label=2.4.20-9.21.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-020stab009.24.777-enterprise<br />
label=2.4.20-9.24.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-021stab028.3.777-enterprise<br />
label=2.4.20-28.3.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-021stab028.5.777-enterprise<br />
label=old_linux+vz<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-021stab028.12.777-enterprise<br />
label=linux+virtuozzo<br />
read-only<br />
root=/dev/sda1</pre><br />
<br />
With this configuration, the new kernel is the default kernel to use at boot. The original kernel entry has been retained with the label "old_linux+vz ".<br />
<br />
Grub:<br />
<br />
vi /boot/grub /menu.lst<br />
<br />
Add new entry at the top.<br />
For example:<br />
<br />
<pre>serial --unit=0 --speed=38400<br />
terminal --timeout=10 serial console<br />
default=0<br />
timeout=10<br />
splashimage=(hd0,0)/boot/grub/splash.xpm.gz<br />
title Virtuozzo 2.6.1 (2.4.20-021stab028.12.777-enterprise)<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-22.4.20-021stab028.12.777-enterprise root=/dev/sda1 console=tty0 \ console=ttyS0,38400<br />
title Virtuozzo 2.6.1 (2.4.20-021stab028.3.777-enterprise)<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.20-021stab028.3.777-enterprise root=/dev/sda1 console=tty0 \ console=ttyS0,38400<br />
title Red Hat Linux (2.4.18-3smp)<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.18-3smp ro root=/dev/sda1<br />
initrd /boot/initrd-2.4.18-3smp.img<br />
title Red Hat Linux-up (2.4.18-3)<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.18-3 ro root=/dev/sda1<br />
initrd /boot/initrd-2.4.18-3.img<br />
title Linux with Virtuozzo 2.5<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.20-020stab009.3.777-smp ro root=/dev/sda1<br />
title vz-enterpriseo<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.20-020stab009.21.777-enterprise ro root=/dev/sda1<br />
title vz-enterprise<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.20-020stab009.24.777-enterprise ro root=/dev/sda1 \ console=tty0 console=ttyS0,38400</pre><br />
<br />
3. run the lilo (if using lilo)<br />
<br />
<pre># lilo<br />
Added linux<br />
Added linux-up<br />
Added virtuozzo_old<br />
Added linux+virtuozzo *</pre><br />
<br />
4. reboot the server<br />
shutdown -r 23:05<br />
<br />
when it comes time for the shutdown, run [[VPS_Management#stopvirt|stopvirt]] to get things shut down faster<br />
<br />
<br />
= Remaking service VE (<=3.x only) =<br />
<br />
Occasionally the control panel for VEs (aka VZPP) will stop working and you just need to reinstall the service VE (which runs the control panels for all VEs on the HN). Here's how that's done :<br />
<br />
<pre>vzctl stop 1<br />
ipdel 1 69.55.234.152<br />
vzmlocal 1:2<br />
vzctl set 2 --save --onboot no<br />
vzsveinstall -t redhat-as3-minimal -d /root/Rel261/HW/RPMS -s 69.55.234.152 --skip-client</pre><br />
<br />
on 3.0:<br />
<br />
<pre>vzsveinstall -t redhat-as3-minimal -d /vz/Rel300/HW/RPMS -s 69.55.229.151 --skip-client<br />
<br />
vzctl stop 1<br />
vzctl mount 1<br />
vzctl mount 2<br />
/bin/cp -aRf /vz/root/2/home/vzagent0 /vz/root/1/home<br />
/bin/cp -aRf /vz/root/2/var/lib/mysql/VZADB /vz/root/1/var/lib/mysql<br />
/bin/cp -af /vz/root/2/etc/shadow /vz/root/1/etc/<br />
/bin/cp -aRf /vz/root/2/var/vzcp/static/vz/skins/ /vz/root/1/var/vzcp/static/vz<br />
/bin/cp -af /vz/root/2/etc/vzcp/pp/menu.xml /vz/root/1/etc/vzcp/pp/<br />
/bin/cp -af /vz/root/2/etc/vzcp/pp/dashboard.xml /vz/root/1/etc/vzcp/pp/<br />
<br />
vzctl umount 2<br />
vzctl umount 1<br />
vzctl start 1</pre><br />
<br />
<br />
= GRUB: boot once to a new kernel =<br />
<br />
To use a different kernel for the next reboot only:<br />
<pre>grub --no-floppy<br />
savedefault --default=N --once<br />
(where N is the number of the kernel you want to boot once)<br />
quit</pre><br />
<br />
Reboot your machine. It will boot using kernel "N".<br />
<br />
grub-reboot N<br />
will do essentially the same thing and prompt to reboot the machine<br />
<br />
<br />
= Recovering from GRUB failure =<br />
<br />
(example from DPE 2950)<br />
<br />
Boot to CEOS4 server CD (original install cd). Did this example with iso mounted via drac<br />
<br />
Boot: linux rescue<br />
<br />
<br />
= Make/edit an ISO on Linux =<br />
<br />
<pre>mount file.iso /mnt/iso/ -t iso9660 -o ro,loop=/dev/loop0<br />
mkdir /tmp/iso<br />
cp -aR /mnt/iso/* /tmp/iso/<br />
umount /mnt/iso/<br />
mkisofs -iso-level 2 -o /tmp/new.iso /tmp/iso/<br />
rm -fr /tmp/iso</pre><br />
<br />
http://www.damnsmalllinux.org/wiki/index.php/Installing_to_a_USB_Flash_Drive<br />
<br />
= Mount/edit USB drive on Linux =<br />
<br />
mount -t vfat -o loop /tmp/3wfirmware.iso /mnt/usb/</div>Supporthttps://wiki.jcihosting.com/index.php?title=Management_System_/_Public_Website_/_Signup_/_Account_Manager&diff=1043Management System / Public Website / Signup / Account Manager2013-03-11T23:55:24Z<p>Support: /* layout */</p>
<hr />
<div>= Overview =<br />
These systems are all running under the same apache instance on the mail server.<br><br />
<br />
The main ingredients are: apache 1.3.31, mysql, perl, mod_perl and template toolkit.<br />
<br />
Apache is listening on 69.55.230.9 ports 80 and 443<br />
<br />
The webserver can be stopped with:<br />
apachectl stop<br />
<br />
and must be started with:<br />
apachectl startssl<br />
<br />
If you run <tt>apachectl start</tt> none of the ssl-enabled pages will work.<br />
<br />
The database may be stopped and started as follows:<br />
/usr/local/etc/rc.d/mysql-server.sh stop<br />
/usr/local/etc/rc.d/mysql-server.sh start<br />
<br />
* webroot: <tt>/usr/local/www/</tt><br />
* logs: <tt>/var/log/httpd-error.log /var/log/httpd-access.log</tt><br />
* Apache config: <br />
<pre><br />
/usr/local/etc/apache/httpd.conf:<br />
<br />
-SNIP-<br />
<br />
<Directory /usr/local/www><br />
Options none<br />
</Directory><br />
<br />
<br />
<VirtualHost *><br />
SSLDisable<br />
DocumentRoot /usr/local/www/jc_pub<br />
ServerName johncompanies.com<br />
ServerAlias www.johncompanies.com<br />
ServerAlias newwww.johncompanies.com<br />
<br />
<Directory /usr/local/www/jc_pub><br />
Options FollowSymLinks<br />
</Directory><br />
<br />
Redirect /collocation http://www.johncompanies.com<br />
Redirect /colocation http://www.johncompanies.com<br />
<br />
ScriptAlias /cgi-bin/ "/usr/local/www/jc_pub/cgi-bin/"<br />
<Directory /usr/local/www/jc_pub/cgi-bin><br />
AllowOverride None<br />
Options None<br />
Order allow,deny<br />
Allow from all<br />
SetHandler None<br />
</Directory><br />
<br />
</VirtualHost><br />
<br />
<VirtualHost *><br />
ServerName mini.johncompanies.com<br />
SSLDisable<br />
DocumentRoot /usr/local/www/mini<br />
<br />
<Directory /usr/local/www/mini><br />
Options FollowSymLinks<br />
</Directory><br />
<br />
</VirtualHost><br />
<br />
<br />
<VirtualHost _default_:443><br />
<br />
DocumentRoot "/usr/local/www"<br />
<br />
DocumentRoot /usr/local/www<br />
<Directory /usr/local/www><br />
Deny from All<br />
</Directory><br />
<Directory /usr/local/www/mgmt><br />
Allow from All<br />
</Directory><br />
<Directory /usr/local/www/am><br />
Allow from All<br />
</Directory><br />
<Directory /usr/local/www/signup><br />
Allow from All<br />
</Directory><br />
<Directory /usr/local/www/images><br />
Allow from All<br />
</Directory><br />
<Directory /usr/local/www/media><br />
Allow from All<br />
</Directory><br />
<Files favicon.ico><br />
Allow from All<br />
</Files><br />
<br />
SSLEngine on<br />
SSLCipherSuite ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP:+eNULL<br />
<br />
# SSLCertificateFile /usr/local/etc/apache/mail.cert<br />
# SSLCertificateKeyFile /usr/local/etc/apache/mail.key<br />
#SSLCertificateFile /usr/local/etc/apache/ssl.crt/secure.crt<br />
SSLCertificateFile /usr/local/etc/apache/ssl.crt/secure.johncompanies.com.crt<br />
SSLCertificateChainFile /usr/local/etc/apache/ssl.crt/gd_bundle.crt<br />
SSLCertificateKeyFile /usr/local/etc/apache/ssl.key/secure.key<br />
<br />
PerlModule Apache<br />
<br />
# debug stuff<br />
PerlWarn On<br />
PerlTaintCheck Off<br />
PerlModule Apache::StatINC<br />
PerlInitHandler Apache::Reload<br />
# CustomLog /usr/local/apache/logs/access_log.mgmt common<br />
# ErrorLog /usr/local/apache/logs/error_log.mgmt<br />
<br />
PerlModule Apache::DBI<br />
# PerlModule Mgmt<br />
# PerlModule Auth<br />
<br />
PerlSetEnv MGMT_BASE /usr/local/www/mgmt<br />
PerlSetEnv COMMON_BASE /usr/local/www/common<br />
PerlSetEnv SIGNUP_BASE /usr/local/www/signup<br />
PerlSetEnv AM_BASE /usr/local/www/am<br />
PerlSetEnv MGMT_INDEX /mgmt/index.html<br />
PerlSetEnv SIGNUP_INDEX /signup/step1.html<br />
PerlSetEnv AM_INDEX /am/dashboard.html<br />
PerlSetEnv MGMT_LOG_LEVEL debug<br />
PerlSetEnv MGMT_LOG_FILE /usr/local/www/mgmt/Log/mgmt.log<br />
PerlSetEnv SIGNUP_LOG_LEVEL info<br />
PerlSetEnv SIGNUP_LOG_FILE /usr/local/www/signup/Log/signup.log<br />
PerlSetEnv AM_LOG_LEVEL debug<br />
PerlSetEnv AM_LOG_FILE /usr/local/www/am/Log/am.log<br />
PerlSetEnv PP_AUTH_TOKEN Pe3aLk5GdMblAyyLAv5vNYqipcynWdZJKdw1CmcGcIdOz74ujMrDYIov27i<br />
PerlSetEnv PP_URL 'https://www.paypal.com/cgi-bin/webscr'<br />
PerlSetEnv PP_EMAIL 'payments@johncompanies.com'<br />
PerlSetEnv JC_DOMAIN 'secure.johncompanies.com'<br />
PerlSetEnv CC_LOG_FILE /usr/local/www/mgmt/Log/cc.log<br />
PerlSetEnv DEV 0<br />
PerlRequire /usr/local/www/common/conf/startup.pl<br />
<br />
<Directory "/usr/local/www/mgmt"><br />
SetHandler perl-script<br />
PerlHandler Mgmt<br />
PerlSetVar JCMGMTPath /mgmt<br />
PerlSetVar JCMGMTLoginScript /mgmt/login.html<br />
<br />
<Files LOGIN><br />
AuthType Auth<br />
AuthName JCMGMT<br />
SetHandler perl-script<br />
PerlHandler Auth->login<br />
</Files><br />
<br />
<FilesMatch ".*\.html$|.*\.cgi|.*\.png"><br />
AuthType Auth<br />
AuthName JCMGMT<br />
PerlAuthenHandler Auth->authenticate<br />
PerlAuthzHandler Auth->authorize<br />
require valid-user<br />
</FilesMatch><br />
</Directory><br />
<br />
<Directory "/usr/local/www/am"><br />
SetHandler perl-script<br />
PerlHandler AM<br />
PerlSetVar JCAMPath /am<br />
PerlSetVar JCAMLoginScript /am/login.html<br />
#PerlSetVar JCAMExpires +1m<br />
<br />
<Files LOGINAM><br />
AuthType AMAuth<br />
AuthName JCAM<br />
SetHandler perl-script<br />
PerlHandler AMAuth->login<br />
</Files><br />
<br />
<Files PASSRESET><br />
SetHandler perl-script<br />
PerlHandler AMPassR<br />
</Files><br />
<br />
<FilesMatch ".*\.html$|.*\.cgi"><br />
AuthType AMAuth<br />
AuthName JCAM<br />
PerlAuthenHandler AMAuth->authenticate<br />
PerlAuthzHandler AMAuth->authorize<br />
require valid-user<br />
</FilesMatch><br />
</Directory><br />
<br />
<Directory /usr/local/www/mgmt/static><br />
SetHandler None<br />
</Directory><br />
<br />
<Directory /usr/local/www/am/static><br />
SetHandler None<br />
</Directory><br />
<br />
<Directory /usr/local/www/mgmt/bwgraphs><br />
SetHandler None<br />
</Directory><br />
<br />
<Directory /usr/local/www/am/bwgraphs><br />
SetHandler None<br />
</Directory><br />
<br />
<Directory /usr/local/www/images><br />
SetHandler None<br />
</Directory><br />
<br />
<Directory /usr/local/www/media><br />
SetHandler None<br />
</Directory><br />
<Directory /usr/local/www><br />
Options FollowSymlinks<br />
</Directory><br />
<br />
ScriptAlias /mgmt/cgi/ "/usr/local/www/mgmt/cgi/"<br />
<br />
<Directory /usr/local/www/mgmt/cgi><br />
AllowOverride None<br />
Options None<br />
Order allow,deny<br />
Allow from all<br />
SetHandler None<br />
AuthType Auth<br />
AuthName JCMGMT<br />
PerlAuthenHandler Auth->authenticate<br />
PerlAuthzHandler Auth->authorize<br />
require valid-user<br />
</Directory><br />
<br />
<Directory "/usr/local/www/signup"><br />
SetHandler perl-script<br />
PerlHandler Signup<br />
</Directory><br />
<br />
Alias /mgmt/mrtg "/usr/local/www/mgmt/mrtg/data"<br />
<Directory /usr/local/www/mgmt/mrtg/data/><br />
DirectoryIndex index.cgi<br />
SetHandler None<br />
Options ExecCGI<br />
AddHandler cgi-script .cgi<br />
</Directory><br />
<br />
Alias /mgmt/rrd "/usr/local/www/mgmt/mrtg/rrd"<br />
<Directory /usr/local/www/mgmt/mrtg/rrd/><br />
DirectoryIndex index.html<br />
SetHandler None<br />
</Directory><br />
<br />
ScriptAlias /mgmt/bb/cgi-bin/ /usr/home/bb/bbsrc/bb1.9i-btf/web/<br />
Alias /mgmt/bb "/usr/home/bb/bbsrc/bb1.9i-btf/www"<br />
<Directory /usr/home/bb/bbsrc/bb1.9i-btf/www/gifs><br />
SetHandler None<br />
</Directory><br />
<Directory /usr/home/bb/bbsrc/bb1.9i-btf/web><br />
AllowOverride None<br />
Options None<br />
Order allow,deny<br />
Allow from all<br />
PerlSetVar JCMGMTLoginScript /mgmt/login.html<br />
AuthType Auth<br />
AuthName JCMGMT<br />
PerlAuthenHandler Auth->authenticate<br />
PerlAuthzHandler Auth->authorize<br />
require valid-user<br />
</Directory><br />
<Directory /usr/home/bb/bbsrc/bb1.9i-btf/www><br />
PerlSetVar JCMGMTLoginScript /mgmt/login.html<br />
AuthType Auth<br />
AuthName JCMGMT<br />
PerlAuthenHandler Auth->authenticate<br />
PerlAuthzHandler Auth->authorize<br />
require valid-user<br />
</Directory><br />
<br />
Alias /mgmt/awstatsclasses "/usr/local/www/mgmt/awstats/wwwroot/classes/"<br />
Alias /mgmt/awstatscss "/usr/local/www/mgmt/awstats/wwwroot/css/"<br />
Alias /mgmt/awstatsicons "/usr/local/www/mgmt/awstats/wwwroot/icon/"<br />
ScriptAlias /mgmt/awstats/ "/usr/local/www/mgmt/awstats/wwwroot/cgi-bin/"<br />
Alias /mgmt/icon/ "/usr/local/www/mgmt/awstats/wwwroot/icon/"<br />
<br />
<Directory "/usr/local/www/mgmt/awstats/wwwroot/icon"><br />
SetHandler None<br />
Options Indexes MultiViews<br />
AllowOverride None<br />
Order allow,deny<br />
Allow from all<br />
</Directory><br />
<br />
<br />
<Directory "/usr/local/www/mgmt/awstats/wwwroot"><br />
PerlSetVar JCMGMTLoginScript /mgmt/login.html<br />
AuthType Auth<br />
AuthName JCMGMT<br />
PerlAuthenHandler Auth->authenticate<br />
PerlAuthzHandler Auth->authorize<br />
require valid-user<br />
SetHandler None<br />
Options None<br />
AllowOverride None<br />
Order allow,deny<br />
Allow from all<br />
</Directory><br />
</VirtualHost><br />
</pre><br />
<br />
== Template Toolkit ==<br />
<br />
http://www.template-toolkit.org/docs/manual/index.html<br />
<br />
All our dynamic content pages (i.e. anything but jcpub) makes use of the Template Toolkit (TT) framework to create/display our pages. This package allows us to create customied apache handlers via the use of mod_perl. The result is an easy to deploy, easy to develop, flexible and efficient platform.<br />
<br />
All these sites are organized under <tt>/usr/local/www</tt> and consist/rely on several components:<br />
<br />
* common/conf/startup.pl - this contains all the use/require code that brings in all the required libraries and modules required to run our sites and TT. It is run once as apache starts up and applies to all sites.<br />
<br />
* common/conf/Lib - this directory contains all common libraries, primarly those dealing with database access and access to individual tables. <br />
<br />
* mysql:jc - the main table containing everything for mgmt/am/signup sites is contained in the <tt>jc</tt> database. All data is accessed via the mysql user <tt>jc</tt> - this is so we can impose restrictions on what that user may do. That access info is stored in <tt>common/Lib/DBI.pm</tt><br />
The exception to that is the <tt>traffic</tt> database which used to be stored on mail, but has since been exported to run locally on the bwdb server itself. i.e. mgmt/am contact the remote traffic database running elsewhere for that data. That access data is stored in <tt>common/Lib/DBI_BWDB.pm</tt><br />
<br />
Part of TT's efficiency comes from static and natively compiled code/pages. Those pages are stored in:<br />
* mgmt: <tt>/tmp/mgmt_templates/</tt><br />
* am: <tt>/tmp/am_templates</tt><br />
* signup: <tt>/tmp/signup_templates</tt><br />
You could safely delete all of these and apache will just recreate them - though the directory holding the templates must exist, with the right permissions.<br />
<br />
To make editing of pages easier we enable apache modules/directives which direct apache to rebuild these pages when it notices there's a difference (httpd.conf):<br />
PerlModule Apache::StatINC<br />
PerlInitHandler Apache::Reload<br />
<br />
However, any changes made to anything under <tt>Lib</tt> requires an immediate apache restart to take effect and to keep the site running- any changes w/o a restart will break the site.<br />
<br />
Among the directives in httpd.conf you will see a series of variables which are important as they, among other things, direct the mod_perl handlers where certain data is stored:<br />
<br />
This is where all the files for a particular set of pages and unique handler are located:<br />
PerlSetEnv MGMT_BASE /usr/local/www/mgmt<br />
PerlSetEnv COMMON_BASE /usr/local/www/common<br />
PerlSetEnv SIGNUP_BASE /usr/local/www/signup<br />
PerlSetEnv AM_BASE /usr/local/www/am<br />
<br />
This is the default page to present, if none is specified:<br />
PerlSetEnv MGMT_INDEX /mgmt/index.html<br />
PerlSetEnv SIGNUP_INDEX /signup/step1.html<br />
PerlSetEnv AM_INDEX /am/dashboard.html<br />
<br />
Indicates the verbosity and location of the handler logfile (these log files only contain logging which we do internally from our code- these differ from the apache logs which log simple hits and server errors):<br />
PerlSetEnv MGMT_LOG_LEVEL debug<br />
PerlSetEnv MGMT_LOG_FILE /usr/local/www/mgmt/Log/mgmt.log<br />
PerlSetEnv SIGNUP_LOG_LEVEL info<br />
PerlSetEnv SIGNUP_LOG_FILE /usr/local/www/signup/Log/signup.log<br />
PerlSetEnv AM_LOG_LEVEL debug<br />
PerlSetEnv AM_LOG_FILE /usr/local/www/am/Log/am.log<br />
<br />
Variables used for pp API access:<br />
PerlSetEnv PP_AUTH_TOKEN Pe3aLk5GdMblAyyLAv5vNYqipcynWdZJKdw1CmcGcIdOz74ujMrDYIov27i<br />
PerlSetEnv PP_URL 'https://www.paypal.com/cgi-bin/webscr'<br />
PerlSetEnv PP_EMAIL 'payments@johncompanies.com'<br />
<br />
Our base url:<br />
PerlSetEnv JC_DOMAIN 'secure.johncompanies.com'<br />
<br />
Our credit card info logging file:<br />
PerlSetEnv CC_LOG_FILE /usr/local/www/mgmt/Log/cc.log<br />
<br />
Indicates we are/not in a dev environment:<br />
PerlSetEnv DEV 0<br />
<br />
= Public Website (jcpub) =<br />
<br />
* domain: <tt>http://www.johncompanies.com http://johncompanies.com</tt><br />
* webroot: <tt>/usr/local/www/jc_pub</tt><br />
<br />
Our public-facing website is all static, standard HTML. We have some light javascripting on some pages, but by in large it's a very WYSIWYG site setup.<br />
<br><br />
<br><br />
<br />
= Signup (signup) =<br />
== Setup / organization / notes ==<br />
<br />
* domain: <tt>https://secure.johncompanies.com/signup/step1.html</tt><br />
* webroot: <tt>/usr/local/www/signup</tt><br />
<br />
Directory structure:<br />
* am: contains the main handler <tt>Signup.pm</tt> and the ipn activity log <tt>ipn.log</tt><br />
* signup/Lib: contains the log module <tt>SLog.pm</tt><br />
* signup/Log: contains the log <tt>signup.log</tt>, and cardit card activity log <tt>cc.log</tt><br />
* signup/Plugin: contains the plugin module <tt>AMP.pm</tt> and the form fill in module <tt>FillInForm.pm</tt><br />
* signup/html: contains all web pages<br />
<br />
=== Usage - server order ===<br />
<br />
Once a customer leaves our public site and visits any page under https://secure.johncompanies.com/signup/ they are in the signup section of our site. There isn't a strict rule that all these pages are signup-related, for instance there are some pages related to payments that run in the signup namespace. This is for convenience since this codebase doesn't require a login/authentication.<br />
<br />
The signup code primarly deals with the ordering process. Customers entering the signup process via https://secure.johncompanies.com/signup/step1.html will have the ability to order any product. Several links throughout our site pass optional parameters to this page to pre-select the product the customer has selected. For instance, a link to order a linux package looks like https://secure.johncompanies.com/signup/step1.html?svc=linux<br />
<br />
In step 1 the customer is prompted to select the package/service level, their OS, any backup space (a la carte), ownership info and contact info. All customers must provide a billing and admin (technical) contact- both may be the same person. They may also (optionally) provide an alternate contact for the billing and admin contacts. What this means is if we are unable to reach them (perhaps due to some outage on their server, where their primary email is hosted) this is how we'd reach them. They must provide at least 1 non-alternate billing and admin contact (i.e. if they provide just 1 contact and mark it billing and admin, it cannot also be marked alternate. They would need to add another contact and mark it as alternate.)<br />
<br />
To update the packages available or OS choices for colo's, edit:<br><br />
<tt>signup/html/step1.html</tt><br />
<br />
When editing the OS templates for VPSs, make sure the values for the input options matches the existing template_id (jc:ref_templates.template_id). Also remember that there are 2 sets of linux and freebsd order options to edit (1 normal and 1 oss discount).<br />
<br />
<br />
In step 2 the customer is prompted for more info based upon their service selection:<br />
<br />
<br />
any server:<br />
* hostname<br />
* ips - if the package allows add-on ips they are given the option to purchase more<br />
* bandwidth - they are given the option to purchase additional bandwidth<br />
* bandwidth overage - they are prompted to choose from 3 options for when they use all their bandwidth in a given month: 1. stop traffic, 2. bill for additional usage up to an amount they specify, 3. allow unlimited bandwidth and bill accordingly<br />
* payment method (cc, paypal, echeck (paypal), invoice)<br />
* existing customer - we ask if they are an existing customer so we know if this server order is new or a replacement on an existing account or if it is to be set up under a new account<br />
<br />
<br />
vps:<br />
* disk space - if the package allows add-on space they are given the option to purchase more<br />
<br />
<br />
colo: <br />
* time zone to set the server to<br />
* # ips to assign initially, and a reason to assign more<br />
* disk partitioning (this should add up to the amount of the disk included with the system)<br />
* special instructions<br />
* (nfs) backup space - provided the option to purchase extra a la carte<br />
<br />
<br />
In step 3 they are shown the total price for the selected services and given the choice of billing intervals: 1/6/12 months. Discounts and savings are displayed. They are requested to tell us how they found us here. However, if they indicated they were an existing customer on step 2 and they are replacing or adding to an existing account, they are also prompted for info:<br />
* hostname/ip of the server being replaced or the account to which we are adding the server<br />
* selection of a new, permanent ip or a temporary ip<br />
* how to handle the information provided in the signup in regards to existing data on file (replace/merge)<br />
<br />
<br />
In step 4 they are shown a summary of the entire order, and prompted to accept our TOS.<br />
Once they submit this page, their signup is registered regardless of whether they go on to make payment.<br />
<br />
The next step depends on their selected payment option. If they are paying via credit card they are prompted to enter their card info, then shown a payment confirmation screen upon successful payment.<br />
<br />
If they are paying via paypal, they're sent off to paypal to setup payment, then returned to our site to complete the payment and shown a payment confirmation screen upon successful payment.<br />
<br />
Note on payment- they can reload this page and it will not result in a reorder, in fact they cannot go back and accidentally reorder at all. If the customer is ording a colo no payment is taken, rather we do an authorization or setup their paypal payment, but we don't actually take funds until the server is in service.<br />
<br />
=== Usage - colo quote ===<br />
<br />
Another facility provided by the signup code is our colo quote tool. A customer visiting this page https://secure.johncompanies.com/signup/colo_quote.html is able to request a quote on a customized server. The requests end up in the New Signups section of mgmt. Once responded to, the customer is given a special signup link which leads them into the signup process where they may order the customized server.<br />
<br />
In the colo quote tool they are asked to provide:<br />
* processor<br />
* RAM<br />
* raid level<br />
* drive size<br />
* IPs<br />
* bandwidth<br />
* nfs backup space<br />
* OS<br />
* requested delivery date<br />
* OS install option (JC-installed or self-install via IPKVM)<br />
* current customer Y/N<br />
* referral source<br />
* comments/questions<br />
<br />
To update the list of OS's or the hardware options, you'll edit:<br />
<tt>signup/html/colo_quote.html</tt><br />
<br />
As long as the field names don't change, this will not break compatibility with the quote build tool in mgmt.<br />
<br />
= Management System (mgmt) =<br />
== Setup / organization / notes ==<br />
<br />
* domain: <tt>https://secure.johncompanies.com/mgmt</tt><br />
* webroot: <tt>/usr/local/www/mgmt</tt><br />
<br />
Directory structure:<br />
* mgmt: contains the main handler <tt>Mgmt.pm</tt> and the authentication (<tt>Auth.pm</tt>) handler, the bigbrother status file <tt>bb.html</tt>, the credit card gpg key <tt>pubring.gpg</tt><br />
* mgmt/Lib: contains the log module <tt>MLog.pm</tt><br />
* mgmt/Log: contains the log <tt>am.log</tt>, and ats activity log <tt>ats.log</tt><br />
* mgmt/Plugin: contains the plugins required for the various mgmt pages<br />
* mgmt/html: contains all web pages<br />
* mgmt/static: contains static, non-interpreted content like graphics and javascripts<br />
* mgmt/awstats: our stats package. Not controlled by TT, just lives here so you have to authenticate in mgmt to see it<br />
* mgmt/bwgraphs: deprecated? b/w graphing libraries and code<br />
* mgmt/mrtg: mrtg graphs. Not controlled by TT, just lives here so you have to authenticate in mgmt to see it<br />
<br />
All functions located in mgmt are password-protected. Once you're logged in, as long as you keep your browser open (maintain session cookie) you will not need to login again. Users are authenticated via table: <tt>jc.users</tt> To add users or change password, update this table.<br />
<br />
=== menu ===<br />
<br />
==== layout ====<br />
<br />
* Pastes<br />
This link is twofold: first it is a dropdown menu of some frequently-used responses to customer questions. Clicking on any of these items will copy the text of the response into your clipboard- provided you are running in a compatible browser (IE). Clicking on the "Pastes" button itself will open up a new window containing many more pastes. Clicking on the links at the top will also copy in the text, which you may review in the page below. <br />
<br />
Updating: <tt>mgmt/html/pastes.html</tt><br />
<br />
* Load <br />
Opens up a new window showing mrtg-style load figures for all our hardware, and network traffic for several important servers (firewall, backup). Clicking on each graph drills down for more data.<br><br />
See [[#mrtg|mrtg]]<br />
<br />
* BigBrother <br />
Opens up a new window to the Big Brother monitoring system.<br><br />
See [[BigBrother|BigBrother]]<br />
<br />
* New Signups <br />
Opens up a new window to the order processing page.<br><br />
See [[#New_Signups|New Signups]]<br />
<br />
* PayFlow<br />
Opens up a new window to the the PayFlow manager.<br />
See [[Payment_System#PayFlow|PayFlow]]<br />
<br />
* Cabinet map<br />
Opens up a new window to the [[Cabinetmap|cabinet map]] in this wiki<br />
<br />
* Monitoring<br />
Opens up windows to monitor the following:<br><br />
ATS - monitor/manage ATSs at i2b. See [[#ats|ATS control]]<br><br />
Backups - monitor the size and timing of backups running from our servers to backup1. See [[#backupmonitor|backup monitoring]]<br><br />
Colo Monitor - a bb instance that only monitors customers who have requested/receive the service. See [[BigBrother#monitor.johncompanies.com|monitor.johncompanies.com]]<br><br />
ATS-n - the web-based interface for the ATS. See [[ATS|ATS]]<br><br />
Switches - mrtg pages for our switches<br><br />
System Counts - display's how many VPSs are on each system (according to the database- may not be 100% accurate)<br><br />
XX VZPP - VZPP (Virtuozzo Power Panel) for each server. See [[Virtuozzo_Reference|Virtuozzo Reference]]<br><br />
<br />
* Payments<br />
Paypal history - takes you to the paypal page where you can do advanced searches based on email/subid/txnid<br><br />
2Checkout V2 - 2Checkout site<br><br />
Invoice Pmt Entry - allows easy entry of payments for any customer paying via invoice<br><br />
Arrears list - shows customers who have a balance due<br><br />
Pmt notices - shows outstanding payment invoices, primarily for customers paying via paypal<br><br />
Pfp batch - shows items pending processing via payflow. See [[#pfp_batch|pfp_batch]]<br><br />
Payment links - deprecated<br><br />
Payment audit - show(ed) discrepancies between pp sub and billing sub<br><br />
Packages - shows all our current/past packages. See [[#pkgs|packages]]<br><br />
Orphaned R.pmts - shows all paypal subs in the system which are not associated with an account. These should only/all be from people who signed up for payment fraudulently, and all subs should be in cancelled/disabled state. The only real reason to visit this page is in the case of a customer who is adding a server and who's signed up, you would link his paypal sub to his account here.<br><br />
IPN Input - allows you to manually input a paypal IPN into our system<br><br />
Request pmt - broken. ??<br><br />
<br />
* Reference<br />
IPKVM - links to our IPKVMs. See [[IPKVM|IPKVM]]<br><br />
VZ Ticket - submit a trouble ticket for help with VZ. See [[Virtuozzo_/_Linux_Reference#getting_help|getting help]]<br><br />
IP Map - display's ip allocation information. See [[#ipmap|ipmap]]<br><br />
Acct Mgr - quick link to the [[#Account_Manager_.28AM.29|Account Manager]]<br><br />
Asset DB - our asset database. See [[#asset_database|asset database]]<br><br />
Bandwidth - bandwidth query page. See [[Bandwidth_Management#Reporting|Reporting]]<br><br />
Web stats - awstats page<br><br />
Domaintools - handy whois resource<br><br />
CrashLog - tool to log the details of a crash. See [[#crash_log|crash log]]<br><br />
DosLog - tool to log the details of a dos attack. See [[#dos_log|dos log]]<br><br />
URL: Linux VPS - takes you right to the linux vps order page<br><br />
URL: FreeBSD VPS - takes you right to the linux freebsd order page<br><br />
URL: Dedicated - takes you right to the dedicated server order page<br><br />
Notices - takes you to the notices tool. See [[#notices|notices]]<br><br />
Templates - template management/tracking. See [[#templates|templates]]<br><br />
VZ Reference - VZ reference/knowledge base at the parallels website<br><br />
VZ Templates - template repository at parallels website. The best method for installing new templates is <tt>vzup2date</tt>. See [[Virtuozzo_Reference#adding_new_templates|adding new templates]]<br><br />
Bugzilla - our bugzilla site<br><br />
redit.com - customer portal at redit. Get realtime bandwidth and cabinet power usage<br><br />
Manual - old/deprecated<br><br />
johncompanies.com - our website<br><br />
Linux welcome - generic welcome email<br><br />
BSD welcome - freebsd welcome email<br><br />
Debian welcome - debian/ubuntu welcome email<br><br />
Spam check - site to check if an ip is blacklisted<br><br />
DEV mgmt - mgmt site on dev server<br><br />
DEV am - AM on dev server<br><br />
DEV jcpub - public site on dev server<br><br />
<br />
* Wiki<br />
This.<br><br />
<br />
==== editing ====<br />
The dropdown menu is driven by a couple files:<br><br />
<tt>mgmt/static/HM_Arrays.js<br><br />
mgmt/static/HM_Loader.js</tt><br />
<br />
to change or add menu items, edit HM_Arrays.js<br><br />
covering every topic would be lengthy but here are some tips:<br />
<br />
* in each section, each entry but the last must end with a comma<br />
* each entry is formatted: <tt>["name", action, 1,0,N]</tt> where N is 0 that means this is a simple button, when N is 1 it is a dropdown and another menu is attached.<br />
* in the case of the first, top-level group:<br />
<pre>["Pastes","JavaScript:win1=window.open('','win1'); win1.location='https://secure.johncompanies.com/mgmt/pastes.html'; void(0)",1,0,1],<br />
["Load","JavaScript:win7=window.open('','win7'); win7.location='https://secure.johncompanies.com/mgmt/mrtg/index.cgi'; void(0)",1,0,0],<br />
["BigBrother","JavaScript:win9=window.open('','win9'); win9.location='https://secure.johncompanies.com/mgmt/bb/'; void(0)",1,0,0],<br />
["New Signups","https://secure.johncompanies.com/mgmt/pending.html",1,0,0],<br />
["PayFlow","JavaScript:winpf=window.open('','winpf'); winpf.location='https://manager.paypal.com/'; void(0)",1,0,0],<br />
["Cabinet map","JavaScript:win25=window.open('','win25'); win25.location='https://secure.johncompanies.com/mgmt/cabinetmap.html'; void(0)",1,0,0],<br />
["Monitoring","",1,0,1],<br />
["Payments","",1,0,1],<br />
["Reference","",1,0,1],<br />
["Wiki","JavaScript:winwiki=window.open('','winwiki'); winwiki.location='https://wiki.johncompanies.com/'; void(0)",1,0,0]<br />
],</pre><br />
<br />
You see that Pastes is the first menu item and that it has a submenu. That submenu is defined in the <tt>HM_Array2_1</tt> array. Likewise, the 7th menu item, Monitoring, has a submenu defined in <tt>HM_Array2_7</tt>. And that furter has nested menus, the first item in the Monitoring dropdown has a nested menu in <tt>HM_Array2_7_1</tt> , and the 13th item in <tt>HM_Array2_7_13</tt><br />
<br />
* when setting a window to open a new window (most menu items) it's important to keep track of the window name if you want each link to open a unique window. i.e. <tt>JavaScript:win10a=window.open('','win10a'...</tt> make the <tt>win10a</tt> name is unique.<br />
<br />
== Searching ==<br />
<br />
== Customer main view page ==<br />
Most interaction in mgmt is done on or from the main customer page. This is the main landing page you'll get if you're searching for a customer. From this page you can see and edit:<br />
<br />
* systems/services <br />
* ownership info<br />
* bandwidth usage info<br />
* contact info<br />
* billing info<br />
<br />
=== Systems ===<br />
This table displays the systems and services to which a customer has subscribed. Depending on the type of system, certain data will appear. Namely:<br />
* hostname<br />
* root directory (VPS only)<br />
* OS (VPS only)<br />
* disk space allocated (VPS only)<br />
* CT id (linux VPS only)<br />
* sysid<br />
* IPs <br />
* dates of start/cancel/stop<br />
* package ID/smount+interval/next bill date<br />
<br />
<br />
Any item black and underlined (this goes for anything in mgmt) is clickable and will copy into your clipboard (IE only). Items in this table with special background coloring indicates:<br />
* grey = cancelled system. To un-stop a system, databases need to be manually edited- so be careful.<br />
* blue = monotired by castle. If you uncheck and save the monitor flag (in the system edit screen), you must also update the tracking sheet (a0, p4). Before setting up a probe with castle, make sure that there are no notes indicating the customer has previously requested not to be monitored ("don't monitor"), or “can't monitor” which means castle just can't monitor them for some reason. <br />
* yellow = pending shutdown (cancel date set)<br />
<br />
To setup the system for (future) cancel, click the "cancel" link. You will be prompted to provide a cancel date. This will indicate the date you intend to shut down the system. Usually this is in the future since the customer pre-pays for service. Also, you want to set at least 1 day in the future to allow the customer to get the "[[System-generated_Notifications#.22IMPORTANT_REMINDER_-_SYSTEM_SHUT_DOWN_TODAY.22|Shutdown today]]" email, clueing them in to the fact that their server is going away. The 2 quick links for 5-days and 10-days will set the date in the future (5 or 10 days), both optimized to allow the customer to receive the shutdown reminder notices. If you want the customer not to receive shutdown reminder notices, you should set that field to no. If you want to un-cancel a system, go into the edit screen and remove/clear the cancel date.<br />
<br />
<br />
When a system has been shut down (or pulled from the rack in the case of a colo), click the "stop" link and set the date it was stopped. If this is the last active system on the account, it will mark the customer cancelled <br />
<br />
<br />
To edit aspects of the system click the "edit" link under the "actions" column.<br />
<br />
==== edit ====<br />
From this screen you may edit most if not all aspects of the system in question. <br />
* machine - pretty simple, what host is the VPS on or is this a (jc owned) managedcolo or (customer owned) colo <br />
* directory - (VPS only) where the VPS is located on the host<br />
* disk - (VPS only) how much they actually have. If you need to set this higher, edit systems.disk<br />
* hostname<br />
* veid - linux (VPS only) aka CT id<br />
* os tempalte - (VPS only) this will auto-populate with the templates/os on the host selected in the machine field. To add templates you need to do so via the [[#Templates|Templates]] page<br />
* password - (VPS only) if this is not set this system was setup with our universal/standard password: p455agfa or 8ico2987<br />
* ats port - (colo only) port and ATS to which this server is connected<br />
* cabinet - (colo only) location of colo<br />
* ips - ips assigned to server and the ability to assign more (space delimited if adding mult). A handy link will open ipmap and clicking ips should copy back the ip to the new ip field.<br />
* dates - start/stop/cancel<br />
* monitored - indicate whether this system is on castle's probe map<br />
* base, disk, ip packages - adjust all aspects of packages. If a field is red that means there's a mismatch between what the customer has and what the package covers. i.e. if they have 50 GB of disk and the package only gives 30, you'd need to add a disk package for 20 GB or adjust the base package to include 50 GB. Anything yellow indicates an altered/custom value. i.e. if the package comes with 6 IPs and you set it to 5, that will be highlighted in yellow. Or if the price was $19 and you make it $17, prices will be yellow'd. Green fields in the Totals row indicate a good matchup for the given attribute (i.e. disk allocated to server is the same as disk provided by package(s)). <br />
<br />
If you are moving a customer to a new machine, that may be done here- the old machine will be marked as stopped and show up as an additional row in the view screen. So really that is a stop and add operation behind the scenes in mgmt.<br />
<br />
==== add ====<br />
<br />
This works very similar to edit, and like it implies will add a new system to the customer.<br />
<br />
=== Owner Info ===<br />
<br />
=== Contact Info ===<br />
<br />
=== Notes ===<br />
<br />
=== Bandwidth ===<br />
<br />
=== Billing History ===<br />
<br />
=== Payment Sources ===<br />
<br />
=== Outstanding Invoices ===<br />
<br />
=== (Legacy) Billing ===<br />
<br />
=== (Legacy) Recurring Payments ===<br />
<br />
== ats ==<br />
<br />
== backupmonitor ==<br />
<br />
== asset database ==<br />
<br />
== crash log ==<br />
<br />
== dos log ==<br />
<br />
== notices ==<br />
<br />
== New Signups ==<br />
<br />
== Templates ==<br />
<br />
== mrtg ==<br />
* domain: <tt>https://secure.johncompanies.com/mgmt/mrtg/index.cgi/ https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p3&path=</tt><br />
* webroot: <tt>/usr/local/www/mgmt/mrtg</tt><br />
All configuration is done via *.cfg files. The main load graph is found in mrtg1.cfg<br />
All other config files are for various switches. Switch config files are rebuilt out of a cron jobs running on mail. This ensures if we change a port name (desc) that the mrtg we look at has the latest info. So if you want to change port naming, please do it in the switch itself. If you have problems getting new devices setup or change existing devices you may need to change permissions on the cfg file as well as the data file in <tt>/usr/local/www/mgmt/mrtg/data</tt>, including removal of the rrd file if necessary.<br />
<br />
== Errors ==<br />
<br />
=== "Lock wait timeout exceeded" ===<br />
<br />
<pre><br />
delete error - Can't delete a2206e24: DBD::mysql::st execute failed: Lock wait timeout exceeded; try restarting transaction [for Statement "DELETE FROM invoice WHERE inv_ref=? "] at /usr/local/lib/perl5/site_perl/5.6.1/DBIx/ContextualFetch.pm line 51. at /usr/local/www/mgmt/Plugin/Billing.pm line 1934<br />
</pre><br />
<br />
This is the result of an unclean submit/commit. Usually from an error or a double click on something that should have been single click. To clear this up, restart the database:<br />
<br />
<pre><br />
mail /usr/local/www/scripts# /usr/local/etc/rc.d/mysql-server.sh stop<br />
mysqldmail /usr/local/www/scripts#<br />
</pre><br />
<br />
It takes a minute to shutdown. I keep running the command until it says it isn't running, then I start it:<br />
<br />
<pre>mail /usr/local/www/scripts# /usr/local/etc/rc.d/mysql-server.sh stop<br />
mysqldmail /usr/local/www/scripts# /usr/local/etc/rc.d/mysql-server.sh stop<br />
mysqldmail /usr/local/www/scripts# /usr/local/etc/rc.d/mysql-server.sh stop<br />
mysql-server isn't running<br />
mail /usr/local/www/scripts# /usr/local/etc/rc.d/mysql-server.sh start<br />
mysqldmail /usr/local/www/scripts#</pre><br />
<br />
= Account Manager (AM) =<br />
== Setup / organization / notes ==<br />
<br />
* domain: <tt>https://secure.johncompanies.com/am</tt><br />
* webroot: <tt>/usr/local/www/am</tt><br />
<br />
Directory structure:<br />
* am: contains the main handler <tt>AM.pm</tt>, the authentication (<tt>AMAuth.pm</tt>), and password reset (<tt>AMPassR.pm</tt>) handlers<br />
* am/Lib: contains the log module <tt>AMLog.pm</tt><br />
* am/Log: contains the log <tt>am.log</tt>, and ats activity log <tt>ats.log</tt><br />
* am/Plugin: contains the plugin module <tt>AMP.pm</tt> and the form fill in module <tt>FillInForm.pm</tt><br />
* am/html: contains all web pages<br />
* am/static: contains static, non-interpreted content like graphics and javascripts<br />
<br />
<br />
The Account Manager was designed and intended to allow customers to do the following:<br />
* see their services and various details about it<br />
** IP<br />
** type service<br />
** cost of service<br />
** link to control panel (virtuozzo CT only)<br />
* to manage their service level<br />
* to power cycle their server (if at i2b)<br />
* to see their bandwidth usage and allocation<br />
* to manage their contact info<br />
* to review their billing history<br />
* to update their billing method info<br />
<br />
It is accessed via our secure website. Customers must login using their CID (colNNNNN) and a password which is created specifically for use with the AM. It is not tied in any way to their server, though some customers may be confused by that or think that changing the password in one location will affect the other.<br />
<br />
JC staff may login to any customer's AM by using a special (all access) password.<br />
<br />
In order to access the AM for any given customer, a few things must be in place:<br />
# they must have a password issued for their AM access. In order to determine if a password has been set you can look at the "AM pass" (jc:customers.am_auth) field in the customer's page in mgmt. If it says "(hidden)" a password has been established. If instead you see 3 links:<br>reset activate<br>activate+email<br>that means the customer has no password and access has not been enabled.<br />
# the "owner email" (jc:customers.owner_email) field must contain the owner's email address. If a customer has been issued the pass to their AM but an owner email has not been selected, they will be prompted to select one from the existing contacts upon logging in for the first time. The selected owner will be emailed. It's for this reason that if you access the AM for a customer with the global pass, you should make sure a owner has been selected before hand (enter the info in mgmt) and don't select it in the AM, lest you want them to get a random email.<br />
<br />
If no password has been established (or it has been forgotten- it's stored as a 1 way hash) it must be reset. The customer may do this themselves via the password reset page (linked off the AM login page). This will require them to have on file and use/match the owner_email field. You can also reset the password via mgmt via 2 methods:<br />
# reset: will just reset their password and show it to you in mgmt. It will also give you handy instructions (commands) on how to place their CID and AM password in a file on their VPS. We prefer to do it this way since we then don't have to decide if the person requesting access is authorized or not. If we place the CID/password in a file on their VPS and make it viewable by root only, then only someone with root access could get to that and if you have root access, you're considered authorized.<br />
# activate+email: this will set/reset their password and send an email to the customer with access instructions and it will instruct them to find the AM access info in the <tt>/root/ampass</tt> on their VPS- which you will create for them as soon as you click this link. If the customer has only colo's, the site will recognize that and email them the instructions/password.<br />
# activate: this will enable AM access. The customer may then use the password reset function to get a password set for access.<br />
<br />
There is lots of contextual/hover help throughout the AM to help customers understand what they're looking at.<br />
<br />
Note - if a(n old) customer with no AM access gets a new VPS, the welcome email will indicate they have AM access established- this isn't true. You will need to at least activate the AM for them and possibly also reset their pass and update the ampass file accordingly.<br />
<br />
== Usage ==<br />
<br />
=== Dashboard ===<br />
This is the main/landing page for the AM. It has several sections:<br />
<br />
* Announcements: This is where we make available any announcement we might have for the customer. If the customer had a billing issue, it would be listed here.<br />
<br />
* Services: lists the services (servers) which the customer has and basic info about it (service type, IP, hostname, OS)<br />
<br />
* Bandwidth: shows condensed usage/allocation, overage and how we deal with overage (per the customer's preference).<br />
<br />
* Contacts: shows the contacts on file for the account and their role.<br />
<br />
=== Services ===<br />
<br />
This page shows in detail what services they have with us. For instance, their package and what it comes with in terms of IP, disk space, bandwidth, price. There is a link for them to change service on this page- this just trigger's an email to support so they can't actually change their service instantly.<br />
<br />
If they have a colo @ i2b, the power controls are available on this page.<br />
The status of the power outlet to which their server is connected is displayed ("Current Status") and if the ATS is responding (see below) 2 radio options are given for controlling the port: "Off" and "On". Pretty self explanatory, "Off" turns the port off and "On" turns it on. In order to power cycle the server, the will need to first select the "Off" option and click the "Submit" button. A window will popup and display the status of the port until the status switches to off (it will auto refresh every 10sec). To power back on, they just need to set the option to "On" and click "Submit".<br />
<br />
If the customer has a virtuozzo VPS, the link to their VZ control panel is provided.<br />
<br />
=== Bandwidth ===<br />
<br />
This page allows the customer to pull up bandwidth usage statistics. The reporting of those statistics is available in several flavors and can be customized to show data based on:<br />
* date range<br />
* server<br />
* IP(s)<br />
* data flowing in a particular direction<br />
* protocol (ICMP, TCP, UDP) <br />
* port(s)<br />
<br />
Further, data can be output as:<br />
* daily bar graph<br />
* 15min granularity bar graph<br />
* pie chart<br />
* table/raw<br />
<br />
The only thing to keep in mind here is that anything but the daily bar graph query will take up a bit more time as all other queries come from much larger tables and thus require more time to seek through the database.<br />
<br />
The graphs generated by this activity actually generate files which live on the server for a period of time, and are removed by cronjob.<br />
<br />
=== Billing ===<br />
<br />
This screen displays the payment method currently active for the customer, and allows them to update that info (in the case of credit card payments). <br />
<br />
It also lists their payment history, oldest to newest, and their outstanding balance.<br />
<br />
It displays their ongoing JC charges.<br />
<br />
If they have a paypal subscription setup it displays that.<br />
<br />
Finally, if they have an outstanding (paypal) invoice, it will display here and give them a link to (pay) it.<br />
<br />
=== Account Info ===<br />
<br />
Allows the customer to see / update:<br />
* account owner info<br />
* contact info<br />
* the account manager password<br />
* bandwidth preferences<br />
<br />
Any modifications to these settings will require the user to request a security token to be emailed to the owner (jc:customers.owner_email) which they are required to provide in order to make updates to this page. Once the customer provides this key and goes on to edit the info it is no longer valid and any subsequent visit to the edit page will require the generation of a new security token. Obviously, this requires the customer to have the ability to retrieve email at the owner's email address on file.<br />
<br />
=== Support ===<br />
<br />
This page allows the customer to quickly send us a message. They are allowed to send it as/from one of the contacts on file (or provide a new one) and they can indicate which server the issue is regarding. The email is cc'd back to the customer for their records. This form is intelligent and will send the email to the right place based on the server/product they select- support@ or linux@<br />
<br />
== Problems ==<br />
=== Power management: Status and control temporarily unavailable ===<br />
<br />
ATS isn't responding. See [[ATS#Rebooting_and_Recovering|Rebooting and Recovering ]]<br />
<br />
=== Power control doesn't work ===<br />
<br />
If after requesting a change in power status, the popup status window never reflects a change to the requested status, that means the ATS isn't responding and the customer should try setting the power again. If that still doesn't work, the ATS is out and needs to be reset. See [[ATS#Rebooting and Recovering|Rebooting and Recovering]]<br />
<br />
= Database =<br />
<br />
The database behind AM/Signup/MGMT runs on mysql on the [[Infrastructure_Machines#mail|mail]] server.<br />
<br />
== jc:billing_history ==<br />
<br />
== jc:ref_templates ==</div>Supporthttps://wiki.jcihosting.com/index.php?title=Virtuozzo_Reference&diff=1042Virtuozzo Reference2013-03-11T23:55:00Z<p>Support: Support moved page Virtuozzo Reference to Virtuozzo / Linux Reference</p>
<hr />
<div>#REDIRECT [[Virtuozzo / Linux Reference]]</div>Supporthttps://wiki.jcihosting.com/index.php?title=Virtuozzo_/_Linux_Reference&diff=1041Virtuozzo / Linux Reference2013-03-11T23:55:00Z<p>Support: Support moved page Virtuozzo Reference to Virtuozzo / Linux Reference</p>
<hr />
<div>= Updating VZ =<br />
<br />
Update Virtuozzo to version 4.0.0<br />
Apply minor updates for Virtuozzo 3.0.0<br />
<br />
...<br />
When you get to the last screen, you will be given an option to <br />
<br />
( ) Automatically reboot after update<br />
(*) Reboot later manually<br />
<br />
make sure you choose to reboot later.<br />
<br />
= Migrating Templates =<br />
One nice feature VZ offers is the ability to run a virtual environment (VE or CT as it's now called) based on a particular ditribution on a host which may or may not share the same distribution. For instance, you could run a CT running Ubuntu while the host machine (Hardware Node or HN) is running CentOS. This is accomplished via the use of OS templates. As long as the HN contains the OS template on which a particular CT relies, it will run there. As such, if you want to move a CT from one HN to another, you will need to make sure the OS template exists there.<br />
<br />
Modern versions of VZ (>= 4.x) will check for and automatically move the OS template over to the host as you're (vz)migrating the CT over to a new HN. However we like to make sure the template is in place prior to moving the CT. <br />
<br />
The first step is to see if the template exists on the host. To see OS templates installed:<br />
<br />
2.x (shows OS and application templates):<br />
<pre># vzpkgls<br />
mod_ssl-rh9 20040624<br />
mysql-rh9 20030814 20050412<br />
openwebmail-rh9 20050427<br />
php-rh9 20050506<br />
phpBB-rh9 20041229<br />
postgresql-rh9 20050719<br />
sl-webalizer-rh9 20040119<br />
spamassassin-rh9 20040127<br />
tomcat-rh9 20040524<br />
usermin-rh9 20040116</pre><br />
<br />
>= 3.x (shows just OS templates, run without -O to see application as well):<br />
<pre># vzpkg list -O<br />
ubuntu-10.04-x86 2010-06-01 11:39:05<br />
ubuntu-11.04-x86 2011-06-22 14:23:59<br />
ubuntu-9.10-x86 2010-03-11 17:39:51<br />
centos-5-x86 2010-03-11 17:12:27<br />
debian-5.0-x86 2010-03-11 17:33:34</pre><br />
<br />
<br />
All template files are located:<br />
<br />
<pre># ls /vz/template/<br />
ColdFusion-fc1 jre-fc1 openwebmail-fc1 sl-webalizer-fc1<br />
ColdFusion-fc2 jre-fc2 openwebmail-fc2 sl-webalizer-fc2<br />
ColdFusion-rh9 jre-rh9 openwebmail-rh9 sl-webalizer-rh9<br />
PostNuke-fc1 jre-suse92 openwebmail-suse92 spamassassin-fc1<br />
PostNuke-fc2 jrun php-deb31 spamassassin-fc2<br />
PostNuke-rh9 mailman-deb31 php-fc1 spamassassin-rh9<br />
analog-fc1 mailman-fc1 php-fc2 spamassassin-suse92<br />
analog-fc2 mailman-fc2 php-rh9 suse-9.2<br />
analog-rh9 mailman-rh9 php-suse92 tomcat-fc1<br />
awstats-fc1 mailman-suse92 phpBB-fc1 tomcat-fc2<br />
awstats-rh9 mod_frontpage-fc1 phpBB-fc2 tomcat-rh9<br />
bbClone-fc1 mod_frontpage-fc2 phpBB-rh9 tomcat-suse92<br />
bbClone-fc2 mod_frontpage-rh9 phpmyadmin-deb31 urchin-fc1<br />
bbClone-rh9 mod_frontpage-suse92 postgresql-deb31 usermin-fc1<br />
conf mod_perl-deb31 postgresql-fc1 usermin-fc2<br />
debian-3.1 mod_perl-fc1 postgresql-fc2 usermin-rh9<br />
devel-deb31 mod_perl-fc2 postgresql-rh9 uw-imap-fc1<br />
devel-fc2 mod_perl-rh9 postgresql-suse92 uw-imap-fc2<br />
devel-suse92 mod_perl-suse92 proftpd-deb31 uw-imap-rh9<br />
fedora-core-1 mod_ssl-fc1 proftpd-fc1 vzredhat-7.3<br />
fedora-core-2 mod_ssl-fc2 proftpd-fc2 vzredhat-8.0<br />
jdk-deb31 mod_ssl-rh9 proftpd-rh9 webalizer-suse92<br />
jdk-fc1 mysql-deb31 proftpd-suse92 webmin-fc1<br />
jdk-fc2 mysql-fc1 redhat-7.2 webmin-fc2<br />
jdk-rh9 mysql-fc2 redhat-9 webmin-rh9<br />
jdk-suse92 mysql-rh9 redhat-as3-minimal<br />
jre-deb31 mysql-suse92 redhat-devel-9<br />
</pre><br />
<br />
What we see here are the base OS templates: <tt>redhat-9, fedora-core-1</tt> and supporting application templates: <tt>postgresql-rh9, mailman-rh9, jdk-fc1, mod_ssl-fc1</tt>. So if you wanted to copy over all of redhat 9 you'd need to copy the contents of:<br />
<pre># ls -d /vz/template/*rh9*<br />
/vz/template/ColdFusion-rh9 /vz/template/mod_frontpage-rh9 /vz/template/proftpd-rh9<br />
/vz/template/PostNuke-rh9 /vz/template/mod_perl-rh9 /vz/template/sl-webalizer-rh9<br />
/vz/template/analog-rh9 /vz/template/mod_ssl-rh9 /vz/template/spamassassin-rh9<br />
/vz/template/awstats-rh9 /vz/template/mysql-rh9 /vz/template/tomcat-rh9<br />
/vz/template/bbClone-rh9 /vz/template/openwebmail-rh9 /vz/template/usermin-rh9<br />
/vz/template/jdk-rh9 /vz/template/php-rh9 /vz/template/uw-imap-rh9<br />
/vz/template/jre-rh9 /vz/template/phpBB-rh9 /vz/template/webmin-rh9<br />
/vz/template/mailman-rh9 /vz/template/postgresql-rh9<br />
<br />
# ls -d /vz/template/*redhat-9*<br />
/vz/template/redhat-9<br />
</pre><br />
<br />
To get a template from one HN to another, it simply needs to be rsync'd over to the target HN. It's rsync'd over in this fashion:<br />
<br />
rsync -av -e ssh /vz/template/redhat-9 root@10.1.4.65:/vz/template<br />
rsync -av -e ssh /vz/template/*rh9 root@10.1.4.65:/vz/template<br />
<br />
Newer (>= 3.x) VZ employs "EZ" templates which are organized a little differently:<br />
<br />
<pre># ls /vz/template/<br />
cache CmdTool.log debian redhat-as3-minimal-20080630.tar.gz vc<br />
centos conf redhat-as3-minimal ubuntu<br />
<br />
# ls /vz/template/ubuntu/<br />
10.04 11.04 9.10<br />
# ls /vz/template/ubuntu/10.04/<br />
x86<br />
# ls /vz/template/ubuntu/10.04/x86/<br />
aap_1.091-1_all<br />
aap-doc_1.091-1_all<br />
adduser_3.112ubuntu1_all<br />
anacron_2.3-13.1ubuntu11_i386<br />
apache2_2.2.14-5ubuntu8.2_i386<br />
apache2_2.2.14-5ubuntu8.4_i386<br />
apache2_2.2.14-5ubuntu8.6_i386<br />
apache2_2.2.14-5ubuntu8.7_i386<br />
apache2_2.2.14-5ubuntu8.8_i386<br />
apache2_2.2.14-5ubuntu8.9_i386<br />
<...snip><br />
<br />
# ls /vz/template/cache/<br />
centos-5-x86.tar.gz suse-11.3-x86.tar.gz ubuntu-9.10-x86.tar.gz<br />
debian-5.0-x86.tar.gz ubuntu-10.04-x86.tar.gz<br />
fedora-core-14-x86.tar.gz ubuntu-11.04-x86.tar.gz<br />
</pre><br />
<br />
So what we see is the entire OS and all applications are in 1 directory, and organized by distribution and release version. So to move Ubuntu 10.04:<br />
<br />
rsync -av -e ssh /vz/template/ubuntu/10.04 root@10.1.4.69:/vz/template/ubuntu<br />
<br />
and you also need to move the cache:<br />
<br />
rsync -av -e ssh /vz/template/cache/ubuntu-10.04-x86.tar.gz root@10.1.4.69:/vz/template/cache/<br />
<br />
Once the transfer is complete, you will be able to run <tt>vzpkgls</tt> or <tt>vzpkg list -O</tt> and see the template(s) listed on the target HN.<br />
<br />
So far, this all supposes template doesn't exist on the target- very simple, just copy it over. If the template exists already on the target HN, this requires closer inspection. With older templates, simply moving over the templates would be the end of it- you see the version listed when you do a <tt>vzplgls</tt>:<br />
<br />
mysql-rh9 20030814 20050412<br />
<br />
this means there are 2 versions of the mysql package for RH9: 20030814 and 20050412<br />
As an aside, you can't copy over just 1 of them, but if you rsync'd over the <tt>mysql-rh9</tt> directory, you'd see 20030814 and 20050412 on the target HN. As long as the versions match up, you're good to go.<br />
<br />
With EZ templates, the versions are not as easy to decipher. When you look at the <tt>vzpkg list</tt> output, that simply tells you when a cache was created. A cache is simply a snapshot of all the data needed for the latest versions of the OS and it's applications at the time the cache was created. It is a date, and thus tells you nothing about the versions within. So for instance, if you had a cache for Ubuntu 10.04 from 2007 and you ran <tt>vzpkg create cache ubuntu-10.04-x86</tt> it would re-download the latest version of apache, mysql and so on. And any ''subsequent'' ubuntu 10.04 CT's created thereafter would use those newer versions, whereas older CT's based on 10.04 would use older templates. You can see how this works by looking at the files under:<br />
<br />
<pre># ls /vz/template/ubuntu/10.04/x86/<br />
aap_1.091-1_all<br />
aap-doc_1.091-1_all<br />
adduser_3.112ubuntu1_all<br />
anacron_2.3-13.1ubuntu11_i386<br />
apache2_2.2.14-5ubuntu8.2_i386<br />
apache2_2.2.14-5ubuntu8.4_i386<br />
apache2_2.2.14-5ubuntu8.6_i386<br />
apache2_2.2.14-5ubuntu8.7_i386<br />
apache2_2.2.14-5ubuntu8.8_i386<br />
apache2_2.2.14-5ubuntu8.9_i386<br />
<...snip><br />
</pre><br />
<br />
You can see that this template has in it 6 different versions of apache2. This means that this template was merged with other 10.04 templates and/or it has been cached a few times, each time a new apache was available. The unfortunate thing is it's almost impossible to know which CT's are using which version of apache so it's hard to try to prune this down and remove unwanted/unneeded applications.<br />
<br />
So, if you see another HN has Ubuntu 10.04 on it, you need to see/confirm that it has all the exact files your CT needs. You can run a test rsync to see what ''would'' be transferred (what's missing):<br />
<br />
rsync -avn --stats -e ssh /vz/template/ubuntu/10.04 root@10.1.4.69:/vz/template/ubuntu<br />
<br />
Based on the amount of data you get back from the rsync, you will be able to decide whether or not to remove the -n and allow the full transfer to go through. The downside to doing the transfer is the situation described above- you've permanently joined these templates and now you may have 10 versions of apache on the target HN. There's one other potential pitfall to this kind of merging- it will also potentially copy over shared files, for which there is no particular version, most of which are in <tt>/vz/template/ubuntu/10.04/x86/config</tt>: <br />
<br />
cat /vz/template/ubuntu/10.04/x86/config/os/default/repositories<br />
/vz/template/ubuntu/10.04/x86/config/os/default/repositories<br />
<br />
Here are some specific things to look out for. Case: copying centos-5 on virt19 to virt12. We dumped the rsync output to a file so we could inspect:<br />
<br />
[root@virt19 /vz/template]# rsync -an -e ssh /vz/template/centos/ root@10.1.4.62:/vz/template/centos/ > v12_c5<br />
<br />
(note, that can be dangerous, better to add on full path <tt>/vz/template/centos/5</tt>)<br />
<br />
So in the output file we see things like:<br />
<br />
<pre><br />
x86/base0/cachecookie<br />
x86/base0/primary.xml.gz <br />
x86/base0/primary.xml.gz.sqlite <br />
x86/base0/repomd.xml <br />
x86/base1/cachecookie <br />
x86/base1/filelists.xml.gz <br />
x86/base1/filelists.xml.gz.sqlite <br />
x86/base1/primary.xml.gz <br />
x86/base1/primary.xml.gz.sqlite <br />
x86/base1/repomd.xml<br />
x86/base2/cachecookie<br />
</pre><br />
<br />
Which make us nervous cause those are not versioned files, i.e.:<br />
5/x86/MAKEDEV-3.23-1.2.i386/usr/sbin/mksock<br />
<br />
So those files will replace existing files on v12 rather than the case of the latter where it's likely being added. So caution should be given and deference to whichever version is newer (which rsync should take care of, but that will depend on the date each file was created and perhaps not the actual data within).<br />
<br />
If we look further in the file we see:<br />
<br />
x86/psa-appvault-moodle-1.8-8202920080409011254.noarch/usr/local/psa/var/cgitory/moodle-1.<br />
9/htdocs/lib/editor/htmlarea/plugins/TableOperations/img/cell-split.gif<br />
<br />
and other files looking like:<br />
5/x86/plesk-base-9.0.1-cos5.build90090127.18.i586/etc/sw/keys/info<br />
<br />
Which means plesk is here. We don't need plesk on v12 (the CT moving over doesn't use it) so we rerun the rsync and exclude it:<br />
<br />
[root@virt19 /vz/template]# rsync -an -e ssh --exclude="plesk*" --exclude="psa*" /vz/template/centos/ root@10.1.4.62:/vz/template/centos/ > v12_c5<br />
<br />
and now it's much smaller:<br />
<br />
[root@virt19 /vz/template]# cat v12_c5|wc -l<br />
97104<br />
<br />
Before running with excludes it was:<br />
[root@virt19 /vz/template]# cat v12_c5|wc -l <br />
238238<br />
<br />
So again, look at what you're doing. Generally, combining templates is fine however.<br />
<br />
If you're happy with the merge, run the rsync again without the "-n" to let it actually do the transfer.<br />
<br />
Once done, don't forget to add the new template to the HN in Management -> Reference -> Templates<br />
<br />
= getting help =<br />
<br />
= vzup2date =<br />
TODO <br />
= Adding new OS/application templates =<br />
<br />
Virtuozzo will lag a bit behind the initial release of an OS, perhaps by a month or more. Further, a newly-released OS may be available, but there may not be many/any application templates available initially. It pays to wait a bit.<br />
<br />
A template is easily installed via [[#vzup2date|vzup2date]]:<br />
<br />
vzup2date -z<br />
<br />
You will be given a list of OS's to install. Select an OS, then customize that selection by checking/unchecking the application templates we do/don't want to include/offer.<br />
<br />
Generally, here are the app templates we include/offer:<br />
<pre>cyrus-imap<br />
devel<br />
imp<br />
jre<br />
jsdk<br />
mailman<br />
mod_perl<br />
mod_ssl<br />
mysql<br />
php<br />
phpmyadmin<br />
phppgadmin<br />
postgresql<br />
proftpd<br />
spamassassin<br />
squirrelmail<br />
tomcat<br />
vsftpd<br />
</pre><br />
<br />
We do not (any longer) offer webalizer or analog.<br />
<br />
After selecting and installing the OS and apps, you should ensure it's installed:<br />
<br />
vzpkg list -O<br />
<br />
Your new OS (ex. fedora-core-13-x86_64) will be listed, without a corresponding cache date:<br />
<pre><br />
[root@virt13 /vzconf]# vzpkg list -O<br />
centos-6-x86 2011-07-22 13:24:41<br />
centos-6-x86_64 2012-03-09 20:42:22<br />
centos-5-x86 2009-07-27 16:58:24<br />
centos-5-x86_64 2012-03-09 22:38:53<br />
ubuntu-11.10-x86_64 2012-03-01 23:13:33<br />
ubuntu-9.04-x86 2009-06-02 11:32:39<br />
ubuntu-12.04-x86_64 2012-05-29 13:50:50<br />
ubuntu-8.04-x86 2008-09-17 21:27:20<br />
ubuntu-8.10-x86 2009-03-13 13:58:57<br />
ubuntu-11.04-x86 2011-12-05 11:15:46<br />
ubuntu-7.04-x86 2008-09-24 16:42:50<br />
redhat-el6-x86_64<br />
fedora-core-13-x86_64 <br />
fedora-core-12-x86 2010-02-06 13:24:32<br />
fedora-core-15-x86 2011-12-05 11:36:43<br />
fedora-core-11-x86 2009-10-01 16:30:59<br />
fedora-core-14-x86<br />
fedora-core-16-x86_64 2012-03-13 11:40:23<br />
fedora-core-9-x86 2008-11-29 10:52:18<br />
debian-6.0-x86 2011-07-29 16:00:35<br />
debian-6.0-x86_64 2012-03-12 19:53:33<br />
debian-5.0-x86 2009-03-12 12:39:11<br />
</pre><br />
<br />
You can also confirm the apps installed:<br />
<pre>[root@virt13 /vzconf]# vzpkg list fedora-core-13-x86_64<br />
fedora-core-13-x86_64 2013-03-08 11:39:44<br />
fedora-core-13-x86_64 devel<br />
fedora-core-13-x86_64 php<br />
fedora-core-13-x86_64 proftpd<br />
fedora-core-13-x86_64 postgresql<br />
fedora-core-13-x86_64 phpmyadmin<br />
fedora-core-13-x86_64 mysql<br />
</pre><br />
<br />
Before you can use the OS you must cache it. To create the cache run:<br />
vzpkg cache [OS]<br />
<br />
ex: <tt>vzpkg cache fedora-core-13-x86_64</tt><br />
<br />
Now it's ready to use. In order to be able to use it with the [[VPS_Management#vm|vm]] command, you must create a file:<br />
jctmpl-[OS]<br />
<br />
ex: <tt>jctmpl-fedora-core-13-x86_64</tt><br />
<br />
In it, on line 1 we place the OS, followed by the app names, ex:<br />
<pre>more jctmpl-fedora-core-13-x86_64<br />
fedora-core-13-x86_64<br />
postgresql<br />
jsdk<br />
mod_ssl<br />
phpmyadmin<br />
mod_perl<br />
tomcat<br />
devel<br />
php<br />
mailman<br />
cyrus-imap<br />
squirrelmail<br />
proftpd<br />
mysql<br />
phppgadmin<br />
spamassassin<br />
</pre><br />
<br />
The packages will be installed in that order, so any pre-req's should be handled there.<br />
<br />
Now, when you run [[VPS_Management#vm|vm]] it will show you the newly-added OS and you may create a CT using it.<br />
<br />
The last few tasks are related to mgmt:<br />
<br />
1. add to mgmt->reference->templates (use the existing templates as an example). Once added, the new template will be included with the list of OS's available for the given HN.<br />
2. if this is going to be a new OS offered for ordering, it needs to be added to the order form and webpage.<br />
<br />
a. to get a list of applications and versions in the new OS, you should create a test CT, enter it and run <tt>dpkg -l|sort</tt> (or <tt>rpm -aq|sort</tt> in centos/fedora) and enter that list in the following places:<br><br />
<tt>/usr/local/www/jc_pub/[osname].html</tt><br><br />
<tt>/usr/local/www/signup/html/[osname].html</tt><br><br />
Note, you will have to move the most recent OS data from <tt>/usr/local/www/signup/html/[osname].html</tt> into the corresponding <tt>/usr/local/www/signup/html/[osname]_old.html</tt> updating the version links in both.<br />
<br />
b. to add the new OS to the order form, you will need to update <tt>/usr/local/www/signup/html/step1.html</tt> in several places (for regular and OSS orders) as well as updating the templateID which you can get from mgmt->reference->templates (or [[Management_System_/_Public_Website_/_Signup_/_Account_Manager#jc:ref_templates|jc:ref_templates]])<br />
<br />
<br />
<br />
Old method to obtain the templates:<br />
<br />
Go to http://www.parallels.com/en/virtuozzo/templates/catalog/<br />
<br />
Download the RPM's and install them<br />
<br />
vzpkg list<br />
to see the new packages/os templates<br />
<br />
vzpkg create cache "OS_EZ_template_name"<br />
to create a cache<br />
<br />
<br />
= Update OS template =<br />
<br />
Installation and update of the new version of OS template will not affect the existing containers. It will affect only the new containers, created after this update operation. In terms of fixing "vzctl enter" error and SSH connectivity to Ubuntu based containers, it is necessary to recreate the template and not to update it, so the commands to run are:<br />
<br />
<pre>rpm -Uhv ubuntu-8.04-x86-ez-4.0.0-9.swsoft.noarch.rpm<br />
vzpkg clean ubuntu-8.04-x86<br />
vzpkg remove cache ubuntu-8.04-x86<br />
vzpkg create cache ubuntu-8.04-x86</pre><br />
<br />
<br />
= Installing new SSL cert on VZCP =<br />
<br />
<pre>openssl req -days 1999 -new -x509 -nodes -out server.crt -keyout server.key<br />
<br />
US<br />
CA<br />
San Diego<br />
johncompanies.com<br />
johncompanies.com<br />
virt15ohncompanies.com<br />
linux@johncompanies.com<br />
<br />
cp -p /etc/httpd/conf/ssl.crt/server.crt /etc/httpd/conf/ssl.crt/server.crt.bak<br />
cp -p /etc/httpd/conf/ssl.key/server.key /etc/httpd/conf/ssl.key/server.key.bak<br />
cp server.crt /etc/httpd/conf/ssl.crt/server.crt<br />
cp server.key /etc/httpd/conf/ssl.key/server.key<br />
/etc/init.d/vzcp restart</pre><br />
<br />
= Moving virt (drives) to new hardware =<br />
<br />
If you have a case where a virt is dead (due to hardware failure), you can move the drives to a new hardware setup.<br />
Here are the steps and things to look for:<br />
<br />
Move the drives to the new box. Most likely, the drives will be recognized and there will be no problems booting up. On Dell PERC 5/6 cards you will need to import the new "foreign" volume via the raid BIOS.<br />
<br />
If the hardware (motherboard) is different, it’s possible the nic’s won’t be recognized. Look at /etc/modules.conf the ethernet drivers are either:<br />
<br />
<pre>alias eth0 tg3<br />
alias eth1 tg3</pre><br />
(supermicro)<br />
<br />
Or<br />
<br />
<pre>alias eth0 e100<br />
alias eth1 e1000</pre><br />
(intel)<br />
<br />
Or<br />
<br />
<pre>alias eth0 bnx2<br />
alias eth1 bnx2</pre><br />
(dell 29500)<br />
<br />
Also, you may see eth0 and eth1 swapped so if after confirming the eth devices are present (may require a reboot), then you may still need to swap the green/yellow network cables in the back to get them on the right port.<br />
<br />
VZ will not run (for long) on the new hardware, so you will need to get the license reissued. You may create/download a temporary license via their website for the interim.<br />
<br />
= Updating kernel on virt =<br />
<br />
We now use [[#vzup2date|vzup2date]] to update systems and kernels. What follows is what we used to do when we had to download kernels and install manually on old systems:<br />
<br />
<br />
1. install kernel and modules<br />
<pre># rpm -ivh vzkernel-enterprise-2.4.20-021stab028.12.777.i686.rpm \<br />
vzmodules-enterprise-2.4.20-021stab028.12.swsoft.i686.rpm<br />
vzkernel-smp ##################################################<br />
vzmodules-smp ##################################################</pre><br />
<br />
DO NOT USE the "rpm -Uhv" command to install the kernel, otherwise all the previously installed kernels may be removed from your system. <br />
<br />
2. update lilo/grub<br />
<br />
Lilo:<br />
<br />
vi /etc/lilo.conf<br />
<br />
rename old Virtuozzo kernel label from "linux+virtuozzo" to "virtuozzo_old"<br />
and add a new entry pointing to the newly installed virtuozzo kernel image.<br />
For example:<br />
<br />
<pre>prompt<br />
timeout=50<br />
default=linux+virtuozzo<br />
append="debug panic=5 console=tty console=ttyS0,38400"<br />
boot=/dev/sda<br />
map=/boot/map<br />
install=/boot/boot.b<br />
message=/boot/message<br />
linear<br />
<br />
image=/boot/vmlinuz-2.4.18-3smp<br />
label=linux<br />
initrd=/boot/initrd-2.4.18-3smp.img<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.18-3<br />
label=linux-up<br />
initrd=/boot/initrd-2.4.18-3.img<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-020stab009.5.777-enterprise<br />
label=2.4.20-9.5.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-020stab009.8.777-enterprise<br />
label=2.4.20-9.8.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-020stab009.21.777-enterprise<br />
label=2.4.20-9.21.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-020stab009.24.777-enterprise<br />
label=2.4.20-9.24.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-021stab028.3.777-enterprise<br />
label=2.4.20-28.3.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-021stab028.5.777-enterprise<br />
label=old_linux+vz<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-021stab028.12.777-enterprise<br />
label=linux+virtuozzo<br />
read-only<br />
root=/dev/sda1</pre><br />
<br />
With this configuration, the new kernel is the default kernel to use at boot. The original kernel entry has been retained with the label "old_linux+vz ".<br />
<br />
Grub:<br />
<br />
vi /boot/grub /menu.lst<br />
<br />
Add new entry at the top.<br />
For example:<br />
<br />
<pre>serial --unit=0 --speed=38400<br />
terminal --timeout=10 serial console<br />
default=0<br />
timeout=10<br />
splashimage=(hd0,0)/boot/grub/splash.xpm.gz<br />
title Virtuozzo 2.6.1 (2.4.20-021stab028.12.777-enterprise)<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-22.4.20-021stab028.12.777-enterprise root=/dev/sda1 console=tty0 \ console=ttyS0,38400<br />
title Virtuozzo 2.6.1 (2.4.20-021stab028.3.777-enterprise)<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.20-021stab028.3.777-enterprise root=/dev/sda1 console=tty0 \ console=ttyS0,38400<br />
title Red Hat Linux (2.4.18-3smp)<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.18-3smp ro root=/dev/sda1<br />
initrd /boot/initrd-2.4.18-3smp.img<br />
title Red Hat Linux-up (2.4.18-3)<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.18-3 ro root=/dev/sda1<br />
initrd /boot/initrd-2.4.18-3.img<br />
title Linux with Virtuozzo 2.5<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.20-020stab009.3.777-smp ro root=/dev/sda1<br />
title vz-enterpriseo<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.20-020stab009.21.777-enterprise ro root=/dev/sda1<br />
title vz-enterprise<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.20-020stab009.24.777-enterprise ro root=/dev/sda1 \ console=tty0 console=ttyS0,38400</pre><br />
<br />
3. run the lilo (if using lilo)<br />
<br />
<pre># lilo<br />
Added linux<br />
Added linux-up<br />
Added virtuozzo_old<br />
Added linux+virtuozzo *</pre><br />
<br />
4. reboot the server<br />
shutdown -r 23:05<br />
<br />
when it comes time for the shutdown, run [[VPS_Management#stopvirt|stopvirt]] to get things shut down faster<br />
<br />
<br />
= Remaking service VE (<=3.x only) =<br />
<br />
Occasionally the control panel for VEs (aka VZPP) will stop working and you just need to reinstall the service VE (which runs the control panels for all VEs on the HN). Here's how that's done :<br />
<br />
<pre>vzctl stop 1<br />
ipdel 1 69.55.234.152<br />
vzmlocal 1:2<br />
vzctl set 2 --save --onboot no<br />
vzsveinstall -t redhat-as3-minimal -d /root/Rel261/HW/RPMS -s 69.55.234.152 --skip-client</pre><br />
<br />
on 3.0:<br />
<br />
<pre>vzsveinstall -t redhat-as3-minimal -d /vz/Rel300/HW/RPMS -s 69.55.229.151 --skip-client<br />
<br />
vzctl stop 1<br />
vzctl mount 1<br />
vzctl mount 2<br />
/bin/cp -aRf /vz/root/2/home/vzagent0 /vz/root/1/home<br />
/bin/cp -aRf /vz/root/2/var/lib/mysql/VZADB /vz/root/1/var/lib/mysql<br />
/bin/cp -af /vz/root/2/etc/shadow /vz/root/1/etc/<br />
/bin/cp -aRf /vz/root/2/var/vzcp/static/vz/skins/ /vz/root/1/var/vzcp/static/vz<br />
/bin/cp -af /vz/root/2/etc/vzcp/pp/menu.xml /vz/root/1/etc/vzcp/pp/<br />
/bin/cp -af /vz/root/2/etc/vzcp/pp/dashboard.xml /vz/root/1/etc/vzcp/pp/<br />
<br />
vzctl umount 2<br />
vzctl umount 1<br />
vzctl start 1</pre><br />
<br />
<br />
= GRUB: boot once to a new kernel =<br />
<br />
To use a different kernel for the next reboot only:<br />
<pre>grub --no-floppy<br />
savedefault --default=N --once<br />
(where N is the number of the kernel you want to boot once)<br />
quit</pre><br />
<br />
Reboot your machine. It will boot using kernel "N".<br />
<br />
grub-reboot N<br />
will do essentially the same thing and prompt to reboot the machine<br />
<br />
<br />
= Recovering from GRUB failure =<br />
<br />
(example from DPE 2950)<br />
<br />
Boot to CEOS4 server CD (original install cd). Did this example with iso mounted via drac<br />
<br />
Boot: linux rescue</div>Supporthttps://wiki.jcihosting.com/index.php?title=Main_Page&diff=1040Main Page2013-03-11T23:53:56Z<p>Support: </p>
<hr />
<div>* [[Special:AllPages|All Pages]]<br />
* [[Payment_System|Payment System]]<br />
* [[Management_System_/_Public_Website_/_Signup_/_Account_Manager|Management System / Public Website / Signup / Account Manager]]<br />
* [[New_Signups|New Signups]]<br />
* [[Shutting_Down_Service|Shutting Down Service]]<br />
* [[Routine_Maintenance|Routine Maintenance]]<br />
* Reference<br />
** [[VPS_Management|VPS Management]]<br />
** [[Jail_Server_Install|Jail Server Install]]<br />
** [[Infrastructure_Machines|Infrastructure Machines]]<br />
** [[Colo_Server_Setup|Colo Server Setup]]<br />
** [[DNS]]<br />
** [[FreeBSD_Reference|FreeBSD Reference]]<br />
** [[Virtuozzo_/_Linux_Reference|Virtuozzo / Linux Reference]]<br />
** [[Cabinetmap]]<br />
** [[Switch_Control|Switch Control]]<br />
** [[Private_IP_Mapping|Private IP Mapping]]<br />
** [[Data_Center_Notes|Data Center Notes]]<br />
** [[Crash_Reference|Crash Reference]]<br />
** [[Hardware_Reference|Hardware Reference]]<br />
** [[ATS]]<br />
** [[Wiring_Reference|Wiring Reference]]<br />
** [[Screen]]<br />
** [[RAID_Cards|RAID Cards]]<br />
** [[BigBrother]]<br />
** [[Bandwidth_Management|Bandwidth Management]]<br />
** [[Network_Time_(ntp)|Network Time (ntp)]]<br />
** [[Isys_-_out_of_inodes|Isys - out of inodes]]<br />
** [[DRAC/RMM|DRAC/RMM]]<br />
** [[Customer_Backups|Customer Backups]]<br />
** [[DoS_Attacks|DoS Attacks]]<br />
** [[IPKVM|IPKVM]]<br />
* [[System-generated_Notifications|System-generated Notifications]]<br />
* [[Customer_Interaction|Customer Interaction]]<br />
* [[Backup_Accounts_(rsync.net)| Backup Accounts (rsync.net)]]<br />
* [[Todo]]<br />
<br />
<br />
== Getting started ==<br />
* [//www.mediawiki.org/wiki/Manual:Configuration_settings Configuration settings list]<br />
* [//www.mediawiki.org/wiki/Manual:FAQ MediaWiki FAQ]<br />
* [https://lists.wikimedia.org/mailman/listinfo/mediawiki-announce MediaWiki release mailing list]<br />
* [https://meta.wikimedia.org/wiki/Help:Advanced_editing Advanced Editing]</div>Supporthttps://wiki.jcihosting.com/index.php?title=Virtuozzo_/_Linux_Reference&diff=1039Virtuozzo / Linux Reference2013-03-11T23:52:10Z<p>Support: /* Remaking service VE (<=3.x only) */</p>
<hr />
<div>= Updating VZ =<br />
<br />
Update Virtuozzo to version 4.0.0<br />
Apply minor updates for Virtuozzo 3.0.0<br />
<br />
...<br />
When you get to the last screen, you will be given an option to <br />
<br />
( ) Automatically reboot after update<br />
(*) Reboot later manually<br />
<br />
make sure you choose to reboot later.<br />
<br />
= Migrating Templates =<br />
One nice feature VZ offers is the ability to run a virtual environment (VE or CT as it's now called) based on a particular ditribution on a host which may or may not share the same distribution. For instance, you could run a CT running Ubuntu while the host machine (Hardware Node or HN) is running CentOS. This is accomplished via the use of OS templates. As long as the HN contains the OS template on which a particular CT relies, it will run there. As such, if you want to move a CT from one HN to another, you will need to make sure the OS template exists there.<br />
<br />
Modern versions of VZ (>= 4.x) will check for and automatically move the OS template over to the host as you're (vz)migrating the CT over to a new HN. However we like to make sure the template is in place prior to moving the CT. <br />
<br />
The first step is to see if the template exists on the host. To see OS templates installed:<br />
<br />
2.x (shows OS and application templates):<br />
<pre># vzpkgls<br />
mod_ssl-rh9 20040624<br />
mysql-rh9 20030814 20050412<br />
openwebmail-rh9 20050427<br />
php-rh9 20050506<br />
phpBB-rh9 20041229<br />
postgresql-rh9 20050719<br />
sl-webalizer-rh9 20040119<br />
spamassassin-rh9 20040127<br />
tomcat-rh9 20040524<br />
usermin-rh9 20040116</pre><br />
<br />
>= 3.x (shows just OS templates, run without -O to see application as well):<br />
<pre># vzpkg list -O<br />
ubuntu-10.04-x86 2010-06-01 11:39:05<br />
ubuntu-11.04-x86 2011-06-22 14:23:59<br />
ubuntu-9.10-x86 2010-03-11 17:39:51<br />
centos-5-x86 2010-03-11 17:12:27<br />
debian-5.0-x86 2010-03-11 17:33:34</pre><br />
<br />
<br />
All template files are located:<br />
<br />
<pre># ls /vz/template/<br />
ColdFusion-fc1 jre-fc1 openwebmail-fc1 sl-webalizer-fc1<br />
ColdFusion-fc2 jre-fc2 openwebmail-fc2 sl-webalizer-fc2<br />
ColdFusion-rh9 jre-rh9 openwebmail-rh9 sl-webalizer-rh9<br />
PostNuke-fc1 jre-suse92 openwebmail-suse92 spamassassin-fc1<br />
PostNuke-fc2 jrun php-deb31 spamassassin-fc2<br />
PostNuke-rh9 mailman-deb31 php-fc1 spamassassin-rh9<br />
analog-fc1 mailman-fc1 php-fc2 spamassassin-suse92<br />
analog-fc2 mailman-fc2 php-rh9 suse-9.2<br />
analog-rh9 mailman-rh9 php-suse92 tomcat-fc1<br />
awstats-fc1 mailman-suse92 phpBB-fc1 tomcat-fc2<br />
awstats-rh9 mod_frontpage-fc1 phpBB-fc2 tomcat-rh9<br />
bbClone-fc1 mod_frontpage-fc2 phpBB-rh9 tomcat-suse92<br />
bbClone-fc2 mod_frontpage-rh9 phpmyadmin-deb31 urchin-fc1<br />
bbClone-rh9 mod_frontpage-suse92 postgresql-deb31 usermin-fc1<br />
conf mod_perl-deb31 postgresql-fc1 usermin-fc2<br />
debian-3.1 mod_perl-fc1 postgresql-fc2 usermin-rh9<br />
devel-deb31 mod_perl-fc2 postgresql-rh9 uw-imap-fc1<br />
devel-fc2 mod_perl-rh9 postgresql-suse92 uw-imap-fc2<br />
devel-suse92 mod_perl-suse92 proftpd-deb31 uw-imap-rh9<br />
fedora-core-1 mod_ssl-fc1 proftpd-fc1 vzredhat-7.3<br />
fedora-core-2 mod_ssl-fc2 proftpd-fc2 vzredhat-8.0<br />
jdk-deb31 mod_ssl-rh9 proftpd-rh9 webalizer-suse92<br />
jdk-fc1 mysql-deb31 proftpd-suse92 webmin-fc1<br />
jdk-fc2 mysql-fc1 redhat-7.2 webmin-fc2<br />
jdk-rh9 mysql-fc2 redhat-9 webmin-rh9<br />
jdk-suse92 mysql-rh9 redhat-as3-minimal<br />
jre-deb31 mysql-suse92 redhat-devel-9<br />
</pre><br />
<br />
What we see here are the base OS templates: <tt>redhat-9, fedora-core-1</tt> and supporting application templates: <tt>postgresql-rh9, mailman-rh9, jdk-fc1, mod_ssl-fc1</tt>. So if you wanted to copy over all of redhat 9 you'd need to copy the contents of:<br />
<pre># ls -d /vz/template/*rh9*<br />
/vz/template/ColdFusion-rh9 /vz/template/mod_frontpage-rh9 /vz/template/proftpd-rh9<br />
/vz/template/PostNuke-rh9 /vz/template/mod_perl-rh9 /vz/template/sl-webalizer-rh9<br />
/vz/template/analog-rh9 /vz/template/mod_ssl-rh9 /vz/template/spamassassin-rh9<br />
/vz/template/awstats-rh9 /vz/template/mysql-rh9 /vz/template/tomcat-rh9<br />
/vz/template/bbClone-rh9 /vz/template/openwebmail-rh9 /vz/template/usermin-rh9<br />
/vz/template/jdk-rh9 /vz/template/php-rh9 /vz/template/uw-imap-rh9<br />
/vz/template/jre-rh9 /vz/template/phpBB-rh9 /vz/template/webmin-rh9<br />
/vz/template/mailman-rh9 /vz/template/postgresql-rh9<br />
<br />
# ls -d /vz/template/*redhat-9*<br />
/vz/template/redhat-9<br />
</pre><br />
<br />
To get a template from one HN to another, it simply needs to be rsync'd over to the target HN. It's rsync'd over in this fashion:<br />
<br />
rsync -av -e ssh /vz/template/redhat-9 root@10.1.4.65:/vz/template<br />
rsync -av -e ssh /vz/template/*rh9 root@10.1.4.65:/vz/template<br />
<br />
Newer (>= 3.x) VZ employs "EZ" templates which are organized a little differently:<br />
<br />
<pre># ls /vz/template/<br />
cache CmdTool.log debian redhat-as3-minimal-20080630.tar.gz vc<br />
centos conf redhat-as3-minimal ubuntu<br />
<br />
# ls /vz/template/ubuntu/<br />
10.04 11.04 9.10<br />
# ls /vz/template/ubuntu/10.04/<br />
x86<br />
# ls /vz/template/ubuntu/10.04/x86/<br />
aap_1.091-1_all<br />
aap-doc_1.091-1_all<br />
adduser_3.112ubuntu1_all<br />
anacron_2.3-13.1ubuntu11_i386<br />
apache2_2.2.14-5ubuntu8.2_i386<br />
apache2_2.2.14-5ubuntu8.4_i386<br />
apache2_2.2.14-5ubuntu8.6_i386<br />
apache2_2.2.14-5ubuntu8.7_i386<br />
apache2_2.2.14-5ubuntu8.8_i386<br />
apache2_2.2.14-5ubuntu8.9_i386<br />
<...snip><br />
<br />
# ls /vz/template/cache/<br />
centos-5-x86.tar.gz suse-11.3-x86.tar.gz ubuntu-9.10-x86.tar.gz<br />
debian-5.0-x86.tar.gz ubuntu-10.04-x86.tar.gz<br />
fedora-core-14-x86.tar.gz ubuntu-11.04-x86.tar.gz<br />
</pre><br />
<br />
So what we see is the entire OS and all applications are in 1 directory, and organized by distribution and release version. So to move Ubuntu 10.04:<br />
<br />
rsync -av -e ssh /vz/template/ubuntu/10.04 root@10.1.4.69:/vz/template/ubuntu<br />
<br />
and you also need to move the cache:<br />
<br />
rsync -av -e ssh /vz/template/cache/ubuntu-10.04-x86.tar.gz root@10.1.4.69:/vz/template/cache/<br />
<br />
Once the transfer is complete, you will be able to run <tt>vzpkgls</tt> or <tt>vzpkg list -O</tt> and see the template(s) listed on the target HN.<br />
<br />
So far, this all supposes template doesn't exist on the target- very simple, just copy it over. If the template exists already on the target HN, this requires closer inspection. With older templates, simply moving over the templates would be the end of it- you see the version listed when you do a <tt>vzplgls</tt>:<br />
<br />
mysql-rh9 20030814 20050412<br />
<br />
this means there are 2 versions of the mysql package for RH9: 20030814 and 20050412<br />
As an aside, you can't copy over just 1 of them, but if you rsync'd over the <tt>mysql-rh9</tt> directory, you'd see 20030814 and 20050412 on the target HN. As long as the versions match up, you're good to go.<br />
<br />
With EZ templates, the versions are not as easy to decipher. When you look at the <tt>vzpkg list</tt> output, that simply tells you when a cache was created. A cache is simply a snapshot of all the data needed for the latest versions of the OS and it's applications at the time the cache was created. It is a date, and thus tells you nothing about the versions within. So for instance, if you had a cache for Ubuntu 10.04 from 2007 and you ran <tt>vzpkg create cache ubuntu-10.04-x86</tt> it would re-download the latest version of apache, mysql and so on. And any ''subsequent'' ubuntu 10.04 CT's created thereafter would use those newer versions, whereas older CT's based on 10.04 would use older templates. You can see how this works by looking at the files under:<br />
<br />
<pre># ls /vz/template/ubuntu/10.04/x86/<br />
aap_1.091-1_all<br />
aap-doc_1.091-1_all<br />
adduser_3.112ubuntu1_all<br />
anacron_2.3-13.1ubuntu11_i386<br />
apache2_2.2.14-5ubuntu8.2_i386<br />
apache2_2.2.14-5ubuntu8.4_i386<br />
apache2_2.2.14-5ubuntu8.6_i386<br />
apache2_2.2.14-5ubuntu8.7_i386<br />
apache2_2.2.14-5ubuntu8.8_i386<br />
apache2_2.2.14-5ubuntu8.9_i386<br />
<...snip><br />
</pre><br />
<br />
You can see that this template has in it 6 different versions of apache2. This means that this template was merged with other 10.04 templates and/or it has been cached a few times, each time a new apache was available. The unfortunate thing is it's almost impossible to know which CT's are using which version of apache so it's hard to try to prune this down and remove unwanted/unneeded applications.<br />
<br />
So, if you see another HN has Ubuntu 10.04 on it, you need to see/confirm that it has all the exact files your CT needs. You can run a test rsync to see what ''would'' be transferred (what's missing):<br />
<br />
rsync -avn --stats -e ssh /vz/template/ubuntu/10.04 root@10.1.4.69:/vz/template/ubuntu<br />
<br />
Based on the amount of data you get back from the rsync, you will be able to decide whether or not to remove the -n and allow the full transfer to go through. The downside to doing the transfer is the situation described above- you've permanently joined these templates and now you may have 10 versions of apache on the target HN. There's one other potential pitfall to this kind of merging- it will also potentially copy over shared files, for which there is no particular version, most of which are in <tt>/vz/template/ubuntu/10.04/x86/config</tt>: <br />
<br />
cat /vz/template/ubuntu/10.04/x86/config/os/default/repositories<br />
/vz/template/ubuntu/10.04/x86/config/os/default/repositories<br />
<br />
Here are some specific things to look out for. Case: copying centos-5 on virt19 to virt12. We dumped the rsync output to a file so we could inspect:<br />
<br />
[root@virt19 /vz/template]# rsync -an -e ssh /vz/template/centos/ root@10.1.4.62:/vz/template/centos/ > v12_c5<br />
<br />
(note, that can be dangerous, better to add on full path <tt>/vz/template/centos/5</tt>)<br />
<br />
So in the output file we see things like:<br />
<br />
<pre><br />
x86/base0/cachecookie<br />
x86/base0/primary.xml.gz <br />
x86/base0/primary.xml.gz.sqlite <br />
x86/base0/repomd.xml <br />
x86/base1/cachecookie <br />
x86/base1/filelists.xml.gz <br />
x86/base1/filelists.xml.gz.sqlite <br />
x86/base1/primary.xml.gz <br />
x86/base1/primary.xml.gz.sqlite <br />
x86/base1/repomd.xml<br />
x86/base2/cachecookie<br />
</pre><br />
<br />
Which make us nervous cause those are not versioned files, i.e.:<br />
5/x86/MAKEDEV-3.23-1.2.i386/usr/sbin/mksock<br />
<br />
So those files will replace existing files on v12 rather than the case of the latter where it's likely being added. So caution should be given and deference to whichever version is newer (which rsync should take care of, but that will depend on the date each file was created and perhaps not the actual data within).<br />
<br />
If we look further in the file we see:<br />
<br />
x86/psa-appvault-moodle-1.8-8202920080409011254.noarch/usr/local/psa/var/cgitory/moodle-1.<br />
9/htdocs/lib/editor/htmlarea/plugins/TableOperations/img/cell-split.gif<br />
<br />
and other files looking like:<br />
5/x86/plesk-base-9.0.1-cos5.build90090127.18.i586/etc/sw/keys/info<br />
<br />
Which means plesk is here. We don't need plesk on v12 (the CT moving over doesn't use it) so we rerun the rsync and exclude it:<br />
<br />
[root@virt19 /vz/template]# rsync -an -e ssh --exclude="plesk*" --exclude="psa*" /vz/template/centos/ root@10.1.4.62:/vz/template/centos/ > v12_c5<br />
<br />
and now it's much smaller:<br />
<br />
[root@virt19 /vz/template]# cat v12_c5|wc -l<br />
97104<br />
<br />
Before running with excludes it was:<br />
[root@virt19 /vz/template]# cat v12_c5|wc -l <br />
238238<br />
<br />
So again, look at what you're doing. Generally, combining templates is fine however.<br />
<br />
If you're happy with the merge, run the rsync again without the "-n" to let it actually do the transfer.<br />
<br />
Once done, don't forget to add the new template to the HN in Management -> Reference -> Templates<br />
<br />
= getting help =<br />
<br />
= vzup2date =<br />
TODO <br />
= Adding new OS/application templates =<br />
<br />
Virtuozzo will lag a bit behind the initial release of an OS, perhaps by a month or more. Further, a newly-released OS may be available, but there may not be many/any application templates available initially. It pays to wait a bit.<br />
<br />
A template is easily installed via [[#vzup2date|vzup2date]]:<br />
<br />
vzup2date -z<br />
<br />
You will be given a list of OS's to install. Select an OS, then customize that selection by checking/unchecking the application templates we do/don't want to include/offer.<br />
<br />
Generally, here are the app templates we include/offer:<br />
<pre>cyrus-imap<br />
devel<br />
imp<br />
jre<br />
jsdk<br />
mailman<br />
mod_perl<br />
mod_ssl<br />
mysql<br />
php<br />
phpmyadmin<br />
phppgadmin<br />
postgresql<br />
proftpd<br />
spamassassin<br />
squirrelmail<br />
tomcat<br />
vsftpd<br />
</pre><br />
<br />
We do not (any longer) offer webalizer or analog.<br />
<br />
After selecting and installing the OS and apps, you should ensure it's installed:<br />
<br />
vzpkg list -O<br />
<br />
Your new OS (ex. fedora-core-13-x86_64) will be listed, without a corresponding cache date:<br />
<pre><br />
[root@virt13 /vzconf]# vzpkg list -O<br />
centos-6-x86 2011-07-22 13:24:41<br />
centos-6-x86_64 2012-03-09 20:42:22<br />
centos-5-x86 2009-07-27 16:58:24<br />
centos-5-x86_64 2012-03-09 22:38:53<br />
ubuntu-11.10-x86_64 2012-03-01 23:13:33<br />
ubuntu-9.04-x86 2009-06-02 11:32:39<br />
ubuntu-12.04-x86_64 2012-05-29 13:50:50<br />
ubuntu-8.04-x86 2008-09-17 21:27:20<br />
ubuntu-8.10-x86 2009-03-13 13:58:57<br />
ubuntu-11.04-x86 2011-12-05 11:15:46<br />
ubuntu-7.04-x86 2008-09-24 16:42:50<br />
redhat-el6-x86_64<br />
fedora-core-13-x86_64 <br />
fedora-core-12-x86 2010-02-06 13:24:32<br />
fedora-core-15-x86 2011-12-05 11:36:43<br />
fedora-core-11-x86 2009-10-01 16:30:59<br />
fedora-core-14-x86<br />
fedora-core-16-x86_64 2012-03-13 11:40:23<br />
fedora-core-9-x86 2008-11-29 10:52:18<br />
debian-6.0-x86 2011-07-29 16:00:35<br />
debian-6.0-x86_64 2012-03-12 19:53:33<br />
debian-5.0-x86 2009-03-12 12:39:11<br />
</pre><br />
<br />
You can also confirm the apps installed:<br />
<pre>[root@virt13 /vzconf]# vzpkg list fedora-core-13-x86_64<br />
fedora-core-13-x86_64 2013-03-08 11:39:44<br />
fedora-core-13-x86_64 devel<br />
fedora-core-13-x86_64 php<br />
fedora-core-13-x86_64 proftpd<br />
fedora-core-13-x86_64 postgresql<br />
fedora-core-13-x86_64 phpmyadmin<br />
fedora-core-13-x86_64 mysql<br />
</pre><br />
<br />
Before you can use the OS you must cache it. To create the cache run:<br />
vzpkg cache [OS]<br />
<br />
ex: <tt>vzpkg cache fedora-core-13-x86_64</tt><br />
<br />
Now it's ready to use. In order to be able to use it with the [[VPS_Management#vm|vm]] command, you must create a file:<br />
jctmpl-[OS]<br />
<br />
ex: <tt>jctmpl-fedora-core-13-x86_64</tt><br />
<br />
In it, on line 1 we place the OS, followed by the app names, ex:<br />
<pre>more jctmpl-fedora-core-13-x86_64<br />
fedora-core-13-x86_64<br />
postgresql<br />
jsdk<br />
mod_ssl<br />
phpmyadmin<br />
mod_perl<br />
tomcat<br />
devel<br />
php<br />
mailman<br />
cyrus-imap<br />
squirrelmail<br />
proftpd<br />
mysql<br />
phppgadmin<br />
spamassassin<br />
</pre><br />
<br />
The packages will be installed in that order, so any pre-req's should be handled there.<br />
<br />
Now, when you run [[VPS_Management#vm|vm]] it will show you the newly-added OS and you may create a CT using it.<br />
<br />
The last few tasks are related to mgmt:<br />
<br />
1. add to mgmt->reference->templates (use the existing templates as an example). Once added, the new template will be included with the list of OS's available for the given HN.<br />
2. if this is going to be a new OS offered for ordering, it needs to be added to the order form and webpage.<br />
<br />
a. to get a list of applications and versions in the new OS, you should create a test CT, enter it and run <tt>dpkg -l|sort</tt> (or <tt>rpm -aq|sort</tt> in centos/fedora) and enter that list in the following places:<br><br />
<tt>/usr/local/www/jc_pub/[osname].html</tt><br><br />
<tt>/usr/local/www/signup/html/[osname].html</tt><br><br />
Note, you will have to move the most recent OS data from <tt>/usr/local/www/signup/html/[osname].html</tt> into the corresponding <tt>/usr/local/www/signup/html/[osname]_old.html</tt> updating the version links in both.<br />
<br />
b. to add the new OS to the order form, you will need to update <tt>/usr/local/www/signup/html/step1.html</tt> in several places (for regular and OSS orders) as well as updating the templateID which you can get from mgmt->reference->templates (or [[Management_System_/_Public_Website_/_Signup_/_Account_Manager#jc:ref_templates|jc:ref_templates]])<br />
<br />
<br />
<br />
Old method to obtain the templates:<br />
<br />
Go to http://www.parallels.com/en/virtuozzo/templates/catalog/<br />
<br />
Download the RPM's and install them<br />
<br />
vzpkg list<br />
to see the new packages/os templates<br />
<br />
vzpkg create cache "OS_EZ_template_name"<br />
to create a cache<br />
<br />
<br />
= Update OS template =<br />
<br />
Installation and update of the new version of OS template will not affect the existing containers. It will affect only the new containers, created after this update operation. In terms of fixing "vzctl enter" error and SSH connectivity to Ubuntu based containers, it is necessary to recreate the template and not to update it, so the commands to run are:<br />
<br />
<pre>rpm -Uhv ubuntu-8.04-x86-ez-4.0.0-9.swsoft.noarch.rpm<br />
vzpkg clean ubuntu-8.04-x86<br />
vzpkg remove cache ubuntu-8.04-x86<br />
vzpkg create cache ubuntu-8.04-x86</pre><br />
<br />
<br />
= Installing new SSL cert on VZCP =<br />
<br />
<pre>openssl req -days 1999 -new -x509 -nodes -out server.crt -keyout server.key<br />
<br />
US<br />
CA<br />
San Diego<br />
johncompanies.com<br />
johncompanies.com<br />
virt15ohncompanies.com<br />
linux@johncompanies.com<br />
<br />
cp -p /etc/httpd/conf/ssl.crt/server.crt /etc/httpd/conf/ssl.crt/server.crt.bak<br />
cp -p /etc/httpd/conf/ssl.key/server.key /etc/httpd/conf/ssl.key/server.key.bak<br />
cp server.crt /etc/httpd/conf/ssl.crt/server.crt<br />
cp server.key /etc/httpd/conf/ssl.key/server.key<br />
/etc/init.d/vzcp restart</pre><br />
<br />
= Moving virt (drives) to new hardware =<br />
<br />
If you have a case where a virt is dead (due to hardware failure), you can move the drives to a new hardware setup.<br />
Here are the steps and things to look for:<br />
<br />
Move the drives to the new box. Most likely, the drives will be recognized and there will be no problems booting up. On Dell PERC 5/6 cards you will need to import the new "foreign" volume via the raid BIOS.<br />
<br />
If the hardware (motherboard) is different, it’s possible the nic’s won’t be recognized. Look at /etc/modules.conf the ethernet drivers are either:<br />
<br />
<pre>alias eth0 tg3<br />
alias eth1 tg3</pre><br />
(supermicro)<br />
<br />
Or<br />
<br />
<pre>alias eth0 e100<br />
alias eth1 e1000</pre><br />
(intel)<br />
<br />
Or<br />
<br />
<pre>alias eth0 bnx2<br />
alias eth1 bnx2</pre><br />
(dell 29500)<br />
<br />
Also, you may see eth0 and eth1 swapped so if after confirming the eth devices are present (may require a reboot), then you may still need to swap the green/yellow network cables in the back to get them on the right port.<br />
<br />
VZ will not run (for long) on the new hardware, so you will need to get the license reissued. You may create/download a temporary license via their website for the interim.<br />
<br />
= Updating kernel on virt =<br />
<br />
We now use [[#vzup2date|vzup2date]] to update systems and kernels. What follows is what we used to do when we had to download kernels and install manually on old systems:<br />
<br />
<br />
1. install kernel and modules<br />
<pre># rpm -ivh vzkernel-enterprise-2.4.20-021stab028.12.777.i686.rpm \<br />
vzmodules-enterprise-2.4.20-021stab028.12.swsoft.i686.rpm<br />
vzkernel-smp ##################################################<br />
vzmodules-smp ##################################################</pre><br />
<br />
DO NOT USE the "rpm -Uhv" command to install the kernel, otherwise all the previously installed kernels may be removed from your system. <br />
<br />
2. update lilo/grub<br />
<br />
Lilo:<br />
<br />
vi /etc/lilo.conf<br />
<br />
rename old Virtuozzo kernel label from "linux+virtuozzo" to "virtuozzo_old"<br />
and add a new entry pointing to the newly installed virtuozzo kernel image.<br />
For example:<br />
<br />
<pre>prompt<br />
timeout=50<br />
default=linux+virtuozzo<br />
append="debug panic=5 console=tty console=ttyS0,38400"<br />
boot=/dev/sda<br />
map=/boot/map<br />
install=/boot/boot.b<br />
message=/boot/message<br />
linear<br />
<br />
image=/boot/vmlinuz-2.4.18-3smp<br />
label=linux<br />
initrd=/boot/initrd-2.4.18-3smp.img<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.18-3<br />
label=linux-up<br />
initrd=/boot/initrd-2.4.18-3.img<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-020stab009.5.777-enterprise<br />
label=2.4.20-9.5.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-020stab009.8.777-enterprise<br />
label=2.4.20-9.8.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-020stab009.21.777-enterprise<br />
label=2.4.20-9.21.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-020stab009.24.777-enterprise<br />
label=2.4.20-9.24.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-021stab028.3.777-enterprise<br />
label=2.4.20-28.3.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-021stab028.5.777-enterprise<br />
label=old_linux+vz<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-021stab028.12.777-enterprise<br />
label=linux+virtuozzo<br />
read-only<br />
root=/dev/sda1</pre><br />
<br />
With this configuration, the new kernel is the default kernel to use at boot. The original kernel entry has been retained with the label "old_linux+vz ".<br />
<br />
Grub:<br />
<br />
vi /boot/grub /menu.lst<br />
<br />
Add new entry at the top.<br />
For example:<br />
<br />
<pre>serial --unit=0 --speed=38400<br />
terminal --timeout=10 serial console<br />
default=0<br />
timeout=10<br />
splashimage=(hd0,0)/boot/grub/splash.xpm.gz<br />
title Virtuozzo 2.6.1 (2.4.20-021stab028.12.777-enterprise)<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-22.4.20-021stab028.12.777-enterprise root=/dev/sda1 console=tty0 \ console=ttyS0,38400<br />
title Virtuozzo 2.6.1 (2.4.20-021stab028.3.777-enterprise)<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.20-021stab028.3.777-enterprise root=/dev/sda1 console=tty0 \ console=ttyS0,38400<br />
title Red Hat Linux (2.4.18-3smp)<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.18-3smp ro root=/dev/sda1<br />
initrd /boot/initrd-2.4.18-3smp.img<br />
title Red Hat Linux-up (2.4.18-3)<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.18-3 ro root=/dev/sda1<br />
initrd /boot/initrd-2.4.18-3.img<br />
title Linux with Virtuozzo 2.5<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.20-020stab009.3.777-smp ro root=/dev/sda1<br />
title vz-enterpriseo<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.20-020stab009.21.777-enterprise ro root=/dev/sda1<br />
title vz-enterprise<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.20-020stab009.24.777-enterprise ro root=/dev/sda1 \ console=tty0 console=ttyS0,38400</pre><br />
<br />
3. run the lilo (if using lilo)<br />
<br />
<pre># lilo<br />
Added linux<br />
Added linux-up<br />
Added virtuozzo_old<br />
Added linux+virtuozzo *</pre><br />
<br />
4. reboot the server<br />
shutdown -r 23:05<br />
<br />
when it comes time for the shutdown, run [[VPS_Management#stopvirt|stopvirt]] to get things shut down faster<br />
<br />
<br />
= Remaking service VE (<=3.x only) =<br />
<br />
Occasionally the control panel for VEs (aka VZPP) will stop working and you just need to reinstall the service VE (which runs the control panels for all VEs on the HN). Here's how that's done :<br />
<br />
<pre>vzctl stop 1<br />
ipdel 1 69.55.234.152<br />
vzmlocal 1:2<br />
vzctl set 2 --save --onboot no<br />
vzsveinstall -t redhat-as3-minimal -d /root/Rel261/HW/RPMS -s 69.55.234.152 --skip-client</pre><br />
<br />
on 3.0:<br />
<br />
<pre>vzsveinstall -t redhat-as3-minimal -d /vz/Rel300/HW/RPMS -s 69.55.229.151 --skip-client<br />
<br />
vzctl stop 1<br />
vzctl mount 1<br />
vzctl mount 2<br />
/bin/cp -aRf /vz/root/2/home/vzagent0 /vz/root/1/home<br />
/bin/cp -aRf /vz/root/2/var/lib/mysql/VZADB /vz/root/1/var/lib/mysql<br />
/bin/cp -af /vz/root/2/etc/shadow /vz/root/1/etc/<br />
/bin/cp -aRf /vz/root/2/var/vzcp/static/vz/skins/ /vz/root/1/var/vzcp/static/vz<br />
/bin/cp -af /vz/root/2/etc/vzcp/pp/menu.xml /vz/root/1/etc/vzcp/pp/<br />
/bin/cp -af /vz/root/2/etc/vzcp/pp/dashboard.xml /vz/root/1/etc/vzcp/pp/<br />
<br />
vzctl umount 2<br />
vzctl umount 1<br />
vzctl start 1</pre><br />
<br />
<br />
= GRUB: boot once to a new kernel =<br />
<br />
To use a different kernel for the next reboot only:<br />
<pre>grub --no-floppy<br />
savedefault --default=N --once<br />
(where N is the number of the kernel you want to boot once)<br />
quit</pre><br />
<br />
Reboot your machine. It will boot using kernel "N".<br />
<br />
grub-reboot N<br />
will do essentially the same thing and prompt to reboot the machine<br />
<br />
<br />
= Recovering from GRUB failure =<br />
<br />
(example from DPE 2950)<br />
<br />
Boot to CEOS4 server CD (original install cd). Did this example with iso mounted via drac<br />
<br />
Boot: linux rescue</div>Supporthttps://wiki.jcihosting.com/index.php?title=VPS_Management&diff=1038VPS Management2013-03-11T23:49:46Z<p>Support: /* Making new customer VE (vm) */</p>
<hr />
<div>= Common Problems =<br />
== Login to any machine without a password ==<br />
<br />
This is possible via the use of ssh keys. The process is thus:<br />
<br />
1. place the public key for your user (root@mail) in the /root/.ssh/authorized_keys file on the server you wish to login to<br />
cat /root/.ssh/id_dsa.pub<br />
(paste that into authorized_keys on the target server). If the file doesn't exist, create it.<br />
<br />
2. enable root login (usually only applies to FreeBSD). Edit the /etc/ssh/sshd_config on the target server and change:<br />
<tt>#PermitRootLogin no</tt><br />
to<br />
<tt>PermitRootLogin yes</tt><br />
<br />
3. Restart the sshd on the target machine. First, find the sshd process: <br />
jailps <hostname> | grep sshd <br />
or <br />
vp <VEID> | grep sshd<br />
<br />
Look for the process resembling:<br />
root 17296 0.0 0.0 5280 1036 ? Ss 2011 4:27 /usr/sbin/sshd <br />
(this is the sshd)<br />
<br />
Not:<br />
root 6270 0.5 0.0 6808 2536 ? Ss 14:33 0:00 sshd: root [priv]<br />
(this is an sshd child- someone already ssh'd in as root)<br />
<br />
Restart the sshd: <br />
kill -1 <PID><br />
<br />
Ex:<br />
kill -1 17296<br />
<br />
You may now ssh in.<br />
<br />
Once you're done, IF you enabled root login, you should repeat steps 2 and 3 to disable root logins.<br />
<br />
== Letting someone in who has locked themselves out (killed sshd, lost pwd) ==<br />
<br />
There are two ways people frequently lock themselves out - either they forget a password, or they kill off sshd somehow.<br />
<br />
These are actually both fairly easy to solve. First, let's say someone kills off their sshd, or somehow mangles /etc/ssh/sshd_config such that it no longer lets them in.<br />
<br />
Their email may be very short, or it may have all sorts of details about how you should fix sshd_config to let them in ... just ignore all of this. They can fix their own mangled sshd. Fixing this is very simple. First, edit the /etc/inetd.conf on their system and uncomment the telnet line:<br />
<br />
telnet stream tcp nowait root /usr/libexec/telnetd telnetd<br />
#telnet stream tcp6 nowait root /usr/libexec/telnetd telnetd<br />
<br />
(just leave the tcp6 version of telnet commented)<br />
<br />
Then, use jailps to list the processes on their system, and find their inetd process. Then simply:<br />
<br />
kill -HUP (pid)<br />
<br />
where (pid) is the PID of their inetd process. Now they have telnet running on their system and they can log in and do whatever they need to do.<br />
<br />
The only complications that could occur are:<br />
<br />
a) their firewall config on our firewall has port 23 blocked, in which case you will need to open that - will be covered in a different lesson.<br />
<br />
b) they are not running inetd, so you can't HUP it. If this happens, edit their /etc/rc.conf, add the inetd_enable="YES" line, and then kill<br />
their jail with /tmp/jailkill.pl - then restart their jail with the jail line from their quad/safe file. Easy.<br />
<br />
If they have forgotten a password,<br />
<br />
On 6.x+ you can reset their password with:<br />
jexec <jailID from jls> passwd root<br />
<br />
Note: the default password for 6.x jails is 8ico2987, for 4.x it is p455agfa<br />
<br />
On 4.x, you need to cd to their etc directory<br />
... for instance:<br />
<br />
cd /mnt/data2/198.78.65.136-col00261-DIR/etc<br />
<br />
and run:<br />
<br />
vipw -d .<br />
<br />
Then paste in these two lines (theres a paste with these):<br />
<br />
root:$1$krszPxhk$xkCepSnz3mIikT3vCtJCt0:0:0::0:0:Charlie &:/root:/bin/csh<br />
user:$1$Mx9p5Npk$QdMU6c8YQqp2FW2M3irEh/:1001:1001::0:0:User &:/home/user:/bin/sh<br />
<br />
overwriting the lines they already have for "user" and "root" - then just tell them that both user and root have been reset to the default password of p455agfa.<br />
<br />
For linux, just passwd inside shell or <br />
vzctl set <veid> --userpasswd root:p455agfa –save<br />
<br />
Starting in 2009 we began giving out randomized passwords for FreeBSD and Linux as the default password. That is stored with each system in Mgmt. You should look for and reset the password to that password in the event of a reset and refer the customer to use their original password from their welcome email- this way we don’t have to send the password again via email (in clear text).<br />
<br />
<br />
== sendmail can’t be contacted from ext ip (only locally) ==<br />
<br />
By default redhat puts this line in sendmail.mc:<br />
<br />
DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')<br />
<br />
which makes it only answer on localhost. Comment it out like:<br />
<br />
dnl DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')<br />
<br />
and then rebuild sendmail.cf with:<br />
<br />
m4 /etc/mail/sendmail.mc > /etc/sendmail.cf<br />
<br />
== virt doesn’t properly let go of ve’s ip(s) when moved to another system ==<br />
<br />
On virtuozzo 2.6 systems, it's been observed that when moving ips from one virt to another that sometimes the routing table will not get updated to reflect the removal of the ip addresses.<br />
<br />
A recent example was a customer that was moving to a new ve on a new virt and the ip addresses were traded between the two ve's. After the trade the two systems were not able to talk to each other. When looking at the routing table for the old system all the ip addresses were still in the routing table as being local, like so:<br />
<br />
<pre>netstat -rn | grep 69.55.225.149<br />
69.55.225.149 0.0.0.0 255.255.255.255 UH 40 0 0 venet0</pre><br />
<br />
This was preventing traffic to the other system from being routed properly.<br />
The solution is to manually delete the route:<br />
<br />
route delete 69.55.225.149 gw 0.0.0.0<br />
<br />
Supposedly, this was fixed in 2.6.1<br />
<br />
== sshd on FreeBSD 6.2 segfaults ==<br />
<br />
First try to reinstall ssh<br />
<br />
<pre>cd /usr/src/secure<br />
cd lib/libssh<br />
make depend && make all install<br />
cd ../../usr.sbin/sshd<br />
make depend && make all install<br />
cd ../../usr.bin/ssh<br />
make depend && make all install</pre><br />
<br />
Failing that, find the library that’s messed up:<br />
<br />
<pre>ldd /usr/sbin/sshd<br />
libssh.so.3 => /usr/lib/libssh.so.3 (0x280a3000) <br />
libutil.so.5 => /lib/libutil.so.5 (0x280d8000) <br />
libz.so.3 => /lib/libz.so.3 (0x280e4000) <br />
libwrap.so.4 => /usr/lib/libwrap.so.4 (0x280f5000) <br />
libpam.so.3 => /usr/lib/libpam.so.3 (0x280fc000) <br />
libbsm.so.1 => /usr/lib/libbsm.so.1 (0x28103000) <br />
libgssapi.so.8 => /usr/lib/libgssapi.so.8 (0x28112000) <br />
libkrb5.so.8 => /usr/lib/libkrb5.so.8 (0x28120000) <br />
libasn1.so.8 => /usr/lib/libasn1.so.8 (0x28154000) <br />
libcom_err.so.3 => /usr/lib/libcom_err.so.3 (0x28175000) <br />
libroken.so.8 => /usr/lib/libroken.so.8 (0x28177000) <br />
libcrypto.so.4 => /lib/libcrypto.so.4 (0x28183000) <br />
libcrypt.so.3 => /lib/libcrypt.so.3 (0x28276000) <br />
libc.so.6 => /lib/libc.so.6 (0x2828e000) <br />
libmd.so.3 => /lib/libmd.so.3 (0x28373000)</pre><br />
<br />
md5 them and compare to other jail hosts or jails running on host<br />
<br />
for libcrypto reinstall:<br />
<pre>/usr/src/crypto<br />
make depend && make all install</pre><br />
<br />
= FreeBSD VPS =<br />
<br />
== Starting jails: Quad/Safe Files ==<br />
<br />
FreeBSD customer systems do not start up automatically at boot time. When one of our freebsd machines boots up, it boots up, and does nothing else. To start jails, we put the commands to start each jail into a shell script(s) and run the script(s). Jail startup is something that needs to be actively monitored, which is why we don’t just run the script automatically. More on monitoring later.<br />
<br />
NOTE: >=7.x we have moved to 1 quad file: <tt>quad1</tt>. Startups are not done by running each quad, but rather [[#startalljails|startalljails]] which relies on the contents of <tt>quad1</tt>. The specifics of this are lower in this article. What follows here applies for pre 7.x systems.<br />
<br />
There are eight files in <tt>/usr/local/jail/rc.d</tt>:<br />
<br />
<pre>jail3# ls /usr/local/jail/rc.d/<br />
quad1 quad2 quad3 quad4 safe1 safe2 safe3 safe4<br />
jail3#</pre><br />
<br />
four quad files and four safe files.<br />
<br />
Each file contains an even number of system startup blocks (total number of jails divided by 4)<br />
<br />
The reason for this is, if we make one large script to startup all the systems at boot time, it will take too long - the first system in the script will start up right after system boot, which is great, but the last system may not start for another 20 minutes.<br />
<br />
Since there is no way to parralelize this during the startup procedure, we simply open four terminals (in screen window 9) and run each script, one in each terminal. This way they all run simultaneously, and the very last system in each startup script gets started in 1/4th the time it would if there was one large file<br />
<br />
The files are generally organized so that quad/safe 1&2 have only jails from disk 1, and quad/safe 3&4 have jails from disk 2. This helps ensure that only 2 fscks on any disk are going on at once. Further, they are balanced so that all quad/safe’s finish executing around the same time. We do this by making sure each quad/safe has a similar number of jails and represents a similar number of inodes (see js).<br />
<br />
The other, very important reason we do it this way, and this is the reason there are quad files and safe files, is that in the event of a system crash, every single vn-backed filesystem that was mounted at the time of system crash needs to be fsck'd. However, fsck'ing takes time, so if we shut the system down gracefully, we don't want to fsck.<br />
<br />
Therefore, we have two sets of scripts - the four quad scripts are identical to the four safe scripts except for the fact that the quad scripts contain fsck commands for each filesystem.<br />
<br />
So, if you shut a system down gracefully, start four terminals and run safe1 in window one, and safe2 in window 2, and so on.<br />
<br />
If you crash, start four terminals (or go to screen window 9) and run quad1 in window one, and quad2 in window 2, and so on.<br />
<br />
Here is a snip of (a 4.x version) quad2 from jail17:<br />
<br />
<pre>vnconfig /dev/vn16 /mnt/data2/69.55.228.7-col00820<br />
fsck -y /dev/vn16<br />
mount /dev/vn16c /mnt/data2/69.55.228.7-col00820-DIR<br />
chmod 0666 /mnt/data2/69.55.228.7-col00820-DIR/dev/null<br />
jail /mnt/data2/69.55.228.7-col00820-DIR mail1.phimail.com 69.55.228.7 /bin/sh /etc/rc<br />
<br />
# moved to data2 col00368<br />
#vnconfig /dev/vn28 /mnt/data2/69.55.236.132-col00368<br />
#fsck -y /dev/vn28<br />
#mount /dev/vn28c /mnt/data2/69.55.236.132-col00368-DIR<br />
#chmod 0666 /mnt/data2/69.55.236.132-col00368-DIR/dev/null<br />
#jail /mnt/data2/69.55.236.132-col00368-DIR limehouse.org 69.55.236.132 /bin/sh /etc/rc<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
vnconfig /dev/vn22 /mnt/data2/69.55.228.13-col01063<br />
fsck -y /dev/vn22<br />
mount /dev/vn22c /mnt/data2/69.55.228.13-col01063-DIR<br />
chmod 0666 /mnt/data2/69.55.228.13-col01063-DIR/dev/null<br />
jail /mnt/data2/69.55.228.13-col01063-DIR www.widestream.com.au 69.55.228.13 /bin/sh /etc/rc<br />
<br />
# cancelled col00106<br />
#vnconfig /dev/vn15 /mnt/data2/69.55.238.5-col00106<br />
#fsck -y /dev/vn15<br />
#mount /dev/vn15c /mnt/data2/69.55.238.5-col00106-DIR<br />
#chmod 0666 /mnt/data2/69.55.238.5-col00106-DIR/dev/null<br />
#jail /mnt/data2/69.55.238.5-col00106-DIR mail.azebu.net 69.55.238.5 /bin/sh /etc/rc</pre><br />
<br />
As you can see, two of the systems specified are commented out - presumably those customers cancelled, or were moved to new servers.<br />
<br />
As you can see, the vnconfig line is the simpler command line, not the longer one that was used when it was first configured. As you can see, all that is done is, vnconfig the filesystem, then fsck it, then mount it. The fourth command is the `jail` command used to start the system – but that will be covered later.<br />
<br />
Here is the safe2 file from jail17:<br />
<br />
<pre>vnconfig /dev/vn16 /mnt/data2/69.55.228.7-col00820<br />
mount /dev/vn16c /mnt/data2/69.55.228.7-col00820-DIR<br />
chmod 0666 /mnt/data2/69.55.228.7-col00820-DIR/dev/null<br />
jail /mnt/data2/69.55.228.7-col00820-DIR mail1.phimail.com 69.55.228.7 /bin/sh /etc/rc<br />
<br />
# moved to data2 col00368<br />
#vnconfig /dev/vn28 /mnt/data2/69.55.236.132-col00368<br />
#mount /dev/vn28c /mnt/data2/69.55.236.132-col00368-DIR<br />
#chmod 0666 /mnt/data2/69.55.236.132-col00368-DIR/dev/null<br />
#jail /mnt/data2/69.55.236.132-col00368-DIR limehouse.org 69.55.236.132 /bin/sh /etc/rc<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
vnconfig /dev/vn22 /mnt/data2/69.55.228.13-col01063<br />
mount /dev/vn22c /mnt/data2/69.55.228.13-col01063-DIR<br />
chmod 0666 /mnt/data2/69.55.228.13-col01063-DIR/dev/null<br />
jail /mnt/data2/69.55.228.13-col01063-DIR www.widestream.com.au 69.55.228.13 /bin/sh /etc/rc<br />
<br />
# cancelled col00106<br />
#vnconfig /dev/vn15 /mnt/data2/69.55.238.5-col00106<br />
#mount /dev/vn15c /mnt/data2/69.55.238.5-col00106-DIR<br />
#chmod 0666 /mnt/data2/69.55.238.5-col00106-DIR/dev/null<br />
#jail /mnt/data2/69.55.238.5-col00106-DIR mail.azebu.net 69.55.238.5 /bin/sh /etc/rc</pre><br />
<br />
As you can see, it is exactly the same, but it does not have the fsck lines.<br />
<br />
Take a look at the last entry - note that the file is named:<br />
<br />
/mnt/data2/69.55.238.5-col00106<br />
<br />
and the mount point is named:<br />
<br />
/mnt/data2/69.55.238.5-col00106-DIR<br />
<br />
This is the general format on all the FreeBSD systems. The file is always named:<br />
<br />
IP-custnumber<br />
<br />
and the directory is named:<br />
<br />
IP-custnumber-DIR<br />
<br />
If you run safe when you need a fsck, the mount will fail and jail will fail:<br />
<br />
# mount /dev/vn1c /mnt/data2/jails/65.248.2.131-ns1.kozubik.com-DIR<br />
mount: /dev/vn1c: Operation not permitted<br />
<br />
No reboot needed, just run the quad script<br />
<br />
Starting with 6.x jails, we added block delimiters to the quad/safe files, the block looks like:<br />
<br />
<pre>echo '## begin ##: nuie.solaris.mu'<br />
fsck -y /dev/concat/v30v31a<br />
mount /dev/concat/v30v31a /mnt/data1/69.55.228.218-col01441-DIR<br />
mount_devfs devfs /mnt/data1/69.55.228.218-col01441-DIR/dev<br />
devfs -m /mnt/data1/69.55.228.218-col01441-DIR/dev rule -s 3 applyset<br />
jail /mnt/data1/69.55.228.218-col01441-DIR nuie.solaris.mu 69.55.228.218 /bin/sh /etc/rc<br />
echo '## end ##: nuie.solaris.mu'</pre><br />
<br />
These are more than just informative when running quad/safe’s, the echo lines MUST be present for certain tools to work properly. So it’s important that any updates to the hostname also be updated on the 2 echo lines. For example, if you try to startjail a jail with a hostname which is on the jail line but not the echo lines, the command will return with host not found.<br />
<br />
=== FreeBSD 7.x+ notes ===<br />
<br />
Starting with the release of FreeBSD 7.x, we are doing jail startups in a slightly different way. First, thereis only 1 file: <tt>/usr/local/jail/rc.d/quad1</tt><br />
There are no other quads or corresponding safe files. The reason for this is twofold, 1. We can pass –C to fsck which will tell is to skip the fsck if the fs is clean (no more need for safe files), 2. We have a new startup script which can be launched multiple times, running in parallel to start jails, where quad1 is the master jail file. <br />
Quad1 could still be run as a shell script, but it would take a very long time for it to run completely so it’s not advisable; or you should break it down into smaller chunks (like quad1, quad2, quad3, etc)<br />
<br />
Here is a snip of (a 7.x version) quad1 from jail2:<br />
<br />
<pre>echo '## begin ##: projects.tw.com'<br />
mdconfig -a -t vnode -f /mnt/data1/69.55.230.46-col01213 -u 50<br />
fsck -Cy /dev/md50c<br />
mount /dev/md50c /mnt/data1/69.55.230.46-col01213-DIR<br />
mount -t devfs devfs /mnt/data1/69.55.230.46-col01213-DIR/dev<br />
devfs -m /mnt/data1/69.55.230.46-col01213-DIR/dev rule -s 3 applyset<br />
jail /mnt/data1/69.55.230.46-col01213-DIR projects.tw.com 69.55.230.46 /bin/sh /etc/rc<br />
echo '## end ##: projects.tw.com'</pre><br />
<br />
Cancelled jails are no longer commented out and stored in quad1, rather they’re moved to <tt>/usr/local/jail/rc.d/deprecated</tt> <br />
<br />
To start these jails, start the 4 ssh sessions as you would for a normal crash and then instead of running quad1-4, instead run startalljails in each window. IMPORTANT- before running startalljails you should make sure you ran preboot once as it will clear out all the lockfiles and enable startalljails to work properly.<br />
<br />
== Problems with the quad/safe files ==<br />
<br />
When you run the quad/safe files, there are two problems that can occur - either a particular system will hang during initialization, OR a system will spit out output to the screen, impeding your ability to do anything. Or both.<br />
<br />
First off, when you start a jail, you see output like this:<br />
<br />
<pre>Skipping disk checks ...<br />
adjkerntz[25285]: sysctl(put_wallclock): Operation not permitted<br />
Doing initial network setup:.<br />
ifconfig: ioctl (SIOCDIFADDR): permission denied<br />
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384<br />
Additional routing options: TCP keepalive=YESsysctl:<br />
net.inet.tcp.always_keepalive: Operation not permitted.<br />
Routing daemons:.<br />
Additional daemons: syslogd.<br />
Doing additional network setup:.<br />
Starting final network daemons:.<br />
ELF ldconfig path: /usr/lib /usr/lib/compat /usr/X11R6/lib /usr/local/lib<br />
a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout /usr/X11R6/lib/aout<br />
Starting standard daemons: inetd cron sshd sendmail sendmail-clientmqueue.<br />
Initial rc.i386 initialization:.<br />
Configuring syscons: blanktime.<br />
Additional ABI support:.<br />
Local package initialization:.<br />
Additional TCP options:.</pre><br />
<br />
Now, let's look at this line, near the end:<br />
<br />
Local package initialization:.<br />
<br />
This is where a list of daemons that are set to start at boot time willshow up. You might see something like:<br />
<br />
Local package initialization: mysqld apache sendmail sendmail-clientmqueue<br />
<br />
Or something like this:<br />
<br />
Local package initialization: postgres postfix apache<br />
<br />
The problem is that many systems (about 4-5 per machine) will hang on that line. Basically it will get to some of the way through the total daemons to be started:<br />
<br />
Local package initialization: mysqld apache<br />
<br />
and will just sit there. Forever.<br />
<br />
Fortunately, pressing ctrl-c will break out of it. Not only will it break out of it, but it will also continue on that same line and start the other daemons:<br />
<br />
Local package initialization: mysqld apache ^c sendmail-clientmqueue<br />
<br />
and then continue on to finish the startup, and then move to the next system to be started.<br />
<br />
So what does this mean? It means that if a machine crashes, and you start four screen-windows to run four quads or four safes, you need to periodically cycle between them and see if any systems are stuck at that point, causing their quad/safe file to hang. A good rule of thumb is, if you see a system at that point in the startup, give it another 100 seconds - if it is still at the exact same spot, hit ctrl-c. Its also a good idea to go back into the quad file (just before the first command in the jail startup block) and note that this jail tends to need a control-c or more time as follows:<br />
<pre>echo '### NOTE ### slow sendmail'<br />
echo '### NOTE ###: ^C @ Starting sendmail.'</pre><br />
<br />
'''NEVER''' hit ctrl-c repeatedly if you don't get an immediate response - that will cause the following jail’s startup commands to be aborted.<br />
<br />
A second problem that can occur is that a jail - maybe the first one in that particular quad/safe, maybe the last one, or maybe one in the middle, will start spitting out status or error messages from one of its init scripts. This is not a problem - basically, hit enter a few times and see if you get a prompt - if you do get a prompt, that means that the quad/safe script has already completed. Therefore it is safe to log out (and log out of the user that you su'd from) and then log back in (if necessary).<br />
<br />
The tricky thing is, if a system in the middle starts flooding with messages, and you hit enter a few times and don't get a prompt. Are you not getting a prompt because some subsequent system is hanging at the initialization, as we discussede above ? Or are you not getting a prompt because that quad file is currently running an fsck ? Usually you can tell by scrolling back in screen’s history to see what it was doing before you started getting the messages.<br />
<br />
If you don’t get clues from history, you have to use your judgement - instead of giving it 100 seconds to respond, perhaps give it 2-3 mins ... if you still get no response (no prompt) when you hit enter, hit ctrl-c. However, be aware that you might still be hitting ctrl-c in the middle of an fsck. This means you will get an error like "filesystem still marked dirty" and then the vnconfig for it will fail and so will the jail command, and the next system in the quad file will then start starting up.<br />
<br />
If this happens, just wait until the end of all the quad files have finished, and start that system manually.<br />
<br />
If things really get weird, like a screen flooded with errors, and you can't get a prompt, and ctrl-c does nothing, then you need to just eventually (give it ten mins or so) just kill that window with ctrl-p, then k, and then log in again and manually check which systems are now running and which aren't, and manually start up any that are not.<br />
<br />
Don't EVER risk running a particular quad/safe file a second time.<br />
If the quad/safe script gets executed twice, reboot the machine immediately.<br />
<br />
So, for all the above reasons, anytime a machine crashes and you run all the quads or all the safes, '''always''' check every jail afterwards to make sure it is running - even if you have no hangs or complications at all.<br />
Run this command:<br />
<br />
<tt>[[#jailpsall|jailpsall]]</tt> <br />
<br />
Note: [[#postboot|postboot]] also populates ipfw counts, so it '''should not be run multiple times''', use <tt>jailpsall</tt> for subsequent extensive ps’ing<br />
<br />
And make sure they all show as running. If one does not show as running, check its /etc/rc.conf file first to see if maybe it is using a different hostname first before starting it manually.<br />
<br />
One thing we have implemented to alleviate these startup hangs and noisy jails, is to put jail start blocks that are slow or hangy at the bottom of the safe/quad file. Further, for each bad jail we note in each quad/safe just before the start block something like:<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
<br />
That way we’ll be prepared to ^C when we see that message appear during the quad/safe startup process. If you observe a new, undocumented hang, '''after''' the quad/safe has finished, place a line similar to the above in the quad file, move the jail start block to the end of the file, then run [[#buildsafe|buildsafe]]<br />
<br />
<br />
== Recovering from a crash (FreeBSD) ==<br />
<br />
=== Diagnose whether you have a crash ===<br />
The most important thing is to get the machine and all jails back up as soon as possible. Note the time, you’ll need to create a crash log entry (Mgmt. -> Reference -> CrashLog). The first thing to do is head over to the [[Screen#Screen_Organization|serial console screen]] and see if there’s any kernel error messages output. Try to copy any messages (or just a sample of repeating messages) you see into the notes section of the crash log. If there are no messages, the machine may just be really busy- wait a bit (5-10min) to see if it comes back. If it's still pinging, odds are its very busy. Note, if you see messages about swap space exhausted, the server is obviously out of memory, however it may recover briefly enough for you to get a jtop in to see who's lauched a ton of procs (most likely) and then issue a quick jailkill to get it back under control.<br />
<br />
If it doesn't come back, or the messages indicate a fatal error, you will need to proceed with a power cycle (ctrl+alt+del will not work).<br />
<br />
=== Power cycle the server ===<br />
If this machine is not a Dell 2950 with a [[DRAC/RMM#DRAC|DRAC card]] (i.e. if you can’t ssh into the DRAC card (as root, using the standard root pass) and issue <br />
racadm serveraction hardreset<br />
then you will need someone at the data center power the macine off, wait 30 sec, then turn it back on. Make sure to re-attach via console:<br />
tip jailX<br />
immediately after power down. <br />
<br />
=== (Re)attach to the console ===<br />
Stay on the console the entire time during boot. As the BIOS posts- look out for the RAID card output- does everything look healthy? The output may be scrambled, look for "DEGRADED" or "FAILED". Once the OS starts booting you will be disconnected (dropped back to the shell on the console server) a couple times during the boot up. The reason you want to quickly re-attach is two-fold: 1. If you don’t reattach quickly then you won’t get any console output, 2. you want to be attached before the server ''potentially'' starts (an extensive) fsck. If you attach after the fsck begins, you’ll have seen no indication it started an fsck and the server will appear frozen during startup- no output, no response. <br />
<br />
IMPORTANT NOTE: on some older FreeBSD systems, there will be no output to the video (KVM) console as it boots up. The console output is redirected to the serial port ... so if a jail crashes, and you attach a kvm, the output during the bootup procedure will not be shown on the screen. However, when the bootup is done, you will get a login prompt on the screen and will be able to log in as normal. <tt>/boot/loader.conf</tt> is where serial console redirect output lives, so comment that if you want to catch output on kvm.<br />
On newer systems it sends most output to both locations. <br />
<br />
=== Assess the heath of the server ===<br />
Once the server boots up fully, you should be able to ssh in. Look around- make sure all the mounts are there and reporting the correct size/usage (i.e. /mnt/data1 /mnt/data2 /mnt/data3 - look in /etc/fstab to determine which mount points should be there), check to see if RAID mirrors are healthy. See [[RAID_Cards#Common_CLI_commands_.28megacli.29|megacli]], [[#aaccheck|aaccheck]]<br />
<br />
Before you start the jails, you need to run [[#preboot|preboot]]. This will do some assurance checks to make sure things are prepped to start the jails. Any issues that come out of preboot need to be addressed before starting jails.<br />
<br />
=== Start jails ===<br />
[[#Starting_jails:_Quad.2FSafe_Files|More on starting jails]]<br />
Customer jails (the VPSs) do not start up automatically at boot time. When a FreeBSD machines boots up, it boots up, and does nothing else. To start jails, we put the commands to start each jail into a shell script(s) and run the script(s). Jail startup is something that needs to be actively monitored, which is why we don’t just run the script automatically. <br />
<br />
In order to start jails, we run the quad files: quad1 quad2 quad3 and quad4 (on new systems there is only quad1). If the machine was cleanly rebooted- which wouldn't be the case if this was a crash, you may run the safe files (safe1 safe2 safe3 safe4) in lieu of quads. <br />
<br />
Open up 4 logins to the server (use the windows in [[Screen#Screen_Organization|a9]])<br />
In each of the 4 windows you will:<br />
<br />
If there is a [[#startalljails|startalljails]] script (and only quad1), run that command in each of the 4 windows. It will parse through the quad1 file and start each jail. Follow the instructions [[#Problems_with_the_quad.2Fsafe_files|here]] for monitoring startup. Note that you can be a little more lenient with jails that take awhile to start- startalljails will work around the slow jails and start the rest. As long as there aren't 4 jails which are "hung" during startup, the rest will get started eventually.<br />
-or-<br />
If there is no startalljails script, there will be multiple quad files. In each of the 4 windows, start each of the quads. i.e. start quad1 in window1, quad2 in window2 and so on. DO NOT start any quad twice. It will crash the server. If you accidentally do this, just jailkill all the jails which are in the quad and run the quad again. Follow the instructions here for monitoring quad startup.<br />
<br />
Note the time the last jail boots- this is what you will enter in the crash log.<br />
<br />
Save the crash log.<br />
<br />
=== Check to make sure all jails have started ===<br />
There's a simple script which will make sure all jails have started, and enter the ipfw counter rules: [[#postboot|postboot]] <br />
Run postboot, which will do a jailps on each jail it finds (excluding commented out jails) in the quad file(s). We're looking for 2 things:<br />
# systems spawning out of control or too many procs<br />
# jails which haven't started<br />
On 7.x and newer systems it will print out the problems (which jails haven't started) at the conclusion of postboot. <br />
On older systems you will need to watch closely to see if/when there's a problem, namely:<br />
<br />
[hostname] doesnt exist on this server<br />
<br />
When you get this message, it means one of 2 things:<br />
1. the jail really didn't start:<br />
When a jail doesn't start it usually boils down to a problem in the quad file. Perhaps the path name is wrong (data1 vs data2) or the name of the vn/mdfile is wrong. Once this is corrected, you will need to run the commands from the quad file manually, or you may use <tt>startjail <hostname></tt><br />
<br />
2. the customer has changed their hostname (and not told us) so their jail ''is'' running, just under a different hostname:<br />
On systems with jls, this is easy to rectify. First, get the customer info: <tt>g <hostname></tt><br />
Then look for the customer in jls: <tt>jls | grep <col0XXXX></tt><br />
From there you will see their new hostname- you should update that hostname in the quad file: don't forget to edit it on the <tt>## begin ##</tt> and <tt>## end ##</tt> lines, and in mgmt. <br />
On older systems without jls, this will be harder, you will need to look further to see their hostname- perhaps its in their /etc/rc.conf<br />
<br />
<br />
Once all jails are started, do some spot checks- try to ssh or browse to some customers, just to make sure things are really ok.<br />
<br />
== Adding disk to a 7.x/8.x jail ==<br />
or<br />
== Moving customer to a different drive (md) ==<br />
<br />
NOTE: this doesn’t apply to mx2 which uses gvinum. Use same procedure as 6.x<br />
NOTE: if you unmount before mdconfig, re-mdconfig (attach) then unmount then mdconfig -u again <br />
<br />
<br />
(parts to change/customize are <tt><span style="color:red">red</span></tt>)<br />
<br />
<br />
If someone wants more disk space, there’s a paste for it, send it to them (explains about downtime, etc).<br />
<br />
1. Figure out the space avail from <tt>js</tt>. Ideally, you want to put the customers new space on a different partition (and create the new md on the new partition). <br />
<br />
<br />
2. make a mental note of how much space they're currently using<br />
<br />
<br />
3. <tt>jailkill <hostname></tt> <br />
<br />
<br />
4. Umount it (including their devfs) but leave the md config’d (so if you use stopjail, you will have to re-mdconfig it)<br />
<br />
<br />
5. <tt>g <customerID></tt> to get the info (IP/cust#) needed to feed the new mdfile and mount name, and to see the current md device. <br />
<br />
<br />
6a. When there's enough room to place new system on an alternate, or the same drive:<br />
USE CAUTION not to overwrite (touch, mdconfig) existing md!!<br><br />
<tt>touch /mnt/data<span style="color:red">3/69.55.234.66-col01334</span><br><br />
mdconfig -a -t vnode -s 10g -f /mnt/data3/69.55.234.66-col01334 -u 97<br><br />
newfs /dev/md97</tt><br />
<br />
<br />
Optional- if new space is on a different drive, move the mount point directory AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
<tt>mount /dev/md<span style="color:red">97</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<br />
confirm you are mounted to /dev/md<span style="color:red">97</span> and space is correct:<br><br />
<tt>df .</tt> <br />
<br />
<br />
do the dump and pipe directly to restore:<br><br />
<tt>dump -0a -f - /dev/md<span style="color:red">1</span> | restore -r -f - <br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use <tt>df</tt> to confirm the restored data size matches the original usage figure<br />
<br />
<br />
md-unconfig old system:<br><br />
<tt>mdconfig -d -u <span style="color:red">1</span></tt><br />
<br />
<br />
archive old mdfile. <br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.237.26-col00241</span> /mnt/data<span style="color:red">1/old-col00241-mdfile-noarchive-20091211</span></tt><br />
<br />
<br />
edit quad (vq1) to point to new (/mnt/data<span style="color:red">3</span>) location AND new md number (md<span style="color:red">97</span>)<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
6b. When there's not enough room on an alternate partition, or on the same drive...but there is enough room if you were to remove the existing customer's space:<br />
<br />
<br />
mount backup nfs mounts:<br><br />
<tt>mbm</tt><br> <br />
(run <tt>df</tt> to confirm backup mounts are mounted)<br />
<br />
<br />
dump the customer to backup2 or backup1<br><br />
<tt>dump -0a -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> /dev/md<span style="color:red">1</span></tt><br><br />
(when complete WITHOUT errors, <tt>du</tt> the dump file to confirm it matches size, roughly, with usage)<br />
<br />
<br />
unconfigure and remove old mdfile<br><br />
<tt>mdconfig -d -u <span style="color:red">1</span><br><br />
rm /mnt/data<span style="color:red">1/69.55.237.26-col00241</span></tt><br><br />
(there should now be enough space to recreate your bigger system. If not, run sync a couple times)<br />
<br />
<br />
create the new system (ok to reuse old mdfile and md#):<br><br />
<tt>touch /mnt/data<span style="color:red">1/69.55.234.66-col01334</span><br><br />
mdconfig -a -t vnode -s <span style="color:red">10</span>g -f /mnt/data<span style="color:red">1/69.55.234.66-col01334</span> -u <span style="color:red">1</span><br><br />
newfs /dev/md<span style="color:red">1</span><br><br />
mount /dev/md<span style="color:red">1</span> /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> <br><br />
cd /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<br />
confirm you are mounted to /dev/md<span style="color:red">1</span> and space is correct:<br><br />
<tt>df .</tt> <br />
<br />
<br />
do the restore from the dumpfile on the backup server:<br><br />
<tt>restore -r -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> .<br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
umount nfs:<br><br />
<tt>mbu</tt><br />
<br />
<br />
If md# changed (or mount point), edit quad (<tt>vq1</tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new md number (md<span style="color:red">1</span>)<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
7. update disk (and dir if applicable) in mgmt screen<br />
<br />
<br />
8. update backup list AND move backups, if applicatble<br />
<br />
Ex: <tt>mvbackups <span style="color:red">69.55.237.26-col00241</span> jail<span style="color:red">9</span> data<span style="color:red">3</span></tt><br />
<br />
<br />
9. Optional: archive old mdfile<br />
<br />
<tt>mbm<br><br />
gzip -c old-col01588-mdfile-noarchive-20120329 > /deprecated/old-col01588-mdfile-noarchive-20120329.gz<br><br />
mbu<br><br />
rm old-col01588-mdfile-noarchive-20120329</tt><br />
<br />
== Adding disk to a 6.x jail (gvinum/gconcat) ==<br />
or<br />
== Moving customer to a different drive (gvinum/gconcat) ==<br />
<br />
(parts to change are <span style="color:red">highlighted</span>)<br />
<br />
<br />
If someone wants more disk space, there’s a paste for it, send it to them (explains about downtime, etc).<br />
<br />
<br />
1. Figure out the space avail from [[#js|js]]. Ideally, you want to put the customers new space on a different partition (and create the new md on the new partition). <br />
<br />
<br />
2. make a mental note of how much space they're currently using<br />
<br />
<br />
3. <tt>[[#stopjail|stopjail]] <hostname></tt> <br />
<br />
<br />
4. <tt>[[#g|g]] <customerID></tt> to get the info (IP/cust#) needed to feed the new mount name and existing volume/device. <br />
<br />
<br />
5a. When there's enough room to place new system on an alternate, or the same drive (using only UNUSED - including if it's in use by the system in question - gvinum volumes):<br />
<br />
<br />
Configure the new device:<br><br />
A. for a 2G system (single gvinum volume):<br><br />
<tt>bsdlabel -r -w /dev/gvinum/v<span style="color:red">123</span><br><br />
newfs /dev/gvinum/v<span style="color:red">123</span>a</tt><br><br />
<br />
-or- <br />
<br />
<br />
B. for a >2G system (create a gconcat volume):<br><br />
try to grab a contiguous block of gvinum volumes. gconcat volumes MAY NOT span drives (i.e. you cannot use a gvinum volume from data3 and a volume from data2 in the same gconcat volume). <br />
<br />
<tt>gconcat label <span style="color:red">v82-v84 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84</span><br><br />
bsdlabel -r -w /dev/concat/<span style="color:red">v82-v84</span><br><br />
newfs /dev/concat/<span style="color:red">v82-v84</span>a</tt><br />
<br />
<br />
Other valid gconcat examples:<br><br />
<tt>gconcat label v82-v84v109v112 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84 /dev/gvinum/v109 /dev/gvinum/v112<br><br />
gconcat label v82v83 /dev/gvinum/v82 /dev/gvinum/v83</tt> <br><br />
Note, long names will truncate: v144v145v148-v115 will truncate to v144v145v148-v1 (so you will refer to it as v144v145v148-v1 thereafter)<br />
<br />
<br />
Optional- if new volume is on a different drive, move the mount point directory (get the drive from js output) AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
<br />
confirm you are mounted to the device (<tt>/dev/gvinum/v<span style="color:red">123</span>a</tt> OR <tt>/dev/concat/<span style="color:red">v82-v84</span></tt>) and space is correct:<br><br />
A. <tt>mount /dev/gvinum/v<span style="color:red">123</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br><br />
-or-<br><br />
B. <tt>mount /dev/concat/<span style="color:red">v82-v84</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<tt>cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
df .</tt> <br />
<br />
<br />
do the dump and pipe directly to restore:<br><br />
<tt>dump -0a -f - /dev/gvinum/v<span style="color:red">1</span> | restore -r -f - <br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
edit quad (<tt>vq<span style="color:red">1</span></tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new volume (<tt>/dev/gvinum/v<span style="color:red">123</span>a</tt> or <tt>/dev/concat/<span style="color:red">v82-v84</span>a</tt>) , run <tt>buildsafe</tt><br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
5b. When there's not enough room on an alternate partition, or on the same drive...but there is enough room if you were to remove the existing customer's space (i.e. if you want/need to reuse the existing gvinum volumes and add on more):<br />
<br />
<br />
mount backup nfs mounts:<br><br />
<tt>mbm</tt> <br><br />
(run df to confirm backup mounts are mounted)<br />
<br />
<br />
dump the customer to backup2 or backup1<br><br />
<tt>dump -0a -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> /dev/<span style="color:red">gconcat/v106-v107</span></tt><br><br />
(when complete WITHOUT errors, du the dump file to confirm it matches size, roughly, with usage)<br />
<br />
<br />
unconfigure the old gconcat volume<br><br />
list member gvinum volumes:<br />
<br />
<tt>gconcat list <span style="color:red">v106v107</span></tt><br />
<br />
<br />
Output will resemble:<br><br />
<pre>Geom name: v106v107<br />
State: UP<br />
Status: Total=2, Online=2<br />
Type: AUTOMATIC<br />
ID: 3530663882<br />
Providers:<br />
1. Name: concat/v106v107<br />
Mediasize: 4294966272 (4.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e2<br />
Consumers:<br />
1. Name: gvinum/sd/v106.p0.s0<br />
Mediasize: 2147483648 (2.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e3<br />
Start: 0<br />
End: 2147483136<br />
2. Name: gvinum/sd/v107.p0.s0<br />
Mediasize: 2147483648 (2.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e3<br />
Start: 2147483136<br />
End: 4294966272</pre><br />
<br />
stop volume and clear members<br><br />
<tt>gconcat stop <span style="color:red">v106v107</span><br><br />
gconcat clear <span style="color:red">gvinum/sd/v106.p0.s0 gvinum/sd/v107.p0.s0</span></tt><br />
<br />
<br />
create new device- and its ok to reuse old/former members<br><br />
try to grab a contiguous block of gvinum volumes. gconcat volumes MAY NOT span drives (i.e. you cannot use a gvinum volume from data3 and a volume from data2 in the same gconcat volume). <br><br />
<tt>gconcat label <span style="color:red">v82-v84v106v107 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84 /dev/gvinum/v106 /dev/gvinum/v107</span> <br><br />
bsdlabel -r -w /dev/concat/<span style="color:red">v82-v84v106v107</span><br><br />
newfs /dev/concat/<span style="color:red">v82-v84v106v107</span>a</tt><br />
<br />
<br />
Optional- if new volume is on a different drive, move the mount point directory (get the drive from js output) AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
confirm you are mounted to the device (/dev/concat/<span style="color:red">v82-v84v106v107</span>) and space is correct:<br><br />
<tt>mount /dev/concat/<span style="color:red">v82-v84v106v107</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<tt>cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
df .</tt> <br />
<br />
do the restore from the dumpfile on the backup server:<br><br />
<tt>restore -r -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> .<br><br />
rm restoresymtable</tt><br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
edit quad (<tt>vq<span style="color:red">1</span></tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new volume (<tt>/dev/concat/<span style="color:red">v82-v84v106v107</span>a</tt>), run buildsafe<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
TODO: clean up/clear old gvin/gconcat vol<br />
<br />
<br />
6. update disk (and dir if applicable) in mgmt screen<br />
<br />
<br />
7. update backup list AND move backups, if applicatble<br />
<br />
<br />
Ex: <tt>mvbackups <span style="color:red">69.55.237.26-col00241</span> jail<span style="color:red">9</span> data<span style="color:red">3</span></tt><br />
<br />
<br />
<br />
DEPRECATED - steps to tack on a new gvin to existing gconcat- leads to corrupted fs<br />
bsdlabel -e /dev/concat/v82-v84<br />
<br />
To figure out new size of the c partition, multiply 4194304 by the # of 2G gvinum volumes and subtract the # of 2G volumes:<br />
10G: 4194304 * 5 – 5 = 20971515<br />
8G: 4194304 * 4 – 4 = 16777212<br />
6G: 4194304 * 3 – 3 = 12582909<br />
4G: 4194304 * 2 – 2 = 8388606<br />
<br />
To figure out the new size of the a partition, subtract 16 from the c partition:<br />
10G: 20971515 – 16 = 20971499<br />
8G: 16777212 – 16 = 16777196<br />
6G: 12582909 – 16 = 12582893<br />
4G: 8388606 – 16 = 8388590<br />
<br />
Orig:<br />
8 partitions:<br />
# size offset fstype [fsize bsize bps/cpg]<br />
a: 8388590 16 4.2BSD 2048 16384 28552<br />
c: 8388606 0 unused 0 0 # "raw" part, don't edit<br />
<br />
New:<br />
8 partitions:<br />
# size offset fstype [fsize bsize bps/cpg]<br />
a: 12582893 16 4.2BSD 2048 16384 28552<br />
c: 12582909 0 unused 0 0 # "raw" part, don't edit<br />
<br />
sync; sync<br />
<br />
growfs /dev/concat/v82-v84a<br />
<br />
fsck –fy /dev/concat/v82-v84a<br />
<br />
sync<br />
<br />
fsck –fy /dev/concat/v82-v84a<br />
<br />
(keep running fsck’s till NO errors)<br />
<br />
== Problems un-mounting - and with mount_null’s ==<br />
<br />
If you cannot unmount a filesystem, beacuse it says the filesystem is busy, it is because of three things:<br />
<br />
a) the jail is still running<br />
<br />
b) you are actually in that directory, even though the jail is stopped<br />
<br />
c) there are still dev, null_mount or linprocfs mount points mounted inside that directory.<br />
<br />
d) when trying to umount null_mounts that are really long and you get an error like “No such file or directory”, it’s an OS bug where the dir is truncated. No known fix<br />
<br />
e) there are still files open somewhere inside the dir. Use <tt>fstat | grep <cid></tt> to find the process that has files open<br />
<br />
f) Starting with 6.x, the jail mechanism does a poor job of keeping track of processes running in a jail and if it thinks there are still procs running, it will refuse to umount the disk. If this is happening you should see a low number in the #REF column when you run jls. In this case you ''can'' safely <tt>umount –f</tt> the mount. <br />
<br />
Please note -if you forcibly unmount a (4.x) filesystem that has null_mounts<br />
still mounted in it, the system '''will crash''' within 10-15 mins.<br />
<br />
== Misc jail Items ==<br />
<br />
We are overselling hard drive space on jail2, jail8, jail9, a couple jails on jail17, jail4, jail12 and jail18.<br />
Even though the vn file shows 4G size, it doesn’t actually occupy that amount of space on the disk. So be careful not to fill up drives where we’re overselling – use oversellcheck to confirm you’re not oversold by more than 10G.<br />
There are other truncated jails, they are generally noted in a the file on the root system: /root/truncated<br />
<br />
The act of moving a truncated vn to another system un-does the truncating- the truncated vn is filled with 0’s and it occupies physical disk space for which it’s configured. So, you should use dumpremote to preserve the truncation.<br />
<br />
= FreeBSD VPS Management Tools =<br />
<br />
These files are located in /usr/local/jail/rc.d and /usr/local/jail/bin<br />
<br />
== jailmake ==<br />
<br />
Applies to 7.x+ <br />
On older systems syntax differs, run jailmake once to see.<br />
<br />
Note: this procedure differs on mx2 which is 7.x but still uses gvinum<br />
<br />
# run js to figure out which md’s are in use, which disk has enough space, IP to put it on<br />
# use col00xxx for both hostnames if they don’t give you a hostname<br />
# copy over dir, ip and password to pending customer screen<br />
<br />
<tt>Usage: jailmake IP[,IP] CID disk[1|2|3] md# hostname shorthost ipfw# email [size in GB]</tt><br />
<br />
Ex: <br />
<br />
Jail2# jailmake 69.55.234.66 col01334 3 97 vps.bsd.it vps 1334 fb@bsd.it<br />
<br />
== jailps ==<br />
jailps [hostname]<br />
DEPRECATED FOR jps: displays processes belonging to/running inside a jail. The command<br />
takes one (optional) argument – the hostname of the jail you wish to query. If you don’t <br />
supply an argument, all processes on the machine are listed and grouped by jail. <br />
<br />
== jps ==<br />
jps [hostname]<br />
displays processes belonging to/running inside a jail. The command<br />
takes one (optional) argument – the hostname or ID of the jail you wish to query. <br />
<br />
== jailkill ==<br />
jailkill <hostname><br />
stops all process running in a jail.<br />
<br />
You can also run:<br />
jailkill <JID><br />
<br />
=== problems ===<br />
Occasionally you will hit an issue where jail will not kill off:<br />
<br />
<pre>jail9# jailkill www.domain.com<br />
www.domain.com .. killed: none<br />
jail9#</pre><br />
Because no processes are running under that hostname. You cannot use jailps.pl either:<br />
<br />
<pre>jail9# jailps www.domain.com<br />
www.domain.com doesn’t exist on this server<br />
jail9#</pre><br />
<br />
The reasons for this are usually:<br />
* the jail is no longer running<br />
<br />
* the jail's hostname has changed<br />
In this case, <br />
<br />
>=6.x: run a <tt>jls|grep <jail's IP></tt> to find the correct hostname, then update the quad file, then kill the jail.<br />
<br />
<6.x: the first step is to cat their /etc/rc.conf file to see if you can tell what they set the new hostname to. This very often works. For example:<br />
<br />
cat /mnt/data2/198.78.65.136-col00261-DIR/etc/rc.conf<br />
<br />
But maybe they set the hostname with the hostname command, and the original hostname is still in /etc/rc.conf.<br />
<br />
The welcome email clearly states that they should tell us if they change their hostname, so there is no problem in just emailing them and asking them what they set the new hostname to.<br />
<br />
Once you know the new hostname OR if a customer simply emails to inform you that they have set the hostname to something different, you need to edit the quad and safe files that their system is in to input the new hostname.<br />
<br />
However, if push comes to shove and you cannot find out the hostname from them or from their system, then you need to start doing some detective work.<br />
<br />
The easiest thing to do is run jailps looking for a hostname similar to their original hostname. Or you could get into the /bin/sh shell by running:<br />
<br />
/bin/sh<br />
<br />
and then looking at every hostname of every process:<br />
<br />
for f in `ls /proc` ; do cat /proc/$f/status ; done<br />
<br />
and scanning for a hostname that is either similar to their original hostname, or that you don't see in any of the quad safe files.<br />
<br />
This is very brute force though, and it is possible that catting every file in /proc is dangerous - I don't recommend it. A better thing would be to identify any processes that you know belong to this system – perhaps the reason you are trying to find this system is because they are running something bad - and just catting the status from only that PID.<br />
<br />
Somewhere there’s a jail where there may be 2 systems named www. Look at /etc/rc.conf and make sure they’re both really www. If they are, jailkill www, jailps www to make sure not running. Then immediately restart the other one, as the fqdn (as found from a rev nslookup)<br />
<br />
* on >=6.x the hostname may not yet be hashed:<br />
<pre>jail9 /# jls<br />
JID Hostname Path IP Address(es)<br />
1 bitnet.dgate.org /mnt/data1/69.55.232.50-col02094-DIR 69.55.232.50<br />
2 ns3.hctc.net /mnt/data1/69.55.234.52-col01925-DIR 69.55.234.52<br />
3 bsd1 /mnt/data1/69.55.232.44-col00155-DIR 69.55.232.44<br />
4 let2.bbag.org /mnt/data1/69.55.230.92-col00202-DIR 69.55.230.92<br />
5 post.org /mnt/data2/69.55.232.51-col02095-DIR 69.55.232.51 ...<br />
6 ns2 /mnt/data1/69.55.232.47-col01506-DIR 69.55.232.47 ...<br />
7 arlen.server.net /mnt/data1/69.55.232.52-col01171-DIR 69.55.232.52<br />
8 deskfood.com /mnt/data1/69.55.232.71-col00419-DIR 69.55.232.71<br />
9 mirage.confluentforms.com /mnt/data1/69.55.232.54-col02105-DIR 69.55.232.54 ...<br />
10 beachmember.com /mnt/data1/69.55.232.59-col02107-DIR 69.55.232.59<br />
11 www.agottem.com /mnt/data1/69.55.232.60-col02109-DIR 69.55.232.60<br />
12 sdhobbit.myglance.org /mnt/data1/69.55.236.82-col01708-DIR 69.55.236.82<br />
13 ns1.jnielsen.net /mnt/data1/69.55.234.48-col00204-DIR 69.55.234.48 ...<br />
14 ymt.rollingegg.net /mnt/data2/69.55.236.71-col01678-DIR 69.55.236.71<br />
15 verse.unixlore.net /mnt/data1/69.55.232.58-col02131-DIR 69.55.232.58<br />
16 smcc-mail.org /mnt/data2/69.55.232.68-col02144-DIR 69.55.232.68<br />
17 kasoutsuki.w4jdh.net /mnt/data2/69.55.232.46-col02147-DIR 69.55.232.46<br />
18 dili.thium.net /mnt/data2/69.55.232.80-col01901-DIR 69.55.232.80<br />
20 www.tekmarsis.com /mnt/data2/69.55.232.66-col02155-DIR 69.55.232.66<br />
21 vps.yoxel.net /mnt/data2/69.55.236.67-col01673-DIR 69.55.236.67<br />
22 smitty.twitalertz.com /mnt/data2/69.55.232.84-col02153-DIR 69.55.232.84<br />
23 deliver4.klatha.com /mnt/data2/69.55.232.67-col02160-DIR 69.55.232.67<br />
24 nideffer.com /mnt/data2/69.55.232.65-col00412-DIR 69.55.232.65<br />
25 usa.hanyuan.com /mnt/data2/69.55.232.57-col02163-DIR 69.55.232.57<br />
26 daifuku.ppbh.com /mnt/data2/69.55.236.91-col01720-DIR 69.55.236.91<br />
27 collins.greencape.net /mnt/data2/69.55.232.83-col01294-DIR 69.55.232.83<br />
28 ragebox.com /mnt/data2/69.55.230.104-col01278-DIR 69.55.230.104<br />
29 outside.mt.net /mnt/data2/69.55.232.72-col02166-DIR 69.55.232.72<br />
30 vps.payneful.ca /mnt/data2/69.55.234.98-col01999-DIR 69.55.234.98<br />
31 higgins /mnt/data2/69.55.232.87-col02165-DIR 69.55.232.87 ...<br />
32 ozymandius /mnt/data2/69.55.228.96-col01233-DIR 69.55.228.96<br />
33 trusted.realtors.org /mnt/data2/69.55.238.72-col02170-DIR 69.55.238.72<br />
34 jc1.flanderous.com /mnt/data2/69.55.239.22-col01504-DIR 69.55.239.22<br />
36 guppylog.com /mnt/data2/69.55.238.73-col00036-DIR 69.55.238.73<br />
40 haliohost.com /mnt/data2/69.55.234.41-col01916-DIR 69.55.234.41 ...<br />
41 satyr.jorge.cc /mnt/data1/69.55.232.70-col01963-DIR 69.55.232.70<br />
jail9 /# jailkill satyr.jorge.cc<br />
ERROR: jail_: jail "satyr,jorge,cc" not found<br />
</pre><br />
<br />
Note how it's saying <tt>satyr,jorge,cc</tt> is not found, and not <tt>satyr.jorge.cc</tt>. <br />
<br />
The jail subsystem tracks things using comma-delimited hostnames. That is created every few hours:<br />
<br />
jail9 /# crontab -l<br />
0 0,6,12,18 * * * /usr/local/jail/bin/sync_jail_names<br />
<br />
So if we run this manually:<br />
jail9 /# /usr/local/jail/bin/sync_jail_names<br />
<br />
Then kill the jail:<br />
jail9 /# jailkill satyr.jorge.cc<br />
successfully killed: satyr,jorge,cc</pre><br />
<br />
It worked.<br />
<br />
<br />
If you ever see this when trying to kill a jail:<br />
<br />
<pre># jailkill e-scribe.com<br />
killing JID: 6 hostname: e-scribe.com<br />
3 procs running<br />
3 procs running<br />
3 procs running<br />
3 procs running<br />
...</pre><br />
<br />
<tt>[[#jailkill|jailkill]]</tt> probably got lost trying to kill off the jail. Just ctrl-c the jailkill process, then run a jailps on the hostname, and <tt>kill -9</tt> any process which is still running. Keep running jailps and <tt>kill -9</tt> till all processes are gone.<br />
<br />
== jailpsall ==<br />
jailpsall<br />
will run a jailps on all jails configured in the quad files (this is different from<br />
jailps with no arguments as it won’t help you find a “hidden” system)<br />
<br />
== jailpsw ==<br />
jailpsw<br />
will run a jailps with an extra -w to provide wider output<br />
<br />
== jt (>=7.x) ==<br />
jt<br />
displays the top 20 processes on the server (the top 20 processes from top) and <br />
which jail owns them. This is very helpful for determining who is doing what when<br />
the server is very busy.<br />
<br />
== jtop (>=7.x) ==<br />
jtop<br />
a wrapper for top displaying processes on the server and which jail owns them. Constantly updates, like top. <br />
<br />
== jtop (<7.x) ==<br />
jtop<br />
displays the top 20 processes on the server (the top 20 processes from top) and <br />
which jail owns them. This is very helpful for determining who is doing what when<br />
the server is very busy.<br />
<br />
== stopjail ==<br />
stopjail <hostname> [1]<br />
this will jailkill, umount and vnconfig –u a jail. If passed an optional 2nd<br />
argument, it will not exit before umounting and un-vnconfig’ing in the event<br />
jailkill returns no processes killed. This is useful if you just want to umount<br />
and vnconfig –u a jail you’ve already killed. It is intelligent in that it won’t <br />
try to umount or vnconfig –u if it’s not necessary.<br />
<br />
== startjail ==<br />
startjail <hostname><br />
this will start vnconfig, mount (including linprocfs and null-mounts), and start a jail.<br />
Essentially, it reads the jail’s relevant block from the right quad file and executes it.<br />
It is intelligent in that it won’t try to mount or vnconfig if it’s not necessary.<br />
<br />
== jpid ==<br />
jpid <pid><br />
displays information about a process – including which jail owns it.<br />
It’s the equivalent of running cat /proc/<pid>/status<br />
<br />
== canceljail ==<br />
canceljail <hostname> [1]<br />
this will stop a jail (the equivalent of stopjail), check for backups (offer to remove them <br />
from the backup server and the backup.config), rename the vnfile, remove the dir, and <br />
edit quad/safe. If passed an optional 2nd argument, it will not exit upon failing to kill<br />
and processes owned by the jail. This is useful if you just want to cancel a jail which <br />
is already stopped.<br />
<br />
== jls ==<br />
jls [-v]<br />
Lists all jails running:<br />
<pre>JID #REF IP Address Hostname Path<br />
101 135 69.55.224.148 mail.pc9.org /mnt/data2/69.55.224.148-col01034-DIR</pre><br />
<br />
#REF is the number of references or procs(?) running<br />
<br />
Running with -v will give you all IPs assigned to each jail (7.2 up)<br />
<pre>JID #REF Hostname Path IP Address(es)<br />
101 139 mail.pc9.org /mnt/data2/69.55.224.148-col01034-DIR 69.55.224.14869.55.234.85</pre><br />
<br />
== startalljails ==<br />
startalljails<br />
7.2+ only. This will parse through quad1 and start all jails. It utilizes lockfiles so it won’t try to start a jail more than once- therefore multiple instances can be running in parallel without fear of starting a jail twice. If a jail startup gets stuck, you can ^C without fear of killing the script. IMPORTANT- before running startalljails you should make sure you ran preboot once as it will clear out all the lockfiles and enable startalljails to work properly.<br />
<br />
== aaccheck.sh ==<br />
aaccheck.sh<br />
displayes the output of container list and task list from aaccli<br />
<br />
== backup ==<br />
backup<br />
backup script called nightly to update jail scripts and do customer backups<br />
<br />
== buildsafe ==<br />
buildsafe<br />
creates safe files based on quads (automatically removing the fsck’s). This will destructively overwrite safe files<br />
<br />
== checkload.pl ==<br />
checkload.pl<br />
this was intended to be setup as a cronjob to watch processes on a jail when the load <br />
rises above a certain level. Not currently in use.<br />
<br />
== checkprio.pl ==<br />
checkprio.pl<br />
will look for any process (other than the current shell’s csh, sh, sshd procs) with a non-normal priority and normalize it<br />
<br />
== diskusagemon == <br />
diskusagemon <mount point> <1k blocks><br />
watches a mount point’s disk use, when it reaches the level specified in the 2nd argument,<br />
it exits. This is useful when doing a restore and you want to be paged as it’s nearing completion.<br />
Best used as: <tt>diskusagemon /asd/asd 1234; pagexxx</tt><br />
<br />
== dumprestore ==<br />
dumprestore <dumpfile><br />
this is a perl expect script which automatically enters ‘1’ and ‘y’. It seems to cause restore to fail<br />
to set owner permissions on large restores.<br />
<br />
== g ==<br />
g <search><br />
greps the quad/safe files for the given search parameter<br />
<br />
== gather.pl ==<br />
gather.pl<br />
gathers up data about jails configured and writes to a file. Used for audits against the db<br />
<br />
== gb ==<br />
gb <search><br />
greps backup.config for the given search parameter<br />
<br />
== gbg ==<br />
gbg <search><br />
greps backup.config for the given search parameter and presents just the directories (for clean pasting)<br />
<br />
== ipfwbackup ==<br />
ipfwbackup<br />
writes ipfw traffic count data to a logfile<br />
<br />
== ipfwreset ==<br />
ipfwreset<br />
writes ipfw traffic count data to a logfile and resets counters to 0<br />
<br />
== js ==<br />
js<br />
output varies by OS version, but generally provides information about the base jail:<br />
- which vn’s are in use<br />
- disk usage<br />
- info about the contents of quads<br />
- the # of inodes represented by the jails contained in the group (133.2 in the example below), and how many jails per data mount, as well as subtotals<br />
- ips bound to the base machine but not in use by a jail<br />
- free gvinum volumes, or unused vn’s or used md’s<br />
<br />
<pre>/usr/local/jail/rc.d/quad1:<br />
/mnt/data1 133.2 (1)<br />
/mnt/data2 1040.5 (7)<br />
total 1173.7 (8)<br />
/usr/local/jail/rc.d/quad2:<br />
/mnt/data1 983.4 (6)<br />
total 983.4 (6)<br />
/usr/local/jail/rc.d/quad3:<br />
/mnt/data1 693.4 (4)<br />
/mnt/data2 371.6 (3)<br />
total 1065 (7)<br />
/usr/local/jail/rc.d/quad4:<br />
/mnt/data1 466.6 (3)<br />
/mnt/data2 882.2 (5)<br />
total 1348.8 (8)<br />
/mnt/data1: 2276.6 (14)<br />
/mnt/data2: 2294.3 (15)<br />
<br />
Available IPs:<br />
69.55.230.11 69.55.230.13 69.55.228.200<br />
<br />
Available volumes:<br />
v78 /mnt/data2 2G<br />
v79 /mnt/data2 2G<br />
v80 /mnt/data2 2G</pre> <br />
<br />
== load.pl ==<br />
load.pl<br />
feeds info to load mrtg - executed by inetd.<br />
<br />
== makevirginjail ==<br />
makevirginjail<br />
Only on some systems, makes an empty jail (doesn't do restore step)<br />
<br />
== mb == <br />
mb <mount|umount><br />
(nfs) mounts and umounts dirs to backup2. Shortcuts are mbm and mbu to mount and unmount. <br />
<br />
== notify.sh ==<br />
notify.sh<br />
emails reboot@johncompanies.com – intended to be called at boot time to alert us to a machine which panics and reboots and isn’t caught by bb or castle.<br />
<br />
== orphanedbackupwatch ==<br />
orphanedbackupwatch<br />
looks for directories on backup2 which aren’t configured in backup.config and offers to delete them<br />
<br />
== postboot ==<br />
postboot<br />
to be run after a machine reboot and quad/safe’s are done executing. It will:<br />
* do chmod 666 on each jail’s /dev/null<br />
* add ipfw counts<br />
* run jailpsall (so you can see if a configured jail isn’t running)<br />
<br />
== preboot ==<br />
preboot<br />
to be run before running quad/safe – checks for misconfigurations: <br />
* a jail configured in a quad but not a safe<br />
* a jail is listed more than once in a quad<br />
* the ip assigned to a jail isn’t configured on the machine<br />
* alias numbering skips in the rc.conf (resulting in the above)<br />
* orphaned vnfile's that aren't mentioned in a quad/safe<br />
* ip mismatches between dir/vnfile name and the jail’s ip<br />
* dir/vnfiles's in quad/safe that don’t exist <br />
<br />
== quadanalyze.pl ==<br />
quadanalyze.pl<br />
called by js, produces the info (seen above with js explanation) about the contents of quad (inode count, # of jails, etc.)<br />
<br />
== rsync.backup ==<br />
rsync.backup<br />
does customer backups (relies on backup.config)<br />
<br />
== taskdone ==<br />
taskdone<br />
when called will email support@johncompanies.com with the hostname of the machine from which it was executed as the subject<br />
<br />
== topten ==<br />
topten<br />
summarizes the top 10 traffic users (called by ipfwreset)<br />
<br />
== trafficgather.pl ==<br />
trafficgather.pl [yy-mm]<br />
sends a traffic usage summary by jail to support@johncomapnies.com and payments@johncompanies.com. Optional arguments are year and month (must be in the past). If not passed, assumes last month. Relies on traffic logs created by ipfwreset and ipfwbackup<br />
<br />
== trafficwatch.pl ==<br />
trafficwatch.pl<br />
checks traffic usage and emails support@johncomapnies.com when a jail reaches the warning level (35G) and the limit (40G). We really aren’t using this anymore now that we have netflow.<br />
<br />
== trafstats ==<br />
trafstats<br />
writes ipfw traffic usage info by jail to a file called jc_traffic_dump in each jail’s / dir<br />
<br />
== truncate_jailmake ==<br />
truncate_jailmake<br />
a version of jailmake which creates truncated vnfiles.<br />
<br />
== vb ==<br />
vb<br />
the equivalent of: <tt>vi /usr/local/jail/bin/backup.config</tt><br />
<br />
== vs (freebsd) ==<br />
vs<n><br />
the equivalent of: <tt>vi /usr/local/jail/rc.d/safe<n></tt><br />
<br />
vq<n><br />
the equivalent of: vi /usr/local/jail/rc.d/quad<n><br />
<br />
== dumpremote ==<br />
dumpremote <user@machine> </remote/location/file-dump> <vnX><br />
ex: dumpremote user@10.1.4.117 /mnt/data3/remote.echoditto.com-dump 7<br />
this will dump a vn filesystem to a remote machine and location<br />
<br />
== oversellcheck ==<br />
oversellcheck<br />
displays how much a disk is oversold or undersold taking into account truncated vn files. Only for use on 4.x systems<br />
<br />
== mvbackups (freebsd) ==<br />
mvbackups <dir> (1.1.1.1-col00001-DIR) <target_machine> (jail1) <target_dir> (data1)<br />
moves backups from one location to another on the backup server, and provides you with option to remove entries from current backup.config, and simple paste command to add the config to backup.config on the target server<br />
<br />
== jailnice ==<br />
jailnice <hostname><br />
applies <tt>renice 19 [PID]</tt> and <tt>rtprio 31 –[PID]</tt> to each process in the given jail<br />
<br />
== dumpremoterestore ==<br />
dumpremoterestore <device> <ip of target machine> <dir on target machine><br />
ex: dumpremoterestore /dev/vn51 10.1.4.118 /mnt/data2/69.55.239.45-col00688-DIR<br />
dumps a device and restores it to a directory on a remote machine. Requires that you enable root ssh on the <br />
remote machine.<br />
<br />
== psj ==<br />
psj<br />
shows just the procs running on the base system – a ps auxw but without jail’d procs present<br />
<br />
== perc5iraidchk ==<br />
perc5iraidchk<br />
checks for degraded arrays on Dell 2950 systems with Perc5/6 controllers<br />
<br />
== perc4eraidchk ==<br />
perc4eraidchk<br />
checks for degraded arrays on Dell 2850 systems with Perc4e/Di controllers<br />
<br />
= Virtuozzo VPS =<br />
<br />
== Making new customer VE (vm) ==<br />
<br />
This applies only to new virts >= 4.x<br />
<br />
grab ip from ipmap (if opened from the pending cust screen it should take you to the right block). You can also run vzlist -a to see what block is in use, generally. Try to find an IP that's in the same block of class C IP's already on the box.<br />
<br />
1. confirm ip you want to use isn’t in use via Mgmt. -> IP Map in management screens<br />
<br />
2. put CT on whichever partition has more space<br />
<br />
3. vm CID IP hostname /vz[1|2] email[,email] template ( <LB|LBP|LS|LM|LMP|LP> [disk] | <gb_disk/mb_ram/gb_burst> ) <br />
vm col00009 69.55.230.238 centos.testdave.com /vz1 dsmith@n2.net centos-6-x86_64 LM<br />
<br />
4. copy veid, dir, ip and password to pending customer screen. activate customer<br />
<br />
<br />
== Making new customer VE (vemakexxx) ==<br />
<br />
This applies to older virts with old templates. This should probably not be used at all anymore.<br />
<br />
<br />
1. look thru hist for ip<br />
<br />
2. confirm ip you want to use isn’t in use via Mgmt. -> IP Map in management screens<br />
<br />
3. put ve on whichever partition has more space<br />
vemakerh9 <veid> <ip> <hostname> <mount> <email> [gb disk]; <256|384|512> <veid><br />
vemakerh9 866 69.55.226.109 ngentu.com /vz1 ayo@ngantu.com,asd@asd.com 5; 256 866<br />
<br />
4. copy (veid), dir, and ip to pending customer screen (pass set to p455agfa)<br />
<br />
<br />
<br />
Note: We use VEID (Virtual Environment ID) and CTID (Container ID) interchangably. Similarly, VE and CT. They mean the same thing.<br />
VZPP = VirtuoZzo Power Panel (the control panel for each CT)<br />
<br />
All linux systems exist in /vz, /vz1 or /vz2 - since each linux machine holds roughly 60-90 customers, there will be roughly 30-45 in each partition.<br />
<br />
The actual filesystem of the system in question is in:<br />
<br />
/vz(1-2)/private/(VEID)<br />
<br />
Where VEID is the identifier for that system - an all-numeric string larger than 100.<br />
<br />
The actual mounted and running systems are in the corresponding:<br />
<br />
/vz(1-2)/root/(VEID)<br />
<br />
But we rarely interact with any system from this mount point.<br />
<br />
You should never need to touch the root portion of their system – however you can traverse their filesystem by going to <tt>/vz(1-2)/private/(VEID)/root</tt> (<tt>/vz(1-2)/private/(VEID)/fs/root</tt> on 4.x systems) the root of their filesystem is in that directory, and their entire system is underneath that.<br />
<br />
Every VE has a startup script in <tt>/etc/sysconfig/vz-scripts</tt> (which is symlinked as <tt>/vzconf</tt> on all systems) - the VE startup script is simply named <tt>(VEID).conf</tt> - it contains all the system parameters for that VE:<br />
<br />
<pre># Configuration file generated by vzsplit for 60 VE<br />
# on HN with total amount of physical mem 2011 Mb<br />
<br />
VERSION="2"<br />
CLASSID="2"<br />
<br />
ONBOOT="yes"<br />
<br />
KMEMSIZE="8100000:8200000"<br />
LOCKEDPAGES="322:322"<br />
PRIVVMPAGES="610000:615000"<br />
SHMPAGES="33000:34500"<br />
NUMPROC="410:415"<br />
PHYSPAGES="0:2147483647"<br />
VMGUARPAGES="13019:2147483647"<br />
OOMGUARPAGES="13019:2147483647"<br />
NUMTCPSOCK="1210:1215"<br />
NUMFLOCK="107:117"<br />
NUMPTY="19:19"<br />
NUMSIGINFO="274:274"<br />
TCPSNDBUF="1800000:1900000"<br />
TCPRCVBUF="1800000:1900000"<br />
OTHERSOCKBUF="900000:950000"<br />
DGRAMRCVBUF="200000:200000"<br />
NUMOTHERSOCK="650:660"<br />
DCACHE="786432:818029"<br />
NUMFILE="7500:7600"<br />
AVNUMPROC="51:51"<br />
IPTENTRIES="155:155"<br />
DISKSPACE="4194304:4613734"<br />
DISKINODES="400000:420000"<br />
CPUUNITS="1412"<br />
QUOTAUGIDLIMIT="2000"<br />
VE_ROOT="/vz1/root/636"<br />
VE_PRIVATE="/vz1/private/636"<br />
NAMESERVER="69.55.225.225 69.55.230.3"<br />
OSTEMPLATE="vzredhat-7.3/20030305"<br />
VE_TYPE="regular"<br />
IP_ADDRESS="69.55.225.229"<br />
HOSTNAME="textengine.net"</pre><br />
<br />
<br />
As you can see, the hostname is set here, the disk space is set here, the number of inodes, the number of files that can be open, the number of tcp sockets, etc. - all are set here.<br />
<br />
In fact, everything that can be set on this customer system is set in this conf file.<br />
<br />
<br />
All interaction with the customer system is done with the VEID. You start the system by running:<br />
<br />
vzctl start 999<br />
<br />
You stop it by running:<br />
<br />
vzctl stop 999<br />
<br />
You execute commands in it by running:<br />
<br />
vzctl exec 999 df -k<br />
<br />
You enter into it, via a root-shell backdoor with:<br />
<br />
vzctl enter 999<br />
<br />
and you set parameters for the system, while it is still running, with:<br />
<br />
vzctl set 999 --diskspace 6100000:6200000 --save<br />
<br />
<tt>vzctl</tt> is the most commonly used command - we have aliased <tt>v</tt> to <tt>vzctl</tt> since we use it so often. We’ll continue to use <tt>vzctl</tt> in our examples, but feel free to use just <tt>v</tt>.<br />
<br />
Let's say the user wants more diskspace. You can cat their conf file and see:<br />
<br />
DISKSPACE="4194304:4613734"<br />
<br />
So right now they have 4gigs of space. You can then change it to 6 with:<br />
<br />
vzctl set 999 --diskspace 6100000:6200000 --save<br />
<br />
IMPORTANT: all issuances of the vzctl set command need to end with <tt>–save</tt> - if they don't, the setting will be set, but it will not be saved to the conf file, and they will not have those settings next time they boot.<br />
<br />
All of the tunables in the conf file can be set with the vzctl set command. Note that in the conf file, and on the vzctl set command line, we always issue two numbers seperated by a colon - that is because we are setting the hard and soft limits. Always set the hard limit slightly above the soft limit, as you see it is in the conf file for all those settings.<br />
<br />
There are also things you can set with `<tt>vzctl set</tt>` that are not in the conf file as settings, per se. For instance, you can add IPs:<br />
<br />
vzctl set 999 --ipadd 10.10.10.10 --save<br />
<br />
or multiple IPs:<br />
<br />
vzctl set 999 --ipadd 10.10.10.10 --ipadd 10.10.20.30 --save<br />
<br />
or change the hostname:<br />
<br />
vzctl set 999 --hostname www.example.com --save<br />
<br />
You can even set the nameservers:<br />
<br />
vzctl set 999 --nameserver 198.78.66.4 --nameserver 198.78.70.180 --save<br />
<br />
Although you probably will never do that.<br />
<br />
You can disable a VPS from being started (by VZPP or reboot) (4.x):<br />
<br />
vzctl set 999 --disabled yes --save <br />
<br />
You can disable a VPS from being started (by VZPP or reboot) (<=3.x):<br />
<br />
vzctl set 999 --onboot=no --save <br />
<br />
You can disable a VPS from using his control panel:<br />
<br />
vzctl set 999 --offline_management=no --save <br />
<br />
You can suspend a VPS, so it can be resumed in the same state it was in when it was stopped (4.x):<br />
<br />
vzctl suspend 999<br />
<br />
and to resume it:<br />
<br />
vzctl resume 999<br />
<br />
to see who owns process:<br />
vzpid <PID><br />
<br />
to mount up an unmounted ve:<br />
vzctl mount 827<br />
<br />
<br />
To see network stats for CT's:<br />
<pre># vznetstat<br />
VEID Net.Class Output(bytes) Input(bytes)<br />
-----------------------------------------------<br />
24218 1 484M 39M<br />
24245 1 463M 143M<br />
2451 1 2224M 265M<br />
2454 1 2616M 385M<br />
4149 1 125M 68M<br />
418 1 1560M 34M<br />
472 1 1219M 315M<br />
726 1 628M 317M<br />
763 1 223M 82M<br />
771 1 4234M 437M<br />
-----------------------------------------------<br />
Total: 13780M 2090M</pre><br />
<br />
<br />
One thing that sometimes comes up on older systems that we created with smaller defaults is that the system would run out of inodes. The user will email and say they cannot create any more files or grow any files larger, but they will also say that they are not out of diskspace ... they are running:<br />
<br />
df -k<br />
<br />
and seeing how much space is free - and they are not out of space. They are most likely out of inodes - which they would see by running:<br />
<br />
df -i<br />
<br />
So, the first thing you should do is enter their system with:<br />
<br />
vzctl enter 999<br />
<br />
and run: <tt>df -i</tt><br />
<br />
to confirm your theory. Then exit their system. Then simply cat their conf file and see what their inodes are set to (probably 200000:200000, since that was the old default on the older systems) and run:<br />
<br />
vzctl set 999 --diskinodes 400000:400000 --save<br />
<br />
If they are not out of inodes, then a good possibility is that they have maxed out their numfile configuration variable, which controls how many files they can have in their system. The current default is 7500 (which nobody has ever hit), but the old default was as low as 2000, so you would run something like:<br />
<br />
vzctl set 999 --numfile 7500:7500 --save<br />
<br />
----<br />
<br />
You cannot start or stop a VE if your pwd is its private (/vz/private/999) or root (/vz/root/999) directories, or anywhere below them.<br />
<br />
== Recovering from a crash (linux) ==<br />
<br />
=== Diagnose whether you have a crash ===<br />
The most important thing is to get the machine and all ve’s back up as soon as possible. Note the time, you’ll need to create a crash log entry (Mgmt. -> Reference -> CrashLog). The first thing to do is head over to the [[Screen#Screen_Organization|serial console screen]] and see if there’s any kernel error messages output. Try to copy any messages (or just a sample of repeating messages) you see into the notes section of the crash log – these will also likely need to be sent to virtuozzo for interpretation. If the messages are spewing too fast, hit ^O + H to start a screen log dump which you can ob1182.pts-38.bb serve after the machine is rebooted. Additionally, if the machine is responsive, you can get a trace to send to virtuozzo by hooking up a kvm and entering these 3 sequences:<br />
<pre>alt+print screen+m<br />
alt+print screen+p<br />
alt+print screen+t</pre><br />
<br />
If there are no messages, the machine may just be really busy- wait a bit (5-10min) to see if it comes back. If it's still pinging, odds are its very busy. If it doesn't come back, or the messages indicate a fatal error, you will need to proceed with a power cycle (ctrl+alt+del will not work).<br />
<br />
=== Power cycle the server ===<br />
If this machine is not a Dell 2950 with a [[DRAC/RMM#DRAC|DRAC card]] (i.e. if you can’t ssh into the DRAC card and issue racadm serveraction hardreset, then you will need someone at the data center to power the macine off, wait 30 sec, then turn it back on. Make sure to re-attach via console (<tt>tip virtxx</tt>) immediately after power down. <br />
<br />
=== (Re)attach to the console ===<br />
Stay on the console the entire time during boot. As the BIOS posts- look out for the RAID card output- does everything look healthy? The output may be scrambled, look for "DEGRADED" or "FAILED". Once the OS starts booting you will be disconnected (dropped back to the shell on the console server) a couple times during the boot up. The reason you want to quickly re-attach is two-fold: 1. If you don’t reattach quickly then you won’t get any console output, 2. you want to be attached before the server ''potentially'' starts (an extensive) fsck. If you attach after the fsck begins, you’ll have seen no indication it started an fsck and the server will appear frozen during startup- no output, no response. <br />
<br />
=== Start containers/VE's/VPSs ===<br />
When the machine begins to start VE’s, it’s safe to leave the console and login via ssh. All virts should be set to auto start all the VEs after a crash. Further, most (newer) virts are set to “fastboot” it’s VE’s (to find out, do:<br />
grep -i fast /etc/sysconfig/vz <br />
and look for <tt>VZFASTBOOT=yes</tt>). If this was set prior to the machine’s crash (setting it after the machine boots will not have any effect until the vz service is restarted) it will start each ve as fast as possible, in serial, then go thru each VE (serially), shutting it down running a vzquota (disk usage) check, then bringing it back up. The benefit is that all VE’s are brought up quickly (within 15min or so depending on the #), the downside is a customer watching closely will notice 2 outages – 1st the machine crash, 2nd their quota check (which will be a much shorter downtime- on the order of a few minutes). <br />
<br />
Where “fastboot” is not set to yes (i.e on quar1), vz will start them consecutively, checking the quotas one at a time, and the 60th VE may not start until an hour or two later - this is not acceptable.<br />
<br />
The good news is, if you run vzctl start for a VE that is already started, you will simply get an error: <tt>VE is already started</tt>. Further, if you attempt to vzctl start a VE that is in the process of being started, you will simply get an error: unable to lock VE. So, there is no danger in simply running scripts to start smaller sets of VEs. If the system is not autostarting, then there is no issue, and even if it does, when it conflicts, one process (yours or the autostart) will lose, and just move on to the next one.<br />
<br />
A script has been written to assist with ve starts: [[#startvirt.pl|startvirt.pl]] which will start 6 ve’s at once until there are no more left. If startvirt.pl is used on a system where “fastboot” was on, it will circumvent the fastboot for ve’s started by startvirt.pl – they will go through the complete quota check before starting- therefore this is not advisable when a system has crashed. When a system is booted cleanly, and there's no need for vzquota checks, then startvirt.pl is safe and advisable to run.<br />
<br />
=== Make sure all containers are running ===<br />
You can quickly get a feel for how many ve’s are started by running:<br />
<br />
<pre>[root@virt4 log]# vs<br />
VEID 16066 exist mounted running<br />
VEID 16067 exist mounted running<br />
VEID 4102 exist mounted running<br />
VEID 4112 exist mounted running<br />
VEID 4116 exist mounted running<br />
VEID 4122 exist mounted running<br />
VEID 4123 exist mounted running<br />
VEID 4124 exist mounted running<br />
VEID 4132 exist mounted running<br />
VEID 4148 exist mounted running<br />
VEID 4151 exist mounted running<br />
VEID 4155 exist mounted running<br />
VEID 42 exist mounted running<br />
VEID 432 exist mounted running<br />
VEID 434 exist mounted running<br />
VEID 442 exist mounted running<br />
VEID 450 exist mounted running<br />
VEID 452 exist mounted running<br />
VEID 453 exist mounted running<br />
VEID 454 exist mounted running<br />
VEID 462 exist mounted running<br />
VEID 463 exist mounted running<br />
VEID 464 exist mounted running<br />
VEID 465 exist mounted running<br />
VEID 477 exist mounted running<br />
VEID 484 exist mounted running<br />
VEID 486 exist mounted running<br />
VEID 490 exist mounted running</pre><br />
<br />
So to see how many ve’s have started:<br />
<pre>[root@virt11 root]# vs | grep running | wc -l<br />
39</pre><br />
<br />
And to see how many haven’t:<br />
<pre>[root@virt11 root]# vs | grep down | wc -l<br />
0</pre><br />
<br />
And how many we should have running:<br />
<pre>[root@virt11 root]# vs | wc -l<br />
39</pre><br />
<br />
Another tool you can use to see which ve’s have started, among other things is [[#vzstat|vzstat]]. It will give you CPU, memory, and other stats on each ve and the overall system. It’s a good thing to watch as ve’s are starting (note the VENum parameter, it will tell you how many have started):<br />
<br />
<pre>4:37pm, up 3 days, 5:31, 1 user, load average: 1.57, 1.68, 1.79<br />
VENum 40, procs 1705: running 2, sleeping 1694, unint 0, zombie 9, stopped 0<br />
CPU [ OK ]: VEs 57%, VE0 0%, user 8%, sys 7%, idle 85%, lat(ms) 412/2<br />
Mem [ OK ]: total 6057MB, free 9MB/54MB (low/high), lat(ms) 0/0<br />
Swap [ OK ]: tot 6142MB, free 4953MB, in 0.000MB/s, out 0.000MB/s<br />
Net [ OK ]: tot: in 0.043MB/s 402pkt/s, out 0.382MB/s 4116pkt/s<br />
Disks [ OK ]: in 0.002MB/s, out 0.000MB/s<br />
<br />
VEID ST %VM %KM PROC CPU SOCK FCNT MLAT IP<br />
1 OK 1.0/17 0.0/0.4 0/32/256 0.0/0.5 39/1256 0 9 69.55.227.152<br />
21 OK 1.3/39 0.1/0.2 0/46/410 0.2/2.8 23/1860 0 6 69.55.239.60<br />
133 OK 3.1/39 0.1/0.3 1/34/410 6.3/2.8 98/1860 0 0 69.55.227.147<br />
263 OK 2.3/39 0.1/0.2 0/56/410 0.3/2.8 34/1860 0 1 69.55.237.74<br />
456 OK 17/39 0.1/0.2 0/100/410 0.1/2.8 48/1860 0 11 69.55.236.65<br />
476 OK 0.6/39 0.0/0.2 0/33/410 0.1/2.8 96/1860 0 10 69.55.227.151<br />
524 OK 1.8/39 0.1/0.2 0/33/410 0.0/2.8 28/1860 0 0 69.55.227.153<br />
594 OK 3.1/39 0.1/0.2 0/45/410 0.0/2.8 87/1860 0 1 69.55.239.40<br />
670 OK 7.7/39 0.2/0.3 0/98/410 0.0/2.8 64/1860 0 216 69.55.225.136<br />
691 OK 2.0/39 0.1/0.2 0/31/410 0.0/0.7 25/1860 0 1 69.55.234.96<br />
744 OK 0.1/17 0.0/0.5 0/10/410 0.0/0.7 7/1860 0 6 69.55.224.253<br />
755 OK 1.1/39 0.0/0.2 0/27/410 0.0/2.8 33/1860 0 0 192.168.1.4<br />
835 OK 1.1/39 0.0/0.2 0/19/410 0.0/2.8 5/1860 0 0 69.55.227.134<br />
856 OK 0.3/39 0.0/0.2 0/13/410 0.0/2.8 16/1860 0 0 69.55.227.137<br />
936 OK 3.2/52 0.2/0.4 0/75/410 0.2/0.7 69/1910 0 8 69.55.224.181<br />
1020 OK 3.9/39 0.1/0.2 0/60/410 0.1/0.7 55/1860 0 8 69.55.227.52<br />
1027 OK 0.3/39 0.0/0.2 0/14/410 0.0/2.8 17/1860 0 0 69.55.227.83<br />
1029 OK 1.9/39 0.1/0.2 0/48/410 0.2/2.8 25/1860 0 5 69.55.227.85<br />
1032 OK 12/39 0.1/0.4 0/80/410 0.0/2.8 41/1860 0 8 69.55.227.90</pre><br />
<br />
When you are all done, you will want to make sure that all the VEs really did get started, run vs one more time.<br />
<br />
Note the time all ve’s are back up and enter that into and save the crash log entry.<br />
<br />
Occasionally, a ve will not start automatically. The most common reason for a ve not to come up normally is the ve was at it’s disk limit before the crash, and will not start since they’re over the limit. To overcome this, set the disk space to current usage level (the system will give this to you when it fails to start), start the ve, then re-set the disk space back to the prior level. Lastly, contact the customer to let them know they’re out of disk (or allocate more disk if they're entitled to more).<br />
<br />
== Hitting performance barriers and fixing them ==<br />
<br />
There are multiple modes virtuozzo offers to allocate resources to a ve. We utilize 2: SLM and UBC parameters<br />
On our 4.x systems, we use all SLM – it’s simpler to manage and understand. There are a few systems on virt19/18 that may also use SLM. Everything else uses UBC. <br />
You can tell a SLM ve by:<br />
<br />
SLMMODE="all"<br />
<br />
in their conf file. <br />
<br />
TODO: detail SLM modes and parameters.<br />
<br />
If someone is in SLM mode and they hit memory resource limits, they simply need to upgrade to more memory.<br />
<br />
The following applies to everyone else (UBC).<br />
<br />
Customers will often email and say that they are getting out of memory errors - a common one is "cannot fork" ... basically, anytime you see something odd like this, it means they are hitting one of their limits that is in place in their conf file.<br />
<br />
The conf file, however, simply shows their limits - how do we know what they are currently at ?<br />
<br />
The answer is a file called v - this file contains the current status (and peaks) of their performance settings, and also counts how many times they have hit the barrier. The output of the file looks like this:<br />
<br />
<pre>764: kmemsize 384113 898185 8100000 8200000 0<br />
lockedpages 0 0 322 322 0<br />
privvmpages 1292 7108 610000 615000 0<br />
shmpages 270 528 33000 34500 0<br />
dummy 0 0 0 0 0<br />
numproc 8 23 410 415 0<br />
physpages 48 5624 0 2147483647 0<br />
vmguarpages 0 0 13019 2147483647 0<br />
oomguarpages 641 6389 13019 2147483647 0<br />
numtcpsock 3 21 1210 1215 0<br />
numflock 1 3 107 117 0<br />
numpty 0 2 19 19 0<br />
numsiginfo 0 4 274 274 0<br />
tcpsndbuf 0 80928 1800000 1900000 0 <br />
tcprcvbuf 0 108976 1800000 1900000 0<br />
othersockbuf 2224 37568 900000 950000 0<br />
dgramrcvbuf 0 4272 200000 200000 0<br />
numothersock 3 9 650 660 0<br />
dcachesize 53922 100320 786432 818029 0<br />
numfile 161 382 7500 7600 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
numiptent 4 4 155 155 0</pre><br />
<br />
The first column is the name of the counter in question - the same names we saw in the systems conf file. The second column is the _current_ value of that counter, the third column is the max that that counter has ever risen to, the fourth column is the soft limit, and the fifth column is the hard limit (which is the same as the numbers in that systems conf file).<br />
<br />
The sixth number is the failcount - how many times the current usage has risen to hit the barrier. It will increase as soon as the current usage hits the soft limit.<br />
<br />
The problem with /proc/user_beancounters is that it actually contains that set of data for every running VE - so you can't just cat /proc/user_beancounters - it is too long and you get info for every other running system.<br />
<br />
You can vzctl enter the system and run:<br />
<br />
vzctl enter 9999<br />
cat /proc/user_beancounters<br />
<br />
inside their system, and you will just see the stats for their particular system, but entering their system every time you want to see it is combersome.<br />
<br />
So, I wrote a simple script called "vzs" which simply greps for the VEID, and spits out the next 20 or so lines (however many lines there are in the output, I forget) after it. For instance:<br />
<br />
<pre># vzs 765:<br />
765: kmemsize 2007936 2562780 8100000 8200000 0<br />
lockedpages 0 8 322 322 0<br />
privvmpages 26925 71126 610000 615000 0<br />
shmpages 16654 16750 33000 34500 0<br />
dummy 0 0 0 0 0<br />
numproc 41 57 410 415 0<br />
physpages 1794 49160 0 2147483647 0<br />
vmguarpages 0 0 13019 2147483647 0<br />
oomguarpages 4780 51270 13019 2147483647 0<br />
numtcpsock 23 37 1210 1215 0<br />
numflock 17 39 107 117 0<br />
numpty 1 3 19 19 0<br />
numsiginfo 0 6 274 274 0<br />
tcpsndbuf 22240 333600 1800000 1900000 0<br />
tcprcvbuf 0 222656 1800000 1900000 0<br />
othersockbuf 104528 414944 900000 950000 0<br />
dgramrcvbuf 0 4448 200000 200000 0<br />
numothersock 73 105 650 660 0<br />
dcachesize 247038 309111 786432 818029 0<br />
numfile 904 1231 7500 7600 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
numiptent 4 4 155 155 0</pre><br />
<br />
That showed us just the portion of /proc/user_beancounters for system 765.<br />
<br />
When you run the vzs command, always add a : after the VEID.<br />
<br />
So, if a customer complains about some out of memory errors, or no more files, or no more ptys, or just has an unspecific complain about processes dying, etc., the very first thing you need to do is check their beancounters with vzs. Usually you will spot an item that has a high failcount and needs to be upped.<br />
<br />
At that point you could simply up the counter with `vzctl set`. Generally pick a number 10-20% higher than the old one, and make the hard limit slightly larger than the the soft limit. However our systems now come in several levels and those levels have more/different memory allocations. If someone is complaining about something other than a memory limit (pty, numiptent, numflock), it’s generally safe to increase it, at least to the same level as what’s in the /vzconf/4unlimited file on the newest virt. If someone is hitting a memory limit, first make sure they are given what they deserve:<br />
<br />
(refer to mgmt -> payments -> packages)<br />
<br />
To set those levels, you use the [[#setmem|setmem]] command. <br />
<br />
The alternate (DEPRECATED) method would be to use one of 3 commands:<br />
256 <veid><br />
300 <veid><br />
384 <veid><br />
512 <veid><br />
<br />
If the levels were not right (you’d run vzs <veid> before and after to see the effect) tell the customer they’ve been adjusted and be done with it. If the levels were right, tell the customer they must upgrade to a higher package, tell them how to see level (control panel) and that they can reboot their system to escape this lockup contidion.<br />
<br />
Customers can also complain that their site is totally unreachable, or complain that it is down ... if the underlying machine is up, and all seems well, you may notice in the beancounters that network-specific counters are failing - such as numtcpsock, tcpsndbuf or tcprcvbuf. This will keep them from talking on the network and make it seem like their system is down. Again, just up the limits and things should be fine.<br />
<br />
On virts 1-4, you should first look at the default settings for that item on a later virt, such as virt 8 - we have increased the defaults a lot since the early machines. So, if you are going to up a counter on virt2, instead of upping it by 10-20%, instead up it to the new default that you see on virt8.<br />
<br />
== Moving a VE to another virt (migrate/migrateonline) ==<br />
<br />
This will take a while to complete - and it is best to do this at night when the load is light on both machines.<br />
<br />
There are different methods for this, depending on which version of virtuozzo is installed on the src. and dst. virt. <br />
To check which version is running: <br />
[root@virt12 private]# cat /etc/virtuozzo-release<br />
Virtuozzo release 2.6.0<br />
<br />
Ok, let's say that the VE is 1212, and vital stats are:<br />
<pre>[root@virt12 sbin]# vc 1212<br />
VE_ROOT="/vz1/root/1212"<br />
VE_PRIVATE="/vz1/private/1212"<br />
OSTEMPLATE="fedora-core-2/20040903"<br />
IP_ADDRESS="69.55.229.84"<br />
TEMPLATES="devel-fc2/20040903 php-fc2/20040813 mysql-fc2/20040812 postgresql-fc2/20040813 mod_perl-fc2/20040812 mod_ssl-fc2/20040811 jre-fc2/20040823 jdk-fc2/20040823 mailman-fc2/20040823 analog-fc2/20040824 proftpd-fc2/20040818 tomcat-fc2/20040823 usermin-fc2/20040909 webmin-fc2/20040909 uw-imap-fc2/20040830 phpBB-fc2/20040831 spamassassin-fc2/20040910 PostNuke-fc2/20040824 sl-webalizer-fc2/20040<br />
818"<br />
<br />
[root@virt12 sbin]# vzctl exec 1212 df -h<br />
Filesystem Size Used Avail Use% Mounted on<br />
/dev/vzfs 4.0G 405M 3.7G 10% /</pre><br />
<br />
From this you can see that he’s using (and will minimally need free on the dst server) ~400MB, and he’s running on a Fedora 2 template, version 20040903. He’s also got a bunch of other templates installed. It’s is '''vital''' that '''all''' these templates exist on the dst system. To confirm that, on the dst system run:<br />
<br />
For < 3.0:<br />
<br />
<pre>[root@virt14 private]# vzpkgls | grep fc2<br />
devel-fc2 20040903<br />
PostNuke-fc2 20040824<br />
analog-fc2 20040824<br />
awstats-fc2 20040824<br />
bbClone-fc2 20040824<br />
jdk-fc2 20040823<br />
jre-fc2 20040823<br />
mailman-fc2 20040823<br />
mod_frontpage-fc2 20040816<br />
mod_perl-fc2 20040812<br />
mod_ssl-fc2 20040811<br />
mysql-fc2 20040812<br />
openwebmail-fc2 20040817<br />
php-fc2 20040813<br />
phpBB-fc2 20040831<br />
postgresql-fc2 20040813<br />
proftpd-fc2 20040818<br />
sl-webalizer-fc2 20040818<br />
spamassassin-fc2 20040910<br />
tomcat-fc2 20040823<br />
usermin-fc2 20040909<br />
uw-imap-fc2 20040830<br />
webmin-fc2 20040909<br />
[root@virt14 private]# vzpkgls | grep fedora<br />
fedora-core-1 20040121 20040818<br />
fedora-core-devel-1 20040121 20040818<br />
fedora-core-2 20040903<br />
[root@virt14 private]#</pre><br />
<br />
For these older systems, you can simply match up the date on the template. <br />
<br />
For >= 3.0:<br />
<br />
<pre>[root@virt19 /vz2/private]# vzpkg list<br />
centos-5-x86 2008-01-07 22:05:57<br />
centos-5-x86 devel<br />
centos-5-x86 jre<br />
centos-5-x86 jsdk<br />
centos-5-x86 mod_perl<br />
centos-5-x86 mod_ssl<br />
centos-5-x86 mysql<br />
centos-5-x86 php<br />
centos-5-x86 plesk9<br />
centos-5-x86 plesk9-antivirus<br />
centos-5-x86 plesk9-api<br />
centos-5-x86 plesk9-atmail<br />
centos-5-x86 plesk9-backup<br />
centos-5-x86 plesk9-horde<br />
centos-5-x86 plesk9-mailman<br />
centos-5-x86 plesk9-mod-bw<br />
centos-5-x86 plesk9-postfix<br />
centos-5-x86 plesk9-ppwse<br />
centos-5-x86 plesk9-psa-firewall<br />
centos-5-x86 plesk9-psa-vpn<br />
centos-5-x86 plesk9-psa-fileserver<br />
centos-5-x86 plesk9-qmail<br />
centos-5-x86 plesk9-sb-publish<br />
centos-5-x86 plesk9-vault<br />
centos-5-x86 plesk9-vault-most-popular<br />
centos-5-x86 plesk9-watchdog</pre><br />
<br />
On these newer systems, it's difficult to tell whether the template on the dst matches exactly the src. Just cause a centos-5-x86 is listed on both servers doesn't mean all the same packages are there on the dst. To truly know, you must perform a sample rsync:<br />
<br />
rsync -avn /vz/template/centos/5/x86/ root@10.1.4.61:/vz/template/centos/5/x86/<br />
<br />
if you see a ton of output from the dry run command, then clearly there are some differences. You may opt to let the rsync complete (without running in dry run mode) the only downside is you've now used up more space on the dst and also the centos template will be a mess with old and new data- it will be difficult if not impossible to undo (if someday we wanted to reclaim the space).<br />
<br />
If you choose to merge templates, you should closely inspect the dry run output. You should also take care to exclude anything in the /config directory. For example:<br />
<br />
rsync -av -e ssh --stats --exclude=x86/config /vz/template/ubuntu/10.04/ root@10.1.4.62:/vz/template/ubuntu/10.04/<br />
<br />
Which will avoid this directory and contents:<br />
<pre>[root@virt11 /vz2/private]# ls /vz/template/ubuntu/10.04/x86/config*<br />
app os</pre><br />
<br />
This is important to avoid since the config may differ on the destination and we are really only interested in making sure the pacakges are there, not overwriting a newer config with an older one.<br />
<br />
If the dst system was missing a template, you have 2 choices: <br />
# put the missing template on the dst system. 2 choices here: <br />
## Install the template from rpm (found under backup2: /mnt/data4/vzrpms/distro/) or <br />
## rsync over the template (found under /vz/template) - see above<br />
# put the ve on a system which has all the proper templates<br />
<br />
=== pre-seeding a migration ===<br />
<br />
When migrating a customer (or when doing many) depending on how much data you have to transfer, it can take some time. Further, it can be difficult to gauge when a migration will complete or how long it will take. To help speed up the process and get a better idea about how long it will take you can pre-transfer a customer's data to the destination server. If done correctly, vzmigrate will see the pre-transferred data and pick up where you left off, having much less to transfer (just changed/new files). <br />
<br />
We believe vzmigrate uses rsync to do it's transfer. Therefore not only can you use rsync to do a pre-seed, you can also run rsync to see what is causing a repeatedly-failing vzmigrate to fail. <br />
<br />
There's no magic to a pre-seed, you just need to make sure it's named correctly.<br />
<br />
Given:<br />
<br />
source: /vz1/private/1234<br />
<br />
and you want to migrate to /vz2 on the target system, your rsync would look like:<br />
<br />
rsync -av /vz1/private/1234/ root@x.x.x.x:/vz2/private/1234.migrated/<br />
<br />
After running that successful rsync, the ensuing migrateonline (or migrate) will take much less time to complete- depending on the # of files to be analyzed and the # of changed files. In any case, it'll be much much faster than had you just started the migration from scratch.<br />
<br />
Further, as we discuss elsewhere in this topic, a failed migration can be moved from <tt>/vz/private/1234</tt> to <tt>/vz/private/1234.migrated</tt> on the destination if you want to restart a failed migration. This should '''only''' be done if the migration failed and the CT is not running on the destination HN.<br />
<br />
=== migrateonline intructions: src >=3.x -> dst>=3.x ===<br />
<br />
A script called [[#migrateonline|migrateonline]] was written to handle this kind of move. It is basically a wrapper for <tt>vzmigrate</tt> – vzmigrate is a util to seamlessly- as no no reboot of the ve necessary- move a ve from one host to another. This wrapper was initially written cause vz virtuozzo version 2.6.0 has a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables, causing problems when the ve was started up on the dst system. [[#migrate|migrate]] mitigates that. Since it makes multiple ssh connections to the dst virt, it’s a good idea to put the pub key for the src virt in the authorized_keys file on the dst virt. In addition, migrateonline emails ve owners when their migration starts and stops. For this to happen they need to put email addresses (on a single line, space delimited) in a file on their system: /migrate_notify. If the optional target dir is not specified, the ve will be moved to the same private/root location as it was on the src virt. Note: <tt>migrate</tt> is equivalent to <tt>migrateonline</tt>, but will <tt>migrate</tt> a ve AND restart it in the process.<br />
<br />
<pre>[root@virt12 sbin]# migrateonline<br />
usage: /usr/local/sbin/migrateonline <ip of node migrating to> <veid> [target dir: vz | vz1 | vz2]<br />
[root@virt12 sbin]# migrateonline 10.1.4.64 1212 vz<br />
starting to migrate 1212 at Sat Mar 26 22:40:38 PST 2005<br />
Turning off offline_management<br />
Saved parameters for VE 1212<br />
migrating with no start on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
clearing the arp cache<br />
now going to 10.1.4.64 and clear cache and starting<br />
starting it<br />
Delete port redirection<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
finished migrating 1212 at Sat Mar 26 22 22:52:01 PST 2005<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine.<br />
<br />
If they had backups, use the mvbackups command to move their backups to the new server:<br />
<br />
mvbackups 1212 virt14 vz<br />
<br />
Rename the ve<br />
<br />
[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/migrated-1212<br />
<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/old-1212-migrated-20120404-noarchive<br />
<br />
Update the customer’s systems in mgmt to reflect the new path and server.<br />
<br />
<br />
IF migrateonline does not work, you can try again using simply migrate- this will result in a brief reboot for the ve.<br />
Before you try again, make sure of a few things:<br />
<br />
Depending on where in the migration died, there may be partial data on the dst system in 1 of 2 places:<br />
(given the example above)<br />
<br />
/vz/private/1212<br />
<br />
or <br />
<br />
/vz/private/1212.migrated<br />
<br />
before you run migrate again, you'll want to rename so that all data is in <br />
1212.migrated:<br />
<br />
mv /vz/private/1212 /vz/private/1212.migrated<br />
<br />
this way, it will pick up where it left off and transfer only new files.<br />
<br />
Likewise, if you want to speed up a migration, you can pre-seed the dst as follows:<br />
<br />
[root@virt12 sbin]# rsync -avSH /vz/private/1212/ root@10.1.4.64:/vz/private/1212.migrated/<br />
<br />
then when you run migrate or migrateonline, it will only need to move the changed files- the migration will complete quickly<br />
<br />
=== migrateonline/migrate failures (migrate manually) ===<br />
<br />
Lets say for whatever reason the migration fails. If it fails with [[#migrateonline|migrateonline]], you should try [[#migrate|migrate]] (which will reboot the customer, so notify them ahead of time).<br />
<br />
You may want to run a [[#pre-seeding_a_migration|pre-seed]] rsync to see if you can find the problem. On older virts, we've seen this problem due to a large logfile (which you can find and encourage the customer to remove/compress):<br />
for f in `find / -size +1048576k`; do ls -lh $f; done<br />
<br />
You may also see migration failing due to quota issues.<br />
<br />
You can try to resolve by copying any quota file into the file you need:<br />
<br />
cp /var/vzquota/quota.1 /var/vzquota/quota.xxx<br />
<br />
If it complains about quota running you should then be able to stop it<br />
<br />
vzquota off xxxx<br />
<br />
If all else fails, migrate to a new VEID<br />
i.e. 1234 becomes 12341<br />
<br />
<br />
If the rsync or [[#migrate|migrate]] fails, you can always move someone manually:<br />
<br />
1. stop ve: <br><br />
v stop 1234<br />
<br />
2. copy over data<br><br />
rsync -avSH /vz/private/1234/ root@1.1.1.1:/vzX/private/1234/<br />
<br />
NOTE: if you've previously seeded the data (run rsync while the VE was up/running), and this is a subsequent rsync, make sure the last rsync you do (while the VE is not running, has the --delete option in the rsync)<br />
<br />
3. copy over conf<br><br />
scp /vzconf/1234.conf root@1.1.1.1:/vzconf<br />
<br />
4. on dst, edit the conf to reflect the right vzX dir<br><br />
vi /vzconf/1234.conf<br />
<br />
5. on src remove the IPs<br><br />
ipdel 1234 2.2.2.2 3.3.3.3<br />
<br />
6. on dst add IPs <br><br />
ipadd 1234 2.2.2.2 3.3.3.3<br />
<br />
7. on dst, start ve: <br><br />
v start 1324<br />
<br />
8. cancel, then archive ve on src per above instrs.<br />
<br />
=== migrate src=2.6.0 -> dst>=2.6.0, or mass-migration with customer notify ===<br />
<br />
A script called <tt>migrate</tt> was written to handle this kind of move. It is basically a wrapper for vzmigrate – vzmigrate is a util to seamlessly move a ve from one host to another. This wrapper was initially written cause vz virtuozzo version 2.6.0 has a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables, causing problems when the ve was started up on the dst system. migrate mitigates that. Since it makes multiple ssh connections to the dst virt, it’s a good idea to put the pub key for the src virt in the authorized_keys file on the dst virt. In addition, migrate emails ve owners when their migration starts and stops. For this to happen they need to put email addresses (on a single line, space delimited) in a file on their system: /migrate_notify. If the optional target dir is not specified, the ve will be moved to the same private/root location as it was on the src virt. Note: migrateonline is equivalent to migrate, but will migrate a ve from one 2.6 '''kernel''' machine to another 2.6 kernel machine without restarting the ve.<br />
<br />
<pre>[root@virt12 sbin]# migrate<br />
usage: /usr/local/sbin/migrate <ip of node migrating to> <veid> [target dir: vz | vz1 | vz2]<br />
[root@virt12 sbin]# migrate 10.1.4.64 1212 vz<br />
starting to migrate 1212 at Sat Mar 26 22:40:38 PST 2005<br />
Turning off offline_management<br />
Saved parameters for VE 1212<br />
migrating with no start on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
clearing the arp cache<br />
now going to 10.1.4.64 and clear cache and starting<br />
starting it<br />
Delete port redirection<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
finished migrating 1212 at Sat Mar 26 22 22:52:01 PST 2005<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine (backup2 or mail).<br />
Cancel the ve (first we have to rename things which migrate changed so cancelve will find them):<br />
<br />
[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/1212.conf<br />
<br />
On 2.6.1 you’ll also have to move the private area:<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/1212<br />
<br />
<pre>[root@virt12 sbin]# cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, [[#cancelve|cancelve]] would offer to remove them. You want to say '''no''' to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
To move everyone off a system, you’d do:<br />
for f in `vl`; do migrate <ip> $f; done<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
=== vzmigrate: src=2.6.1 -> dst>=2.6.0 ===<br />
<br />
This version of vzmigrate works properly with regard to handling ips. It will not notify ve owners of moves as in the above example. Other than that it’s essentially the same.<br />
<br />
<pre>[root@virt12 sbin]# vzmigrate 10.1.4.64 -r no 1212:1212:/vz/private/1212:/vz/root/1212<br />
migrating on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine (backup2 or mail).<br />
Cancel the ve (first we have to rename things which vzmigrate changed so cancelve will find them):<br />
<br />
<pre>[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/1212.conf<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/1212<br />
<br />
[root@virt12 sbin]# cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, <tt>cancelve</tt> would offer to remove them. You want to say no to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
=== src=2.5.x ===<br />
<br />
First, go to the private dir:<br />
<br />
cd /vz1/private/<br />
<br />
Stop the VE - make sure it stops totally cleanly.<br />
<br />
vzctl stop 1212<br />
<br />
Then you’d use vemove - a script written to copy over the config, create tarballs of the ve’s data on the destination virt, and cancel the ve on the source system (in this example we’re going to put a ve that was in /vz1/private on the src virt, in /vz/private on the dst virt):<br />
<br />
<pre>[root@virt12 sbin]# vemove<br />
ERROR: Usage: vemove veid target_ip target_path_dir<br />
[root@virt12 sbin]# vemove 1212 10.1.4.64 /vz/private/1212<br />
tar cfpP - 1212 --ignore-failed-read | (ssh -2 -c arcfour 10.1.4.64 "split - -b 1024m /vz/private/1212.tar" )<br />
scp /vzconf/1212.conf 10.1.4.64:/vzconf<br />
cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, cancelve would offer to remove them. You want to say no to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
When you are done, go to /vz/private on the dst virt you will have files like this:<br />
<br />
<pre>1212.taraa<br />
1212.tarab<br />
1212.tarac</pre><br />
<br />
Each one 1024m (or less, for the last one) in size.<br />
<br />
on the dst server and run:<br />
<br />
cat 1212.tar?? | tar xpPBf -<br />
<br />
and after 20 mins or so it will be totally untarred. Now since the conf<br />
file is already there, you can go ahead and start the system.<br />
<br />
vzctl start 1212<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
NOTE: you MUST tar the system up using the virtuozzo version of tar that<br />
is on all the virt systems, and further you MUST untar the tarball with<br />
the virtuozzo tar, using these options: `<tt>tar xpPBf -</tt>`<br />
<br />
If you tar up an entire VE and move it to a non-virtuozzo machine, that is<br />
ok, and you can untar it there with normal tar commands, but do not untar<br />
it and then repack it with a normal tar and expect it to work - you need<br />
to use virtuozzo tar commands on virtuozzo tarballs to make it work.<br />
<br />
The backups are sort of an exception, since we are just (usually)<br />
restoring user data that was created after we gave them the system, and<br />
therefore has nothing to do with magic symlinks or vz-rpms, etc.<br />
<br />
== Moving a VE on the same virt ==<br />
<br />
Easy way:<br><br />
Scenario 1: ve 123 is to be renamed 1231 and moved from vz1 to vz<br />
<br />
vzmlocal 123:1231:/vz/private/1231:/vz/root/1231<br />
<br />
Scenario 2: ve 123 is to be moved vz1 to vz<br />
<br />
vzmlocal 123:123:/vz/private/123:/vz/root/123<br />
<br />
vzmlocal will reboot the ve at the end of the move<br />
<br />
Manual/old way:<br />
<br />
1) <tt>vzctl stop 123</tt><br><br />
2) <tt>mv /vz1/private/123 /vz/private/.</tt><br><br />
(or cp -a if you want to copy)<br />
3) in <tt>/etc/sysconfig/vz-scripts/123.conf</tt> change value<br><br />
of '<tt>VE_PRIVATE</tt>' variable to point to a new private area location<br />
4) <tt>vzctl start 123</tt><br><br />
5) update backups if needed: <tt>mvbackups 123 virtX virt1 vz</tt><br><br />
6) update management scerens<br><br />
<br />
Notes: a) absolute path to private area is stored in quota file <tt>/var/vzquota/quota.123</tt> - so during first startup quota will be recalculated.<br><br />
b) if you're going to write some script to do a job, you MUST be sure that $VEID won't be expanded to '' in ve config file - ie. you need to escape '$'. Otherwise you might have:<br />
<br />
VE_PRIVATE="/vz/private/"<br />
<br />
in config, and 'vzctl destroy' for this VE ID '''will remove everything under /vz/private/ directory'''.<br />
<br />
== Adding a veth device to a VE ==<br />
<br />
Not totally sure what this is, but a customer asked for it and here's what we did (as instructed by vz support):<br />
<br />
<pre>v set 99 --netif_add eth99 --save<br />
ipdel 99 69.55.230.58<br />
v set 99 --ifname eth99 --ipadd 69.55.230.58 --save<br />
v set 99 --ifname eth99 --gateway 69.55.230.1 --save<br />
<br />
vznetcfg net new veth_net<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
net99 active eth0 veth77.77,veth99.99<br />
veth_net active<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
br0 bridge veth_net<br />
br99 bridge net99<br />
veth99.99 veth net99<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic net99 69.55.227.70/24<br />
<br />
vznetcfg br attach br0 eth0<br />
<br />
(will remove 99 from orig net and move to veth_net)<br />
vznetcfg net addif veth_net veth99.99<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
net99 active<br />
veth_net active eth0 veth77.77,veth99.99<br />
<br />
(delete the old crap)<br />
vznetcfg net del net99<br />
<br />
<br />
Then, to add another device in<br />
<br />
v set 77 --netif_add eth77 --save<br />
ipdel 77 69.55.230.78<br />
v set 77 --ifname eth77 --ipadd 69.55.230.78 --save<br />
v set 77 --ifname eth77 --gateway 69.55.230.1 --save<br />
v set 77 --save --ifname eth77 --network veth_net<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
veth77.77 veth<br />
br0 bridge veth_net<br />
veth99.99 veth veth_net<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic veth_net 69.55.227.70/24<br />
<br />
vznetcfg net addif veth_net veth77.77<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
veth77.77 veth veth_net<br />
br0 bridge veth_net<br />
veth99.99 veth veth_net<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic veth_net 69.55.227.70/24<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
veth_net active eth0 veth77.77,veth99.99<br />
<br />
another example<br />
<br />
v set 1182 --netif_add eth1182 --save<br />
ipdel 1182 69.55.236.217<br />
v set 1182 --ifname eth1182 --ipadd 69.55.236.217 --save<br />
v set 1182 --ifname eth1182 --gateway 69.55.236.1 --save<br />
vznetcfg net addif veth_net veth1182.1182<br />
v set 1182 --save --ifname eth1182 --network veth_net<br />
<br />
<br />
unused/not working commands:<br />
ifconfig veth99.0 0<br />
vznetcfg net list<br />
vznetcfg br new br99 net99<br />
vznetcfg br attach br99 eth0<br />
vznetcfg br show<br />
vznetcfg if list<br />
vznetcfg net addif net99 veth99.99<br />
vznetcfg net addif eth0 net99<br />
<br />
---------<br />
<br />
vznetcfg br new br1182 net1182<br />
<br />
vznetcfg br attach br99 eth0<br />
vznetcfg if list<br />
vznetcfg net addif net99 veth99.99<br />
vznetcfg net addif eth0 net99<br />
<br />
<br />
<br />
vznetcfg net addif eth0 net1182<br />
<br />
<br />
# vznetcfg net addif veth99.0 net99<br />
--- 8< ---<br />
<br />
vznetcfg net new net<br />
# vznetcfg net addif eth0 net99<br />
<br />
<br />
# vzctl set 99 --save --netif_add eth0 (at this stage veth99.0 interface have to appear<br />
on node)<br />
# vzctl set 99 --save --ifname eth0 --ipadd 69.55.230.58 (and probably few more arguments<br />
here - see 'man vzctl')<br />
# vznetcfg net addif veth99.0 net99</pre><br />
<br />
<br />
== Assigning/remove ip from a VE ==<br />
<br />
1. Add or remove ips:<br />
ipdel 1234 1.1.1.1 2.2.2.2<br />
ipadd 1234 1.1.1.1 2.2.2.2<br />
<br />
2. update Mgmt screens<br />
<br />
3. offer to update any DNS we do for them<br />
<br />
4. check to see if we had rules for old IP in firwall<br />
<br />
== Enabling tun device for a ve ==<br />
Note, there’s a command for this: [[#addtun|addtun]]<br />
<br />
For posterity:<br />
Make sure the tun.o module is already loaded before Virtuozzo is started: <br />
lsmod <br />
Allow the VPS to use the TUN/TAP device: <br />
vzctl set 101 --devices c:10:200:rw --save <br />
Create the corresponding device inside the VPS and set the proper permissions: <br />
vzctl exec 101 mkdir -p /dev/net <br />
vzctl exec 101 mknod /dev/net/tun c 10 200 <br />
vzctl exec 101 chmod 600 /dev/net/tun<br />
<br />
== Remaking a system (on same virt) ==<br />
<br />
1. [[#cancelve|cancelve]] (or v destroy x - ONLY if you're POSITIVE no data needs to be saved)<br />
<br />
2. [[#vemake|vemake]] using same veid<br />
<br />
3. [[#mvbackups|mvbackups]] or [[#vb|vb]] (if new mount point)<br />
<br />
4. update mgmt with new dir/ip <br />
<br />
5. update firewall if changed ip<br />
<br />
== Re-initialize quota for a VE ==<br />
<br />
There’s a commamd for this now: [[#clearquota|clearquota]]<br />
<br />
For posterity:<br />
<br />
vzctl stop 1<br />
vzquota drop 1<br />
vzctl start 1<br />
<br />
<br />
== Traffic accounting on linux ==<br />
<br />
DEPRECATED - all tracking is done via bwdb now. This is how we used to track traffic.<br />
<br />
TODO: update for diff versions of vz<br />
<br />
Unlike FreeBSD, where we have to add firewall count rules to the system to count the traffic, on virtuozzo counts the traffic for us. You an see the current traffic stats by running `vznetstat`:<br />
<br />
<pre># vznetstat<br />
VEID Net.Class Output(bytes) Input(bytes)<br />
-----------------------------------------------<br />
24218 1 484M 39M<br />
24245 1 463M 143M<br />
2451 1 2224M 265M<br />
2454 1 2616M 385M<br />
4149 1 125M 68M<br />
418 1 1560M 34M<br />
472 1 1219M 315M<br />
726 1 628M 317M<br />
763 1 223M 82M<br />
771 1 4234M 437M<br />
-----------------------------------------------<br />
Total: 13780M 2090M</pre><br />
<br />
As you can see the VEID is on a line with the in and out bytes. So, we simply run a cron job:<br />
<br />
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /root/vztrafdump.sh<br />
<br />
Just like we do on FreeBSD - this one goes through all the VEs in /vz/private and greps the line from vznetstat that matches them and dumps it in /jc_traffic_dump on their system. Then it does it again for all the VEs in /vz1/private. It is important to note that vznetstat runs only once, and the grepping is done from a temporary file that contains that output - we do this because running vznetstat once for each VE that we read out of /vz/private and /vz1/private would take way too long and be too intensive.<br />
<br />
You do not need to do anything to facilitate this other than make sure that that cron job is running - the vznetstat counters are always running, and any new VEs that are added to the system will be accounted for automatically.<br />
<br />
Traffic resetting no longer works with vz 2.6, so we disable the vztrafdump.sh on those virts.<br />
<br />
== Watchdog script ==<br />
<br />
On some of the older virts, we have a watchdog running that kills procs that are deemed bad per the following:<br />
<br />
/root/watchdog from quar1<br />
<br />
<pre>if echo $line | awk '{print $(NF-3)}' | grep [5-9]...<br />
then<br />
# 50-90%<br />
if echo $line | awk '{print $(NF-1)}' | grep "...:.."<br />
then<br />
# running for > 99min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
if echo $line | awk '{print $(NF-1)}' | grep "....m"<br />
then<br />
# running for > 1000min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
fi<br />
<br />
if echo $line | awk '{print $(NF-3)}' | grep [1-9]...<br />
then<br />
# running for 10-90 percent<br />
if echo $line | awk '{print $NF}' | egrep 'cfusion|counter|vchkpw'<br />
then<br />
<br />
if echo $line | awk '{print $(NF-1)}' | grep "[2-9]:.."<br />
then<br />
# between 2-9min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
elif echo $line | awk '{print $(NF-1)}' | grep "[0-9][0-9]:.."<br />
then<br />
# up to 99min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
fi<br />
fi</pre><br />
<br />
<br />
== Misc Linux Items ==<br />
<br />
We are overselling hard drive space ... when you configure a linux system with a certain amount of disk space (the default is 4gigs) you do not actually use up 4gigs of space on the system. The diskspace setting for a user is simply a cap, and they only use up as much space on the actual disk drive as they are actually using.<br />
<br />
When you create a new linux system, even though there are some 300 RPMs or so installed, if you run `df -k` you will see that the entire 4gig partition is empty - no space is being used. This is because the files in their system are "magic symlinks" to the template for their OS that is in /vz/template - however, any changes to any of those files will "disconnect" them and they will immediately begin using space in their system. Further, any new files uploaded (even if those new files overwrite existing files) will take up space on the partition.<br />
<br />
<br />
if you see this:<br />
<br />
<pre>[root@virt8 root]# vzctl stop 160 ; vzctl start 160<br />
VE is not running<br />
Starting VE ...<br />
VE is unmounted<br />
VE is mounted<br />
Adding IP address(es): 69.55.226.83 69.55.226.84<br />
bash ERROR: Can't change file /etc/sysconfig/network<br />
Deleting IP address(es): 69.55.226.83 69.55.226.84<br />
VE is unmounted<br />
[root@virt8 root]#</pre><br />
<br />
it probably means they no longer have /bin/bash - copy one in for them<br />
<br />
ALSO: another possibility is that they have removed the `ed` RPM from their system - it needs to be reinstalled into their system. But since their system is down, this is tricky ...<br />
<br />
VE startup scripts used by 'vzctl' want package 'ed' to be available inside VE. So if package 'ed' will be enabled in OS template config and OS template itself VE #827 is based on - this error should be fixed.<br />
<br />
yes, it is possible to add RPM to VE while it not running.<br />
Try to do following:<br />
<br />
<pre># cd /vz/template/<OS_template_with_ed_package>/<br />
# vzctl mount 827<br />
# rpm -Uvh --root /vz/root/827 --veid 827 ed-0.2-25.i386.vz.rpm</pre><br />
<br />
Usually theres an error, but its ok<br />
<br />
Note: replace 'ed-0.2-25.i386.vz.rpm' in last command with actual<br />
version of 'ed' package you have.<br />
<br />
----<br />
<br />
So how do I know what template the user has ? cat their conf file and it is listed in there. For example, if the conf file has:<br />
<br />
<pre>[root@virt12 sbin]# vc 1103<br />
…snip…<br />
OSTEMPLATE="debian-3.0/20030822"<br />
TEMPLATES="mod_perl-deb30/20030707 mod_ssl-deb30/20030703 mysql-deb30/20030707 proftpd-deb30/20030703 webmin-deb30/20030823 "<br />
…</pre><br />
<br />
then they are on debian 3.0, all of their system RPMs are in /vz/template/debian-3.0, and they are using version 20030822 of that debian 3.0 template. Also, they’ve also got additional packages installed (mod_perl, mod_ssl, etc). Those are also found under /vz/template<br />
<br />
----<br />
<br />
Edits needed to run java:<br />
<br />
When we first created the VEs, the default setting for privvmpages was 93000:94000 ... which was high enough that most people never had problems ... however, you can;t run java or jdk or tomcat or anything java related with that setting. We have found that by setting privvmpages to 610000:615000 that java runs just fine. That is now the default setting. It is exceedingly rare that anyone needs it higher than that, although we have seen it once or twice.<br />
<br />
Any problems with java at all - the first thing you need to do is see if the failcnt has raised for privvmpages.<br />
<br />
<pre># vzctl start 160<br />
Starting VE ...<br />
vzquota : (error) Quota on syscall for 160: Device or resource busy<br />
Running vzquota on failed for VE 160 [3]</pre><br />
<br />
This is because my pwd is _in_ their private directory - you can't start it until you move out<br />
<br />
People seem to have trouble with php if they are clueless newbies. Here are two common problems/solutions:<br />
<br />
no... but i figured it out myself. problem was the php.ini file that came<br />
vanilla with the account was not configured to work with apache (the<br />
ENGINE directive was set to off).<br />
<br />
everything else seems fine now.<br />
<br />
----<br />
<br />
the problem was in the php.ini file. I noticed that is wasnt showing<br />
the code when it was in an html file so I looked at the php.ini file<br />
and had to change it so it recognized <? tags aswell as <?php tags.<br />
<br />
Also, make sure added to httpd.conf<br />
AddType application/x-httpd-php .php<br />
<br />
----<br />
<br />
You can set the time zone:<br />
<br />
You can change the timezone by doing this:<br />
<br />
ln -sf /usr/share/zoneinfo/<zone> /etc/localtime<br />
<br />
where <zone> is the zone you want in the /usr/share/zoneinfo/ directory.<br />
<br />
----<br />
<br />
Failing shm_open calls:<br />
<br />
first, please check if /dev/shm is mounted inside VE.<br />
'cat /proc/mounts' command should show something like this:<br />
tmpfs /dev/shm tmpfs rw 0 0<br />
<br />
If /dev/shm is not mounted you have 2 ways to solve issue:<br />
1. execute following command inside VE (doesn't require VE reboot):<br />
mount -t tmpfs none /dev/shm<br />
2. add following string to /etc/fstab inside VE and reboot it:<br />
tmpfs /dev/shm tmpfs defaults 0 0<br />
<br />
----<br />
<br />
You can have a mounted but not running ve<br />
Just:<br />
vzctl mount <veid><br />
<br />
----<br />
<br />
When a debian sys can’t get on the network, and you try:<br />
<br />
<pre>vzctl set 1046 --ipadd 69.55.227.117<br />
Adding IP address(es): 69.55.227.117<br />
Failed to bring up lo.<br />
Failed to bring up venet0.</pre><br />
<br />
They probably removed iproute package, which must be the one from swsoft. To restore:<br />
<pre># dpkg -i --veid=1046 --admindir=/vz1/private/1046/root/var/lib/dpkg --instdir=/vz1/private/1046/root/ /vz/template/debian-3.0/iproute_20010824-8_i386.vz.deb<br />
(Reading database ... 16007 files and directories currently installed.)<br />
Preparing to replace iproute 20010824-8 (using .../iproute_20010824-8_i386.vz.deb) ...<br />
Unpacking replacement iproute ...<br />
Setting up iproute (20010824-8) ...</pre><br />
<br />
Then restart their ve<br />
<br />
----<br />
<br />
in a ve i do:<br />
<br />
<pre>cd /<br />
du -h .</pre><br />
<br />
and get: 483M .<br />
<br />
i do:<br />
<br />
<pre>bash-2.05a# df -h<br />
Filesystem Size Used Avail Use% Mounted on<br />
/dev/vzfs 4.0G 2.3G 1.7G 56% /</pre><br />
<br />
how can this be?<br />
<br />
Is it possible that quota file was corrupted somehow? Please try to: <br />
<pre>vzctl stop <VEID><br />
vzquota drop <VEID><br />
vzquota init <VEID><br />
vzctl start <VEID></pre><br />
<br />
----<br />
<br />
How to stop vz from starting after reboot:<br />
<br />
VIRTUOZZO=no <br />
in <br />
/etc/sysconfig/vz<br />
<br />
To start: <br />
service vz start<br />
(after setting VIRTUOZZO=yes in /etc/sysconfig/vz)<br />
<br />
service vz restart will do some kind of 'soft reboot' -- restart all<br />
VPSes and reload modules without rebooting the node<br />
<br />
if you need to shut down all VPSes really really fast, run killall -9 init<br />
<br />
----<br />
<br />
Postfix tip:<br />
<br />
You may want to tweak settings: default_process_limit=10<br />
<br />
= Virtuozzo VPS Management Tools =<br />
<br />
== vm ==<br />
<br />
TODO<br />
<br />
== cancelve ==<br />
cancelve <veid><br />
this will:<br />
* stop a ve<br />
* check for backups (offer to remove them from the backup server <br />
and the backup.config)<br />
* rename the private dir<br />
* check for PTR, provide the commands to reset to default<br />
* and rename the ve’s config<br />
* remind you to remove firewall rules<br />
* remind you to remove DNS entries<br />
<br />
== ipadd ==<br />
ipadd <veid> <ip> [ip] [ip]<br />
add’s ip(s) to a ve<br />
<br />
== ipdel ==<br />
ipdel <veid> <ip> [ip] [ip]<br />
removes ip(s) from a ve<br />
<br />
== vc ==<br />
vc <veid><br />
the equivalent of: <tt>cat /vzconf/<veid>.conf</tt><br />
<br />
== vl ==<br />
vl<br />
will displays a list of ve #’s, 1 per line. (ostensibly to use in a for loop)<br />
<br />
== vp ==<br />
vp <veid><br />
the equivalent of: <tt>vzps auxww –E <veid></tt><br />
<br />
== vpe ==<br />
DEPRECATED<br />
vpe <veid> <br />
this will allow you to do a vp when a ve is running out of control, the equivalent of (deprecated since vp operates outside the VPS): <br />
<pre>vzctl set <veid> --kmemsize 2100000:2200000<br />
vzctl exec <veid> ps auxw<br />
vzctl set <veid> --kmemsize (ve’s orig lvalue):(ve’s orig hvalue)</pre><br />
<br />
== vt ==<br />
vt <veid><br />
the equivalent of: <tt>vztop –E <veid></tt><br />
<br />
== vr ==<br />
vr <veid><br />
the equivalent of: <tt>vzctl stop <veid>; vzctl start <veid></tt><br />
You can run this even if the ve is down - the stop command will just fail<br />
<br />
== vs ==<br />
vs [veid]<br />
displays status (output of <tt>vzctl status <veid></tt>) on each ve configured on the system (as determined by <tt>/vzconf/*.conf</tt>)<br />
If passed an argument, gives the status for just that ve. <br />
A running system looks like:<br />
<tt>VEID 16066 exist mounted running</tt><br />
<br />
A ve which is not running (but does exist) looks like:<br />
<tt>VEID 9990 exist unmounted down</tt><br />
<br />
A ve which is not running and doesn’t exist looks like:<br />
<tt>VEID 421 deleted unmounted down</tt><br />
<br />
== vs2 ==<br />
vs2 [veid]<br />
this is similar to vs in that it displays status (output of <tt>vzctl status <veid></tt>) on each ve,<br />
but the difference is it’s list comes from doing an ls on the data dirs. This was meant to catch <br />
the rare case where a ve configured but exists. <br />
<br />
== vw ==<br />
vw [veid]<br />
displays the output of ‘<tt>w</tt>’ (the equivalent of <tt>vzctl exec <veid> w</tt>) for each configured ve (as determined by <tt>/vzconf/*.conf</tt>). Useful for determing which ve is contributing to a heavily-loaded system.<br />
If passed an argument, gives ‘<tt>w</tt>‘ output for just that ve. <br />
Ex:<br />
<pre>[root@virt2 etc]# vw<br />
134<br />
10:52pm up 79 days, 6:14, 0 users, load average: 0.02, 0.02, 0.00<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT<br />
16027<br />
2:52pm up 7 days, 19:54, 0 users, load average: 0.00, 0.00, 0.00<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT<br />
16055<br />
2:52pm up 79 days, 6:38, 0 users, load average: 0.00, 0.04, 0.07<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT</pre><br />
<br />
== vwe ==<br />
vwe [constraint]<br />
just like <tt>vw</tt>, but takes a constraint as an argument, only show’s ve’s with loads >= the constraint provided. If no constraint is provided, 1 is used by default<br />
<br />
== vzs ==<br />
vzs [veid]<br />
displays the beancounter status for all ve’s, or a particular ve if an argument is passed<br />
<br />
== ve ==<br />
ve <veid><br />
the equivalent of: <tt>vzctl enter <veid></tt><br />
<br />
== vx ==<br />
vx <veid> <command><br />
the equivalent of: <tt>/usr/sbin/vzctl exec <veid> <command></tt><br />
<br />
== dprocs ==<br />
dprocs [count]<br />
a script which outputs a continuous report (or a certain number of reports if an option is passed) of processes stuck in the D state and which VPS’s those procs belong to.<br />
<br />
== setmem ==<br />
setmem <VEID> <256|512|768|1024|2048><br />
adjusts the memory resources for the VE<br />
<br />
== afacheck.sh ==<br />
afacheck.sh<br />
displays the health/status of containers and mirrors on an adaptec card (currently quar1, tempvirt1-2, virt9, virt10)- all other are LSI<br />
<br />
== backup ==<br />
backup<br />
backup script called nightly to update virt scripts and do customer backups<br />
<br />
== checkload.pl ==<br />
checkload.pl<br />
this was intended to be setup as a cronjob to watch processes on a virt when the load <br />
rises above a certain level. Not currently in use.<br />
<br />
== findbackuppigs.pl ==<br />
findbackuppigs.pl<br />
looks for files larger than 50MB which customers have asked us to backup. Emails matches<br />
to linux@johncompanies.com<br />
<br />
== gatherlinux.pl ==<br />
gatherlinux.pl<br />
gathers up data about ve’s configured and writes to a file. Used for audits against the db<br />
<br />
== gb ==<br />
gb <search><br />
greps backup.config for the given search parameter<br />
<br />
== gbg ==<br />
gbg <search><br />
greps backup.config for the given search parameter and presents just the directories (for clean pasting)<br />
<br />
== linuxtrafficgather.pl ==<br />
linuxtrafficgather.pl [yy-mm]<br />
sends a traffic usage summary by ve to support@johncomapnies.com and payments@johncompanies.com.<br />
Optional arguments are year and month (must be in the past). If not passed, assumes last month. Relies on <br />
traffic logs created by netstatreset and netstatbackup<br />
<br />
== linuxtrafficwatch.pl ==<br />
linuxtrafficwatch.pl<br />
checks traffic usage and emails support@johncomapnies.com when a ve reaches the warning level (35G) and<br />
the limit (40G). Works on virtuozzo versions <= 2.6.x. We really aren’t using this anymore now that we have netflow.<br />
<br />
== linuxtrafficwatch2.pl ==<br />
linuxtrafficwatch2.pl<br />
checks traffic usage and emails support@johncomapnies.com when a ve reaches the warning level (35G) and<br />
the limit (40G). Works on virtuozzo version 2.6.x. We really aren’t using this anymore now that we have netflow.<br />
<br />
== load.pl ==<br />
load.pl<br />
feeds info to load mrtg - executed by inetd.<br />
<br />
== mb (linux) ==<br />
mb <mount|umount><br />
(nfs) mounts and umounts dirs to backup2. Shortcuts are mbm and mbu to mount and unmount. <br />
<br />
== migrate ==<br />
migrate <ip of node migrating to> <veid> <target_veid> [target dir: vz | vz1 | vz2]<br />
this is basically a wrapper for <tt>vzmigrate</tt> – vzmigrate is a util to seamlessly move a ve from one host to another. This wrapper was written cause vz virtuozzo version 2.6 had a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables. This script mitigates that. Since it makes multiple ssh connections to the target host, it’s a good idea to put the pub key for the src system in the authorized_keys file on the target host. In addition, it emails ve owners when their migration starts and stops (if they place email addresses in a file on their system: /migrate_notify). To move everyone off a system, you’d do:<br />
for f in `vl`; do migrate <ip> $f; done<br />
<br />
== migrateonline ==<br />
migrateonline <ip of node migrating to> <veid> <target_veid> [target dir: vz | vz1 | vz2]<br />
this is the same as migrate but will migrate a ve in <tt>–online</tt> mode which means it won’t be shut down at the end of the migration. This only works when migrating ve’s between 2 machines running a 2.6 kernel (currently tempvirt1-2. virt16-19, virt12). If you get an error that the machine you’re trying to migrate to has a different CPU or features, etc, then you have to edit the file and add the –f switch to the vzmigrate line- you can basically ignore this kind of warning (but never ignore a warning about missing templates on the destination node). NOTE: This edit (if made to migrateonline) will be overwritten by the base script during each night’s backup.<br />
<br />
== netstatbackup ==<br />
DEPRECATED<br />
netstatbackup <br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.5.x. <br />
<br />
== netstatbackup2 ==<br />
DEPRECATED<br />
netstatbackup2<br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.6.x. <br />
<br />
== netstatreset ==<br />
DEPRECATED<br />
netstatreset<br />
writes traffic count data to a logfile and resets counters to 0. Works on virtuozzo versions 2.5.x <br />
<br />
== netstatreset2 ==<br />
DEPRECATED<br />
netstatreset2<br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.6.x. <br />
<br />
== orphanedbackupwatchlinux ==<br />
orphanedbackupwatchlinux <br />
looks for directories on backup2 which aren’t configured in backup.config and offers to <br />
delete them<br />
<br />
== rsync.backup (linux) ==<br />
rsync.backup<br />
does customer backups (relies on backup.config)<br />
<br />
== startvirt.pl ==<br />
startvirt.pl<br />
forks off start ve commands – keeps 6 running at a time. This is not to be used on systems where fastboot is enabled as it circumvents the benefit of the fastboot. The script will occasionally not exit gracefully and will continue to use up CPU, so it should be watched. Also, don’t exit from the script till you’re sure all ve’s are started – if you do you need to start them manually and may have to free up locks. On some systems, the startvirt script doesn’t exit cleanly and you have to ^C out of it. Be careful though- doing so can leave some VE’s in an odd bootup state and you may need to ‘vr’ them manually. You should check to see which ve’s aren’t running and/or confirm all have started when ^C’ing out of startvirt.<br />
<br />
== taskdone (linux) ==<br />
taskdone<br />
when called will email support@johncompanies.com with the hostname of the machine from which it was <br />
executed as the subject<br />
<br />
== vb (linux) ==<br />
vb<br />
the equivalent of: <tt>vi /usr/local/sbin/backup.config</tt><br />
<br />
== vemakeXX ==<br />
DEPRECATED<br />
vemakerh9 <br />
ve create script for RH9 (see vemake)<br />
<br />
vemakedebian30 <br />
ve create script for debian 3.0 (Woody) (see vemake)<br />
<br />
vemakedebian31 <br />
ve create script for debian 3.1 (Sarge) (see vemake)<br />
<br />
vemakedebian40 <br />
ve create script for debian 4.0 (Etch) (see vemake)<br />
<br />
vemakefedora, vemakefedora2, vemakefedora4, vemakefedora5, vemakefedora6, vemakefedora7<br />
ve create script for fedora core 1, 2, 4, 5, 6, 7 respectively (see vemake)<br />
<br />
vemakecentos3, vemakecentos4<br />
ve create script for centos 3, 4 respectively (see vemake)<br />
<br />
vemakesuse, vemakesuse93, vemakesuse100<br />
ve create script for suse 9.2, 9.3, 10.0 respectively (see vemake)<br />
<br />
vemakeubuntu5, vemakeubuntu606, vemakeubuntu606 vemakeubuntu704<br />
ve create script for ubuntu 5.10, 6.06, 6.10, 7.04 respectively (see vemake)<br />
<br />
== vemove ==<br />
DEPRECATED<br />
vemove <veid> <target_ip> </vz/private/123><br />
this script simplifies the old way of moving ve’s from one system to another - in short moving a ve to or from a virt running virtuozzo < 2.6.x<br />
It’s the equivalent of:<br />
<tt>tar cfpP - <veid> --ignore-failed-read | (ssh -2 -c arcfour <target_ip> "split - -b 1024m </vz/private/123>.tar" )</tt>This should only be used if migrate/vzmigrate can’t be used. <br />
<br />
== vim.watchdog ==<br />
vim.watchdog <br />
looks for and kills procs matching “vi|vim|nano|pine|elm” that are running for a long time & consuming lots of cpu. Works on virtuozzo versions 2.5.x<br />
<br />
== vim.watchdog2 ==<br />
vim.watchdog2<br />
looks for and kills procs matching “vi|vim|nano|pine|elm” that are running for a long time & consuming lots of cpu.<br />
Works on virtuozzo versions 2.6.x.<br />
<br />
== vzmigrate ==<br />
vzmigrate <target_ip> -r no <veid>:[dst veid]:[dst /vzX/private/veid]:[dst /vzX/root/veid]<br />
(this is the raw command “wrapped” by migrate/migrateonline) this will seamlessly move a ve from one host to another. The ve will run for the duration of the migration till the very end when it’s shut down, ip moved and started up on the target system. The filesystem on the src will remain. This should be watched – occasionally the move will timeout and leave the system shut down. If target private and root aren’t specified it just puts it in /vz. Only works when both systems are running virtuozzo 2.6.x<br />
<br />
== vztrafdump.sh ==<br />
DEPRECATED<br />
vztrafdump.sh<br />
writes traffic usage info by ve to a file called jc_traffic_dump in each ve’s / dir. Works on virtuozzo versions <= 2.5.x. <br />
<br />
== vztrafdump2.sh ==<br />
DEPRECATED<br />
vztrafdump2.sh<br />
writes traffic usage info by ve to a file called jc_traffic_dump in each ve’s / dir. Works on virtuozzo versions 2.6.x. <br />
<br />
== addtun ==<br />
addtun <veid><br />
Add’s tun device to ve.<br />
<br />
== bwcap ==<br />
bwcap <veid> <kbps><br />
Ex: <tt>bwcap 1234 512</tt><br />
Caps a VE’s bandwidth to the amount given<br />
<br />
== setdisk ==<br />
setdisk <veid> <diskspace in GB><br />
Ex: <tt>setdisk 1234 5</tt><br />
Gives a VE’s a given amount of disk space<br />
<br />
== vdf ==<br />
vdf <veid> <br />
the equivalent of: <tt>vzctl exec <veid> df –h</tt><br />
<br />
== vdff ==<br />
vdff<br />
runs a (condensed) vdf for all ve’s in your pwd (must be run from /vz/privateN)<br />
<br />
== mvbackups ==<br />
mvbackups <veid> <target_machine> (virt1) <target_dir> (vz1)<br />
moves backups from one location to another on the backup server, and provides you with option to remove entries from current backup.config, and simple paste command to add the config to backup.config on the target server<br />
<br />
== checkquota ==<br />
checkquota<br />
for all the ve’s in the cwd (run from /vz/private, /vz1/private, etc) reports what vz quota says they’re using and what the actual usage is (as reported by du)<br />
<br />
== clearquota ==<br />
clearquota <veid><br />
Recalculates a ve’s quota, prints out the usage before and after. The equivalent of:<br />
<tt>vdf <veid>; v stop <veid>; vzquota drop <veid>; v start <veid>; vdf <veid></tt><br />
<br />
== dprocs ==<br />
dprocs<br />
Sometimes the server’s have a large number of processes get stuck in the D state- this script shows (every 3 secs) which VE’s have D procs, which procs<br />
are stuck and a running average of the top “offenders”<br />
<br />
== vzstat ==<br />
vstat<br />
sort of like top for VZ. sort VEs by CPU usage by pressing 'o' and then 'c' keys<br />
<br />
== stopvirt ==<br />
stopvirt<br />
will stop VEs as fast as it can, 6 at a time. May not exit when complete so you should watch [[#vzstat|vzstat]] in another window.</div>Supporthttps://wiki.jcihosting.com/index.php?title=VPS_Management&diff=1037VPS Management2013-03-11T23:49:14Z<p>Support: /* Making new customer VE (vemakexxx) */</p>
<hr />
<div>= Common Problems =<br />
== Login to any machine without a password ==<br />
<br />
This is possible via the use of ssh keys. The process is thus:<br />
<br />
1. place the public key for your user (root@mail) in the /root/.ssh/authorized_keys file on the server you wish to login to<br />
cat /root/.ssh/id_dsa.pub<br />
(paste that into authorized_keys on the target server). If the file doesn't exist, create it.<br />
<br />
2. enable root login (usually only applies to FreeBSD). Edit the /etc/ssh/sshd_config on the target server and change:<br />
<tt>#PermitRootLogin no</tt><br />
to<br />
<tt>PermitRootLogin yes</tt><br />
<br />
3. Restart the sshd on the target machine. First, find the sshd process: <br />
jailps <hostname> | grep sshd <br />
or <br />
vp <VEID> | grep sshd<br />
<br />
Look for the process resembling:<br />
root 17296 0.0 0.0 5280 1036 ? Ss 2011 4:27 /usr/sbin/sshd <br />
(this is the sshd)<br />
<br />
Not:<br />
root 6270 0.5 0.0 6808 2536 ? Ss 14:33 0:00 sshd: root [priv]<br />
(this is an sshd child- someone already ssh'd in as root)<br />
<br />
Restart the sshd: <br />
kill -1 <PID><br />
<br />
Ex:<br />
kill -1 17296<br />
<br />
You may now ssh in.<br />
<br />
Once you're done, IF you enabled root login, you should repeat steps 2 and 3 to disable root logins.<br />
<br />
== Letting someone in who has locked themselves out (killed sshd, lost pwd) ==<br />
<br />
There are two ways people frequently lock themselves out - either they forget a password, or they kill off sshd somehow.<br />
<br />
These are actually both fairly easy to solve. First, let's say someone kills off their sshd, or somehow mangles /etc/ssh/sshd_config such that it no longer lets them in.<br />
<br />
Their email may be very short, or it may have all sorts of details about how you should fix sshd_config to let them in ... just ignore all of this. They can fix their own mangled sshd. Fixing this is very simple. First, edit the /etc/inetd.conf on their system and uncomment the telnet line:<br />
<br />
telnet stream tcp nowait root /usr/libexec/telnetd telnetd<br />
#telnet stream tcp6 nowait root /usr/libexec/telnetd telnetd<br />
<br />
(just leave the tcp6 version of telnet commented)<br />
<br />
Then, use jailps to list the processes on their system, and find their inetd process. Then simply:<br />
<br />
kill -HUP (pid)<br />
<br />
where (pid) is the PID of their inetd process. Now they have telnet running on their system and they can log in and do whatever they need to do.<br />
<br />
The only complications that could occur are:<br />
<br />
a) their firewall config on our firewall has port 23 blocked, in which case you will need to open that - will be covered in a different lesson.<br />
<br />
b) they are not running inetd, so you can't HUP it. If this happens, edit their /etc/rc.conf, add the inetd_enable="YES" line, and then kill<br />
their jail with /tmp/jailkill.pl - then restart their jail with the jail line from their quad/safe file. Easy.<br />
<br />
If they have forgotten a password,<br />
<br />
On 6.x+ you can reset their password with:<br />
jexec <jailID from jls> passwd root<br />
<br />
Note: the default password for 6.x jails is 8ico2987, for 4.x it is p455agfa<br />
<br />
On 4.x, you need to cd to their etc directory<br />
... for instance:<br />
<br />
cd /mnt/data2/198.78.65.136-col00261-DIR/etc<br />
<br />
and run:<br />
<br />
vipw -d .<br />
<br />
Then paste in these two lines (theres a paste with these):<br />
<br />
root:$1$krszPxhk$xkCepSnz3mIikT3vCtJCt0:0:0::0:0:Charlie &:/root:/bin/csh<br />
user:$1$Mx9p5Npk$QdMU6c8YQqp2FW2M3irEh/:1001:1001::0:0:User &:/home/user:/bin/sh<br />
<br />
overwriting the lines they already have for "user" and "root" - then just tell them that both user and root have been reset to the default password of p455agfa.<br />
<br />
For linux, just passwd inside shell or <br />
vzctl set <veid> --userpasswd root:p455agfa –save<br />
<br />
Starting in 2009 we began giving out randomized passwords for FreeBSD and Linux as the default password. That is stored with each system in Mgmt. You should look for and reset the password to that password in the event of a reset and refer the customer to use their original password from their welcome email- this way we don’t have to send the password again via email (in clear text).<br />
<br />
<br />
== sendmail can’t be contacted from ext ip (only locally) ==<br />
<br />
By default redhat puts this line in sendmail.mc:<br />
<br />
DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')<br />
<br />
which makes it only answer on localhost. Comment it out like:<br />
<br />
dnl DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')<br />
<br />
and then rebuild sendmail.cf with:<br />
<br />
m4 /etc/mail/sendmail.mc > /etc/sendmail.cf<br />
<br />
== virt doesn’t properly let go of ve’s ip(s) when moved to another system ==<br />
<br />
On virtuozzo 2.6 systems, it's been observed that when moving ips from one virt to another that sometimes the routing table will not get updated to reflect the removal of the ip addresses.<br />
<br />
A recent example was a customer that was moving to a new ve on a new virt and the ip addresses were traded between the two ve's. After the trade the two systems were not able to talk to each other. When looking at the routing table for the old system all the ip addresses were still in the routing table as being local, like so:<br />
<br />
<pre>netstat -rn | grep 69.55.225.149<br />
69.55.225.149 0.0.0.0 255.255.255.255 UH 40 0 0 venet0</pre><br />
<br />
This was preventing traffic to the other system from being routed properly.<br />
The solution is to manually delete the route:<br />
<br />
route delete 69.55.225.149 gw 0.0.0.0<br />
<br />
Supposedly, this was fixed in 2.6.1<br />
<br />
== sshd on FreeBSD 6.2 segfaults ==<br />
<br />
First try to reinstall ssh<br />
<br />
<pre>cd /usr/src/secure<br />
cd lib/libssh<br />
make depend && make all install<br />
cd ../../usr.sbin/sshd<br />
make depend && make all install<br />
cd ../../usr.bin/ssh<br />
make depend && make all install</pre><br />
<br />
Failing that, find the library that’s messed up:<br />
<br />
<pre>ldd /usr/sbin/sshd<br />
libssh.so.3 => /usr/lib/libssh.so.3 (0x280a3000) <br />
libutil.so.5 => /lib/libutil.so.5 (0x280d8000) <br />
libz.so.3 => /lib/libz.so.3 (0x280e4000) <br />
libwrap.so.4 => /usr/lib/libwrap.so.4 (0x280f5000) <br />
libpam.so.3 => /usr/lib/libpam.so.3 (0x280fc000) <br />
libbsm.so.1 => /usr/lib/libbsm.so.1 (0x28103000) <br />
libgssapi.so.8 => /usr/lib/libgssapi.so.8 (0x28112000) <br />
libkrb5.so.8 => /usr/lib/libkrb5.so.8 (0x28120000) <br />
libasn1.so.8 => /usr/lib/libasn1.so.8 (0x28154000) <br />
libcom_err.so.3 => /usr/lib/libcom_err.so.3 (0x28175000) <br />
libroken.so.8 => /usr/lib/libroken.so.8 (0x28177000) <br />
libcrypto.so.4 => /lib/libcrypto.so.4 (0x28183000) <br />
libcrypt.so.3 => /lib/libcrypt.so.3 (0x28276000) <br />
libc.so.6 => /lib/libc.so.6 (0x2828e000) <br />
libmd.so.3 => /lib/libmd.so.3 (0x28373000)</pre><br />
<br />
md5 them and compare to other jail hosts or jails running on host<br />
<br />
for libcrypto reinstall:<br />
<pre>/usr/src/crypto<br />
make depend && make all install</pre><br />
<br />
= FreeBSD VPS =<br />
<br />
== Starting jails: Quad/Safe Files ==<br />
<br />
FreeBSD customer systems do not start up automatically at boot time. When one of our freebsd machines boots up, it boots up, and does nothing else. To start jails, we put the commands to start each jail into a shell script(s) and run the script(s). Jail startup is something that needs to be actively monitored, which is why we don’t just run the script automatically. More on monitoring later.<br />
<br />
NOTE: >=7.x we have moved to 1 quad file: <tt>quad1</tt>. Startups are not done by running each quad, but rather [[#startalljails|startalljails]] which relies on the contents of <tt>quad1</tt>. The specifics of this are lower in this article. What follows here applies for pre 7.x systems.<br />
<br />
There are eight files in <tt>/usr/local/jail/rc.d</tt>:<br />
<br />
<pre>jail3# ls /usr/local/jail/rc.d/<br />
quad1 quad2 quad3 quad4 safe1 safe2 safe3 safe4<br />
jail3#</pre><br />
<br />
four quad files and four safe files.<br />
<br />
Each file contains an even number of system startup blocks (total number of jails divided by 4)<br />
<br />
The reason for this is, if we make one large script to startup all the systems at boot time, it will take too long - the first system in the script will start up right after system boot, which is great, but the last system may not start for another 20 minutes.<br />
<br />
Since there is no way to parralelize this during the startup procedure, we simply open four terminals (in screen window 9) and run each script, one in each terminal. This way they all run simultaneously, and the very last system in each startup script gets started in 1/4th the time it would if there was one large file<br />
<br />
The files are generally organized so that quad/safe 1&2 have only jails from disk 1, and quad/safe 3&4 have jails from disk 2. This helps ensure that only 2 fscks on any disk are going on at once. Further, they are balanced so that all quad/safe’s finish executing around the same time. We do this by making sure each quad/safe has a similar number of jails and represents a similar number of inodes (see js).<br />
<br />
The other, very important reason we do it this way, and this is the reason there are quad files and safe files, is that in the event of a system crash, every single vn-backed filesystem that was mounted at the time of system crash needs to be fsck'd. However, fsck'ing takes time, so if we shut the system down gracefully, we don't want to fsck.<br />
<br />
Therefore, we have two sets of scripts - the four quad scripts are identical to the four safe scripts except for the fact that the quad scripts contain fsck commands for each filesystem.<br />
<br />
So, if you shut a system down gracefully, start four terminals and run safe1 in window one, and safe2 in window 2, and so on.<br />
<br />
If you crash, start four terminals (or go to screen window 9) and run quad1 in window one, and quad2 in window 2, and so on.<br />
<br />
Here is a snip of (a 4.x version) quad2 from jail17:<br />
<br />
<pre>vnconfig /dev/vn16 /mnt/data2/69.55.228.7-col00820<br />
fsck -y /dev/vn16<br />
mount /dev/vn16c /mnt/data2/69.55.228.7-col00820-DIR<br />
chmod 0666 /mnt/data2/69.55.228.7-col00820-DIR/dev/null<br />
jail /mnt/data2/69.55.228.7-col00820-DIR mail1.phimail.com 69.55.228.7 /bin/sh /etc/rc<br />
<br />
# moved to data2 col00368<br />
#vnconfig /dev/vn28 /mnt/data2/69.55.236.132-col00368<br />
#fsck -y /dev/vn28<br />
#mount /dev/vn28c /mnt/data2/69.55.236.132-col00368-DIR<br />
#chmod 0666 /mnt/data2/69.55.236.132-col00368-DIR/dev/null<br />
#jail /mnt/data2/69.55.236.132-col00368-DIR limehouse.org 69.55.236.132 /bin/sh /etc/rc<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
vnconfig /dev/vn22 /mnt/data2/69.55.228.13-col01063<br />
fsck -y /dev/vn22<br />
mount /dev/vn22c /mnt/data2/69.55.228.13-col01063-DIR<br />
chmod 0666 /mnt/data2/69.55.228.13-col01063-DIR/dev/null<br />
jail /mnt/data2/69.55.228.13-col01063-DIR www.widestream.com.au 69.55.228.13 /bin/sh /etc/rc<br />
<br />
# cancelled col00106<br />
#vnconfig /dev/vn15 /mnt/data2/69.55.238.5-col00106<br />
#fsck -y /dev/vn15<br />
#mount /dev/vn15c /mnt/data2/69.55.238.5-col00106-DIR<br />
#chmod 0666 /mnt/data2/69.55.238.5-col00106-DIR/dev/null<br />
#jail /mnt/data2/69.55.238.5-col00106-DIR mail.azebu.net 69.55.238.5 /bin/sh /etc/rc</pre><br />
<br />
As you can see, two of the systems specified are commented out - presumably those customers cancelled, or were moved to new servers.<br />
<br />
As you can see, the vnconfig line is the simpler command line, not the longer one that was used when it was first configured. As you can see, all that is done is, vnconfig the filesystem, then fsck it, then mount it. The fourth command is the `jail` command used to start the system – but that will be covered later.<br />
<br />
Here is the safe2 file from jail17:<br />
<br />
<pre>vnconfig /dev/vn16 /mnt/data2/69.55.228.7-col00820<br />
mount /dev/vn16c /mnt/data2/69.55.228.7-col00820-DIR<br />
chmod 0666 /mnt/data2/69.55.228.7-col00820-DIR/dev/null<br />
jail /mnt/data2/69.55.228.7-col00820-DIR mail1.phimail.com 69.55.228.7 /bin/sh /etc/rc<br />
<br />
# moved to data2 col00368<br />
#vnconfig /dev/vn28 /mnt/data2/69.55.236.132-col00368<br />
#mount /dev/vn28c /mnt/data2/69.55.236.132-col00368-DIR<br />
#chmod 0666 /mnt/data2/69.55.236.132-col00368-DIR/dev/null<br />
#jail /mnt/data2/69.55.236.132-col00368-DIR limehouse.org 69.55.236.132 /bin/sh /etc/rc<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
vnconfig /dev/vn22 /mnt/data2/69.55.228.13-col01063<br />
mount /dev/vn22c /mnt/data2/69.55.228.13-col01063-DIR<br />
chmod 0666 /mnt/data2/69.55.228.13-col01063-DIR/dev/null<br />
jail /mnt/data2/69.55.228.13-col01063-DIR www.widestream.com.au 69.55.228.13 /bin/sh /etc/rc<br />
<br />
# cancelled col00106<br />
#vnconfig /dev/vn15 /mnt/data2/69.55.238.5-col00106<br />
#mount /dev/vn15c /mnt/data2/69.55.238.5-col00106-DIR<br />
#chmod 0666 /mnt/data2/69.55.238.5-col00106-DIR/dev/null<br />
#jail /mnt/data2/69.55.238.5-col00106-DIR mail.azebu.net 69.55.238.5 /bin/sh /etc/rc</pre><br />
<br />
As you can see, it is exactly the same, but it does not have the fsck lines.<br />
<br />
Take a look at the last entry - note that the file is named:<br />
<br />
/mnt/data2/69.55.238.5-col00106<br />
<br />
and the mount point is named:<br />
<br />
/mnt/data2/69.55.238.5-col00106-DIR<br />
<br />
This is the general format on all the FreeBSD systems. The file is always named:<br />
<br />
IP-custnumber<br />
<br />
and the directory is named:<br />
<br />
IP-custnumber-DIR<br />
<br />
If you run safe when you need a fsck, the mount will fail and jail will fail:<br />
<br />
# mount /dev/vn1c /mnt/data2/jails/65.248.2.131-ns1.kozubik.com-DIR<br />
mount: /dev/vn1c: Operation not permitted<br />
<br />
No reboot needed, just run the quad script<br />
<br />
Starting with 6.x jails, we added block delimiters to the quad/safe files, the block looks like:<br />
<br />
<pre>echo '## begin ##: nuie.solaris.mu'<br />
fsck -y /dev/concat/v30v31a<br />
mount /dev/concat/v30v31a /mnt/data1/69.55.228.218-col01441-DIR<br />
mount_devfs devfs /mnt/data1/69.55.228.218-col01441-DIR/dev<br />
devfs -m /mnt/data1/69.55.228.218-col01441-DIR/dev rule -s 3 applyset<br />
jail /mnt/data1/69.55.228.218-col01441-DIR nuie.solaris.mu 69.55.228.218 /bin/sh /etc/rc<br />
echo '## end ##: nuie.solaris.mu'</pre><br />
<br />
These are more than just informative when running quad/safe’s, the echo lines MUST be present for certain tools to work properly. So it’s important that any updates to the hostname also be updated on the 2 echo lines. For example, if you try to startjail a jail with a hostname which is on the jail line but not the echo lines, the command will return with host not found.<br />
<br />
=== FreeBSD 7.x+ notes ===<br />
<br />
Starting with the release of FreeBSD 7.x, we are doing jail startups in a slightly different way. First, thereis only 1 file: <tt>/usr/local/jail/rc.d/quad1</tt><br />
There are no other quads or corresponding safe files. The reason for this is twofold, 1. We can pass –C to fsck which will tell is to skip the fsck if the fs is clean (no more need for safe files), 2. We have a new startup script which can be launched multiple times, running in parallel to start jails, where quad1 is the master jail file. <br />
Quad1 could still be run as a shell script, but it would take a very long time for it to run completely so it’s not advisable; or you should break it down into smaller chunks (like quad1, quad2, quad3, etc)<br />
<br />
Here is a snip of (a 7.x version) quad1 from jail2:<br />
<br />
<pre>echo '## begin ##: projects.tw.com'<br />
mdconfig -a -t vnode -f /mnt/data1/69.55.230.46-col01213 -u 50<br />
fsck -Cy /dev/md50c<br />
mount /dev/md50c /mnt/data1/69.55.230.46-col01213-DIR<br />
mount -t devfs devfs /mnt/data1/69.55.230.46-col01213-DIR/dev<br />
devfs -m /mnt/data1/69.55.230.46-col01213-DIR/dev rule -s 3 applyset<br />
jail /mnt/data1/69.55.230.46-col01213-DIR projects.tw.com 69.55.230.46 /bin/sh /etc/rc<br />
echo '## end ##: projects.tw.com'</pre><br />
<br />
Cancelled jails are no longer commented out and stored in quad1, rather they’re moved to <tt>/usr/local/jail/rc.d/deprecated</tt> <br />
<br />
To start these jails, start the 4 ssh sessions as you would for a normal crash and then instead of running quad1-4, instead run startalljails in each window. IMPORTANT- before running startalljails you should make sure you ran preboot once as it will clear out all the lockfiles and enable startalljails to work properly.<br />
<br />
== Problems with the quad/safe files ==<br />
<br />
When you run the quad/safe files, there are two problems that can occur - either a particular system will hang during initialization, OR a system will spit out output to the screen, impeding your ability to do anything. Or both.<br />
<br />
First off, when you start a jail, you see output like this:<br />
<br />
<pre>Skipping disk checks ...<br />
adjkerntz[25285]: sysctl(put_wallclock): Operation not permitted<br />
Doing initial network setup:.<br />
ifconfig: ioctl (SIOCDIFADDR): permission denied<br />
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384<br />
Additional routing options: TCP keepalive=YESsysctl:<br />
net.inet.tcp.always_keepalive: Operation not permitted.<br />
Routing daemons:.<br />
Additional daemons: syslogd.<br />
Doing additional network setup:.<br />
Starting final network daemons:.<br />
ELF ldconfig path: /usr/lib /usr/lib/compat /usr/X11R6/lib /usr/local/lib<br />
a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout /usr/X11R6/lib/aout<br />
Starting standard daemons: inetd cron sshd sendmail sendmail-clientmqueue.<br />
Initial rc.i386 initialization:.<br />
Configuring syscons: blanktime.<br />
Additional ABI support:.<br />
Local package initialization:.<br />
Additional TCP options:.</pre><br />
<br />
Now, let's look at this line, near the end:<br />
<br />
Local package initialization:.<br />
<br />
This is where a list of daemons that are set to start at boot time willshow up. You might see something like:<br />
<br />
Local package initialization: mysqld apache sendmail sendmail-clientmqueue<br />
<br />
Or something like this:<br />
<br />
Local package initialization: postgres postfix apache<br />
<br />
The problem is that many systems (about 4-5 per machine) will hang on that line. Basically it will get to some of the way through the total daemons to be started:<br />
<br />
Local package initialization: mysqld apache<br />
<br />
and will just sit there. Forever.<br />
<br />
Fortunately, pressing ctrl-c will break out of it. Not only will it break out of it, but it will also continue on that same line and start the other daemons:<br />
<br />
Local package initialization: mysqld apache ^c sendmail-clientmqueue<br />
<br />
and then continue on to finish the startup, and then move to the next system to be started.<br />
<br />
So what does this mean? It means that if a machine crashes, and you start four screen-windows to run four quads or four safes, you need to periodically cycle between them and see if any systems are stuck at that point, causing their quad/safe file to hang. A good rule of thumb is, if you see a system at that point in the startup, give it another 100 seconds - if it is still at the exact same spot, hit ctrl-c. Its also a good idea to go back into the quad file (just before the first command in the jail startup block) and note that this jail tends to need a control-c or more time as follows:<br />
<pre>echo '### NOTE ### slow sendmail'<br />
echo '### NOTE ###: ^C @ Starting sendmail.'</pre><br />
<br />
'''NEVER''' hit ctrl-c repeatedly if you don't get an immediate response - that will cause the following jail’s startup commands to be aborted.<br />
<br />
A second problem that can occur is that a jail - maybe the first one in that particular quad/safe, maybe the last one, or maybe one in the middle, will start spitting out status or error messages from one of its init scripts. This is not a problem - basically, hit enter a few times and see if you get a prompt - if you do get a prompt, that means that the quad/safe script has already completed. Therefore it is safe to log out (and log out of the user that you su'd from) and then log back in (if necessary).<br />
<br />
The tricky thing is, if a system in the middle starts flooding with messages, and you hit enter a few times and don't get a prompt. Are you not getting a prompt because some subsequent system is hanging at the initialization, as we discussede above ? Or are you not getting a prompt because that quad file is currently running an fsck ? Usually you can tell by scrolling back in screen’s history to see what it was doing before you started getting the messages.<br />
<br />
If you don’t get clues from history, you have to use your judgement - instead of giving it 100 seconds to respond, perhaps give it 2-3 mins ... if you still get no response (no prompt) when you hit enter, hit ctrl-c. However, be aware that you might still be hitting ctrl-c in the middle of an fsck. This means you will get an error like "filesystem still marked dirty" and then the vnconfig for it will fail and so will the jail command, and the next system in the quad file will then start starting up.<br />
<br />
If this happens, just wait until the end of all the quad files have finished, and start that system manually.<br />
<br />
If things really get weird, like a screen flooded with errors, and you can't get a prompt, and ctrl-c does nothing, then you need to just eventually (give it ten mins or so) just kill that window with ctrl-p, then k, and then log in again and manually check which systems are now running and which aren't, and manually start up any that are not.<br />
<br />
Don't EVER risk running a particular quad/safe file a second time.<br />
If the quad/safe script gets executed twice, reboot the machine immediately.<br />
<br />
So, for all the above reasons, anytime a machine crashes and you run all the quads or all the safes, '''always''' check every jail afterwards to make sure it is running - even if you have no hangs or complications at all.<br />
Run this command:<br />
<br />
<tt>[[#jailpsall|jailpsall]]</tt> <br />
<br />
Note: [[#postboot|postboot]] also populates ipfw counts, so it '''should not be run multiple times''', use <tt>jailpsall</tt> for subsequent extensive ps’ing<br />
<br />
And make sure they all show as running. If one does not show as running, check its /etc/rc.conf file first to see if maybe it is using a different hostname first before starting it manually.<br />
<br />
One thing we have implemented to alleviate these startup hangs and noisy jails, is to put jail start blocks that are slow or hangy at the bottom of the safe/quad file. Further, for each bad jail we note in each quad/safe just before the start block something like:<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
<br />
That way we’ll be prepared to ^C when we see that message appear during the quad/safe startup process. If you observe a new, undocumented hang, '''after''' the quad/safe has finished, place a line similar to the above in the quad file, move the jail start block to the end of the file, then run [[#buildsafe|buildsafe]]<br />
<br />
<br />
== Recovering from a crash (FreeBSD) ==<br />
<br />
=== Diagnose whether you have a crash ===<br />
The most important thing is to get the machine and all jails back up as soon as possible. Note the time, you’ll need to create a crash log entry (Mgmt. -> Reference -> CrashLog). The first thing to do is head over to the [[Screen#Screen_Organization|serial console screen]] and see if there’s any kernel error messages output. Try to copy any messages (or just a sample of repeating messages) you see into the notes section of the crash log. If there are no messages, the machine may just be really busy- wait a bit (5-10min) to see if it comes back. If it's still pinging, odds are its very busy. Note, if you see messages about swap space exhausted, the server is obviously out of memory, however it may recover briefly enough for you to get a jtop in to see who's lauched a ton of procs (most likely) and then issue a quick jailkill to get it back under control.<br />
<br />
If it doesn't come back, or the messages indicate a fatal error, you will need to proceed with a power cycle (ctrl+alt+del will not work).<br />
<br />
=== Power cycle the server ===<br />
If this machine is not a Dell 2950 with a [[DRAC/RMM#DRAC|DRAC card]] (i.e. if you can’t ssh into the DRAC card (as root, using the standard root pass) and issue <br />
racadm serveraction hardreset<br />
then you will need someone at the data center power the macine off, wait 30 sec, then turn it back on. Make sure to re-attach via console:<br />
tip jailX<br />
immediately after power down. <br />
<br />
=== (Re)attach to the console ===<br />
Stay on the console the entire time during boot. As the BIOS posts- look out for the RAID card output- does everything look healthy? The output may be scrambled, look for "DEGRADED" or "FAILED". Once the OS starts booting you will be disconnected (dropped back to the shell on the console server) a couple times during the boot up. The reason you want to quickly re-attach is two-fold: 1. If you don’t reattach quickly then you won’t get any console output, 2. you want to be attached before the server ''potentially'' starts (an extensive) fsck. If you attach after the fsck begins, you’ll have seen no indication it started an fsck and the server will appear frozen during startup- no output, no response. <br />
<br />
IMPORTANT NOTE: on some older FreeBSD systems, there will be no output to the video (KVM) console as it boots up. The console output is redirected to the serial port ... so if a jail crashes, and you attach a kvm, the output during the bootup procedure will not be shown on the screen. However, when the bootup is done, you will get a login prompt on the screen and will be able to log in as normal. <tt>/boot/loader.conf</tt> is where serial console redirect output lives, so comment that if you want to catch output on kvm.<br />
On newer systems it sends most output to both locations. <br />
<br />
=== Assess the heath of the server ===<br />
Once the server boots up fully, you should be able to ssh in. Look around- make sure all the mounts are there and reporting the correct size/usage (i.e. /mnt/data1 /mnt/data2 /mnt/data3 - look in /etc/fstab to determine which mount points should be there), check to see if RAID mirrors are healthy. See [[RAID_Cards#Common_CLI_commands_.28megacli.29|megacli]], [[#aaccheck|aaccheck]]<br />
<br />
Before you start the jails, you need to run [[#preboot|preboot]]. This will do some assurance checks to make sure things are prepped to start the jails. Any issues that come out of preboot need to be addressed before starting jails.<br />
<br />
=== Start jails ===<br />
[[#Starting_jails:_Quad.2FSafe_Files|More on starting jails]]<br />
Customer jails (the VPSs) do not start up automatically at boot time. When a FreeBSD machines boots up, it boots up, and does nothing else. To start jails, we put the commands to start each jail into a shell script(s) and run the script(s). Jail startup is something that needs to be actively monitored, which is why we don’t just run the script automatically. <br />
<br />
In order to start jails, we run the quad files: quad1 quad2 quad3 and quad4 (on new systems there is only quad1). If the machine was cleanly rebooted- which wouldn't be the case if this was a crash, you may run the safe files (safe1 safe2 safe3 safe4) in lieu of quads. <br />
<br />
Open up 4 logins to the server (use the windows in [[Screen#Screen_Organization|a9]])<br />
In each of the 4 windows you will:<br />
<br />
If there is a [[#startalljails|startalljails]] script (and only quad1), run that command in each of the 4 windows. It will parse through the quad1 file and start each jail. Follow the instructions [[#Problems_with_the_quad.2Fsafe_files|here]] for monitoring startup. Note that you can be a little more lenient with jails that take awhile to start- startalljails will work around the slow jails and start the rest. As long as there aren't 4 jails which are "hung" during startup, the rest will get started eventually.<br />
-or-<br />
If there is no startalljails script, there will be multiple quad files. In each of the 4 windows, start each of the quads. i.e. start quad1 in window1, quad2 in window2 and so on. DO NOT start any quad twice. It will crash the server. If you accidentally do this, just jailkill all the jails which are in the quad and run the quad again. Follow the instructions here for monitoring quad startup.<br />
<br />
Note the time the last jail boots- this is what you will enter in the crash log.<br />
<br />
Save the crash log.<br />
<br />
=== Check to make sure all jails have started ===<br />
There's a simple script which will make sure all jails have started, and enter the ipfw counter rules: [[#postboot|postboot]] <br />
Run postboot, which will do a jailps on each jail it finds (excluding commented out jails) in the quad file(s). We're looking for 2 things:<br />
# systems spawning out of control or too many procs<br />
# jails which haven't started<br />
On 7.x and newer systems it will print out the problems (which jails haven't started) at the conclusion of postboot. <br />
On older systems you will need to watch closely to see if/when there's a problem, namely:<br />
<br />
[hostname] doesnt exist on this server<br />
<br />
When you get this message, it means one of 2 things:<br />
1. the jail really didn't start:<br />
When a jail doesn't start it usually boils down to a problem in the quad file. Perhaps the path name is wrong (data1 vs data2) or the name of the vn/mdfile is wrong. Once this is corrected, you will need to run the commands from the quad file manually, or you may use <tt>startjail <hostname></tt><br />
<br />
2. the customer has changed their hostname (and not told us) so their jail ''is'' running, just under a different hostname:<br />
On systems with jls, this is easy to rectify. First, get the customer info: <tt>g <hostname></tt><br />
Then look for the customer in jls: <tt>jls | grep <col0XXXX></tt><br />
From there you will see their new hostname- you should update that hostname in the quad file: don't forget to edit it on the <tt>## begin ##</tt> and <tt>## end ##</tt> lines, and in mgmt. <br />
On older systems without jls, this will be harder, you will need to look further to see their hostname- perhaps its in their /etc/rc.conf<br />
<br />
<br />
Once all jails are started, do some spot checks- try to ssh or browse to some customers, just to make sure things are really ok.<br />
<br />
== Adding disk to a 7.x/8.x jail ==<br />
or<br />
== Moving customer to a different drive (md) ==<br />
<br />
NOTE: this doesn’t apply to mx2 which uses gvinum. Use same procedure as 6.x<br />
NOTE: if you unmount before mdconfig, re-mdconfig (attach) then unmount then mdconfig -u again <br />
<br />
<br />
(parts to change/customize are <tt><span style="color:red">red</span></tt>)<br />
<br />
<br />
If someone wants more disk space, there’s a paste for it, send it to them (explains about downtime, etc).<br />
<br />
1. Figure out the space avail from <tt>js</tt>. Ideally, you want to put the customers new space on a different partition (and create the new md on the new partition). <br />
<br />
<br />
2. make a mental note of how much space they're currently using<br />
<br />
<br />
3. <tt>jailkill <hostname></tt> <br />
<br />
<br />
4. Umount it (including their devfs) but leave the md config’d (so if you use stopjail, you will have to re-mdconfig it)<br />
<br />
<br />
5. <tt>g <customerID></tt> to get the info (IP/cust#) needed to feed the new mdfile and mount name, and to see the current md device. <br />
<br />
<br />
6a. When there's enough room to place new system on an alternate, or the same drive:<br />
USE CAUTION not to overwrite (touch, mdconfig) existing md!!<br><br />
<tt>touch /mnt/data<span style="color:red">3/69.55.234.66-col01334</span><br><br />
mdconfig -a -t vnode -s 10g -f /mnt/data3/69.55.234.66-col01334 -u 97<br><br />
newfs /dev/md97</tt><br />
<br />
<br />
Optional- if new space is on a different drive, move the mount point directory AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
<tt>mount /dev/md<span style="color:red">97</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<br />
confirm you are mounted to /dev/md<span style="color:red">97</span> and space is correct:<br><br />
<tt>df .</tt> <br />
<br />
<br />
do the dump and pipe directly to restore:<br><br />
<tt>dump -0a -f - /dev/md<span style="color:red">1</span> | restore -r -f - <br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use <tt>df</tt> to confirm the restored data size matches the original usage figure<br />
<br />
<br />
md-unconfig old system:<br><br />
<tt>mdconfig -d -u <span style="color:red">1</span></tt><br />
<br />
<br />
archive old mdfile. <br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.237.26-col00241</span> /mnt/data<span style="color:red">1/old-col00241-mdfile-noarchive-20091211</span></tt><br />
<br />
<br />
edit quad (vq1) to point to new (/mnt/data<span style="color:red">3</span>) location AND new md number (md<span style="color:red">97</span>)<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
6b. When there's not enough room on an alternate partition, or on the same drive...but there is enough room if you were to remove the existing customer's space:<br />
<br />
<br />
mount backup nfs mounts:<br><br />
<tt>mbm</tt><br> <br />
(run <tt>df</tt> to confirm backup mounts are mounted)<br />
<br />
<br />
dump the customer to backup2 or backup1<br><br />
<tt>dump -0a -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> /dev/md<span style="color:red">1</span></tt><br><br />
(when complete WITHOUT errors, <tt>du</tt> the dump file to confirm it matches size, roughly, with usage)<br />
<br />
<br />
unconfigure and remove old mdfile<br><br />
<tt>mdconfig -d -u <span style="color:red">1</span><br><br />
rm /mnt/data<span style="color:red">1/69.55.237.26-col00241</span></tt><br><br />
(there should now be enough space to recreate your bigger system. If not, run sync a couple times)<br />
<br />
<br />
create the new system (ok to reuse old mdfile and md#):<br><br />
<tt>touch /mnt/data<span style="color:red">1/69.55.234.66-col01334</span><br><br />
mdconfig -a -t vnode -s <span style="color:red">10</span>g -f /mnt/data<span style="color:red">1/69.55.234.66-col01334</span> -u <span style="color:red">1</span><br><br />
newfs /dev/md<span style="color:red">1</span><br><br />
mount /dev/md<span style="color:red">1</span> /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> <br><br />
cd /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<br />
confirm you are mounted to /dev/md<span style="color:red">1</span> and space is correct:<br><br />
<tt>df .</tt> <br />
<br />
<br />
do the restore from the dumpfile on the backup server:<br><br />
<tt>restore -r -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> .<br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
umount nfs:<br><br />
<tt>mbu</tt><br />
<br />
<br />
If md# changed (or mount point), edit quad (<tt>vq1</tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new md number (md<span style="color:red">1</span>)<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
7. update disk (and dir if applicable) in mgmt screen<br />
<br />
<br />
8. update backup list AND move backups, if applicatble<br />
<br />
Ex: <tt>mvbackups <span style="color:red">69.55.237.26-col00241</span> jail<span style="color:red">9</span> data<span style="color:red">3</span></tt><br />
<br />
<br />
9. Optional: archive old mdfile<br />
<br />
<tt>mbm<br><br />
gzip -c old-col01588-mdfile-noarchive-20120329 > /deprecated/old-col01588-mdfile-noarchive-20120329.gz<br><br />
mbu<br><br />
rm old-col01588-mdfile-noarchive-20120329</tt><br />
<br />
== Adding disk to a 6.x jail (gvinum/gconcat) ==<br />
or<br />
== Moving customer to a different drive (gvinum/gconcat) ==<br />
<br />
(parts to change are <span style="color:red">highlighted</span>)<br />
<br />
<br />
If someone wants more disk space, there’s a paste for it, send it to them (explains about downtime, etc).<br />
<br />
<br />
1. Figure out the space avail from [[#js|js]]. Ideally, you want to put the customers new space on a different partition (and create the new md on the new partition). <br />
<br />
<br />
2. make a mental note of how much space they're currently using<br />
<br />
<br />
3. <tt>[[#stopjail|stopjail]] <hostname></tt> <br />
<br />
<br />
4. <tt>[[#g|g]] <customerID></tt> to get the info (IP/cust#) needed to feed the new mount name and existing volume/device. <br />
<br />
<br />
5a. When there's enough room to place new system on an alternate, or the same drive (using only UNUSED - including if it's in use by the system in question - gvinum volumes):<br />
<br />
<br />
Configure the new device:<br><br />
A. for a 2G system (single gvinum volume):<br><br />
<tt>bsdlabel -r -w /dev/gvinum/v<span style="color:red">123</span><br><br />
newfs /dev/gvinum/v<span style="color:red">123</span>a</tt><br><br />
<br />
-or- <br />
<br />
<br />
B. for a >2G system (create a gconcat volume):<br><br />
try to grab a contiguous block of gvinum volumes. gconcat volumes MAY NOT span drives (i.e. you cannot use a gvinum volume from data3 and a volume from data2 in the same gconcat volume). <br />
<br />
<tt>gconcat label <span style="color:red">v82-v84 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84</span><br><br />
bsdlabel -r -w /dev/concat/<span style="color:red">v82-v84</span><br><br />
newfs /dev/concat/<span style="color:red">v82-v84</span>a</tt><br />
<br />
<br />
Other valid gconcat examples:<br><br />
<tt>gconcat label v82-v84v109v112 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84 /dev/gvinum/v109 /dev/gvinum/v112<br><br />
gconcat label v82v83 /dev/gvinum/v82 /dev/gvinum/v83</tt> <br><br />
Note, long names will truncate: v144v145v148-v115 will truncate to v144v145v148-v1 (so you will refer to it as v144v145v148-v1 thereafter)<br />
<br />
<br />
Optional- if new volume is on a different drive, move the mount point directory (get the drive from js output) AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
<br />
confirm you are mounted to the device (<tt>/dev/gvinum/v<span style="color:red">123</span>a</tt> OR <tt>/dev/concat/<span style="color:red">v82-v84</span></tt>) and space is correct:<br><br />
A. <tt>mount /dev/gvinum/v<span style="color:red">123</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br><br />
-or-<br><br />
B. <tt>mount /dev/concat/<span style="color:red">v82-v84</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<tt>cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
df .</tt> <br />
<br />
<br />
do the dump and pipe directly to restore:<br><br />
<tt>dump -0a -f - /dev/gvinum/v<span style="color:red">1</span> | restore -r -f - <br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
edit quad (<tt>vq<span style="color:red">1</span></tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new volume (<tt>/dev/gvinum/v<span style="color:red">123</span>a</tt> or <tt>/dev/concat/<span style="color:red">v82-v84</span>a</tt>) , run <tt>buildsafe</tt><br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
5b. When there's not enough room on an alternate partition, or on the same drive...but there is enough room if you were to remove the existing customer's space (i.e. if you want/need to reuse the existing gvinum volumes and add on more):<br />
<br />
<br />
mount backup nfs mounts:<br><br />
<tt>mbm</tt> <br><br />
(run df to confirm backup mounts are mounted)<br />
<br />
<br />
dump the customer to backup2 or backup1<br><br />
<tt>dump -0a -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> /dev/<span style="color:red">gconcat/v106-v107</span></tt><br><br />
(when complete WITHOUT errors, du the dump file to confirm it matches size, roughly, with usage)<br />
<br />
<br />
unconfigure the old gconcat volume<br><br />
list member gvinum volumes:<br />
<br />
<tt>gconcat list <span style="color:red">v106v107</span></tt><br />
<br />
<br />
Output will resemble:<br><br />
<pre>Geom name: v106v107<br />
State: UP<br />
Status: Total=2, Online=2<br />
Type: AUTOMATIC<br />
ID: 3530663882<br />
Providers:<br />
1. Name: concat/v106v107<br />
Mediasize: 4294966272 (4.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e2<br />
Consumers:<br />
1. Name: gvinum/sd/v106.p0.s0<br />
Mediasize: 2147483648 (2.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e3<br />
Start: 0<br />
End: 2147483136<br />
2. Name: gvinum/sd/v107.p0.s0<br />
Mediasize: 2147483648 (2.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e3<br />
Start: 2147483136<br />
End: 4294966272</pre><br />
<br />
stop volume and clear members<br><br />
<tt>gconcat stop <span style="color:red">v106v107</span><br><br />
gconcat clear <span style="color:red">gvinum/sd/v106.p0.s0 gvinum/sd/v107.p0.s0</span></tt><br />
<br />
<br />
create new device- and its ok to reuse old/former members<br><br />
try to grab a contiguous block of gvinum volumes. gconcat volumes MAY NOT span drives (i.e. you cannot use a gvinum volume from data3 and a volume from data2 in the same gconcat volume). <br><br />
<tt>gconcat label <span style="color:red">v82-v84v106v107 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84 /dev/gvinum/v106 /dev/gvinum/v107</span> <br><br />
bsdlabel -r -w /dev/concat/<span style="color:red">v82-v84v106v107</span><br><br />
newfs /dev/concat/<span style="color:red">v82-v84v106v107</span>a</tt><br />
<br />
<br />
Optional- if new volume is on a different drive, move the mount point directory (get the drive from js output) AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
confirm you are mounted to the device (/dev/concat/<span style="color:red">v82-v84v106v107</span>) and space is correct:<br><br />
<tt>mount /dev/concat/<span style="color:red">v82-v84v106v107</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<tt>cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
df .</tt> <br />
<br />
do the restore from the dumpfile on the backup server:<br><br />
<tt>restore -r -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> .<br><br />
rm restoresymtable</tt><br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
edit quad (<tt>vq<span style="color:red">1</span></tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new volume (<tt>/dev/concat/<span style="color:red">v82-v84v106v107</span>a</tt>), run buildsafe<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
TODO: clean up/clear old gvin/gconcat vol<br />
<br />
<br />
6. update disk (and dir if applicable) in mgmt screen<br />
<br />
<br />
7. update backup list AND move backups, if applicatble<br />
<br />
<br />
Ex: <tt>mvbackups <span style="color:red">69.55.237.26-col00241</span> jail<span style="color:red">9</span> data<span style="color:red">3</span></tt><br />
<br />
<br />
<br />
DEPRECATED - steps to tack on a new gvin to existing gconcat- leads to corrupted fs<br />
bsdlabel -e /dev/concat/v82-v84<br />
<br />
To figure out new size of the c partition, multiply 4194304 by the # of 2G gvinum volumes and subtract the # of 2G volumes:<br />
10G: 4194304 * 5 – 5 = 20971515<br />
8G: 4194304 * 4 – 4 = 16777212<br />
6G: 4194304 * 3 – 3 = 12582909<br />
4G: 4194304 * 2 – 2 = 8388606<br />
<br />
To figure out the new size of the a partition, subtract 16 from the c partition:<br />
10G: 20971515 – 16 = 20971499<br />
8G: 16777212 – 16 = 16777196<br />
6G: 12582909 – 16 = 12582893<br />
4G: 8388606 – 16 = 8388590<br />
<br />
Orig:<br />
8 partitions:<br />
# size offset fstype [fsize bsize bps/cpg]<br />
a: 8388590 16 4.2BSD 2048 16384 28552<br />
c: 8388606 0 unused 0 0 # "raw" part, don't edit<br />
<br />
New:<br />
8 partitions:<br />
# size offset fstype [fsize bsize bps/cpg]<br />
a: 12582893 16 4.2BSD 2048 16384 28552<br />
c: 12582909 0 unused 0 0 # "raw" part, don't edit<br />
<br />
sync; sync<br />
<br />
growfs /dev/concat/v82-v84a<br />
<br />
fsck –fy /dev/concat/v82-v84a<br />
<br />
sync<br />
<br />
fsck –fy /dev/concat/v82-v84a<br />
<br />
(keep running fsck’s till NO errors)<br />
<br />
== Problems un-mounting - and with mount_null’s ==<br />
<br />
If you cannot unmount a filesystem, beacuse it says the filesystem is busy, it is because of three things:<br />
<br />
a) the jail is still running<br />
<br />
b) you are actually in that directory, even though the jail is stopped<br />
<br />
c) there are still dev, null_mount or linprocfs mount points mounted inside that directory.<br />
<br />
d) when trying to umount null_mounts that are really long and you get an error like “No such file or directory”, it’s an OS bug where the dir is truncated. No known fix<br />
<br />
e) there are still files open somewhere inside the dir. Use <tt>fstat | grep <cid></tt> to find the process that has files open<br />
<br />
f) Starting with 6.x, the jail mechanism does a poor job of keeping track of processes running in a jail and if it thinks there are still procs running, it will refuse to umount the disk. If this is happening you should see a low number in the #REF column when you run jls. In this case you ''can'' safely <tt>umount –f</tt> the mount. <br />
<br />
Please note -if you forcibly unmount a (4.x) filesystem that has null_mounts<br />
still mounted in it, the system '''will crash''' within 10-15 mins.<br />
<br />
== Misc jail Items ==<br />
<br />
We are overselling hard drive space on jail2, jail8, jail9, a couple jails on jail17, jail4, jail12 and jail18.<br />
Even though the vn file shows 4G size, it doesn’t actually occupy that amount of space on the disk. So be careful not to fill up drives where we’re overselling – use oversellcheck to confirm you’re not oversold by more than 10G.<br />
There are other truncated jails, they are generally noted in a the file on the root system: /root/truncated<br />
<br />
The act of moving a truncated vn to another system un-does the truncating- the truncated vn is filled with 0’s and it occupies physical disk space for which it’s configured. So, you should use dumpremote to preserve the truncation.<br />
<br />
= FreeBSD VPS Management Tools =<br />
<br />
These files are located in /usr/local/jail/rc.d and /usr/local/jail/bin<br />
<br />
== jailmake ==<br />
<br />
Applies to 7.x+ <br />
On older systems syntax differs, run jailmake once to see.<br />
<br />
Note: this procedure differs on mx2 which is 7.x but still uses gvinum<br />
<br />
# run js to figure out which md’s are in use, which disk has enough space, IP to put it on<br />
# use col00xxx for both hostnames if they don’t give you a hostname<br />
# copy over dir, ip and password to pending customer screen<br />
<br />
<tt>Usage: jailmake IP[,IP] CID disk[1|2|3] md# hostname shorthost ipfw# email [size in GB]</tt><br />
<br />
Ex: <br />
<br />
Jail2# jailmake 69.55.234.66 col01334 3 97 vps.bsd.it vps 1334 fb@bsd.it<br />
<br />
== jailps ==<br />
jailps [hostname]<br />
DEPRECATED FOR jps: displays processes belonging to/running inside a jail. The command<br />
takes one (optional) argument – the hostname of the jail you wish to query. If you don’t <br />
supply an argument, all processes on the machine are listed and grouped by jail. <br />
<br />
== jps ==<br />
jps [hostname]<br />
displays processes belonging to/running inside a jail. The command<br />
takes one (optional) argument – the hostname or ID of the jail you wish to query. <br />
<br />
== jailkill ==<br />
jailkill <hostname><br />
stops all process running in a jail.<br />
<br />
You can also run:<br />
jailkill <JID><br />
<br />
=== problems ===<br />
Occasionally you will hit an issue where jail will not kill off:<br />
<br />
<pre>jail9# jailkill www.domain.com<br />
www.domain.com .. killed: none<br />
jail9#</pre><br />
Because no processes are running under that hostname. You cannot use jailps.pl either:<br />
<br />
<pre>jail9# jailps www.domain.com<br />
www.domain.com doesn’t exist on this server<br />
jail9#</pre><br />
<br />
The reasons for this are usually:<br />
* the jail is no longer running<br />
<br />
* the jail's hostname has changed<br />
In this case, <br />
<br />
>=6.x: run a <tt>jls|grep <jail's IP></tt> to find the correct hostname, then update the quad file, then kill the jail.<br />
<br />
<6.x: the first step is to cat their /etc/rc.conf file to see if you can tell what they set the new hostname to. This very often works. For example:<br />
<br />
cat /mnt/data2/198.78.65.136-col00261-DIR/etc/rc.conf<br />
<br />
But maybe they set the hostname with the hostname command, and the original hostname is still in /etc/rc.conf.<br />
<br />
The welcome email clearly states that they should tell us if they change their hostname, so there is no problem in just emailing them and asking them what they set the new hostname to.<br />
<br />
Once you know the new hostname OR if a customer simply emails to inform you that they have set the hostname to something different, you need to edit the quad and safe files that their system is in to input the new hostname.<br />
<br />
However, if push comes to shove and you cannot find out the hostname from them or from their system, then you need to start doing some detective work.<br />
<br />
The easiest thing to do is run jailps looking for a hostname similar to their original hostname. Or you could get into the /bin/sh shell by running:<br />
<br />
/bin/sh<br />
<br />
and then looking at every hostname of every process:<br />
<br />
for f in `ls /proc` ; do cat /proc/$f/status ; done<br />
<br />
and scanning for a hostname that is either similar to their original hostname, or that you don't see in any of the quad safe files.<br />
<br />
This is very brute force though, and it is possible that catting every file in /proc is dangerous - I don't recommend it. A better thing would be to identify any processes that you know belong to this system – perhaps the reason you are trying to find this system is because they are running something bad - and just catting the status from only that PID.<br />
<br />
Somewhere there’s a jail where there may be 2 systems named www. Look at /etc/rc.conf and make sure they’re both really www. If they are, jailkill www, jailps www to make sure not running. Then immediately restart the other one, as the fqdn (as found from a rev nslookup)<br />
<br />
* on >=6.x the hostname may not yet be hashed:<br />
<pre>jail9 /# jls<br />
JID Hostname Path IP Address(es)<br />
1 bitnet.dgate.org /mnt/data1/69.55.232.50-col02094-DIR 69.55.232.50<br />
2 ns3.hctc.net /mnt/data1/69.55.234.52-col01925-DIR 69.55.234.52<br />
3 bsd1 /mnt/data1/69.55.232.44-col00155-DIR 69.55.232.44<br />
4 let2.bbag.org /mnt/data1/69.55.230.92-col00202-DIR 69.55.230.92<br />
5 post.org /mnt/data2/69.55.232.51-col02095-DIR 69.55.232.51 ...<br />
6 ns2 /mnt/data1/69.55.232.47-col01506-DIR 69.55.232.47 ...<br />
7 arlen.server.net /mnt/data1/69.55.232.52-col01171-DIR 69.55.232.52<br />
8 deskfood.com /mnt/data1/69.55.232.71-col00419-DIR 69.55.232.71<br />
9 mirage.confluentforms.com /mnt/data1/69.55.232.54-col02105-DIR 69.55.232.54 ...<br />
10 beachmember.com /mnt/data1/69.55.232.59-col02107-DIR 69.55.232.59<br />
11 www.agottem.com /mnt/data1/69.55.232.60-col02109-DIR 69.55.232.60<br />
12 sdhobbit.myglance.org /mnt/data1/69.55.236.82-col01708-DIR 69.55.236.82<br />
13 ns1.jnielsen.net /mnt/data1/69.55.234.48-col00204-DIR 69.55.234.48 ...<br />
14 ymt.rollingegg.net /mnt/data2/69.55.236.71-col01678-DIR 69.55.236.71<br />
15 verse.unixlore.net /mnt/data1/69.55.232.58-col02131-DIR 69.55.232.58<br />
16 smcc-mail.org /mnt/data2/69.55.232.68-col02144-DIR 69.55.232.68<br />
17 kasoutsuki.w4jdh.net /mnt/data2/69.55.232.46-col02147-DIR 69.55.232.46<br />
18 dili.thium.net /mnt/data2/69.55.232.80-col01901-DIR 69.55.232.80<br />
20 www.tekmarsis.com /mnt/data2/69.55.232.66-col02155-DIR 69.55.232.66<br />
21 vps.yoxel.net /mnt/data2/69.55.236.67-col01673-DIR 69.55.236.67<br />
22 smitty.twitalertz.com /mnt/data2/69.55.232.84-col02153-DIR 69.55.232.84<br />
23 deliver4.klatha.com /mnt/data2/69.55.232.67-col02160-DIR 69.55.232.67<br />
24 nideffer.com /mnt/data2/69.55.232.65-col00412-DIR 69.55.232.65<br />
25 usa.hanyuan.com /mnt/data2/69.55.232.57-col02163-DIR 69.55.232.57<br />
26 daifuku.ppbh.com /mnt/data2/69.55.236.91-col01720-DIR 69.55.236.91<br />
27 collins.greencape.net /mnt/data2/69.55.232.83-col01294-DIR 69.55.232.83<br />
28 ragebox.com /mnt/data2/69.55.230.104-col01278-DIR 69.55.230.104<br />
29 outside.mt.net /mnt/data2/69.55.232.72-col02166-DIR 69.55.232.72<br />
30 vps.payneful.ca /mnt/data2/69.55.234.98-col01999-DIR 69.55.234.98<br />
31 higgins /mnt/data2/69.55.232.87-col02165-DIR 69.55.232.87 ...<br />
32 ozymandius /mnt/data2/69.55.228.96-col01233-DIR 69.55.228.96<br />
33 trusted.realtors.org /mnt/data2/69.55.238.72-col02170-DIR 69.55.238.72<br />
34 jc1.flanderous.com /mnt/data2/69.55.239.22-col01504-DIR 69.55.239.22<br />
36 guppylog.com /mnt/data2/69.55.238.73-col00036-DIR 69.55.238.73<br />
40 haliohost.com /mnt/data2/69.55.234.41-col01916-DIR 69.55.234.41 ...<br />
41 satyr.jorge.cc /mnt/data1/69.55.232.70-col01963-DIR 69.55.232.70<br />
jail9 /# jailkill satyr.jorge.cc<br />
ERROR: jail_: jail "satyr,jorge,cc" not found<br />
</pre><br />
<br />
Note how it's saying <tt>satyr,jorge,cc</tt> is not found, and not <tt>satyr.jorge.cc</tt>. <br />
<br />
The jail subsystem tracks things using comma-delimited hostnames. That is created every few hours:<br />
<br />
jail9 /# crontab -l<br />
0 0,6,12,18 * * * /usr/local/jail/bin/sync_jail_names<br />
<br />
So if we run this manually:<br />
jail9 /# /usr/local/jail/bin/sync_jail_names<br />
<br />
Then kill the jail:<br />
jail9 /# jailkill satyr.jorge.cc<br />
successfully killed: satyr,jorge,cc</pre><br />
<br />
It worked.<br />
<br />
<br />
If you ever see this when trying to kill a jail:<br />
<br />
<pre># jailkill e-scribe.com<br />
killing JID: 6 hostname: e-scribe.com<br />
3 procs running<br />
3 procs running<br />
3 procs running<br />
3 procs running<br />
...</pre><br />
<br />
<tt>[[#jailkill|jailkill]]</tt> probably got lost trying to kill off the jail. Just ctrl-c the jailkill process, then run a jailps on the hostname, and <tt>kill -9</tt> any process which is still running. Keep running jailps and <tt>kill -9</tt> till all processes are gone.<br />
<br />
== jailpsall ==<br />
jailpsall<br />
will run a jailps on all jails configured in the quad files (this is different from<br />
jailps with no arguments as it won’t help you find a “hidden” system)<br />
<br />
== jailpsw ==<br />
jailpsw<br />
will run a jailps with an extra -w to provide wider output<br />
<br />
== jt (>=7.x) ==<br />
jt<br />
displays the top 20 processes on the server (the top 20 processes from top) and <br />
which jail owns them. This is very helpful for determining who is doing what when<br />
the server is very busy.<br />
<br />
== jtop (>=7.x) ==<br />
jtop<br />
a wrapper for top displaying processes on the server and which jail owns them. Constantly updates, like top. <br />
<br />
== jtop (<7.x) ==<br />
jtop<br />
displays the top 20 processes on the server (the top 20 processes from top) and <br />
which jail owns them. This is very helpful for determining who is doing what when<br />
the server is very busy.<br />
<br />
== stopjail ==<br />
stopjail <hostname> [1]<br />
this will jailkill, umount and vnconfig –u a jail. If passed an optional 2nd<br />
argument, it will not exit before umounting and un-vnconfig’ing in the event<br />
jailkill returns no processes killed. This is useful if you just want to umount<br />
and vnconfig –u a jail you’ve already killed. It is intelligent in that it won’t <br />
try to umount or vnconfig –u if it’s not necessary.<br />
<br />
== startjail ==<br />
startjail <hostname><br />
this will start vnconfig, mount (including linprocfs and null-mounts), and start a jail.<br />
Essentially, it reads the jail’s relevant block from the right quad file and executes it.<br />
It is intelligent in that it won’t try to mount or vnconfig if it’s not necessary.<br />
<br />
== jpid ==<br />
jpid <pid><br />
displays information about a process – including which jail owns it.<br />
It’s the equivalent of running cat /proc/<pid>/status<br />
<br />
== canceljail ==<br />
canceljail <hostname> [1]<br />
this will stop a jail (the equivalent of stopjail), check for backups (offer to remove them <br />
from the backup server and the backup.config), rename the vnfile, remove the dir, and <br />
edit quad/safe. If passed an optional 2nd argument, it will not exit upon failing to kill<br />
and processes owned by the jail. This is useful if you just want to cancel a jail which <br />
is already stopped.<br />
<br />
== jls ==<br />
jls [-v]<br />
Lists all jails running:<br />
<pre>JID #REF IP Address Hostname Path<br />
101 135 69.55.224.148 mail.pc9.org /mnt/data2/69.55.224.148-col01034-DIR</pre><br />
<br />
#REF is the number of references or procs(?) running<br />
<br />
Running with -v will give you all IPs assigned to each jail (7.2 up)<br />
<pre>JID #REF Hostname Path IP Address(es)<br />
101 139 mail.pc9.org /mnt/data2/69.55.224.148-col01034-DIR 69.55.224.14869.55.234.85</pre><br />
<br />
== startalljails ==<br />
startalljails<br />
7.2+ only. This will parse through quad1 and start all jails. It utilizes lockfiles so it won’t try to start a jail more than once- therefore multiple instances can be running in parallel without fear of starting a jail twice. If a jail startup gets stuck, you can ^C without fear of killing the script. IMPORTANT- before running startalljails you should make sure you ran preboot once as it will clear out all the lockfiles and enable startalljails to work properly.<br />
<br />
== aaccheck.sh ==<br />
aaccheck.sh<br />
displayes the output of container list and task list from aaccli<br />
<br />
== backup ==<br />
backup<br />
backup script called nightly to update jail scripts and do customer backups<br />
<br />
== buildsafe ==<br />
buildsafe<br />
creates safe files based on quads (automatically removing the fsck’s). This will destructively overwrite safe files<br />
<br />
== checkload.pl ==<br />
checkload.pl<br />
this was intended to be setup as a cronjob to watch processes on a jail when the load <br />
rises above a certain level. Not currently in use.<br />
<br />
== checkprio.pl ==<br />
checkprio.pl<br />
will look for any process (other than the current shell’s csh, sh, sshd procs) with a non-normal priority and normalize it<br />
<br />
== diskusagemon == <br />
diskusagemon <mount point> <1k blocks><br />
watches a mount point’s disk use, when it reaches the level specified in the 2nd argument,<br />
it exits. This is useful when doing a restore and you want to be paged as it’s nearing completion.<br />
Best used as: <tt>diskusagemon /asd/asd 1234; pagexxx</tt><br />
<br />
== dumprestore ==<br />
dumprestore <dumpfile><br />
this is a perl expect script which automatically enters ‘1’ and ‘y’. It seems to cause restore to fail<br />
to set owner permissions on large restores.<br />
<br />
== g ==<br />
g <search><br />
greps the quad/safe files for the given search parameter<br />
<br />
== gather.pl ==<br />
gather.pl<br />
gathers up data about jails configured and writes to a file. Used for audits against the db<br />
<br />
== gb ==<br />
gb <search><br />
greps backup.config for the given search parameter<br />
<br />
== gbg ==<br />
gbg <search><br />
greps backup.config for the given search parameter and presents just the directories (for clean pasting)<br />
<br />
== ipfwbackup ==<br />
ipfwbackup<br />
writes ipfw traffic count data to a logfile<br />
<br />
== ipfwreset ==<br />
ipfwreset<br />
writes ipfw traffic count data to a logfile and resets counters to 0<br />
<br />
== js ==<br />
js<br />
output varies by OS version, but generally provides information about the base jail:<br />
- which vn’s are in use<br />
- disk usage<br />
- info about the contents of quads<br />
- the # of inodes represented by the jails contained in the group (133.2 in the example below), and how many jails per data mount, as well as subtotals<br />
- ips bound to the base machine but not in use by a jail<br />
- free gvinum volumes, or unused vn’s or used md’s<br />
<br />
<pre>/usr/local/jail/rc.d/quad1:<br />
/mnt/data1 133.2 (1)<br />
/mnt/data2 1040.5 (7)<br />
total 1173.7 (8)<br />
/usr/local/jail/rc.d/quad2:<br />
/mnt/data1 983.4 (6)<br />
total 983.4 (6)<br />
/usr/local/jail/rc.d/quad3:<br />
/mnt/data1 693.4 (4)<br />
/mnt/data2 371.6 (3)<br />
total 1065 (7)<br />
/usr/local/jail/rc.d/quad4:<br />
/mnt/data1 466.6 (3)<br />
/mnt/data2 882.2 (5)<br />
total 1348.8 (8)<br />
/mnt/data1: 2276.6 (14)<br />
/mnt/data2: 2294.3 (15)<br />
<br />
Available IPs:<br />
69.55.230.11 69.55.230.13 69.55.228.200<br />
<br />
Available volumes:<br />
v78 /mnt/data2 2G<br />
v79 /mnt/data2 2G<br />
v80 /mnt/data2 2G</pre> <br />
<br />
== load.pl ==<br />
load.pl<br />
feeds info to load mrtg - executed by inetd.<br />
<br />
== makevirginjail ==<br />
makevirginjail<br />
Only on some systems, makes an empty jail (doesn't do restore step)<br />
<br />
== mb == <br />
mb <mount|umount><br />
(nfs) mounts and umounts dirs to backup2. Shortcuts are mbm and mbu to mount and unmount. <br />
<br />
== notify.sh ==<br />
notify.sh<br />
emails reboot@johncompanies.com – intended to be called at boot time to alert us to a machine which panics and reboots and isn’t caught by bb or castle.<br />
<br />
== orphanedbackupwatch ==<br />
orphanedbackupwatch<br />
looks for directories on backup2 which aren’t configured in backup.config and offers to delete them<br />
<br />
== postboot ==<br />
postboot<br />
to be run after a machine reboot and quad/safe’s are done executing. It will:<br />
* do chmod 666 on each jail’s /dev/null<br />
* add ipfw counts<br />
* run jailpsall (so you can see if a configured jail isn’t running)<br />
<br />
== preboot ==<br />
preboot<br />
to be run before running quad/safe – checks for misconfigurations: <br />
* a jail configured in a quad but not a safe<br />
* a jail is listed more than once in a quad<br />
* the ip assigned to a jail isn’t configured on the machine<br />
* alias numbering skips in the rc.conf (resulting in the above)<br />
* orphaned vnfile's that aren't mentioned in a quad/safe<br />
* ip mismatches between dir/vnfile name and the jail’s ip<br />
* dir/vnfiles's in quad/safe that don’t exist <br />
<br />
== quadanalyze.pl ==<br />
quadanalyze.pl<br />
called by js, produces the info (seen above with js explanation) about the contents of quad (inode count, # of jails, etc.)<br />
<br />
== rsync.backup ==<br />
rsync.backup<br />
does customer backups (relies on backup.config)<br />
<br />
== taskdone ==<br />
taskdone<br />
when called will email support@johncompanies.com with the hostname of the machine from which it was executed as the subject<br />
<br />
== topten ==<br />
topten<br />
summarizes the top 10 traffic users (called by ipfwreset)<br />
<br />
== trafficgather.pl ==<br />
trafficgather.pl [yy-mm]<br />
sends a traffic usage summary by jail to support@johncomapnies.com and payments@johncompanies.com. Optional arguments are year and month (must be in the past). If not passed, assumes last month. Relies on traffic logs created by ipfwreset and ipfwbackup<br />
<br />
== trafficwatch.pl ==<br />
trafficwatch.pl<br />
checks traffic usage and emails support@johncomapnies.com when a jail reaches the warning level (35G) and the limit (40G). We really aren’t using this anymore now that we have netflow.<br />
<br />
== trafstats ==<br />
trafstats<br />
writes ipfw traffic usage info by jail to a file called jc_traffic_dump in each jail’s / dir<br />
<br />
== truncate_jailmake ==<br />
truncate_jailmake<br />
a version of jailmake which creates truncated vnfiles.<br />
<br />
== vb ==<br />
vb<br />
the equivalent of: <tt>vi /usr/local/jail/bin/backup.config</tt><br />
<br />
== vs (freebsd) ==<br />
vs<n><br />
the equivalent of: <tt>vi /usr/local/jail/rc.d/safe<n></tt><br />
<br />
vq<n><br />
the equivalent of: vi /usr/local/jail/rc.d/quad<n><br />
<br />
== dumpremote ==<br />
dumpremote <user@machine> </remote/location/file-dump> <vnX><br />
ex: dumpremote user@10.1.4.117 /mnt/data3/remote.echoditto.com-dump 7<br />
this will dump a vn filesystem to a remote machine and location<br />
<br />
== oversellcheck ==<br />
oversellcheck<br />
displays how much a disk is oversold or undersold taking into account truncated vn files. Only for use on 4.x systems<br />
<br />
== mvbackups (freebsd) ==<br />
mvbackups <dir> (1.1.1.1-col00001-DIR) <target_machine> (jail1) <target_dir> (data1)<br />
moves backups from one location to another on the backup server, and provides you with option to remove entries from current backup.config, and simple paste command to add the config to backup.config on the target server<br />
<br />
== jailnice ==<br />
jailnice <hostname><br />
applies <tt>renice 19 [PID]</tt> and <tt>rtprio 31 –[PID]</tt> to each process in the given jail<br />
<br />
== dumpremoterestore ==<br />
dumpremoterestore <device> <ip of target machine> <dir on target machine><br />
ex: dumpremoterestore /dev/vn51 10.1.4.118 /mnt/data2/69.55.239.45-col00688-DIR<br />
dumps a device and restores it to a directory on a remote machine. Requires that you enable root ssh on the <br />
remote machine.<br />
<br />
== psj ==<br />
psj<br />
shows just the procs running on the base system – a ps auxw but without jail’d procs present<br />
<br />
== perc5iraidchk ==<br />
perc5iraidchk<br />
checks for degraded arrays on Dell 2950 systems with Perc5/6 controllers<br />
<br />
== perc4eraidchk ==<br />
perc4eraidchk<br />
checks for degraded arrays on Dell 2850 systems with Perc4e/Di controllers<br />
<br />
= Virtuozzo VPS =<br />
<br />
== Making new customer VE (vm) ==<br />
<br />
This applies only to new virts >= 4.x<br />
<br />
grab ip from ipmap (if opened from the pending cust screen it should take you to the right block). You can also run vzlist -a to see what block is in use, generally. Try to find an IP that's in the same block of class C IP's already on the box.<br />
<br />
1. confirm ip you want to use isn’t in use via Mgmt. -> IP Map in management screens<br />
<br />
2. put CT on whichever partition has more space<br />
<br />
3. vm CID IP hostname /vz[1|2] email[,email] template ( <LB|LBP|LS|LM|LMP|LP> [disk] | <gb_disk/mb_ram/gb_burst> ) <br />
vm col00009 69.55.230.238 centos.testdave.com /vz1 dsmith@n2.net centos-6-x86_64 LM<br />
<br />
4. copy veid, dir, ip and password to pending customer screen. activate customer<br />
<br />
<br />
Note: We use VEID (Virtual Environment ID) and CTID (Container ID) interchangably. Similarly, VE and CT. They mean the same thing.<br />
VZPP = VirtuoZzo Power Panel (the control panel for each CT)<br />
<br />
All linux systems exist in /vz, /vz1 or /vz2 - since each linux machine holds roughly 60-90 customers, there will be roughly 30-45 in each partition.<br />
<br />
The actual filesystem of the system in question is in:<br />
<br />
/vz(1-2)/private/(VEID)<br />
<br />
Where VEID is the identifier for that system - an all-numeric string larger than 100.<br />
<br />
The actual mounted and running systems are in the corresponding:<br />
<br />
/vz(1-2)/root/(VEID)<br />
<br />
But we rarely interact with any system from this mount point.<br />
<br />
You should never need to touch the root portion of their system – however you can traverse their filesystem by going to <tt>/vz(1-2)/private/(VEID)/root</tt> (<tt>/vz(1-2)/private/(VEID)/fs/root</tt> on 4.x systems) the root of their filesystem is in that directory, and their entire system is underneath that.<br />
<br />
Every VE has a startup script in <tt>/etc/sysconfig/vz-scripts</tt> (which is symlinked as <tt>/vzconf</tt> on all systems) - the VE startup script is simply named <tt>(VEID).conf</tt> - it contains all the system parameters for that VE:<br />
<br />
<pre># Configuration file generated by vzsplit for 60 VE<br />
# on HN with total amount of physical mem 2011 Mb<br />
<br />
VERSION="2"<br />
CLASSID="2"<br />
<br />
ONBOOT="yes"<br />
<br />
KMEMSIZE="8100000:8200000"<br />
LOCKEDPAGES="322:322"<br />
PRIVVMPAGES="610000:615000"<br />
SHMPAGES="33000:34500"<br />
NUMPROC="410:415"<br />
PHYSPAGES="0:2147483647"<br />
VMGUARPAGES="13019:2147483647"<br />
OOMGUARPAGES="13019:2147483647"<br />
NUMTCPSOCK="1210:1215"<br />
NUMFLOCK="107:117"<br />
NUMPTY="19:19"<br />
NUMSIGINFO="274:274"<br />
TCPSNDBUF="1800000:1900000"<br />
TCPRCVBUF="1800000:1900000"<br />
OTHERSOCKBUF="900000:950000"<br />
DGRAMRCVBUF="200000:200000"<br />
NUMOTHERSOCK="650:660"<br />
DCACHE="786432:818029"<br />
NUMFILE="7500:7600"<br />
AVNUMPROC="51:51"<br />
IPTENTRIES="155:155"<br />
DISKSPACE="4194304:4613734"<br />
DISKINODES="400000:420000"<br />
CPUUNITS="1412"<br />
QUOTAUGIDLIMIT="2000"<br />
VE_ROOT="/vz1/root/636"<br />
VE_PRIVATE="/vz1/private/636"<br />
NAMESERVER="69.55.225.225 69.55.230.3"<br />
OSTEMPLATE="vzredhat-7.3/20030305"<br />
VE_TYPE="regular"<br />
IP_ADDRESS="69.55.225.229"<br />
HOSTNAME="textengine.net"</pre><br />
<br />
<br />
As you can see, the hostname is set here, the disk space is set here, the number of inodes, the number of files that can be open, the number of tcp sockets, etc. - all are set here.<br />
<br />
In fact, everything that can be set on this customer system is set in this conf file.<br />
<br />
<br />
All interaction with the customer system is done with the VEID. You start the system by running:<br />
<br />
vzctl start 999<br />
<br />
You stop it by running:<br />
<br />
vzctl stop 999<br />
<br />
You execute commands in it by running:<br />
<br />
vzctl exec 999 df -k<br />
<br />
You enter into it, via a root-shell backdoor with:<br />
<br />
vzctl enter 999<br />
<br />
and you set parameters for the system, while it is still running, with:<br />
<br />
vzctl set 999 --diskspace 6100000:6200000 --save<br />
<br />
<tt>vzctl</tt> is the most commonly used command - we have aliased <tt>v</tt> to <tt>vzctl</tt> since we use it so often. We’ll continue to use <tt>vzctl</tt> in our examples, but feel free to use just <tt>v</tt>.<br />
<br />
Let's say the user wants more diskspace. You can cat their conf file and see:<br />
<br />
DISKSPACE="4194304:4613734"<br />
<br />
So right now they have 4gigs of space. You can then change it to 6 with:<br />
<br />
vzctl set 999 --diskspace 6100000:6200000 --save<br />
<br />
IMPORTANT: all issuances of the vzctl set command need to end with <tt>–save</tt> - if they don't, the setting will be set, but it will not be saved to the conf file, and they will not have those settings next time they boot.<br />
<br />
All of the tunables in the conf file can be set with the vzctl set command. Note that in the conf file, and on the vzctl set command line, we always issue two numbers seperated by a colon - that is because we are setting the hard and soft limits. Always set the hard limit slightly above the soft limit, as you see it is in the conf file for all those settings.<br />
<br />
There are also things you can set with `<tt>vzctl set</tt>` that are not in the conf file as settings, per se. For instance, you can add IPs:<br />
<br />
vzctl set 999 --ipadd 10.10.10.10 --save<br />
<br />
or multiple IPs:<br />
<br />
vzctl set 999 --ipadd 10.10.10.10 --ipadd 10.10.20.30 --save<br />
<br />
or change the hostname:<br />
<br />
vzctl set 999 --hostname www.example.com --save<br />
<br />
You can even set the nameservers:<br />
<br />
vzctl set 999 --nameserver 198.78.66.4 --nameserver 198.78.70.180 --save<br />
<br />
Although you probably will never do that.<br />
<br />
You can disable a VPS from being started (by VZPP or reboot) (4.x):<br />
<br />
vzctl set 999 --disabled yes --save <br />
<br />
You can disable a VPS from being started (by VZPP or reboot) (<=3.x):<br />
<br />
vzctl set 999 --onboot=no --save <br />
<br />
You can disable a VPS from using his control panel:<br />
<br />
vzctl set 999 --offline_management=no --save <br />
<br />
You can suspend a VPS, so it can be resumed in the same state it was in when it was stopped (4.x):<br />
<br />
vzctl suspend 999<br />
<br />
and to resume it:<br />
<br />
vzctl resume 999<br />
<br />
to see who owns process:<br />
vzpid <PID><br />
<br />
to mount up an unmounted ve:<br />
vzctl mount 827<br />
<br />
<br />
To see network stats for CT's:<br />
<pre># vznetstat<br />
VEID Net.Class Output(bytes) Input(bytes)<br />
-----------------------------------------------<br />
24218 1 484M 39M<br />
24245 1 463M 143M<br />
2451 1 2224M 265M<br />
2454 1 2616M 385M<br />
4149 1 125M 68M<br />
418 1 1560M 34M<br />
472 1 1219M 315M<br />
726 1 628M 317M<br />
763 1 223M 82M<br />
771 1 4234M 437M<br />
-----------------------------------------------<br />
Total: 13780M 2090M</pre><br />
<br />
<br />
One thing that sometimes comes up on older systems that we created with smaller defaults is that the system would run out of inodes. The user will email and say they cannot create any more files or grow any files larger, but they will also say that they are not out of diskspace ... they are running:<br />
<br />
df -k<br />
<br />
and seeing how much space is free - and they are not out of space. They are most likely out of inodes - which they would see by running:<br />
<br />
df -i<br />
<br />
So, the first thing you should do is enter their system with:<br />
<br />
vzctl enter 999<br />
<br />
and run: <tt>df -i</tt><br />
<br />
to confirm your theory. Then exit their system. Then simply cat their conf file and see what their inodes are set to (probably 200000:200000, since that was the old default on the older systems) and run:<br />
<br />
vzctl set 999 --diskinodes 400000:400000 --save<br />
<br />
If they are not out of inodes, then a good possibility is that they have maxed out their numfile configuration variable, which controls how many files they can have in their system. The current default is 7500 (which nobody has ever hit), but the old default was as low as 2000, so you would run something like:<br />
<br />
vzctl set 999 --numfile 7500:7500 --save<br />
<br />
----<br />
<br />
You cannot start or stop a VE if your pwd is its private (/vz/private/999) or root (/vz/root/999) directories, or anywhere below them.<br />
<br />
<br />
== Recovering from a crash (linux) ==<br />
<br />
=== Diagnose whether you have a crash ===<br />
The most important thing is to get the machine and all ve’s back up as soon as possible. Note the time, you’ll need to create a crash log entry (Mgmt. -> Reference -> CrashLog). The first thing to do is head over to the [[Screen#Screen_Organization|serial console screen]] and see if there’s any kernel error messages output. Try to copy any messages (or just a sample of repeating messages) you see into the notes section of the crash log – these will also likely need to be sent to virtuozzo for interpretation. If the messages are spewing too fast, hit ^O + H to start a screen log dump which you can ob1182.pts-38.bb serve after the machine is rebooted. Additionally, if the machine is responsive, you can get a trace to send to virtuozzo by hooking up a kvm and entering these 3 sequences:<br />
<pre>alt+print screen+m<br />
alt+print screen+p<br />
alt+print screen+t</pre><br />
<br />
If there are no messages, the machine may just be really busy- wait a bit (5-10min) to see if it comes back. If it's still pinging, odds are its very busy. If it doesn't come back, or the messages indicate a fatal error, you will need to proceed with a power cycle (ctrl+alt+del will not work).<br />
<br />
=== Power cycle the server ===<br />
If this machine is not a Dell 2950 with a [[DRAC/RMM#DRAC|DRAC card]] (i.e. if you can’t ssh into the DRAC card and issue racadm serveraction hardreset, then you will need someone at the data center to power the macine off, wait 30 sec, then turn it back on. Make sure to re-attach via console (<tt>tip virtxx</tt>) immediately after power down. <br />
<br />
=== (Re)attach to the console ===<br />
Stay on the console the entire time during boot. As the BIOS posts- look out for the RAID card output- does everything look healthy? The output may be scrambled, look for "DEGRADED" or "FAILED". Once the OS starts booting you will be disconnected (dropped back to the shell on the console server) a couple times during the boot up. The reason you want to quickly re-attach is two-fold: 1. If you don’t reattach quickly then you won’t get any console output, 2. you want to be attached before the server ''potentially'' starts (an extensive) fsck. If you attach after the fsck begins, you’ll have seen no indication it started an fsck and the server will appear frozen during startup- no output, no response. <br />
<br />
=== Start containers/VE's/VPSs ===<br />
When the machine begins to start VE’s, it’s safe to leave the console and login via ssh. All virts should be set to auto start all the VEs after a crash. Further, most (newer) virts are set to “fastboot” it’s VE’s (to find out, do:<br />
grep -i fast /etc/sysconfig/vz <br />
and look for <tt>VZFASTBOOT=yes</tt>). If this was set prior to the machine’s crash (setting it after the machine boots will not have any effect until the vz service is restarted) it will start each ve as fast as possible, in serial, then go thru each VE (serially), shutting it down running a vzquota (disk usage) check, then bringing it back up. The benefit is that all VE’s are brought up quickly (within 15min or so depending on the #), the downside is a customer watching closely will notice 2 outages – 1st the machine crash, 2nd their quota check (which will be a much shorter downtime- on the order of a few minutes). <br />
<br />
Where “fastboot” is not set to yes (i.e on quar1), vz will start them consecutively, checking the quotas one at a time, and the 60th VE may not start until an hour or two later - this is not acceptable.<br />
<br />
The good news is, if you run vzctl start for a VE that is already started, you will simply get an error: <tt>VE is already started</tt>. Further, if you attempt to vzctl start a VE that is in the process of being started, you will simply get an error: unable to lock VE. So, there is no danger in simply running scripts to start smaller sets of VEs. If the system is not autostarting, then there is no issue, and even if it does, when it conflicts, one process (yours or the autostart) will lose, and just move on to the next one.<br />
<br />
A script has been written to assist with ve starts: [[#startvirt.pl|startvirt.pl]] which will start 6 ve’s at once until there are no more left. If startvirt.pl is used on a system where “fastboot” was on, it will circumvent the fastboot for ve’s started by startvirt.pl – they will go through the complete quota check before starting- therefore this is not advisable when a system has crashed. When a system is booted cleanly, and there's no need for vzquota checks, then startvirt.pl is safe and advisable to run.<br />
<br />
=== Make sure all containers are running ===<br />
You can quickly get a feel for how many ve’s are started by running:<br />
<br />
<pre>[root@virt4 log]# vs<br />
VEID 16066 exist mounted running<br />
VEID 16067 exist mounted running<br />
VEID 4102 exist mounted running<br />
VEID 4112 exist mounted running<br />
VEID 4116 exist mounted running<br />
VEID 4122 exist mounted running<br />
VEID 4123 exist mounted running<br />
VEID 4124 exist mounted running<br />
VEID 4132 exist mounted running<br />
VEID 4148 exist mounted running<br />
VEID 4151 exist mounted running<br />
VEID 4155 exist mounted running<br />
VEID 42 exist mounted running<br />
VEID 432 exist mounted running<br />
VEID 434 exist mounted running<br />
VEID 442 exist mounted running<br />
VEID 450 exist mounted running<br />
VEID 452 exist mounted running<br />
VEID 453 exist mounted running<br />
VEID 454 exist mounted running<br />
VEID 462 exist mounted running<br />
VEID 463 exist mounted running<br />
VEID 464 exist mounted running<br />
VEID 465 exist mounted running<br />
VEID 477 exist mounted running<br />
VEID 484 exist mounted running<br />
VEID 486 exist mounted running<br />
VEID 490 exist mounted running</pre><br />
<br />
So to see how many ve’s have started:<br />
<pre>[root@virt11 root]# vs | grep running | wc -l<br />
39</pre><br />
<br />
And to see how many haven’t:<br />
<pre>[root@virt11 root]# vs | grep down | wc -l<br />
0</pre><br />
<br />
And how many we should have running:<br />
<pre>[root@virt11 root]# vs | wc -l<br />
39</pre><br />
<br />
Another tool you can use to see which ve’s have started, among other things is [[#vzstat|vzstat]]. It will give you CPU, memory, and other stats on each ve and the overall system. It’s a good thing to watch as ve’s are starting (note the VENum parameter, it will tell you how many have started):<br />
<br />
<pre>4:37pm, up 3 days, 5:31, 1 user, load average: 1.57, 1.68, 1.79<br />
VENum 40, procs 1705: running 2, sleeping 1694, unint 0, zombie 9, stopped 0<br />
CPU [ OK ]: VEs 57%, VE0 0%, user 8%, sys 7%, idle 85%, lat(ms) 412/2<br />
Mem [ OK ]: total 6057MB, free 9MB/54MB (low/high), lat(ms) 0/0<br />
Swap [ OK ]: tot 6142MB, free 4953MB, in 0.000MB/s, out 0.000MB/s<br />
Net [ OK ]: tot: in 0.043MB/s 402pkt/s, out 0.382MB/s 4116pkt/s<br />
Disks [ OK ]: in 0.002MB/s, out 0.000MB/s<br />
<br />
VEID ST %VM %KM PROC CPU SOCK FCNT MLAT IP<br />
1 OK 1.0/17 0.0/0.4 0/32/256 0.0/0.5 39/1256 0 9 69.55.227.152<br />
21 OK 1.3/39 0.1/0.2 0/46/410 0.2/2.8 23/1860 0 6 69.55.239.60<br />
133 OK 3.1/39 0.1/0.3 1/34/410 6.3/2.8 98/1860 0 0 69.55.227.147<br />
263 OK 2.3/39 0.1/0.2 0/56/410 0.3/2.8 34/1860 0 1 69.55.237.74<br />
456 OK 17/39 0.1/0.2 0/100/410 0.1/2.8 48/1860 0 11 69.55.236.65<br />
476 OK 0.6/39 0.0/0.2 0/33/410 0.1/2.8 96/1860 0 10 69.55.227.151<br />
524 OK 1.8/39 0.1/0.2 0/33/410 0.0/2.8 28/1860 0 0 69.55.227.153<br />
594 OK 3.1/39 0.1/0.2 0/45/410 0.0/2.8 87/1860 0 1 69.55.239.40<br />
670 OK 7.7/39 0.2/0.3 0/98/410 0.0/2.8 64/1860 0 216 69.55.225.136<br />
691 OK 2.0/39 0.1/0.2 0/31/410 0.0/0.7 25/1860 0 1 69.55.234.96<br />
744 OK 0.1/17 0.0/0.5 0/10/410 0.0/0.7 7/1860 0 6 69.55.224.253<br />
755 OK 1.1/39 0.0/0.2 0/27/410 0.0/2.8 33/1860 0 0 192.168.1.4<br />
835 OK 1.1/39 0.0/0.2 0/19/410 0.0/2.8 5/1860 0 0 69.55.227.134<br />
856 OK 0.3/39 0.0/0.2 0/13/410 0.0/2.8 16/1860 0 0 69.55.227.137<br />
936 OK 3.2/52 0.2/0.4 0/75/410 0.2/0.7 69/1910 0 8 69.55.224.181<br />
1020 OK 3.9/39 0.1/0.2 0/60/410 0.1/0.7 55/1860 0 8 69.55.227.52<br />
1027 OK 0.3/39 0.0/0.2 0/14/410 0.0/2.8 17/1860 0 0 69.55.227.83<br />
1029 OK 1.9/39 0.1/0.2 0/48/410 0.2/2.8 25/1860 0 5 69.55.227.85<br />
1032 OK 12/39 0.1/0.4 0/80/410 0.0/2.8 41/1860 0 8 69.55.227.90</pre><br />
<br />
When you are all done, you will want to make sure that all the VEs really did get started, run vs one more time.<br />
<br />
Note the time all ve’s are back up and enter that into and save the crash log entry.<br />
<br />
Occasionally, a ve will not start automatically. The most common reason for a ve not to come up normally is the ve was at it’s disk limit before the crash, and will not start since they’re over the limit. To overcome this, set the disk space to current usage level (the system will give this to you when it fails to start), start the ve, then re-set the disk space back to the prior level. Lastly, contact the customer to let them know they’re out of disk (or allocate more disk if they're entitled to more).<br />
<br />
== Hitting performance barriers and fixing them ==<br />
<br />
There are multiple modes virtuozzo offers to allocate resources to a ve. We utilize 2: SLM and UBC parameters<br />
On our 4.x systems, we use all SLM – it’s simpler to manage and understand. There are a few systems on virt19/18 that may also use SLM. Everything else uses UBC. <br />
You can tell a SLM ve by:<br />
<br />
SLMMODE="all"<br />
<br />
in their conf file. <br />
<br />
TODO: detail SLM modes and parameters.<br />
<br />
If someone is in SLM mode and they hit memory resource limits, they simply need to upgrade to more memory.<br />
<br />
The following applies to everyone else (UBC).<br />
<br />
Customers will often email and say that they are getting out of memory errors - a common one is "cannot fork" ... basically, anytime you see something odd like this, it means they are hitting one of their limits that is in place in their conf file.<br />
<br />
The conf file, however, simply shows their limits - how do we know what they are currently at ?<br />
<br />
The answer is a file called v - this file contains the current status (and peaks) of their performance settings, and also counts how many times they have hit the barrier. The output of the file looks like this:<br />
<br />
<pre>764: kmemsize 384113 898185 8100000 8200000 0<br />
lockedpages 0 0 322 322 0<br />
privvmpages 1292 7108 610000 615000 0<br />
shmpages 270 528 33000 34500 0<br />
dummy 0 0 0 0 0<br />
numproc 8 23 410 415 0<br />
physpages 48 5624 0 2147483647 0<br />
vmguarpages 0 0 13019 2147483647 0<br />
oomguarpages 641 6389 13019 2147483647 0<br />
numtcpsock 3 21 1210 1215 0<br />
numflock 1 3 107 117 0<br />
numpty 0 2 19 19 0<br />
numsiginfo 0 4 274 274 0<br />
tcpsndbuf 0 80928 1800000 1900000 0 <br />
tcprcvbuf 0 108976 1800000 1900000 0<br />
othersockbuf 2224 37568 900000 950000 0<br />
dgramrcvbuf 0 4272 200000 200000 0<br />
numothersock 3 9 650 660 0<br />
dcachesize 53922 100320 786432 818029 0<br />
numfile 161 382 7500 7600 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
numiptent 4 4 155 155 0</pre><br />
<br />
The first column is the name of the counter in question - the same names we saw in the systems conf file. The second column is the _current_ value of that counter, the third column is the max that that counter has ever risen to, the fourth column is the soft limit, and the fifth column is the hard limit (which is the same as the numbers in that systems conf file).<br />
<br />
The sixth number is the failcount - how many times the current usage has risen to hit the barrier. It will increase as soon as the current usage hits the soft limit.<br />
<br />
The problem with /proc/user_beancounters is that it actually contains that set of data for every running VE - so you can't just cat /proc/user_beancounters - it is too long and you get info for every other running system.<br />
<br />
You can vzctl enter the system and run:<br />
<br />
vzctl enter 9999<br />
cat /proc/user_beancounters<br />
<br />
inside their system, and you will just see the stats for their particular system, but entering their system every time you want to see it is combersome.<br />
<br />
So, I wrote a simple script called "vzs" which simply greps for the VEID, and spits out the next 20 or so lines (however many lines there are in the output, I forget) after it. For instance:<br />
<br />
<pre># vzs 765:<br />
765: kmemsize 2007936 2562780 8100000 8200000 0<br />
lockedpages 0 8 322 322 0<br />
privvmpages 26925 71126 610000 615000 0<br />
shmpages 16654 16750 33000 34500 0<br />
dummy 0 0 0 0 0<br />
numproc 41 57 410 415 0<br />
physpages 1794 49160 0 2147483647 0<br />
vmguarpages 0 0 13019 2147483647 0<br />
oomguarpages 4780 51270 13019 2147483647 0<br />
numtcpsock 23 37 1210 1215 0<br />
numflock 17 39 107 117 0<br />
numpty 1 3 19 19 0<br />
numsiginfo 0 6 274 274 0<br />
tcpsndbuf 22240 333600 1800000 1900000 0<br />
tcprcvbuf 0 222656 1800000 1900000 0<br />
othersockbuf 104528 414944 900000 950000 0<br />
dgramrcvbuf 0 4448 200000 200000 0<br />
numothersock 73 105 650 660 0<br />
dcachesize 247038 309111 786432 818029 0<br />
numfile 904 1231 7500 7600 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
numiptent 4 4 155 155 0</pre><br />
<br />
That showed us just the portion of /proc/user_beancounters for system 765.<br />
<br />
When you run the vzs command, always add a : after the VEID.<br />
<br />
So, if a customer complains about some out of memory errors, or no more files, or no more ptys, or just has an unspecific complain about processes dying, etc., the very first thing you need to do is check their beancounters with vzs. Usually you will spot an item that has a high failcount and needs to be upped.<br />
<br />
At that point you could simply up the counter with `vzctl set`. Generally pick a number 10-20% higher than the old one, and make the hard limit slightly larger than the the soft limit. However our systems now come in several levels and those levels have more/different memory allocations. If someone is complaining about something other than a memory limit (pty, numiptent, numflock), it’s generally safe to increase it, at least to the same level as what’s in the /vzconf/4unlimited file on the newest virt. If someone is hitting a memory limit, first make sure they are given what they deserve:<br />
<br />
(refer to mgmt -> payments -> packages)<br />
<br />
To set those levels, you use the [[#setmem|setmem]] command. <br />
<br />
The alternate (DEPRECATED) method would be to use one of 3 commands:<br />
256 <veid><br />
300 <veid><br />
384 <veid><br />
512 <veid><br />
<br />
If the levels were not right (you’d run vzs <veid> before and after to see the effect) tell the customer they’ve been adjusted and be done with it. If the levels were right, tell the customer they must upgrade to a higher package, tell them how to see level (control panel) and that they can reboot their system to escape this lockup contidion.<br />
<br />
Customers can also complain that their site is totally unreachable, or complain that it is down ... if the underlying machine is up, and all seems well, you may notice in the beancounters that network-specific counters are failing - such as numtcpsock, tcpsndbuf or tcprcvbuf. This will keep them from talking on the network and make it seem like their system is down. Again, just up the limits and things should be fine.<br />
<br />
On virts 1-4, you should first look at the default settings for that item on a later virt, such as virt 8 - we have increased the defaults a lot since the early machines. So, if you are going to up a counter on virt2, instead of upping it by 10-20%, instead up it to the new default that you see on virt8.<br />
<br />
== Moving a VE to another virt (migrate/migrateonline) ==<br />
<br />
This will take a while to complete - and it is best to do this at night when the load is light on both machines.<br />
<br />
There are different methods for this, depending on which version of virtuozzo is installed on the src. and dst. virt. <br />
To check which version is running: <br />
[root@virt12 private]# cat /etc/virtuozzo-release<br />
Virtuozzo release 2.6.0<br />
<br />
Ok, let's say that the VE is 1212, and vital stats are:<br />
<pre>[root@virt12 sbin]# vc 1212<br />
VE_ROOT="/vz1/root/1212"<br />
VE_PRIVATE="/vz1/private/1212"<br />
OSTEMPLATE="fedora-core-2/20040903"<br />
IP_ADDRESS="69.55.229.84"<br />
TEMPLATES="devel-fc2/20040903 php-fc2/20040813 mysql-fc2/20040812 postgresql-fc2/20040813 mod_perl-fc2/20040812 mod_ssl-fc2/20040811 jre-fc2/20040823 jdk-fc2/20040823 mailman-fc2/20040823 analog-fc2/20040824 proftpd-fc2/20040818 tomcat-fc2/20040823 usermin-fc2/20040909 webmin-fc2/20040909 uw-imap-fc2/20040830 phpBB-fc2/20040831 spamassassin-fc2/20040910 PostNuke-fc2/20040824 sl-webalizer-fc2/20040<br />
818"<br />
<br />
[root@virt12 sbin]# vzctl exec 1212 df -h<br />
Filesystem Size Used Avail Use% Mounted on<br />
/dev/vzfs 4.0G 405M 3.7G 10% /</pre><br />
<br />
From this you can see that he’s using (and will minimally need free on the dst server) ~400MB, and he’s running on a Fedora 2 template, version 20040903. He’s also got a bunch of other templates installed. It’s is '''vital''' that '''all''' these templates exist on the dst system. To confirm that, on the dst system run:<br />
<br />
For < 3.0:<br />
<br />
<pre>[root@virt14 private]# vzpkgls | grep fc2<br />
devel-fc2 20040903<br />
PostNuke-fc2 20040824<br />
analog-fc2 20040824<br />
awstats-fc2 20040824<br />
bbClone-fc2 20040824<br />
jdk-fc2 20040823<br />
jre-fc2 20040823<br />
mailman-fc2 20040823<br />
mod_frontpage-fc2 20040816<br />
mod_perl-fc2 20040812<br />
mod_ssl-fc2 20040811<br />
mysql-fc2 20040812<br />
openwebmail-fc2 20040817<br />
php-fc2 20040813<br />
phpBB-fc2 20040831<br />
postgresql-fc2 20040813<br />
proftpd-fc2 20040818<br />
sl-webalizer-fc2 20040818<br />
spamassassin-fc2 20040910<br />
tomcat-fc2 20040823<br />
usermin-fc2 20040909<br />
uw-imap-fc2 20040830<br />
webmin-fc2 20040909<br />
[root@virt14 private]# vzpkgls | grep fedora<br />
fedora-core-1 20040121 20040818<br />
fedora-core-devel-1 20040121 20040818<br />
fedora-core-2 20040903<br />
[root@virt14 private]#</pre><br />
<br />
For these older systems, you can simply match up the date on the template. <br />
<br />
For >= 3.0:<br />
<br />
<pre>[root@virt19 /vz2/private]# vzpkg list<br />
centos-5-x86 2008-01-07 22:05:57<br />
centos-5-x86 devel<br />
centos-5-x86 jre<br />
centos-5-x86 jsdk<br />
centos-5-x86 mod_perl<br />
centos-5-x86 mod_ssl<br />
centos-5-x86 mysql<br />
centos-5-x86 php<br />
centos-5-x86 plesk9<br />
centos-5-x86 plesk9-antivirus<br />
centos-5-x86 plesk9-api<br />
centos-5-x86 plesk9-atmail<br />
centos-5-x86 plesk9-backup<br />
centos-5-x86 plesk9-horde<br />
centos-5-x86 plesk9-mailman<br />
centos-5-x86 plesk9-mod-bw<br />
centos-5-x86 plesk9-postfix<br />
centos-5-x86 plesk9-ppwse<br />
centos-5-x86 plesk9-psa-firewall<br />
centos-5-x86 plesk9-psa-vpn<br />
centos-5-x86 plesk9-psa-fileserver<br />
centos-5-x86 plesk9-qmail<br />
centos-5-x86 plesk9-sb-publish<br />
centos-5-x86 plesk9-vault<br />
centos-5-x86 plesk9-vault-most-popular<br />
centos-5-x86 plesk9-watchdog</pre><br />
<br />
On these newer systems, it's difficult to tell whether the template on the dst matches exactly the src. Just cause a centos-5-x86 is listed on both servers doesn't mean all the same packages are there on the dst. To truly know, you must perform a sample rsync:<br />
<br />
rsync -avn /vz/template/centos/5/x86/ root@10.1.4.61:/vz/template/centos/5/x86/<br />
<br />
if you see a ton of output from the dry run command, then clearly there are some differences. You may opt to let the rsync complete (without running in dry run mode) the only downside is you've now used up more space on the dst and also the centos template will be a mess with old and new data- it will be difficult if not impossible to undo (if someday we wanted to reclaim the space).<br />
<br />
If you choose to merge templates, you should closely inspect the dry run output. You should also take care to exclude anything in the /config directory. For example:<br />
<br />
rsync -av -e ssh --stats --exclude=x86/config /vz/template/ubuntu/10.04/ root@10.1.4.62:/vz/template/ubuntu/10.04/<br />
<br />
Which will avoid this directory and contents:<br />
<pre>[root@virt11 /vz2/private]# ls /vz/template/ubuntu/10.04/x86/config*<br />
app os</pre><br />
<br />
This is important to avoid since the config may differ on the destination and we are really only interested in making sure the pacakges are there, not overwriting a newer config with an older one.<br />
<br />
If the dst system was missing a template, you have 2 choices: <br />
# put the missing template on the dst system. 2 choices here: <br />
## Install the template from rpm (found under backup2: /mnt/data4/vzrpms/distro/) or <br />
## rsync over the template (found under /vz/template) - see above<br />
# put the ve on a system which has all the proper templates<br />
<br />
=== pre-seeding a migration ===<br />
<br />
When migrating a customer (or when doing many) depending on how much data you have to transfer, it can take some time. Further, it can be difficult to gauge when a migration will complete or how long it will take. To help speed up the process and get a better idea about how long it will take you can pre-transfer a customer's data to the destination server. If done correctly, vzmigrate will see the pre-transferred data and pick up where you left off, having much less to transfer (just changed/new files). <br />
<br />
We believe vzmigrate uses rsync to do it's transfer. Therefore not only can you use rsync to do a pre-seed, you can also run rsync to see what is causing a repeatedly-failing vzmigrate to fail. <br />
<br />
There's no magic to a pre-seed, you just need to make sure it's named correctly.<br />
<br />
Given:<br />
<br />
source: /vz1/private/1234<br />
<br />
and you want to migrate to /vz2 on the target system, your rsync would look like:<br />
<br />
rsync -av /vz1/private/1234/ root@x.x.x.x:/vz2/private/1234.migrated/<br />
<br />
After running that successful rsync, the ensuing migrateonline (or migrate) will take much less time to complete- depending on the # of files to be analyzed and the # of changed files. In any case, it'll be much much faster than had you just started the migration from scratch.<br />
<br />
Further, as we discuss elsewhere in this topic, a failed migration can be moved from <tt>/vz/private/1234</tt> to <tt>/vz/private/1234.migrated</tt> on the destination if you want to restart a failed migration. This should '''only''' be done if the migration failed and the CT is not running on the destination HN.<br />
<br />
=== migrateonline intructions: src >=3.x -> dst>=3.x ===<br />
<br />
A script called [[#migrateonline|migrateonline]] was written to handle this kind of move. It is basically a wrapper for <tt>vzmigrate</tt> – vzmigrate is a util to seamlessly- as no no reboot of the ve necessary- move a ve from one host to another. This wrapper was initially written cause vz virtuozzo version 2.6.0 has a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables, causing problems when the ve was started up on the dst system. [[#migrate|migrate]] mitigates that. Since it makes multiple ssh connections to the dst virt, it’s a good idea to put the pub key for the src virt in the authorized_keys file on the dst virt. In addition, migrateonline emails ve owners when their migration starts and stops. For this to happen they need to put email addresses (on a single line, space delimited) in a file on their system: /migrate_notify. If the optional target dir is not specified, the ve will be moved to the same private/root location as it was on the src virt. Note: <tt>migrate</tt> is equivalent to <tt>migrateonline</tt>, but will <tt>migrate</tt> a ve AND restart it in the process.<br />
<br />
<pre>[root@virt12 sbin]# migrateonline<br />
usage: /usr/local/sbin/migrateonline <ip of node migrating to> <veid> [target dir: vz | vz1 | vz2]<br />
[root@virt12 sbin]# migrateonline 10.1.4.64 1212 vz<br />
starting to migrate 1212 at Sat Mar 26 22:40:38 PST 2005<br />
Turning off offline_management<br />
Saved parameters for VE 1212<br />
migrating with no start on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
clearing the arp cache<br />
now going to 10.1.4.64 and clear cache and starting<br />
starting it<br />
Delete port redirection<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
finished migrating 1212 at Sat Mar 26 22 22:52:01 PST 2005<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine.<br />
<br />
If they had backups, use the mvbackups command to move their backups to the new server:<br />
<br />
mvbackups 1212 virt14 vz<br />
<br />
Rename the ve<br />
<br />
[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/migrated-1212<br />
<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/old-1212-migrated-20120404-noarchive<br />
<br />
Update the customer’s systems in mgmt to reflect the new path and server.<br />
<br />
<br />
IF migrateonline does not work, you can try again using simply migrate- this will result in a brief reboot for the ve.<br />
Before you try again, make sure of a few things:<br />
<br />
Depending on where in the migration died, there may be partial data on the dst system in 1 of 2 places:<br />
(given the example above)<br />
<br />
/vz/private/1212<br />
<br />
or <br />
<br />
/vz/private/1212.migrated<br />
<br />
before you run migrate again, you'll want to rename so that all data is in <br />
1212.migrated:<br />
<br />
mv /vz/private/1212 /vz/private/1212.migrated<br />
<br />
this way, it will pick up where it left off and transfer only new files.<br />
<br />
Likewise, if you want to speed up a migration, you can pre-seed the dst as follows:<br />
<br />
[root@virt12 sbin]# rsync -avSH /vz/private/1212/ root@10.1.4.64:/vz/private/1212.migrated/<br />
<br />
then when you run migrate or migrateonline, it will only need to move the changed files- the migration will complete quickly<br />
<br />
=== migrateonline/migrate failures (migrate manually) ===<br />
<br />
Lets say for whatever reason the migration fails. If it fails with [[#migrateonline|migrateonline]], you should try [[#migrate|migrate]] (which will reboot the customer, so notify them ahead of time).<br />
<br />
You may want to run a [[#pre-seeding_a_migration|pre-seed]] rsync to see if you can find the problem. On older virts, we've seen this problem due to a large logfile (which you can find and encourage the customer to remove/compress):<br />
for f in `find / -size +1048576k`; do ls -lh $f; done<br />
<br />
You may also see migration failing due to quota issues.<br />
<br />
You can try to resolve by copying any quota file into the file you need:<br />
<br />
cp /var/vzquota/quota.1 /var/vzquota/quota.xxx<br />
<br />
If it complains about quota running you should then be able to stop it<br />
<br />
vzquota off xxxx<br />
<br />
If all else fails, migrate to a new VEID<br />
i.e. 1234 becomes 12341<br />
<br />
<br />
If the rsync or [[#migrate|migrate]] fails, you can always move someone manually:<br />
<br />
1. stop ve: <br><br />
v stop 1234<br />
<br />
2. copy over data<br><br />
rsync -avSH /vz/private/1234/ root@1.1.1.1:/vzX/private/1234/<br />
<br />
NOTE: if you've previously seeded the data (run rsync while the VE was up/running), and this is a subsequent rsync, make sure the last rsync you do (while the VE is not running, has the --delete option in the rsync)<br />
<br />
3. copy over conf<br><br />
scp /vzconf/1234.conf root@1.1.1.1:/vzconf<br />
<br />
4. on dst, edit the conf to reflect the right vzX dir<br><br />
vi /vzconf/1234.conf<br />
<br />
5. on src remove the IPs<br><br />
ipdel 1234 2.2.2.2 3.3.3.3<br />
<br />
6. on dst add IPs <br><br />
ipadd 1234 2.2.2.2 3.3.3.3<br />
<br />
7. on dst, start ve: <br><br />
v start 1324<br />
<br />
8. cancel, then archive ve on src per above instrs.<br />
<br />
=== migrate src=2.6.0 -> dst>=2.6.0, or mass-migration with customer notify ===<br />
<br />
A script called <tt>migrate</tt> was written to handle this kind of move. It is basically a wrapper for vzmigrate – vzmigrate is a util to seamlessly move a ve from one host to another. This wrapper was initially written cause vz virtuozzo version 2.6.0 has a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables, causing problems when the ve was started up on the dst system. migrate mitigates that. Since it makes multiple ssh connections to the dst virt, it’s a good idea to put the pub key for the src virt in the authorized_keys file on the dst virt. In addition, migrate emails ve owners when their migration starts and stops. For this to happen they need to put email addresses (on a single line, space delimited) in a file on their system: /migrate_notify. If the optional target dir is not specified, the ve will be moved to the same private/root location as it was on the src virt. Note: migrateonline is equivalent to migrate, but will migrate a ve from one 2.6 '''kernel''' machine to another 2.6 kernel machine without restarting the ve.<br />
<br />
<pre>[root@virt12 sbin]# migrate<br />
usage: /usr/local/sbin/migrate <ip of node migrating to> <veid> [target dir: vz | vz1 | vz2]<br />
[root@virt12 sbin]# migrate 10.1.4.64 1212 vz<br />
starting to migrate 1212 at Sat Mar 26 22:40:38 PST 2005<br />
Turning off offline_management<br />
Saved parameters for VE 1212<br />
migrating with no start on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
clearing the arp cache<br />
now going to 10.1.4.64 and clear cache and starting<br />
starting it<br />
Delete port redirection<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
finished migrating 1212 at Sat Mar 26 22 22:52:01 PST 2005<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine (backup2 or mail).<br />
Cancel the ve (first we have to rename things which migrate changed so cancelve will find them):<br />
<br />
[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/1212.conf<br />
<br />
On 2.6.1 you’ll also have to move the private area:<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/1212<br />
<br />
<pre>[root@virt12 sbin]# cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, [[#cancelve|cancelve]] would offer to remove them. You want to say '''no''' to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
To move everyone off a system, you’d do:<br />
for f in `vl`; do migrate <ip> $f; done<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
=== vzmigrate: src=2.6.1 -> dst>=2.6.0 ===<br />
<br />
This version of vzmigrate works properly with regard to handling ips. It will not notify ve owners of moves as in the above example. Other than that it’s essentially the same.<br />
<br />
<pre>[root@virt12 sbin]# vzmigrate 10.1.4.64 -r no 1212:1212:/vz/private/1212:/vz/root/1212<br />
migrating on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine (backup2 or mail).<br />
Cancel the ve (first we have to rename things which vzmigrate changed so cancelve will find them):<br />
<br />
<pre>[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/1212.conf<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/1212<br />
<br />
[root@virt12 sbin]# cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, <tt>cancelve</tt> would offer to remove them. You want to say no to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
=== src=2.5.x ===<br />
<br />
First, go to the private dir:<br />
<br />
cd /vz1/private/<br />
<br />
Stop the VE - make sure it stops totally cleanly.<br />
<br />
vzctl stop 1212<br />
<br />
Then you’d use vemove - a script written to copy over the config, create tarballs of the ve’s data on the destination virt, and cancel the ve on the source system (in this example we’re going to put a ve that was in /vz1/private on the src virt, in /vz/private on the dst virt):<br />
<br />
<pre>[root@virt12 sbin]# vemove<br />
ERROR: Usage: vemove veid target_ip target_path_dir<br />
[root@virt12 sbin]# vemove 1212 10.1.4.64 /vz/private/1212<br />
tar cfpP - 1212 --ignore-failed-read | (ssh -2 -c arcfour 10.1.4.64 "split - -b 1024m /vz/private/1212.tar" )<br />
scp /vzconf/1212.conf 10.1.4.64:/vzconf<br />
cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, cancelve would offer to remove them. You want to say no to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
When you are done, go to /vz/private on the dst virt you will have files like this:<br />
<br />
<pre>1212.taraa<br />
1212.tarab<br />
1212.tarac</pre><br />
<br />
Each one 1024m (or less, for the last one) in size.<br />
<br />
on the dst server and run:<br />
<br />
cat 1212.tar?? | tar xpPBf -<br />
<br />
and after 20 mins or so it will be totally untarred. Now since the conf<br />
file is already there, you can go ahead and start the system.<br />
<br />
vzctl start 1212<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
NOTE: you MUST tar the system up using the virtuozzo version of tar that<br />
is on all the virt systems, and further you MUST untar the tarball with<br />
the virtuozzo tar, using these options: `<tt>tar xpPBf -</tt>`<br />
<br />
If you tar up an entire VE and move it to a non-virtuozzo machine, that is<br />
ok, and you can untar it there with normal tar commands, but do not untar<br />
it and then repack it with a normal tar and expect it to work - you need<br />
to use virtuozzo tar commands on virtuozzo tarballs to make it work.<br />
<br />
The backups are sort of an exception, since we are just (usually)<br />
restoring user data that was created after we gave them the system, and<br />
therefore has nothing to do with magic symlinks or vz-rpms, etc.<br />
<br />
== Moving a VE on the same virt ==<br />
<br />
Easy way:<br><br />
Scenario 1: ve 123 is to be renamed 1231 and moved from vz1 to vz<br />
<br />
vzmlocal 123:1231:/vz/private/1231:/vz/root/1231<br />
<br />
Scenario 2: ve 123 is to be moved vz1 to vz<br />
<br />
vzmlocal 123:123:/vz/private/123:/vz/root/123<br />
<br />
vzmlocal will reboot the ve at the end of the move<br />
<br />
Manual/old way:<br />
<br />
1) <tt>vzctl stop 123</tt><br><br />
2) <tt>mv /vz1/private/123 /vz/private/.</tt><br><br />
(or cp -a if you want to copy)<br />
3) in <tt>/etc/sysconfig/vz-scripts/123.conf</tt> change value<br><br />
of '<tt>VE_PRIVATE</tt>' variable to point to a new private area location<br />
4) <tt>vzctl start 123</tt><br><br />
5) update backups if needed: <tt>mvbackups 123 virtX virt1 vz</tt><br><br />
6) update management scerens<br><br />
<br />
Notes: a) absolute path to private area is stored in quota file <tt>/var/vzquota/quota.123</tt> - so during first startup quota will be recalculated.<br><br />
b) if you're going to write some script to do a job, you MUST be sure that $VEID won't be expanded to '' in ve config file - ie. you need to escape '$'. Otherwise you might have:<br />
<br />
VE_PRIVATE="/vz/private/"<br />
<br />
in config, and 'vzctl destroy' for this VE ID '''will remove everything under /vz/private/ directory'''.<br />
<br />
== Adding a veth device to a VE ==<br />
<br />
Not totally sure what this is, but a customer asked for it and here's what we did (as instructed by vz support):<br />
<br />
<pre>v set 99 --netif_add eth99 --save<br />
ipdel 99 69.55.230.58<br />
v set 99 --ifname eth99 --ipadd 69.55.230.58 --save<br />
v set 99 --ifname eth99 --gateway 69.55.230.1 --save<br />
<br />
vznetcfg net new veth_net<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
net99 active eth0 veth77.77,veth99.99<br />
veth_net active<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
br0 bridge veth_net<br />
br99 bridge net99<br />
veth99.99 veth net99<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic net99 69.55.227.70/24<br />
<br />
vznetcfg br attach br0 eth0<br />
<br />
(will remove 99 from orig net and move to veth_net)<br />
vznetcfg net addif veth_net veth99.99<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
net99 active<br />
veth_net active eth0 veth77.77,veth99.99<br />
<br />
(delete the old crap)<br />
vznetcfg net del net99<br />
<br />
<br />
Then, to add another device in<br />
<br />
v set 77 --netif_add eth77 --save<br />
ipdel 77 69.55.230.78<br />
v set 77 --ifname eth77 --ipadd 69.55.230.78 --save<br />
v set 77 --ifname eth77 --gateway 69.55.230.1 --save<br />
v set 77 --save --ifname eth77 --network veth_net<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
veth77.77 veth<br />
br0 bridge veth_net<br />
veth99.99 veth veth_net<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic veth_net 69.55.227.70/24<br />
<br />
vznetcfg net addif veth_net veth77.77<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
veth77.77 veth veth_net<br />
br0 bridge veth_net<br />
veth99.99 veth veth_net<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic veth_net 69.55.227.70/24<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
veth_net active eth0 veth77.77,veth99.99<br />
<br />
another example<br />
<br />
v set 1182 --netif_add eth1182 --save<br />
ipdel 1182 69.55.236.217<br />
v set 1182 --ifname eth1182 --ipadd 69.55.236.217 --save<br />
v set 1182 --ifname eth1182 --gateway 69.55.236.1 --save<br />
vznetcfg net addif veth_net veth1182.1182<br />
v set 1182 --save --ifname eth1182 --network veth_net<br />
<br />
<br />
unused/not working commands:<br />
ifconfig veth99.0 0<br />
vznetcfg net list<br />
vznetcfg br new br99 net99<br />
vznetcfg br attach br99 eth0<br />
vznetcfg br show<br />
vznetcfg if list<br />
vznetcfg net addif net99 veth99.99<br />
vznetcfg net addif eth0 net99<br />
<br />
---------<br />
<br />
vznetcfg br new br1182 net1182<br />
<br />
vznetcfg br attach br99 eth0<br />
vznetcfg if list<br />
vznetcfg net addif net99 veth99.99<br />
vznetcfg net addif eth0 net99<br />
<br />
<br />
<br />
vznetcfg net addif eth0 net1182<br />
<br />
<br />
# vznetcfg net addif veth99.0 net99<br />
--- 8< ---<br />
<br />
vznetcfg net new net<br />
# vznetcfg net addif eth0 net99<br />
<br />
<br />
# vzctl set 99 --save --netif_add eth0 (at this stage veth99.0 interface have to appear<br />
on node)<br />
# vzctl set 99 --save --ifname eth0 --ipadd 69.55.230.58 (and probably few more arguments<br />
here - see 'man vzctl')<br />
# vznetcfg net addif veth99.0 net99</pre><br />
<br />
<br />
== Assigning/remove ip from a VE ==<br />
<br />
1. Add or remove ips:<br />
ipdel 1234 1.1.1.1 2.2.2.2<br />
ipadd 1234 1.1.1.1 2.2.2.2<br />
<br />
2. update Mgmt screens<br />
<br />
3. offer to update any DNS we do for them<br />
<br />
4. check to see if we had rules for old IP in firwall<br />
<br />
== Enabling tun device for a ve ==<br />
Note, there’s a command for this: [[#addtun|addtun]]<br />
<br />
For posterity:<br />
Make sure the tun.o module is already loaded before Virtuozzo is started: <br />
lsmod <br />
Allow the VPS to use the TUN/TAP device: <br />
vzctl set 101 --devices c:10:200:rw --save <br />
Create the corresponding device inside the VPS and set the proper permissions: <br />
vzctl exec 101 mkdir -p /dev/net <br />
vzctl exec 101 mknod /dev/net/tun c 10 200 <br />
vzctl exec 101 chmod 600 /dev/net/tun<br />
<br />
== Remaking a system (on same virt) ==<br />
<br />
1. [[#cancelve|cancelve]] (or v destroy x - ONLY if you're POSITIVE no data needs to be saved)<br />
<br />
2. [[#vemake|vemake]] using same veid<br />
<br />
3. [[#mvbackups|mvbackups]] or [[#vb|vb]] (if new mount point)<br />
<br />
4. update mgmt with new dir/ip <br />
<br />
5. update firewall if changed ip<br />
<br />
== Re-initialize quota for a VE ==<br />
<br />
There’s a commamd for this now: [[#clearquota|clearquota]]<br />
<br />
For posterity:<br />
<br />
vzctl stop 1<br />
vzquota drop 1<br />
vzctl start 1<br />
<br />
<br />
== Traffic accounting on linux ==<br />
<br />
DEPRECATED - all tracking is done via bwdb now. This is how we used to track traffic.<br />
<br />
TODO: update for diff versions of vz<br />
<br />
Unlike FreeBSD, where we have to add firewall count rules to the system to count the traffic, on virtuozzo counts the traffic for us. You an see the current traffic stats by running `vznetstat`:<br />
<br />
<pre># vznetstat<br />
VEID Net.Class Output(bytes) Input(bytes)<br />
-----------------------------------------------<br />
24218 1 484M 39M<br />
24245 1 463M 143M<br />
2451 1 2224M 265M<br />
2454 1 2616M 385M<br />
4149 1 125M 68M<br />
418 1 1560M 34M<br />
472 1 1219M 315M<br />
726 1 628M 317M<br />
763 1 223M 82M<br />
771 1 4234M 437M<br />
-----------------------------------------------<br />
Total: 13780M 2090M</pre><br />
<br />
As you can see the VEID is on a line with the in and out bytes. So, we simply run a cron job:<br />
<br />
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /root/vztrafdump.sh<br />
<br />
Just like we do on FreeBSD - this one goes through all the VEs in /vz/private and greps the line from vznetstat that matches them and dumps it in /jc_traffic_dump on their system. Then it does it again for all the VEs in /vz1/private. It is important to note that vznetstat runs only once, and the grepping is done from a temporary file that contains that output - we do this because running vznetstat once for each VE that we read out of /vz/private and /vz1/private would take way too long and be too intensive.<br />
<br />
You do not need to do anything to facilitate this other than make sure that that cron job is running - the vznetstat counters are always running, and any new VEs that are added to the system will be accounted for automatically.<br />
<br />
Traffic resetting no longer works with vz 2.6, so we disable the vztrafdump.sh on those virts.<br />
<br />
== Watchdog script ==<br />
<br />
On some of the older virts, we have a watchdog running that kills procs that are deemed bad per the following:<br />
<br />
/root/watchdog from quar1<br />
<br />
<pre>if echo $line | awk '{print $(NF-3)}' | grep [5-9]...<br />
then<br />
# 50-90%<br />
if echo $line | awk '{print $(NF-1)}' | grep "...:.."<br />
then<br />
# running for > 99min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
if echo $line | awk '{print $(NF-1)}' | grep "....m"<br />
then<br />
# running for > 1000min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
fi<br />
<br />
if echo $line | awk '{print $(NF-3)}' | grep [1-9]...<br />
then<br />
# running for 10-90 percent<br />
if echo $line | awk '{print $NF}' | egrep 'cfusion|counter|vchkpw'<br />
then<br />
<br />
if echo $line | awk '{print $(NF-1)}' | grep "[2-9]:.."<br />
then<br />
# between 2-9min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
elif echo $line | awk '{print $(NF-1)}' | grep "[0-9][0-9]:.."<br />
then<br />
# up to 99min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
fi<br />
fi</pre><br />
<br />
<br />
== Misc Linux Items ==<br />
<br />
We are overselling hard drive space ... when you configure a linux system with a certain amount of disk space (the default is 4gigs) you do not actually use up 4gigs of space on the system. The diskspace setting for a user is simply a cap, and they only use up as much space on the actual disk drive as they are actually using.<br />
<br />
When you create a new linux system, even though there are some 300 RPMs or so installed, if you run `df -k` you will see that the entire 4gig partition is empty - no space is being used. This is because the files in their system are "magic symlinks" to the template for their OS that is in /vz/template - however, any changes to any of those files will "disconnect" them and they will immediately begin using space in their system. Further, any new files uploaded (even if those new files overwrite existing files) will take up space on the partition.<br />
<br />
<br />
if you see this:<br />
<br />
<pre>[root@virt8 root]# vzctl stop 160 ; vzctl start 160<br />
VE is not running<br />
Starting VE ...<br />
VE is unmounted<br />
VE is mounted<br />
Adding IP address(es): 69.55.226.83 69.55.226.84<br />
bash ERROR: Can't change file /etc/sysconfig/network<br />
Deleting IP address(es): 69.55.226.83 69.55.226.84<br />
VE is unmounted<br />
[root@virt8 root]#</pre><br />
<br />
it probably means they no longer have /bin/bash - copy one in for them<br />
<br />
ALSO: another possibility is that they have removed the `ed` RPM from their system - it needs to be reinstalled into their system. But since their system is down, this is tricky ...<br />
<br />
VE startup scripts used by 'vzctl' want package 'ed' to be available inside VE. So if package 'ed' will be enabled in OS template config and OS template itself VE #827 is based on - this error should be fixed.<br />
<br />
yes, it is possible to add RPM to VE while it not running.<br />
Try to do following:<br />
<br />
<pre># cd /vz/template/<OS_template_with_ed_package>/<br />
# vzctl mount 827<br />
# rpm -Uvh --root /vz/root/827 --veid 827 ed-0.2-25.i386.vz.rpm</pre><br />
<br />
Usually theres an error, but its ok<br />
<br />
Note: replace 'ed-0.2-25.i386.vz.rpm' in last command with actual<br />
version of 'ed' package you have.<br />
<br />
----<br />
<br />
So how do I know what template the user has ? cat their conf file and it is listed in there. For example, if the conf file has:<br />
<br />
<pre>[root@virt12 sbin]# vc 1103<br />
…snip…<br />
OSTEMPLATE="debian-3.0/20030822"<br />
TEMPLATES="mod_perl-deb30/20030707 mod_ssl-deb30/20030703 mysql-deb30/20030707 proftpd-deb30/20030703 webmin-deb30/20030823 "<br />
…</pre><br />
<br />
then they are on debian 3.0, all of their system RPMs are in /vz/template/debian-3.0, and they are using version 20030822 of that debian 3.0 template. Also, they’ve also got additional packages installed (mod_perl, mod_ssl, etc). Those are also found under /vz/template<br />
<br />
----<br />
<br />
Edits needed to run java:<br />
<br />
When we first created the VEs, the default setting for privvmpages was 93000:94000 ... which was high enough that most people never had problems ... however, you can;t run java or jdk or tomcat or anything java related with that setting. We have found that by setting privvmpages to 610000:615000 that java runs just fine. That is now the default setting. It is exceedingly rare that anyone needs it higher than that, although we have seen it once or twice.<br />
<br />
Any problems with java at all - the first thing you need to do is see if the failcnt has raised for privvmpages.<br />
<br />
<pre># vzctl start 160<br />
Starting VE ...<br />
vzquota : (error) Quota on syscall for 160: Device or resource busy<br />
Running vzquota on failed for VE 160 [3]</pre><br />
<br />
This is because my pwd is _in_ their private directory - you can't start it until you move out<br />
<br />
People seem to have trouble with php if they are clueless newbies. Here are two common problems/solutions:<br />
<br />
no... but i figured it out myself. problem was the php.ini file that came<br />
vanilla with the account was not configured to work with apache (the<br />
ENGINE directive was set to off).<br />
<br />
everything else seems fine now.<br />
<br />
----<br />
<br />
the problem was in the php.ini file. I noticed that is wasnt showing<br />
the code when it was in an html file so I looked at the php.ini file<br />
and had to change it so it recognized <? tags aswell as <?php tags.<br />
<br />
Also, make sure added to httpd.conf<br />
AddType application/x-httpd-php .php<br />
<br />
----<br />
<br />
You can set the time zone:<br />
<br />
You can change the timezone by doing this:<br />
<br />
ln -sf /usr/share/zoneinfo/<zone> /etc/localtime<br />
<br />
where <zone> is the zone you want in the /usr/share/zoneinfo/ directory.<br />
<br />
----<br />
<br />
Failing shm_open calls:<br />
<br />
first, please check if /dev/shm is mounted inside VE.<br />
'cat /proc/mounts' command should show something like this:<br />
tmpfs /dev/shm tmpfs rw 0 0<br />
<br />
If /dev/shm is not mounted you have 2 ways to solve issue:<br />
1. execute following command inside VE (doesn't require VE reboot):<br />
mount -t tmpfs none /dev/shm<br />
2. add following string to /etc/fstab inside VE and reboot it:<br />
tmpfs /dev/shm tmpfs defaults 0 0<br />
<br />
----<br />
<br />
You can have a mounted but not running ve<br />
Just:<br />
vzctl mount <veid><br />
<br />
----<br />
<br />
When a debian sys can’t get on the network, and you try:<br />
<br />
<pre>vzctl set 1046 --ipadd 69.55.227.117<br />
Adding IP address(es): 69.55.227.117<br />
Failed to bring up lo.<br />
Failed to bring up venet0.</pre><br />
<br />
They probably removed iproute package, which must be the one from swsoft. To restore:<br />
<pre># dpkg -i --veid=1046 --admindir=/vz1/private/1046/root/var/lib/dpkg --instdir=/vz1/private/1046/root/ /vz/template/debian-3.0/iproute_20010824-8_i386.vz.deb<br />
(Reading database ... 16007 files and directories currently installed.)<br />
Preparing to replace iproute 20010824-8 (using .../iproute_20010824-8_i386.vz.deb) ...<br />
Unpacking replacement iproute ...<br />
Setting up iproute (20010824-8) ...</pre><br />
<br />
Then restart their ve<br />
<br />
----<br />
<br />
in a ve i do:<br />
<br />
<pre>cd /<br />
du -h .</pre><br />
<br />
and get: 483M .<br />
<br />
i do:<br />
<br />
<pre>bash-2.05a# df -h<br />
Filesystem Size Used Avail Use% Mounted on<br />
/dev/vzfs 4.0G 2.3G 1.7G 56% /</pre><br />
<br />
how can this be?<br />
<br />
Is it possible that quota file was corrupted somehow? Please try to: <br />
<pre>vzctl stop <VEID><br />
vzquota drop <VEID><br />
vzquota init <VEID><br />
vzctl start <VEID></pre><br />
<br />
----<br />
<br />
How to stop vz from starting after reboot:<br />
<br />
VIRTUOZZO=no <br />
in <br />
/etc/sysconfig/vz<br />
<br />
To start: <br />
service vz start<br />
(after setting VIRTUOZZO=yes in /etc/sysconfig/vz)<br />
<br />
service vz restart will do some kind of 'soft reboot' -- restart all<br />
VPSes and reload modules without rebooting the node<br />
<br />
if you need to shut down all VPSes really really fast, run killall -9 init<br />
<br />
----<br />
<br />
Postfix tip:<br />
<br />
You may want to tweak settings: default_process_limit=10<br />
<br />
= Virtuozzo VPS Management Tools =<br />
<br />
== vm ==<br />
<br />
TODO<br />
<br />
== cancelve ==<br />
cancelve <veid><br />
this will:<br />
* stop a ve<br />
* check for backups (offer to remove them from the backup server <br />
and the backup.config)<br />
* rename the private dir<br />
* check for PTR, provide the commands to reset to default<br />
* and rename the ve’s config<br />
* remind you to remove firewall rules<br />
* remind you to remove DNS entries<br />
<br />
== ipadd ==<br />
ipadd <veid> <ip> [ip] [ip]<br />
add’s ip(s) to a ve<br />
<br />
== ipdel ==<br />
ipdel <veid> <ip> [ip] [ip]<br />
removes ip(s) from a ve<br />
<br />
== vc ==<br />
vc <veid><br />
the equivalent of: <tt>cat /vzconf/<veid>.conf</tt><br />
<br />
== vl ==<br />
vl<br />
will displays a list of ve #’s, 1 per line. (ostensibly to use in a for loop)<br />
<br />
== vp ==<br />
vp <veid><br />
the equivalent of: <tt>vzps auxww –E <veid></tt><br />
<br />
== vpe ==<br />
DEPRECATED<br />
vpe <veid> <br />
this will allow you to do a vp when a ve is running out of control, the equivalent of (deprecated since vp operates outside the VPS): <br />
<pre>vzctl set <veid> --kmemsize 2100000:2200000<br />
vzctl exec <veid> ps auxw<br />
vzctl set <veid> --kmemsize (ve’s orig lvalue):(ve’s orig hvalue)</pre><br />
<br />
== vt ==<br />
vt <veid><br />
the equivalent of: <tt>vztop –E <veid></tt><br />
<br />
== vr ==<br />
vr <veid><br />
the equivalent of: <tt>vzctl stop <veid>; vzctl start <veid></tt><br />
You can run this even if the ve is down - the stop command will just fail<br />
<br />
== vs ==<br />
vs [veid]<br />
displays status (output of <tt>vzctl status <veid></tt>) on each ve configured on the system (as determined by <tt>/vzconf/*.conf</tt>)<br />
If passed an argument, gives the status for just that ve. <br />
A running system looks like:<br />
<tt>VEID 16066 exist mounted running</tt><br />
<br />
A ve which is not running (but does exist) looks like:<br />
<tt>VEID 9990 exist unmounted down</tt><br />
<br />
A ve which is not running and doesn’t exist looks like:<br />
<tt>VEID 421 deleted unmounted down</tt><br />
<br />
== vs2 ==<br />
vs2 [veid]<br />
this is similar to vs in that it displays status (output of <tt>vzctl status <veid></tt>) on each ve,<br />
but the difference is it’s list comes from doing an ls on the data dirs. This was meant to catch <br />
the rare case where a ve configured but exists. <br />
<br />
== vw ==<br />
vw [veid]<br />
displays the output of ‘<tt>w</tt>’ (the equivalent of <tt>vzctl exec <veid> w</tt>) for each configured ve (as determined by <tt>/vzconf/*.conf</tt>). Useful for determing which ve is contributing to a heavily-loaded system.<br />
If passed an argument, gives ‘<tt>w</tt>‘ output for just that ve. <br />
Ex:<br />
<pre>[root@virt2 etc]# vw<br />
134<br />
10:52pm up 79 days, 6:14, 0 users, load average: 0.02, 0.02, 0.00<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT<br />
16027<br />
2:52pm up 7 days, 19:54, 0 users, load average: 0.00, 0.00, 0.00<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT<br />
16055<br />
2:52pm up 79 days, 6:38, 0 users, load average: 0.00, 0.04, 0.07<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT</pre><br />
<br />
== vwe ==<br />
vwe [constraint]<br />
just like <tt>vw</tt>, but takes a constraint as an argument, only show’s ve’s with loads >= the constraint provided. If no constraint is provided, 1 is used by default<br />
<br />
== vzs ==<br />
vzs [veid]<br />
displays the beancounter status for all ve’s, or a particular ve if an argument is passed<br />
<br />
== ve ==<br />
ve <veid><br />
the equivalent of: <tt>vzctl enter <veid></tt><br />
<br />
== vx ==<br />
vx <veid> <command><br />
the equivalent of: <tt>/usr/sbin/vzctl exec <veid> <command></tt><br />
<br />
== dprocs ==<br />
dprocs [count]<br />
a script which outputs a continuous report (or a certain number of reports if an option is passed) of processes stuck in the D state and which VPS’s those procs belong to.<br />
<br />
== setmem ==<br />
setmem <VEID> <256|512|768|1024|2048><br />
adjusts the memory resources for the VE<br />
<br />
== afacheck.sh ==<br />
afacheck.sh<br />
displays the health/status of containers and mirrors on an adaptec card (currently quar1, tempvirt1-2, virt9, virt10)- all other are LSI<br />
<br />
== backup ==<br />
backup<br />
backup script called nightly to update virt scripts and do customer backups<br />
<br />
== checkload.pl ==<br />
checkload.pl<br />
this was intended to be setup as a cronjob to watch processes on a virt when the load <br />
rises above a certain level. Not currently in use.<br />
<br />
== findbackuppigs.pl ==<br />
findbackuppigs.pl<br />
looks for files larger than 50MB which customers have asked us to backup. Emails matches<br />
to linux@johncompanies.com<br />
<br />
== gatherlinux.pl ==<br />
gatherlinux.pl<br />
gathers up data about ve’s configured and writes to a file. Used for audits against the db<br />
<br />
== gb ==<br />
gb <search><br />
greps backup.config for the given search parameter<br />
<br />
== gbg ==<br />
gbg <search><br />
greps backup.config for the given search parameter and presents just the directories (for clean pasting)<br />
<br />
== linuxtrafficgather.pl ==<br />
linuxtrafficgather.pl [yy-mm]<br />
sends a traffic usage summary by ve to support@johncomapnies.com and payments@johncompanies.com.<br />
Optional arguments are year and month (must be in the past). If not passed, assumes last month. Relies on <br />
traffic logs created by netstatreset and netstatbackup<br />
<br />
== linuxtrafficwatch.pl ==<br />
linuxtrafficwatch.pl<br />
checks traffic usage and emails support@johncomapnies.com when a ve reaches the warning level (35G) and<br />
the limit (40G). Works on virtuozzo versions <= 2.6.x. We really aren’t using this anymore now that we have netflow.<br />
<br />
== linuxtrafficwatch2.pl ==<br />
linuxtrafficwatch2.pl<br />
checks traffic usage and emails support@johncomapnies.com when a ve reaches the warning level (35G) and<br />
the limit (40G). Works on virtuozzo version 2.6.x. We really aren’t using this anymore now that we have netflow.<br />
<br />
== load.pl ==<br />
load.pl<br />
feeds info to load mrtg - executed by inetd.<br />
<br />
== mb (linux) ==<br />
mb <mount|umount><br />
(nfs) mounts and umounts dirs to backup2. Shortcuts are mbm and mbu to mount and unmount. <br />
<br />
== migrate ==<br />
migrate <ip of node migrating to> <veid> <target_veid> [target dir: vz | vz1 | vz2]<br />
this is basically a wrapper for <tt>vzmigrate</tt> – vzmigrate is a util to seamlessly move a ve from one host to another. This wrapper was written cause vz virtuozzo version 2.6 had a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables. This script mitigates that. Since it makes multiple ssh connections to the target host, it’s a good idea to put the pub key for the src system in the authorized_keys file on the target host. In addition, it emails ve owners when their migration starts and stops (if they place email addresses in a file on their system: /migrate_notify). To move everyone off a system, you’d do:<br />
for f in `vl`; do migrate <ip> $f; done<br />
<br />
== migrateonline ==<br />
migrateonline <ip of node migrating to> <veid> <target_veid> [target dir: vz | vz1 | vz2]<br />
this is the same as migrate but will migrate a ve in <tt>–online</tt> mode which means it won’t be shut down at the end of the migration. This only works when migrating ve’s between 2 machines running a 2.6 kernel (currently tempvirt1-2. virt16-19, virt12). If you get an error that the machine you’re trying to migrate to has a different CPU or features, etc, then you have to edit the file and add the –f switch to the vzmigrate line- you can basically ignore this kind of warning (but never ignore a warning about missing templates on the destination node). NOTE: This edit (if made to migrateonline) will be overwritten by the base script during each night’s backup.<br />
<br />
== netstatbackup ==<br />
DEPRECATED<br />
netstatbackup <br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.5.x. <br />
<br />
== netstatbackup2 ==<br />
DEPRECATED<br />
netstatbackup2<br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.6.x. <br />
<br />
== netstatreset ==<br />
DEPRECATED<br />
netstatreset<br />
writes traffic count data to a logfile and resets counters to 0. Works on virtuozzo versions 2.5.x <br />
<br />
== netstatreset2 ==<br />
DEPRECATED<br />
netstatreset2<br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.6.x. <br />
<br />
== orphanedbackupwatchlinux ==<br />
orphanedbackupwatchlinux <br />
looks for directories on backup2 which aren’t configured in backup.config and offers to <br />
delete them<br />
<br />
== rsync.backup (linux) ==<br />
rsync.backup<br />
does customer backups (relies on backup.config)<br />
<br />
== startvirt.pl ==<br />
startvirt.pl<br />
forks off start ve commands – keeps 6 running at a time. This is not to be used on systems where fastboot is enabled as it circumvents the benefit of the fastboot. The script will occasionally not exit gracefully and will continue to use up CPU, so it should be watched. Also, don’t exit from the script till you’re sure all ve’s are started – if you do you need to start them manually and may have to free up locks. On some systems, the startvirt script doesn’t exit cleanly and you have to ^C out of it. Be careful though- doing so can leave some VE’s in an odd bootup state and you may need to ‘vr’ them manually. You should check to see which ve’s aren’t running and/or confirm all have started when ^C’ing out of startvirt.<br />
<br />
== taskdone (linux) ==<br />
taskdone<br />
when called will email support@johncompanies.com with the hostname of the machine from which it was <br />
executed as the subject<br />
<br />
== vb (linux) ==<br />
vb<br />
the equivalent of: <tt>vi /usr/local/sbin/backup.config</tt><br />
<br />
== vemakeXX ==<br />
DEPRECATED<br />
vemakerh9 <br />
ve create script for RH9 (see vemake)<br />
<br />
vemakedebian30 <br />
ve create script for debian 3.0 (Woody) (see vemake)<br />
<br />
vemakedebian31 <br />
ve create script for debian 3.1 (Sarge) (see vemake)<br />
<br />
vemakedebian40 <br />
ve create script for debian 4.0 (Etch) (see vemake)<br />
<br />
vemakefedora, vemakefedora2, vemakefedora4, vemakefedora5, vemakefedora6, vemakefedora7<br />
ve create script for fedora core 1, 2, 4, 5, 6, 7 respectively (see vemake)<br />
<br />
vemakecentos3, vemakecentos4<br />
ve create script for centos 3, 4 respectively (see vemake)<br />
<br />
vemakesuse, vemakesuse93, vemakesuse100<br />
ve create script for suse 9.2, 9.3, 10.0 respectively (see vemake)<br />
<br />
vemakeubuntu5, vemakeubuntu606, vemakeubuntu606 vemakeubuntu704<br />
ve create script for ubuntu 5.10, 6.06, 6.10, 7.04 respectively (see vemake)<br />
<br />
== vemove ==<br />
DEPRECATED<br />
vemove <veid> <target_ip> </vz/private/123><br />
this script simplifies the old way of moving ve’s from one system to another - in short moving a ve to or from a virt running virtuozzo < 2.6.x<br />
It’s the equivalent of:<br />
<tt>tar cfpP - <veid> --ignore-failed-read | (ssh -2 -c arcfour <target_ip> "split - -b 1024m </vz/private/123>.tar" )</tt>This should only be used if migrate/vzmigrate can’t be used. <br />
<br />
== vim.watchdog ==<br />
vim.watchdog <br />
looks for and kills procs matching “vi|vim|nano|pine|elm” that are running for a long time & consuming lots of cpu. Works on virtuozzo versions 2.5.x<br />
<br />
== vim.watchdog2 ==<br />
vim.watchdog2<br />
looks for and kills procs matching “vi|vim|nano|pine|elm” that are running for a long time & consuming lots of cpu.<br />
Works on virtuozzo versions 2.6.x.<br />
<br />
== vzmigrate ==<br />
vzmigrate <target_ip> -r no <veid>:[dst veid]:[dst /vzX/private/veid]:[dst /vzX/root/veid]<br />
(this is the raw command “wrapped” by migrate/migrateonline) this will seamlessly move a ve from one host to another. The ve will run for the duration of the migration till the very end when it’s shut down, ip moved and started up on the target system. The filesystem on the src will remain. This should be watched – occasionally the move will timeout and leave the system shut down. If target private and root aren’t specified it just puts it in /vz. Only works when both systems are running virtuozzo 2.6.x<br />
<br />
== vztrafdump.sh ==<br />
DEPRECATED<br />
vztrafdump.sh<br />
writes traffic usage info by ve to a file called jc_traffic_dump in each ve’s / dir. Works on virtuozzo versions <= 2.5.x. <br />
<br />
== vztrafdump2.sh ==<br />
DEPRECATED<br />
vztrafdump2.sh<br />
writes traffic usage info by ve to a file called jc_traffic_dump in each ve’s / dir. Works on virtuozzo versions 2.6.x. <br />
<br />
== addtun ==<br />
addtun <veid><br />
Add’s tun device to ve.<br />
<br />
== bwcap ==<br />
bwcap <veid> <kbps><br />
Ex: <tt>bwcap 1234 512</tt><br />
Caps a VE’s bandwidth to the amount given<br />
<br />
== setdisk ==<br />
setdisk <veid> <diskspace in GB><br />
Ex: <tt>setdisk 1234 5</tt><br />
Gives a VE’s a given amount of disk space<br />
<br />
== vdf ==<br />
vdf <veid> <br />
the equivalent of: <tt>vzctl exec <veid> df –h</tt><br />
<br />
== vdff ==<br />
vdff<br />
runs a (condensed) vdf for all ve’s in your pwd (must be run from /vz/privateN)<br />
<br />
== mvbackups ==<br />
mvbackups <veid> <target_machine> (virt1) <target_dir> (vz1)<br />
moves backups from one location to another on the backup server, and provides you with option to remove entries from current backup.config, and simple paste command to add the config to backup.config on the target server<br />
<br />
== checkquota ==<br />
checkquota<br />
for all the ve’s in the cwd (run from /vz/private, /vz1/private, etc) reports what vz quota says they’re using and what the actual usage is (as reported by du)<br />
<br />
== clearquota ==<br />
clearquota <veid><br />
Recalculates a ve’s quota, prints out the usage before and after. The equivalent of:<br />
<tt>vdf <veid>; v stop <veid>; vzquota drop <veid>; v start <veid>; vdf <veid></tt><br />
<br />
== dprocs ==<br />
dprocs<br />
Sometimes the server’s have a large number of processes get stuck in the D state- this script shows (every 3 secs) which VE’s have D procs, which procs<br />
are stuck and a running average of the top “offenders”<br />
<br />
== vzstat ==<br />
vstat<br />
sort of like top for VZ. sort VEs by CPU usage by pressing 'o' and then 'c' keys<br />
<br />
== stopvirt ==<br />
stopvirt<br />
will stop VEs as fast as it can, 6 at a time. May not exit when complete so you should watch [[#vzstat|vzstat]] in another window.</div>Supporthttps://wiki.jcihosting.com/index.php?title=VPS_Management&diff=1036VPS Management2013-03-11T23:49:00Z<p>Support: /* Virtuozzo VPS */</p>
<hr />
<div>= Common Problems =<br />
== Login to any machine without a password ==<br />
<br />
This is possible via the use of ssh keys. The process is thus:<br />
<br />
1. place the public key for your user (root@mail) in the /root/.ssh/authorized_keys file on the server you wish to login to<br />
cat /root/.ssh/id_dsa.pub<br />
(paste that into authorized_keys on the target server). If the file doesn't exist, create it.<br />
<br />
2. enable root login (usually only applies to FreeBSD). Edit the /etc/ssh/sshd_config on the target server and change:<br />
<tt>#PermitRootLogin no</tt><br />
to<br />
<tt>PermitRootLogin yes</tt><br />
<br />
3. Restart the sshd on the target machine. First, find the sshd process: <br />
jailps <hostname> | grep sshd <br />
or <br />
vp <VEID> | grep sshd<br />
<br />
Look for the process resembling:<br />
root 17296 0.0 0.0 5280 1036 ? Ss 2011 4:27 /usr/sbin/sshd <br />
(this is the sshd)<br />
<br />
Not:<br />
root 6270 0.5 0.0 6808 2536 ? Ss 14:33 0:00 sshd: root [priv]<br />
(this is an sshd child- someone already ssh'd in as root)<br />
<br />
Restart the sshd: <br />
kill -1 <PID><br />
<br />
Ex:<br />
kill -1 17296<br />
<br />
You may now ssh in.<br />
<br />
Once you're done, IF you enabled root login, you should repeat steps 2 and 3 to disable root logins.<br />
<br />
== Letting someone in who has locked themselves out (killed sshd, lost pwd) ==<br />
<br />
There are two ways people frequently lock themselves out - either they forget a password, or they kill off sshd somehow.<br />
<br />
These are actually both fairly easy to solve. First, let's say someone kills off their sshd, or somehow mangles /etc/ssh/sshd_config such that it no longer lets them in.<br />
<br />
Their email may be very short, or it may have all sorts of details about how you should fix sshd_config to let them in ... just ignore all of this. They can fix their own mangled sshd. Fixing this is very simple. First, edit the /etc/inetd.conf on their system and uncomment the telnet line:<br />
<br />
telnet stream tcp nowait root /usr/libexec/telnetd telnetd<br />
#telnet stream tcp6 nowait root /usr/libexec/telnetd telnetd<br />
<br />
(just leave the tcp6 version of telnet commented)<br />
<br />
Then, use jailps to list the processes on their system, and find their inetd process. Then simply:<br />
<br />
kill -HUP (pid)<br />
<br />
where (pid) is the PID of their inetd process. Now they have telnet running on their system and they can log in and do whatever they need to do.<br />
<br />
The only complications that could occur are:<br />
<br />
a) their firewall config on our firewall has port 23 blocked, in which case you will need to open that - will be covered in a different lesson.<br />
<br />
b) they are not running inetd, so you can't HUP it. If this happens, edit their /etc/rc.conf, add the inetd_enable="YES" line, and then kill<br />
their jail with /tmp/jailkill.pl - then restart their jail with the jail line from their quad/safe file. Easy.<br />
<br />
If they have forgotten a password,<br />
<br />
On 6.x+ you can reset their password with:<br />
jexec <jailID from jls> passwd root<br />
<br />
Note: the default password for 6.x jails is 8ico2987, for 4.x it is p455agfa<br />
<br />
On 4.x, you need to cd to their etc directory<br />
... for instance:<br />
<br />
cd /mnt/data2/198.78.65.136-col00261-DIR/etc<br />
<br />
and run:<br />
<br />
vipw -d .<br />
<br />
Then paste in these two lines (theres a paste with these):<br />
<br />
root:$1$krszPxhk$xkCepSnz3mIikT3vCtJCt0:0:0::0:0:Charlie &:/root:/bin/csh<br />
user:$1$Mx9p5Npk$QdMU6c8YQqp2FW2M3irEh/:1001:1001::0:0:User &:/home/user:/bin/sh<br />
<br />
overwriting the lines they already have for "user" and "root" - then just tell them that both user and root have been reset to the default password of p455agfa.<br />
<br />
For linux, just passwd inside shell or <br />
vzctl set <veid> --userpasswd root:p455agfa –save<br />
<br />
Starting in 2009 we began giving out randomized passwords for FreeBSD and Linux as the default password. That is stored with each system in Mgmt. You should look for and reset the password to that password in the event of a reset and refer the customer to use their original password from their welcome email- this way we don’t have to send the password again via email (in clear text).<br />
<br />
<br />
== sendmail can’t be contacted from ext ip (only locally) ==<br />
<br />
By default redhat puts this line in sendmail.mc:<br />
<br />
DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')<br />
<br />
which makes it only answer on localhost. Comment it out like:<br />
<br />
dnl DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')<br />
<br />
and then rebuild sendmail.cf with:<br />
<br />
m4 /etc/mail/sendmail.mc > /etc/sendmail.cf<br />
<br />
== virt doesn’t properly let go of ve’s ip(s) when moved to another system ==<br />
<br />
On virtuozzo 2.6 systems, it's been observed that when moving ips from one virt to another that sometimes the routing table will not get updated to reflect the removal of the ip addresses.<br />
<br />
A recent example was a customer that was moving to a new ve on a new virt and the ip addresses were traded between the two ve's. After the trade the two systems were not able to talk to each other. When looking at the routing table for the old system all the ip addresses were still in the routing table as being local, like so:<br />
<br />
<pre>netstat -rn | grep 69.55.225.149<br />
69.55.225.149 0.0.0.0 255.255.255.255 UH 40 0 0 venet0</pre><br />
<br />
This was preventing traffic to the other system from being routed properly.<br />
The solution is to manually delete the route:<br />
<br />
route delete 69.55.225.149 gw 0.0.0.0<br />
<br />
Supposedly, this was fixed in 2.6.1<br />
<br />
== sshd on FreeBSD 6.2 segfaults ==<br />
<br />
First try to reinstall ssh<br />
<br />
<pre>cd /usr/src/secure<br />
cd lib/libssh<br />
make depend && make all install<br />
cd ../../usr.sbin/sshd<br />
make depend && make all install<br />
cd ../../usr.bin/ssh<br />
make depend && make all install</pre><br />
<br />
Failing that, find the library that’s messed up:<br />
<br />
<pre>ldd /usr/sbin/sshd<br />
libssh.so.3 => /usr/lib/libssh.so.3 (0x280a3000) <br />
libutil.so.5 => /lib/libutil.so.5 (0x280d8000) <br />
libz.so.3 => /lib/libz.so.3 (0x280e4000) <br />
libwrap.so.4 => /usr/lib/libwrap.so.4 (0x280f5000) <br />
libpam.so.3 => /usr/lib/libpam.so.3 (0x280fc000) <br />
libbsm.so.1 => /usr/lib/libbsm.so.1 (0x28103000) <br />
libgssapi.so.8 => /usr/lib/libgssapi.so.8 (0x28112000) <br />
libkrb5.so.8 => /usr/lib/libkrb5.so.8 (0x28120000) <br />
libasn1.so.8 => /usr/lib/libasn1.so.8 (0x28154000) <br />
libcom_err.so.3 => /usr/lib/libcom_err.so.3 (0x28175000) <br />
libroken.so.8 => /usr/lib/libroken.so.8 (0x28177000) <br />
libcrypto.so.4 => /lib/libcrypto.so.4 (0x28183000) <br />
libcrypt.so.3 => /lib/libcrypt.so.3 (0x28276000) <br />
libc.so.6 => /lib/libc.so.6 (0x2828e000) <br />
libmd.so.3 => /lib/libmd.so.3 (0x28373000)</pre><br />
<br />
md5 them and compare to other jail hosts or jails running on host<br />
<br />
for libcrypto reinstall:<br />
<pre>/usr/src/crypto<br />
make depend && make all install</pre><br />
<br />
= FreeBSD VPS =<br />
<br />
== Starting jails: Quad/Safe Files ==<br />
<br />
FreeBSD customer systems do not start up automatically at boot time. When one of our freebsd machines boots up, it boots up, and does nothing else. To start jails, we put the commands to start each jail into a shell script(s) and run the script(s). Jail startup is something that needs to be actively monitored, which is why we don’t just run the script automatically. More on monitoring later.<br />
<br />
NOTE: >=7.x we have moved to 1 quad file: <tt>quad1</tt>. Startups are not done by running each quad, but rather [[#startalljails|startalljails]] which relies on the contents of <tt>quad1</tt>. The specifics of this are lower in this article. What follows here applies for pre 7.x systems.<br />
<br />
There are eight files in <tt>/usr/local/jail/rc.d</tt>:<br />
<br />
<pre>jail3# ls /usr/local/jail/rc.d/<br />
quad1 quad2 quad3 quad4 safe1 safe2 safe3 safe4<br />
jail3#</pre><br />
<br />
four quad files and four safe files.<br />
<br />
Each file contains an even number of system startup blocks (total number of jails divided by 4)<br />
<br />
The reason for this is, if we make one large script to startup all the systems at boot time, it will take too long - the first system in the script will start up right after system boot, which is great, but the last system may not start for another 20 minutes.<br />
<br />
Since there is no way to parralelize this during the startup procedure, we simply open four terminals (in screen window 9) and run each script, one in each terminal. This way they all run simultaneously, and the very last system in each startup script gets started in 1/4th the time it would if there was one large file<br />
<br />
The files are generally organized so that quad/safe 1&2 have only jails from disk 1, and quad/safe 3&4 have jails from disk 2. This helps ensure that only 2 fscks on any disk are going on at once. Further, they are balanced so that all quad/safe’s finish executing around the same time. We do this by making sure each quad/safe has a similar number of jails and represents a similar number of inodes (see js).<br />
<br />
The other, very important reason we do it this way, and this is the reason there are quad files and safe files, is that in the event of a system crash, every single vn-backed filesystem that was mounted at the time of system crash needs to be fsck'd. However, fsck'ing takes time, so if we shut the system down gracefully, we don't want to fsck.<br />
<br />
Therefore, we have two sets of scripts - the four quad scripts are identical to the four safe scripts except for the fact that the quad scripts contain fsck commands for each filesystem.<br />
<br />
So, if you shut a system down gracefully, start four terminals and run safe1 in window one, and safe2 in window 2, and so on.<br />
<br />
If you crash, start four terminals (or go to screen window 9) and run quad1 in window one, and quad2 in window 2, and so on.<br />
<br />
Here is a snip of (a 4.x version) quad2 from jail17:<br />
<br />
<pre>vnconfig /dev/vn16 /mnt/data2/69.55.228.7-col00820<br />
fsck -y /dev/vn16<br />
mount /dev/vn16c /mnt/data2/69.55.228.7-col00820-DIR<br />
chmod 0666 /mnt/data2/69.55.228.7-col00820-DIR/dev/null<br />
jail /mnt/data2/69.55.228.7-col00820-DIR mail1.phimail.com 69.55.228.7 /bin/sh /etc/rc<br />
<br />
# moved to data2 col00368<br />
#vnconfig /dev/vn28 /mnt/data2/69.55.236.132-col00368<br />
#fsck -y /dev/vn28<br />
#mount /dev/vn28c /mnt/data2/69.55.236.132-col00368-DIR<br />
#chmod 0666 /mnt/data2/69.55.236.132-col00368-DIR/dev/null<br />
#jail /mnt/data2/69.55.236.132-col00368-DIR limehouse.org 69.55.236.132 /bin/sh /etc/rc<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
vnconfig /dev/vn22 /mnt/data2/69.55.228.13-col01063<br />
fsck -y /dev/vn22<br />
mount /dev/vn22c /mnt/data2/69.55.228.13-col01063-DIR<br />
chmod 0666 /mnt/data2/69.55.228.13-col01063-DIR/dev/null<br />
jail /mnt/data2/69.55.228.13-col01063-DIR www.widestream.com.au 69.55.228.13 /bin/sh /etc/rc<br />
<br />
# cancelled col00106<br />
#vnconfig /dev/vn15 /mnt/data2/69.55.238.5-col00106<br />
#fsck -y /dev/vn15<br />
#mount /dev/vn15c /mnt/data2/69.55.238.5-col00106-DIR<br />
#chmod 0666 /mnt/data2/69.55.238.5-col00106-DIR/dev/null<br />
#jail /mnt/data2/69.55.238.5-col00106-DIR mail.azebu.net 69.55.238.5 /bin/sh /etc/rc</pre><br />
<br />
As you can see, two of the systems specified are commented out - presumably those customers cancelled, or were moved to new servers.<br />
<br />
As you can see, the vnconfig line is the simpler command line, not the longer one that was used when it was first configured. As you can see, all that is done is, vnconfig the filesystem, then fsck it, then mount it. The fourth command is the `jail` command used to start the system – but that will be covered later.<br />
<br />
Here is the safe2 file from jail17:<br />
<br />
<pre>vnconfig /dev/vn16 /mnt/data2/69.55.228.7-col00820<br />
mount /dev/vn16c /mnt/data2/69.55.228.7-col00820-DIR<br />
chmod 0666 /mnt/data2/69.55.228.7-col00820-DIR/dev/null<br />
jail /mnt/data2/69.55.228.7-col00820-DIR mail1.phimail.com 69.55.228.7 /bin/sh /etc/rc<br />
<br />
# moved to data2 col00368<br />
#vnconfig /dev/vn28 /mnt/data2/69.55.236.132-col00368<br />
#mount /dev/vn28c /mnt/data2/69.55.236.132-col00368-DIR<br />
#chmod 0666 /mnt/data2/69.55.236.132-col00368-DIR/dev/null<br />
#jail /mnt/data2/69.55.236.132-col00368-DIR limehouse.org 69.55.236.132 /bin/sh /etc/rc<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
vnconfig /dev/vn22 /mnt/data2/69.55.228.13-col01063<br />
mount /dev/vn22c /mnt/data2/69.55.228.13-col01063-DIR<br />
chmod 0666 /mnt/data2/69.55.228.13-col01063-DIR/dev/null<br />
jail /mnt/data2/69.55.228.13-col01063-DIR www.widestream.com.au 69.55.228.13 /bin/sh /etc/rc<br />
<br />
# cancelled col00106<br />
#vnconfig /dev/vn15 /mnt/data2/69.55.238.5-col00106<br />
#mount /dev/vn15c /mnt/data2/69.55.238.5-col00106-DIR<br />
#chmod 0666 /mnt/data2/69.55.238.5-col00106-DIR/dev/null<br />
#jail /mnt/data2/69.55.238.5-col00106-DIR mail.azebu.net 69.55.238.5 /bin/sh /etc/rc</pre><br />
<br />
As you can see, it is exactly the same, but it does not have the fsck lines.<br />
<br />
Take a look at the last entry - note that the file is named:<br />
<br />
/mnt/data2/69.55.238.5-col00106<br />
<br />
and the mount point is named:<br />
<br />
/mnt/data2/69.55.238.5-col00106-DIR<br />
<br />
This is the general format on all the FreeBSD systems. The file is always named:<br />
<br />
IP-custnumber<br />
<br />
and the directory is named:<br />
<br />
IP-custnumber-DIR<br />
<br />
If you run safe when you need a fsck, the mount will fail and jail will fail:<br />
<br />
# mount /dev/vn1c /mnt/data2/jails/65.248.2.131-ns1.kozubik.com-DIR<br />
mount: /dev/vn1c: Operation not permitted<br />
<br />
No reboot needed, just run the quad script<br />
<br />
Starting with 6.x jails, we added block delimiters to the quad/safe files, the block looks like:<br />
<br />
<pre>echo '## begin ##: nuie.solaris.mu'<br />
fsck -y /dev/concat/v30v31a<br />
mount /dev/concat/v30v31a /mnt/data1/69.55.228.218-col01441-DIR<br />
mount_devfs devfs /mnt/data1/69.55.228.218-col01441-DIR/dev<br />
devfs -m /mnt/data1/69.55.228.218-col01441-DIR/dev rule -s 3 applyset<br />
jail /mnt/data1/69.55.228.218-col01441-DIR nuie.solaris.mu 69.55.228.218 /bin/sh /etc/rc<br />
echo '## end ##: nuie.solaris.mu'</pre><br />
<br />
These are more than just informative when running quad/safe’s, the echo lines MUST be present for certain tools to work properly. So it’s important that any updates to the hostname also be updated on the 2 echo lines. For example, if you try to startjail a jail with a hostname which is on the jail line but not the echo lines, the command will return with host not found.<br />
<br />
=== FreeBSD 7.x+ notes ===<br />
<br />
Starting with the release of FreeBSD 7.x, we are doing jail startups in a slightly different way. First, thereis only 1 file: <tt>/usr/local/jail/rc.d/quad1</tt><br />
There are no other quads or corresponding safe files. The reason for this is twofold, 1. We can pass –C to fsck which will tell is to skip the fsck if the fs is clean (no more need for safe files), 2. We have a new startup script which can be launched multiple times, running in parallel to start jails, where quad1 is the master jail file. <br />
Quad1 could still be run as a shell script, but it would take a very long time for it to run completely so it’s not advisable; or you should break it down into smaller chunks (like quad1, quad2, quad3, etc)<br />
<br />
Here is a snip of (a 7.x version) quad1 from jail2:<br />
<br />
<pre>echo '## begin ##: projects.tw.com'<br />
mdconfig -a -t vnode -f /mnt/data1/69.55.230.46-col01213 -u 50<br />
fsck -Cy /dev/md50c<br />
mount /dev/md50c /mnt/data1/69.55.230.46-col01213-DIR<br />
mount -t devfs devfs /mnt/data1/69.55.230.46-col01213-DIR/dev<br />
devfs -m /mnt/data1/69.55.230.46-col01213-DIR/dev rule -s 3 applyset<br />
jail /mnt/data1/69.55.230.46-col01213-DIR projects.tw.com 69.55.230.46 /bin/sh /etc/rc<br />
echo '## end ##: projects.tw.com'</pre><br />
<br />
Cancelled jails are no longer commented out and stored in quad1, rather they’re moved to <tt>/usr/local/jail/rc.d/deprecated</tt> <br />
<br />
To start these jails, start the 4 ssh sessions as you would for a normal crash and then instead of running quad1-4, instead run startalljails in each window. IMPORTANT- before running startalljails you should make sure you ran preboot once as it will clear out all the lockfiles and enable startalljails to work properly.<br />
<br />
== Problems with the quad/safe files ==<br />
<br />
When you run the quad/safe files, there are two problems that can occur - either a particular system will hang during initialization, OR a system will spit out output to the screen, impeding your ability to do anything. Or both.<br />
<br />
First off, when you start a jail, you see output like this:<br />
<br />
<pre>Skipping disk checks ...<br />
adjkerntz[25285]: sysctl(put_wallclock): Operation not permitted<br />
Doing initial network setup:.<br />
ifconfig: ioctl (SIOCDIFADDR): permission denied<br />
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384<br />
Additional routing options: TCP keepalive=YESsysctl:<br />
net.inet.tcp.always_keepalive: Operation not permitted.<br />
Routing daemons:.<br />
Additional daemons: syslogd.<br />
Doing additional network setup:.<br />
Starting final network daemons:.<br />
ELF ldconfig path: /usr/lib /usr/lib/compat /usr/X11R6/lib /usr/local/lib<br />
a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout /usr/X11R6/lib/aout<br />
Starting standard daemons: inetd cron sshd sendmail sendmail-clientmqueue.<br />
Initial rc.i386 initialization:.<br />
Configuring syscons: blanktime.<br />
Additional ABI support:.<br />
Local package initialization:.<br />
Additional TCP options:.</pre><br />
<br />
Now, let's look at this line, near the end:<br />
<br />
Local package initialization:.<br />
<br />
This is where a list of daemons that are set to start at boot time willshow up. You might see something like:<br />
<br />
Local package initialization: mysqld apache sendmail sendmail-clientmqueue<br />
<br />
Or something like this:<br />
<br />
Local package initialization: postgres postfix apache<br />
<br />
The problem is that many systems (about 4-5 per machine) will hang on that line. Basically it will get to some of the way through the total daemons to be started:<br />
<br />
Local package initialization: mysqld apache<br />
<br />
and will just sit there. Forever.<br />
<br />
Fortunately, pressing ctrl-c will break out of it. Not only will it break out of it, but it will also continue on that same line and start the other daemons:<br />
<br />
Local package initialization: mysqld apache ^c sendmail-clientmqueue<br />
<br />
and then continue on to finish the startup, and then move to the next system to be started.<br />
<br />
So what does this mean? It means that if a machine crashes, and you start four screen-windows to run four quads or four safes, you need to periodically cycle between them and see if any systems are stuck at that point, causing their quad/safe file to hang. A good rule of thumb is, if you see a system at that point in the startup, give it another 100 seconds - if it is still at the exact same spot, hit ctrl-c. Its also a good idea to go back into the quad file (just before the first command in the jail startup block) and note that this jail tends to need a control-c or more time as follows:<br />
<pre>echo '### NOTE ### slow sendmail'<br />
echo '### NOTE ###: ^C @ Starting sendmail.'</pre><br />
<br />
'''NEVER''' hit ctrl-c repeatedly if you don't get an immediate response - that will cause the following jail’s startup commands to be aborted.<br />
<br />
A second problem that can occur is that a jail - maybe the first one in that particular quad/safe, maybe the last one, or maybe one in the middle, will start spitting out status or error messages from one of its init scripts. This is not a problem - basically, hit enter a few times and see if you get a prompt - if you do get a prompt, that means that the quad/safe script has already completed. Therefore it is safe to log out (and log out of the user that you su'd from) and then log back in (if necessary).<br />
<br />
The tricky thing is, if a system in the middle starts flooding with messages, and you hit enter a few times and don't get a prompt. Are you not getting a prompt because some subsequent system is hanging at the initialization, as we discussede above ? Or are you not getting a prompt because that quad file is currently running an fsck ? Usually you can tell by scrolling back in screen’s history to see what it was doing before you started getting the messages.<br />
<br />
If you don’t get clues from history, you have to use your judgement - instead of giving it 100 seconds to respond, perhaps give it 2-3 mins ... if you still get no response (no prompt) when you hit enter, hit ctrl-c. However, be aware that you might still be hitting ctrl-c in the middle of an fsck. This means you will get an error like "filesystem still marked dirty" and then the vnconfig for it will fail and so will the jail command, and the next system in the quad file will then start starting up.<br />
<br />
If this happens, just wait until the end of all the quad files have finished, and start that system manually.<br />
<br />
If things really get weird, like a screen flooded with errors, and you can't get a prompt, and ctrl-c does nothing, then you need to just eventually (give it ten mins or so) just kill that window with ctrl-p, then k, and then log in again and manually check which systems are now running and which aren't, and manually start up any that are not.<br />
<br />
Don't EVER risk running a particular quad/safe file a second time.<br />
If the quad/safe script gets executed twice, reboot the machine immediately.<br />
<br />
So, for all the above reasons, anytime a machine crashes and you run all the quads or all the safes, '''always''' check every jail afterwards to make sure it is running - even if you have no hangs or complications at all.<br />
Run this command:<br />
<br />
<tt>[[#jailpsall|jailpsall]]</tt> <br />
<br />
Note: [[#postboot|postboot]] also populates ipfw counts, so it '''should not be run multiple times''', use <tt>jailpsall</tt> for subsequent extensive ps’ing<br />
<br />
And make sure they all show as running. If one does not show as running, check its /etc/rc.conf file first to see if maybe it is using a different hostname first before starting it manually.<br />
<br />
One thing we have implemented to alleviate these startup hangs and noisy jails, is to put jail start blocks that are slow or hangy at the bottom of the safe/quad file. Further, for each bad jail we note in each quad/safe just before the start block something like:<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
<br />
That way we’ll be prepared to ^C when we see that message appear during the quad/safe startup process. If you observe a new, undocumented hang, '''after''' the quad/safe has finished, place a line similar to the above in the quad file, move the jail start block to the end of the file, then run [[#buildsafe|buildsafe]]<br />
<br />
<br />
== Recovering from a crash (FreeBSD) ==<br />
<br />
=== Diagnose whether you have a crash ===<br />
The most important thing is to get the machine and all jails back up as soon as possible. Note the time, you’ll need to create a crash log entry (Mgmt. -> Reference -> CrashLog). The first thing to do is head over to the [[Screen#Screen_Organization|serial console screen]] and see if there’s any kernel error messages output. Try to copy any messages (or just a sample of repeating messages) you see into the notes section of the crash log. If there are no messages, the machine may just be really busy- wait a bit (5-10min) to see if it comes back. If it's still pinging, odds are its very busy. Note, if you see messages about swap space exhausted, the server is obviously out of memory, however it may recover briefly enough for you to get a jtop in to see who's lauched a ton of procs (most likely) and then issue a quick jailkill to get it back under control.<br />
<br />
If it doesn't come back, or the messages indicate a fatal error, you will need to proceed with a power cycle (ctrl+alt+del will not work).<br />
<br />
=== Power cycle the server ===<br />
If this machine is not a Dell 2950 with a [[DRAC/RMM#DRAC|DRAC card]] (i.e. if you can’t ssh into the DRAC card (as root, using the standard root pass) and issue <br />
racadm serveraction hardreset<br />
then you will need someone at the data center power the macine off, wait 30 sec, then turn it back on. Make sure to re-attach via console:<br />
tip jailX<br />
immediately after power down. <br />
<br />
=== (Re)attach to the console ===<br />
Stay on the console the entire time during boot. As the BIOS posts- look out for the RAID card output- does everything look healthy? The output may be scrambled, look for "DEGRADED" or "FAILED". Once the OS starts booting you will be disconnected (dropped back to the shell on the console server) a couple times during the boot up. The reason you want to quickly re-attach is two-fold: 1. If you don’t reattach quickly then you won’t get any console output, 2. you want to be attached before the server ''potentially'' starts (an extensive) fsck. If you attach after the fsck begins, you’ll have seen no indication it started an fsck and the server will appear frozen during startup- no output, no response. <br />
<br />
IMPORTANT NOTE: on some older FreeBSD systems, there will be no output to the video (KVM) console as it boots up. The console output is redirected to the serial port ... so if a jail crashes, and you attach a kvm, the output during the bootup procedure will not be shown on the screen. However, when the bootup is done, you will get a login prompt on the screen and will be able to log in as normal. <tt>/boot/loader.conf</tt> is where serial console redirect output lives, so comment that if you want to catch output on kvm.<br />
On newer systems it sends most output to both locations. <br />
<br />
=== Assess the heath of the server ===<br />
Once the server boots up fully, you should be able to ssh in. Look around- make sure all the mounts are there and reporting the correct size/usage (i.e. /mnt/data1 /mnt/data2 /mnt/data3 - look in /etc/fstab to determine which mount points should be there), check to see if RAID mirrors are healthy. See [[RAID_Cards#Common_CLI_commands_.28megacli.29|megacli]], [[#aaccheck|aaccheck]]<br />
<br />
Before you start the jails, you need to run [[#preboot|preboot]]. This will do some assurance checks to make sure things are prepped to start the jails. Any issues that come out of preboot need to be addressed before starting jails.<br />
<br />
=== Start jails ===<br />
[[#Starting_jails:_Quad.2FSafe_Files|More on starting jails]]<br />
Customer jails (the VPSs) do not start up automatically at boot time. When a FreeBSD machines boots up, it boots up, and does nothing else. To start jails, we put the commands to start each jail into a shell script(s) and run the script(s). Jail startup is something that needs to be actively monitored, which is why we don’t just run the script automatically. <br />
<br />
In order to start jails, we run the quad files: quad1 quad2 quad3 and quad4 (on new systems there is only quad1). If the machine was cleanly rebooted- which wouldn't be the case if this was a crash, you may run the safe files (safe1 safe2 safe3 safe4) in lieu of quads. <br />
<br />
Open up 4 logins to the server (use the windows in [[Screen#Screen_Organization|a9]])<br />
In each of the 4 windows you will:<br />
<br />
If there is a [[#startalljails|startalljails]] script (and only quad1), run that command in each of the 4 windows. It will parse through the quad1 file and start each jail. Follow the instructions [[#Problems_with_the_quad.2Fsafe_files|here]] for monitoring startup. Note that you can be a little more lenient with jails that take awhile to start- startalljails will work around the slow jails and start the rest. As long as there aren't 4 jails which are "hung" during startup, the rest will get started eventually.<br />
-or-<br />
If there is no startalljails script, there will be multiple quad files. In each of the 4 windows, start each of the quads. i.e. start quad1 in window1, quad2 in window2 and so on. DO NOT start any quad twice. It will crash the server. If you accidentally do this, just jailkill all the jails which are in the quad and run the quad again. Follow the instructions here for monitoring quad startup.<br />
<br />
Note the time the last jail boots- this is what you will enter in the crash log.<br />
<br />
Save the crash log.<br />
<br />
=== Check to make sure all jails have started ===<br />
There's a simple script which will make sure all jails have started, and enter the ipfw counter rules: [[#postboot|postboot]] <br />
Run postboot, which will do a jailps on each jail it finds (excluding commented out jails) in the quad file(s). We're looking for 2 things:<br />
# systems spawning out of control or too many procs<br />
# jails which haven't started<br />
On 7.x and newer systems it will print out the problems (which jails haven't started) at the conclusion of postboot. <br />
On older systems you will need to watch closely to see if/when there's a problem, namely:<br />
<br />
[hostname] doesnt exist on this server<br />
<br />
When you get this message, it means one of 2 things:<br />
1. the jail really didn't start:<br />
When a jail doesn't start it usually boils down to a problem in the quad file. Perhaps the path name is wrong (data1 vs data2) or the name of the vn/mdfile is wrong. Once this is corrected, you will need to run the commands from the quad file manually, or you may use <tt>startjail <hostname></tt><br />
<br />
2. the customer has changed their hostname (and not told us) so their jail ''is'' running, just under a different hostname:<br />
On systems with jls, this is easy to rectify. First, get the customer info: <tt>g <hostname></tt><br />
Then look for the customer in jls: <tt>jls | grep <col0XXXX></tt><br />
From there you will see their new hostname- you should update that hostname in the quad file: don't forget to edit it on the <tt>## begin ##</tt> and <tt>## end ##</tt> lines, and in mgmt. <br />
On older systems without jls, this will be harder, you will need to look further to see their hostname- perhaps its in their /etc/rc.conf<br />
<br />
<br />
Once all jails are started, do some spot checks- try to ssh or browse to some customers, just to make sure things are really ok.<br />
<br />
== Adding disk to a 7.x/8.x jail ==<br />
or<br />
== Moving customer to a different drive (md) ==<br />
<br />
NOTE: this doesn’t apply to mx2 which uses gvinum. Use same procedure as 6.x<br />
NOTE: if you unmount before mdconfig, re-mdconfig (attach) then unmount then mdconfig -u again <br />
<br />
<br />
(parts to change/customize are <tt><span style="color:red">red</span></tt>)<br />
<br />
<br />
If someone wants more disk space, there’s a paste for it, send it to them (explains about downtime, etc).<br />
<br />
1. Figure out the space avail from <tt>js</tt>. Ideally, you want to put the customers new space on a different partition (and create the new md on the new partition). <br />
<br />
<br />
2. make a mental note of how much space they're currently using<br />
<br />
<br />
3. <tt>jailkill <hostname></tt> <br />
<br />
<br />
4. Umount it (including their devfs) but leave the md config’d (so if you use stopjail, you will have to re-mdconfig it)<br />
<br />
<br />
5. <tt>g <customerID></tt> to get the info (IP/cust#) needed to feed the new mdfile and mount name, and to see the current md device. <br />
<br />
<br />
6a. When there's enough room to place new system on an alternate, or the same drive:<br />
USE CAUTION not to overwrite (touch, mdconfig) existing md!!<br><br />
<tt>touch /mnt/data<span style="color:red">3/69.55.234.66-col01334</span><br><br />
mdconfig -a -t vnode -s 10g -f /mnt/data3/69.55.234.66-col01334 -u 97<br><br />
newfs /dev/md97</tt><br />
<br />
<br />
Optional- if new space is on a different drive, move the mount point directory AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
<tt>mount /dev/md<span style="color:red">97</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<br />
confirm you are mounted to /dev/md<span style="color:red">97</span> and space is correct:<br><br />
<tt>df .</tt> <br />
<br />
<br />
do the dump and pipe directly to restore:<br><br />
<tt>dump -0a -f - /dev/md<span style="color:red">1</span> | restore -r -f - <br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use <tt>df</tt> to confirm the restored data size matches the original usage figure<br />
<br />
<br />
md-unconfig old system:<br><br />
<tt>mdconfig -d -u <span style="color:red">1</span></tt><br />
<br />
<br />
archive old mdfile. <br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.237.26-col00241</span> /mnt/data<span style="color:red">1/old-col00241-mdfile-noarchive-20091211</span></tt><br />
<br />
<br />
edit quad (vq1) to point to new (/mnt/data<span style="color:red">3</span>) location AND new md number (md<span style="color:red">97</span>)<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
6b. When there's not enough room on an alternate partition, or on the same drive...but there is enough room if you were to remove the existing customer's space:<br />
<br />
<br />
mount backup nfs mounts:<br><br />
<tt>mbm</tt><br> <br />
(run <tt>df</tt> to confirm backup mounts are mounted)<br />
<br />
<br />
dump the customer to backup2 or backup1<br><br />
<tt>dump -0a -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> /dev/md<span style="color:red">1</span></tt><br><br />
(when complete WITHOUT errors, <tt>du</tt> the dump file to confirm it matches size, roughly, with usage)<br />
<br />
<br />
unconfigure and remove old mdfile<br><br />
<tt>mdconfig -d -u <span style="color:red">1</span><br><br />
rm /mnt/data<span style="color:red">1/69.55.237.26-col00241</span></tt><br><br />
(there should now be enough space to recreate your bigger system. If not, run sync a couple times)<br />
<br />
<br />
create the new system (ok to reuse old mdfile and md#):<br><br />
<tt>touch /mnt/data<span style="color:red">1/69.55.234.66-col01334</span><br><br />
mdconfig -a -t vnode -s <span style="color:red">10</span>g -f /mnt/data<span style="color:red">1/69.55.234.66-col01334</span> -u <span style="color:red">1</span><br><br />
newfs /dev/md<span style="color:red">1</span><br><br />
mount /dev/md<span style="color:red">1</span> /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> <br><br />
cd /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<br />
confirm you are mounted to /dev/md<span style="color:red">1</span> and space is correct:<br><br />
<tt>df .</tt> <br />
<br />
<br />
do the restore from the dumpfile on the backup server:<br><br />
<tt>restore -r -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> .<br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
umount nfs:<br><br />
<tt>mbu</tt><br />
<br />
<br />
If md# changed (or mount point), edit quad (<tt>vq1</tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new md number (md<span style="color:red">1</span>)<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
7. update disk (and dir if applicable) in mgmt screen<br />
<br />
<br />
8. update backup list AND move backups, if applicatble<br />
<br />
Ex: <tt>mvbackups <span style="color:red">69.55.237.26-col00241</span> jail<span style="color:red">9</span> data<span style="color:red">3</span></tt><br />
<br />
<br />
9. Optional: archive old mdfile<br />
<br />
<tt>mbm<br><br />
gzip -c old-col01588-mdfile-noarchive-20120329 > /deprecated/old-col01588-mdfile-noarchive-20120329.gz<br><br />
mbu<br><br />
rm old-col01588-mdfile-noarchive-20120329</tt><br />
<br />
== Adding disk to a 6.x jail (gvinum/gconcat) ==<br />
or<br />
== Moving customer to a different drive (gvinum/gconcat) ==<br />
<br />
(parts to change are <span style="color:red">highlighted</span>)<br />
<br />
<br />
If someone wants more disk space, there’s a paste for it, send it to them (explains about downtime, etc).<br />
<br />
<br />
1. Figure out the space avail from [[#js|js]]. Ideally, you want to put the customers new space on a different partition (and create the new md on the new partition). <br />
<br />
<br />
2. make a mental note of how much space they're currently using<br />
<br />
<br />
3. <tt>[[#stopjail|stopjail]] <hostname></tt> <br />
<br />
<br />
4. <tt>[[#g|g]] <customerID></tt> to get the info (IP/cust#) needed to feed the new mount name and existing volume/device. <br />
<br />
<br />
5a. When there's enough room to place new system on an alternate, or the same drive (using only UNUSED - including if it's in use by the system in question - gvinum volumes):<br />
<br />
<br />
Configure the new device:<br><br />
A. for a 2G system (single gvinum volume):<br><br />
<tt>bsdlabel -r -w /dev/gvinum/v<span style="color:red">123</span><br><br />
newfs /dev/gvinum/v<span style="color:red">123</span>a</tt><br><br />
<br />
-or- <br />
<br />
<br />
B. for a >2G system (create a gconcat volume):<br><br />
try to grab a contiguous block of gvinum volumes. gconcat volumes MAY NOT span drives (i.e. you cannot use a gvinum volume from data3 and a volume from data2 in the same gconcat volume). <br />
<br />
<tt>gconcat label <span style="color:red">v82-v84 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84</span><br><br />
bsdlabel -r -w /dev/concat/<span style="color:red">v82-v84</span><br><br />
newfs /dev/concat/<span style="color:red">v82-v84</span>a</tt><br />
<br />
<br />
Other valid gconcat examples:<br><br />
<tt>gconcat label v82-v84v109v112 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84 /dev/gvinum/v109 /dev/gvinum/v112<br><br />
gconcat label v82v83 /dev/gvinum/v82 /dev/gvinum/v83</tt> <br><br />
Note, long names will truncate: v144v145v148-v115 will truncate to v144v145v148-v1 (so you will refer to it as v144v145v148-v1 thereafter)<br />
<br />
<br />
Optional- if new volume is on a different drive, move the mount point directory (get the drive from js output) AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
<br />
confirm you are mounted to the device (<tt>/dev/gvinum/v<span style="color:red">123</span>a</tt> OR <tt>/dev/concat/<span style="color:red">v82-v84</span></tt>) and space is correct:<br><br />
A. <tt>mount /dev/gvinum/v<span style="color:red">123</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br><br />
-or-<br><br />
B. <tt>mount /dev/concat/<span style="color:red">v82-v84</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<tt>cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
df .</tt> <br />
<br />
<br />
do the dump and pipe directly to restore:<br><br />
<tt>dump -0a -f - /dev/gvinum/v<span style="color:red">1</span> | restore -r -f - <br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
edit quad (<tt>vq<span style="color:red">1</span></tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new volume (<tt>/dev/gvinum/v<span style="color:red">123</span>a</tt> or <tt>/dev/concat/<span style="color:red">v82-v84</span>a</tt>) , run <tt>buildsafe</tt><br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
5b. When there's not enough room on an alternate partition, or on the same drive...but there is enough room if you were to remove the existing customer's space (i.e. if you want/need to reuse the existing gvinum volumes and add on more):<br />
<br />
<br />
mount backup nfs mounts:<br><br />
<tt>mbm</tt> <br><br />
(run df to confirm backup mounts are mounted)<br />
<br />
<br />
dump the customer to backup2 or backup1<br><br />
<tt>dump -0a -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> /dev/<span style="color:red">gconcat/v106-v107</span></tt><br><br />
(when complete WITHOUT errors, du the dump file to confirm it matches size, roughly, with usage)<br />
<br />
<br />
unconfigure the old gconcat volume<br><br />
list member gvinum volumes:<br />
<br />
<tt>gconcat list <span style="color:red">v106v107</span></tt><br />
<br />
<br />
Output will resemble:<br><br />
<pre>Geom name: v106v107<br />
State: UP<br />
Status: Total=2, Online=2<br />
Type: AUTOMATIC<br />
ID: 3530663882<br />
Providers:<br />
1. Name: concat/v106v107<br />
Mediasize: 4294966272 (4.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e2<br />
Consumers:<br />
1. Name: gvinum/sd/v106.p0.s0<br />
Mediasize: 2147483648 (2.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e3<br />
Start: 0<br />
End: 2147483136<br />
2. Name: gvinum/sd/v107.p0.s0<br />
Mediasize: 2147483648 (2.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e3<br />
Start: 2147483136<br />
End: 4294966272</pre><br />
<br />
stop volume and clear members<br><br />
<tt>gconcat stop <span style="color:red">v106v107</span><br><br />
gconcat clear <span style="color:red">gvinum/sd/v106.p0.s0 gvinum/sd/v107.p0.s0</span></tt><br />
<br />
<br />
create new device- and its ok to reuse old/former members<br><br />
try to grab a contiguous block of gvinum volumes. gconcat volumes MAY NOT span drives (i.e. you cannot use a gvinum volume from data3 and a volume from data2 in the same gconcat volume). <br><br />
<tt>gconcat label <span style="color:red">v82-v84v106v107 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84 /dev/gvinum/v106 /dev/gvinum/v107</span> <br><br />
bsdlabel -r -w /dev/concat/<span style="color:red">v82-v84v106v107</span><br><br />
newfs /dev/concat/<span style="color:red">v82-v84v106v107</span>a</tt><br />
<br />
<br />
Optional- if new volume is on a different drive, move the mount point directory (get the drive from js output) AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
confirm you are mounted to the device (/dev/concat/<span style="color:red">v82-v84v106v107</span>) and space is correct:<br><br />
<tt>mount /dev/concat/<span style="color:red">v82-v84v106v107</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<tt>cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
df .</tt> <br />
<br />
do the restore from the dumpfile on the backup server:<br><br />
<tt>restore -r -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> .<br><br />
rm restoresymtable</tt><br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
edit quad (<tt>vq<span style="color:red">1</span></tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new volume (<tt>/dev/concat/<span style="color:red">v82-v84v106v107</span>a</tt>), run buildsafe<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
TODO: clean up/clear old gvin/gconcat vol<br />
<br />
<br />
6. update disk (and dir if applicable) in mgmt screen<br />
<br />
<br />
7. update backup list AND move backups, if applicatble<br />
<br />
<br />
Ex: <tt>mvbackups <span style="color:red">69.55.237.26-col00241</span> jail<span style="color:red">9</span> data<span style="color:red">3</span></tt><br />
<br />
<br />
<br />
DEPRECATED - steps to tack on a new gvin to existing gconcat- leads to corrupted fs<br />
bsdlabel -e /dev/concat/v82-v84<br />
<br />
To figure out new size of the c partition, multiply 4194304 by the # of 2G gvinum volumes and subtract the # of 2G volumes:<br />
10G: 4194304 * 5 – 5 = 20971515<br />
8G: 4194304 * 4 – 4 = 16777212<br />
6G: 4194304 * 3 – 3 = 12582909<br />
4G: 4194304 * 2 – 2 = 8388606<br />
<br />
To figure out the new size of the a partition, subtract 16 from the c partition:<br />
10G: 20971515 – 16 = 20971499<br />
8G: 16777212 – 16 = 16777196<br />
6G: 12582909 – 16 = 12582893<br />
4G: 8388606 – 16 = 8388590<br />
<br />
Orig:<br />
8 partitions:<br />
# size offset fstype [fsize bsize bps/cpg]<br />
a: 8388590 16 4.2BSD 2048 16384 28552<br />
c: 8388606 0 unused 0 0 # "raw" part, don't edit<br />
<br />
New:<br />
8 partitions:<br />
# size offset fstype [fsize bsize bps/cpg]<br />
a: 12582893 16 4.2BSD 2048 16384 28552<br />
c: 12582909 0 unused 0 0 # "raw" part, don't edit<br />
<br />
sync; sync<br />
<br />
growfs /dev/concat/v82-v84a<br />
<br />
fsck –fy /dev/concat/v82-v84a<br />
<br />
sync<br />
<br />
fsck –fy /dev/concat/v82-v84a<br />
<br />
(keep running fsck’s till NO errors)<br />
<br />
== Problems un-mounting - and with mount_null’s ==<br />
<br />
If you cannot unmount a filesystem, beacuse it says the filesystem is busy, it is because of three things:<br />
<br />
a) the jail is still running<br />
<br />
b) you are actually in that directory, even though the jail is stopped<br />
<br />
c) there are still dev, null_mount or linprocfs mount points mounted inside that directory.<br />
<br />
d) when trying to umount null_mounts that are really long and you get an error like “No such file or directory”, it’s an OS bug where the dir is truncated. No known fix<br />
<br />
e) there are still files open somewhere inside the dir. Use <tt>fstat | grep <cid></tt> to find the process that has files open<br />
<br />
f) Starting with 6.x, the jail mechanism does a poor job of keeping track of processes running in a jail and if it thinks there are still procs running, it will refuse to umount the disk. If this is happening you should see a low number in the #REF column when you run jls. In this case you ''can'' safely <tt>umount –f</tt> the mount. <br />
<br />
Please note -if you forcibly unmount a (4.x) filesystem that has null_mounts<br />
still mounted in it, the system '''will crash''' within 10-15 mins.<br />
<br />
== Misc jail Items ==<br />
<br />
We are overselling hard drive space on jail2, jail8, jail9, a couple jails on jail17, jail4, jail12 and jail18.<br />
Even though the vn file shows 4G size, it doesn’t actually occupy that amount of space on the disk. So be careful not to fill up drives where we’re overselling – use oversellcheck to confirm you’re not oversold by more than 10G.<br />
There are other truncated jails, they are generally noted in a the file on the root system: /root/truncated<br />
<br />
The act of moving a truncated vn to another system un-does the truncating- the truncated vn is filled with 0’s and it occupies physical disk space for which it’s configured. So, you should use dumpremote to preserve the truncation.<br />
<br />
= FreeBSD VPS Management Tools =<br />
<br />
These files are located in /usr/local/jail/rc.d and /usr/local/jail/bin<br />
<br />
== jailmake ==<br />
<br />
Applies to 7.x+ <br />
On older systems syntax differs, run jailmake once to see.<br />
<br />
Note: this procedure differs on mx2 which is 7.x but still uses gvinum<br />
<br />
# run js to figure out which md’s are in use, which disk has enough space, IP to put it on<br />
# use col00xxx for both hostnames if they don’t give you a hostname<br />
# copy over dir, ip and password to pending customer screen<br />
<br />
<tt>Usage: jailmake IP[,IP] CID disk[1|2|3] md# hostname shorthost ipfw# email [size in GB]</tt><br />
<br />
Ex: <br />
<br />
Jail2# jailmake 69.55.234.66 col01334 3 97 vps.bsd.it vps 1334 fb@bsd.it<br />
<br />
== jailps ==<br />
jailps [hostname]<br />
DEPRECATED FOR jps: displays processes belonging to/running inside a jail. The command<br />
takes one (optional) argument – the hostname of the jail you wish to query. If you don’t <br />
supply an argument, all processes on the machine are listed and grouped by jail. <br />
<br />
== jps ==<br />
jps [hostname]<br />
displays processes belonging to/running inside a jail. The command<br />
takes one (optional) argument – the hostname or ID of the jail you wish to query. <br />
<br />
== jailkill ==<br />
jailkill <hostname><br />
stops all process running in a jail.<br />
<br />
You can also run:<br />
jailkill <JID><br />
<br />
=== problems ===<br />
Occasionally you will hit an issue where jail will not kill off:<br />
<br />
<pre>jail9# jailkill www.domain.com<br />
www.domain.com .. killed: none<br />
jail9#</pre><br />
Because no processes are running under that hostname. You cannot use jailps.pl either:<br />
<br />
<pre>jail9# jailps www.domain.com<br />
www.domain.com doesn’t exist on this server<br />
jail9#</pre><br />
<br />
The reasons for this are usually:<br />
* the jail is no longer running<br />
<br />
* the jail's hostname has changed<br />
In this case, <br />
<br />
>=6.x: run a <tt>jls|grep <jail's IP></tt> to find the correct hostname, then update the quad file, then kill the jail.<br />
<br />
<6.x: the first step is to cat their /etc/rc.conf file to see if you can tell what they set the new hostname to. This very often works. For example:<br />
<br />
cat /mnt/data2/198.78.65.136-col00261-DIR/etc/rc.conf<br />
<br />
But maybe they set the hostname with the hostname command, and the original hostname is still in /etc/rc.conf.<br />
<br />
The welcome email clearly states that they should tell us if they change their hostname, so there is no problem in just emailing them and asking them what they set the new hostname to.<br />
<br />
Once you know the new hostname OR if a customer simply emails to inform you that they have set the hostname to something different, you need to edit the quad and safe files that their system is in to input the new hostname.<br />
<br />
However, if push comes to shove and you cannot find out the hostname from them or from their system, then you need to start doing some detective work.<br />
<br />
The easiest thing to do is run jailps looking for a hostname similar to their original hostname. Or you could get into the /bin/sh shell by running:<br />
<br />
/bin/sh<br />
<br />
and then looking at every hostname of every process:<br />
<br />
for f in `ls /proc` ; do cat /proc/$f/status ; done<br />
<br />
and scanning for a hostname that is either similar to their original hostname, or that you don't see in any of the quad safe files.<br />
<br />
This is very brute force though, and it is possible that catting every file in /proc is dangerous - I don't recommend it. A better thing would be to identify any processes that you know belong to this system – perhaps the reason you are trying to find this system is because they are running something bad - and just catting the status from only that PID.<br />
<br />
Somewhere there’s a jail where there may be 2 systems named www. Look at /etc/rc.conf and make sure they’re both really www. If they are, jailkill www, jailps www to make sure not running. Then immediately restart the other one, as the fqdn (as found from a rev nslookup)<br />
<br />
* on >=6.x the hostname may not yet be hashed:<br />
<pre>jail9 /# jls<br />
JID Hostname Path IP Address(es)<br />
1 bitnet.dgate.org /mnt/data1/69.55.232.50-col02094-DIR 69.55.232.50<br />
2 ns3.hctc.net /mnt/data1/69.55.234.52-col01925-DIR 69.55.234.52<br />
3 bsd1 /mnt/data1/69.55.232.44-col00155-DIR 69.55.232.44<br />
4 let2.bbag.org /mnt/data1/69.55.230.92-col00202-DIR 69.55.230.92<br />
5 post.org /mnt/data2/69.55.232.51-col02095-DIR 69.55.232.51 ...<br />
6 ns2 /mnt/data1/69.55.232.47-col01506-DIR 69.55.232.47 ...<br />
7 arlen.server.net /mnt/data1/69.55.232.52-col01171-DIR 69.55.232.52<br />
8 deskfood.com /mnt/data1/69.55.232.71-col00419-DIR 69.55.232.71<br />
9 mirage.confluentforms.com /mnt/data1/69.55.232.54-col02105-DIR 69.55.232.54 ...<br />
10 beachmember.com /mnt/data1/69.55.232.59-col02107-DIR 69.55.232.59<br />
11 www.agottem.com /mnt/data1/69.55.232.60-col02109-DIR 69.55.232.60<br />
12 sdhobbit.myglance.org /mnt/data1/69.55.236.82-col01708-DIR 69.55.236.82<br />
13 ns1.jnielsen.net /mnt/data1/69.55.234.48-col00204-DIR 69.55.234.48 ...<br />
14 ymt.rollingegg.net /mnt/data2/69.55.236.71-col01678-DIR 69.55.236.71<br />
15 verse.unixlore.net /mnt/data1/69.55.232.58-col02131-DIR 69.55.232.58<br />
16 smcc-mail.org /mnt/data2/69.55.232.68-col02144-DIR 69.55.232.68<br />
17 kasoutsuki.w4jdh.net /mnt/data2/69.55.232.46-col02147-DIR 69.55.232.46<br />
18 dili.thium.net /mnt/data2/69.55.232.80-col01901-DIR 69.55.232.80<br />
20 www.tekmarsis.com /mnt/data2/69.55.232.66-col02155-DIR 69.55.232.66<br />
21 vps.yoxel.net /mnt/data2/69.55.236.67-col01673-DIR 69.55.236.67<br />
22 smitty.twitalertz.com /mnt/data2/69.55.232.84-col02153-DIR 69.55.232.84<br />
23 deliver4.klatha.com /mnt/data2/69.55.232.67-col02160-DIR 69.55.232.67<br />
24 nideffer.com /mnt/data2/69.55.232.65-col00412-DIR 69.55.232.65<br />
25 usa.hanyuan.com /mnt/data2/69.55.232.57-col02163-DIR 69.55.232.57<br />
26 daifuku.ppbh.com /mnt/data2/69.55.236.91-col01720-DIR 69.55.236.91<br />
27 collins.greencape.net /mnt/data2/69.55.232.83-col01294-DIR 69.55.232.83<br />
28 ragebox.com /mnt/data2/69.55.230.104-col01278-DIR 69.55.230.104<br />
29 outside.mt.net /mnt/data2/69.55.232.72-col02166-DIR 69.55.232.72<br />
30 vps.payneful.ca /mnt/data2/69.55.234.98-col01999-DIR 69.55.234.98<br />
31 higgins /mnt/data2/69.55.232.87-col02165-DIR 69.55.232.87 ...<br />
32 ozymandius /mnt/data2/69.55.228.96-col01233-DIR 69.55.228.96<br />
33 trusted.realtors.org /mnt/data2/69.55.238.72-col02170-DIR 69.55.238.72<br />
34 jc1.flanderous.com /mnt/data2/69.55.239.22-col01504-DIR 69.55.239.22<br />
36 guppylog.com /mnt/data2/69.55.238.73-col00036-DIR 69.55.238.73<br />
40 haliohost.com /mnt/data2/69.55.234.41-col01916-DIR 69.55.234.41 ...<br />
41 satyr.jorge.cc /mnt/data1/69.55.232.70-col01963-DIR 69.55.232.70<br />
jail9 /# jailkill satyr.jorge.cc<br />
ERROR: jail_: jail "satyr,jorge,cc" not found<br />
</pre><br />
<br />
Note how it's saying <tt>satyr,jorge,cc</tt> is not found, and not <tt>satyr.jorge.cc</tt>. <br />
<br />
The jail subsystem tracks things using comma-delimited hostnames. That is created every few hours:<br />
<br />
jail9 /# crontab -l<br />
0 0,6,12,18 * * * /usr/local/jail/bin/sync_jail_names<br />
<br />
So if we run this manually:<br />
jail9 /# /usr/local/jail/bin/sync_jail_names<br />
<br />
Then kill the jail:<br />
jail9 /# jailkill satyr.jorge.cc<br />
successfully killed: satyr,jorge,cc</pre><br />
<br />
It worked.<br />
<br />
<br />
If you ever see this when trying to kill a jail:<br />
<br />
<pre># jailkill e-scribe.com<br />
killing JID: 6 hostname: e-scribe.com<br />
3 procs running<br />
3 procs running<br />
3 procs running<br />
3 procs running<br />
...</pre><br />
<br />
<tt>[[#jailkill|jailkill]]</tt> probably got lost trying to kill off the jail. Just ctrl-c the jailkill process, then run a jailps on the hostname, and <tt>kill -9</tt> any process which is still running. Keep running jailps and <tt>kill -9</tt> till all processes are gone.<br />
<br />
== jailpsall ==<br />
jailpsall<br />
will run a jailps on all jails configured in the quad files (this is different from<br />
jailps with no arguments as it won’t help you find a “hidden” system)<br />
<br />
== jailpsw ==<br />
jailpsw<br />
will run a jailps with an extra -w to provide wider output<br />
<br />
== jt (>=7.x) ==<br />
jt<br />
displays the top 20 processes on the server (the top 20 processes from top) and <br />
which jail owns them. This is very helpful for determining who is doing what when<br />
the server is very busy.<br />
<br />
== jtop (>=7.x) ==<br />
jtop<br />
a wrapper for top displaying processes on the server and which jail owns them. Constantly updates, like top. <br />
<br />
== jtop (<7.x) ==<br />
jtop<br />
displays the top 20 processes on the server (the top 20 processes from top) and <br />
which jail owns them. This is very helpful for determining who is doing what when<br />
the server is very busy.<br />
<br />
== stopjail ==<br />
stopjail <hostname> [1]<br />
this will jailkill, umount and vnconfig –u a jail. If passed an optional 2nd<br />
argument, it will not exit before umounting and un-vnconfig’ing in the event<br />
jailkill returns no processes killed. This is useful if you just want to umount<br />
and vnconfig –u a jail you’ve already killed. It is intelligent in that it won’t <br />
try to umount or vnconfig –u if it’s not necessary.<br />
<br />
== startjail ==<br />
startjail <hostname><br />
this will start vnconfig, mount (including linprocfs and null-mounts), and start a jail.<br />
Essentially, it reads the jail’s relevant block from the right quad file and executes it.<br />
It is intelligent in that it won’t try to mount or vnconfig if it’s not necessary.<br />
<br />
== jpid ==<br />
jpid <pid><br />
displays information about a process – including which jail owns it.<br />
It’s the equivalent of running cat /proc/<pid>/status<br />
<br />
== canceljail ==<br />
canceljail <hostname> [1]<br />
this will stop a jail (the equivalent of stopjail), check for backups (offer to remove them <br />
from the backup server and the backup.config), rename the vnfile, remove the dir, and <br />
edit quad/safe. If passed an optional 2nd argument, it will not exit upon failing to kill<br />
and processes owned by the jail. This is useful if you just want to cancel a jail which <br />
is already stopped.<br />
<br />
== jls ==<br />
jls [-v]<br />
Lists all jails running:<br />
<pre>JID #REF IP Address Hostname Path<br />
101 135 69.55.224.148 mail.pc9.org /mnt/data2/69.55.224.148-col01034-DIR</pre><br />
<br />
#REF is the number of references or procs(?) running<br />
<br />
Running with -v will give you all IPs assigned to each jail (7.2 up)<br />
<pre>JID #REF Hostname Path IP Address(es)<br />
101 139 mail.pc9.org /mnt/data2/69.55.224.148-col01034-DIR 69.55.224.14869.55.234.85</pre><br />
<br />
== startalljails ==<br />
startalljails<br />
7.2+ only. This will parse through quad1 and start all jails. It utilizes lockfiles so it won’t try to start a jail more than once- therefore multiple instances can be running in parallel without fear of starting a jail twice. If a jail startup gets stuck, you can ^C without fear of killing the script. IMPORTANT- before running startalljails you should make sure you ran preboot once as it will clear out all the lockfiles and enable startalljails to work properly.<br />
<br />
== aaccheck.sh ==<br />
aaccheck.sh<br />
displayes the output of container list and task list from aaccli<br />
<br />
== backup ==<br />
backup<br />
backup script called nightly to update jail scripts and do customer backups<br />
<br />
== buildsafe ==<br />
buildsafe<br />
creates safe files based on quads (automatically removing the fsck’s). This will destructively overwrite safe files<br />
<br />
== checkload.pl ==<br />
checkload.pl<br />
this was intended to be setup as a cronjob to watch processes on a jail when the load <br />
rises above a certain level. Not currently in use.<br />
<br />
== checkprio.pl ==<br />
checkprio.pl<br />
will look for any process (other than the current shell’s csh, sh, sshd procs) with a non-normal priority and normalize it<br />
<br />
== diskusagemon == <br />
diskusagemon <mount point> <1k blocks><br />
watches a mount point’s disk use, when it reaches the level specified in the 2nd argument,<br />
it exits. This is useful when doing a restore and you want to be paged as it’s nearing completion.<br />
Best used as: <tt>diskusagemon /asd/asd 1234; pagexxx</tt><br />
<br />
== dumprestore ==<br />
dumprestore <dumpfile><br />
this is a perl expect script which automatically enters ‘1’ and ‘y’. It seems to cause restore to fail<br />
to set owner permissions on large restores.<br />
<br />
== g ==<br />
g <search><br />
greps the quad/safe files for the given search parameter<br />
<br />
== gather.pl ==<br />
gather.pl<br />
gathers up data about jails configured and writes to a file. Used for audits against the db<br />
<br />
== gb ==<br />
gb <search><br />
greps backup.config for the given search parameter<br />
<br />
== gbg ==<br />
gbg <search><br />
greps backup.config for the given search parameter and presents just the directories (for clean pasting)<br />
<br />
== ipfwbackup ==<br />
ipfwbackup<br />
writes ipfw traffic count data to a logfile<br />
<br />
== ipfwreset ==<br />
ipfwreset<br />
writes ipfw traffic count data to a logfile and resets counters to 0<br />
<br />
== js ==<br />
js<br />
output varies by OS version, but generally provides information about the base jail:<br />
- which vn’s are in use<br />
- disk usage<br />
- info about the contents of quads<br />
- the # of inodes represented by the jails contained in the group (133.2 in the example below), and how many jails per data mount, as well as subtotals<br />
- ips bound to the base machine but not in use by a jail<br />
- free gvinum volumes, or unused vn’s or used md’s<br />
<br />
<pre>/usr/local/jail/rc.d/quad1:<br />
/mnt/data1 133.2 (1)<br />
/mnt/data2 1040.5 (7)<br />
total 1173.7 (8)<br />
/usr/local/jail/rc.d/quad2:<br />
/mnt/data1 983.4 (6)<br />
total 983.4 (6)<br />
/usr/local/jail/rc.d/quad3:<br />
/mnt/data1 693.4 (4)<br />
/mnt/data2 371.6 (3)<br />
total 1065 (7)<br />
/usr/local/jail/rc.d/quad4:<br />
/mnt/data1 466.6 (3)<br />
/mnt/data2 882.2 (5)<br />
total 1348.8 (8)<br />
/mnt/data1: 2276.6 (14)<br />
/mnt/data2: 2294.3 (15)<br />
<br />
Available IPs:<br />
69.55.230.11 69.55.230.13 69.55.228.200<br />
<br />
Available volumes:<br />
v78 /mnt/data2 2G<br />
v79 /mnt/data2 2G<br />
v80 /mnt/data2 2G</pre> <br />
<br />
== load.pl ==<br />
load.pl<br />
feeds info to load mrtg - executed by inetd.<br />
<br />
== makevirginjail ==<br />
makevirginjail<br />
Only on some systems, makes an empty jail (doesn't do restore step)<br />
<br />
== mb == <br />
mb <mount|umount><br />
(nfs) mounts and umounts dirs to backup2. Shortcuts are mbm and mbu to mount and unmount. <br />
<br />
== notify.sh ==<br />
notify.sh<br />
emails reboot@johncompanies.com – intended to be called at boot time to alert us to a machine which panics and reboots and isn’t caught by bb or castle.<br />
<br />
== orphanedbackupwatch ==<br />
orphanedbackupwatch<br />
looks for directories on backup2 which aren’t configured in backup.config and offers to delete them<br />
<br />
== postboot ==<br />
postboot<br />
to be run after a machine reboot and quad/safe’s are done executing. It will:<br />
* do chmod 666 on each jail’s /dev/null<br />
* add ipfw counts<br />
* run jailpsall (so you can see if a configured jail isn’t running)<br />
<br />
== preboot ==<br />
preboot<br />
to be run before running quad/safe – checks for misconfigurations: <br />
* a jail configured in a quad but not a safe<br />
* a jail is listed more than once in a quad<br />
* the ip assigned to a jail isn’t configured on the machine<br />
* alias numbering skips in the rc.conf (resulting in the above)<br />
* orphaned vnfile's that aren't mentioned in a quad/safe<br />
* ip mismatches between dir/vnfile name and the jail’s ip<br />
* dir/vnfiles's in quad/safe that don’t exist <br />
<br />
== quadanalyze.pl ==<br />
quadanalyze.pl<br />
called by js, produces the info (seen above with js explanation) about the contents of quad (inode count, # of jails, etc.)<br />
<br />
== rsync.backup ==<br />
rsync.backup<br />
does customer backups (relies on backup.config)<br />
<br />
== taskdone ==<br />
taskdone<br />
when called will email support@johncompanies.com with the hostname of the machine from which it was executed as the subject<br />
<br />
== topten ==<br />
topten<br />
summarizes the top 10 traffic users (called by ipfwreset)<br />
<br />
== trafficgather.pl ==<br />
trafficgather.pl [yy-mm]<br />
sends a traffic usage summary by jail to support@johncomapnies.com and payments@johncompanies.com. Optional arguments are year and month (must be in the past). If not passed, assumes last month. Relies on traffic logs created by ipfwreset and ipfwbackup<br />
<br />
== trafficwatch.pl ==<br />
trafficwatch.pl<br />
checks traffic usage and emails support@johncomapnies.com when a jail reaches the warning level (35G) and the limit (40G). We really aren’t using this anymore now that we have netflow.<br />
<br />
== trafstats ==<br />
trafstats<br />
writes ipfw traffic usage info by jail to a file called jc_traffic_dump in each jail’s / dir<br />
<br />
== truncate_jailmake ==<br />
truncate_jailmake<br />
a version of jailmake which creates truncated vnfiles.<br />
<br />
== vb ==<br />
vb<br />
the equivalent of: <tt>vi /usr/local/jail/bin/backup.config</tt><br />
<br />
== vs (freebsd) ==<br />
vs<n><br />
the equivalent of: <tt>vi /usr/local/jail/rc.d/safe<n></tt><br />
<br />
vq<n><br />
the equivalent of: vi /usr/local/jail/rc.d/quad<n><br />
<br />
== dumpremote ==<br />
dumpremote <user@machine> </remote/location/file-dump> <vnX><br />
ex: dumpremote user@10.1.4.117 /mnt/data3/remote.echoditto.com-dump 7<br />
this will dump a vn filesystem to a remote machine and location<br />
<br />
== oversellcheck ==<br />
oversellcheck<br />
displays how much a disk is oversold or undersold taking into account truncated vn files. Only for use on 4.x systems<br />
<br />
== mvbackups (freebsd) ==<br />
mvbackups <dir> (1.1.1.1-col00001-DIR) <target_machine> (jail1) <target_dir> (data1)<br />
moves backups from one location to another on the backup server, and provides you with option to remove entries from current backup.config, and simple paste command to add the config to backup.config on the target server<br />
<br />
== jailnice ==<br />
jailnice <hostname><br />
applies <tt>renice 19 [PID]</tt> and <tt>rtprio 31 –[PID]</tt> to each process in the given jail<br />
<br />
== dumpremoterestore ==<br />
dumpremoterestore <device> <ip of target machine> <dir on target machine><br />
ex: dumpremoterestore /dev/vn51 10.1.4.118 /mnt/data2/69.55.239.45-col00688-DIR<br />
dumps a device and restores it to a directory on a remote machine. Requires that you enable root ssh on the <br />
remote machine.<br />
<br />
== psj ==<br />
psj<br />
shows just the procs running on the base system – a ps auxw but without jail’d procs present<br />
<br />
== perc5iraidchk ==<br />
perc5iraidchk<br />
checks for degraded arrays on Dell 2950 systems with Perc5/6 controllers<br />
<br />
== perc4eraidchk ==<br />
perc4eraidchk<br />
checks for degraded arrays on Dell 2850 systems with Perc4e/Di controllers<br />
<br />
== Making new customer VE (vemakexxx) ==<br />
<br />
This applies to older virts with old templates. This should probably not be used at all anymore.<br />
<br />
<br />
1. look thru hist for ip<br />
<br />
2. confirm ip you want to use isn’t in use via Mgmt. -> IP Map in management screens<br />
<br />
3. put ve on whichever partition has more space<br />
vemakerh9 <veid> <ip> <hostname> <mount> <email> [gb disk]; <256|384|512> <veid><br />
vemakerh9 866 69.55.226.109 ngentu.com /vz1 ayo@ngantu.com,asd@asd.com 5; 256 866<br />
<br />
4. copy (veid), dir, and ip to pending customer screen (pass set to p455agfa)<br />
<br />
= Virtuozzo VPS =<br />
<br />
== Making new customer VE (vm) ==<br />
<br />
This applies only to new virts >= 4.x<br />
<br />
grab ip from ipmap (if opened from the pending cust screen it should take you to the right block). You can also run vzlist -a to see what block is in use, generally. Try to find an IP that's in the same block of class C IP's already on the box.<br />
<br />
1. confirm ip you want to use isn’t in use via Mgmt. -> IP Map in management screens<br />
<br />
2. put CT on whichever partition has more space<br />
<br />
3. vm CID IP hostname /vz[1|2] email[,email] template ( <LB|LBP|LS|LM|LMP|LP> [disk] | <gb_disk/mb_ram/gb_burst> ) <br />
vm col00009 69.55.230.238 centos.testdave.com /vz1 dsmith@n2.net centos-6-x86_64 LM<br />
<br />
4. copy veid, dir, ip and password to pending customer screen. activate customer<br />
<br />
<br />
Note: We use VEID (Virtual Environment ID) and CTID (Container ID) interchangably. Similarly, VE and CT. They mean the same thing.<br />
VZPP = VirtuoZzo Power Panel (the control panel for each CT)<br />
<br />
All linux systems exist in /vz, /vz1 or /vz2 - since each linux machine holds roughly 60-90 customers, there will be roughly 30-45 in each partition.<br />
<br />
The actual filesystem of the system in question is in:<br />
<br />
/vz(1-2)/private/(VEID)<br />
<br />
Where VEID is the identifier for that system - an all-numeric string larger than 100.<br />
<br />
The actual mounted and running systems are in the corresponding:<br />
<br />
/vz(1-2)/root/(VEID)<br />
<br />
But we rarely interact with any system from this mount point.<br />
<br />
You should never need to touch the root portion of their system – however you can traverse their filesystem by going to <tt>/vz(1-2)/private/(VEID)/root</tt> (<tt>/vz(1-2)/private/(VEID)/fs/root</tt> on 4.x systems) the root of their filesystem is in that directory, and their entire system is underneath that.<br />
<br />
Every VE has a startup script in <tt>/etc/sysconfig/vz-scripts</tt> (which is symlinked as <tt>/vzconf</tt> on all systems) - the VE startup script is simply named <tt>(VEID).conf</tt> - it contains all the system parameters for that VE:<br />
<br />
<pre># Configuration file generated by vzsplit for 60 VE<br />
# on HN with total amount of physical mem 2011 Mb<br />
<br />
VERSION="2"<br />
CLASSID="2"<br />
<br />
ONBOOT="yes"<br />
<br />
KMEMSIZE="8100000:8200000"<br />
LOCKEDPAGES="322:322"<br />
PRIVVMPAGES="610000:615000"<br />
SHMPAGES="33000:34500"<br />
NUMPROC="410:415"<br />
PHYSPAGES="0:2147483647"<br />
VMGUARPAGES="13019:2147483647"<br />
OOMGUARPAGES="13019:2147483647"<br />
NUMTCPSOCK="1210:1215"<br />
NUMFLOCK="107:117"<br />
NUMPTY="19:19"<br />
NUMSIGINFO="274:274"<br />
TCPSNDBUF="1800000:1900000"<br />
TCPRCVBUF="1800000:1900000"<br />
OTHERSOCKBUF="900000:950000"<br />
DGRAMRCVBUF="200000:200000"<br />
NUMOTHERSOCK="650:660"<br />
DCACHE="786432:818029"<br />
NUMFILE="7500:7600"<br />
AVNUMPROC="51:51"<br />
IPTENTRIES="155:155"<br />
DISKSPACE="4194304:4613734"<br />
DISKINODES="400000:420000"<br />
CPUUNITS="1412"<br />
QUOTAUGIDLIMIT="2000"<br />
VE_ROOT="/vz1/root/636"<br />
VE_PRIVATE="/vz1/private/636"<br />
NAMESERVER="69.55.225.225 69.55.230.3"<br />
OSTEMPLATE="vzredhat-7.3/20030305"<br />
VE_TYPE="regular"<br />
IP_ADDRESS="69.55.225.229"<br />
HOSTNAME="textengine.net"</pre><br />
<br />
<br />
As you can see, the hostname is set here, the disk space is set here, the number of inodes, the number of files that can be open, the number of tcp sockets, etc. - all are set here.<br />
<br />
In fact, everything that can be set on this customer system is set in this conf file.<br />
<br />
<br />
All interaction with the customer system is done with the VEID. You start the system by running:<br />
<br />
vzctl start 999<br />
<br />
You stop it by running:<br />
<br />
vzctl stop 999<br />
<br />
You execute commands in it by running:<br />
<br />
vzctl exec 999 df -k<br />
<br />
You enter into it, via a root-shell backdoor with:<br />
<br />
vzctl enter 999<br />
<br />
and you set parameters for the system, while it is still running, with:<br />
<br />
vzctl set 999 --diskspace 6100000:6200000 --save<br />
<br />
<tt>vzctl</tt> is the most commonly used command - we have aliased <tt>v</tt> to <tt>vzctl</tt> since we use it so often. We’ll continue to use <tt>vzctl</tt> in our examples, but feel free to use just <tt>v</tt>.<br />
<br />
Let's say the user wants more diskspace. You can cat their conf file and see:<br />
<br />
DISKSPACE="4194304:4613734"<br />
<br />
So right now they have 4gigs of space. You can then change it to 6 with:<br />
<br />
vzctl set 999 --diskspace 6100000:6200000 --save<br />
<br />
IMPORTANT: all issuances of the vzctl set command need to end with <tt>–save</tt> - if they don't, the setting will be set, but it will not be saved to the conf file, and they will not have those settings next time they boot.<br />
<br />
All of the tunables in the conf file can be set with the vzctl set command. Note that in the conf file, and on the vzctl set command line, we always issue two numbers seperated by a colon - that is because we are setting the hard and soft limits. Always set the hard limit slightly above the soft limit, as you see it is in the conf file for all those settings.<br />
<br />
There are also things you can set with `<tt>vzctl set</tt>` that are not in the conf file as settings, per se. For instance, you can add IPs:<br />
<br />
vzctl set 999 --ipadd 10.10.10.10 --save<br />
<br />
or multiple IPs:<br />
<br />
vzctl set 999 --ipadd 10.10.10.10 --ipadd 10.10.20.30 --save<br />
<br />
or change the hostname:<br />
<br />
vzctl set 999 --hostname www.example.com --save<br />
<br />
You can even set the nameservers:<br />
<br />
vzctl set 999 --nameserver 198.78.66.4 --nameserver 198.78.70.180 --save<br />
<br />
Although you probably will never do that.<br />
<br />
You can disable a VPS from being started (by VZPP or reboot) (4.x):<br />
<br />
vzctl set 999 --disabled yes --save <br />
<br />
You can disable a VPS from being started (by VZPP or reboot) (<=3.x):<br />
<br />
vzctl set 999 --onboot=no --save <br />
<br />
You can disable a VPS from using his control panel:<br />
<br />
vzctl set 999 --offline_management=no --save <br />
<br />
You can suspend a VPS, so it can be resumed in the same state it was in when it was stopped (4.x):<br />
<br />
vzctl suspend 999<br />
<br />
and to resume it:<br />
<br />
vzctl resume 999<br />
<br />
to see who owns process:<br />
vzpid <PID><br />
<br />
to mount up an unmounted ve:<br />
vzctl mount 827<br />
<br />
<br />
To see network stats for CT's:<br />
<pre># vznetstat<br />
VEID Net.Class Output(bytes) Input(bytes)<br />
-----------------------------------------------<br />
24218 1 484M 39M<br />
24245 1 463M 143M<br />
2451 1 2224M 265M<br />
2454 1 2616M 385M<br />
4149 1 125M 68M<br />
418 1 1560M 34M<br />
472 1 1219M 315M<br />
726 1 628M 317M<br />
763 1 223M 82M<br />
771 1 4234M 437M<br />
-----------------------------------------------<br />
Total: 13780M 2090M</pre><br />
<br />
<br />
One thing that sometimes comes up on older systems that we created with smaller defaults is that the system would run out of inodes. The user will email and say they cannot create any more files or grow any files larger, but they will also say that they are not out of diskspace ... they are running:<br />
<br />
df -k<br />
<br />
and seeing how much space is free - and they are not out of space. They are most likely out of inodes - which they would see by running:<br />
<br />
df -i<br />
<br />
So, the first thing you should do is enter their system with:<br />
<br />
vzctl enter 999<br />
<br />
and run: <tt>df -i</tt><br />
<br />
to confirm your theory. Then exit their system. Then simply cat their conf file and see what their inodes are set to (probably 200000:200000, since that was the old default on the older systems) and run:<br />
<br />
vzctl set 999 --diskinodes 400000:400000 --save<br />
<br />
If they are not out of inodes, then a good possibility is that they have maxed out their numfile configuration variable, which controls how many files they can have in their system. The current default is 7500 (which nobody has ever hit), but the old default was as low as 2000, so you would run something like:<br />
<br />
vzctl set 999 --numfile 7500:7500 --save<br />
<br />
----<br />
<br />
You cannot start or stop a VE if your pwd is its private (/vz/private/999) or root (/vz/root/999) directories, or anywhere below them.<br />
<br />
<br />
== Recovering from a crash (linux) ==<br />
<br />
=== Diagnose whether you have a crash ===<br />
The most important thing is to get the machine and all ve’s back up as soon as possible. Note the time, you’ll need to create a crash log entry (Mgmt. -> Reference -> CrashLog). The first thing to do is head over to the [[Screen#Screen_Organization|serial console screen]] and see if there’s any kernel error messages output. Try to copy any messages (or just a sample of repeating messages) you see into the notes section of the crash log – these will also likely need to be sent to virtuozzo for interpretation. If the messages are spewing too fast, hit ^O + H to start a screen log dump which you can ob1182.pts-38.bb serve after the machine is rebooted. Additionally, if the machine is responsive, you can get a trace to send to virtuozzo by hooking up a kvm and entering these 3 sequences:<br />
<pre>alt+print screen+m<br />
alt+print screen+p<br />
alt+print screen+t</pre><br />
<br />
If there are no messages, the machine may just be really busy- wait a bit (5-10min) to see if it comes back. If it's still pinging, odds are its very busy. If it doesn't come back, or the messages indicate a fatal error, you will need to proceed with a power cycle (ctrl+alt+del will not work).<br />
<br />
=== Power cycle the server ===<br />
If this machine is not a Dell 2950 with a [[DRAC/RMM#DRAC|DRAC card]] (i.e. if you can’t ssh into the DRAC card and issue racadm serveraction hardreset, then you will need someone at the data center to power the macine off, wait 30 sec, then turn it back on. Make sure to re-attach via console (<tt>tip virtxx</tt>) immediately after power down. <br />
<br />
=== (Re)attach to the console ===<br />
Stay on the console the entire time during boot. As the BIOS posts- look out for the RAID card output- does everything look healthy? The output may be scrambled, look for "DEGRADED" or "FAILED". Once the OS starts booting you will be disconnected (dropped back to the shell on the console server) a couple times during the boot up. The reason you want to quickly re-attach is two-fold: 1. If you don’t reattach quickly then you won’t get any console output, 2. you want to be attached before the server ''potentially'' starts (an extensive) fsck. If you attach after the fsck begins, you’ll have seen no indication it started an fsck and the server will appear frozen during startup- no output, no response. <br />
<br />
=== Start containers/VE's/VPSs ===<br />
When the machine begins to start VE’s, it’s safe to leave the console and login via ssh. All virts should be set to auto start all the VEs after a crash. Further, most (newer) virts are set to “fastboot” it’s VE’s (to find out, do:<br />
grep -i fast /etc/sysconfig/vz <br />
and look for <tt>VZFASTBOOT=yes</tt>). If this was set prior to the machine’s crash (setting it after the machine boots will not have any effect until the vz service is restarted) it will start each ve as fast as possible, in serial, then go thru each VE (serially), shutting it down running a vzquota (disk usage) check, then bringing it back up. The benefit is that all VE’s are brought up quickly (within 15min or so depending on the #), the downside is a customer watching closely will notice 2 outages – 1st the machine crash, 2nd their quota check (which will be a much shorter downtime- on the order of a few minutes). <br />
<br />
Where “fastboot” is not set to yes (i.e on quar1), vz will start them consecutively, checking the quotas one at a time, and the 60th VE may not start until an hour or two later - this is not acceptable.<br />
<br />
The good news is, if you run vzctl start for a VE that is already started, you will simply get an error: <tt>VE is already started</tt>. Further, if you attempt to vzctl start a VE that is in the process of being started, you will simply get an error: unable to lock VE. So, there is no danger in simply running scripts to start smaller sets of VEs. If the system is not autostarting, then there is no issue, and even if it does, when it conflicts, one process (yours or the autostart) will lose, and just move on to the next one.<br />
<br />
A script has been written to assist with ve starts: [[#startvirt.pl|startvirt.pl]] which will start 6 ve’s at once until there are no more left. If startvirt.pl is used on a system where “fastboot” was on, it will circumvent the fastboot for ve’s started by startvirt.pl – they will go through the complete quota check before starting- therefore this is not advisable when a system has crashed. When a system is booted cleanly, and there's no need for vzquota checks, then startvirt.pl is safe and advisable to run.<br />
<br />
=== Make sure all containers are running ===<br />
You can quickly get a feel for how many ve’s are started by running:<br />
<br />
<pre>[root@virt4 log]# vs<br />
VEID 16066 exist mounted running<br />
VEID 16067 exist mounted running<br />
VEID 4102 exist mounted running<br />
VEID 4112 exist mounted running<br />
VEID 4116 exist mounted running<br />
VEID 4122 exist mounted running<br />
VEID 4123 exist mounted running<br />
VEID 4124 exist mounted running<br />
VEID 4132 exist mounted running<br />
VEID 4148 exist mounted running<br />
VEID 4151 exist mounted running<br />
VEID 4155 exist mounted running<br />
VEID 42 exist mounted running<br />
VEID 432 exist mounted running<br />
VEID 434 exist mounted running<br />
VEID 442 exist mounted running<br />
VEID 450 exist mounted running<br />
VEID 452 exist mounted running<br />
VEID 453 exist mounted running<br />
VEID 454 exist mounted running<br />
VEID 462 exist mounted running<br />
VEID 463 exist mounted running<br />
VEID 464 exist mounted running<br />
VEID 465 exist mounted running<br />
VEID 477 exist mounted running<br />
VEID 484 exist mounted running<br />
VEID 486 exist mounted running<br />
VEID 490 exist mounted running</pre><br />
<br />
So to see how many ve’s have started:<br />
<pre>[root@virt11 root]# vs | grep running | wc -l<br />
39</pre><br />
<br />
And to see how many haven’t:<br />
<pre>[root@virt11 root]# vs | grep down | wc -l<br />
0</pre><br />
<br />
And how many we should have running:<br />
<pre>[root@virt11 root]# vs | wc -l<br />
39</pre><br />
<br />
Another tool you can use to see which ve’s have started, among other things is [[#vzstat|vzstat]]. It will give you CPU, memory, and other stats on each ve and the overall system. It’s a good thing to watch as ve’s are starting (note the VENum parameter, it will tell you how many have started):<br />
<br />
<pre>4:37pm, up 3 days, 5:31, 1 user, load average: 1.57, 1.68, 1.79<br />
VENum 40, procs 1705: running 2, sleeping 1694, unint 0, zombie 9, stopped 0<br />
CPU [ OK ]: VEs 57%, VE0 0%, user 8%, sys 7%, idle 85%, lat(ms) 412/2<br />
Mem [ OK ]: total 6057MB, free 9MB/54MB (low/high), lat(ms) 0/0<br />
Swap [ OK ]: tot 6142MB, free 4953MB, in 0.000MB/s, out 0.000MB/s<br />
Net [ OK ]: tot: in 0.043MB/s 402pkt/s, out 0.382MB/s 4116pkt/s<br />
Disks [ OK ]: in 0.002MB/s, out 0.000MB/s<br />
<br />
VEID ST %VM %KM PROC CPU SOCK FCNT MLAT IP<br />
1 OK 1.0/17 0.0/0.4 0/32/256 0.0/0.5 39/1256 0 9 69.55.227.152<br />
21 OK 1.3/39 0.1/0.2 0/46/410 0.2/2.8 23/1860 0 6 69.55.239.60<br />
133 OK 3.1/39 0.1/0.3 1/34/410 6.3/2.8 98/1860 0 0 69.55.227.147<br />
263 OK 2.3/39 0.1/0.2 0/56/410 0.3/2.8 34/1860 0 1 69.55.237.74<br />
456 OK 17/39 0.1/0.2 0/100/410 0.1/2.8 48/1860 0 11 69.55.236.65<br />
476 OK 0.6/39 0.0/0.2 0/33/410 0.1/2.8 96/1860 0 10 69.55.227.151<br />
524 OK 1.8/39 0.1/0.2 0/33/410 0.0/2.8 28/1860 0 0 69.55.227.153<br />
594 OK 3.1/39 0.1/0.2 0/45/410 0.0/2.8 87/1860 0 1 69.55.239.40<br />
670 OK 7.7/39 0.2/0.3 0/98/410 0.0/2.8 64/1860 0 216 69.55.225.136<br />
691 OK 2.0/39 0.1/0.2 0/31/410 0.0/0.7 25/1860 0 1 69.55.234.96<br />
744 OK 0.1/17 0.0/0.5 0/10/410 0.0/0.7 7/1860 0 6 69.55.224.253<br />
755 OK 1.1/39 0.0/0.2 0/27/410 0.0/2.8 33/1860 0 0 192.168.1.4<br />
835 OK 1.1/39 0.0/0.2 0/19/410 0.0/2.8 5/1860 0 0 69.55.227.134<br />
856 OK 0.3/39 0.0/0.2 0/13/410 0.0/2.8 16/1860 0 0 69.55.227.137<br />
936 OK 3.2/52 0.2/0.4 0/75/410 0.2/0.7 69/1910 0 8 69.55.224.181<br />
1020 OK 3.9/39 0.1/0.2 0/60/410 0.1/0.7 55/1860 0 8 69.55.227.52<br />
1027 OK 0.3/39 0.0/0.2 0/14/410 0.0/2.8 17/1860 0 0 69.55.227.83<br />
1029 OK 1.9/39 0.1/0.2 0/48/410 0.2/2.8 25/1860 0 5 69.55.227.85<br />
1032 OK 12/39 0.1/0.4 0/80/410 0.0/2.8 41/1860 0 8 69.55.227.90</pre><br />
<br />
When you are all done, you will want to make sure that all the VEs really did get started, run vs one more time.<br />
<br />
Note the time all ve’s are back up and enter that into and save the crash log entry.<br />
<br />
Occasionally, a ve will not start automatically. The most common reason for a ve not to come up normally is the ve was at it’s disk limit before the crash, and will not start since they’re over the limit. To overcome this, set the disk space to current usage level (the system will give this to you when it fails to start), start the ve, then re-set the disk space back to the prior level. Lastly, contact the customer to let them know they’re out of disk (or allocate more disk if they're entitled to more).<br />
<br />
== Hitting performance barriers and fixing them ==<br />
<br />
There are multiple modes virtuozzo offers to allocate resources to a ve. We utilize 2: SLM and UBC parameters<br />
On our 4.x systems, we use all SLM – it’s simpler to manage and understand. There are a few systems on virt19/18 that may also use SLM. Everything else uses UBC. <br />
You can tell a SLM ve by:<br />
<br />
SLMMODE="all"<br />
<br />
in their conf file. <br />
<br />
TODO: detail SLM modes and parameters.<br />
<br />
If someone is in SLM mode and they hit memory resource limits, they simply need to upgrade to more memory.<br />
<br />
The following applies to everyone else (UBC).<br />
<br />
Customers will often email and say that they are getting out of memory errors - a common one is "cannot fork" ... basically, anytime you see something odd like this, it means they are hitting one of their limits that is in place in their conf file.<br />
<br />
The conf file, however, simply shows their limits - how do we know what they are currently at ?<br />
<br />
The answer is a file called v - this file contains the current status (and peaks) of their performance settings, and also counts how many times they have hit the barrier. The output of the file looks like this:<br />
<br />
<pre>764: kmemsize 384113 898185 8100000 8200000 0<br />
lockedpages 0 0 322 322 0<br />
privvmpages 1292 7108 610000 615000 0<br />
shmpages 270 528 33000 34500 0<br />
dummy 0 0 0 0 0<br />
numproc 8 23 410 415 0<br />
physpages 48 5624 0 2147483647 0<br />
vmguarpages 0 0 13019 2147483647 0<br />
oomguarpages 641 6389 13019 2147483647 0<br />
numtcpsock 3 21 1210 1215 0<br />
numflock 1 3 107 117 0<br />
numpty 0 2 19 19 0<br />
numsiginfo 0 4 274 274 0<br />
tcpsndbuf 0 80928 1800000 1900000 0 <br />
tcprcvbuf 0 108976 1800000 1900000 0<br />
othersockbuf 2224 37568 900000 950000 0<br />
dgramrcvbuf 0 4272 200000 200000 0<br />
numothersock 3 9 650 660 0<br />
dcachesize 53922 100320 786432 818029 0<br />
numfile 161 382 7500 7600 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
numiptent 4 4 155 155 0</pre><br />
<br />
The first column is the name of the counter in question - the same names we saw in the systems conf file. The second column is the _current_ value of that counter, the third column is the max that that counter has ever risen to, the fourth column is the soft limit, and the fifth column is the hard limit (which is the same as the numbers in that systems conf file).<br />
<br />
The sixth number is the failcount - how many times the current usage has risen to hit the barrier. It will increase as soon as the current usage hits the soft limit.<br />
<br />
The problem with /proc/user_beancounters is that it actually contains that set of data for every running VE - so you can't just cat /proc/user_beancounters - it is too long and you get info for every other running system.<br />
<br />
You can vzctl enter the system and run:<br />
<br />
vzctl enter 9999<br />
cat /proc/user_beancounters<br />
<br />
inside their system, and you will just see the stats for their particular system, but entering their system every time you want to see it is combersome.<br />
<br />
So, I wrote a simple script called "vzs" which simply greps for the VEID, and spits out the next 20 or so lines (however many lines there are in the output, I forget) after it. For instance:<br />
<br />
<pre># vzs 765:<br />
765: kmemsize 2007936 2562780 8100000 8200000 0<br />
lockedpages 0 8 322 322 0<br />
privvmpages 26925 71126 610000 615000 0<br />
shmpages 16654 16750 33000 34500 0<br />
dummy 0 0 0 0 0<br />
numproc 41 57 410 415 0<br />
physpages 1794 49160 0 2147483647 0<br />
vmguarpages 0 0 13019 2147483647 0<br />
oomguarpages 4780 51270 13019 2147483647 0<br />
numtcpsock 23 37 1210 1215 0<br />
numflock 17 39 107 117 0<br />
numpty 1 3 19 19 0<br />
numsiginfo 0 6 274 274 0<br />
tcpsndbuf 22240 333600 1800000 1900000 0<br />
tcprcvbuf 0 222656 1800000 1900000 0<br />
othersockbuf 104528 414944 900000 950000 0<br />
dgramrcvbuf 0 4448 200000 200000 0<br />
numothersock 73 105 650 660 0<br />
dcachesize 247038 309111 786432 818029 0<br />
numfile 904 1231 7500 7600 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
numiptent 4 4 155 155 0</pre><br />
<br />
That showed us just the portion of /proc/user_beancounters for system 765.<br />
<br />
When you run the vzs command, always add a : after the VEID.<br />
<br />
So, if a customer complains about some out of memory errors, or no more files, or no more ptys, or just has an unspecific complain about processes dying, etc., the very first thing you need to do is check their beancounters with vzs. Usually you will spot an item that has a high failcount and needs to be upped.<br />
<br />
At that point you could simply up the counter with `vzctl set`. Generally pick a number 10-20% higher than the old one, and make the hard limit slightly larger than the the soft limit. However our systems now come in several levels and those levels have more/different memory allocations. If someone is complaining about something other than a memory limit (pty, numiptent, numflock), it’s generally safe to increase it, at least to the same level as what’s in the /vzconf/4unlimited file on the newest virt. If someone is hitting a memory limit, first make sure they are given what they deserve:<br />
<br />
(refer to mgmt -> payments -> packages)<br />
<br />
To set those levels, you use the [[#setmem|setmem]] command. <br />
<br />
The alternate (DEPRECATED) method would be to use one of 3 commands:<br />
256 <veid><br />
300 <veid><br />
384 <veid><br />
512 <veid><br />
<br />
If the levels were not right (you’d run vzs <veid> before and after to see the effect) tell the customer they’ve been adjusted and be done with it. If the levels were right, tell the customer they must upgrade to a higher package, tell them how to see level (control panel) and that they can reboot their system to escape this lockup contidion.<br />
<br />
Customers can also complain that their site is totally unreachable, or complain that it is down ... if the underlying machine is up, and all seems well, you may notice in the beancounters that network-specific counters are failing - such as numtcpsock, tcpsndbuf or tcprcvbuf. This will keep them from talking on the network and make it seem like their system is down. Again, just up the limits and things should be fine.<br />
<br />
On virts 1-4, you should first look at the default settings for that item on a later virt, such as virt 8 - we have increased the defaults a lot since the early machines. So, if you are going to up a counter on virt2, instead of upping it by 10-20%, instead up it to the new default that you see on virt8.<br />
<br />
== Moving a VE to another virt (migrate/migrateonline) ==<br />
<br />
This will take a while to complete - and it is best to do this at night when the load is light on both machines.<br />
<br />
There are different methods for this, depending on which version of virtuozzo is installed on the src. and dst. virt. <br />
To check which version is running: <br />
[root@virt12 private]# cat /etc/virtuozzo-release<br />
Virtuozzo release 2.6.0<br />
<br />
Ok, let's say that the VE is 1212, and vital stats are:<br />
<pre>[root@virt12 sbin]# vc 1212<br />
VE_ROOT="/vz1/root/1212"<br />
VE_PRIVATE="/vz1/private/1212"<br />
OSTEMPLATE="fedora-core-2/20040903"<br />
IP_ADDRESS="69.55.229.84"<br />
TEMPLATES="devel-fc2/20040903 php-fc2/20040813 mysql-fc2/20040812 postgresql-fc2/20040813 mod_perl-fc2/20040812 mod_ssl-fc2/20040811 jre-fc2/20040823 jdk-fc2/20040823 mailman-fc2/20040823 analog-fc2/20040824 proftpd-fc2/20040818 tomcat-fc2/20040823 usermin-fc2/20040909 webmin-fc2/20040909 uw-imap-fc2/20040830 phpBB-fc2/20040831 spamassassin-fc2/20040910 PostNuke-fc2/20040824 sl-webalizer-fc2/20040<br />
818"<br />
<br />
[root@virt12 sbin]# vzctl exec 1212 df -h<br />
Filesystem Size Used Avail Use% Mounted on<br />
/dev/vzfs 4.0G 405M 3.7G 10% /</pre><br />
<br />
From this you can see that he’s using (and will minimally need free on the dst server) ~400MB, and he’s running on a Fedora 2 template, version 20040903. He’s also got a bunch of other templates installed. It’s is '''vital''' that '''all''' these templates exist on the dst system. To confirm that, on the dst system run:<br />
<br />
For < 3.0:<br />
<br />
<pre>[root@virt14 private]# vzpkgls | grep fc2<br />
devel-fc2 20040903<br />
PostNuke-fc2 20040824<br />
analog-fc2 20040824<br />
awstats-fc2 20040824<br />
bbClone-fc2 20040824<br />
jdk-fc2 20040823<br />
jre-fc2 20040823<br />
mailman-fc2 20040823<br />
mod_frontpage-fc2 20040816<br />
mod_perl-fc2 20040812<br />
mod_ssl-fc2 20040811<br />
mysql-fc2 20040812<br />
openwebmail-fc2 20040817<br />
php-fc2 20040813<br />
phpBB-fc2 20040831<br />
postgresql-fc2 20040813<br />
proftpd-fc2 20040818<br />
sl-webalizer-fc2 20040818<br />
spamassassin-fc2 20040910<br />
tomcat-fc2 20040823<br />
usermin-fc2 20040909<br />
uw-imap-fc2 20040830<br />
webmin-fc2 20040909<br />
[root@virt14 private]# vzpkgls | grep fedora<br />
fedora-core-1 20040121 20040818<br />
fedora-core-devel-1 20040121 20040818<br />
fedora-core-2 20040903<br />
[root@virt14 private]#</pre><br />
<br />
For these older systems, you can simply match up the date on the template. <br />
<br />
For >= 3.0:<br />
<br />
<pre>[root@virt19 /vz2/private]# vzpkg list<br />
centos-5-x86 2008-01-07 22:05:57<br />
centos-5-x86 devel<br />
centos-5-x86 jre<br />
centos-5-x86 jsdk<br />
centos-5-x86 mod_perl<br />
centos-5-x86 mod_ssl<br />
centos-5-x86 mysql<br />
centos-5-x86 php<br />
centos-5-x86 plesk9<br />
centos-5-x86 plesk9-antivirus<br />
centos-5-x86 plesk9-api<br />
centos-5-x86 plesk9-atmail<br />
centos-5-x86 plesk9-backup<br />
centos-5-x86 plesk9-horde<br />
centos-5-x86 plesk9-mailman<br />
centos-5-x86 plesk9-mod-bw<br />
centos-5-x86 plesk9-postfix<br />
centos-5-x86 plesk9-ppwse<br />
centos-5-x86 plesk9-psa-firewall<br />
centos-5-x86 plesk9-psa-vpn<br />
centos-5-x86 plesk9-psa-fileserver<br />
centos-5-x86 plesk9-qmail<br />
centos-5-x86 plesk9-sb-publish<br />
centos-5-x86 plesk9-vault<br />
centos-5-x86 plesk9-vault-most-popular<br />
centos-5-x86 plesk9-watchdog</pre><br />
<br />
On these newer systems, it's difficult to tell whether the template on the dst matches exactly the src. Just cause a centos-5-x86 is listed on both servers doesn't mean all the same packages are there on the dst. To truly know, you must perform a sample rsync:<br />
<br />
rsync -avn /vz/template/centos/5/x86/ root@10.1.4.61:/vz/template/centos/5/x86/<br />
<br />
if you see a ton of output from the dry run command, then clearly there are some differences. You may opt to let the rsync complete (without running in dry run mode) the only downside is you've now used up more space on the dst and also the centos template will be a mess with old and new data- it will be difficult if not impossible to undo (if someday we wanted to reclaim the space).<br />
<br />
If you choose to merge templates, you should closely inspect the dry run output. You should also take care to exclude anything in the /config directory. For example:<br />
<br />
rsync -av -e ssh --stats --exclude=x86/config /vz/template/ubuntu/10.04/ root@10.1.4.62:/vz/template/ubuntu/10.04/<br />
<br />
Which will avoid this directory and contents:<br />
<pre>[root@virt11 /vz2/private]# ls /vz/template/ubuntu/10.04/x86/config*<br />
app os</pre><br />
<br />
This is important to avoid since the config may differ on the destination and we are really only interested in making sure the pacakges are there, not overwriting a newer config with an older one.<br />
<br />
If the dst system was missing a template, you have 2 choices: <br />
# put the missing template on the dst system. 2 choices here: <br />
## Install the template from rpm (found under backup2: /mnt/data4/vzrpms/distro/) or <br />
## rsync over the template (found under /vz/template) - see above<br />
# put the ve on a system which has all the proper templates<br />
<br />
=== pre-seeding a migration ===<br />
<br />
When migrating a customer (or when doing many) depending on how much data you have to transfer, it can take some time. Further, it can be difficult to gauge when a migration will complete or how long it will take. To help speed up the process and get a better idea about how long it will take you can pre-transfer a customer's data to the destination server. If done correctly, vzmigrate will see the pre-transferred data and pick up where you left off, having much less to transfer (just changed/new files). <br />
<br />
We believe vzmigrate uses rsync to do it's transfer. Therefore not only can you use rsync to do a pre-seed, you can also run rsync to see what is causing a repeatedly-failing vzmigrate to fail. <br />
<br />
There's no magic to a pre-seed, you just need to make sure it's named correctly.<br />
<br />
Given:<br />
<br />
source: /vz1/private/1234<br />
<br />
and you want to migrate to /vz2 on the target system, your rsync would look like:<br />
<br />
rsync -av /vz1/private/1234/ root@x.x.x.x:/vz2/private/1234.migrated/<br />
<br />
After running that successful rsync, the ensuing migrateonline (or migrate) will take much less time to complete- depending on the # of files to be analyzed and the # of changed files. In any case, it'll be much much faster than had you just started the migration from scratch.<br />
<br />
Further, as we discuss elsewhere in this topic, a failed migration can be moved from <tt>/vz/private/1234</tt> to <tt>/vz/private/1234.migrated</tt> on the destination if you want to restart a failed migration. This should '''only''' be done if the migration failed and the CT is not running on the destination HN.<br />
<br />
=== migrateonline intructions: src >=3.x -> dst>=3.x ===<br />
<br />
A script called [[#migrateonline|migrateonline]] was written to handle this kind of move. It is basically a wrapper for <tt>vzmigrate</tt> – vzmigrate is a util to seamlessly- as no no reboot of the ve necessary- move a ve from one host to another. This wrapper was initially written cause vz virtuozzo version 2.6.0 has a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables, causing problems when the ve was started up on the dst system. [[#migrate|migrate]] mitigates that. Since it makes multiple ssh connections to the dst virt, it’s a good idea to put the pub key for the src virt in the authorized_keys file on the dst virt. In addition, migrateonline emails ve owners when their migration starts and stops. For this to happen they need to put email addresses (on a single line, space delimited) in a file on their system: /migrate_notify. If the optional target dir is not specified, the ve will be moved to the same private/root location as it was on the src virt. Note: <tt>migrate</tt> is equivalent to <tt>migrateonline</tt>, but will <tt>migrate</tt> a ve AND restart it in the process.<br />
<br />
<pre>[root@virt12 sbin]# migrateonline<br />
usage: /usr/local/sbin/migrateonline <ip of node migrating to> <veid> [target dir: vz | vz1 | vz2]<br />
[root@virt12 sbin]# migrateonline 10.1.4.64 1212 vz<br />
starting to migrate 1212 at Sat Mar 26 22:40:38 PST 2005<br />
Turning off offline_management<br />
Saved parameters for VE 1212<br />
migrating with no start on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
clearing the arp cache<br />
now going to 10.1.4.64 and clear cache and starting<br />
starting it<br />
Delete port redirection<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
finished migrating 1212 at Sat Mar 26 22 22:52:01 PST 2005<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine.<br />
<br />
If they had backups, use the mvbackups command to move their backups to the new server:<br />
<br />
mvbackups 1212 virt14 vz<br />
<br />
Rename the ve<br />
<br />
[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/migrated-1212<br />
<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/old-1212-migrated-20120404-noarchive<br />
<br />
Update the customer’s systems in mgmt to reflect the new path and server.<br />
<br />
<br />
IF migrateonline does not work, you can try again using simply migrate- this will result in a brief reboot for the ve.<br />
Before you try again, make sure of a few things:<br />
<br />
Depending on where in the migration died, there may be partial data on the dst system in 1 of 2 places:<br />
(given the example above)<br />
<br />
/vz/private/1212<br />
<br />
or <br />
<br />
/vz/private/1212.migrated<br />
<br />
before you run migrate again, you'll want to rename so that all data is in <br />
1212.migrated:<br />
<br />
mv /vz/private/1212 /vz/private/1212.migrated<br />
<br />
this way, it will pick up where it left off and transfer only new files.<br />
<br />
Likewise, if you want to speed up a migration, you can pre-seed the dst as follows:<br />
<br />
[root@virt12 sbin]# rsync -avSH /vz/private/1212/ root@10.1.4.64:/vz/private/1212.migrated/<br />
<br />
then when you run migrate or migrateonline, it will only need to move the changed files- the migration will complete quickly<br />
<br />
=== migrateonline/migrate failures (migrate manually) ===<br />
<br />
Lets say for whatever reason the migration fails. If it fails with [[#migrateonline|migrateonline]], you should try [[#migrate|migrate]] (which will reboot the customer, so notify them ahead of time).<br />
<br />
You may want to run a [[#pre-seeding_a_migration|pre-seed]] rsync to see if you can find the problem. On older virts, we've seen this problem due to a large logfile (which you can find and encourage the customer to remove/compress):<br />
for f in `find / -size +1048576k`; do ls -lh $f; done<br />
<br />
You may also see migration failing due to quota issues.<br />
<br />
You can try to resolve by copying any quota file into the file you need:<br />
<br />
cp /var/vzquota/quota.1 /var/vzquota/quota.xxx<br />
<br />
If it complains about quota running you should then be able to stop it<br />
<br />
vzquota off xxxx<br />
<br />
If all else fails, migrate to a new VEID<br />
i.e. 1234 becomes 12341<br />
<br />
<br />
If the rsync or [[#migrate|migrate]] fails, you can always move someone manually:<br />
<br />
1. stop ve: <br><br />
v stop 1234<br />
<br />
2. copy over data<br><br />
rsync -avSH /vz/private/1234/ root@1.1.1.1:/vzX/private/1234/<br />
<br />
NOTE: if you've previously seeded the data (run rsync while the VE was up/running), and this is a subsequent rsync, make sure the last rsync you do (while the VE is not running, has the --delete option in the rsync)<br />
<br />
3. copy over conf<br><br />
scp /vzconf/1234.conf root@1.1.1.1:/vzconf<br />
<br />
4. on dst, edit the conf to reflect the right vzX dir<br><br />
vi /vzconf/1234.conf<br />
<br />
5. on src remove the IPs<br><br />
ipdel 1234 2.2.2.2 3.3.3.3<br />
<br />
6. on dst add IPs <br><br />
ipadd 1234 2.2.2.2 3.3.3.3<br />
<br />
7. on dst, start ve: <br><br />
v start 1324<br />
<br />
8. cancel, then archive ve on src per above instrs.<br />
<br />
=== migrate src=2.6.0 -> dst>=2.6.0, or mass-migration with customer notify ===<br />
<br />
A script called <tt>migrate</tt> was written to handle this kind of move. It is basically a wrapper for vzmigrate – vzmigrate is a util to seamlessly move a ve from one host to another. This wrapper was initially written cause vz virtuozzo version 2.6.0 has a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables, causing problems when the ve was started up on the dst system. migrate mitigates that. Since it makes multiple ssh connections to the dst virt, it’s a good idea to put the pub key for the src virt in the authorized_keys file on the dst virt. In addition, migrate emails ve owners when their migration starts and stops. For this to happen they need to put email addresses (on a single line, space delimited) in a file on their system: /migrate_notify. If the optional target dir is not specified, the ve will be moved to the same private/root location as it was on the src virt. Note: migrateonline is equivalent to migrate, but will migrate a ve from one 2.6 '''kernel''' machine to another 2.6 kernel machine without restarting the ve.<br />
<br />
<pre>[root@virt12 sbin]# migrate<br />
usage: /usr/local/sbin/migrate <ip of node migrating to> <veid> [target dir: vz | vz1 | vz2]<br />
[root@virt12 sbin]# migrate 10.1.4.64 1212 vz<br />
starting to migrate 1212 at Sat Mar 26 22:40:38 PST 2005<br />
Turning off offline_management<br />
Saved parameters for VE 1212<br />
migrating with no start on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
clearing the arp cache<br />
now going to 10.1.4.64 and clear cache and starting<br />
starting it<br />
Delete port redirection<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
finished migrating 1212 at Sat Mar 26 22 22:52:01 PST 2005<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine (backup2 or mail).<br />
Cancel the ve (first we have to rename things which migrate changed so cancelve will find them):<br />
<br />
[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/1212.conf<br />
<br />
On 2.6.1 you’ll also have to move the private area:<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/1212<br />
<br />
<pre>[root@virt12 sbin]# cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, [[#cancelve|cancelve]] would offer to remove them. You want to say '''no''' to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
To move everyone off a system, you’d do:<br />
for f in `vl`; do migrate <ip> $f; done<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
=== vzmigrate: src=2.6.1 -> dst>=2.6.0 ===<br />
<br />
This version of vzmigrate works properly with regard to handling ips. It will not notify ve owners of moves as in the above example. Other than that it’s essentially the same.<br />
<br />
<pre>[root@virt12 sbin]# vzmigrate 10.1.4.64 -r no 1212:1212:/vz/private/1212:/vz/root/1212<br />
migrating on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine (backup2 or mail).<br />
Cancel the ve (first we have to rename things which vzmigrate changed so cancelve will find them):<br />
<br />
<pre>[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/1212.conf<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/1212<br />
<br />
[root@virt12 sbin]# cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, <tt>cancelve</tt> would offer to remove them. You want to say no to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
=== src=2.5.x ===<br />
<br />
First, go to the private dir:<br />
<br />
cd /vz1/private/<br />
<br />
Stop the VE - make sure it stops totally cleanly.<br />
<br />
vzctl stop 1212<br />
<br />
Then you’d use vemove - a script written to copy over the config, create tarballs of the ve’s data on the destination virt, and cancel the ve on the source system (in this example we’re going to put a ve that was in /vz1/private on the src virt, in /vz/private on the dst virt):<br />
<br />
<pre>[root@virt12 sbin]# vemove<br />
ERROR: Usage: vemove veid target_ip target_path_dir<br />
[root@virt12 sbin]# vemove 1212 10.1.4.64 /vz/private/1212<br />
tar cfpP - 1212 --ignore-failed-read | (ssh -2 -c arcfour 10.1.4.64 "split - -b 1024m /vz/private/1212.tar" )<br />
scp /vzconf/1212.conf 10.1.4.64:/vzconf<br />
cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, cancelve would offer to remove them. You want to say no to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
When you are done, go to /vz/private on the dst virt you will have files like this:<br />
<br />
<pre>1212.taraa<br />
1212.tarab<br />
1212.tarac</pre><br />
<br />
Each one 1024m (or less, for the last one) in size.<br />
<br />
on the dst server and run:<br />
<br />
cat 1212.tar?? | tar xpPBf -<br />
<br />
and after 20 mins or so it will be totally untarred. Now since the conf<br />
file is already there, you can go ahead and start the system.<br />
<br />
vzctl start 1212<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
NOTE: you MUST tar the system up using the virtuozzo version of tar that<br />
is on all the virt systems, and further you MUST untar the tarball with<br />
the virtuozzo tar, using these options: `<tt>tar xpPBf -</tt>`<br />
<br />
If you tar up an entire VE and move it to a non-virtuozzo machine, that is<br />
ok, and you can untar it there with normal tar commands, but do not untar<br />
it and then repack it with a normal tar and expect it to work - you need<br />
to use virtuozzo tar commands on virtuozzo tarballs to make it work.<br />
<br />
The backups are sort of an exception, since we are just (usually)<br />
restoring user data that was created after we gave them the system, and<br />
therefore has nothing to do with magic symlinks or vz-rpms, etc.<br />
<br />
== Moving a VE on the same virt ==<br />
<br />
Easy way:<br><br />
Scenario 1: ve 123 is to be renamed 1231 and moved from vz1 to vz<br />
<br />
vzmlocal 123:1231:/vz/private/1231:/vz/root/1231<br />
<br />
Scenario 2: ve 123 is to be moved vz1 to vz<br />
<br />
vzmlocal 123:123:/vz/private/123:/vz/root/123<br />
<br />
vzmlocal will reboot the ve at the end of the move<br />
<br />
Manual/old way:<br />
<br />
1) <tt>vzctl stop 123</tt><br><br />
2) <tt>mv /vz1/private/123 /vz/private/.</tt><br><br />
(or cp -a if you want to copy)<br />
3) in <tt>/etc/sysconfig/vz-scripts/123.conf</tt> change value<br><br />
of '<tt>VE_PRIVATE</tt>' variable to point to a new private area location<br />
4) <tt>vzctl start 123</tt><br><br />
5) update backups if needed: <tt>mvbackups 123 virtX virt1 vz</tt><br><br />
6) update management scerens<br><br />
<br />
Notes: a) absolute path to private area is stored in quota file <tt>/var/vzquota/quota.123</tt> - so during first startup quota will be recalculated.<br><br />
b) if you're going to write some script to do a job, you MUST be sure that $VEID won't be expanded to '' in ve config file - ie. you need to escape '$'. Otherwise you might have:<br />
<br />
VE_PRIVATE="/vz/private/"<br />
<br />
in config, and 'vzctl destroy' for this VE ID '''will remove everything under /vz/private/ directory'''.<br />
<br />
== Adding a veth device to a VE ==<br />
<br />
Not totally sure what this is, but a customer asked for it and here's what we did (as instructed by vz support):<br />
<br />
<pre>v set 99 --netif_add eth99 --save<br />
ipdel 99 69.55.230.58<br />
v set 99 --ifname eth99 --ipadd 69.55.230.58 --save<br />
v set 99 --ifname eth99 --gateway 69.55.230.1 --save<br />
<br />
vznetcfg net new veth_net<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
net99 active eth0 veth77.77,veth99.99<br />
veth_net active<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
br0 bridge veth_net<br />
br99 bridge net99<br />
veth99.99 veth net99<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic net99 69.55.227.70/24<br />
<br />
vznetcfg br attach br0 eth0<br />
<br />
(will remove 99 from orig net and move to veth_net)<br />
vznetcfg net addif veth_net veth99.99<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
net99 active<br />
veth_net active eth0 veth77.77,veth99.99<br />
<br />
(delete the old crap)<br />
vznetcfg net del net99<br />
<br />
<br />
Then, to add another device in<br />
<br />
v set 77 --netif_add eth77 --save<br />
ipdel 77 69.55.230.78<br />
v set 77 --ifname eth77 --ipadd 69.55.230.78 --save<br />
v set 77 --ifname eth77 --gateway 69.55.230.1 --save<br />
v set 77 --save --ifname eth77 --network veth_net<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
veth77.77 veth<br />
br0 bridge veth_net<br />
veth99.99 veth veth_net<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic veth_net 69.55.227.70/24<br />
<br />
vznetcfg net addif veth_net veth77.77<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
veth77.77 veth veth_net<br />
br0 bridge veth_net<br />
veth99.99 veth veth_net<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic veth_net 69.55.227.70/24<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
veth_net active eth0 veth77.77,veth99.99<br />
<br />
another example<br />
<br />
v set 1182 --netif_add eth1182 --save<br />
ipdel 1182 69.55.236.217<br />
v set 1182 --ifname eth1182 --ipadd 69.55.236.217 --save<br />
v set 1182 --ifname eth1182 --gateway 69.55.236.1 --save<br />
vznetcfg net addif veth_net veth1182.1182<br />
v set 1182 --save --ifname eth1182 --network veth_net<br />
<br />
<br />
unused/not working commands:<br />
ifconfig veth99.0 0<br />
vznetcfg net list<br />
vznetcfg br new br99 net99<br />
vznetcfg br attach br99 eth0<br />
vznetcfg br show<br />
vznetcfg if list<br />
vznetcfg net addif net99 veth99.99<br />
vznetcfg net addif eth0 net99<br />
<br />
---------<br />
<br />
vznetcfg br new br1182 net1182<br />
<br />
vznetcfg br attach br99 eth0<br />
vznetcfg if list<br />
vznetcfg net addif net99 veth99.99<br />
vznetcfg net addif eth0 net99<br />
<br />
<br />
<br />
vznetcfg net addif eth0 net1182<br />
<br />
<br />
# vznetcfg net addif veth99.0 net99<br />
--- 8< ---<br />
<br />
vznetcfg net new net<br />
# vznetcfg net addif eth0 net99<br />
<br />
<br />
# vzctl set 99 --save --netif_add eth0 (at this stage veth99.0 interface have to appear<br />
on node)<br />
# vzctl set 99 --save --ifname eth0 --ipadd 69.55.230.58 (and probably few more arguments<br />
here - see 'man vzctl')<br />
# vznetcfg net addif veth99.0 net99</pre><br />
<br />
<br />
== Assigning/remove ip from a VE ==<br />
<br />
1. Add or remove ips:<br />
ipdel 1234 1.1.1.1 2.2.2.2<br />
ipadd 1234 1.1.1.1 2.2.2.2<br />
<br />
2. update Mgmt screens<br />
<br />
3. offer to update any DNS we do for them<br />
<br />
4. check to see if we had rules for old IP in firwall<br />
<br />
== Enabling tun device for a ve ==<br />
Note, there’s a command for this: [[#addtun|addtun]]<br />
<br />
For posterity:<br />
Make sure the tun.o module is already loaded before Virtuozzo is started: <br />
lsmod <br />
Allow the VPS to use the TUN/TAP device: <br />
vzctl set 101 --devices c:10:200:rw --save <br />
Create the corresponding device inside the VPS and set the proper permissions: <br />
vzctl exec 101 mkdir -p /dev/net <br />
vzctl exec 101 mknod /dev/net/tun c 10 200 <br />
vzctl exec 101 chmod 600 /dev/net/tun<br />
<br />
== Remaking a system (on same virt) ==<br />
<br />
1. [[#cancelve|cancelve]] (or v destroy x - ONLY if you're POSITIVE no data needs to be saved)<br />
<br />
2. [[#vemake|vemake]] using same veid<br />
<br />
3. [[#mvbackups|mvbackups]] or [[#vb|vb]] (if new mount point)<br />
<br />
4. update mgmt with new dir/ip <br />
<br />
5. update firewall if changed ip<br />
<br />
== Re-initialize quota for a VE ==<br />
<br />
There’s a commamd for this now: [[#clearquota|clearquota]]<br />
<br />
For posterity:<br />
<br />
vzctl stop 1<br />
vzquota drop 1<br />
vzctl start 1<br />
<br />
<br />
== Traffic accounting on linux ==<br />
<br />
DEPRECATED - all tracking is done via bwdb now. This is how we used to track traffic.<br />
<br />
TODO: update for diff versions of vz<br />
<br />
Unlike FreeBSD, where we have to add firewall count rules to the system to count the traffic, on virtuozzo counts the traffic for us. You an see the current traffic stats by running `vznetstat`:<br />
<br />
<pre># vznetstat<br />
VEID Net.Class Output(bytes) Input(bytes)<br />
-----------------------------------------------<br />
24218 1 484M 39M<br />
24245 1 463M 143M<br />
2451 1 2224M 265M<br />
2454 1 2616M 385M<br />
4149 1 125M 68M<br />
418 1 1560M 34M<br />
472 1 1219M 315M<br />
726 1 628M 317M<br />
763 1 223M 82M<br />
771 1 4234M 437M<br />
-----------------------------------------------<br />
Total: 13780M 2090M</pre><br />
<br />
As you can see the VEID is on a line with the in and out bytes. So, we simply run a cron job:<br />
<br />
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /root/vztrafdump.sh<br />
<br />
Just like we do on FreeBSD - this one goes through all the VEs in /vz/private and greps the line from vznetstat that matches them and dumps it in /jc_traffic_dump on their system. Then it does it again for all the VEs in /vz1/private. It is important to note that vznetstat runs only once, and the grepping is done from a temporary file that contains that output - we do this because running vznetstat once for each VE that we read out of /vz/private and /vz1/private would take way too long and be too intensive.<br />
<br />
You do not need to do anything to facilitate this other than make sure that that cron job is running - the vznetstat counters are always running, and any new VEs that are added to the system will be accounted for automatically.<br />
<br />
Traffic resetting no longer works with vz 2.6, so we disable the vztrafdump.sh on those virts.<br />
<br />
== Watchdog script ==<br />
<br />
On some of the older virts, we have a watchdog running that kills procs that are deemed bad per the following:<br />
<br />
/root/watchdog from quar1<br />
<br />
<pre>if echo $line | awk '{print $(NF-3)}' | grep [5-9]...<br />
then<br />
# 50-90%<br />
if echo $line | awk '{print $(NF-1)}' | grep "...:.."<br />
then<br />
# running for > 99min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
if echo $line | awk '{print $(NF-1)}' | grep "....m"<br />
then<br />
# running for > 1000min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
fi<br />
<br />
if echo $line | awk '{print $(NF-3)}' | grep [1-9]...<br />
then<br />
# running for 10-90 percent<br />
if echo $line | awk '{print $NF}' | egrep 'cfusion|counter|vchkpw'<br />
then<br />
<br />
if echo $line | awk '{print $(NF-1)}' | grep "[2-9]:.."<br />
then<br />
# between 2-9min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
elif echo $line | awk '{print $(NF-1)}' | grep "[0-9][0-9]:.."<br />
then<br />
# up to 99min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
fi<br />
fi</pre><br />
<br />
<br />
== Misc Linux Items ==<br />
<br />
We are overselling hard drive space ... when you configure a linux system with a certain amount of disk space (the default is 4gigs) you do not actually use up 4gigs of space on the system. The diskspace setting for a user is simply a cap, and they only use up as much space on the actual disk drive as they are actually using.<br />
<br />
When you create a new linux system, even though there are some 300 RPMs or so installed, if you run `df -k` you will see that the entire 4gig partition is empty - no space is being used. This is because the files in their system are "magic symlinks" to the template for their OS that is in /vz/template - however, any changes to any of those files will "disconnect" them and they will immediately begin using space in their system. Further, any new files uploaded (even if those new files overwrite existing files) will take up space on the partition.<br />
<br />
<br />
if you see this:<br />
<br />
<pre>[root@virt8 root]# vzctl stop 160 ; vzctl start 160<br />
VE is not running<br />
Starting VE ...<br />
VE is unmounted<br />
VE is mounted<br />
Adding IP address(es): 69.55.226.83 69.55.226.84<br />
bash ERROR: Can't change file /etc/sysconfig/network<br />
Deleting IP address(es): 69.55.226.83 69.55.226.84<br />
VE is unmounted<br />
[root@virt8 root]#</pre><br />
<br />
it probably means they no longer have /bin/bash - copy one in for them<br />
<br />
ALSO: another possibility is that they have removed the `ed` RPM from their system - it needs to be reinstalled into their system. But since their system is down, this is tricky ...<br />
<br />
VE startup scripts used by 'vzctl' want package 'ed' to be available inside VE. So if package 'ed' will be enabled in OS template config and OS template itself VE #827 is based on - this error should be fixed.<br />
<br />
yes, it is possible to add RPM to VE while it not running.<br />
Try to do following:<br />
<br />
<pre># cd /vz/template/<OS_template_with_ed_package>/<br />
# vzctl mount 827<br />
# rpm -Uvh --root /vz/root/827 --veid 827 ed-0.2-25.i386.vz.rpm</pre><br />
<br />
Usually theres an error, but its ok<br />
<br />
Note: replace 'ed-0.2-25.i386.vz.rpm' in last command with actual<br />
version of 'ed' package you have.<br />
<br />
----<br />
<br />
So how do I know what template the user has ? cat their conf file and it is listed in there. For example, if the conf file has:<br />
<br />
<pre>[root@virt12 sbin]# vc 1103<br />
…snip…<br />
OSTEMPLATE="debian-3.0/20030822"<br />
TEMPLATES="mod_perl-deb30/20030707 mod_ssl-deb30/20030703 mysql-deb30/20030707 proftpd-deb30/20030703 webmin-deb30/20030823 "<br />
…</pre><br />
<br />
then they are on debian 3.0, all of their system RPMs are in /vz/template/debian-3.0, and they are using version 20030822 of that debian 3.0 template. Also, they’ve also got additional packages installed (mod_perl, mod_ssl, etc). Those are also found under /vz/template<br />
<br />
----<br />
<br />
Edits needed to run java:<br />
<br />
When we first created the VEs, the default setting for privvmpages was 93000:94000 ... which was high enough that most people never had problems ... however, you can;t run java or jdk or tomcat or anything java related with that setting. We have found that by setting privvmpages to 610000:615000 that java runs just fine. That is now the default setting. It is exceedingly rare that anyone needs it higher than that, although we have seen it once or twice.<br />
<br />
Any problems with java at all - the first thing you need to do is see if the failcnt has raised for privvmpages.<br />
<br />
<pre># vzctl start 160<br />
Starting VE ...<br />
vzquota : (error) Quota on syscall for 160: Device or resource busy<br />
Running vzquota on failed for VE 160 [3]</pre><br />
<br />
This is because my pwd is _in_ their private directory - you can't start it until you move out<br />
<br />
People seem to have trouble with php if they are clueless newbies. Here are two common problems/solutions:<br />
<br />
no... but i figured it out myself. problem was the php.ini file that came<br />
vanilla with the account was not configured to work with apache (the<br />
ENGINE directive was set to off).<br />
<br />
everything else seems fine now.<br />
<br />
----<br />
<br />
the problem was in the php.ini file. I noticed that is wasnt showing<br />
the code when it was in an html file so I looked at the php.ini file<br />
and had to change it so it recognized <? tags aswell as <?php tags.<br />
<br />
Also, make sure added to httpd.conf<br />
AddType application/x-httpd-php .php<br />
<br />
----<br />
<br />
You can set the time zone:<br />
<br />
You can change the timezone by doing this:<br />
<br />
ln -sf /usr/share/zoneinfo/<zone> /etc/localtime<br />
<br />
where <zone> is the zone you want in the /usr/share/zoneinfo/ directory.<br />
<br />
----<br />
<br />
Failing shm_open calls:<br />
<br />
first, please check if /dev/shm is mounted inside VE.<br />
'cat /proc/mounts' command should show something like this:<br />
tmpfs /dev/shm tmpfs rw 0 0<br />
<br />
If /dev/shm is not mounted you have 2 ways to solve issue:<br />
1. execute following command inside VE (doesn't require VE reboot):<br />
mount -t tmpfs none /dev/shm<br />
2. add following string to /etc/fstab inside VE and reboot it:<br />
tmpfs /dev/shm tmpfs defaults 0 0<br />
<br />
----<br />
<br />
You can have a mounted but not running ve<br />
Just:<br />
vzctl mount <veid><br />
<br />
----<br />
<br />
When a debian sys can’t get on the network, and you try:<br />
<br />
<pre>vzctl set 1046 --ipadd 69.55.227.117<br />
Adding IP address(es): 69.55.227.117<br />
Failed to bring up lo.<br />
Failed to bring up venet0.</pre><br />
<br />
They probably removed iproute package, which must be the one from swsoft. To restore:<br />
<pre># dpkg -i --veid=1046 --admindir=/vz1/private/1046/root/var/lib/dpkg --instdir=/vz1/private/1046/root/ /vz/template/debian-3.0/iproute_20010824-8_i386.vz.deb<br />
(Reading database ... 16007 files and directories currently installed.)<br />
Preparing to replace iproute 20010824-8 (using .../iproute_20010824-8_i386.vz.deb) ...<br />
Unpacking replacement iproute ...<br />
Setting up iproute (20010824-8) ...</pre><br />
<br />
Then restart their ve<br />
<br />
----<br />
<br />
in a ve i do:<br />
<br />
<pre>cd /<br />
du -h .</pre><br />
<br />
and get: 483M .<br />
<br />
i do:<br />
<br />
<pre>bash-2.05a# df -h<br />
Filesystem Size Used Avail Use% Mounted on<br />
/dev/vzfs 4.0G 2.3G 1.7G 56% /</pre><br />
<br />
how can this be?<br />
<br />
Is it possible that quota file was corrupted somehow? Please try to: <br />
<pre>vzctl stop <VEID><br />
vzquota drop <VEID><br />
vzquota init <VEID><br />
vzctl start <VEID></pre><br />
<br />
----<br />
<br />
How to stop vz from starting after reboot:<br />
<br />
VIRTUOZZO=no <br />
in <br />
/etc/sysconfig/vz<br />
<br />
To start: <br />
service vz start<br />
(after setting VIRTUOZZO=yes in /etc/sysconfig/vz)<br />
<br />
service vz restart will do some kind of 'soft reboot' -- restart all<br />
VPSes and reload modules without rebooting the node<br />
<br />
if you need to shut down all VPSes really really fast, run killall -9 init<br />
<br />
----<br />
<br />
Postfix tip:<br />
<br />
You may want to tweak settings: default_process_limit=10<br />
<br />
= Virtuozzo VPS Management Tools =<br />
<br />
== vm ==<br />
<br />
TODO<br />
<br />
== cancelve ==<br />
cancelve <veid><br />
this will:<br />
* stop a ve<br />
* check for backups (offer to remove them from the backup server <br />
and the backup.config)<br />
* rename the private dir<br />
* check for PTR, provide the commands to reset to default<br />
* and rename the ve’s config<br />
* remind you to remove firewall rules<br />
* remind you to remove DNS entries<br />
<br />
== ipadd ==<br />
ipadd <veid> <ip> [ip] [ip]<br />
add’s ip(s) to a ve<br />
<br />
== ipdel ==<br />
ipdel <veid> <ip> [ip] [ip]<br />
removes ip(s) from a ve<br />
<br />
== vc ==<br />
vc <veid><br />
the equivalent of: <tt>cat /vzconf/<veid>.conf</tt><br />
<br />
== vl ==<br />
vl<br />
will displays a list of ve #’s, 1 per line. (ostensibly to use in a for loop)<br />
<br />
== vp ==<br />
vp <veid><br />
the equivalent of: <tt>vzps auxww –E <veid></tt><br />
<br />
== vpe ==<br />
DEPRECATED<br />
vpe <veid> <br />
this will allow you to do a vp when a ve is running out of control, the equivalent of (deprecated since vp operates outside the VPS): <br />
<pre>vzctl set <veid> --kmemsize 2100000:2200000<br />
vzctl exec <veid> ps auxw<br />
vzctl set <veid> --kmemsize (ve’s orig lvalue):(ve’s orig hvalue)</pre><br />
<br />
== vt ==<br />
vt <veid><br />
the equivalent of: <tt>vztop –E <veid></tt><br />
<br />
== vr ==<br />
vr <veid><br />
the equivalent of: <tt>vzctl stop <veid>; vzctl start <veid></tt><br />
You can run this even if the ve is down - the stop command will just fail<br />
<br />
== vs ==<br />
vs [veid]<br />
displays status (output of <tt>vzctl status <veid></tt>) on each ve configured on the system (as determined by <tt>/vzconf/*.conf</tt>)<br />
If passed an argument, gives the status for just that ve. <br />
A running system looks like:<br />
<tt>VEID 16066 exist mounted running</tt><br />
<br />
A ve which is not running (but does exist) looks like:<br />
<tt>VEID 9990 exist unmounted down</tt><br />
<br />
A ve which is not running and doesn’t exist looks like:<br />
<tt>VEID 421 deleted unmounted down</tt><br />
<br />
== vs2 ==<br />
vs2 [veid]<br />
this is similar to vs in that it displays status (output of <tt>vzctl status <veid></tt>) on each ve,<br />
but the difference is it’s list comes from doing an ls on the data dirs. This was meant to catch <br />
the rare case where a ve configured but exists. <br />
<br />
== vw ==<br />
vw [veid]<br />
displays the output of ‘<tt>w</tt>’ (the equivalent of <tt>vzctl exec <veid> w</tt>) for each configured ve (as determined by <tt>/vzconf/*.conf</tt>). Useful for determing which ve is contributing to a heavily-loaded system.<br />
If passed an argument, gives ‘<tt>w</tt>‘ output for just that ve. <br />
Ex:<br />
<pre>[root@virt2 etc]# vw<br />
134<br />
10:52pm up 79 days, 6:14, 0 users, load average: 0.02, 0.02, 0.00<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT<br />
16027<br />
2:52pm up 7 days, 19:54, 0 users, load average: 0.00, 0.00, 0.00<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT<br />
16055<br />
2:52pm up 79 days, 6:38, 0 users, load average: 0.00, 0.04, 0.07<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT</pre><br />
<br />
== vwe ==<br />
vwe [constraint]<br />
just like <tt>vw</tt>, but takes a constraint as an argument, only show’s ve’s with loads >= the constraint provided. If no constraint is provided, 1 is used by default<br />
<br />
== vzs ==<br />
vzs [veid]<br />
displays the beancounter status for all ve’s, or a particular ve if an argument is passed<br />
<br />
== ve ==<br />
ve <veid><br />
the equivalent of: <tt>vzctl enter <veid></tt><br />
<br />
== vx ==<br />
vx <veid> <command><br />
the equivalent of: <tt>/usr/sbin/vzctl exec <veid> <command></tt><br />
<br />
== dprocs ==<br />
dprocs [count]<br />
a script which outputs a continuous report (or a certain number of reports if an option is passed) of processes stuck in the D state and which VPS’s those procs belong to.<br />
<br />
== setmem ==<br />
setmem <VEID> <256|512|768|1024|2048><br />
adjusts the memory resources for the VE<br />
<br />
== afacheck.sh ==<br />
afacheck.sh<br />
displays the health/status of containers and mirrors on an adaptec card (currently quar1, tempvirt1-2, virt9, virt10)- all other are LSI<br />
<br />
== backup ==<br />
backup<br />
backup script called nightly to update virt scripts and do customer backups<br />
<br />
== checkload.pl ==<br />
checkload.pl<br />
this was intended to be setup as a cronjob to watch processes on a virt when the load <br />
rises above a certain level. Not currently in use.<br />
<br />
== findbackuppigs.pl ==<br />
findbackuppigs.pl<br />
looks for files larger than 50MB which customers have asked us to backup. Emails matches<br />
to linux@johncompanies.com<br />
<br />
== gatherlinux.pl ==<br />
gatherlinux.pl<br />
gathers up data about ve’s configured and writes to a file. Used for audits against the db<br />
<br />
== gb ==<br />
gb <search><br />
greps backup.config for the given search parameter<br />
<br />
== gbg ==<br />
gbg <search><br />
greps backup.config for the given search parameter and presents just the directories (for clean pasting)<br />
<br />
== linuxtrafficgather.pl ==<br />
linuxtrafficgather.pl [yy-mm]<br />
sends a traffic usage summary by ve to support@johncomapnies.com and payments@johncompanies.com.<br />
Optional arguments are year and month (must be in the past). If not passed, assumes last month. Relies on <br />
traffic logs created by netstatreset and netstatbackup<br />
<br />
== linuxtrafficwatch.pl ==<br />
linuxtrafficwatch.pl<br />
checks traffic usage and emails support@johncomapnies.com when a ve reaches the warning level (35G) and<br />
the limit (40G). Works on virtuozzo versions <= 2.6.x. We really aren’t using this anymore now that we have netflow.<br />
<br />
== linuxtrafficwatch2.pl ==<br />
linuxtrafficwatch2.pl<br />
checks traffic usage and emails support@johncomapnies.com when a ve reaches the warning level (35G) and<br />
the limit (40G). Works on virtuozzo version 2.6.x. We really aren’t using this anymore now that we have netflow.<br />
<br />
== load.pl ==<br />
load.pl<br />
feeds info to load mrtg - executed by inetd.<br />
<br />
== mb (linux) ==<br />
mb <mount|umount><br />
(nfs) mounts and umounts dirs to backup2. Shortcuts are mbm and mbu to mount and unmount. <br />
<br />
== migrate ==<br />
migrate <ip of node migrating to> <veid> <target_veid> [target dir: vz | vz1 | vz2]<br />
this is basically a wrapper for <tt>vzmigrate</tt> – vzmigrate is a util to seamlessly move a ve from one host to another. This wrapper was written cause vz virtuozzo version 2.6 had a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables. This script mitigates that. Since it makes multiple ssh connections to the target host, it’s a good idea to put the pub key for the src system in the authorized_keys file on the target host. In addition, it emails ve owners when their migration starts and stops (if they place email addresses in a file on their system: /migrate_notify). To move everyone off a system, you’d do:<br />
for f in `vl`; do migrate <ip> $f; done<br />
<br />
== migrateonline ==<br />
migrateonline <ip of node migrating to> <veid> <target_veid> [target dir: vz | vz1 | vz2]<br />
this is the same as migrate but will migrate a ve in <tt>–online</tt> mode which means it won’t be shut down at the end of the migration. This only works when migrating ve’s between 2 machines running a 2.6 kernel (currently tempvirt1-2. virt16-19, virt12). If you get an error that the machine you’re trying to migrate to has a different CPU or features, etc, then you have to edit the file and add the –f switch to the vzmigrate line- you can basically ignore this kind of warning (but never ignore a warning about missing templates on the destination node). NOTE: This edit (if made to migrateonline) will be overwritten by the base script during each night’s backup.<br />
<br />
== netstatbackup ==<br />
DEPRECATED<br />
netstatbackup <br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.5.x. <br />
<br />
== netstatbackup2 ==<br />
DEPRECATED<br />
netstatbackup2<br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.6.x. <br />
<br />
== netstatreset ==<br />
DEPRECATED<br />
netstatreset<br />
writes traffic count data to a logfile and resets counters to 0. Works on virtuozzo versions 2.5.x <br />
<br />
== netstatreset2 ==<br />
DEPRECATED<br />
netstatreset2<br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.6.x. <br />
<br />
== orphanedbackupwatchlinux ==<br />
orphanedbackupwatchlinux <br />
looks for directories on backup2 which aren’t configured in backup.config and offers to <br />
delete them<br />
<br />
== rsync.backup (linux) ==<br />
rsync.backup<br />
does customer backups (relies on backup.config)<br />
<br />
== startvirt.pl ==<br />
startvirt.pl<br />
forks off start ve commands – keeps 6 running at a time. This is not to be used on systems where fastboot is enabled as it circumvents the benefit of the fastboot. The script will occasionally not exit gracefully and will continue to use up CPU, so it should be watched. Also, don’t exit from the script till you’re sure all ve’s are started – if you do you need to start them manually and may have to free up locks. On some systems, the startvirt script doesn’t exit cleanly and you have to ^C out of it. Be careful though- doing so can leave some VE’s in an odd bootup state and you may need to ‘vr’ them manually. You should check to see which ve’s aren’t running and/or confirm all have started when ^C’ing out of startvirt.<br />
<br />
== taskdone (linux) ==<br />
taskdone<br />
when called will email support@johncompanies.com with the hostname of the machine from which it was <br />
executed as the subject<br />
<br />
== vb (linux) ==<br />
vb<br />
the equivalent of: <tt>vi /usr/local/sbin/backup.config</tt><br />
<br />
== vemakeXX ==<br />
DEPRECATED<br />
vemakerh9 <br />
ve create script for RH9 (see vemake)<br />
<br />
vemakedebian30 <br />
ve create script for debian 3.0 (Woody) (see vemake)<br />
<br />
vemakedebian31 <br />
ve create script for debian 3.1 (Sarge) (see vemake)<br />
<br />
vemakedebian40 <br />
ve create script for debian 4.0 (Etch) (see vemake)<br />
<br />
vemakefedora, vemakefedora2, vemakefedora4, vemakefedora5, vemakefedora6, vemakefedora7<br />
ve create script for fedora core 1, 2, 4, 5, 6, 7 respectively (see vemake)<br />
<br />
vemakecentos3, vemakecentos4<br />
ve create script for centos 3, 4 respectively (see vemake)<br />
<br />
vemakesuse, vemakesuse93, vemakesuse100<br />
ve create script for suse 9.2, 9.3, 10.0 respectively (see vemake)<br />
<br />
vemakeubuntu5, vemakeubuntu606, vemakeubuntu606 vemakeubuntu704<br />
ve create script for ubuntu 5.10, 6.06, 6.10, 7.04 respectively (see vemake)<br />
<br />
== vemove ==<br />
DEPRECATED<br />
vemove <veid> <target_ip> </vz/private/123><br />
this script simplifies the old way of moving ve’s from one system to another - in short moving a ve to or from a virt running virtuozzo < 2.6.x<br />
It’s the equivalent of:<br />
<tt>tar cfpP - <veid> --ignore-failed-read | (ssh -2 -c arcfour <target_ip> "split - -b 1024m </vz/private/123>.tar" )</tt>This should only be used if migrate/vzmigrate can’t be used. <br />
<br />
== vim.watchdog ==<br />
vim.watchdog <br />
looks for and kills procs matching “vi|vim|nano|pine|elm” that are running for a long time & consuming lots of cpu. Works on virtuozzo versions 2.5.x<br />
<br />
== vim.watchdog2 ==<br />
vim.watchdog2<br />
looks for and kills procs matching “vi|vim|nano|pine|elm” that are running for a long time & consuming lots of cpu.<br />
Works on virtuozzo versions 2.6.x.<br />
<br />
== vzmigrate ==<br />
vzmigrate <target_ip> -r no <veid>:[dst veid]:[dst /vzX/private/veid]:[dst /vzX/root/veid]<br />
(this is the raw command “wrapped” by migrate/migrateonline) this will seamlessly move a ve from one host to another. The ve will run for the duration of the migration till the very end when it’s shut down, ip moved and started up on the target system. The filesystem on the src will remain. This should be watched – occasionally the move will timeout and leave the system shut down. If target private and root aren’t specified it just puts it in /vz. Only works when both systems are running virtuozzo 2.6.x<br />
<br />
== vztrafdump.sh ==<br />
DEPRECATED<br />
vztrafdump.sh<br />
writes traffic usage info by ve to a file called jc_traffic_dump in each ve’s / dir. Works on virtuozzo versions <= 2.5.x. <br />
<br />
== vztrafdump2.sh ==<br />
DEPRECATED<br />
vztrafdump2.sh<br />
writes traffic usage info by ve to a file called jc_traffic_dump in each ve’s / dir. Works on virtuozzo versions 2.6.x. <br />
<br />
== addtun ==<br />
addtun <veid><br />
Add’s tun device to ve.<br />
<br />
== bwcap ==<br />
bwcap <veid> <kbps><br />
Ex: <tt>bwcap 1234 512</tt><br />
Caps a VE’s bandwidth to the amount given<br />
<br />
== setdisk ==<br />
setdisk <veid> <diskspace in GB><br />
Ex: <tt>setdisk 1234 5</tt><br />
Gives a VE’s a given amount of disk space<br />
<br />
== vdf ==<br />
vdf <veid> <br />
the equivalent of: <tt>vzctl exec <veid> df –h</tt><br />
<br />
== vdff ==<br />
vdff<br />
runs a (condensed) vdf for all ve’s in your pwd (must be run from /vz/privateN)<br />
<br />
== mvbackups ==<br />
mvbackups <veid> <target_machine> (virt1) <target_dir> (vz1)<br />
moves backups from one location to another on the backup server, and provides you with option to remove entries from current backup.config, and simple paste command to add the config to backup.config on the target server<br />
<br />
== checkquota ==<br />
checkquota<br />
for all the ve’s in the cwd (run from /vz/private, /vz1/private, etc) reports what vz quota says they’re using and what the actual usage is (as reported by du)<br />
<br />
== clearquota ==<br />
clearquota <veid><br />
Recalculates a ve’s quota, prints out the usage before and after. The equivalent of:<br />
<tt>vdf <veid>; v stop <veid>; vzquota drop <veid>; v start <veid>; vdf <veid></tt><br />
<br />
== dprocs ==<br />
dprocs<br />
Sometimes the server’s have a large number of processes get stuck in the D state- this script shows (every 3 secs) which VE’s have D procs, which procs<br />
are stuck and a running average of the top “offenders”<br />
<br />
== vzstat ==<br />
vstat<br />
sort of like top for VZ. sort VEs by CPU usage by pressing 'o' and then 'c' keys<br />
<br />
== stopvirt ==<br />
stopvirt<br />
will stop VEs as fast as it can, 6 at a time. May not exit when complete so you should watch [[#vzstat|vzstat]] in another window.</div>Supporthttps://wiki.jcihosting.com/index.php?title=VPS_Management&diff=1035VPS Management2013-03-11T23:48:46Z<p>Support: /* Making new customer VE (vm) */</p>
<hr />
<div>= Common Problems =<br />
== Login to any machine without a password ==<br />
<br />
This is possible via the use of ssh keys. The process is thus:<br />
<br />
1. place the public key for your user (root@mail) in the /root/.ssh/authorized_keys file on the server you wish to login to<br />
cat /root/.ssh/id_dsa.pub<br />
(paste that into authorized_keys on the target server). If the file doesn't exist, create it.<br />
<br />
2. enable root login (usually only applies to FreeBSD). Edit the /etc/ssh/sshd_config on the target server and change:<br />
<tt>#PermitRootLogin no</tt><br />
to<br />
<tt>PermitRootLogin yes</tt><br />
<br />
3. Restart the sshd on the target machine. First, find the sshd process: <br />
jailps <hostname> | grep sshd <br />
or <br />
vp <VEID> | grep sshd<br />
<br />
Look for the process resembling:<br />
root 17296 0.0 0.0 5280 1036 ? Ss 2011 4:27 /usr/sbin/sshd <br />
(this is the sshd)<br />
<br />
Not:<br />
root 6270 0.5 0.0 6808 2536 ? Ss 14:33 0:00 sshd: root [priv]<br />
(this is an sshd child- someone already ssh'd in as root)<br />
<br />
Restart the sshd: <br />
kill -1 <PID><br />
<br />
Ex:<br />
kill -1 17296<br />
<br />
You may now ssh in.<br />
<br />
Once you're done, IF you enabled root login, you should repeat steps 2 and 3 to disable root logins.<br />
<br />
== Letting someone in who has locked themselves out (killed sshd, lost pwd) ==<br />
<br />
There are two ways people frequently lock themselves out - either they forget a password, or they kill off sshd somehow.<br />
<br />
These are actually both fairly easy to solve. First, let's say someone kills off their sshd, or somehow mangles /etc/ssh/sshd_config such that it no longer lets them in.<br />
<br />
Their email may be very short, or it may have all sorts of details about how you should fix sshd_config to let them in ... just ignore all of this. They can fix their own mangled sshd. Fixing this is very simple. First, edit the /etc/inetd.conf on their system and uncomment the telnet line:<br />
<br />
telnet stream tcp nowait root /usr/libexec/telnetd telnetd<br />
#telnet stream tcp6 nowait root /usr/libexec/telnetd telnetd<br />
<br />
(just leave the tcp6 version of telnet commented)<br />
<br />
Then, use jailps to list the processes on their system, and find their inetd process. Then simply:<br />
<br />
kill -HUP (pid)<br />
<br />
where (pid) is the PID of their inetd process. Now they have telnet running on their system and they can log in and do whatever they need to do.<br />
<br />
The only complications that could occur are:<br />
<br />
a) their firewall config on our firewall has port 23 blocked, in which case you will need to open that - will be covered in a different lesson.<br />
<br />
b) they are not running inetd, so you can't HUP it. If this happens, edit their /etc/rc.conf, add the inetd_enable="YES" line, and then kill<br />
their jail with /tmp/jailkill.pl - then restart their jail with the jail line from their quad/safe file. Easy.<br />
<br />
If they have forgotten a password,<br />
<br />
On 6.x+ you can reset their password with:<br />
jexec <jailID from jls> passwd root<br />
<br />
Note: the default password for 6.x jails is 8ico2987, for 4.x it is p455agfa<br />
<br />
On 4.x, you need to cd to their etc directory<br />
... for instance:<br />
<br />
cd /mnt/data2/198.78.65.136-col00261-DIR/etc<br />
<br />
and run:<br />
<br />
vipw -d .<br />
<br />
Then paste in these two lines (theres a paste with these):<br />
<br />
root:$1$krszPxhk$xkCepSnz3mIikT3vCtJCt0:0:0::0:0:Charlie &:/root:/bin/csh<br />
user:$1$Mx9p5Npk$QdMU6c8YQqp2FW2M3irEh/:1001:1001::0:0:User &:/home/user:/bin/sh<br />
<br />
overwriting the lines they already have for "user" and "root" - then just tell them that both user and root have been reset to the default password of p455agfa.<br />
<br />
For linux, just passwd inside shell or <br />
vzctl set <veid> --userpasswd root:p455agfa –save<br />
<br />
Starting in 2009 we began giving out randomized passwords for FreeBSD and Linux as the default password. That is stored with each system in Mgmt. You should look for and reset the password to that password in the event of a reset and refer the customer to use their original password from their welcome email- this way we don’t have to send the password again via email (in clear text).<br />
<br />
<br />
== sendmail can’t be contacted from ext ip (only locally) ==<br />
<br />
By default redhat puts this line in sendmail.mc:<br />
<br />
DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')<br />
<br />
which makes it only answer on localhost. Comment it out like:<br />
<br />
dnl DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')<br />
<br />
and then rebuild sendmail.cf with:<br />
<br />
m4 /etc/mail/sendmail.mc > /etc/sendmail.cf<br />
<br />
== virt doesn’t properly let go of ve’s ip(s) when moved to another system ==<br />
<br />
On virtuozzo 2.6 systems, it's been observed that when moving ips from one virt to another that sometimes the routing table will not get updated to reflect the removal of the ip addresses.<br />
<br />
A recent example was a customer that was moving to a new ve on a new virt and the ip addresses were traded between the two ve's. After the trade the two systems were not able to talk to each other. When looking at the routing table for the old system all the ip addresses were still in the routing table as being local, like so:<br />
<br />
<pre>netstat -rn | grep 69.55.225.149<br />
69.55.225.149 0.0.0.0 255.255.255.255 UH 40 0 0 venet0</pre><br />
<br />
This was preventing traffic to the other system from being routed properly.<br />
The solution is to manually delete the route:<br />
<br />
route delete 69.55.225.149 gw 0.0.0.0<br />
<br />
Supposedly, this was fixed in 2.6.1<br />
<br />
== sshd on FreeBSD 6.2 segfaults ==<br />
<br />
First try to reinstall ssh<br />
<br />
<pre>cd /usr/src/secure<br />
cd lib/libssh<br />
make depend && make all install<br />
cd ../../usr.sbin/sshd<br />
make depend && make all install<br />
cd ../../usr.bin/ssh<br />
make depend && make all install</pre><br />
<br />
Failing that, find the library that’s messed up:<br />
<br />
<pre>ldd /usr/sbin/sshd<br />
libssh.so.3 => /usr/lib/libssh.so.3 (0x280a3000) <br />
libutil.so.5 => /lib/libutil.so.5 (0x280d8000) <br />
libz.so.3 => /lib/libz.so.3 (0x280e4000) <br />
libwrap.so.4 => /usr/lib/libwrap.so.4 (0x280f5000) <br />
libpam.so.3 => /usr/lib/libpam.so.3 (0x280fc000) <br />
libbsm.so.1 => /usr/lib/libbsm.so.1 (0x28103000) <br />
libgssapi.so.8 => /usr/lib/libgssapi.so.8 (0x28112000) <br />
libkrb5.so.8 => /usr/lib/libkrb5.so.8 (0x28120000) <br />
libasn1.so.8 => /usr/lib/libasn1.so.8 (0x28154000) <br />
libcom_err.so.3 => /usr/lib/libcom_err.so.3 (0x28175000) <br />
libroken.so.8 => /usr/lib/libroken.so.8 (0x28177000) <br />
libcrypto.so.4 => /lib/libcrypto.so.4 (0x28183000) <br />
libcrypt.so.3 => /lib/libcrypt.so.3 (0x28276000) <br />
libc.so.6 => /lib/libc.so.6 (0x2828e000) <br />
libmd.so.3 => /lib/libmd.so.3 (0x28373000)</pre><br />
<br />
md5 them and compare to other jail hosts or jails running on host<br />
<br />
for libcrypto reinstall:<br />
<pre>/usr/src/crypto<br />
make depend && make all install</pre><br />
<br />
= FreeBSD VPS =<br />
<br />
== Starting jails: Quad/Safe Files ==<br />
<br />
FreeBSD customer systems do not start up automatically at boot time. When one of our freebsd machines boots up, it boots up, and does nothing else. To start jails, we put the commands to start each jail into a shell script(s) and run the script(s). Jail startup is something that needs to be actively monitored, which is why we don’t just run the script automatically. More on monitoring later.<br />
<br />
NOTE: >=7.x we have moved to 1 quad file: <tt>quad1</tt>. Startups are not done by running each quad, but rather [[#startalljails|startalljails]] which relies on the contents of <tt>quad1</tt>. The specifics of this are lower in this article. What follows here applies for pre 7.x systems.<br />
<br />
There are eight files in <tt>/usr/local/jail/rc.d</tt>:<br />
<br />
<pre>jail3# ls /usr/local/jail/rc.d/<br />
quad1 quad2 quad3 quad4 safe1 safe2 safe3 safe4<br />
jail3#</pre><br />
<br />
four quad files and four safe files.<br />
<br />
Each file contains an even number of system startup blocks (total number of jails divided by 4)<br />
<br />
The reason for this is, if we make one large script to startup all the systems at boot time, it will take too long - the first system in the script will start up right after system boot, which is great, but the last system may not start for another 20 minutes.<br />
<br />
Since there is no way to parralelize this during the startup procedure, we simply open four terminals (in screen window 9) and run each script, one in each terminal. This way they all run simultaneously, and the very last system in each startup script gets started in 1/4th the time it would if there was one large file<br />
<br />
The files are generally organized so that quad/safe 1&2 have only jails from disk 1, and quad/safe 3&4 have jails from disk 2. This helps ensure that only 2 fscks on any disk are going on at once. Further, they are balanced so that all quad/safe’s finish executing around the same time. We do this by making sure each quad/safe has a similar number of jails and represents a similar number of inodes (see js).<br />
<br />
The other, very important reason we do it this way, and this is the reason there are quad files and safe files, is that in the event of a system crash, every single vn-backed filesystem that was mounted at the time of system crash needs to be fsck'd. However, fsck'ing takes time, so if we shut the system down gracefully, we don't want to fsck.<br />
<br />
Therefore, we have two sets of scripts - the four quad scripts are identical to the four safe scripts except for the fact that the quad scripts contain fsck commands for each filesystem.<br />
<br />
So, if you shut a system down gracefully, start four terminals and run safe1 in window one, and safe2 in window 2, and so on.<br />
<br />
If you crash, start four terminals (or go to screen window 9) and run quad1 in window one, and quad2 in window 2, and so on.<br />
<br />
Here is a snip of (a 4.x version) quad2 from jail17:<br />
<br />
<pre>vnconfig /dev/vn16 /mnt/data2/69.55.228.7-col00820<br />
fsck -y /dev/vn16<br />
mount /dev/vn16c /mnt/data2/69.55.228.7-col00820-DIR<br />
chmod 0666 /mnt/data2/69.55.228.7-col00820-DIR/dev/null<br />
jail /mnt/data2/69.55.228.7-col00820-DIR mail1.phimail.com 69.55.228.7 /bin/sh /etc/rc<br />
<br />
# moved to data2 col00368<br />
#vnconfig /dev/vn28 /mnt/data2/69.55.236.132-col00368<br />
#fsck -y /dev/vn28<br />
#mount /dev/vn28c /mnt/data2/69.55.236.132-col00368-DIR<br />
#chmod 0666 /mnt/data2/69.55.236.132-col00368-DIR/dev/null<br />
#jail /mnt/data2/69.55.236.132-col00368-DIR limehouse.org 69.55.236.132 /bin/sh /etc/rc<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
vnconfig /dev/vn22 /mnt/data2/69.55.228.13-col01063<br />
fsck -y /dev/vn22<br />
mount /dev/vn22c /mnt/data2/69.55.228.13-col01063-DIR<br />
chmod 0666 /mnt/data2/69.55.228.13-col01063-DIR/dev/null<br />
jail /mnt/data2/69.55.228.13-col01063-DIR www.widestream.com.au 69.55.228.13 /bin/sh /etc/rc<br />
<br />
# cancelled col00106<br />
#vnconfig /dev/vn15 /mnt/data2/69.55.238.5-col00106<br />
#fsck -y /dev/vn15<br />
#mount /dev/vn15c /mnt/data2/69.55.238.5-col00106-DIR<br />
#chmod 0666 /mnt/data2/69.55.238.5-col00106-DIR/dev/null<br />
#jail /mnt/data2/69.55.238.5-col00106-DIR mail.azebu.net 69.55.238.5 /bin/sh /etc/rc</pre><br />
<br />
As you can see, two of the systems specified are commented out - presumably those customers cancelled, or were moved to new servers.<br />
<br />
As you can see, the vnconfig line is the simpler command line, not the longer one that was used when it was first configured. As you can see, all that is done is, vnconfig the filesystem, then fsck it, then mount it. The fourth command is the `jail` command used to start the system – but that will be covered later.<br />
<br />
Here is the safe2 file from jail17:<br />
<br />
<pre>vnconfig /dev/vn16 /mnt/data2/69.55.228.7-col00820<br />
mount /dev/vn16c /mnt/data2/69.55.228.7-col00820-DIR<br />
chmod 0666 /mnt/data2/69.55.228.7-col00820-DIR/dev/null<br />
jail /mnt/data2/69.55.228.7-col00820-DIR mail1.phimail.com 69.55.228.7 /bin/sh /etc/rc<br />
<br />
# moved to data2 col00368<br />
#vnconfig /dev/vn28 /mnt/data2/69.55.236.132-col00368<br />
#mount /dev/vn28c /mnt/data2/69.55.236.132-col00368-DIR<br />
#chmod 0666 /mnt/data2/69.55.236.132-col00368-DIR/dev/null<br />
#jail /mnt/data2/69.55.236.132-col00368-DIR limehouse.org 69.55.236.132 /bin/sh /etc/rc<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
vnconfig /dev/vn22 /mnt/data2/69.55.228.13-col01063<br />
mount /dev/vn22c /mnt/data2/69.55.228.13-col01063-DIR<br />
chmod 0666 /mnt/data2/69.55.228.13-col01063-DIR/dev/null<br />
jail /mnt/data2/69.55.228.13-col01063-DIR www.widestream.com.au 69.55.228.13 /bin/sh /etc/rc<br />
<br />
# cancelled col00106<br />
#vnconfig /dev/vn15 /mnt/data2/69.55.238.5-col00106<br />
#mount /dev/vn15c /mnt/data2/69.55.238.5-col00106-DIR<br />
#chmod 0666 /mnt/data2/69.55.238.5-col00106-DIR/dev/null<br />
#jail /mnt/data2/69.55.238.5-col00106-DIR mail.azebu.net 69.55.238.5 /bin/sh /etc/rc</pre><br />
<br />
As you can see, it is exactly the same, but it does not have the fsck lines.<br />
<br />
Take a look at the last entry - note that the file is named:<br />
<br />
/mnt/data2/69.55.238.5-col00106<br />
<br />
and the mount point is named:<br />
<br />
/mnt/data2/69.55.238.5-col00106-DIR<br />
<br />
This is the general format on all the FreeBSD systems. The file is always named:<br />
<br />
IP-custnumber<br />
<br />
and the directory is named:<br />
<br />
IP-custnumber-DIR<br />
<br />
If you run safe when you need a fsck, the mount will fail and jail will fail:<br />
<br />
# mount /dev/vn1c /mnt/data2/jails/65.248.2.131-ns1.kozubik.com-DIR<br />
mount: /dev/vn1c: Operation not permitted<br />
<br />
No reboot needed, just run the quad script<br />
<br />
Starting with 6.x jails, we added block delimiters to the quad/safe files, the block looks like:<br />
<br />
<pre>echo '## begin ##: nuie.solaris.mu'<br />
fsck -y /dev/concat/v30v31a<br />
mount /dev/concat/v30v31a /mnt/data1/69.55.228.218-col01441-DIR<br />
mount_devfs devfs /mnt/data1/69.55.228.218-col01441-DIR/dev<br />
devfs -m /mnt/data1/69.55.228.218-col01441-DIR/dev rule -s 3 applyset<br />
jail /mnt/data1/69.55.228.218-col01441-DIR nuie.solaris.mu 69.55.228.218 /bin/sh /etc/rc<br />
echo '## end ##: nuie.solaris.mu'</pre><br />
<br />
These are more than just informative when running quad/safe’s, the echo lines MUST be present for certain tools to work properly. So it’s important that any updates to the hostname also be updated on the 2 echo lines. For example, if you try to startjail a jail with a hostname which is on the jail line but not the echo lines, the command will return with host not found.<br />
<br />
=== FreeBSD 7.x+ notes ===<br />
<br />
Starting with the release of FreeBSD 7.x, we are doing jail startups in a slightly different way. First, thereis only 1 file: <tt>/usr/local/jail/rc.d/quad1</tt><br />
There are no other quads or corresponding safe files. The reason for this is twofold, 1. We can pass –C to fsck which will tell is to skip the fsck if the fs is clean (no more need for safe files), 2. We have a new startup script which can be launched multiple times, running in parallel to start jails, where quad1 is the master jail file. <br />
Quad1 could still be run as a shell script, but it would take a very long time for it to run completely so it’s not advisable; or you should break it down into smaller chunks (like quad1, quad2, quad3, etc)<br />
<br />
Here is a snip of (a 7.x version) quad1 from jail2:<br />
<br />
<pre>echo '## begin ##: projects.tw.com'<br />
mdconfig -a -t vnode -f /mnt/data1/69.55.230.46-col01213 -u 50<br />
fsck -Cy /dev/md50c<br />
mount /dev/md50c /mnt/data1/69.55.230.46-col01213-DIR<br />
mount -t devfs devfs /mnt/data1/69.55.230.46-col01213-DIR/dev<br />
devfs -m /mnt/data1/69.55.230.46-col01213-DIR/dev rule -s 3 applyset<br />
jail /mnt/data1/69.55.230.46-col01213-DIR projects.tw.com 69.55.230.46 /bin/sh /etc/rc<br />
echo '## end ##: projects.tw.com'</pre><br />
<br />
Cancelled jails are no longer commented out and stored in quad1, rather they’re moved to <tt>/usr/local/jail/rc.d/deprecated</tt> <br />
<br />
To start these jails, start the 4 ssh sessions as you would for a normal crash and then instead of running quad1-4, instead run startalljails in each window. IMPORTANT- before running startalljails you should make sure you ran preboot once as it will clear out all the lockfiles and enable startalljails to work properly.<br />
<br />
== Problems with the quad/safe files ==<br />
<br />
When you run the quad/safe files, there are two problems that can occur - either a particular system will hang during initialization, OR a system will spit out output to the screen, impeding your ability to do anything. Or both.<br />
<br />
First off, when you start a jail, you see output like this:<br />
<br />
<pre>Skipping disk checks ...<br />
adjkerntz[25285]: sysctl(put_wallclock): Operation not permitted<br />
Doing initial network setup:.<br />
ifconfig: ioctl (SIOCDIFADDR): permission denied<br />
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384<br />
Additional routing options: TCP keepalive=YESsysctl:<br />
net.inet.tcp.always_keepalive: Operation not permitted.<br />
Routing daemons:.<br />
Additional daemons: syslogd.<br />
Doing additional network setup:.<br />
Starting final network daemons:.<br />
ELF ldconfig path: /usr/lib /usr/lib/compat /usr/X11R6/lib /usr/local/lib<br />
a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout /usr/X11R6/lib/aout<br />
Starting standard daemons: inetd cron sshd sendmail sendmail-clientmqueue.<br />
Initial rc.i386 initialization:.<br />
Configuring syscons: blanktime.<br />
Additional ABI support:.<br />
Local package initialization:.<br />
Additional TCP options:.</pre><br />
<br />
Now, let's look at this line, near the end:<br />
<br />
Local package initialization:.<br />
<br />
This is where a list of daemons that are set to start at boot time willshow up. You might see something like:<br />
<br />
Local package initialization: mysqld apache sendmail sendmail-clientmqueue<br />
<br />
Or something like this:<br />
<br />
Local package initialization: postgres postfix apache<br />
<br />
The problem is that many systems (about 4-5 per machine) will hang on that line. Basically it will get to some of the way through the total daemons to be started:<br />
<br />
Local package initialization: mysqld apache<br />
<br />
and will just sit there. Forever.<br />
<br />
Fortunately, pressing ctrl-c will break out of it. Not only will it break out of it, but it will also continue on that same line and start the other daemons:<br />
<br />
Local package initialization: mysqld apache ^c sendmail-clientmqueue<br />
<br />
and then continue on to finish the startup, and then move to the next system to be started.<br />
<br />
So what does this mean? It means that if a machine crashes, and you start four screen-windows to run four quads or four safes, you need to periodically cycle between them and see if any systems are stuck at that point, causing their quad/safe file to hang. A good rule of thumb is, if you see a system at that point in the startup, give it another 100 seconds - if it is still at the exact same spot, hit ctrl-c. Its also a good idea to go back into the quad file (just before the first command in the jail startup block) and note that this jail tends to need a control-c or more time as follows:<br />
<pre>echo '### NOTE ### slow sendmail'<br />
echo '### NOTE ###: ^C @ Starting sendmail.'</pre><br />
<br />
'''NEVER''' hit ctrl-c repeatedly if you don't get an immediate response - that will cause the following jail’s startup commands to be aborted.<br />
<br />
A second problem that can occur is that a jail - maybe the first one in that particular quad/safe, maybe the last one, or maybe one in the middle, will start spitting out status or error messages from one of its init scripts. This is not a problem - basically, hit enter a few times and see if you get a prompt - if you do get a prompt, that means that the quad/safe script has already completed. Therefore it is safe to log out (and log out of the user that you su'd from) and then log back in (if necessary).<br />
<br />
The tricky thing is, if a system in the middle starts flooding with messages, and you hit enter a few times and don't get a prompt. Are you not getting a prompt because some subsequent system is hanging at the initialization, as we discussede above ? Or are you not getting a prompt because that quad file is currently running an fsck ? Usually you can tell by scrolling back in screen’s history to see what it was doing before you started getting the messages.<br />
<br />
If you don’t get clues from history, you have to use your judgement - instead of giving it 100 seconds to respond, perhaps give it 2-3 mins ... if you still get no response (no prompt) when you hit enter, hit ctrl-c. However, be aware that you might still be hitting ctrl-c in the middle of an fsck. This means you will get an error like "filesystem still marked dirty" and then the vnconfig for it will fail and so will the jail command, and the next system in the quad file will then start starting up.<br />
<br />
If this happens, just wait until the end of all the quad files have finished, and start that system manually.<br />
<br />
If things really get weird, like a screen flooded with errors, and you can't get a prompt, and ctrl-c does nothing, then you need to just eventually (give it ten mins or so) just kill that window with ctrl-p, then k, and then log in again and manually check which systems are now running and which aren't, and manually start up any that are not.<br />
<br />
Don't EVER risk running a particular quad/safe file a second time.<br />
If the quad/safe script gets executed twice, reboot the machine immediately.<br />
<br />
So, for all the above reasons, anytime a machine crashes and you run all the quads or all the safes, '''always''' check every jail afterwards to make sure it is running - even if you have no hangs or complications at all.<br />
Run this command:<br />
<br />
<tt>[[#jailpsall|jailpsall]]</tt> <br />
<br />
Note: [[#postboot|postboot]] also populates ipfw counts, so it '''should not be run multiple times''', use <tt>jailpsall</tt> for subsequent extensive ps’ing<br />
<br />
And make sure they all show as running. If one does not show as running, check its /etc/rc.conf file first to see if maybe it is using a different hostname first before starting it manually.<br />
<br />
One thing we have implemented to alleviate these startup hangs and noisy jails, is to put jail start blocks that are slow or hangy at the bottom of the safe/quad file. Further, for each bad jail we note in each quad/safe just before the start block something like:<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
<br />
That way we’ll be prepared to ^C when we see that message appear during the quad/safe startup process. If you observe a new, undocumented hang, '''after''' the quad/safe has finished, place a line similar to the above in the quad file, move the jail start block to the end of the file, then run [[#buildsafe|buildsafe]]<br />
<br />
<br />
== Recovering from a crash (FreeBSD) ==<br />
<br />
=== Diagnose whether you have a crash ===<br />
The most important thing is to get the machine and all jails back up as soon as possible. Note the time, you’ll need to create a crash log entry (Mgmt. -> Reference -> CrashLog). The first thing to do is head over to the [[Screen#Screen_Organization|serial console screen]] and see if there’s any kernel error messages output. Try to copy any messages (or just a sample of repeating messages) you see into the notes section of the crash log. If there are no messages, the machine may just be really busy- wait a bit (5-10min) to see if it comes back. If it's still pinging, odds are its very busy. Note, if you see messages about swap space exhausted, the server is obviously out of memory, however it may recover briefly enough for you to get a jtop in to see who's lauched a ton of procs (most likely) and then issue a quick jailkill to get it back under control.<br />
<br />
If it doesn't come back, or the messages indicate a fatal error, you will need to proceed with a power cycle (ctrl+alt+del will not work).<br />
<br />
=== Power cycle the server ===<br />
If this machine is not a Dell 2950 with a [[DRAC/RMM#DRAC|DRAC card]] (i.e. if you can’t ssh into the DRAC card (as root, using the standard root pass) and issue <br />
racadm serveraction hardreset<br />
then you will need someone at the data center power the macine off, wait 30 sec, then turn it back on. Make sure to re-attach via console:<br />
tip jailX<br />
immediately after power down. <br />
<br />
=== (Re)attach to the console ===<br />
Stay on the console the entire time during boot. As the BIOS posts- look out for the RAID card output- does everything look healthy? The output may be scrambled, look for "DEGRADED" or "FAILED". Once the OS starts booting you will be disconnected (dropped back to the shell on the console server) a couple times during the boot up. The reason you want to quickly re-attach is two-fold: 1. If you don’t reattach quickly then you won’t get any console output, 2. you want to be attached before the server ''potentially'' starts (an extensive) fsck. If you attach after the fsck begins, you’ll have seen no indication it started an fsck and the server will appear frozen during startup- no output, no response. <br />
<br />
IMPORTANT NOTE: on some older FreeBSD systems, there will be no output to the video (KVM) console as it boots up. The console output is redirected to the serial port ... so if a jail crashes, and you attach a kvm, the output during the bootup procedure will not be shown on the screen. However, when the bootup is done, you will get a login prompt on the screen and will be able to log in as normal. <tt>/boot/loader.conf</tt> is where serial console redirect output lives, so comment that if you want to catch output on kvm.<br />
On newer systems it sends most output to both locations. <br />
<br />
=== Assess the heath of the server ===<br />
Once the server boots up fully, you should be able to ssh in. Look around- make sure all the mounts are there and reporting the correct size/usage (i.e. /mnt/data1 /mnt/data2 /mnt/data3 - look in /etc/fstab to determine which mount points should be there), check to see if RAID mirrors are healthy. See [[RAID_Cards#Common_CLI_commands_.28megacli.29|megacli]], [[#aaccheck|aaccheck]]<br />
<br />
Before you start the jails, you need to run [[#preboot|preboot]]. This will do some assurance checks to make sure things are prepped to start the jails. Any issues that come out of preboot need to be addressed before starting jails.<br />
<br />
=== Start jails ===<br />
[[#Starting_jails:_Quad.2FSafe_Files|More on starting jails]]<br />
Customer jails (the VPSs) do not start up automatically at boot time. When a FreeBSD machines boots up, it boots up, and does nothing else. To start jails, we put the commands to start each jail into a shell script(s) and run the script(s). Jail startup is something that needs to be actively monitored, which is why we don’t just run the script automatically. <br />
<br />
In order to start jails, we run the quad files: quad1 quad2 quad3 and quad4 (on new systems there is only quad1). If the machine was cleanly rebooted- which wouldn't be the case if this was a crash, you may run the safe files (safe1 safe2 safe3 safe4) in lieu of quads. <br />
<br />
Open up 4 logins to the server (use the windows in [[Screen#Screen_Organization|a9]])<br />
In each of the 4 windows you will:<br />
<br />
If there is a [[#startalljails|startalljails]] script (and only quad1), run that command in each of the 4 windows. It will parse through the quad1 file and start each jail. Follow the instructions [[#Problems_with_the_quad.2Fsafe_files|here]] for monitoring startup. Note that you can be a little more lenient with jails that take awhile to start- startalljails will work around the slow jails and start the rest. As long as there aren't 4 jails which are "hung" during startup, the rest will get started eventually.<br />
-or-<br />
If there is no startalljails script, there will be multiple quad files. In each of the 4 windows, start each of the quads. i.e. start quad1 in window1, quad2 in window2 and so on. DO NOT start any quad twice. It will crash the server. If you accidentally do this, just jailkill all the jails which are in the quad and run the quad again. Follow the instructions here for monitoring quad startup.<br />
<br />
Note the time the last jail boots- this is what you will enter in the crash log.<br />
<br />
Save the crash log.<br />
<br />
=== Check to make sure all jails have started ===<br />
There's a simple script which will make sure all jails have started, and enter the ipfw counter rules: [[#postboot|postboot]] <br />
Run postboot, which will do a jailps on each jail it finds (excluding commented out jails) in the quad file(s). We're looking for 2 things:<br />
# systems spawning out of control or too many procs<br />
# jails which haven't started<br />
On 7.x and newer systems it will print out the problems (which jails haven't started) at the conclusion of postboot. <br />
On older systems you will need to watch closely to see if/when there's a problem, namely:<br />
<br />
[hostname] doesnt exist on this server<br />
<br />
When you get this message, it means one of 2 things:<br />
1. the jail really didn't start:<br />
When a jail doesn't start it usually boils down to a problem in the quad file. Perhaps the path name is wrong (data1 vs data2) or the name of the vn/mdfile is wrong. Once this is corrected, you will need to run the commands from the quad file manually, or you may use <tt>startjail <hostname></tt><br />
<br />
2. the customer has changed their hostname (and not told us) so their jail ''is'' running, just under a different hostname:<br />
On systems with jls, this is easy to rectify. First, get the customer info: <tt>g <hostname></tt><br />
Then look for the customer in jls: <tt>jls | grep <col0XXXX></tt><br />
From there you will see their new hostname- you should update that hostname in the quad file: don't forget to edit it on the <tt>## begin ##</tt> and <tt>## end ##</tt> lines, and in mgmt. <br />
On older systems without jls, this will be harder, you will need to look further to see their hostname- perhaps its in their /etc/rc.conf<br />
<br />
<br />
Once all jails are started, do some spot checks- try to ssh or browse to some customers, just to make sure things are really ok.<br />
<br />
== Adding disk to a 7.x/8.x jail ==<br />
or<br />
== Moving customer to a different drive (md) ==<br />
<br />
NOTE: this doesn’t apply to mx2 which uses gvinum. Use same procedure as 6.x<br />
NOTE: if you unmount before mdconfig, re-mdconfig (attach) then unmount then mdconfig -u again <br />
<br />
<br />
(parts to change/customize are <tt><span style="color:red">red</span></tt>)<br />
<br />
<br />
If someone wants more disk space, there’s a paste for it, send it to them (explains about downtime, etc).<br />
<br />
1. Figure out the space avail from <tt>js</tt>. Ideally, you want to put the customers new space on a different partition (and create the new md on the new partition). <br />
<br />
<br />
2. make a mental note of how much space they're currently using<br />
<br />
<br />
3. <tt>jailkill <hostname></tt> <br />
<br />
<br />
4. Umount it (including their devfs) but leave the md config’d (so if you use stopjail, you will have to re-mdconfig it)<br />
<br />
<br />
5. <tt>g <customerID></tt> to get the info (IP/cust#) needed to feed the new mdfile and mount name, and to see the current md device. <br />
<br />
<br />
6a. When there's enough room to place new system on an alternate, or the same drive:<br />
USE CAUTION not to overwrite (touch, mdconfig) existing md!!<br><br />
<tt>touch /mnt/data<span style="color:red">3/69.55.234.66-col01334</span><br><br />
mdconfig -a -t vnode -s 10g -f /mnt/data3/69.55.234.66-col01334 -u 97<br><br />
newfs /dev/md97</tt><br />
<br />
<br />
Optional- if new space is on a different drive, move the mount point directory AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
<tt>mount /dev/md<span style="color:red">97</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<br />
confirm you are mounted to /dev/md<span style="color:red">97</span> and space is correct:<br><br />
<tt>df .</tt> <br />
<br />
<br />
do the dump and pipe directly to restore:<br><br />
<tt>dump -0a -f - /dev/md<span style="color:red">1</span> | restore -r -f - <br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use <tt>df</tt> to confirm the restored data size matches the original usage figure<br />
<br />
<br />
md-unconfig old system:<br><br />
<tt>mdconfig -d -u <span style="color:red">1</span></tt><br />
<br />
<br />
archive old mdfile. <br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.237.26-col00241</span> /mnt/data<span style="color:red">1/old-col00241-mdfile-noarchive-20091211</span></tt><br />
<br />
<br />
edit quad (vq1) to point to new (/mnt/data<span style="color:red">3</span>) location AND new md number (md<span style="color:red">97</span>)<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
6b. When there's not enough room on an alternate partition, or on the same drive...but there is enough room if you were to remove the existing customer's space:<br />
<br />
<br />
mount backup nfs mounts:<br><br />
<tt>mbm</tt><br> <br />
(run <tt>df</tt> to confirm backup mounts are mounted)<br />
<br />
<br />
dump the customer to backup2 or backup1<br><br />
<tt>dump -0a -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> /dev/md<span style="color:red">1</span></tt><br><br />
(when complete WITHOUT errors, <tt>du</tt> the dump file to confirm it matches size, roughly, with usage)<br />
<br />
<br />
unconfigure and remove old mdfile<br><br />
<tt>mdconfig -d -u <span style="color:red">1</span><br><br />
rm /mnt/data<span style="color:red">1/69.55.237.26-col00241</span></tt><br><br />
(there should now be enough space to recreate your bigger system. If not, run sync a couple times)<br />
<br />
<br />
create the new system (ok to reuse old mdfile and md#):<br><br />
<tt>touch /mnt/data<span style="color:red">1/69.55.234.66-col01334</span><br><br />
mdconfig -a -t vnode -s <span style="color:red">10</span>g -f /mnt/data<span style="color:red">1/69.55.234.66-col01334</span> -u <span style="color:red">1</span><br><br />
newfs /dev/md<span style="color:red">1</span><br><br />
mount /dev/md<span style="color:red">1</span> /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> <br><br />
cd /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<br />
confirm you are mounted to /dev/md<span style="color:red">1</span> and space is correct:<br><br />
<tt>df .</tt> <br />
<br />
<br />
do the restore from the dumpfile on the backup server:<br><br />
<tt>restore -r -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> .<br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
umount nfs:<br><br />
<tt>mbu</tt><br />
<br />
<br />
If md# changed (or mount point), edit quad (<tt>vq1</tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new md number (md<span style="color:red">1</span>)<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
7. update disk (and dir if applicable) in mgmt screen<br />
<br />
<br />
8. update backup list AND move backups, if applicatble<br />
<br />
Ex: <tt>mvbackups <span style="color:red">69.55.237.26-col00241</span> jail<span style="color:red">9</span> data<span style="color:red">3</span></tt><br />
<br />
<br />
9. Optional: archive old mdfile<br />
<br />
<tt>mbm<br><br />
gzip -c old-col01588-mdfile-noarchive-20120329 > /deprecated/old-col01588-mdfile-noarchive-20120329.gz<br><br />
mbu<br><br />
rm old-col01588-mdfile-noarchive-20120329</tt><br />
<br />
== Adding disk to a 6.x jail (gvinum/gconcat) ==<br />
or<br />
== Moving customer to a different drive (gvinum/gconcat) ==<br />
<br />
(parts to change are <span style="color:red">highlighted</span>)<br />
<br />
<br />
If someone wants more disk space, there’s a paste for it, send it to them (explains about downtime, etc).<br />
<br />
<br />
1. Figure out the space avail from [[#js|js]]. Ideally, you want to put the customers new space on a different partition (and create the new md on the new partition). <br />
<br />
<br />
2. make a mental note of how much space they're currently using<br />
<br />
<br />
3. <tt>[[#stopjail|stopjail]] <hostname></tt> <br />
<br />
<br />
4. <tt>[[#g|g]] <customerID></tt> to get the info (IP/cust#) needed to feed the new mount name and existing volume/device. <br />
<br />
<br />
5a. When there's enough room to place new system on an alternate, or the same drive (using only UNUSED - including if it's in use by the system in question - gvinum volumes):<br />
<br />
<br />
Configure the new device:<br><br />
A. for a 2G system (single gvinum volume):<br><br />
<tt>bsdlabel -r -w /dev/gvinum/v<span style="color:red">123</span><br><br />
newfs /dev/gvinum/v<span style="color:red">123</span>a</tt><br><br />
<br />
-or- <br />
<br />
<br />
B. for a >2G system (create a gconcat volume):<br><br />
try to grab a contiguous block of gvinum volumes. gconcat volumes MAY NOT span drives (i.e. you cannot use a gvinum volume from data3 and a volume from data2 in the same gconcat volume). <br />
<br />
<tt>gconcat label <span style="color:red">v82-v84 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84</span><br><br />
bsdlabel -r -w /dev/concat/<span style="color:red">v82-v84</span><br><br />
newfs /dev/concat/<span style="color:red">v82-v84</span>a</tt><br />
<br />
<br />
Other valid gconcat examples:<br><br />
<tt>gconcat label v82-v84v109v112 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84 /dev/gvinum/v109 /dev/gvinum/v112<br><br />
gconcat label v82v83 /dev/gvinum/v82 /dev/gvinum/v83</tt> <br><br />
Note, long names will truncate: v144v145v148-v115 will truncate to v144v145v148-v1 (so you will refer to it as v144v145v148-v1 thereafter)<br />
<br />
<br />
Optional- if new volume is on a different drive, move the mount point directory (get the drive from js output) AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
<br />
confirm you are mounted to the device (<tt>/dev/gvinum/v<span style="color:red">123</span>a</tt> OR <tt>/dev/concat/<span style="color:red">v82-v84</span></tt>) and space is correct:<br><br />
A. <tt>mount /dev/gvinum/v<span style="color:red">123</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br><br />
-or-<br><br />
B. <tt>mount /dev/concat/<span style="color:red">v82-v84</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<tt>cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
df .</tt> <br />
<br />
<br />
do the dump and pipe directly to restore:<br><br />
<tt>dump -0a -f - /dev/gvinum/v<span style="color:red">1</span> | restore -r -f - <br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
edit quad (<tt>vq<span style="color:red">1</span></tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new volume (<tt>/dev/gvinum/v<span style="color:red">123</span>a</tt> or <tt>/dev/concat/<span style="color:red">v82-v84</span>a</tt>) , run <tt>buildsafe</tt><br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
5b. When there's not enough room on an alternate partition, or on the same drive...but there is enough room if you were to remove the existing customer's space (i.e. if you want/need to reuse the existing gvinum volumes and add on more):<br />
<br />
<br />
mount backup nfs mounts:<br><br />
<tt>mbm</tt> <br><br />
(run df to confirm backup mounts are mounted)<br />
<br />
<br />
dump the customer to backup2 or backup1<br><br />
<tt>dump -0a -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> /dev/<span style="color:red">gconcat/v106-v107</span></tt><br><br />
(when complete WITHOUT errors, du the dump file to confirm it matches size, roughly, with usage)<br />
<br />
<br />
unconfigure the old gconcat volume<br><br />
list member gvinum volumes:<br />
<br />
<tt>gconcat list <span style="color:red">v106v107</span></tt><br />
<br />
<br />
Output will resemble:<br><br />
<pre>Geom name: v106v107<br />
State: UP<br />
Status: Total=2, Online=2<br />
Type: AUTOMATIC<br />
ID: 3530663882<br />
Providers:<br />
1. Name: concat/v106v107<br />
Mediasize: 4294966272 (4.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e2<br />
Consumers:<br />
1. Name: gvinum/sd/v106.p0.s0<br />
Mediasize: 2147483648 (2.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e3<br />
Start: 0<br />
End: 2147483136<br />
2. Name: gvinum/sd/v107.p0.s0<br />
Mediasize: 2147483648 (2.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e3<br />
Start: 2147483136<br />
End: 4294966272</pre><br />
<br />
stop volume and clear members<br><br />
<tt>gconcat stop <span style="color:red">v106v107</span><br><br />
gconcat clear <span style="color:red">gvinum/sd/v106.p0.s0 gvinum/sd/v107.p0.s0</span></tt><br />
<br />
<br />
create new device- and its ok to reuse old/former members<br><br />
try to grab a contiguous block of gvinum volumes. gconcat volumes MAY NOT span drives (i.e. you cannot use a gvinum volume from data3 and a volume from data2 in the same gconcat volume). <br><br />
<tt>gconcat label <span style="color:red">v82-v84v106v107 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84 /dev/gvinum/v106 /dev/gvinum/v107</span> <br><br />
bsdlabel -r -w /dev/concat/<span style="color:red">v82-v84v106v107</span><br><br />
newfs /dev/concat/<span style="color:red">v82-v84v106v107</span>a</tt><br />
<br />
<br />
Optional- if new volume is on a different drive, move the mount point directory (get the drive from js output) AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
confirm you are mounted to the device (/dev/concat/<span style="color:red">v82-v84v106v107</span>) and space is correct:<br><br />
<tt>mount /dev/concat/<span style="color:red">v82-v84v106v107</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<tt>cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
df .</tt> <br />
<br />
do the restore from the dumpfile on the backup server:<br><br />
<tt>restore -r -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> .<br><br />
rm restoresymtable</tt><br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
edit quad (<tt>vq<span style="color:red">1</span></tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new volume (<tt>/dev/concat/<span style="color:red">v82-v84v106v107</span>a</tt>), run buildsafe<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
TODO: clean up/clear old gvin/gconcat vol<br />
<br />
<br />
6. update disk (and dir if applicable) in mgmt screen<br />
<br />
<br />
7. update backup list AND move backups, if applicatble<br />
<br />
<br />
Ex: <tt>mvbackups <span style="color:red">69.55.237.26-col00241</span> jail<span style="color:red">9</span> data<span style="color:red">3</span></tt><br />
<br />
<br />
<br />
DEPRECATED - steps to tack on a new gvin to existing gconcat- leads to corrupted fs<br />
bsdlabel -e /dev/concat/v82-v84<br />
<br />
To figure out new size of the c partition, multiply 4194304 by the # of 2G gvinum volumes and subtract the # of 2G volumes:<br />
10G: 4194304 * 5 – 5 = 20971515<br />
8G: 4194304 * 4 – 4 = 16777212<br />
6G: 4194304 * 3 – 3 = 12582909<br />
4G: 4194304 * 2 – 2 = 8388606<br />
<br />
To figure out the new size of the a partition, subtract 16 from the c partition:<br />
10G: 20971515 – 16 = 20971499<br />
8G: 16777212 – 16 = 16777196<br />
6G: 12582909 – 16 = 12582893<br />
4G: 8388606 – 16 = 8388590<br />
<br />
Orig:<br />
8 partitions:<br />
# size offset fstype [fsize bsize bps/cpg]<br />
a: 8388590 16 4.2BSD 2048 16384 28552<br />
c: 8388606 0 unused 0 0 # "raw" part, don't edit<br />
<br />
New:<br />
8 partitions:<br />
# size offset fstype [fsize bsize bps/cpg]<br />
a: 12582893 16 4.2BSD 2048 16384 28552<br />
c: 12582909 0 unused 0 0 # "raw" part, don't edit<br />
<br />
sync; sync<br />
<br />
growfs /dev/concat/v82-v84a<br />
<br />
fsck –fy /dev/concat/v82-v84a<br />
<br />
sync<br />
<br />
fsck –fy /dev/concat/v82-v84a<br />
<br />
(keep running fsck’s till NO errors)<br />
<br />
== Problems un-mounting - and with mount_null’s ==<br />
<br />
If you cannot unmount a filesystem, beacuse it says the filesystem is busy, it is because of three things:<br />
<br />
a) the jail is still running<br />
<br />
b) you are actually in that directory, even though the jail is stopped<br />
<br />
c) there are still dev, null_mount or linprocfs mount points mounted inside that directory.<br />
<br />
d) when trying to umount null_mounts that are really long and you get an error like “No such file or directory”, it’s an OS bug where the dir is truncated. No known fix<br />
<br />
e) there are still files open somewhere inside the dir. Use <tt>fstat | grep <cid></tt> to find the process that has files open<br />
<br />
f) Starting with 6.x, the jail mechanism does a poor job of keeping track of processes running in a jail and if it thinks there are still procs running, it will refuse to umount the disk. If this is happening you should see a low number in the #REF column when you run jls. In this case you ''can'' safely <tt>umount –f</tt> the mount. <br />
<br />
Please note -if you forcibly unmount a (4.x) filesystem that has null_mounts<br />
still mounted in it, the system '''will crash''' within 10-15 mins.<br />
<br />
== Misc jail Items ==<br />
<br />
We are overselling hard drive space on jail2, jail8, jail9, a couple jails on jail17, jail4, jail12 and jail18.<br />
Even though the vn file shows 4G size, it doesn’t actually occupy that amount of space on the disk. So be careful not to fill up drives where we’re overselling – use oversellcheck to confirm you’re not oversold by more than 10G.<br />
There are other truncated jails, they are generally noted in a the file on the root system: /root/truncated<br />
<br />
The act of moving a truncated vn to another system un-does the truncating- the truncated vn is filled with 0’s and it occupies physical disk space for which it’s configured. So, you should use dumpremote to preserve the truncation.<br />
<br />
= FreeBSD VPS Management Tools =<br />
<br />
These files are located in /usr/local/jail/rc.d and /usr/local/jail/bin<br />
<br />
== jailmake ==<br />
<br />
Applies to 7.x+ <br />
On older systems syntax differs, run jailmake once to see.<br />
<br />
Note: this procedure differs on mx2 which is 7.x but still uses gvinum<br />
<br />
# run js to figure out which md’s are in use, which disk has enough space, IP to put it on<br />
# use col00xxx for both hostnames if they don’t give you a hostname<br />
# copy over dir, ip and password to pending customer screen<br />
<br />
<tt>Usage: jailmake IP[,IP] CID disk[1|2|3] md# hostname shorthost ipfw# email [size in GB]</tt><br />
<br />
Ex: <br />
<br />
Jail2# jailmake 69.55.234.66 col01334 3 97 vps.bsd.it vps 1334 fb@bsd.it<br />
<br />
== jailps ==<br />
jailps [hostname]<br />
DEPRECATED FOR jps: displays processes belonging to/running inside a jail. The command<br />
takes one (optional) argument – the hostname of the jail you wish to query. If you don’t <br />
supply an argument, all processes on the machine are listed and grouped by jail. <br />
<br />
== jps ==<br />
jps [hostname]<br />
displays processes belonging to/running inside a jail. The command<br />
takes one (optional) argument – the hostname or ID of the jail you wish to query. <br />
<br />
== jailkill ==<br />
jailkill <hostname><br />
stops all process running in a jail.<br />
<br />
You can also run:<br />
jailkill <JID><br />
<br />
=== problems ===<br />
Occasionally you will hit an issue where jail will not kill off:<br />
<br />
<pre>jail9# jailkill www.domain.com<br />
www.domain.com .. killed: none<br />
jail9#</pre><br />
Because no processes are running under that hostname. You cannot use jailps.pl either:<br />
<br />
<pre>jail9# jailps www.domain.com<br />
www.domain.com doesn’t exist on this server<br />
jail9#</pre><br />
<br />
The reasons for this are usually:<br />
* the jail is no longer running<br />
<br />
* the jail's hostname has changed<br />
In this case, <br />
<br />
>=6.x: run a <tt>jls|grep <jail's IP></tt> to find the correct hostname, then update the quad file, then kill the jail.<br />
<br />
<6.x: the first step is to cat their /etc/rc.conf file to see if you can tell what they set the new hostname to. This very often works. For example:<br />
<br />
cat /mnt/data2/198.78.65.136-col00261-DIR/etc/rc.conf<br />
<br />
But maybe they set the hostname with the hostname command, and the original hostname is still in /etc/rc.conf.<br />
<br />
The welcome email clearly states that they should tell us if they change their hostname, so there is no problem in just emailing them and asking them what they set the new hostname to.<br />
<br />
Once you know the new hostname OR if a customer simply emails to inform you that they have set the hostname to something different, you need to edit the quad and safe files that their system is in to input the new hostname.<br />
<br />
However, if push comes to shove and you cannot find out the hostname from them or from their system, then you need to start doing some detective work.<br />
<br />
The easiest thing to do is run jailps looking for a hostname similar to their original hostname. Or you could get into the /bin/sh shell by running:<br />
<br />
/bin/sh<br />
<br />
and then looking at every hostname of every process:<br />
<br />
for f in `ls /proc` ; do cat /proc/$f/status ; done<br />
<br />
and scanning for a hostname that is either similar to their original hostname, or that you don't see in any of the quad safe files.<br />
<br />
This is very brute force though, and it is possible that catting every file in /proc is dangerous - I don't recommend it. A better thing would be to identify any processes that you know belong to this system – perhaps the reason you are trying to find this system is because they are running something bad - and just catting the status from only that PID.<br />
<br />
Somewhere there’s a jail where there may be 2 systems named www. Look at /etc/rc.conf and make sure they’re both really www. If they are, jailkill www, jailps www to make sure not running. Then immediately restart the other one, as the fqdn (as found from a rev nslookup)<br />
<br />
* on >=6.x the hostname may not yet be hashed:<br />
<pre>jail9 /# jls<br />
JID Hostname Path IP Address(es)<br />
1 bitnet.dgate.org /mnt/data1/69.55.232.50-col02094-DIR 69.55.232.50<br />
2 ns3.hctc.net /mnt/data1/69.55.234.52-col01925-DIR 69.55.234.52<br />
3 bsd1 /mnt/data1/69.55.232.44-col00155-DIR 69.55.232.44<br />
4 let2.bbag.org /mnt/data1/69.55.230.92-col00202-DIR 69.55.230.92<br />
5 post.org /mnt/data2/69.55.232.51-col02095-DIR 69.55.232.51 ...<br />
6 ns2 /mnt/data1/69.55.232.47-col01506-DIR 69.55.232.47 ...<br />
7 arlen.server.net /mnt/data1/69.55.232.52-col01171-DIR 69.55.232.52<br />
8 deskfood.com /mnt/data1/69.55.232.71-col00419-DIR 69.55.232.71<br />
9 mirage.confluentforms.com /mnt/data1/69.55.232.54-col02105-DIR 69.55.232.54 ...<br />
10 beachmember.com /mnt/data1/69.55.232.59-col02107-DIR 69.55.232.59<br />
11 www.agottem.com /mnt/data1/69.55.232.60-col02109-DIR 69.55.232.60<br />
12 sdhobbit.myglance.org /mnt/data1/69.55.236.82-col01708-DIR 69.55.236.82<br />
13 ns1.jnielsen.net /mnt/data1/69.55.234.48-col00204-DIR 69.55.234.48 ...<br />
14 ymt.rollingegg.net /mnt/data2/69.55.236.71-col01678-DIR 69.55.236.71<br />
15 verse.unixlore.net /mnt/data1/69.55.232.58-col02131-DIR 69.55.232.58<br />
16 smcc-mail.org /mnt/data2/69.55.232.68-col02144-DIR 69.55.232.68<br />
17 kasoutsuki.w4jdh.net /mnt/data2/69.55.232.46-col02147-DIR 69.55.232.46<br />
18 dili.thium.net /mnt/data2/69.55.232.80-col01901-DIR 69.55.232.80<br />
20 www.tekmarsis.com /mnt/data2/69.55.232.66-col02155-DIR 69.55.232.66<br />
21 vps.yoxel.net /mnt/data2/69.55.236.67-col01673-DIR 69.55.236.67<br />
22 smitty.twitalertz.com /mnt/data2/69.55.232.84-col02153-DIR 69.55.232.84<br />
23 deliver4.klatha.com /mnt/data2/69.55.232.67-col02160-DIR 69.55.232.67<br />
24 nideffer.com /mnt/data2/69.55.232.65-col00412-DIR 69.55.232.65<br />
25 usa.hanyuan.com /mnt/data2/69.55.232.57-col02163-DIR 69.55.232.57<br />
26 daifuku.ppbh.com /mnt/data2/69.55.236.91-col01720-DIR 69.55.236.91<br />
27 collins.greencape.net /mnt/data2/69.55.232.83-col01294-DIR 69.55.232.83<br />
28 ragebox.com /mnt/data2/69.55.230.104-col01278-DIR 69.55.230.104<br />
29 outside.mt.net /mnt/data2/69.55.232.72-col02166-DIR 69.55.232.72<br />
30 vps.payneful.ca /mnt/data2/69.55.234.98-col01999-DIR 69.55.234.98<br />
31 higgins /mnt/data2/69.55.232.87-col02165-DIR 69.55.232.87 ...<br />
32 ozymandius /mnt/data2/69.55.228.96-col01233-DIR 69.55.228.96<br />
33 trusted.realtors.org /mnt/data2/69.55.238.72-col02170-DIR 69.55.238.72<br />
34 jc1.flanderous.com /mnt/data2/69.55.239.22-col01504-DIR 69.55.239.22<br />
36 guppylog.com /mnt/data2/69.55.238.73-col00036-DIR 69.55.238.73<br />
40 haliohost.com /mnt/data2/69.55.234.41-col01916-DIR 69.55.234.41 ...<br />
41 satyr.jorge.cc /mnt/data1/69.55.232.70-col01963-DIR 69.55.232.70<br />
jail9 /# jailkill satyr.jorge.cc<br />
ERROR: jail_: jail "satyr,jorge,cc" not found<br />
</pre><br />
<br />
Note how it's saying <tt>satyr,jorge,cc</tt> is not found, and not <tt>satyr.jorge.cc</tt>. <br />
<br />
The jail subsystem tracks things using comma-delimited hostnames. That is created every few hours:<br />
<br />
jail9 /# crontab -l<br />
0 0,6,12,18 * * * /usr/local/jail/bin/sync_jail_names<br />
<br />
So if we run this manually:<br />
jail9 /# /usr/local/jail/bin/sync_jail_names<br />
<br />
Then kill the jail:<br />
jail9 /# jailkill satyr.jorge.cc<br />
successfully killed: satyr,jorge,cc</pre><br />
<br />
It worked.<br />
<br />
<br />
If you ever see this when trying to kill a jail:<br />
<br />
<pre># jailkill e-scribe.com<br />
killing JID: 6 hostname: e-scribe.com<br />
3 procs running<br />
3 procs running<br />
3 procs running<br />
3 procs running<br />
...</pre><br />
<br />
<tt>[[#jailkill|jailkill]]</tt> probably got lost trying to kill off the jail. Just ctrl-c the jailkill process, then run a jailps on the hostname, and <tt>kill -9</tt> any process which is still running. Keep running jailps and <tt>kill -9</tt> till all processes are gone.<br />
<br />
== jailpsall ==<br />
jailpsall<br />
will run a jailps on all jails configured in the quad files (this is different from<br />
jailps with no arguments as it won’t help you find a “hidden” system)<br />
<br />
== jailpsw ==<br />
jailpsw<br />
will run a jailps with an extra -w to provide wider output<br />
<br />
== jt (>=7.x) ==<br />
jt<br />
displays the top 20 processes on the server (the top 20 processes from top) and <br />
which jail owns them. This is very helpful for determining who is doing what when<br />
the server is very busy.<br />
<br />
== jtop (>=7.x) ==<br />
jtop<br />
a wrapper for top displaying processes on the server and which jail owns them. Constantly updates, like top. <br />
<br />
== jtop (<7.x) ==<br />
jtop<br />
displays the top 20 processes on the server (the top 20 processes from top) and <br />
which jail owns them. This is very helpful for determining who is doing what when<br />
the server is very busy.<br />
<br />
== stopjail ==<br />
stopjail <hostname> [1]<br />
this will jailkill, umount and vnconfig –u a jail. If passed an optional 2nd<br />
argument, it will not exit before umounting and un-vnconfig’ing in the event<br />
jailkill returns no processes killed. This is useful if you just want to umount<br />
and vnconfig –u a jail you’ve already killed. It is intelligent in that it won’t <br />
try to umount or vnconfig –u if it’s not necessary.<br />
<br />
== startjail ==<br />
startjail <hostname><br />
this will start vnconfig, mount (including linprocfs and null-mounts), and start a jail.<br />
Essentially, it reads the jail’s relevant block from the right quad file and executes it.<br />
It is intelligent in that it won’t try to mount or vnconfig if it’s not necessary.<br />
<br />
== jpid ==<br />
jpid <pid><br />
displays information about a process – including which jail owns it.<br />
It’s the equivalent of running cat /proc/<pid>/status<br />
<br />
== canceljail ==<br />
canceljail <hostname> [1]<br />
this will stop a jail (the equivalent of stopjail), check for backups (offer to remove them <br />
from the backup server and the backup.config), rename the vnfile, remove the dir, and <br />
edit quad/safe. If passed an optional 2nd argument, it will not exit upon failing to kill<br />
and processes owned by the jail. This is useful if you just want to cancel a jail which <br />
is already stopped.<br />
<br />
== jls ==<br />
jls [-v]<br />
Lists all jails running:<br />
<pre>JID #REF IP Address Hostname Path<br />
101 135 69.55.224.148 mail.pc9.org /mnt/data2/69.55.224.148-col01034-DIR</pre><br />
<br />
#REF is the number of references or procs(?) running<br />
<br />
Running with -v will give you all IPs assigned to each jail (7.2 up)<br />
<pre>JID #REF Hostname Path IP Address(es)<br />
101 139 mail.pc9.org /mnt/data2/69.55.224.148-col01034-DIR 69.55.224.14869.55.234.85</pre><br />
<br />
== startalljails ==<br />
startalljails<br />
7.2+ only. This will parse through quad1 and start all jails. It utilizes lockfiles so it won’t try to start a jail more than once- therefore multiple instances can be running in parallel without fear of starting a jail twice. If a jail startup gets stuck, you can ^C without fear of killing the script. IMPORTANT- before running startalljails you should make sure you ran preboot once as it will clear out all the lockfiles and enable startalljails to work properly.<br />
<br />
== aaccheck.sh ==<br />
aaccheck.sh<br />
displayes the output of container list and task list from aaccli<br />
<br />
== backup ==<br />
backup<br />
backup script called nightly to update jail scripts and do customer backups<br />
<br />
== buildsafe ==<br />
buildsafe<br />
creates safe files based on quads (automatically removing the fsck’s). This will destructively overwrite safe files<br />
<br />
== checkload.pl ==<br />
checkload.pl<br />
this was intended to be setup as a cronjob to watch processes on a jail when the load <br />
rises above a certain level. Not currently in use.<br />
<br />
== checkprio.pl ==<br />
checkprio.pl<br />
will look for any process (other than the current shell’s csh, sh, sshd procs) with a non-normal priority and normalize it<br />
<br />
== diskusagemon == <br />
diskusagemon <mount point> <1k blocks><br />
watches a mount point’s disk use, when it reaches the level specified in the 2nd argument,<br />
it exits. This is useful when doing a restore and you want to be paged as it’s nearing completion.<br />
Best used as: <tt>diskusagemon /asd/asd 1234; pagexxx</tt><br />
<br />
== dumprestore ==<br />
dumprestore <dumpfile><br />
this is a perl expect script which automatically enters ‘1’ and ‘y’. It seems to cause restore to fail<br />
to set owner permissions on large restores.<br />
<br />
== g ==<br />
g <search><br />
greps the quad/safe files for the given search parameter<br />
<br />
== gather.pl ==<br />
gather.pl<br />
gathers up data about jails configured and writes to a file. Used for audits against the db<br />
<br />
== gb ==<br />
gb <search><br />
greps backup.config for the given search parameter<br />
<br />
== gbg ==<br />
gbg <search><br />
greps backup.config for the given search parameter and presents just the directories (for clean pasting)<br />
<br />
== ipfwbackup ==<br />
ipfwbackup<br />
writes ipfw traffic count data to a logfile<br />
<br />
== ipfwreset ==<br />
ipfwreset<br />
writes ipfw traffic count data to a logfile and resets counters to 0<br />
<br />
== js ==<br />
js<br />
output varies by OS version, but generally provides information about the base jail:<br />
- which vn’s are in use<br />
- disk usage<br />
- info about the contents of quads<br />
- the # of inodes represented by the jails contained in the group (133.2 in the example below), and how many jails per data mount, as well as subtotals<br />
- ips bound to the base machine but not in use by a jail<br />
- free gvinum volumes, or unused vn’s or used md’s<br />
<br />
<pre>/usr/local/jail/rc.d/quad1:<br />
/mnt/data1 133.2 (1)<br />
/mnt/data2 1040.5 (7)<br />
total 1173.7 (8)<br />
/usr/local/jail/rc.d/quad2:<br />
/mnt/data1 983.4 (6)<br />
total 983.4 (6)<br />
/usr/local/jail/rc.d/quad3:<br />
/mnt/data1 693.4 (4)<br />
/mnt/data2 371.6 (3)<br />
total 1065 (7)<br />
/usr/local/jail/rc.d/quad4:<br />
/mnt/data1 466.6 (3)<br />
/mnt/data2 882.2 (5)<br />
total 1348.8 (8)<br />
/mnt/data1: 2276.6 (14)<br />
/mnt/data2: 2294.3 (15)<br />
<br />
Available IPs:<br />
69.55.230.11 69.55.230.13 69.55.228.200<br />
<br />
Available volumes:<br />
v78 /mnt/data2 2G<br />
v79 /mnt/data2 2G<br />
v80 /mnt/data2 2G</pre> <br />
<br />
== load.pl ==<br />
load.pl<br />
feeds info to load mrtg - executed by inetd.<br />
<br />
== makevirginjail ==<br />
makevirginjail<br />
Only on some systems, makes an empty jail (doesn't do restore step)<br />
<br />
== mb == <br />
mb <mount|umount><br />
(nfs) mounts and umounts dirs to backup2. Shortcuts are mbm and mbu to mount and unmount. <br />
<br />
== notify.sh ==<br />
notify.sh<br />
emails reboot@johncompanies.com – intended to be called at boot time to alert us to a machine which panics and reboots and isn’t caught by bb or castle.<br />
<br />
== orphanedbackupwatch ==<br />
orphanedbackupwatch<br />
looks for directories on backup2 which aren’t configured in backup.config and offers to delete them<br />
<br />
== postboot ==<br />
postboot<br />
to be run after a machine reboot and quad/safe’s are done executing. It will:<br />
* do chmod 666 on each jail’s /dev/null<br />
* add ipfw counts<br />
* run jailpsall (so you can see if a configured jail isn’t running)<br />
<br />
== preboot ==<br />
preboot<br />
to be run before running quad/safe – checks for misconfigurations: <br />
* a jail configured in a quad but not a safe<br />
* a jail is listed more than once in a quad<br />
* the ip assigned to a jail isn’t configured on the machine<br />
* alias numbering skips in the rc.conf (resulting in the above)<br />
* orphaned vnfile's that aren't mentioned in a quad/safe<br />
* ip mismatches between dir/vnfile name and the jail’s ip<br />
* dir/vnfiles's in quad/safe that don’t exist <br />
<br />
== quadanalyze.pl ==<br />
quadanalyze.pl<br />
called by js, produces the info (seen above with js explanation) about the contents of quad (inode count, # of jails, etc.)<br />
<br />
== rsync.backup ==<br />
rsync.backup<br />
does customer backups (relies on backup.config)<br />
<br />
== taskdone ==<br />
taskdone<br />
when called will email support@johncompanies.com with the hostname of the machine from which it was executed as the subject<br />
<br />
== topten ==<br />
topten<br />
summarizes the top 10 traffic users (called by ipfwreset)<br />
<br />
== trafficgather.pl ==<br />
trafficgather.pl [yy-mm]<br />
sends a traffic usage summary by jail to support@johncomapnies.com and payments@johncompanies.com. Optional arguments are year and month (must be in the past). If not passed, assumes last month. Relies on traffic logs created by ipfwreset and ipfwbackup<br />
<br />
== trafficwatch.pl ==<br />
trafficwatch.pl<br />
checks traffic usage and emails support@johncomapnies.com when a jail reaches the warning level (35G) and the limit (40G). We really aren’t using this anymore now that we have netflow.<br />
<br />
== trafstats ==<br />
trafstats<br />
writes ipfw traffic usage info by jail to a file called jc_traffic_dump in each jail’s / dir<br />
<br />
== truncate_jailmake ==<br />
truncate_jailmake<br />
a version of jailmake which creates truncated vnfiles.<br />
<br />
== vb ==<br />
vb<br />
the equivalent of: <tt>vi /usr/local/jail/bin/backup.config</tt><br />
<br />
== vs (freebsd) ==<br />
vs<n><br />
the equivalent of: <tt>vi /usr/local/jail/rc.d/safe<n></tt><br />
<br />
vq<n><br />
the equivalent of: vi /usr/local/jail/rc.d/quad<n><br />
<br />
== dumpremote ==<br />
dumpremote <user@machine> </remote/location/file-dump> <vnX><br />
ex: dumpremote user@10.1.4.117 /mnt/data3/remote.echoditto.com-dump 7<br />
this will dump a vn filesystem to a remote machine and location<br />
<br />
== oversellcheck ==<br />
oversellcheck<br />
displays how much a disk is oversold or undersold taking into account truncated vn files. Only for use on 4.x systems<br />
<br />
== mvbackups (freebsd) ==<br />
mvbackups <dir> (1.1.1.1-col00001-DIR) <target_machine> (jail1) <target_dir> (data1)<br />
moves backups from one location to another on the backup server, and provides you with option to remove entries from current backup.config, and simple paste command to add the config to backup.config on the target server<br />
<br />
== jailnice ==<br />
jailnice <hostname><br />
applies <tt>renice 19 [PID]</tt> and <tt>rtprio 31 –[PID]</tt> to each process in the given jail<br />
<br />
== dumpremoterestore ==<br />
dumpremoterestore <device> <ip of target machine> <dir on target machine><br />
ex: dumpremoterestore /dev/vn51 10.1.4.118 /mnt/data2/69.55.239.45-col00688-DIR<br />
dumps a device and restores it to a directory on a remote machine. Requires that you enable root ssh on the <br />
remote machine.<br />
<br />
== psj ==<br />
psj<br />
shows just the procs running on the base system – a ps auxw but without jail’d procs present<br />
<br />
== perc5iraidchk ==<br />
perc5iraidchk<br />
checks for degraded arrays on Dell 2950 systems with Perc5/6 controllers<br />
<br />
== perc4eraidchk ==<br />
perc4eraidchk<br />
checks for degraded arrays on Dell 2850 systems with Perc4e/Di controllers<br />
<br />
== Making new customer VE (vemakexxx) ==<br />
<br />
This applies to older virts with old templates. This should probably not be used at all anymore.<br />
<br />
<br />
1. look thru hist for ip<br />
<br />
2. confirm ip you want to use isn’t in use via Mgmt. -> IP Map in management screens<br />
<br />
3. put ve on whichever partition has more space<br />
vemakerh9 <veid> <ip> <hostname> <mount> <email> [gb disk]; <256|384|512> <veid><br />
vemakerh9 866 69.55.226.109 ngentu.com /vz1 ayo@ngantu.com,asd@asd.com 5; 256 866<br />
<br />
4. copy (veid), dir, and ip to pending customer screen (pass set to p455agfa)<br />
<br />
= Virtuozzo VPS =<br />
<br />
Note: We use VEID (Virtual Environment ID) and CTID (Container ID) interchangably. Similarly, VE and CT. They mean the same thing.<br />
VZPP = VirtuoZzo Power Panel (the control panel for each CT)<br />
<br />
All linux systems exist in /vz, /vz1 or /vz2 - since each linux machine holds roughly 60-90 customers, there will be roughly 30-45 in each partition.<br />
<br />
The actual filesystem of the system in question is in:<br />
<br />
/vz(1-2)/private/(VEID)<br />
<br />
Where VEID is the identifier for that system - an all-numeric string larger than 100.<br />
<br />
The actual mounted and running systems are in the corresponding:<br />
<br />
/vz(1-2)/root/(VEID)<br />
<br />
But we rarely interact with any system from this mount point.<br />
<br />
You should never need to touch the root portion of their system – however you can traverse their filesystem by going to <tt>/vz(1-2)/private/(VEID)/root</tt> (<tt>/vz(1-2)/private/(VEID)/fs/root</tt> on 4.x systems) the root of their filesystem is in that directory, and their entire system is underneath that.<br />
<br />
Every VE has a startup script in <tt>/etc/sysconfig/vz-scripts</tt> (which is symlinked as <tt>/vzconf</tt> on all systems) - the VE startup script is simply named <tt>(VEID).conf</tt> - it contains all the system parameters for that VE:<br />
<br />
<pre># Configuration file generated by vzsplit for 60 VE<br />
# on HN with total amount of physical mem 2011 Mb<br />
<br />
VERSION="2"<br />
CLASSID="2"<br />
<br />
ONBOOT="yes"<br />
<br />
KMEMSIZE="8100000:8200000"<br />
LOCKEDPAGES="322:322"<br />
PRIVVMPAGES="610000:615000"<br />
SHMPAGES="33000:34500"<br />
NUMPROC="410:415"<br />
PHYSPAGES="0:2147483647"<br />
VMGUARPAGES="13019:2147483647"<br />
OOMGUARPAGES="13019:2147483647"<br />
NUMTCPSOCK="1210:1215"<br />
NUMFLOCK="107:117"<br />
NUMPTY="19:19"<br />
NUMSIGINFO="274:274"<br />
TCPSNDBUF="1800000:1900000"<br />
TCPRCVBUF="1800000:1900000"<br />
OTHERSOCKBUF="900000:950000"<br />
DGRAMRCVBUF="200000:200000"<br />
NUMOTHERSOCK="650:660"<br />
DCACHE="786432:818029"<br />
NUMFILE="7500:7600"<br />
AVNUMPROC="51:51"<br />
IPTENTRIES="155:155"<br />
DISKSPACE="4194304:4613734"<br />
DISKINODES="400000:420000"<br />
CPUUNITS="1412"<br />
QUOTAUGIDLIMIT="2000"<br />
VE_ROOT="/vz1/root/636"<br />
VE_PRIVATE="/vz1/private/636"<br />
NAMESERVER="69.55.225.225 69.55.230.3"<br />
OSTEMPLATE="vzredhat-7.3/20030305"<br />
VE_TYPE="regular"<br />
IP_ADDRESS="69.55.225.229"<br />
HOSTNAME="textengine.net"</pre><br />
<br />
<br />
As you can see, the hostname is set here, the disk space is set here, the number of inodes, the number of files that can be open, the number of tcp sockets, etc. - all are set here.<br />
<br />
In fact, everything that can be set on this customer system is set in this conf file.<br />
<br />
<br />
All interaction with the customer system is done with the VEID. You start the system by running:<br />
<br />
vzctl start 999<br />
<br />
You stop it by running:<br />
<br />
vzctl stop 999<br />
<br />
You execute commands in it by running:<br />
<br />
vzctl exec 999 df -k<br />
<br />
You enter into it, via a root-shell backdoor with:<br />
<br />
vzctl enter 999<br />
<br />
and you set parameters for the system, while it is still running, with:<br />
<br />
vzctl set 999 --diskspace 6100000:6200000 --save<br />
<br />
<tt>vzctl</tt> is the most commonly used command - we have aliased <tt>v</tt> to <tt>vzctl</tt> since we use it so often. We’ll continue to use <tt>vzctl</tt> in our examples, but feel free to use just <tt>v</tt>.<br />
<br />
Let's say the user wants more diskspace. You can cat their conf file and see:<br />
<br />
DISKSPACE="4194304:4613734"<br />
<br />
So right now they have 4gigs of space. You can then change it to 6 with:<br />
<br />
vzctl set 999 --diskspace 6100000:6200000 --save<br />
<br />
IMPORTANT: all issuances of the vzctl set command need to end with <tt>–save</tt> - if they don't, the setting will be set, but it will not be saved to the conf file, and they will not have those settings next time they boot.<br />
<br />
All of the tunables in the conf file can be set with the vzctl set command. Note that in the conf file, and on the vzctl set command line, we always issue two numbers seperated by a colon - that is because we are setting the hard and soft limits. Always set the hard limit slightly above the soft limit, as you see it is in the conf file for all those settings.<br />
<br />
There are also things you can set with `<tt>vzctl set</tt>` that are not in the conf file as settings, per se. For instance, you can add IPs:<br />
<br />
vzctl set 999 --ipadd 10.10.10.10 --save<br />
<br />
or multiple IPs:<br />
<br />
vzctl set 999 --ipadd 10.10.10.10 --ipadd 10.10.20.30 --save<br />
<br />
or change the hostname:<br />
<br />
vzctl set 999 --hostname www.example.com --save<br />
<br />
You can even set the nameservers:<br />
<br />
vzctl set 999 --nameserver 198.78.66.4 --nameserver 198.78.70.180 --save<br />
<br />
Although you probably will never do that.<br />
<br />
You can disable a VPS from being started (by VZPP or reboot) (4.x):<br />
<br />
vzctl set 999 --disabled yes --save <br />
<br />
You can disable a VPS from being started (by VZPP or reboot) (<=3.x):<br />
<br />
vzctl set 999 --onboot=no --save <br />
<br />
You can disable a VPS from using his control panel:<br />
<br />
vzctl set 999 --offline_management=no --save <br />
<br />
You can suspend a VPS, so it can be resumed in the same state it was in when it was stopped (4.x):<br />
<br />
vzctl suspend 999<br />
<br />
and to resume it:<br />
<br />
vzctl resume 999<br />
<br />
to see who owns process:<br />
vzpid <PID><br />
<br />
to mount up an unmounted ve:<br />
vzctl mount 827<br />
<br />
<br />
To see network stats for CT's:<br />
<pre># vznetstat<br />
VEID Net.Class Output(bytes) Input(bytes)<br />
-----------------------------------------------<br />
24218 1 484M 39M<br />
24245 1 463M 143M<br />
2451 1 2224M 265M<br />
2454 1 2616M 385M<br />
4149 1 125M 68M<br />
418 1 1560M 34M<br />
472 1 1219M 315M<br />
726 1 628M 317M<br />
763 1 223M 82M<br />
771 1 4234M 437M<br />
-----------------------------------------------<br />
Total: 13780M 2090M</pre><br />
<br />
<br />
One thing that sometimes comes up on older systems that we created with smaller defaults is that the system would run out of inodes. The user will email and say they cannot create any more files or grow any files larger, but they will also say that they are not out of diskspace ... they are running:<br />
<br />
df -k<br />
<br />
and seeing how much space is free - and they are not out of space. They are most likely out of inodes - which they would see by running:<br />
<br />
df -i<br />
<br />
So, the first thing you should do is enter their system with:<br />
<br />
vzctl enter 999<br />
<br />
and run: <tt>df -i</tt><br />
<br />
to confirm your theory. Then exit their system. Then simply cat their conf file and see what their inodes are set to (probably 200000:200000, since that was the old default on the older systems) and run:<br />
<br />
vzctl set 999 --diskinodes 400000:400000 --save<br />
<br />
If they are not out of inodes, then a good possibility is that they have maxed out their numfile configuration variable, which controls how many files they can have in their system. The current default is 7500 (which nobody has ever hit), but the old default was as low as 2000, so you would run something like:<br />
<br />
vzctl set 999 --numfile 7500:7500 --save<br />
<br />
----<br />
<br />
You cannot start or stop a VE if your pwd is its private (/vz/private/999) or root (/vz/root/999) directories, or anywhere below them.<br />
<br />
<br />
== Recovering from a crash (linux) ==<br />
<br />
=== Diagnose whether you have a crash ===<br />
The most important thing is to get the machine and all ve’s back up as soon as possible. Note the time, you’ll need to create a crash log entry (Mgmt. -> Reference -> CrashLog). The first thing to do is head over to the [[Screen#Screen_Organization|serial console screen]] and see if there’s any kernel error messages output. Try to copy any messages (or just a sample of repeating messages) you see into the notes section of the crash log – these will also likely need to be sent to virtuozzo for interpretation. If the messages are spewing too fast, hit ^O + H to start a screen log dump which you can ob1182.pts-38.bb serve after the machine is rebooted. Additionally, if the machine is responsive, you can get a trace to send to virtuozzo by hooking up a kvm and entering these 3 sequences:<br />
<pre>alt+print screen+m<br />
alt+print screen+p<br />
alt+print screen+t</pre><br />
<br />
If there are no messages, the machine may just be really busy- wait a bit (5-10min) to see if it comes back. If it's still pinging, odds are its very busy. If it doesn't come back, or the messages indicate a fatal error, you will need to proceed with a power cycle (ctrl+alt+del will not work).<br />
<br />
=== Power cycle the server ===<br />
If this machine is not a Dell 2950 with a [[DRAC/RMM#DRAC|DRAC card]] (i.e. if you can’t ssh into the DRAC card and issue racadm serveraction hardreset, then you will need someone at the data center to power the macine off, wait 30 sec, then turn it back on. Make sure to re-attach via console (<tt>tip virtxx</tt>) immediately after power down. <br />
<br />
=== (Re)attach to the console ===<br />
Stay on the console the entire time during boot. As the BIOS posts- look out for the RAID card output- does everything look healthy? The output may be scrambled, look for "DEGRADED" or "FAILED". Once the OS starts booting you will be disconnected (dropped back to the shell on the console server) a couple times during the boot up. The reason you want to quickly re-attach is two-fold: 1. If you don’t reattach quickly then you won’t get any console output, 2. you want to be attached before the server ''potentially'' starts (an extensive) fsck. If you attach after the fsck begins, you’ll have seen no indication it started an fsck and the server will appear frozen during startup- no output, no response. <br />
<br />
=== Start containers/VE's/VPSs ===<br />
When the machine begins to start VE’s, it’s safe to leave the console and login via ssh. All virts should be set to auto start all the VEs after a crash. Further, most (newer) virts are set to “fastboot” it’s VE’s (to find out, do:<br />
grep -i fast /etc/sysconfig/vz <br />
and look for <tt>VZFASTBOOT=yes</tt>). If this was set prior to the machine’s crash (setting it after the machine boots will not have any effect until the vz service is restarted) it will start each ve as fast as possible, in serial, then go thru each VE (serially), shutting it down running a vzquota (disk usage) check, then bringing it back up. The benefit is that all VE’s are brought up quickly (within 15min or so depending on the #), the downside is a customer watching closely will notice 2 outages – 1st the machine crash, 2nd their quota check (which will be a much shorter downtime- on the order of a few minutes). <br />
<br />
Where “fastboot” is not set to yes (i.e on quar1), vz will start them consecutively, checking the quotas one at a time, and the 60th VE may not start until an hour or two later - this is not acceptable.<br />
<br />
The good news is, if you run vzctl start for a VE that is already started, you will simply get an error: <tt>VE is already started</tt>. Further, if you attempt to vzctl start a VE that is in the process of being started, you will simply get an error: unable to lock VE. So, there is no danger in simply running scripts to start smaller sets of VEs. If the system is not autostarting, then there is no issue, and even if it does, when it conflicts, one process (yours or the autostart) will lose, and just move on to the next one.<br />
<br />
A script has been written to assist with ve starts: [[#startvirt.pl|startvirt.pl]] which will start 6 ve’s at once until there are no more left. If startvirt.pl is used on a system where “fastboot” was on, it will circumvent the fastboot for ve’s started by startvirt.pl – they will go through the complete quota check before starting- therefore this is not advisable when a system has crashed. When a system is booted cleanly, and there's no need for vzquota checks, then startvirt.pl is safe and advisable to run.<br />
<br />
=== Make sure all containers are running ===<br />
You can quickly get a feel for how many ve’s are started by running:<br />
<br />
<pre>[root@virt4 log]# vs<br />
VEID 16066 exist mounted running<br />
VEID 16067 exist mounted running<br />
VEID 4102 exist mounted running<br />
VEID 4112 exist mounted running<br />
VEID 4116 exist mounted running<br />
VEID 4122 exist mounted running<br />
VEID 4123 exist mounted running<br />
VEID 4124 exist mounted running<br />
VEID 4132 exist mounted running<br />
VEID 4148 exist mounted running<br />
VEID 4151 exist mounted running<br />
VEID 4155 exist mounted running<br />
VEID 42 exist mounted running<br />
VEID 432 exist mounted running<br />
VEID 434 exist mounted running<br />
VEID 442 exist mounted running<br />
VEID 450 exist mounted running<br />
VEID 452 exist mounted running<br />
VEID 453 exist mounted running<br />
VEID 454 exist mounted running<br />
VEID 462 exist mounted running<br />
VEID 463 exist mounted running<br />
VEID 464 exist mounted running<br />
VEID 465 exist mounted running<br />
VEID 477 exist mounted running<br />
VEID 484 exist mounted running<br />
VEID 486 exist mounted running<br />
VEID 490 exist mounted running</pre><br />
<br />
So to see how many ve’s have started:<br />
<pre>[root@virt11 root]# vs | grep running | wc -l<br />
39</pre><br />
<br />
And to see how many haven’t:<br />
<pre>[root@virt11 root]# vs | grep down | wc -l<br />
0</pre><br />
<br />
And how many we should have running:<br />
<pre>[root@virt11 root]# vs | wc -l<br />
39</pre><br />
<br />
Another tool you can use to see which ve’s have started, among other things is [[#vzstat|vzstat]]. It will give you CPU, memory, and other stats on each ve and the overall system. It’s a good thing to watch as ve’s are starting (note the VENum parameter, it will tell you how many have started):<br />
<br />
<pre>4:37pm, up 3 days, 5:31, 1 user, load average: 1.57, 1.68, 1.79<br />
VENum 40, procs 1705: running 2, sleeping 1694, unint 0, zombie 9, stopped 0<br />
CPU [ OK ]: VEs 57%, VE0 0%, user 8%, sys 7%, idle 85%, lat(ms) 412/2<br />
Mem [ OK ]: total 6057MB, free 9MB/54MB (low/high), lat(ms) 0/0<br />
Swap [ OK ]: tot 6142MB, free 4953MB, in 0.000MB/s, out 0.000MB/s<br />
Net [ OK ]: tot: in 0.043MB/s 402pkt/s, out 0.382MB/s 4116pkt/s<br />
Disks [ OK ]: in 0.002MB/s, out 0.000MB/s<br />
<br />
VEID ST %VM %KM PROC CPU SOCK FCNT MLAT IP<br />
1 OK 1.0/17 0.0/0.4 0/32/256 0.0/0.5 39/1256 0 9 69.55.227.152<br />
21 OK 1.3/39 0.1/0.2 0/46/410 0.2/2.8 23/1860 0 6 69.55.239.60<br />
133 OK 3.1/39 0.1/0.3 1/34/410 6.3/2.8 98/1860 0 0 69.55.227.147<br />
263 OK 2.3/39 0.1/0.2 0/56/410 0.3/2.8 34/1860 0 1 69.55.237.74<br />
456 OK 17/39 0.1/0.2 0/100/410 0.1/2.8 48/1860 0 11 69.55.236.65<br />
476 OK 0.6/39 0.0/0.2 0/33/410 0.1/2.8 96/1860 0 10 69.55.227.151<br />
524 OK 1.8/39 0.1/0.2 0/33/410 0.0/2.8 28/1860 0 0 69.55.227.153<br />
594 OK 3.1/39 0.1/0.2 0/45/410 0.0/2.8 87/1860 0 1 69.55.239.40<br />
670 OK 7.7/39 0.2/0.3 0/98/410 0.0/2.8 64/1860 0 216 69.55.225.136<br />
691 OK 2.0/39 0.1/0.2 0/31/410 0.0/0.7 25/1860 0 1 69.55.234.96<br />
744 OK 0.1/17 0.0/0.5 0/10/410 0.0/0.7 7/1860 0 6 69.55.224.253<br />
755 OK 1.1/39 0.0/0.2 0/27/410 0.0/2.8 33/1860 0 0 192.168.1.4<br />
835 OK 1.1/39 0.0/0.2 0/19/410 0.0/2.8 5/1860 0 0 69.55.227.134<br />
856 OK 0.3/39 0.0/0.2 0/13/410 0.0/2.8 16/1860 0 0 69.55.227.137<br />
936 OK 3.2/52 0.2/0.4 0/75/410 0.2/0.7 69/1910 0 8 69.55.224.181<br />
1020 OK 3.9/39 0.1/0.2 0/60/410 0.1/0.7 55/1860 0 8 69.55.227.52<br />
1027 OK 0.3/39 0.0/0.2 0/14/410 0.0/2.8 17/1860 0 0 69.55.227.83<br />
1029 OK 1.9/39 0.1/0.2 0/48/410 0.2/2.8 25/1860 0 5 69.55.227.85<br />
1032 OK 12/39 0.1/0.4 0/80/410 0.0/2.8 41/1860 0 8 69.55.227.90</pre><br />
<br />
When you are all done, you will want to make sure that all the VEs really did get started, run vs one more time.<br />
<br />
Note the time all ve’s are back up and enter that into and save the crash log entry.<br />
<br />
Occasionally, a ve will not start automatically. The most common reason for a ve not to come up normally is the ve was at it’s disk limit before the crash, and will not start since they’re over the limit. To overcome this, set the disk space to current usage level (the system will give this to you when it fails to start), start the ve, then re-set the disk space back to the prior level. Lastly, contact the customer to let them know they’re out of disk (or allocate more disk if they're entitled to more).<br />
<br />
== Hitting performance barriers and fixing them ==<br />
<br />
There are multiple modes virtuozzo offers to allocate resources to a ve. We utilize 2: SLM and UBC parameters<br />
On our 4.x systems, we use all SLM – it’s simpler to manage and understand. There are a few systems on virt19/18 that may also use SLM. Everything else uses UBC. <br />
You can tell a SLM ve by:<br />
<br />
SLMMODE="all"<br />
<br />
in their conf file. <br />
<br />
TODO: detail SLM modes and parameters.<br />
<br />
If someone is in SLM mode and they hit memory resource limits, they simply need to upgrade to more memory.<br />
<br />
The following applies to everyone else (UBC).<br />
<br />
Customers will often email and say that they are getting out of memory errors - a common one is "cannot fork" ... basically, anytime you see something odd like this, it means they are hitting one of their limits that is in place in their conf file.<br />
<br />
The conf file, however, simply shows their limits - how do we know what they are currently at ?<br />
<br />
The answer is a file called v - this file contains the current status (and peaks) of their performance settings, and also counts how many times they have hit the barrier. The output of the file looks like this:<br />
<br />
<pre>764: kmemsize 384113 898185 8100000 8200000 0<br />
lockedpages 0 0 322 322 0<br />
privvmpages 1292 7108 610000 615000 0<br />
shmpages 270 528 33000 34500 0<br />
dummy 0 0 0 0 0<br />
numproc 8 23 410 415 0<br />
physpages 48 5624 0 2147483647 0<br />
vmguarpages 0 0 13019 2147483647 0<br />
oomguarpages 641 6389 13019 2147483647 0<br />
numtcpsock 3 21 1210 1215 0<br />
numflock 1 3 107 117 0<br />
numpty 0 2 19 19 0<br />
numsiginfo 0 4 274 274 0<br />
tcpsndbuf 0 80928 1800000 1900000 0 <br />
tcprcvbuf 0 108976 1800000 1900000 0<br />
othersockbuf 2224 37568 900000 950000 0<br />
dgramrcvbuf 0 4272 200000 200000 0<br />
numothersock 3 9 650 660 0<br />
dcachesize 53922 100320 786432 818029 0<br />
numfile 161 382 7500 7600 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
numiptent 4 4 155 155 0</pre><br />
<br />
The first column is the name of the counter in question - the same names we saw in the systems conf file. The second column is the _current_ value of that counter, the third column is the max that that counter has ever risen to, the fourth column is the soft limit, and the fifth column is the hard limit (which is the same as the numbers in that systems conf file).<br />
<br />
The sixth number is the failcount - how many times the current usage has risen to hit the barrier. It will increase as soon as the current usage hits the soft limit.<br />
<br />
The problem with /proc/user_beancounters is that it actually contains that set of data for every running VE - so you can't just cat /proc/user_beancounters - it is too long and you get info for every other running system.<br />
<br />
You can vzctl enter the system and run:<br />
<br />
vzctl enter 9999<br />
cat /proc/user_beancounters<br />
<br />
inside their system, and you will just see the stats for their particular system, but entering their system every time you want to see it is combersome.<br />
<br />
So, I wrote a simple script called "vzs" which simply greps for the VEID, and spits out the next 20 or so lines (however many lines there are in the output, I forget) after it. For instance:<br />
<br />
<pre># vzs 765:<br />
765: kmemsize 2007936 2562780 8100000 8200000 0<br />
lockedpages 0 8 322 322 0<br />
privvmpages 26925 71126 610000 615000 0<br />
shmpages 16654 16750 33000 34500 0<br />
dummy 0 0 0 0 0<br />
numproc 41 57 410 415 0<br />
physpages 1794 49160 0 2147483647 0<br />
vmguarpages 0 0 13019 2147483647 0<br />
oomguarpages 4780 51270 13019 2147483647 0<br />
numtcpsock 23 37 1210 1215 0<br />
numflock 17 39 107 117 0<br />
numpty 1 3 19 19 0<br />
numsiginfo 0 6 274 274 0<br />
tcpsndbuf 22240 333600 1800000 1900000 0<br />
tcprcvbuf 0 222656 1800000 1900000 0<br />
othersockbuf 104528 414944 900000 950000 0<br />
dgramrcvbuf 0 4448 200000 200000 0<br />
numothersock 73 105 650 660 0<br />
dcachesize 247038 309111 786432 818029 0<br />
numfile 904 1231 7500 7600 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
numiptent 4 4 155 155 0</pre><br />
<br />
That showed us just the portion of /proc/user_beancounters for system 765.<br />
<br />
When you run the vzs command, always add a : after the VEID.<br />
<br />
So, if a customer complains about some out of memory errors, or no more files, or no more ptys, or just has an unspecific complain about processes dying, etc., the very first thing you need to do is check their beancounters with vzs. Usually you will spot an item that has a high failcount and needs to be upped.<br />
<br />
At that point you could simply up the counter with `vzctl set`. Generally pick a number 10-20% higher than the old one, and make the hard limit slightly larger than the the soft limit. However our systems now come in several levels and those levels have more/different memory allocations. If someone is complaining about something other than a memory limit (pty, numiptent, numflock), it’s generally safe to increase it, at least to the same level as what’s in the /vzconf/4unlimited file on the newest virt. If someone is hitting a memory limit, first make sure they are given what they deserve:<br />
<br />
(refer to mgmt -> payments -> packages)<br />
<br />
To set those levels, you use the [[#setmem|setmem]] command. <br />
<br />
The alternate (DEPRECATED) method would be to use one of 3 commands:<br />
256 <veid><br />
300 <veid><br />
384 <veid><br />
512 <veid><br />
<br />
If the levels were not right (you’d run vzs <veid> before and after to see the effect) tell the customer they’ve been adjusted and be done with it. If the levels were right, tell the customer they must upgrade to a higher package, tell them how to see level (control panel) and that they can reboot their system to escape this lockup contidion.<br />
<br />
Customers can also complain that their site is totally unreachable, or complain that it is down ... if the underlying machine is up, and all seems well, you may notice in the beancounters that network-specific counters are failing - such as numtcpsock, tcpsndbuf or tcprcvbuf. This will keep them from talking on the network and make it seem like their system is down. Again, just up the limits and things should be fine.<br />
<br />
On virts 1-4, you should first look at the default settings for that item on a later virt, such as virt 8 - we have increased the defaults a lot since the early machines. So, if you are going to up a counter on virt2, instead of upping it by 10-20%, instead up it to the new default that you see on virt8.<br />
<br />
== Moving a VE to another virt (migrate/migrateonline) ==<br />
<br />
This will take a while to complete - and it is best to do this at night when the load is light on both machines.<br />
<br />
There are different methods for this, depending on which version of virtuozzo is installed on the src. and dst. virt. <br />
To check which version is running: <br />
[root@virt12 private]# cat /etc/virtuozzo-release<br />
Virtuozzo release 2.6.0<br />
<br />
Ok, let's say that the VE is 1212, and vital stats are:<br />
<pre>[root@virt12 sbin]# vc 1212<br />
VE_ROOT="/vz1/root/1212"<br />
VE_PRIVATE="/vz1/private/1212"<br />
OSTEMPLATE="fedora-core-2/20040903"<br />
IP_ADDRESS="69.55.229.84"<br />
TEMPLATES="devel-fc2/20040903 php-fc2/20040813 mysql-fc2/20040812 postgresql-fc2/20040813 mod_perl-fc2/20040812 mod_ssl-fc2/20040811 jre-fc2/20040823 jdk-fc2/20040823 mailman-fc2/20040823 analog-fc2/20040824 proftpd-fc2/20040818 tomcat-fc2/20040823 usermin-fc2/20040909 webmin-fc2/20040909 uw-imap-fc2/20040830 phpBB-fc2/20040831 spamassassin-fc2/20040910 PostNuke-fc2/20040824 sl-webalizer-fc2/20040<br />
818"<br />
<br />
[root@virt12 sbin]# vzctl exec 1212 df -h<br />
Filesystem Size Used Avail Use% Mounted on<br />
/dev/vzfs 4.0G 405M 3.7G 10% /</pre><br />
<br />
From this you can see that he’s using (and will minimally need free on the dst server) ~400MB, and he’s running on a Fedora 2 template, version 20040903. He’s also got a bunch of other templates installed. It’s is '''vital''' that '''all''' these templates exist on the dst system. To confirm that, on the dst system run:<br />
<br />
For < 3.0:<br />
<br />
<pre>[root@virt14 private]# vzpkgls | grep fc2<br />
devel-fc2 20040903<br />
PostNuke-fc2 20040824<br />
analog-fc2 20040824<br />
awstats-fc2 20040824<br />
bbClone-fc2 20040824<br />
jdk-fc2 20040823<br />
jre-fc2 20040823<br />
mailman-fc2 20040823<br />
mod_frontpage-fc2 20040816<br />
mod_perl-fc2 20040812<br />
mod_ssl-fc2 20040811<br />
mysql-fc2 20040812<br />
openwebmail-fc2 20040817<br />
php-fc2 20040813<br />
phpBB-fc2 20040831<br />
postgresql-fc2 20040813<br />
proftpd-fc2 20040818<br />
sl-webalizer-fc2 20040818<br />
spamassassin-fc2 20040910<br />
tomcat-fc2 20040823<br />
usermin-fc2 20040909<br />
uw-imap-fc2 20040830<br />
webmin-fc2 20040909<br />
[root@virt14 private]# vzpkgls | grep fedora<br />
fedora-core-1 20040121 20040818<br />
fedora-core-devel-1 20040121 20040818<br />
fedora-core-2 20040903<br />
[root@virt14 private]#</pre><br />
<br />
For these older systems, you can simply match up the date on the template. <br />
<br />
For >= 3.0:<br />
<br />
<pre>[root@virt19 /vz2/private]# vzpkg list<br />
centos-5-x86 2008-01-07 22:05:57<br />
centos-5-x86 devel<br />
centos-5-x86 jre<br />
centos-5-x86 jsdk<br />
centos-5-x86 mod_perl<br />
centos-5-x86 mod_ssl<br />
centos-5-x86 mysql<br />
centos-5-x86 php<br />
centos-5-x86 plesk9<br />
centos-5-x86 plesk9-antivirus<br />
centos-5-x86 plesk9-api<br />
centos-5-x86 plesk9-atmail<br />
centos-5-x86 plesk9-backup<br />
centos-5-x86 plesk9-horde<br />
centos-5-x86 plesk9-mailman<br />
centos-5-x86 plesk9-mod-bw<br />
centos-5-x86 plesk9-postfix<br />
centos-5-x86 plesk9-ppwse<br />
centos-5-x86 plesk9-psa-firewall<br />
centos-5-x86 plesk9-psa-vpn<br />
centos-5-x86 plesk9-psa-fileserver<br />
centos-5-x86 plesk9-qmail<br />
centos-5-x86 plesk9-sb-publish<br />
centos-5-x86 plesk9-vault<br />
centos-5-x86 plesk9-vault-most-popular<br />
centos-5-x86 plesk9-watchdog</pre><br />
<br />
On these newer systems, it's difficult to tell whether the template on the dst matches exactly the src. Just cause a centos-5-x86 is listed on both servers doesn't mean all the same packages are there on the dst. To truly know, you must perform a sample rsync:<br />
<br />
rsync -avn /vz/template/centos/5/x86/ root@10.1.4.61:/vz/template/centos/5/x86/<br />
<br />
if you see a ton of output from the dry run command, then clearly there are some differences. You may opt to let the rsync complete (without running in dry run mode) the only downside is you've now used up more space on the dst and also the centos template will be a mess with old and new data- it will be difficult if not impossible to undo (if someday we wanted to reclaim the space).<br />
<br />
If you choose to merge templates, you should closely inspect the dry run output. You should also take care to exclude anything in the /config directory. For example:<br />
<br />
rsync -av -e ssh --stats --exclude=x86/config /vz/template/ubuntu/10.04/ root@10.1.4.62:/vz/template/ubuntu/10.04/<br />
<br />
Which will avoid this directory and contents:<br />
<pre>[root@virt11 /vz2/private]# ls /vz/template/ubuntu/10.04/x86/config*<br />
app os</pre><br />
<br />
This is important to avoid since the config may differ on the destination and we are really only interested in making sure the pacakges are there, not overwriting a newer config with an older one.<br />
<br />
If the dst system was missing a template, you have 2 choices: <br />
# put the missing template on the dst system. 2 choices here: <br />
## Install the template from rpm (found under backup2: /mnt/data4/vzrpms/distro/) or <br />
## rsync over the template (found under /vz/template) - see above<br />
# put the ve on a system which has all the proper templates<br />
<br />
=== pre-seeding a migration ===<br />
<br />
When migrating a customer (or when doing many) depending on how much data you have to transfer, it can take some time. Further, it can be difficult to gauge when a migration will complete or how long it will take. To help speed up the process and get a better idea about how long it will take you can pre-transfer a customer's data to the destination server. If done correctly, vzmigrate will see the pre-transferred data and pick up where you left off, having much less to transfer (just changed/new files). <br />
<br />
We believe vzmigrate uses rsync to do it's transfer. Therefore not only can you use rsync to do a pre-seed, you can also run rsync to see what is causing a repeatedly-failing vzmigrate to fail. <br />
<br />
There's no magic to a pre-seed, you just need to make sure it's named correctly.<br />
<br />
Given:<br />
<br />
source: /vz1/private/1234<br />
<br />
and you want to migrate to /vz2 on the target system, your rsync would look like:<br />
<br />
rsync -av /vz1/private/1234/ root@x.x.x.x:/vz2/private/1234.migrated/<br />
<br />
After running that successful rsync, the ensuing migrateonline (or migrate) will take much less time to complete- depending on the # of files to be analyzed and the # of changed files. In any case, it'll be much much faster than had you just started the migration from scratch.<br />
<br />
Further, as we discuss elsewhere in this topic, a failed migration can be moved from <tt>/vz/private/1234</tt> to <tt>/vz/private/1234.migrated</tt> on the destination if you want to restart a failed migration. This should '''only''' be done if the migration failed and the CT is not running on the destination HN.<br />
<br />
=== migrateonline intructions: src >=3.x -> dst>=3.x ===<br />
<br />
A script called [[#migrateonline|migrateonline]] was written to handle this kind of move. It is basically a wrapper for <tt>vzmigrate</tt> – vzmigrate is a util to seamlessly- as no no reboot of the ve necessary- move a ve from one host to another. This wrapper was initially written cause vz virtuozzo version 2.6.0 has a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables, causing problems when the ve was started up on the dst system. [[#migrate|migrate]] mitigates that. Since it makes multiple ssh connections to the dst virt, it’s a good idea to put the pub key for the src virt in the authorized_keys file on the dst virt. In addition, migrateonline emails ve owners when their migration starts and stops. For this to happen they need to put email addresses (on a single line, space delimited) in a file on their system: /migrate_notify. If the optional target dir is not specified, the ve will be moved to the same private/root location as it was on the src virt. Note: <tt>migrate</tt> is equivalent to <tt>migrateonline</tt>, but will <tt>migrate</tt> a ve AND restart it in the process.<br />
<br />
<pre>[root@virt12 sbin]# migrateonline<br />
usage: /usr/local/sbin/migrateonline <ip of node migrating to> <veid> [target dir: vz | vz1 | vz2]<br />
[root@virt12 sbin]# migrateonline 10.1.4.64 1212 vz<br />
starting to migrate 1212 at Sat Mar 26 22:40:38 PST 2005<br />
Turning off offline_management<br />
Saved parameters for VE 1212<br />
migrating with no start on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
clearing the arp cache<br />
now going to 10.1.4.64 and clear cache and starting<br />
starting it<br />
Delete port redirection<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
finished migrating 1212 at Sat Mar 26 22 22:52:01 PST 2005<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine.<br />
<br />
If they had backups, use the mvbackups command to move their backups to the new server:<br />
<br />
mvbackups 1212 virt14 vz<br />
<br />
Rename the ve<br />
<br />
[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/migrated-1212<br />
<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/old-1212-migrated-20120404-noarchive<br />
<br />
Update the customer’s systems in mgmt to reflect the new path and server.<br />
<br />
<br />
IF migrateonline does not work, you can try again using simply migrate- this will result in a brief reboot for the ve.<br />
Before you try again, make sure of a few things:<br />
<br />
Depending on where in the migration died, there may be partial data on the dst system in 1 of 2 places:<br />
(given the example above)<br />
<br />
/vz/private/1212<br />
<br />
or <br />
<br />
/vz/private/1212.migrated<br />
<br />
before you run migrate again, you'll want to rename so that all data is in <br />
1212.migrated:<br />
<br />
mv /vz/private/1212 /vz/private/1212.migrated<br />
<br />
this way, it will pick up where it left off and transfer only new files.<br />
<br />
Likewise, if you want to speed up a migration, you can pre-seed the dst as follows:<br />
<br />
[root@virt12 sbin]# rsync -avSH /vz/private/1212/ root@10.1.4.64:/vz/private/1212.migrated/<br />
<br />
then when you run migrate or migrateonline, it will only need to move the changed files- the migration will complete quickly<br />
<br />
=== migrateonline/migrate failures (migrate manually) ===<br />
<br />
Lets say for whatever reason the migration fails. If it fails with [[#migrateonline|migrateonline]], you should try [[#migrate|migrate]] (which will reboot the customer, so notify them ahead of time).<br />
<br />
You may want to run a [[#pre-seeding_a_migration|pre-seed]] rsync to see if you can find the problem. On older virts, we've seen this problem due to a large logfile (which you can find and encourage the customer to remove/compress):<br />
for f in `find / -size +1048576k`; do ls -lh $f; done<br />
<br />
You may also see migration failing due to quota issues.<br />
<br />
You can try to resolve by copying any quota file into the file you need:<br />
<br />
cp /var/vzquota/quota.1 /var/vzquota/quota.xxx<br />
<br />
If it complains about quota running you should then be able to stop it<br />
<br />
vzquota off xxxx<br />
<br />
If all else fails, migrate to a new VEID<br />
i.e. 1234 becomes 12341<br />
<br />
<br />
If the rsync or [[#migrate|migrate]] fails, you can always move someone manually:<br />
<br />
1. stop ve: <br><br />
v stop 1234<br />
<br />
2. copy over data<br><br />
rsync -avSH /vz/private/1234/ root@1.1.1.1:/vzX/private/1234/<br />
<br />
NOTE: if you've previously seeded the data (run rsync while the VE was up/running), and this is a subsequent rsync, make sure the last rsync you do (while the VE is not running, has the --delete option in the rsync)<br />
<br />
3. copy over conf<br><br />
scp /vzconf/1234.conf root@1.1.1.1:/vzconf<br />
<br />
4. on dst, edit the conf to reflect the right vzX dir<br><br />
vi /vzconf/1234.conf<br />
<br />
5. on src remove the IPs<br><br />
ipdel 1234 2.2.2.2 3.3.3.3<br />
<br />
6. on dst add IPs <br><br />
ipadd 1234 2.2.2.2 3.3.3.3<br />
<br />
7. on dst, start ve: <br><br />
v start 1324<br />
<br />
8. cancel, then archive ve on src per above instrs.<br />
<br />
=== migrate src=2.6.0 -> dst>=2.6.0, or mass-migration with customer notify ===<br />
<br />
A script called <tt>migrate</tt> was written to handle this kind of move. It is basically a wrapper for vzmigrate – vzmigrate is a util to seamlessly move a ve from one host to another. This wrapper was initially written cause vz virtuozzo version 2.6.0 has a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables, causing problems when the ve was started up on the dst system. migrate mitigates that. Since it makes multiple ssh connections to the dst virt, it’s a good idea to put the pub key for the src virt in the authorized_keys file on the dst virt. In addition, migrate emails ve owners when their migration starts and stops. For this to happen they need to put email addresses (on a single line, space delimited) in a file on their system: /migrate_notify. If the optional target dir is not specified, the ve will be moved to the same private/root location as it was on the src virt. Note: migrateonline is equivalent to migrate, but will migrate a ve from one 2.6 '''kernel''' machine to another 2.6 kernel machine without restarting the ve.<br />
<br />
<pre>[root@virt12 sbin]# migrate<br />
usage: /usr/local/sbin/migrate <ip of node migrating to> <veid> [target dir: vz | vz1 | vz2]<br />
[root@virt12 sbin]# migrate 10.1.4.64 1212 vz<br />
starting to migrate 1212 at Sat Mar 26 22:40:38 PST 2005<br />
Turning off offline_management<br />
Saved parameters for VE 1212<br />
migrating with no start on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
clearing the arp cache<br />
now going to 10.1.4.64 and clear cache and starting<br />
starting it<br />
Delete port redirection<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
finished migrating 1212 at Sat Mar 26 22 22:52:01 PST 2005<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine (backup2 or mail).<br />
Cancel the ve (first we have to rename things which migrate changed so cancelve will find them):<br />
<br />
[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/1212.conf<br />
<br />
On 2.6.1 you’ll also have to move the private area:<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/1212<br />
<br />
<pre>[root@virt12 sbin]# cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, [[#cancelve|cancelve]] would offer to remove them. You want to say '''no''' to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
To move everyone off a system, you’d do:<br />
for f in `vl`; do migrate <ip> $f; done<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
=== vzmigrate: src=2.6.1 -> dst>=2.6.0 ===<br />
<br />
This version of vzmigrate works properly with regard to handling ips. It will not notify ve owners of moves as in the above example. Other than that it’s essentially the same.<br />
<br />
<pre>[root@virt12 sbin]# vzmigrate 10.1.4.64 -r no 1212:1212:/vz/private/1212:/vz/root/1212<br />
migrating on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine (backup2 or mail).<br />
Cancel the ve (first we have to rename things which vzmigrate changed so cancelve will find them):<br />
<br />
<pre>[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/1212.conf<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/1212<br />
<br />
[root@virt12 sbin]# cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, <tt>cancelve</tt> would offer to remove them. You want to say no to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
=== src=2.5.x ===<br />
<br />
First, go to the private dir:<br />
<br />
cd /vz1/private/<br />
<br />
Stop the VE - make sure it stops totally cleanly.<br />
<br />
vzctl stop 1212<br />
<br />
Then you’d use vemove - a script written to copy over the config, create tarballs of the ve’s data on the destination virt, and cancel the ve on the source system (in this example we’re going to put a ve that was in /vz1/private on the src virt, in /vz/private on the dst virt):<br />
<br />
<pre>[root@virt12 sbin]# vemove<br />
ERROR: Usage: vemove veid target_ip target_path_dir<br />
[root@virt12 sbin]# vemove 1212 10.1.4.64 /vz/private/1212<br />
tar cfpP - 1212 --ignore-failed-read | (ssh -2 -c arcfour 10.1.4.64 "split - -b 1024m /vz/private/1212.tar" )<br />
scp /vzconf/1212.conf 10.1.4.64:/vzconf<br />
cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, cancelve would offer to remove them. You want to say no to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
When you are done, go to /vz/private on the dst virt you will have files like this:<br />
<br />
<pre>1212.taraa<br />
1212.tarab<br />
1212.tarac</pre><br />
<br />
Each one 1024m (or less, for the last one) in size.<br />
<br />
on the dst server and run:<br />
<br />
cat 1212.tar?? | tar xpPBf -<br />
<br />
and after 20 mins or so it will be totally untarred. Now since the conf<br />
file is already there, you can go ahead and start the system.<br />
<br />
vzctl start 1212<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
NOTE: you MUST tar the system up using the virtuozzo version of tar that<br />
is on all the virt systems, and further you MUST untar the tarball with<br />
the virtuozzo tar, using these options: `<tt>tar xpPBf -</tt>`<br />
<br />
If you tar up an entire VE and move it to a non-virtuozzo machine, that is<br />
ok, and you can untar it there with normal tar commands, but do not untar<br />
it and then repack it with a normal tar and expect it to work - you need<br />
to use virtuozzo tar commands on virtuozzo tarballs to make it work.<br />
<br />
The backups are sort of an exception, since we are just (usually)<br />
restoring user data that was created after we gave them the system, and<br />
therefore has nothing to do with magic symlinks or vz-rpms, etc.<br />
<br />
== Moving a VE on the same virt ==<br />
<br />
Easy way:<br><br />
Scenario 1: ve 123 is to be renamed 1231 and moved from vz1 to vz<br />
<br />
vzmlocal 123:1231:/vz/private/1231:/vz/root/1231<br />
<br />
Scenario 2: ve 123 is to be moved vz1 to vz<br />
<br />
vzmlocal 123:123:/vz/private/123:/vz/root/123<br />
<br />
vzmlocal will reboot the ve at the end of the move<br />
<br />
Manual/old way:<br />
<br />
1) <tt>vzctl stop 123</tt><br><br />
2) <tt>mv /vz1/private/123 /vz/private/.</tt><br><br />
(or cp -a if you want to copy)<br />
3) in <tt>/etc/sysconfig/vz-scripts/123.conf</tt> change value<br><br />
of '<tt>VE_PRIVATE</tt>' variable to point to a new private area location<br />
4) <tt>vzctl start 123</tt><br><br />
5) update backups if needed: <tt>mvbackups 123 virtX virt1 vz</tt><br><br />
6) update management scerens<br><br />
<br />
Notes: a) absolute path to private area is stored in quota file <tt>/var/vzquota/quota.123</tt> - so during first startup quota will be recalculated.<br><br />
b) if you're going to write some script to do a job, you MUST be sure that $VEID won't be expanded to '' in ve config file - ie. you need to escape '$'. Otherwise you might have:<br />
<br />
VE_PRIVATE="/vz/private/"<br />
<br />
in config, and 'vzctl destroy' for this VE ID '''will remove everything under /vz/private/ directory'''.<br />
<br />
== Adding a veth device to a VE ==<br />
<br />
Not totally sure what this is, but a customer asked for it and here's what we did (as instructed by vz support):<br />
<br />
<pre>v set 99 --netif_add eth99 --save<br />
ipdel 99 69.55.230.58<br />
v set 99 --ifname eth99 --ipadd 69.55.230.58 --save<br />
v set 99 --ifname eth99 --gateway 69.55.230.1 --save<br />
<br />
vznetcfg net new veth_net<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
net99 active eth0 veth77.77,veth99.99<br />
veth_net active<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
br0 bridge veth_net<br />
br99 bridge net99<br />
veth99.99 veth net99<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic net99 69.55.227.70/24<br />
<br />
vznetcfg br attach br0 eth0<br />
<br />
(will remove 99 from orig net and move to veth_net)<br />
vznetcfg net addif veth_net veth99.99<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
net99 active<br />
veth_net active eth0 veth77.77,veth99.99<br />
<br />
(delete the old crap)<br />
vznetcfg net del net99<br />
<br />
<br />
Then, to add another device in<br />
<br />
v set 77 --netif_add eth77 --save<br />
ipdel 77 69.55.230.78<br />
v set 77 --ifname eth77 --ipadd 69.55.230.78 --save<br />
v set 77 --ifname eth77 --gateway 69.55.230.1 --save<br />
v set 77 --save --ifname eth77 --network veth_net<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
veth77.77 veth<br />
br0 bridge veth_net<br />
veth99.99 veth veth_net<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic veth_net 69.55.227.70/24<br />
<br />
vznetcfg net addif veth_net veth77.77<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
veth77.77 veth veth_net<br />
br0 bridge veth_net<br />
veth99.99 veth veth_net<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic veth_net 69.55.227.70/24<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
veth_net active eth0 veth77.77,veth99.99<br />
<br />
another example<br />
<br />
v set 1182 --netif_add eth1182 --save<br />
ipdel 1182 69.55.236.217<br />
v set 1182 --ifname eth1182 --ipadd 69.55.236.217 --save<br />
v set 1182 --ifname eth1182 --gateway 69.55.236.1 --save<br />
vznetcfg net addif veth_net veth1182.1182<br />
v set 1182 --save --ifname eth1182 --network veth_net<br />
<br />
<br />
unused/not working commands:<br />
ifconfig veth99.0 0<br />
vznetcfg net list<br />
vznetcfg br new br99 net99<br />
vznetcfg br attach br99 eth0<br />
vznetcfg br show<br />
vznetcfg if list<br />
vznetcfg net addif net99 veth99.99<br />
vznetcfg net addif eth0 net99<br />
<br />
---------<br />
<br />
vznetcfg br new br1182 net1182<br />
<br />
vznetcfg br attach br99 eth0<br />
vznetcfg if list<br />
vznetcfg net addif net99 veth99.99<br />
vznetcfg net addif eth0 net99<br />
<br />
<br />
<br />
vznetcfg net addif eth0 net1182<br />
<br />
<br />
# vznetcfg net addif veth99.0 net99<br />
--- 8< ---<br />
<br />
vznetcfg net new net<br />
# vznetcfg net addif eth0 net99<br />
<br />
<br />
# vzctl set 99 --save --netif_add eth0 (at this stage veth99.0 interface have to appear<br />
on node)<br />
# vzctl set 99 --save --ifname eth0 --ipadd 69.55.230.58 (and probably few more arguments<br />
here - see 'man vzctl')<br />
# vznetcfg net addif veth99.0 net99</pre><br />
<br />
<br />
== Assigning/remove ip from a VE ==<br />
<br />
1. Add or remove ips:<br />
ipdel 1234 1.1.1.1 2.2.2.2<br />
ipadd 1234 1.1.1.1 2.2.2.2<br />
<br />
2. update Mgmt screens<br />
<br />
3. offer to update any DNS we do for them<br />
<br />
4. check to see if we had rules for old IP in firwall<br />
<br />
== Enabling tun device for a ve ==<br />
Note, there’s a command for this: [[#addtun|addtun]]<br />
<br />
For posterity:<br />
Make sure the tun.o module is already loaded before Virtuozzo is started: <br />
lsmod <br />
Allow the VPS to use the TUN/TAP device: <br />
vzctl set 101 --devices c:10:200:rw --save <br />
Create the corresponding device inside the VPS and set the proper permissions: <br />
vzctl exec 101 mkdir -p /dev/net <br />
vzctl exec 101 mknod /dev/net/tun c 10 200 <br />
vzctl exec 101 chmod 600 /dev/net/tun<br />
<br />
== Remaking a system (on same virt) ==<br />
<br />
1. [[#cancelve|cancelve]] (or v destroy x - ONLY if you're POSITIVE no data needs to be saved)<br />
<br />
2. [[#vemake|vemake]] using same veid<br />
<br />
3. [[#mvbackups|mvbackups]] or [[#vb|vb]] (if new mount point)<br />
<br />
4. update mgmt with new dir/ip <br />
<br />
5. update firewall if changed ip<br />
<br />
== Re-initialize quota for a VE ==<br />
<br />
There’s a commamd for this now: [[#clearquota|clearquota]]<br />
<br />
For posterity:<br />
<br />
vzctl stop 1<br />
vzquota drop 1<br />
vzctl start 1<br />
<br />
<br />
== Traffic accounting on linux ==<br />
<br />
DEPRECATED - all tracking is done via bwdb now. This is how we used to track traffic.<br />
<br />
TODO: update for diff versions of vz<br />
<br />
Unlike FreeBSD, where we have to add firewall count rules to the system to count the traffic, on virtuozzo counts the traffic for us. You an see the current traffic stats by running `vznetstat`:<br />
<br />
<pre># vznetstat<br />
VEID Net.Class Output(bytes) Input(bytes)<br />
-----------------------------------------------<br />
24218 1 484M 39M<br />
24245 1 463M 143M<br />
2451 1 2224M 265M<br />
2454 1 2616M 385M<br />
4149 1 125M 68M<br />
418 1 1560M 34M<br />
472 1 1219M 315M<br />
726 1 628M 317M<br />
763 1 223M 82M<br />
771 1 4234M 437M<br />
-----------------------------------------------<br />
Total: 13780M 2090M</pre><br />
<br />
As you can see the VEID is on a line with the in and out bytes. So, we simply run a cron job:<br />
<br />
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /root/vztrafdump.sh<br />
<br />
Just like we do on FreeBSD - this one goes through all the VEs in /vz/private and greps the line from vznetstat that matches them and dumps it in /jc_traffic_dump on their system. Then it does it again for all the VEs in /vz1/private. It is important to note that vznetstat runs only once, and the grepping is done from a temporary file that contains that output - we do this because running vznetstat once for each VE that we read out of /vz/private and /vz1/private would take way too long and be too intensive.<br />
<br />
You do not need to do anything to facilitate this other than make sure that that cron job is running - the vznetstat counters are always running, and any new VEs that are added to the system will be accounted for automatically.<br />
<br />
Traffic resetting no longer works with vz 2.6, so we disable the vztrafdump.sh on those virts.<br />
<br />
== Watchdog script ==<br />
<br />
On some of the older virts, we have a watchdog running that kills procs that are deemed bad per the following:<br />
<br />
/root/watchdog from quar1<br />
<br />
<pre>if echo $line | awk '{print $(NF-3)}' | grep [5-9]...<br />
then<br />
# 50-90%<br />
if echo $line | awk '{print $(NF-1)}' | grep "...:.."<br />
then<br />
# running for > 99min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
if echo $line | awk '{print $(NF-1)}' | grep "....m"<br />
then<br />
# running for > 1000min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
fi<br />
<br />
if echo $line | awk '{print $(NF-3)}' | grep [1-9]...<br />
then<br />
# running for 10-90 percent<br />
if echo $line | awk '{print $NF}' | egrep 'cfusion|counter|vchkpw'<br />
then<br />
<br />
if echo $line | awk '{print $(NF-1)}' | grep "[2-9]:.."<br />
then<br />
# between 2-9min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
elif echo $line | awk '{print $(NF-1)}' | grep "[0-9][0-9]:.."<br />
then<br />
# up to 99min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
fi<br />
fi</pre><br />
<br />
<br />
== Misc Linux Items ==<br />
<br />
We are overselling hard drive space ... when you configure a linux system with a certain amount of disk space (the default is 4gigs) you do not actually use up 4gigs of space on the system. The diskspace setting for a user is simply a cap, and they only use up as much space on the actual disk drive as they are actually using.<br />
<br />
When you create a new linux system, even though there are some 300 RPMs or so installed, if you run `df -k` you will see that the entire 4gig partition is empty - no space is being used. This is because the files in their system are "magic symlinks" to the template for their OS that is in /vz/template - however, any changes to any of those files will "disconnect" them and they will immediately begin using space in their system. Further, any new files uploaded (even if those new files overwrite existing files) will take up space on the partition.<br />
<br />
<br />
if you see this:<br />
<br />
<pre>[root@virt8 root]# vzctl stop 160 ; vzctl start 160<br />
VE is not running<br />
Starting VE ...<br />
VE is unmounted<br />
VE is mounted<br />
Adding IP address(es): 69.55.226.83 69.55.226.84<br />
bash ERROR: Can't change file /etc/sysconfig/network<br />
Deleting IP address(es): 69.55.226.83 69.55.226.84<br />
VE is unmounted<br />
[root@virt8 root]#</pre><br />
<br />
it probably means they no longer have /bin/bash - copy one in for them<br />
<br />
ALSO: another possibility is that they have removed the `ed` RPM from their system - it needs to be reinstalled into their system. But since their system is down, this is tricky ...<br />
<br />
VE startup scripts used by 'vzctl' want package 'ed' to be available inside VE. So if package 'ed' will be enabled in OS template config and OS template itself VE #827 is based on - this error should be fixed.<br />
<br />
yes, it is possible to add RPM to VE while it not running.<br />
Try to do following:<br />
<br />
<pre># cd /vz/template/<OS_template_with_ed_package>/<br />
# vzctl mount 827<br />
# rpm -Uvh --root /vz/root/827 --veid 827 ed-0.2-25.i386.vz.rpm</pre><br />
<br />
Usually theres an error, but its ok<br />
<br />
Note: replace 'ed-0.2-25.i386.vz.rpm' in last command with actual<br />
version of 'ed' package you have.<br />
<br />
----<br />
<br />
So how do I know what template the user has ? cat their conf file and it is listed in there. For example, if the conf file has:<br />
<br />
<pre>[root@virt12 sbin]# vc 1103<br />
…snip…<br />
OSTEMPLATE="debian-3.0/20030822"<br />
TEMPLATES="mod_perl-deb30/20030707 mod_ssl-deb30/20030703 mysql-deb30/20030707 proftpd-deb30/20030703 webmin-deb30/20030823 "<br />
…</pre><br />
<br />
then they are on debian 3.0, all of their system RPMs are in /vz/template/debian-3.0, and they are using version 20030822 of that debian 3.0 template. Also, they’ve also got additional packages installed (mod_perl, mod_ssl, etc). Those are also found under /vz/template<br />
<br />
----<br />
<br />
Edits needed to run java:<br />
<br />
When we first created the VEs, the default setting for privvmpages was 93000:94000 ... which was high enough that most people never had problems ... however, you can;t run java or jdk or tomcat or anything java related with that setting. We have found that by setting privvmpages to 610000:615000 that java runs just fine. That is now the default setting. It is exceedingly rare that anyone needs it higher than that, although we have seen it once or twice.<br />
<br />
Any problems with java at all - the first thing you need to do is see if the failcnt has raised for privvmpages.<br />
<br />
<pre># vzctl start 160<br />
Starting VE ...<br />
vzquota : (error) Quota on syscall for 160: Device or resource busy<br />
Running vzquota on failed for VE 160 [3]</pre><br />
<br />
This is because my pwd is _in_ their private directory - you can't start it until you move out<br />
<br />
People seem to have trouble with php if they are clueless newbies. Here are two common problems/solutions:<br />
<br />
no... but i figured it out myself. problem was the php.ini file that came<br />
vanilla with the account was not configured to work with apache (the<br />
ENGINE directive was set to off).<br />
<br />
everything else seems fine now.<br />
<br />
----<br />
<br />
the problem was in the php.ini file. I noticed that is wasnt showing<br />
the code when it was in an html file so I looked at the php.ini file<br />
and had to change it so it recognized <? tags aswell as <?php tags.<br />
<br />
Also, make sure added to httpd.conf<br />
AddType application/x-httpd-php .php<br />
<br />
----<br />
<br />
You can set the time zone:<br />
<br />
You can change the timezone by doing this:<br />
<br />
ln -sf /usr/share/zoneinfo/<zone> /etc/localtime<br />
<br />
where <zone> is the zone you want in the /usr/share/zoneinfo/ directory.<br />
<br />
----<br />
<br />
Failing shm_open calls:<br />
<br />
first, please check if /dev/shm is mounted inside VE.<br />
'cat /proc/mounts' command should show something like this:<br />
tmpfs /dev/shm tmpfs rw 0 0<br />
<br />
If /dev/shm is not mounted you have 2 ways to solve issue:<br />
1. execute following command inside VE (doesn't require VE reboot):<br />
mount -t tmpfs none /dev/shm<br />
2. add following string to /etc/fstab inside VE and reboot it:<br />
tmpfs /dev/shm tmpfs defaults 0 0<br />
<br />
----<br />
<br />
You can have a mounted but not running ve<br />
Just:<br />
vzctl mount <veid><br />
<br />
----<br />
<br />
When a debian sys can’t get on the network, and you try:<br />
<br />
<pre>vzctl set 1046 --ipadd 69.55.227.117<br />
Adding IP address(es): 69.55.227.117<br />
Failed to bring up lo.<br />
Failed to bring up venet0.</pre><br />
<br />
They probably removed iproute package, which must be the one from swsoft. To restore:<br />
<pre># dpkg -i --veid=1046 --admindir=/vz1/private/1046/root/var/lib/dpkg --instdir=/vz1/private/1046/root/ /vz/template/debian-3.0/iproute_20010824-8_i386.vz.deb<br />
(Reading database ... 16007 files and directories currently installed.)<br />
Preparing to replace iproute 20010824-8 (using .../iproute_20010824-8_i386.vz.deb) ...<br />
Unpacking replacement iproute ...<br />
Setting up iproute (20010824-8) ...</pre><br />
<br />
Then restart their ve<br />
<br />
----<br />
<br />
in a ve i do:<br />
<br />
<pre>cd /<br />
du -h .</pre><br />
<br />
and get: 483M .<br />
<br />
i do:<br />
<br />
<pre>bash-2.05a# df -h<br />
Filesystem Size Used Avail Use% Mounted on<br />
/dev/vzfs 4.0G 2.3G 1.7G 56% /</pre><br />
<br />
how can this be?<br />
<br />
Is it possible that quota file was corrupted somehow? Please try to: <br />
<pre>vzctl stop <VEID><br />
vzquota drop <VEID><br />
vzquota init <VEID><br />
vzctl start <VEID></pre><br />
<br />
----<br />
<br />
How to stop vz from starting after reboot:<br />
<br />
VIRTUOZZO=no <br />
in <br />
/etc/sysconfig/vz<br />
<br />
To start: <br />
service vz start<br />
(after setting VIRTUOZZO=yes in /etc/sysconfig/vz)<br />
<br />
service vz restart will do some kind of 'soft reboot' -- restart all<br />
VPSes and reload modules without rebooting the node<br />
<br />
if you need to shut down all VPSes really really fast, run killall -9 init<br />
<br />
----<br />
<br />
Postfix tip:<br />
<br />
You may want to tweak settings: default_process_limit=10<br />
<br />
= Virtuozzo VPS Management Tools =<br />
<br />
== vm ==<br />
<br />
TODO<br />
<br />
== cancelve ==<br />
cancelve <veid><br />
this will:<br />
* stop a ve<br />
* check for backups (offer to remove them from the backup server <br />
and the backup.config)<br />
* rename the private dir<br />
* check for PTR, provide the commands to reset to default<br />
* and rename the ve’s config<br />
* remind you to remove firewall rules<br />
* remind you to remove DNS entries<br />
<br />
== ipadd ==<br />
ipadd <veid> <ip> [ip] [ip]<br />
add’s ip(s) to a ve<br />
<br />
== ipdel ==<br />
ipdel <veid> <ip> [ip] [ip]<br />
removes ip(s) from a ve<br />
<br />
== vc ==<br />
vc <veid><br />
the equivalent of: <tt>cat /vzconf/<veid>.conf</tt><br />
<br />
== vl ==<br />
vl<br />
will displays a list of ve #’s, 1 per line. (ostensibly to use in a for loop)<br />
<br />
== vp ==<br />
vp <veid><br />
the equivalent of: <tt>vzps auxww –E <veid></tt><br />
<br />
== vpe ==<br />
DEPRECATED<br />
vpe <veid> <br />
this will allow you to do a vp when a ve is running out of control, the equivalent of (deprecated since vp operates outside the VPS): <br />
<pre>vzctl set <veid> --kmemsize 2100000:2200000<br />
vzctl exec <veid> ps auxw<br />
vzctl set <veid> --kmemsize (ve’s orig lvalue):(ve’s orig hvalue)</pre><br />
<br />
== vt ==<br />
vt <veid><br />
the equivalent of: <tt>vztop –E <veid></tt><br />
<br />
== vr ==<br />
vr <veid><br />
the equivalent of: <tt>vzctl stop <veid>; vzctl start <veid></tt><br />
You can run this even if the ve is down - the stop command will just fail<br />
<br />
== vs ==<br />
vs [veid]<br />
displays status (output of <tt>vzctl status <veid></tt>) on each ve configured on the system (as determined by <tt>/vzconf/*.conf</tt>)<br />
If passed an argument, gives the status for just that ve. <br />
A running system looks like:<br />
<tt>VEID 16066 exist mounted running</tt><br />
<br />
A ve which is not running (but does exist) looks like:<br />
<tt>VEID 9990 exist unmounted down</tt><br />
<br />
A ve which is not running and doesn’t exist looks like:<br />
<tt>VEID 421 deleted unmounted down</tt><br />
<br />
== vs2 ==<br />
vs2 [veid]<br />
this is similar to vs in that it displays status (output of <tt>vzctl status <veid></tt>) on each ve,<br />
but the difference is it’s list comes from doing an ls on the data dirs. This was meant to catch <br />
the rare case where a ve configured but exists. <br />
<br />
== vw ==<br />
vw [veid]<br />
displays the output of ‘<tt>w</tt>’ (the equivalent of <tt>vzctl exec <veid> w</tt>) for each configured ve (as determined by <tt>/vzconf/*.conf</tt>). Useful for determing which ve is contributing to a heavily-loaded system.<br />
If passed an argument, gives ‘<tt>w</tt>‘ output for just that ve. <br />
Ex:<br />
<pre>[root@virt2 etc]# vw<br />
134<br />
10:52pm up 79 days, 6:14, 0 users, load average: 0.02, 0.02, 0.00<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT<br />
16027<br />
2:52pm up 7 days, 19:54, 0 users, load average: 0.00, 0.00, 0.00<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT<br />
16055<br />
2:52pm up 79 days, 6:38, 0 users, load average: 0.00, 0.04, 0.07<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT</pre><br />
<br />
== vwe ==<br />
vwe [constraint]<br />
just like <tt>vw</tt>, but takes a constraint as an argument, only show’s ve’s with loads >= the constraint provided. If no constraint is provided, 1 is used by default<br />
<br />
== vzs ==<br />
vzs [veid]<br />
displays the beancounter status for all ve’s, or a particular ve if an argument is passed<br />
<br />
== ve ==<br />
ve <veid><br />
the equivalent of: <tt>vzctl enter <veid></tt><br />
<br />
== vx ==<br />
vx <veid> <command><br />
the equivalent of: <tt>/usr/sbin/vzctl exec <veid> <command></tt><br />
<br />
== dprocs ==<br />
dprocs [count]<br />
a script which outputs a continuous report (or a certain number of reports if an option is passed) of processes stuck in the D state and which VPS’s those procs belong to.<br />
<br />
== setmem ==<br />
setmem <VEID> <256|512|768|1024|2048><br />
adjusts the memory resources for the VE<br />
<br />
== afacheck.sh ==<br />
afacheck.sh<br />
displays the health/status of containers and mirrors on an adaptec card (currently quar1, tempvirt1-2, virt9, virt10)- all other are LSI<br />
<br />
== backup ==<br />
backup<br />
backup script called nightly to update virt scripts and do customer backups<br />
<br />
== checkload.pl ==<br />
checkload.pl<br />
this was intended to be setup as a cronjob to watch processes on a virt when the load <br />
rises above a certain level. Not currently in use.<br />
<br />
== findbackuppigs.pl ==<br />
findbackuppigs.pl<br />
looks for files larger than 50MB which customers have asked us to backup. Emails matches<br />
to linux@johncompanies.com<br />
<br />
== gatherlinux.pl ==<br />
gatherlinux.pl<br />
gathers up data about ve’s configured and writes to a file. Used for audits against the db<br />
<br />
== gb ==<br />
gb <search><br />
greps backup.config for the given search parameter<br />
<br />
== gbg ==<br />
gbg <search><br />
greps backup.config for the given search parameter and presents just the directories (for clean pasting)<br />
<br />
== linuxtrafficgather.pl ==<br />
linuxtrafficgather.pl [yy-mm]<br />
sends a traffic usage summary by ve to support@johncomapnies.com and payments@johncompanies.com.<br />
Optional arguments are year and month (must be in the past). If not passed, assumes last month. Relies on <br />
traffic logs created by netstatreset and netstatbackup<br />
<br />
== linuxtrafficwatch.pl ==<br />
linuxtrafficwatch.pl<br />
checks traffic usage and emails support@johncomapnies.com when a ve reaches the warning level (35G) and<br />
the limit (40G). Works on virtuozzo versions <= 2.6.x. We really aren’t using this anymore now that we have netflow.<br />
<br />
== linuxtrafficwatch2.pl ==<br />
linuxtrafficwatch2.pl<br />
checks traffic usage and emails support@johncomapnies.com when a ve reaches the warning level (35G) and<br />
the limit (40G). Works on virtuozzo version 2.6.x. We really aren’t using this anymore now that we have netflow.<br />
<br />
== load.pl ==<br />
load.pl<br />
feeds info to load mrtg - executed by inetd.<br />
<br />
== mb (linux) ==<br />
mb <mount|umount><br />
(nfs) mounts and umounts dirs to backup2. Shortcuts are mbm and mbu to mount and unmount. <br />
<br />
== migrate ==<br />
migrate <ip of node migrating to> <veid> <target_veid> [target dir: vz | vz1 | vz2]<br />
this is basically a wrapper for <tt>vzmigrate</tt> – vzmigrate is a util to seamlessly move a ve from one host to another. This wrapper was written cause vz virtuozzo version 2.6 had a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables. This script mitigates that. Since it makes multiple ssh connections to the target host, it’s a good idea to put the pub key for the src system in the authorized_keys file on the target host. In addition, it emails ve owners when their migration starts and stops (if they place email addresses in a file on their system: /migrate_notify). To move everyone off a system, you’d do:<br />
for f in `vl`; do migrate <ip> $f; done<br />
<br />
== migrateonline ==<br />
migrateonline <ip of node migrating to> <veid> <target_veid> [target dir: vz | vz1 | vz2]<br />
this is the same as migrate but will migrate a ve in <tt>–online</tt> mode which means it won’t be shut down at the end of the migration. This only works when migrating ve’s between 2 machines running a 2.6 kernel (currently tempvirt1-2. virt16-19, virt12). If you get an error that the machine you’re trying to migrate to has a different CPU or features, etc, then you have to edit the file and add the –f switch to the vzmigrate line- you can basically ignore this kind of warning (but never ignore a warning about missing templates on the destination node). NOTE: This edit (if made to migrateonline) will be overwritten by the base script during each night’s backup.<br />
<br />
== netstatbackup ==<br />
DEPRECATED<br />
netstatbackup <br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.5.x. <br />
<br />
== netstatbackup2 ==<br />
DEPRECATED<br />
netstatbackup2<br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.6.x. <br />
<br />
== netstatreset ==<br />
DEPRECATED<br />
netstatreset<br />
writes traffic count data to a logfile and resets counters to 0. Works on virtuozzo versions 2.5.x <br />
<br />
== netstatreset2 ==<br />
DEPRECATED<br />
netstatreset2<br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.6.x. <br />
<br />
== orphanedbackupwatchlinux ==<br />
orphanedbackupwatchlinux <br />
looks for directories on backup2 which aren’t configured in backup.config and offers to <br />
delete them<br />
<br />
== rsync.backup (linux) ==<br />
rsync.backup<br />
does customer backups (relies on backup.config)<br />
<br />
== startvirt.pl ==<br />
startvirt.pl<br />
forks off start ve commands – keeps 6 running at a time. This is not to be used on systems where fastboot is enabled as it circumvents the benefit of the fastboot. The script will occasionally not exit gracefully and will continue to use up CPU, so it should be watched. Also, don’t exit from the script till you’re sure all ve’s are started – if you do you need to start them manually and may have to free up locks. On some systems, the startvirt script doesn’t exit cleanly and you have to ^C out of it. Be careful though- doing so can leave some VE’s in an odd bootup state and you may need to ‘vr’ them manually. You should check to see which ve’s aren’t running and/or confirm all have started when ^C’ing out of startvirt.<br />
<br />
== taskdone (linux) ==<br />
taskdone<br />
when called will email support@johncompanies.com with the hostname of the machine from which it was <br />
executed as the subject<br />
<br />
== vb (linux) ==<br />
vb<br />
the equivalent of: <tt>vi /usr/local/sbin/backup.config</tt><br />
<br />
== vemakeXX ==<br />
DEPRECATED<br />
vemakerh9 <br />
ve create script for RH9 (see vemake)<br />
<br />
vemakedebian30 <br />
ve create script for debian 3.0 (Woody) (see vemake)<br />
<br />
vemakedebian31 <br />
ve create script for debian 3.1 (Sarge) (see vemake)<br />
<br />
vemakedebian40 <br />
ve create script for debian 4.0 (Etch) (see vemake)<br />
<br />
vemakefedora, vemakefedora2, vemakefedora4, vemakefedora5, vemakefedora6, vemakefedora7<br />
ve create script for fedora core 1, 2, 4, 5, 6, 7 respectively (see vemake)<br />
<br />
vemakecentos3, vemakecentos4<br />
ve create script for centos 3, 4 respectively (see vemake)<br />
<br />
vemakesuse, vemakesuse93, vemakesuse100<br />
ve create script for suse 9.2, 9.3, 10.0 respectively (see vemake)<br />
<br />
vemakeubuntu5, vemakeubuntu606, vemakeubuntu606 vemakeubuntu704<br />
ve create script for ubuntu 5.10, 6.06, 6.10, 7.04 respectively (see vemake)<br />
<br />
== vemove ==<br />
DEPRECATED<br />
vemove <veid> <target_ip> </vz/private/123><br />
this script simplifies the old way of moving ve’s from one system to another - in short moving a ve to or from a virt running virtuozzo < 2.6.x<br />
It’s the equivalent of:<br />
<tt>tar cfpP - <veid> --ignore-failed-read | (ssh -2 -c arcfour <target_ip> "split - -b 1024m </vz/private/123>.tar" )</tt>This should only be used if migrate/vzmigrate can’t be used. <br />
<br />
== vim.watchdog ==<br />
vim.watchdog <br />
looks for and kills procs matching “vi|vim|nano|pine|elm” that are running for a long time & consuming lots of cpu. Works on virtuozzo versions 2.5.x<br />
<br />
== vim.watchdog2 ==<br />
vim.watchdog2<br />
looks for and kills procs matching “vi|vim|nano|pine|elm” that are running for a long time & consuming lots of cpu.<br />
Works on virtuozzo versions 2.6.x.<br />
<br />
== vzmigrate ==<br />
vzmigrate <target_ip> -r no <veid>:[dst veid]:[dst /vzX/private/veid]:[dst /vzX/root/veid]<br />
(this is the raw command “wrapped” by migrate/migrateonline) this will seamlessly move a ve from one host to another. The ve will run for the duration of the migration till the very end when it’s shut down, ip moved and started up on the target system. The filesystem on the src will remain. This should be watched – occasionally the move will timeout and leave the system shut down. If target private and root aren’t specified it just puts it in /vz. Only works when both systems are running virtuozzo 2.6.x<br />
<br />
== vztrafdump.sh ==<br />
DEPRECATED<br />
vztrafdump.sh<br />
writes traffic usage info by ve to a file called jc_traffic_dump in each ve’s / dir. Works on virtuozzo versions <= 2.5.x. <br />
<br />
== vztrafdump2.sh ==<br />
DEPRECATED<br />
vztrafdump2.sh<br />
writes traffic usage info by ve to a file called jc_traffic_dump in each ve’s / dir. Works on virtuozzo versions 2.6.x. <br />
<br />
== addtun ==<br />
addtun <veid><br />
Add’s tun device to ve.<br />
<br />
== bwcap ==<br />
bwcap <veid> <kbps><br />
Ex: <tt>bwcap 1234 512</tt><br />
Caps a VE’s bandwidth to the amount given<br />
<br />
== setdisk ==<br />
setdisk <veid> <diskspace in GB><br />
Ex: <tt>setdisk 1234 5</tt><br />
Gives a VE’s a given amount of disk space<br />
<br />
== vdf ==<br />
vdf <veid> <br />
the equivalent of: <tt>vzctl exec <veid> df –h</tt><br />
<br />
== vdff ==<br />
vdff<br />
runs a (condensed) vdf for all ve’s in your pwd (must be run from /vz/privateN)<br />
<br />
== mvbackups ==<br />
mvbackups <veid> <target_machine> (virt1) <target_dir> (vz1)<br />
moves backups from one location to another on the backup server, and provides you with option to remove entries from current backup.config, and simple paste command to add the config to backup.config on the target server<br />
<br />
== checkquota ==<br />
checkquota<br />
for all the ve’s in the cwd (run from /vz/private, /vz1/private, etc) reports what vz quota says they’re using and what the actual usage is (as reported by du)<br />
<br />
== clearquota ==<br />
clearquota <veid><br />
Recalculates a ve’s quota, prints out the usage before and after. The equivalent of:<br />
<tt>vdf <veid>; v stop <veid>; vzquota drop <veid>; v start <veid>; vdf <veid></tt><br />
<br />
== dprocs ==<br />
dprocs<br />
Sometimes the server’s have a large number of processes get stuck in the D state- this script shows (every 3 secs) which VE’s have D procs, which procs<br />
are stuck and a running average of the top “offenders”<br />
<br />
== vzstat ==<br />
vstat<br />
sort of like top for VZ. sort VEs by CPU usage by pressing 'o' and then 'c' keys<br />
<br />
== stopvirt ==<br />
stopvirt<br />
will stop VEs as fast as it can, 6 at a time. May not exit when complete so you should watch [[#vzstat|vzstat]] in another window.</div>Supporthttps://wiki.jcihosting.com/index.php?title=VPS_Management&diff=1034VPS Management2013-03-11T23:48:32Z<p>Support: /* Making new customer VE (vm) */</p>
<hr />
<div>= Common Problems =<br />
== Login to any machine without a password ==<br />
<br />
This is possible via the use of ssh keys. The process is thus:<br />
<br />
1. place the public key for your user (root@mail) in the /root/.ssh/authorized_keys file on the server you wish to login to<br />
cat /root/.ssh/id_dsa.pub<br />
(paste that into authorized_keys on the target server). If the file doesn't exist, create it.<br />
<br />
2. enable root login (usually only applies to FreeBSD). Edit the /etc/ssh/sshd_config on the target server and change:<br />
<tt>#PermitRootLogin no</tt><br />
to<br />
<tt>PermitRootLogin yes</tt><br />
<br />
3. Restart the sshd on the target machine. First, find the sshd process: <br />
jailps <hostname> | grep sshd <br />
or <br />
vp <VEID> | grep sshd<br />
<br />
Look for the process resembling:<br />
root 17296 0.0 0.0 5280 1036 ? Ss 2011 4:27 /usr/sbin/sshd <br />
(this is the sshd)<br />
<br />
Not:<br />
root 6270 0.5 0.0 6808 2536 ? Ss 14:33 0:00 sshd: root [priv]<br />
(this is an sshd child- someone already ssh'd in as root)<br />
<br />
Restart the sshd: <br />
kill -1 <PID><br />
<br />
Ex:<br />
kill -1 17296<br />
<br />
You may now ssh in.<br />
<br />
Once you're done, IF you enabled root login, you should repeat steps 2 and 3 to disable root logins.<br />
<br />
== Letting someone in who has locked themselves out (killed sshd, lost pwd) ==<br />
<br />
There are two ways people frequently lock themselves out - either they forget a password, or they kill off sshd somehow.<br />
<br />
These are actually both fairly easy to solve. First, let's say someone kills off their sshd, or somehow mangles /etc/ssh/sshd_config such that it no longer lets them in.<br />
<br />
Their email may be very short, or it may have all sorts of details about how you should fix sshd_config to let them in ... just ignore all of this. They can fix their own mangled sshd. Fixing this is very simple. First, edit the /etc/inetd.conf on their system and uncomment the telnet line:<br />
<br />
telnet stream tcp nowait root /usr/libexec/telnetd telnetd<br />
#telnet stream tcp6 nowait root /usr/libexec/telnetd telnetd<br />
<br />
(just leave the tcp6 version of telnet commented)<br />
<br />
Then, use jailps to list the processes on their system, and find their inetd process. Then simply:<br />
<br />
kill -HUP (pid)<br />
<br />
where (pid) is the PID of their inetd process. Now they have telnet running on their system and they can log in and do whatever they need to do.<br />
<br />
The only complications that could occur are:<br />
<br />
a) their firewall config on our firewall has port 23 blocked, in which case you will need to open that - will be covered in a different lesson.<br />
<br />
b) they are not running inetd, so you can't HUP it. If this happens, edit their /etc/rc.conf, add the inetd_enable="YES" line, and then kill<br />
their jail with /tmp/jailkill.pl - then restart their jail with the jail line from their quad/safe file. Easy.<br />
<br />
If they have forgotten a password,<br />
<br />
On 6.x+ you can reset their password with:<br />
jexec <jailID from jls> passwd root<br />
<br />
Note: the default password for 6.x jails is 8ico2987, for 4.x it is p455agfa<br />
<br />
On 4.x, you need to cd to their etc directory<br />
... for instance:<br />
<br />
cd /mnt/data2/198.78.65.136-col00261-DIR/etc<br />
<br />
and run:<br />
<br />
vipw -d .<br />
<br />
Then paste in these two lines (theres a paste with these):<br />
<br />
root:$1$krszPxhk$xkCepSnz3mIikT3vCtJCt0:0:0::0:0:Charlie &:/root:/bin/csh<br />
user:$1$Mx9p5Npk$QdMU6c8YQqp2FW2M3irEh/:1001:1001::0:0:User &:/home/user:/bin/sh<br />
<br />
overwriting the lines they already have for "user" and "root" - then just tell them that both user and root have been reset to the default password of p455agfa.<br />
<br />
For linux, just passwd inside shell or <br />
vzctl set <veid> --userpasswd root:p455agfa –save<br />
<br />
Starting in 2009 we began giving out randomized passwords for FreeBSD and Linux as the default password. That is stored with each system in Mgmt. You should look for and reset the password to that password in the event of a reset and refer the customer to use their original password from their welcome email- this way we don’t have to send the password again via email (in clear text).<br />
<br />
<br />
== sendmail can’t be contacted from ext ip (only locally) ==<br />
<br />
By default redhat puts this line in sendmail.mc:<br />
<br />
DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')<br />
<br />
which makes it only answer on localhost. Comment it out like:<br />
<br />
dnl DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')<br />
<br />
and then rebuild sendmail.cf with:<br />
<br />
m4 /etc/mail/sendmail.mc > /etc/sendmail.cf<br />
<br />
== virt doesn’t properly let go of ve’s ip(s) when moved to another system ==<br />
<br />
On virtuozzo 2.6 systems, it's been observed that when moving ips from one virt to another that sometimes the routing table will not get updated to reflect the removal of the ip addresses.<br />
<br />
A recent example was a customer that was moving to a new ve on a new virt and the ip addresses were traded between the two ve's. After the trade the two systems were not able to talk to each other. When looking at the routing table for the old system all the ip addresses were still in the routing table as being local, like so:<br />
<br />
<pre>netstat -rn | grep 69.55.225.149<br />
69.55.225.149 0.0.0.0 255.255.255.255 UH 40 0 0 venet0</pre><br />
<br />
This was preventing traffic to the other system from being routed properly.<br />
The solution is to manually delete the route:<br />
<br />
route delete 69.55.225.149 gw 0.0.0.0<br />
<br />
Supposedly, this was fixed in 2.6.1<br />
<br />
== sshd on FreeBSD 6.2 segfaults ==<br />
<br />
First try to reinstall ssh<br />
<br />
<pre>cd /usr/src/secure<br />
cd lib/libssh<br />
make depend && make all install<br />
cd ../../usr.sbin/sshd<br />
make depend && make all install<br />
cd ../../usr.bin/ssh<br />
make depend && make all install</pre><br />
<br />
Failing that, find the library that’s messed up:<br />
<br />
<pre>ldd /usr/sbin/sshd<br />
libssh.so.3 => /usr/lib/libssh.so.3 (0x280a3000) <br />
libutil.so.5 => /lib/libutil.so.5 (0x280d8000) <br />
libz.so.3 => /lib/libz.so.3 (0x280e4000) <br />
libwrap.so.4 => /usr/lib/libwrap.so.4 (0x280f5000) <br />
libpam.so.3 => /usr/lib/libpam.so.3 (0x280fc000) <br />
libbsm.so.1 => /usr/lib/libbsm.so.1 (0x28103000) <br />
libgssapi.so.8 => /usr/lib/libgssapi.so.8 (0x28112000) <br />
libkrb5.so.8 => /usr/lib/libkrb5.so.8 (0x28120000) <br />
libasn1.so.8 => /usr/lib/libasn1.so.8 (0x28154000) <br />
libcom_err.so.3 => /usr/lib/libcom_err.so.3 (0x28175000) <br />
libroken.so.8 => /usr/lib/libroken.so.8 (0x28177000) <br />
libcrypto.so.4 => /lib/libcrypto.so.4 (0x28183000) <br />
libcrypt.so.3 => /lib/libcrypt.so.3 (0x28276000) <br />
libc.so.6 => /lib/libc.so.6 (0x2828e000) <br />
libmd.so.3 => /lib/libmd.so.3 (0x28373000)</pre><br />
<br />
md5 them and compare to other jail hosts or jails running on host<br />
<br />
for libcrypto reinstall:<br />
<pre>/usr/src/crypto<br />
make depend && make all install</pre><br />
<br />
= FreeBSD VPS =<br />
<br />
== Starting jails: Quad/Safe Files ==<br />
<br />
FreeBSD customer systems do not start up automatically at boot time. When one of our freebsd machines boots up, it boots up, and does nothing else. To start jails, we put the commands to start each jail into a shell script(s) and run the script(s). Jail startup is something that needs to be actively monitored, which is why we don’t just run the script automatically. More on monitoring later.<br />
<br />
NOTE: >=7.x we have moved to 1 quad file: <tt>quad1</tt>. Startups are not done by running each quad, but rather [[#startalljails|startalljails]] which relies on the contents of <tt>quad1</tt>. The specifics of this are lower in this article. What follows here applies for pre 7.x systems.<br />
<br />
There are eight files in <tt>/usr/local/jail/rc.d</tt>:<br />
<br />
<pre>jail3# ls /usr/local/jail/rc.d/<br />
quad1 quad2 quad3 quad4 safe1 safe2 safe3 safe4<br />
jail3#</pre><br />
<br />
four quad files and four safe files.<br />
<br />
Each file contains an even number of system startup blocks (total number of jails divided by 4)<br />
<br />
The reason for this is, if we make one large script to startup all the systems at boot time, it will take too long - the first system in the script will start up right after system boot, which is great, but the last system may not start for another 20 minutes.<br />
<br />
Since there is no way to parralelize this during the startup procedure, we simply open four terminals (in screen window 9) and run each script, one in each terminal. This way they all run simultaneously, and the very last system in each startup script gets started in 1/4th the time it would if there was one large file<br />
<br />
The files are generally organized so that quad/safe 1&2 have only jails from disk 1, and quad/safe 3&4 have jails from disk 2. This helps ensure that only 2 fscks on any disk are going on at once. Further, they are balanced so that all quad/safe’s finish executing around the same time. We do this by making sure each quad/safe has a similar number of jails and represents a similar number of inodes (see js).<br />
<br />
The other, very important reason we do it this way, and this is the reason there are quad files and safe files, is that in the event of a system crash, every single vn-backed filesystem that was mounted at the time of system crash needs to be fsck'd. However, fsck'ing takes time, so if we shut the system down gracefully, we don't want to fsck.<br />
<br />
Therefore, we have two sets of scripts - the four quad scripts are identical to the four safe scripts except for the fact that the quad scripts contain fsck commands for each filesystem.<br />
<br />
So, if you shut a system down gracefully, start four terminals and run safe1 in window one, and safe2 in window 2, and so on.<br />
<br />
If you crash, start four terminals (or go to screen window 9) and run quad1 in window one, and quad2 in window 2, and so on.<br />
<br />
Here is a snip of (a 4.x version) quad2 from jail17:<br />
<br />
<pre>vnconfig /dev/vn16 /mnt/data2/69.55.228.7-col00820<br />
fsck -y /dev/vn16<br />
mount /dev/vn16c /mnt/data2/69.55.228.7-col00820-DIR<br />
chmod 0666 /mnt/data2/69.55.228.7-col00820-DIR/dev/null<br />
jail /mnt/data2/69.55.228.7-col00820-DIR mail1.phimail.com 69.55.228.7 /bin/sh /etc/rc<br />
<br />
# moved to data2 col00368<br />
#vnconfig /dev/vn28 /mnt/data2/69.55.236.132-col00368<br />
#fsck -y /dev/vn28<br />
#mount /dev/vn28c /mnt/data2/69.55.236.132-col00368-DIR<br />
#chmod 0666 /mnt/data2/69.55.236.132-col00368-DIR/dev/null<br />
#jail /mnt/data2/69.55.236.132-col00368-DIR limehouse.org 69.55.236.132 /bin/sh /etc/rc<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
vnconfig /dev/vn22 /mnt/data2/69.55.228.13-col01063<br />
fsck -y /dev/vn22<br />
mount /dev/vn22c /mnt/data2/69.55.228.13-col01063-DIR<br />
chmod 0666 /mnt/data2/69.55.228.13-col01063-DIR/dev/null<br />
jail /mnt/data2/69.55.228.13-col01063-DIR www.widestream.com.au 69.55.228.13 /bin/sh /etc/rc<br />
<br />
# cancelled col00106<br />
#vnconfig /dev/vn15 /mnt/data2/69.55.238.5-col00106<br />
#fsck -y /dev/vn15<br />
#mount /dev/vn15c /mnt/data2/69.55.238.5-col00106-DIR<br />
#chmod 0666 /mnt/data2/69.55.238.5-col00106-DIR/dev/null<br />
#jail /mnt/data2/69.55.238.5-col00106-DIR mail.azebu.net 69.55.238.5 /bin/sh /etc/rc</pre><br />
<br />
As you can see, two of the systems specified are commented out - presumably those customers cancelled, or were moved to new servers.<br />
<br />
As you can see, the vnconfig line is the simpler command line, not the longer one that was used when it was first configured. As you can see, all that is done is, vnconfig the filesystem, then fsck it, then mount it. The fourth command is the `jail` command used to start the system – but that will be covered later.<br />
<br />
Here is the safe2 file from jail17:<br />
<br />
<pre>vnconfig /dev/vn16 /mnt/data2/69.55.228.7-col00820<br />
mount /dev/vn16c /mnt/data2/69.55.228.7-col00820-DIR<br />
chmod 0666 /mnt/data2/69.55.228.7-col00820-DIR/dev/null<br />
jail /mnt/data2/69.55.228.7-col00820-DIR mail1.phimail.com 69.55.228.7 /bin/sh /etc/rc<br />
<br />
# moved to data2 col00368<br />
#vnconfig /dev/vn28 /mnt/data2/69.55.236.132-col00368<br />
#mount /dev/vn28c /mnt/data2/69.55.236.132-col00368-DIR<br />
#chmod 0666 /mnt/data2/69.55.236.132-col00368-DIR/dev/null<br />
#jail /mnt/data2/69.55.236.132-col00368-DIR limehouse.org 69.55.236.132 /bin/sh /etc/rc<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
vnconfig /dev/vn22 /mnt/data2/69.55.228.13-col01063<br />
mount /dev/vn22c /mnt/data2/69.55.228.13-col01063-DIR<br />
chmod 0666 /mnt/data2/69.55.228.13-col01063-DIR/dev/null<br />
jail /mnt/data2/69.55.228.13-col01063-DIR www.widestream.com.au 69.55.228.13 /bin/sh /etc/rc<br />
<br />
# cancelled col00106<br />
#vnconfig /dev/vn15 /mnt/data2/69.55.238.5-col00106<br />
#mount /dev/vn15c /mnt/data2/69.55.238.5-col00106-DIR<br />
#chmod 0666 /mnt/data2/69.55.238.5-col00106-DIR/dev/null<br />
#jail /mnt/data2/69.55.238.5-col00106-DIR mail.azebu.net 69.55.238.5 /bin/sh /etc/rc</pre><br />
<br />
As you can see, it is exactly the same, but it does not have the fsck lines.<br />
<br />
Take a look at the last entry - note that the file is named:<br />
<br />
/mnt/data2/69.55.238.5-col00106<br />
<br />
and the mount point is named:<br />
<br />
/mnt/data2/69.55.238.5-col00106-DIR<br />
<br />
This is the general format on all the FreeBSD systems. The file is always named:<br />
<br />
IP-custnumber<br />
<br />
and the directory is named:<br />
<br />
IP-custnumber-DIR<br />
<br />
If you run safe when you need a fsck, the mount will fail and jail will fail:<br />
<br />
# mount /dev/vn1c /mnt/data2/jails/65.248.2.131-ns1.kozubik.com-DIR<br />
mount: /dev/vn1c: Operation not permitted<br />
<br />
No reboot needed, just run the quad script<br />
<br />
Starting with 6.x jails, we added block delimiters to the quad/safe files, the block looks like:<br />
<br />
<pre>echo '## begin ##: nuie.solaris.mu'<br />
fsck -y /dev/concat/v30v31a<br />
mount /dev/concat/v30v31a /mnt/data1/69.55.228.218-col01441-DIR<br />
mount_devfs devfs /mnt/data1/69.55.228.218-col01441-DIR/dev<br />
devfs -m /mnt/data1/69.55.228.218-col01441-DIR/dev rule -s 3 applyset<br />
jail /mnt/data1/69.55.228.218-col01441-DIR nuie.solaris.mu 69.55.228.218 /bin/sh /etc/rc<br />
echo '## end ##: nuie.solaris.mu'</pre><br />
<br />
These are more than just informative when running quad/safe’s, the echo lines MUST be present for certain tools to work properly. So it’s important that any updates to the hostname also be updated on the 2 echo lines. For example, if you try to startjail a jail with a hostname which is on the jail line but not the echo lines, the command will return with host not found.<br />
<br />
=== FreeBSD 7.x+ notes ===<br />
<br />
Starting with the release of FreeBSD 7.x, we are doing jail startups in a slightly different way. First, thereis only 1 file: <tt>/usr/local/jail/rc.d/quad1</tt><br />
There are no other quads or corresponding safe files. The reason for this is twofold, 1. We can pass –C to fsck which will tell is to skip the fsck if the fs is clean (no more need for safe files), 2. We have a new startup script which can be launched multiple times, running in parallel to start jails, where quad1 is the master jail file. <br />
Quad1 could still be run as a shell script, but it would take a very long time for it to run completely so it’s not advisable; or you should break it down into smaller chunks (like quad1, quad2, quad3, etc)<br />
<br />
Here is a snip of (a 7.x version) quad1 from jail2:<br />
<br />
<pre>echo '## begin ##: projects.tw.com'<br />
mdconfig -a -t vnode -f /mnt/data1/69.55.230.46-col01213 -u 50<br />
fsck -Cy /dev/md50c<br />
mount /dev/md50c /mnt/data1/69.55.230.46-col01213-DIR<br />
mount -t devfs devfs /mnt/data1/69.55.230.46-col01213-DIR/dev<br />
devfs -m /mnt/data1/69.55.230.46-col01213-DIR/dev rule -s 3 applyset<br />
jail /mnt/data1/69.55.230.46-col01213-DIR projects.tw.com 69.55.230.46 /bin/sh /etc/rc<br />
echo '## end ##: projects.tw.com'</pre><br />
<br />
Cancelled jails are no longer commented out and stored in quad1, rather they’re moved to <tt>/usr/local/jail/rc.d/deprecated</tt> <br />
<br />
To start these jails, start the 4 ssh sessions as you would for a normal crash and then instead of running quad1-4, instead run startalljails in each window. IMPORTANT- before running startalljails you should make sure you ran preboot once as it will clear out all the lockfiles and enable startalljails to work properly.<br />
<br />
== Problems with the quad/safe files ==<br />
<br />
When you run the quad/safe files, there are two problems that can occur - either a particular system will hang during initialization, OR a system will spit out output to the screen, impeding your ability to do anything. Or both.<br />
<br />
First off, when you start a jail, you see output like this:<br />
<br />
<pre>Skipping disk checks ...<br />
adjkerntz[25285]: sysctl(put_wallclock): Operation not permitted<br />
Doing initial network setup:.<br />
ifconfig: ioctl (SIOCDIFADDR): permission denied<br />
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384<br />
Additional routing options: TCP keepalive=YESsysctl:<br />
net.inet.tcp.always_keepalive: Operation not permitted.<br />
Routing daemons:.<br />
Additional daemons: syslogd.<br />
Doing additional network setup:.<br />
Starting final network daemons:.<br />
ELF ldconfig path: /usr/lib /usr/lib/compat /usr/X11R6/lib /usr/local/lib<br />
a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout /usr/X11R6/lib/aout<br />
Starting standard daemons: inetd cron sshd sendmail sendmail-clientmqueue.<br />
Initial rc.i386 initialization:.<br />
Configuring syscons: blanktime.<br />
Additional ABI support:.<br />
Local package initialization:.<br />
Additional TCP options:.</pre><br />
<br />
Now, let's look at this line, near the end:<br />
<br />
Local package initialization:.<br />
<br />
This is where a list of daemons that are set to start at boot time willshow up. You might see something like:<br />
<br />
Local package initialization: mysqld apache sendmail sendmail-clientmqueue<br />
<br />
Or something like this:<br />
<br />
Local package initialization: postgres postfix apache<br />
<br />
The problem is that many systems (about 4-5 per machine) will hang on that line. Basically it will get to some of the way through the total daemons to be started:<br />
<br />
Local package initialization: mysqld apache<br />
<br />
and will just sit there. Forever.<br />
<br />
Fortunately, pressing ctrl-c will break out of it. Not only will it break out of it, but it will also continue on that same line and start the other daemons:<br />
<br />
Local package initialization: mysqld apache ^c sendmail-clientmqueue<br />
<br />
and then continue on to finish the startup, and then move to the next system to be started.<br />
<br />
So what does this mean? It means that if a machine crashes, and you start four screen-windows to run four quads or four safes, you need to periodically cycle between them and see if any systems are stuck at that point, causing their quad/safe file to hang. A good rule of thumb is, if you see a system at that point in the startup, give it another 100 seconds - if it is still at the exact same spot, hit ctrl-c. Its also a good idea to go back into the quad file (just before the first command in the jail startup block) and note that this jail tends to need a control-c or more time as follows:<br />
<pre>echo '### NOTE ### slow sendmail'<br />
echo '### NOTE ###: ^C @ Starting sendmail.'</pre><br />
<br />
'''NEVER''' hit ctrl-c repeatedly if you don't get an immediate response - that will cause the following jail’s startup commands to be aborted.<br />
<br />
A second problem that can occur is that a jail - maybe the first one in that particular quad/safe, maybe the last one, or maybe one in the middle, will start spitting out status or error messages from one of its init scripts. This is not a problem - basically, hit enter a few times and see if you get a prompt - if you do get a prompt, that means that the quad/safe script has already completed. Therefore it is safe to log out (and log out of the user that you su'd from) and then log back in (if necessary).<br />
<br />
The tricky thing is, if a system in the middle starts flooding with messages, and you hit enter a few times and don't get a prompt. Are you not getting a prompt because some subsequent system is hanging at the initialization, as we discussede above ? Or are you not getting a prompt because that quad file is currently running an fsck ? Usually you can tell by scrolling back in screen’s history to see what it was doing before you started getting the messages.<br />
<br />
If you don’t get clues from history, you have to use your judgement - instead of giving it 100 seconds to respond, perhaps give it 2-3 mins ... if you still get no response (no prompt) when you hit enter, hit ctrl-c. However, be aware that you might still be hitting ctrl-c in the middle of an fsck. This means you will get an error like "filesystem still marked dirty" and then the vnconfig for it will fail and so will the jail command, and the next system in the quad file will then start starting up.<br />
<br />
If this happens, just wait until the end of all the quad files have finished, and start that system manually.<br />
<br />
If things really get weird, like a screen flooded with errors, and you can't get a prompt, and ctrl-c does nothing, then you need to just eventually (give it ten mins or so) just kill that window with ctrl-p, then k, and then log in again and manually check which systems are now running and which aren't, and manually start up any that are not.<br />
<br />
Don't EVER risk running a particular quad/safe file a second time.<br />
If the quad/safe script gets executed twice, reboot the machine immediately.<br />
<br />
So, for all the above reasons, anytime a machine crashes and you run all the quads or all the safes, '''always''' check every jail afterwards to make sure it is running - even if you have no hangs or complications at all.<br />
Run this command:<br />
<br />
<tt>[[#jailpsall|jailpsall]]</tt> <br />
<br />
Note: [[#postboot|postboot]] also populates ipfw counts, so it '''should not be run multiple times''', use <tt>jailpsall</tt> for subsequent extensive ps’ing<br />
<br />
And make sure they all show as running. If one does not show as running, check its /etc/rc.conf file first to see if maybe it is using a different hostname first before starting it manually.<br />
<br />
One thing we have implemented to alleviate these startup hangs and noisy jails, is to put jail start blocks that are slow or hangy at the bottom of the safe/quad file. Further, for each bad jail we note in each quad/safe just before the start block something like:<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
<br />
That way we’ll be prepared to ^C when we see that message appear during the quad/safe startup process. If you observe a new, undocumented hang, '''after''' the quad/safe has finished, place a line similar to the above in the quad file, move the jail start block to the end of the file, then run [[#buildsafe|buildsafe]]<br />
<br />
<br />
== Recovering from a crash (FreeBSD) ==<br />
<br />
=== Diagnose whether you have a crash ===<br />
The most important thing is to get the machine and all jails back up as soon as possible. Note the time, you’ll need to create a crash log entry (Mgmt. -> Reference -> CrashLog). The first thing to do is head over to the [[Screen#Screen_Organization|serial console screen]] and see if there’s any kernel error messages output. Try to copy any messages (or just a sample of repeating messages) you see into the notes section of the crash log. If there are no messages, the machine may just be really busy- wait a bit (5-10min) to see if it comes back. If it's still pinging, odds are its very busy. Note, if you see messages about swap space exhausted, the server is obviously out of memory, however it may recover briefly enough for you to get a jtop in to see who's lauched a ton of procs (most likely) and then issue a quick jailkill to get it back under control.<br />
<br />
If it doesn't come back, or the messages indicate a fatal error, you will need to proceed with a power cycle (ctrl+alt+del will not work).<br />
<br />
=== Power cycle the server ===<br />
If this machine is not a Dell 2950 with a [[DRAC/RMM#DRAC|DRAC card]] (i.e. if you can’t ssh into the DRAC card (as root, using the standard root pass) and issue <br />
racadm serveraction hardreset<br />
then you will need someone at the data center power the macine off, wait 30 sec, then turn it back on. Make sure to re-attach via console:<br />
tip jailX<br />
immediately after power down. <br />
<br />
=== (Re)attach to the console ===<br />
Stay on the console the entire time during boot. As the BIOS posts- look out for the RAID card output- does everything look healthy? The output may be scrambled, look for "DEGRADED" or "FAILED". Once the OS starts booting you will be disconnected (dropped back to the shell on the console server) a couple times during the boot up. The reason you want to quickly re-attach is two-fold: 1. If you don’t reattach quickly then you won’t get any console output, 2. you want to be attached before the server ''potentially'' starts (an extensive) fsck. If you attach after the fsck begins, you’ll have seen no indication it started an fsck and the server will appear frozen during startup- no output, no response. <br />
<br />
IMPORTANT NOTE: on some older FreeBSD systems, there will be no output to the video (KVM) console as it boots up. The console output is redirected to the serial port ... so if a jail crashes, and you attach a kvm, the output during the bootup procedure will not be shown on the screen. However, when the bootup is done, you will get a login prompt on the screen and will be able to log in as normal. <tt>/boot/loader.conf</tt> is where serial console redirect output lives, so comment that if you want to catch output on kvm.<br />
On newer systems it sends most output to both locations. <br />
<br />
=== Assess the heath of the server ===<br />
Once the server boots up fully, you should be able to ssh in. Look around- make sure all the mounts are there and reporting the correct size/usage (i.e. /mnt/data1 /mnt/data2 /mnt/data3 - look in /etc/fstab to determine which mount points should be there), check to see if RAID mirrors are healthy. See [[RAID_Cards#Common_CLI_commands_.28megacli.29|megacli]], [[#aaccheck|aaccheck]]<br />
<br />
Before you start the jails, you need to run [[#preboot|preboot]]. This will do some assurance checks to make sure things are prepped to start the jails. Any issues that come out of preboot need to be addressed before starting jails.<br />
<br />
=== Start jails ===<br />
[[#Starting_jails:_Quad.2FSafe_Files|More on starting jails]]<br />
Customer jails (the VPSs) do not start up automatically at boot time. When a FreeBSD machines boots up, it boots up, and does nothing else. To start jails, we put the commands to start each jail into a shell script(s) and run the script(s). Jail startup is something that needs to be actively monitored, which is why we don’t just run the script automatically. <br />
<br />
In order to start jails, we run the quad files: quad1 quad2 quad3 and quad4 (on new systems there is only quad1). If the machine was cleanly rebooted- which wouldn't be the case if this was a crash, you may run the safe files (safe1 safe2 safe3 safe4) in lieu of quads. <br />
<br />
Open up 4 logins to the server (use the windows in [[Screen#Screen_Organization|a9]])<br />
In each of the 4 windows you will:<br />
<br />
If there is a [[#startalljails|startalljails]] script (and only quad1), run that command in each of the 4 windows. It will parse through the quad1 file and start each jail. Follow the instructions [[#Problems_with_the_quad.2Fsafe_files|here]] for monitoring startup. Note that you can be a little more lenient with jails that take awhile to start- startalljails will work around the slow jails and start the rest. As long as there aren't 4 jails which are "hung" during startup, the rest will get started eventually.<br />
-or-<br />
If there is no startalljails script, there will be multiple quad files. In each of the 4 windows, start each of the quads. i.e. start quad1 in window1, quad2 in window2 and so on. DO NOT start any quad twice. It will crash the server. If you accidentally do this, just jailkill all the jails which are in the quad and run the quad again. Follow the instructions here for monitoring quad startup.<br />
<br />
Note the time the last jail boots- this is what you will enter in the crash log.<br />
<br />
Save the crash log.<br />
<br />
=== Check to make sure all jails have started ===<br />
There's a simple script which will make sure all jails have started, and enter the ipfw counter rules: [[#postboot|postboot]] <br />
Run postboot, which will do a jailps on each jail it finds (excluding commented out jails) in the quad file(s). We're looking for 2 things:<br />
# systems spawning out of control or too many procs<br />
# jails which haven't started<br />
On 7.x and newer systems it will print out the problems (which jails haven't started) at the conclusion of postboot. <br />
On older systems you will need to watch closely to see if/when there's a problem, namely:<br />
<br />
[hostname] doesnt exist on this server<br />
<br />
When you get this message, it means one of 2 things:<br />
1. the jail really didn't start:<br />
When a jail doesn't start it usually boils down to a problem in the quad file. Perhaps the path name is wrong (data1 vs data2) or the name of the vn/mdfile is wrong. Once this is corrected, you will need to run the commands from the quad file manually, or you may use <tt>startjail <hostname></tt><br />
<br />
2. the customer has changed their hostname (and not told us) so their jail ''is'' running, just under a different hostname:<br />
On systems with jls, this is easy to rectify. First, get the customer info: <tt>g <hostname></tt><br />
Then look for the customer in jls: <tt>jls | grep <col0XXXX></tt><br />
From there you will see their new hostname- you should update that hostname in the quad file: don't forget to edit it on the <tt>## begin ##</tt> and <tt>## end ##</tt> lines, and in mgmt. <br />
On older systems without jls, this will be harder, you will need to look further to see their hostname- perhaps its in their /etc/rc.conf<br />
<br />
<br />
Once all jails are started, do some spot checks- try to ssh or browse to some customers, just to make sure things are really ok.<br />
<br />
== Adding disk to a 7.x/8.x jail ==<br />
or<br />
== Moving customer to a different drive (md) ==<br />
<br />
NOTE: this doesn’t apply to mx2 which uses gvinum. Use same procedure as 6.x<br />
NOTE: if you unmount before mdconfig, re-mdconfig (attach) then unmount then mdconfig -u again <br />
<br />
<br />
(parts to change/customize are <tt><span style="color:red">red</span></tt>)<br />
<br />
<br />
If someone wants more disk space, there’s a paste for it, send it to them (explains about downtime, etc).<br />
<br />
1. Figure out the space avail from <tt>js</tt>. Ideally, you want to put the customers new space on a different partition (and create the new md on the new partition). <br />
<br />
<br />
2. make a mental note of how much space they're currently using<br />
<br />
<br />
3. <tt>jailkill <hostname></tt> <br />
<br />
<br />
4. Umount it (including their devfs) but leave the md config’d (so if you use stopjail, you will have to re-mdconfig it)<br />
<br />
<br />
5. <tt>g <customerID></tt> to get the info (IP/cust#) needed to feed the new mdfile and mount name, and to see the current md device. <br />
<br />
<br />
6a. When there's enough room to place new system on an alternate, or the same drive:<br />
USE CAUTION not to overwrite (touch, mdconfig) existing md!!<br><br />
<tt>touch /mnt/data<span style="color:red">3/69.55.234.66-col01334</span><br><br />
mdconfig -a -t vnode -s 10g -f /mnt/data3/69.55.234.66-col01334 -u 97<br><br />
newfs /dev/md97</tt><br />
<br />
<br />
Optional- if new space is on a different drive, move the mount point directory AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
<tt>mount /dev/md<span style="color:red">97</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<br />
confirm you are mounted to /dev/md<span style="color:red">97</span> and space is correct:<br><br />
<tt>df .</tt> <br />
<br />
<br />
do the dump and pipe directly to restore:<br><br />
<tt>dump -0a -f - /dev/md<span style="color:red">1</span> | restore -r -f - <br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use <tt>df</tt> to confirm the restored data size matches the original usage figure<br />
<br />
<br />
md-unconfig old system:<br><br />
<tt>mdconfig -d -u <span style="color:red">1</span></tt><br />
<br />
<br />
archive old mdfile. <br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.237.26-col00241</span> /mnt/data<span style="color:red">1/old-col00241-mdfile-noarchive-20091211</span></tt><br />
<br />
<br />
edit quad (vq1) to point to new (/mnt/data<span style="color:red">3</span>) location AND new md number (md<span style="color:red">97</span>)<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
6b. When there's not enough room on an alternate partition, or on the same drive...but there is enough room if you were to remove the existing customer's space:<br />
<br />
<br />
mount backup nfs mounts:<br><br />
<tt>mbm</tt><br> <br />
(run <tt>df</tt> to confirm backup mounts are mounted)<br />
<br />
<br />
dump the customer to backup2 or backup1<br><br />
<tt>dump -0a -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> /dev/md<span style="color:red">1</span></tt><br><br />
(when complete WITHOUT errors, <tt>du</tt> the dump file to confirm it matches size, roughly, with usage)<br />
<br />
<br />
unconfigure and remove old mdfile<br><br />
<tt>mdconfig -d -u <span style="color:red">1</span><br><br />
rm /mnt/data<span style="color:red">1/69.55.237.26-col00241</span></tt><br><br />
(there should now be enough space to recreate your bigger system. If not, run sync a couple times)<br />
<br />
<br />
create the new system (ok to reuse old mdfile and md#):<br><br />
<tt>touch /mnt/data<span style="color:red">1/69.55.234.66-col01334</span><br><br />
mdconfig -a -t vnode -s <span style="color:red">10</span>g -f /mnt/data<span style="color:red">1/69.55.234.66-col01334</span> -u <span style="color:red">1</span><br><br />
newfs /dev/md<span style="color:red">1</span><br><br />
mount /dev/md<span style="color:red">1</span> /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> <br><br />
cd /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<br />
confirm you are mounted to /dev/md<span style="color:red">1</span> and space is correct:<br><br />
<tt>df .</tt> <br />
<br />
<br />
do the restore from the dumpfile on the backup server:<br><br />
<tt>restore -r -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> .<br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
umount nfs:<br><br />
<tt>mbu</tt><br />
<br />
<br />
If md# changed (or mount point), edit quad (<tt>vq1</tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new md number (md<span style="color:red">1</span>)<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
7. update disk (and dir if applicable) in mgmt screen<br />
<br />
<br />
8. update backup list AND move backups, if applicatble<br />
<br />
Ex: <tt>mvbackups <span style="color:red">69.55.237.26-col00241</span> jail<span style="color:red">9</span> data<span style="color:red">3</span></tt><br />
<br />
<br />
9. Optional: archive old mdfile<br />
<br />
<tt>mbm<br><br />
gzip -c old-col01588-mdfile-noarchive-20120329 > /deprecated/old-col01588-mdfile-noarchive-20120329.gz<br><br />
mbu<br><br />
rm old-col01588-mdfile-noarchive-20120329</tt><br />
<br />
== Adding disk to a 6.x jail (gvinum/gconcat) ==<br />
or<br />
== Moving customer to a different drive (gvinum/gconcat) ==<br />
<br />
(parts to change are <span style="color:red">highlighted</span>)<br />
<br />
<br />
If someone wants more disk space, there’s a paste for it, send it to them (explains about downtime, etc).<br />
<br />
<br />
1. Figure out the space avail from [[#js|js]]. Ideally, you want to put the customers new space on a different partition (and create the new md on the new partition). <br />
<br />
<br />
2. make a mental note of how much space they're currently using<br />
<br />
<br />
3. <tt>[[#stopjail|stopjail]] <hostname></tt> <br />
<br />
<br />
4. <tt>[[#g|g]] <customerID></tt> to get the info (IP/cust#) needed to feed the new mount name and existing volume/device. <br />
<br />
<br />
5a. When there's enough room to place new system on an alternate, or the same drive (using only UNUSED - including if it's in use by the system in question - gvinum volumes):<br />
<br />
<br />
Configure the new device:<br><br />
A. for a 2G system (single gvinum volume):<br><br />
<tt>bsdlabel -r -w /dev/gvinum/v<span style="color:red">123</span><br><br />
newfs /dev/gvinum/v<span style="color:red">123</span>a</tt><br><br />
<br />
-or- <br />
<br />
<br />
B. for a >2G system (create a gconcat volume):<br><br />
try to grab a contiguous block of gvinum volumes. gconcat volumes MAY NOT span drives (i.e. you cannot use a gvinum volume from data3 and a volume from data2 in the same gconcat volume). <br />
<br />
<tt>gconcat label <span style="color:red">v82-v84 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84</span><br><br />
bsdlabel -r -w /dev/concat/<span style="color:red">v82-v84</span><br><br />
newfs /dev/concat/<span style="color:red">v82-v84</span>a</tt><br />
<br />
<br />
Other valid gconcat examples:<br><br />
<tt>gconcat label v82-v84v109v112 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84 /dev/gvinum/v109 /dev/gvinum/v112<br><br />
gconcat label v82v83 /dev/gvinum/v82 /dev/gvinum/v83</tt> <br><br />
Note, long names will truncate: v144v145v148-v115 will truncate to v144v145v148-v1 (so you will refer to it as v144v145v148-v1 thereafter)<br />
<br />
<br />
Optional- if new volume is on a different drive, move the mount point directory (get the drive from js output) AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
<br />
confirm you are mounted to the device (<tt>/dev/gvinum/v<span style="color:red">123</span>a</tt> OR <tt>/dev/concat/<span style="color:red">v82-v84</span></tt>) and space is correct:<br><br />
A. <tt>mount /dev/gvinum/v<span style="color:red">123</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br><br />
-or-<br><br />
B. <tt>mount /dev/concat/<span style="color:red">v82-v84</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<tt>cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
df .</tt> <br />
<br />
<br />
do the dump and pipe directly to restore:<br><br />
<tt>dump -0a -f - /dev/gvinum/v<span style="color:red">1</span> | restore -r -f - <br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
edit quad (<tt>vq<span style="color:red">1</span></tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new volume (<tt>/dev/gvinum/v<span style="color:red">123</span>a</tt> or <tt>/dev/concat/<span style="color:red">v82-v84</span>a</tt>) , run <tt>buildsafe</tt><br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
5b. When there's not enough room on an alternate partition, or on the same drive...but there is enough room if you were to remove the existing customer's space (i.e. if you want/need to reuse the existing gvinum volumes and add on more):<br />
<br />
<br />
mount backup nfs mounts:<br><br />
<tt>mbm</tt> <br><br />
(run df to confirm backup mounts are mounted)<br />
<br />
<br />
dump the customer to backup2 or backup1<br><br />
<tt>dump -0a -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> /dev/<span style="color:red">gconcat/v106-v107</span></tt><br><br />
(when complete WITHOUT errors, du the dump file to confirm it matches size, roughly, with usage)<br />
<br />
<br />
unconfigure the old gconcat volume<br><br />
list member gvinum volumes:<br />
<br />
<tt>gconcat list <span style="color:red">v106v107</span></tt><br />
<br />
<br />
Output will resemble:<br><br />
<pre>Geom name: v106v107<br />
State: UP<br />
Status: Total=2, Online=2<br />
Type: AUTOMATIC<br />
ID: 3530663882<br />
Providers:<br />
1. Name: concat/v106v107<br />
Mediasize: 4294966272 (4.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e2<br />
Consumers:<br />
1. Name: gvinum/sd/v106.p0.s0<br />
Mediasize: 2147483648 (2.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e3<br />
Start: 0<br />
End: 2147483136<br />
2. Name: gvinum/sd/v107.p0.s0<br />
Mediasize: 2147483648 (2.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e3<br />
Start: 2147483136<br />
End: 4294966272</pre><br />
<br />
stop volume and clear members<br><br />
<tt>gconcat stop <span style="color:red">v106v107</span><br><br />
gconcat clear <span style="color:red">gvinum/sd/v106.p0.s0 gvinum/sd/v107.p0.s0</span></tt><br />
<br />
<br />
create new device- and its ok to reuse old/former members<br><br />
try to grab a contiguous block of gvinum volumes. gconcat volumes MAY NOT span drives (i.e. you cannot use a gvinum volume from data3 and a volume from data2 in the same gconcat volume). <br><br />
<tt>gconcat label <span style="color:red">v82-v84v106v107 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84 /dev/gvinum/v106 /dev/gvinum/v107</span> <br><br />
bsdlabel -r -w /dev/concat/<span style="color:red">v82-v84v106v107</span><br><br />
newfs /dev/concat/<span style="color:red">v82-v84v106v107</span>a</tt><br />
<br />
<br />
Optional- if new volume is on a different drive, move the mount point directory (get the drive from js output) AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
confirm you are mounted to the device (/dev/concat/<span style="color:red">v82-v84v106v107</span>) and space is correct:<br><br />
<tt>mount /dev/concat/<span style="color:red">v82-v84v106v107</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<tt>cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
df .</tt> <br />
<br />
do the restore from the dumpfile on the backup server:<br><br />
<tt>restore -r -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> .<br><br />
rm restoresymtable</tt><br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
edit quad (<tt>vq<span style="color:red">1</span></tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new volume (<tt>/dev/concat/<span style="color:red">v82-v84v106v107</span>a</tt>), run buildsafe<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
TODO: clean up/clear old gvin/gconcat vol<br />
<br />
<br />
6. update disk (and dir if applicable) in mgmt screen<br />
<br />
<br />
7. update backup list AND move backups, if applicatble<br />
<br />
<br />
Ex: <tt>mvbackups <span style="color:red">69.55.237.26-col00241</span> jail<span style="color:red">9</span> data<span style="color:red">3</span></tt><br />
<br />
<br />
<br />
DEPRECATED - steps to tack on a new gvin to existing gconcat- leads to corrupted fs<br />
bsdlabel -e /dev/concat/v82-v84<br />
<br />
To figure out new size of the c partition, multiply 4194304 by the # of 2G gvinum volumes and subtract the # of 2G volumes:<br />
10G: 4194304 * 5 – 5 = 20971515<br />
8G: 4194304 * 4 – 4 = 16777212<br />
6G: 4194304 * 3 – 3 = 12582909<br />
4G: 4194304 * 2 – 2 = 8388606<br />
<br />
To figure out the new size of the a partition, subtract 16 from the c partition:<br />
10G: 20971515 – 16 = 20971499<br />
8G: 16777212 – 16 = 16777196<br />
6G: 12582909 – 16 = 12582893<br />
4G: 8388606 – 16 = 8388590<br />
<br />
Orig:<br />
8 partitions:<br />
# size offset fstype [fsize bsize bps/cpg]<br />
a: 8388590 16 4.2BSD 2048 16384 28552<br />
c: 8388606 0 unused 0 0 # "raw" part, don't edit<br />
<br />
New:<br />
8 partitions:<br />
# size offset fstype [fsize bsize bps/cpg]<br />
a: 12582893 16 4.2BSD 2048 16384 28552<br />
c: 12582909 0 unused 0 0 # "raw" part, don't edit<br />
<br />
sync; sync<br />
<br />
growfs /dev/concat/v82-v84a<br />
<br />
fsck –fy /dev/concat/v82-v84a<br />
<br />
sync<br />
<br />
fsck –fy /dev/concat/v82-v84a<br />
<br />
(keep running fsck’s till NO errors)<br />
<br />
== Problems un-mounting - and with mount_null’s ==<br />
<br />
If you cannot unmount a filesystem, beacuse it says the filesystem is busy, it is because of three things:<br />
<br />
a) the jail is still running<br />
<br />
b) you are actually in that directory, even though the jail is stopped<br />
<br />
c) there are still dev, null_mount or linprocfs mount points mounted inside that directory.<br />
<br />
d) when trying to umount null_mounts that are really long and you get an error like “No such file or directory”, it’s an OS bug where the dir is truncated. No known fix<br />
<br />
e) there are still files open somewhere inside the dir. Use <tt>fstat | grep <cid></tt> to find the process that has files open<br />
<br />
f) Starting with 6.x, the jail mechanism does a poor job of keeping track of processes running in a jail and if it thinks there are still procs running, it will refuse to umount the disk. If this is happening you should see a low number in the #REF column when you run jls. In this case you ''can'' safely <tt>umount –f</tt> the mount. <br />
<br />
Please note -if you forcibly unmount a (4.x) filesystem that has null_mounts<br />
still mounted in it, the system '''will crash''' within 10-15 mins.<br />
<br />
== Misc jail Items ==<br />
<br />
We are overselling hard drive space on jail2, jail8, jail9, a couple jails on jail17, jail4, jail12 and jail18.<br />
Even though the vn file shows 4G size, it doesn’t actually occupy that amount of space on the disk. So be careful not to fill up drives where we’re overselling – use oversellcheck to confirm you’re not oversold by more than 10G.<br />
There are other truncated jails, they are generally noted in a the file on the root system: /root/truncated<br />
<br />
The act of moving a truncated vn to another system un-does the truncating- the truncated vn is filled with 0’s and it occupies physical disk space for which it’s configured. So, you should use dumpremote to preserve the truncation.<br />
<br />
= FreeBSD VPS Management Tools =<br />
<br />
These files are located in /usr/local/jail/rc.d and /usr/local/jail/bin<br />
<br />
== jailmake ==<br />
<br />
Applies to 7.x+ <br />
On older systems syntax differs, run jailmake once to see.<br />
<br />
Note: this procedure differs on mx2 which is 7.x but still uses gvinum<br />
<br />
# run js to figure out which md’s are in use, which disk has enough space, IP to put it on<br />
# use col00xxx for both hostnames if they don’t give you a hostname<br />
# copy over dir, ip and password to pending customer screen<br />
<br />
<tt>Usage: jailmake IP[,IP] CID disk[1|2|3] md# hostname shorthost ipfw# email [size in GB]</tt><br />
<br />
Ex: <br />
<br />
Jail2# jailmake 69.55.234.66 col01334 3 97 vps.bsd.it vps 1334 fb@bsd.it<br />
<br />
== jailps ==<br />
jailps [hostname]<br />
DEPRECATED FOR jps: displays processes belonging to/running inside a jail. The command<br />
takes one (optional) argument – the hostname of the jail you wish to query. If you don’t <br />
supply an argument, all processes on the machine are listed and grouped by jail. <br />
<br />
== jps ==<br />
jps [hostname]<br />
displays processes belonging to/running inside a jail. The command<br />
takes one (optional) argument – the hostname or ID of the jail you wish to query. <br />
<br />
== jailkill ==<br />
jailkill <hostname><br />
stops all process running in a jail.<br />
<br />
You can also run:<br />
jailkill <JID><br />
<br />
=== problems ===<br />
Occasionally you will hit an issue where jail will not kill off:<br />
<br />
<pre>jail9# jailkill www.domain.com<br />
www.domain.com .. killed: none<br />
jail9#</pre><br />
Because no processes are running under that hostname. You cannot use jailps.pl either:<br />
<br />
<pre>jail9# jailps www.domain.com<br />
www.domain.com doesn’t exist on this server<br />
jail9#</pre><br />
<br />
The reasons for this are usually:<br />
* the jail is no longer running<br />
<br />
* the jail's hostname has changed<br />
In this case, <br />
<br />
>=6.x: run a <tt>jls|grep <jail's IP></tt> to find the correct hostname, then update the quad file, then kill the jail.<br />
<br />
<6.x: the first step is to cat their /etc/rc.conf file to see if you can tell what they set the new hostname to. This very often works. For example:<br />
<br />
cat /mnt/data2/198.78.65.136-col00261-DIR/etc/rc.conf<br />
<br />
But maybe they set the hostname with the hostname command, and the original hostname is still in /etc/rc.conf.<br />
<br />
The welcome email clearly states that they should tell us if they change their hostname, so there is no problem in just emailing them and asking them what they set the new hostname to.<br />
<br />
Once you know the new hostname OR if a customer simply emails to inform you that they have set the hostname to something different, you need to edit the quad and safe files that their system is in to input the new hostname.<br />
<br />
However, if push comes to shove and you cannot find out the hostname from them or from their system, then you need to start doing some detective work.<br />
<br />
The easiest thing to do is run jailps looking for a hostname similar to their original hostname. Or you could get into the /bin/sh shell by running:<br />
<br />
/bin/sh<br />
<br />
and then looking at every hostname of every process:<br />
<br />
for f in `ls /proc` ; do cat /proc/$f/status ; done<br />
<br />
and scanning for a hostname that is either similar to their original hostname, or that you don't see in any of the quad safe files.<br />
<br />
This is very brute force though, and it is possible that catting every file in /proc is dangerous - I don't recommend it. A better thing would be to identify any processes that you know belong to this system – perhaps the reason you are trying to find this system is because they are running something bad - and just catting the status from only that PID.<br />
<br />
Somewhere there’s a jail where there may be 2 systems named www. Look at /etc/rc.conf and make sure they’re both really www. If they are, jailkill www, jailps www to make sure not running. Then immediately restart the other one, as the fqdn (as found from a rev nslookup)<br />
<br />
* on >=6.x the hostname may not yet be hashed:<br />
<pre>jail9 /# jls<br />
JID Hostname Path IP Address(es)<br />
1 bitnet.dgate.org /mnt/data1/69.55.232.50-col02094-DIR 69.55.232.50<br />
2 ns3.hctc.net /mnt/data1/69.55.234.52-col01925-DIR 69.55.234.52<br />
3 bsd1 /mnt/data1/69.55.232.44-col00155-DIR 69.55.232.44<br />
4 let2.bbag.org /mnt/data1/69.55.230.92-col00202-DIR 69.55.230.92<br />
5 post.org /mnt/data2/69.55.232.51-col02095-DIR 69.55.232.51 ...<br />
6 ns2 /mnt/data1/69.55.232.47-col01506-DIR 69.55.232.47 ...<br />
7 arlen.server.net /mnt/data1/69.55.232.52-col01171-DIR 69.55.232.52<br />
8 deskfood.com /mnt/data1/69.55.232.71-col00419-DIR 69.55.232.71<br />
9 mirage.confluentforms.com /mnt/data1/69.55.232.54-col02105-DIR 69.55.232.54 ...<br />
10 beachmember.com /mnt/data1/69.55.232.59-col02107-DIR 69.55.232.59<br />
11 www.agottem.com /mnt/data1/69.55.232.60-col02109-DIR 69.55.232.60<br />
12 sdhobbit.myglance.org /mnt/data1/69.55.236.82-col01708-DIR 69.55.236.82<br />
13 ns1.jnielsen.net /mnt/data1/69.55.234.48-col00204-DIR 69.55.234.48 ...<br />
14 ymt.rollingegg.net /mnt/data2/69.55.236.71-col01678-DIR 69.55.236.71<br />
15 verse.unixlore.net /mnt/data1/69.55.232.58-col02131-DIR 69.55.232.58<br />
16 smcc-mail.org /mnt/data2/69.55.232.68-col02144-DIR 69.55.232.68<br />
17 kasoutsuki.w4jdh.net /mnt/data2/69.55.232.46-col02147-DIR 69.55.232.46<br />
18 dili.thium.net /mnt/data2/69.55.232.80-col01901-DIR 69.55.232.80<br />
20 www.tekmarsis.com /mnt/data2/69.55.232.66-col02155-DIR 69.55.232.66<br />
21 vps.yoxel.net /mnt/data2/69.55.236.67-col01673-DIR 69.55.236.67<br />
22 smitty.twitalertz.com /mnt/data2/69.55.232.84-col02153-DIR 69.55.232.84<br />
23 deliver4.klatha.com /mnt/data2/69.55.232.67-col02160-DIR 69.55.232.67<br />
24 nideffer.com /mnt/data2/69.55.232.65-col00412-DIR 69.55.232.65<br />
25 usa.hanyuan.com /mnt/data2/69.55.232.57-col02163-DIR 69.55.232.57<br />
26 daifuku.ppbh.com /mnt/data2/69.55.236.91-col01720-DIR 69.55.236.91<br />
27 collins.greencape.net /mnt/data2/69.55.232.83-col01294-DIR 69.55.232.83<br />
28 ragebox.com /mnt/data2/69.55.230.104-col01278-DIR 69.55.230.104<br />
29 outside.mt.net /mnt/data2/69.55.232.72-col02166-DIR 69.55.232.72<br />
30 vps.payneful.ca /mnt/data2/69.55.234.98-col01999-DIR 69.55.234.98<br />
31 higgins /mnt/data2/69.55.232.87-col02165-DIR 69.55.232.87 ...<br />
32 ozymandius /mnt/data2/69.55.228.96-col01233-DIR 69.55.228.96<br />
33 trusted.realtors.org /mnt/data2/69.55.238.72-col02170-DIR 69.55.238.72<br />
34 jc1.flanderous.com /mnt/data2/69.55.239.22-col01504-DIR 69.55.239.22<br />
36 guppylog.com /mnt/data2/69.55.238.73-col00036-DIR 69.55.238.73<br />
40 haliohost.com /mnt/data2/69.55.234.41-col01916-DIR 69.55.234.41 ...<br />
41 satyr.jorge.cc /mnt/data1/69.55.232.70-col01963-DIR 69.55.232.70<br />
jail9 /# jailkill satyr.jorge.cc<br />
ERROR: jail_: jail "satyr,jorge,cc" not found<br />
</pre><br />
<br />
Note how it's saying <tt>satyr,jorge,cc</tt> is not found, and not <tt>satyr.jorge.cc</tt>. <br />
<br />
The jail subsystem tracks things using comma-delimited hostnames. That is created every few hours:<br />
<br />
jail9 /# crontab -l<br />
0 0,6,12,18 * * * /usr/local/jail/bin/sync_jail_names<br />
<br />
So if we run this manually:<br />
jail9 /# /usr/local/jail/bin/sync_jail_names<br />
<br />
Then kill the jail:<br />
jail9 /# jailkill satyr.jorge.cc<br />
successfully killed: satyr,jorge,cc</pre><br />
<br />
It worked.<br />
<br />
<br />
If you ever see this when trying to kill a jail:<br />
<br />
<pre># jailkill e-scribe.com<br />
killing JID: 6 hostname: e-scribe.com<br />
3 procs running<br />
3 procs running<br />
3 procs running<br />
3 procs running<br />
...</pre><br />
<br />
<tt>[[#jailkill|jailkill]]</tt> probably got lost trying to kill off the jail. Just ctrl-c the jailkill process, then run a jailps on the hostname, and <tt>kill -9</tt> any process which is still running. Keep running jailps and <tt>kill -9</tt> till all processes are gone.<br />
<br />
== jailpsall ==<br />
jailpsall<br />
will run a jailps on all jails configured in the quad files (this is different from<br />
jailps with no arguments as it won’t help you find a “hidden” system)<br />
<br />
== jailpsw ==<br />
jailpsw<br />
will run a jailps with an extra -w to provide wider output<br />
<br />
== jt (>=7.x) ==<br />
jt<br />
displays the top 20 processes on the server (the top 20 processes from top) and <br />
which jail owns them. This is very helpful for determining who is doing what when<br />
the server is very busy.<br />
<br />
== jtop (>=7.x) ==<br />
jtop<br />
a wrapper for top displaying processes on the server and which jail owns them. Constantly updates, like top. <br />
<br />
== jtop (<7.x) ==<br />
jtop<br />
displays the top 20 processes on the server (the top 20 processes from top) and <br />
which jail owns them. This is very helpful for determining who is doing what when<br />
the server is very busy.<br />
<br />
== stopjail ==<br />
stopjail <hostname> [1]<br />
this will jailkill, umount and vnconfig –u a jail. If passed an optional 2nd<br />
argument, it will not exit before umounting and un-vnconfig’ing in the event<br />
jailkill returns no processes killed. This is useful if you just want to umount<br />
and vnconfig –u a jail you’ve already killed. It is intelligent in that it won’t <br />
try to umount or vnconfig –u if it’s not necessary.<br />
<br />
== startjail ==<br />
startjail <hostname><br />
this will start vnconfig, mount (including linprocfs and null-mounts), and start a jail.<br />
Essentially, it reads the jail’s relevant block from the right quad file and executes it.<br />
It is intelligent in that it won’t try to mount or vnconfig if it’s not necessary.<br />
<br />
== jpid ==<br />
jpid <pid><br />
displays information about a process – including which jail owns it.<br />
It’s the equivalent of running cat /proc/<pid>/status<br />
<br />
== canceljail ==<br />
canceljail <hostname> [1]<br />
this will stop a jail (the equivalent of stopjail), check for backups (offer to remove them <br />
from the backup server and the backup.config), rename the vnfile, remove the dir, and <br />
edit quad/safe. If passed an optional 2nd argument, it will not exit upon failing to kill<br />
and processes owned by the jail. This is useful if you just want to cancel a jail which <br />
is already stopped.<br />
<br />
== jls ==<br />
jls [-v]<br />
Lists all jails running:<br />
<pre>JID #REF IP Address Hostname Path<br />
101 135 69.55.224.148 mail.pc9.org /mnt/data2/69.55.224.148-col01034-DIR</pre><br />
<br />
#REF is the number of references or procs(?) running<br />
<br />
Running with -v will give you all IPs assigned to each jail (7.2 up)<br />
<pre>JID #REF Hostname Path IP Address(es)<br />
101 139 mail.pc9.org /mnt/data2/69.55.224.148-col01034-DIR 69.55.224.14869.55.234.85</pre><br />
<br />
== startalljails ==<br />
startalljails<br />
7.2+ only. This will parse through quad1 and start all jails. It utilizes lockfiles so it won’t try to start a jail more than once- therefore multiple instances can be running in parallel without fear of starting a jail twice. If a jail startup gets stuck, you can ^C without fear of killing the script. IMPORTANT- before running startalljails you should make sure you ran preboot once as it will clear out all the lockfiles and enable startalljails to work properly.<br />
<br />
== aaccheck.sh ==<br />
aaccheck.sh<br />
displayes the output of container list and task list from aaccli<br />
<br />
== backup ==<br />
backup<br />
backup script called nightly to update jail scripts and do customer backups<br />
<br />
== buildsafe ==<br />
buildsafe<br />
creates safe files based on quads (automatically removing the fsck’s). This will destructively overwrite safe files<br />
<br />
== checkload.pl ==<br />
checkload.pl<br />
this was intended to be setup as a cronjob to watch processes on a jail when the load <br />
rises above a certain level. Not currently in use.<br />
<br />
== checkprio.pl ==<br />
checkprio.pl<br />
will look for any process (other than the current shell’s csh, sh, sshd procs) with a non-normal priority and normalize it<br />
<br />
== diskusagemon == <br />
diskusagemon <mount point> <1k blocks><br />
watches a mount point’s disk use, when it reaches the level specified in the 2nd argument,<br />
it exits. This is useful when doing a restore and you want to be paged as it’s nearing completion.<br />
Best used as: <tt>diskusagemon /asd/asd 1234; pagexxx</tt><br />
<br />
== dumprestore ==<br />
dumprestore <dumpfile><br />
this is a perl expect script which automatically enters ‘1’ and ‘y’. It seems to cause restore to fail<br />
to set owner permissions on large restores.<br />
<br />
== g ==<br />
g <search><br />
greps the quad/safe files for the given search parameter<br />
<br />
== gather.pl ==<br />
gather.pl<br />
gathers up data about jails configured and writes to a file. Used for audits against the db<br />
<br />
== gb ==<br />
gb <search><br />
greps backup.config for the given search parameter<br />
<br />
== gbg ==<br />
gbg <search><br />
greps backup.config for the given search parameter and presents just the directories (for clean pasting)<br />
<br />
== ipfwbackup ==<br />
ipfwbackup<br />
writes ipfw traffic count data to a logfile<br />
<br />
== ipfwreset ==<br />
ipfwreset<br />
writes ipfw traffic count data to a logfile and resets counters to 0<br />
<br />
== js ==<br />
js<br />
output varies by OS version, but generally provides information about the base jail:<br />
- which vn’s are in use<br />
- disk usage<br />
- info about the contents of quads<br />
- the # of inodes represented by the jails contained in the group (133.2 in the example below), and how many jails per data mount, as well as subtotals<br />
- ips bound to the base machine but not in use by a jail<br />
- free gvinum volumes, or unused vn’s or used md’s<br />
<br />
<pre>/usr/local/jail/rc.d/quad1:<br />
/mnt/data1 133.2 (1)<br />
/mnt/data2 1040.5 (7)<br />
total 1173.7 (8)<br />
/usr/local/jail/rc.d/quad2:<br />
/mnt/data1 983.4 (6)<br />
total 983.4 (6)<br />
/usr/local/jail/rc.d/quad3:<br />
/mnt/data1 693.4 (4)<br />
/mnt/data2 371.6 (3)<br />
total 1065 (7)<br />
/usr/local/jail/rc.d/quad4:<br />
/mnt/data1 466.6 (3)<br />
/mnt/data2 882.2 (5)<br />
total 1348.8 (8)<br />
/mnt/data1: 2276.6 (14)<br />
/mnt/data2: 2294.3 (15)<br />
<br />
Available IPs:<br />
69.55.230.11 69.55.230.13 69.55.228.200<br />
<br />
Available volumes:<br />
v78 /mnt/data2 2G<br />
v79 /mnt/data2 2G<br />
v80 /mnt/data2 2G</pre> <br />
<br />
== load.pl ==<br />
load.pl<br />
feeds info to load mrtg - executed by inetd.<br />
<br />
== makevirginjail ==<br />
makevirginjail<br />
Only on some systems, makes an empty jail (doesn't do restore step)<br />
<br />
== mb == <br />
mb <mount|umount><br />
(nfs) mounts and umounts dirs to backup2. Shortcuts are mbm and mbu to mount and unmount. <br />
<br />
== notify.sh ==<br />
notify.sh<br />
emails reboot@johncompanies.com – intended to be called at boot time to alert us to a machine which panics and reboots and isn’t caught by bb or castle.<br />
<br />
== orphanedbackupwatch ==<br />
orphanedbackupwatch<br />
looks for directories on backup2 which aren’t configured in backup.config and offers to delete them<br />
<br />
== postboot ==<br />
postboot<br />
to be run after a machine reboot and quad/safe’s are done executing. It will:<br />
* do chmod 666 on each jail’s /dev/null<br />
* add ipfw counts<br />
* run jailpsall (so you can see if a configured jail isn’t running)<br />
<br />
== preboot ==<br />
preboot<br />
to be run before running quad/safe – checks for misconfigurations: <br />
* a jail configured in a quad but not a safe<br />
* a jail is listed more than once in a quad<br />
* the ip assigned to a jail isn’t configured on the machine<br />
* alias numbering skips in the rc.conf (resulting in the above)<br />
* orphaned vnfile's that aren't mentioned in a quad/safe<br />
* ip mismatches between dir/vnfile name and the jail’s ip<br />
* dir/vnfiles's in quad/safe that don’t exist <br />
<br />
== quadanalyze.pl ==<br />
quadanalyze.pl<br />
called by js, produces the info (seen above with js explanation) about the contents of quad (inode count, # of jails, etc.)<br />
<br />
== rsync.backup ==<br />
rsync.backup<br />
does customer backups (relies on backup.config)<br />
<br />
== taskdone ==<br />
taskdone<br />
when called will email support@johncompanies.com with the hostname of the machine from which it was executed as the subject<br />
<br />
== topten ==<br />
topten<br />
summarizes the top 10 traffic users (called by ipfwreset)<br />
<br />
== trafficgather.pl ==<br />
trafficgather.pl [yy-mm]<br />
sends a traffic usage summary by jail to support@johncomapnies.com and payments@johncompanies.com. Optional arguments are year and month (must be in the past). If not passed, assumes last month. Relies on traffic logs created by ipfwreset and ipfwbackup<br />
<br />
== trafficwatch.pl ==<br />
trafficwatch.pl<br />
checks traffic usage and emails support@johncomapnies.com when a jail reaches the warning level (35G) and the limit (40G). We really aren’t using this anymore now that we have netflow.<br />
<br />
== trafstats ==<br />
trafstats<br />
writes ipfw traffic usage info by jail to a file called jc_traffic_dump in each jail’s / dir<br />
<br />
== truncate_jailmake ==<br />
truncate_jailmake<br />
a version of jailmake which creates truncated vnfiles.<br />
<br />
== vb ==<br />
vb<br />
the equivalent of: <tt>vi /usr/local/jail/bin/backup.config</tt><br />
<br />
== vs (freebsd) ==<br />
vs<n><br />
the equivalent of: <tt>vi /usr/local/jail/rc.d/safe<n></tt><br />
<br />
vq<n><br />
the equivalent of: vi /usr/local/jail/rc.d/quad<n><br />
<br />
== dumpremote ==<br />
dumpremote <user@machine> </remote/location/file-dump> <vnX><br />
ex: dumpremote user@10.1.4.117 /mnt/data3/remote.echoditto.com-dump 7<br />
this will dump a vn filesystem to a remote machine and location<br />
<br />
== oversellcheck ==<br />
oversellcheck<br />
displays how much a disk is oversold or undersold taking into account truncated vn files. Only for use on 4.x systems<br />
<br />
== mvbackups (freebsd) ==<br />
mvbackups <dir> (1.1.1.1-col00001-DIR) <target_machine> (jail1) <target_dir> (data1)<br />
moves backups from one location to another on the backup server, and provides you with option to remove entries from current backup.config, and simple paste command to add the config to backup.config on the target server<br />
<br />
== jailnice ==<br />
jailnice <hostname><br />
applies <tt>renice 19 [PID]</tt> and <tt>rtprio 31 –[PID]</tt> to each process in the given jail<br />
<br />
== dumpremoterestore ==<br />
dumpremoterestore <device> <ip of target machine> <dir on target machine><br />
ex: dumpremoterestore /dev/vn51 10.1.4.118 /mnt/data2/69.55.239.45-col00688-DIR<br />
dumps a device and restores it to a directory on a remote machine. Requires that you enable root ssh on the <br />
remote machine.<br />
<br />
== psj ==<br />
psj<br />
shows just the procs running on the base system – a ps auxw but without jail’d procs present<br />
<br />
== perc5iraidchk ==<br />
perc5iraidchk<br />
checks for degraded arrays on Dell 2950 systems with Perc5/6 controllers<br />
<br />
== perc4eraidchk ==<br />
perc4eraidchk<br />
checks for degraded arrays on Dell 2850 systems with Perc4e/Di controllers<br />
<br />
== Making new customer VE (vm) ==<br />
<br />
This applies only to new virts >= 4.x<br />
<br />
grab ip from ipmap (if opened from the pending cust screen it should take you to the right block). You can also run vzlist -a to see what block is in use, generally. Try to find an IP that's in the same block of class C IP's already on the box.<br />
<br />
1. confirm ip you want to use isn’t in use via Mgmt. -> IP Map in management screens<br />
<br />
2. put CT on whichever partition has more space<br />
<br />
3. vm CID IP hostname /vz[1|2] email[,email] template ( <LB|LBP|LS|LM|LMP|LP> [disk] | <gb_disk/mb_ram/gb_burst> ) <br />
vm col00009 69.55.230.238 centos.testdave.com /vz1 dsmith@n2.net centos-6-x86_64 LM<br />
<br />
4. copy veid, dir, ip and password to pending customer screen. activate customer<br />
<br />
<br />
== Making new customer VE (vemakexxx) ==<br />
<br />
This applies to older virts with old templates. This should probably not be used at all anymore.<br />
<br />
<br />
1. look thru hist for ip<br />
<br />
2. confirm ip you want to use isn’t in use via Mgmt. -> IP Map in management screens<br />
<br />
3. put ve on whichever partition has more space<br />
vemakerh9 <veid> <ip> <hostname> <mount> <email> [gb disk]; <256|384|512> <veid><br />
vemakerh9 866 69.55.226.109 ngentu.com /vz1 ayo@ngantu.com,asd@asd.com 5; 256 866<br />
<br />
4. copy (veid), dir, and ip to pending customer screen (pass set to p455agfa)<br />
<br />
= Virtuozzo VPS =<br />
<br />
Note: We use VEID (Virtual Environment ID) and CTID (Container ID) interchangably. Similarly, VE and CT. They mean the same thing.<br />
VZPP = VirtuoZzo Power Panel (the control panel for each CT)<br />
<br />
All linux systems exist in /vz, /vz1 or /vz2 - since each linux machine holds roughly 60-90 customers, there will be roughly 30-45 in each partition.<br />
<br />
The actual filesystem of the system in question is in:<br />
<br />
/vz(1-2)/private/(VEID)<br />
<br />
Where VEID is the identifier for that system - an all-numeric string larger than 100.<br />
<br />
The actual mounted and running systems are in the corresponding:<br />
<br />
/vz(1-2)/root/(VEID)<br />
<br />
But we rarely interact with any system from this mount point.<br />
<br />
You should never need to touch the root portion of their system – however you can traverse their filesystem by going to <tt>/vz(1-2)/private/(VEID)/root</tt> (<tt>/vz(1-2)/private/(VEID)/fs/root</tt> on 4.x systems) the root of their filesystem is in that directory, and their entire system is underneath that.<br />
<br />
Every VE has a startup script in <tt>/etc/sysconfig/vz-scripts</tt> (which is symlinked as <tt>/vzconf</tt> on all systems) - the VE startup script is simply named <tt>(VEID).conf</tt> - it contains all the system parameters for that VE:<br />
<br />
<pre># Configuration file generated by vzsplit for 60 VE<br />
# on HN with total amount of physical mem 2011 Mb<br />
<br />
VERSION="2"<br />
CLASSID="2"<br />
<br />
ONBOOT="yes"<br />
<br />
KMEMSIZE="8100000:8200000"<br />
LOCKEDPAGES="322:322"<br />
PRIVVMPAGES="610000:615000"<br />
SHMPAGES="33000:34500"<br />
NUMPROC="410:415"<br />
PHYSPAGES="0:2147483647"<br />
VMGUARPAGES="13019:2147483647"<br />
OOMGUARPAGES="13019:2147483647"<br />
NUMTCPSOCK="1210:1215"<br />
NUMFLOCK="107:117"<br />
NUMPTY="19:19"<br />
NUMSIGINFO="274:274"<br />
TCPSNDBUF="1800000:1900000"<br />
TCPRCVBUF="1800000:1900000"<br />
OTHERSOCKBUF="900000:950000"<br />
DGRAMRCVBUF="200000:200000"<br />
NUMOTHERSOCK="650:660"<br />
DCACHE="786432:818029"<br />
NUMFILE="7500:7600"<br />
AVNUMPROC="51:51"<br />
IPTENTRIES="155:155"<br />
DISKSPACE="4194304:4613734"<br />
DISKINODES="400000:420000"<br />
CPUUNITS="1412"<br />
QUOTAUGIDLIMIT="2000"<br />
VE_ROOT="/vz1/root/636"<br />
VE_PRIVATE="/vz1/private/636"<br />
NAMESERVER="69.55.225.225 69.55.230.3"<br />
OSTEMPLATE="vzredhat-7.3/20030305"<br />
VE_TYPE="regular"<br />
IP_ADDRESS="69.55.225.229"<br />
HOSTNAME="textengine.net"</pre><br />
<br />
<br />
As you can see, the hostname is set here, the disk space is set here, the number of inodes, the number of files that can be open, the number of tcp sockets, etc. - all are set here.<br />
<br />
In fact, everything that can be set on this customer system is set in this conf file.<br />
<br />
<br />
All interaction with the customer system is done with the VEID. You start the system by running:<br />
<br />
vzctl start 999<br />
<br />
You stop it by running:<br />
<br />
vzctl stop 999<br />
<br />
You execute commands in it by running:<br />
<br />
vzctl exec 999 df -k<br />
<br />
You enter into it, via a root-shell backdoor with:<br />
<br />
vzctl enter 999<br />
<br />
and you set parameters for the system, while it is still running, with:<br />
<br />
vzctl set 999 --diskspace 6100000:6200000 --save<br />
<br />
<tt>vzctl</tt> is the most commonly used command - we have aliased <tt>v</tt> to <tt>vzctl</tt> since we use it so often. We’ll continue to use <tt>vzctl</tt> in our examples, but feel free to use just <tt>v</tt>.<br />
<br />
Let's say the user wants more diskspace. You can cat their conf file and see:<br />
<br />
DISKSPACE="4194304:4613734"<br />
<br />
So right now they have 4gigs of space. You can then change it to 6 with:<br />
<br />
vzctl set 999 --diskspace 6100000:6200000 --save<br />
<br />
IMPORTANT: all issuances of the vzctl set command need to end with <tt>–save</tt> - if they don't, the setting will be set, but it will not be saved to the conf file, and they will not have those settings next time they boot.<br />
<br />
All of the tunables in the conf file can be set with the vzctl set command. Note that in the conf file, and on the vzctl set command line, we always issue two numbers seperated by a colon - that is because we are setting the hard and soft limits. Always set the hard limit slightly above the soft limit, as you see it is in the conf file for all those settings.<br />
<br />
There are also things you can set with `<tt>vzctl set</tt>` that are not in the conf file as settings, per se. For instance, you can add IPs:<br />
<br />
vzctl set 999 --ipadd 10.10.10.10 --save<br />
<br />
or multiple IPs:<br />
<br />
vzctl set 999 --ipadd 10.10.10.10 --ipadd 10.10.20.30 --save<br />
<br />
or change the hostname:<br />
<br />
vzctl set 999 --hostname www.example.com --save<br />
<br />
You can even set the nameservers:<br />
<br />
vzctl set 999 --nameserver 198.78.66.4 --nameserver 198.78.70.180 --save<br />
<br />
Although you probably will never do that.<br />
<br />
You can disable a VPS from being started (by VZPP or reboot) (4.x):<br />
<br />
vzctl set 999 --disabled yes --save <br />
<br />
You can disable a VPS from being started (by VZPP or reboot) (<=3.x):<br />
<br />
vzctl set 999 --onboot=no --save <br />
<br />
You can disable a VPS from using his control panel:<br />
<br />
vzctl set 999 --offline_management=no --save <br />
<br />
You can suspend a VPS, so it can be resumed in the same state it was in when it was stopped (4.x):<br />
<br />
vzctl suspend 999<br />
<br />
and to resume it:<br />
<br />
vzctl resume 999<br />
<br />
to see who owns process:<br />
vzpid <PID><br />
<br />
to mount up an unmounted ve:<br />
vzctl mount 827<br />
<br />
<br />
To see network stats for CT's:<br />
<pre># vznetstat<br />
VEID Net.Class Output(bytes) Input(bytes)<br />
-----------------------------------------------<br />
24218 1 484M 39M<br />
24245 1 463M 143M<br />
2451 1 2224M 265M<br />
2454 1 2616M 385M<br />
4149 1 125M 68M<br />
418 1 1560M 34M<br />
472 1 1219M 315M<br />
726 1 628M 317M<br />
763 1 223M 82M<br />
771 1 4234M 437M<br />
-----------------------------------------------<br />
Total: 13780M 2090M</pre><br />
<br />
<br />
One thing that sometimes comes up on older systems that we created with smaller defaults is that the system would run out of inodes. The user will email and say they cannot create any more files or grow any files larger, but they will also say that they are not out of diskspace ... they are running:<br />
<br />
df -k<br />
<br />
and seeing how much space is free - and they are not out of space. They are most likely out of inodes - which they would see by running:<br />
<br />
df -i<br />
<br />
So, the first thing you should do is enter their system with:<br />
<br />
vzctl enter 999<br />
<br />
and run: <tt>df -i</tt><br />
<br />
to confirm your theory. Then exit their system. Then simply cat their conf file and see what their inodes are set to (probably 200000:200000, since that was the old default on the older systems) and run:<br />
<br />
vzctl set 999 --diskinodes 400000:400000 --save<br />
<br />
If they are not out of inodes, then a good possibility is that they have maxed out their numfile configuration variable, which controls how many files they can have in their system. The current default is 7500 (which nobody has ever hit), but the old default was as low as 2000, so you would run something like:<br />
<br />
vzctl set 999 --numfile 7500:7500 --save<br />
<br />
----<br />
<br />
You cannot start or stop a VE if your pwd is its private (/vz/private/999) or root (/vz/root/999) directories, or anywhere below them.<br />
<br />
<br />
== Recovering from a crash (linux) ==<br />
<br />
=== Diagnose whether you have a crash ===<br />
The most important thing is to get the machine and all ve’s back up as soon as possible. Note the time, you’ll need to create a crash log entry (Mgmt. -> Reference -> CrashLog). The first thing to do is head over to the [[Screen#Screen_Organization|serial console screen]] and see if there’s any kernel error messages output. Try to copy any messages (or just a sample of repeating messages) you see into the notes section of the crash log – these will also likely need to be sent to virtuozzo for interpretation. If the messages are spewing too fast, hit ^O + H to start a screen log dump which you can ob1182.pts-38.bb serve after the machine is rebooted. Additionally, if the machine is responsive, you can get a trace to send to virtuozzo by hooking up a kvm and entering these 3 sequences:<br />
<pre>alt+print screen+m<br />
alt+print screen+p<br />
alt+print screen+t</pre><br />
<br />
If there are no messages, the machine may just be really busy- wait a bit (5-10min) to see if it comes back. If it's still pinging, odds are its very busy. If it doesn't come back, or the messages indicate a fatal error, you will need to proceed with a power cycle (ctrl+alt+del will not work).<br />
<br />
=== Power cycle the server ===<br />
If this machine is not a Dell 2950 with a [[DRAC/RMM#DRAC|DRAC card]] (i.e. if you can’t ssh into the DRAC card and issue racadm serveraction hardreset, then you will need someone at the data center to power the macine off, wait 30 sec, then turn it back on. Make sure to re-attach via console (<tt>tip virtxx</tt>) immediately after power down. <br />
<br />
=== (Re)attach to the console ===<br />
Stay on the console the entire time during boot. As the BIOS posts- look out for the RAID card output- does everything look healthy? The output may be scrambled, look for "DEGRADED" or "FAILED". Once the OS starts booting you will be disconnected (dropped back to the shell on the console server) a couple times during the boot up. The reason you want to quickly re-attach is two-fold: 1. If you don’t reattach quickly then you won’t get any console output, 2. you want to be attached before the server ''potentially'' starts (an extensive) fsck. If you attach after the fsck begins, you’ll have seen no indication it started an fsck and the server will appear frozen during startup- no output, no response. <br />
<br />
=== Start containers/VE's/VPSs ===<br />
When the machine begins to start VE’s, it’s safe to leave the console and login via ssh. All virts should be set to auto start all the VEs after a crash. Further, most (newer) virts are set to “fastboot” it’s VE’s (to find out, do:<br />
grep -i fast /etc/sysconfig/vz <br />
and look for <tt>VZFASTBOOT=yes</tt>). If this was set prior to the machine’s crash (setting it after the machine boots will not have any effect until the vz service is restarted) it will start each ve as fast as possible, in serial, then go thru each VE (serially), shutting it down running a vzquota (disk usage) check, then bringing it back up. The benefit is that all VE’s are brought up quickly (within 15min or so depending on the #), the downside is a customer watching closely will notice 2 outages – 1st the machine crash, 2nd their quota check (which will be a much shorter downtime- on the order of a few minutes). <br />
<br />
Where “fastboot” is not set to yes (i.e on quar1), vz will start them consecutively, checking the quotas one at a time, and the 60th VE may not start until an hour or two later - this is not acceptable.<br />
<br />
The good news is, if you run vzctl start for a VE that is already started, you will simply get an error: <tt>VE is already started</tt>. Further, if you attempt to vzctl start a VE that is in the process of being started, you will simply get an error: unable to lock VE. So, there is no danger in simply running scripts to start smaller sets of VEs. If the system is not autostarting, then there is no issue, and even if it does, when it conflicts, one process (yours or the autostart) will lose, and just move on to the next one.<br />
<br />
A script has been written to assist with ve starts: [[#startvirt.pl|startvirt.pl]] which will start 6 ve’s at once until there are no more left. If startvirt.pl is used on a system where “fastboot” was on, it will circumvent the fastboot for ve’s started by startvirt.pl – they will go through the complete quota check before starting- therefore this is not advisable when a system has crashed. When a system is booted cleanly, and there's no need for vzquota checks, then startvirt.pl is safe and advisable to run.<br />
<br />
=== Make sure all containers are running ===<br />
You can quickly get a feel for how many ve’s are started by running:<br />
<br />
<pre>[root@virt4 log]# vs<br />
VEID 16066 exist mounted running<br />
VEID 16067 exist mounted running<br />
VEID 4102 exist mounted running<br />
VEID 4112 exist mounted running<br />
VEID 4116 exist mounted running<br />
VEID 4122 exist mounted running<br />
VEID 4123 exist mounted running<br />
VEID 4124 exist mounted running<br />
VEID 4132 exist mounted running<br />
VEID 4148 exist mounted running<br />
VEID 4151 exist mounted running<br />
VEID 4155 exist mounted running<br />
VEID 42 exist mounted running<br />
VEID 432 exist mounted running<br />
VEID 434 exist mounted running<br />
VEID 442 exist mounted running<br />
VEID 450 exist mounted running<br />
VEID 452 exist mounted running<br />
VEID 453 exist mounted running<br />
VEID 454 exist mounted running<br />
VEID 462 exist mounted running<br />
VEID 463 exist mounted running<br />
VEID 464 exist mounted running<br />
VEID 465 exist mounted running<br />
VEID 477 exist mounted running<br />
VEID 484 exist mounted running<br />
VEID 486 exist mounted running<br />
VEID 490 exist mounted running</pre><br />
<br />
So to see how many ve’s have started:<br />
<pre>[root@virt11 root]# vs | grep running | wc -l<br />
39</pre><br />
<br />
And to see how many haven’t:<br />
<pre>[root@virt11 root]# vs | grep down | wc -l<br />
0</pre><br />
<br />
And how many we should have running:<br />
<pre>[root@virt11 root]# vs | wc -l<br />
39</pre><br />
<br />
Another tool you can use to see which ve’s have started, among other things is [[#vzstat|vzstat]]. It will give you CPU, memory, and other stats on each ve and the overall system. It’s a good thing to watch as ve’s are starting (note the VENum parameter, it will tell you how many have started):<br />
<br />
<pre>4:37pm, up 3 days, 5:31, 1 user, load average: 1.57, 1.68, 1.79<br />
VENum 40, procs 1705: running 2, sleeping 1694, unint 0, zombie 9, stopped 0<br />
CPU [ OK ]: VEs 57%, VE0 0%, user 8%, sys 7%, idle 85%, lat(ms) 412/2<br />
Mem [ OK ]: total 6057MB, free 9MB/54MB (low/high), lat(ms) 0/0<br />
Swap [ OK ]: tot 6142MB, free 4953MB, in 0.000MB/s, out 0.000MB/s<br />
Net [ OK ]: tot: in 0.043MB/s 402pkt/s, out 0.382MB/s 4116pkt/s<br />
Disks [ OK ]: in 0.002MB/s, out 0.000MB/s<br />
<br />
VEID ST %VM %KM PROC CPU SOCK FCNT MLAT IP<br />
1 OK 1.0/17 0.0/0.4 0/32/256 0.0/0.5 39/1256 0 9 69.55.227.152<br />
21 OK 1.3/39 0.1/0.2 0/46/410 0.2/2.8 23/1860 0 6 69.55.239.60<br />
133 OK 3.1/39 0.1/0.3 1/34/410 6.3/2.8 98/1860 0 0 69.55.227.147<br />
263 OK 2.3/39 0.1/0.2 0/56/410 0.3/2.8 34/1860 0 1 69.55.237.74<br />
456 OK 17/39 0.1/0.2 0/100/410 0.1/2.8 48/1860 0 11 69.55.236.65<br />
476 OK 0.6/39 0.0/0.2 0/33/410 0.1/2.8 96/1860 0 10 69.55.227.151<br />
524 OK 1.8/39 0.1/0.2 0/33/410 0.0/2.8 28/1860 0 0 69.55.227.153<br />
594 OK 3.1/39 0.1/0.2 0/45/410 0.0/2.8 87/1860 0 1 69.55.239.40<br />
670 OK 7.7/39 0.2/0.3 0/98/410 0.0/2.8 64/1860 0 216 69.55.225.136<br />
691 OK 2.0/39 0.1/0.2 0/31/410 0.0/0.7 25/1860 0 1 69.55.234.96<br />
744 OK 0.1/17 0.0/0.5 0/10/410 0.0/0.7 7/1860 0 6 69.55.224.253<br />
755 OK 1.1/39 0.0/0.2 0/27/410 0.0/2.8 33/1860 0 0 192.168.1.4<br />
835 OK 1.1/39 0.0/0.2 0/19/410 0.0/2.8 5/1860 0 0 69.55.227.134<br />
856 OK 0.3/39 0.0/0.2 0/13/410 0.0/2.8 16/1860 0 0 69.55.227.137<br />
936 OK 3.2/52 0.2/0.4 0/75/410 0.2/0.7 69/1910 0 8 69.55.224.181<br />
1020 OK 3.9/39 0.1/0.2 0/60/410 0.1/0.7 55/1860 0 8 69.55.227.52<br />
1027 OK 0.3/39 0.0/0.2 0/14/410 0.0/2.8 17/1860 0 0 69.55.227.83<br />
1029 OK 1.9/39 0.1/0.2 0/48/410 0.2/2.8 25/1860 0 5 69.55.227.85<br />
1032 OK 12/39 0.1/0.4 0/80/410 0.0/2.8 41/1860 0 8 69.55.227.90</pre><br />
<br />
When you are all done, you will want to make sure that all the VEs really did get started, run vs one more time.<br />
<br />
Note the time all ve’s are back up and enter that into and save the crash log entry.<br />
<br />
Occasionally, a ve will not start automatically. The most common reason for a ve not to come up normally is the ve was at it’s disk limit before the crash, and will not start since they’re over the limit. To overcome this, set the disk space to current usage level (the system will give this to you when it fails to start), start the ve, then re-set the disk space back to the prior level. Lastly, contact the customer to let them know they’re out of disk (or allocate more disk if they're entitled to more).<br />
<br />
== Hitting performance barriers and fixing them ==<br />
<br />
There are multiple modes virtuozzo offers to allocate resources to a ve. We utilize 2: SLM and UBC parameters<br />
On our 4.x systems, we use all SLM – it’s simpler to manage and understand. There are a few systems on virt19/18 that may also use SLM. Everything else uses UBC. <br />
You can tell a SLM ve by:<br />
<br />
SLMMODE="all"<br />
<br />
in their conf file. <br />
<br />
TODO: detail SLM modes and parameters.<br />
<br />
If someone is in SLM mode and they hit memory resource limits, they simply need to upgrade to more memory.<br />
<br />
The following applies to everyone else (UBC).<br />
<br />
Customers will often email and say that they are getting out of memory errors - a common one is "cannot fork" ... basically, anytime you see something odd like this, it means they are hitting one of their limits that is in place in their conf file.<br />
<br />
The conf file, however, simply shows their limits - how do we know what they are currently at ?<br />
<br />
The answer is a file called v - this file contains the current status (and peaks) of their performance settings, and also counts how many times they have hit the barrier. The output of the file looks like this:<br />
<br />
<pre>764: kmemsize 384113 898185 8100000 8200000 0<br />
lockedpages 0 0 322 322 0<br />
privvmpages 1292 7108 610000 615000 0<br />
shmpages 270 528 33000 34500 0<br />
dummy 0 0 0 0 0<br />
numproc 8 23 410 415 0<br />
physpages 48 5624 0 2147483647 0<br />
vmguarpages 0 0 13019 2147483647 0<br />
oomguarpages 641 6389 13019 2147483647 0<br />
numtcpsock 3 21 1210 1215 0<br />
numflock 1 3 107 117 0<br />
numpty 0 2 19 19 0<br />
numsiginfo 0 4 274 274 0<br />
tcpsndbuf 0 80928 1800000 1900000 0 <br />
tcprcvbuf 0 108976 1800000 1900000 0<br />
othersockbuf 2224 37568 900000 950000 0<br />
dgramrcvbuf 0 4272 200000 200000 0<br />
numothersock 3 9 650 660 0<br />
dcachesize 53922 100320 786432 818029 0<br />
numfile 161 382 7500 7600 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
numiptent 4 4 155 155 0</pre><br />
<br />
The first column is the name of the counter in question - the same names we saw in the systems conf file. The second column is the _current_ value of that counter, the third column is the max that that counter has ever risen to, the fourth column is the soft limit, and the fifth column is the hard limit (which is the same as the numbers in that systems conf file).<br />
<br />
The sixth number is the failcount - how many times the current usage has risen to hit the barrier. It will increase as soon as the current usage hits the soft limit.<br />
<br />
The problem with /proc/user_beancounters is that it actually contains that set of data for every running VE - so you can't just cat /proc/user_beancounters - it is too long and you get info for every other running system.<br />
<br />
You can vzctl enter the system and run:<br />
<br />
vzctl enter 9999<br />
cat /proc/user_beancounters<br />
<br />
inside their system, and you will just see the stats for their particular system, but entering their system every time you want to see it is combersome.<br />
<br />
So, I wrote a simple script called "vzs" which simply greps for the VEID, and spits out the next 20 or so lines (however many lines there are in the output, I forget) after it. For instance:<br />
<br />
<pre># vzs 765:<br />
765: kmemsize 2007936 2562780 8100000 8200000 0<br />
lockedpages 0 8 322 322 0<br />
privvmpages 26925 71126 610000 615000 0<br />
shmpages 16654 16750 33000 34500 0<br />
dummy 0 0 0 0 0<br />
numproc 41 57 410 415 0<br />
physpages 1794 49160 0 2147483647 0<br />
vmguarpages 0 0 13019 2147483647 0<br />
oomguarpages 4780 51270 13019 2147483647 0<br />
numtcpsock 23 37 1210 1215 0<br />
numflock 17 39 107 117 0<br />
numpty 1 3 19 19 0<br />
numsiginfo 0 6 274 274 0<br />
tcpsndbuf 22240 333600 1800000 1900000 0<br />
tcprcvbuf 0 222656 1800000 1900000 0<br />
othersockbuf 104528 414944 900000 950000 0<br />
dgramrcvbuf 0 4448 200000 200000 0<br />
numothersock 73 105 650 660 0<br />
dcachesize 247038 309111 786432 818029 0<br />
numfile 904 1231 7500 7600 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
numiptent 4 4 155 155 0</pre><br />
<br />
That showed us just the portion of /proc/user_beancounters for system 765.<br />
<br />
When you run the vzs command, always add a : after the VEID.<br />
<br />
So, if a customer complains about some out of memory errors, or no more files, or no more ptys, or just has an unspecific complain about processes dying, etc., the very first thing you need to do is check their beancounters with vzs. Usually you will spot an item that has a high failcount and needs to be upped.<br />
<br />
At that point you could simply up the counter with `vzctl set`. Generally pick a number 10-20% higher than the old one, and make the hard limit slightly larger than the the soft limit. However our systems now come in several levels and those levels have more/different memory allocations. If someone is complaining about something other than a memory limit (pty, numiptent, numflock), it’s generally safe to increase it, at least to the same level as what’s in the /vzconf/4unlimited file on the newest virt. If someone is hitting a memory limit, first make sure they are given what they deserve:<br />
<br />
(refer to mgmt -> payments -> packages)<br />
<br />
To set those levels, you use the [[#setmem|setmem]] command. <br />
<br />
The alternate (DEPRECATED) method would be to use one of 3 commands:<br />
256 <veid><br />
300 <veid><br />
384 <veid><br />
512 <veid><br />
<br />
If the levels were not right (you’d run vzs <veid> before and after to see the effect) tell the customer they’ve been adjusted and be done with it. If the levels were right, tell the customer they must upgrade to a higher package, tell them how to see level (control panel) and that they can reboot their system to escape this lockup contidion.<br />
<br />
Customers can also complain that their site is totally unreachable, or complain that it is down ... if the underlying machine is up, and all seems well, you may notice in the beancounters that network-specific counters are failing - such as numtcpsock, tcpsndbuf or tcprcvbuf. This will keep them from talking on the network and make it seem like their system is down. Again, just up the limits and things should be fine.<br />
<br />
On virts 1-4, you should first look at the default settings for that item on a later virt, such as virt 8 - we have increased the defaults a lot since the early machines. So, if you are going to up a counter on virt2, instead of upping it by 10-20%, instead up it to the new default that you see on virt8.<br />
<br />
== Moving a VE to another virt (migrate/migrateonline) ==<br />
<br />
This will take a while to complete - and it is best to do this at night when the load is light on both machines.<br />
<br />
There are different methods for this, depending on which version of virtuozzo is installed on the src. and dst. virt. <br />
To check which version is running: <br />
[root@virt12 private]# cat /etc/virtuozzo-release<br />
Virtuozzo release 2.6.0<br />
<br />
Ok, let's say that the VE is 1212, and vital stats are:<br />
<pre>[root@virt12 sbin]# vc 1212<br />
VE_ROOT="/vz1/root/1212"<br />
VE_PRIVATE="/vz1/private/1212"<br />
OSTEMPLATE="fedora-core-2/20040903"<br />
IP_ADDRESS="69.55.229.84"<br />
TEMPLATES="devel-fc2/20040903 php-fc2/20040813 mysql-fc2/20040812 postgresql-fc2/20040813 mod_perl-fc2/20040812 mod_ssl-fc2/20040811 jre-fc2/20040823 jdk-fc2/20040823 mailman-fc2/20040823 analog-fc2/20040824 proftpd-fc2/20040818 tomcat-fc2/20040823 usermin-fc2/20040909 webmin-fc2/20040909 uw-imap-fc2/20040830 phpBB-fc2/20040831 spamassassin-fc2/20040910 PostNuke-fc2/20040824 sl-webalizer-fc2/20040<br />
818"<br />
<br />
[root@virt12 sbin]# vzctl exec 1212 df -h<br />
Filesystem Size Used Avail Use% Mounted on<br />
/dev/vzfs 4.0G 405M 3.7G 10% /</pre><br />
<br />
From this you can see that he’s using (and will minimally need free on the dst server) ~400MB, and he’s running on a Fedora 2 template, version 20040903. He’s also got a bunch of other templates installed. It’s is '''vital''' that '''all''' these templates exist on the dst system. To confirm that, on the dst system run:<br />
<br />
For < 3.0:<br />
<br />
<pre>[root@virt14 private]# vzpkgls | grep fc2<br />
devel-fc2 20040903<br />
PostNuke-fc2 20040824<br />
analog-fc2 20040824<br />
awstats-fc2 20040824<br />
bbClone-fc2 20040824<br />
jdk-fc2 20040823<br />
jre-fc2 20040823<br />
mailman-fc2 20040823<br />
mod_frontpage-fc2 20040816<br />
mod_perl-fc2 20040812<br />
mod_ssl-fc2 20040811<br />
mysql-fc2 20040812<br />
openwebmail-fc2 20040817<br />
php-fc2 20040813<br />
phpBB-fc2 20040831<br />
postgresql-fc2 20040813<br />
proftpd-fc2 20040818<br />
sl-webalizer-fc2 20040818<br />
spamassassin-fc2 20040910<br />
tomcat-fc2 20040823<br />
usermin-fc2 20040909<br />
uw-imap-fc2 20040830<br />
webmin-fc2 20040909<br />
[root@virt14 private]# vzpkgls | grep fedora<br />
fedora-core-1 20040121 20040818<br />
fedora-core-devel-1 20040121 20040818<br />
fedora-core-2 20040903<br />
[root@virt14 private]#</pre><br />
<br />
For these older systems, you can simply match up the date on the template. <br />
<br />
For >= 3.0:<br />
<br />
<pre>[root@virt19 /vz2/private]# vzpkg list<br />
centos-5-x86 2008-01-07 22:05:57<br />
centos-5-x86 devel<br />
centos-5-x86 jre<br />
centos-5-x86 jsdk<br />
centos-5-x86 mod_perl<br />
centos-5-x86 mod_ssl<br />
centos-5-x86 mysql<br />
centos-5-x86 php<br />
centos-5-x86 plesk9<br />
centos-5-x86 plesk9-antivirus<br />
centos-5-x86 plesk9-api<br />
centos-5-x86 plesk9-atmail<br />
centos-5-x86 plesk9-backup<br />
centos-5-x86 plesk9-horde<br />
centos-5-x86 plesk9-mailman<br />
centos-5-x86 plesk9-mod-bw<br />
centos-5-x86 plesk9-postfix<br />
centos-5-x86 plesk9-ppwse<br />
centos-5-x86 plesk9-psa-firewall<br />
centos-5-x86 plesk9-psa-vpn<br />
centos-5-x86 plesk9-psa-fileserver<br />
centos-5-x86 plesk9-qmail<br />
centos-5-x86 plesk9-sb-publish<br />
centos-5-x86 plesk9-vault<br />
centos-5-x86 plesk9-vault-most-popular<br />
centos-5-x86 plesk9-watchdog</pre><br />
<br />
On these newer systems, it's difficult to tell whether the template on the dst matches exactly the src. Just cause a centos-5-x86 is listed on both servers doesn't mean all the same packages are there on the dst. To truly know, you must perform a sample rsync:<br />
<br />
rsync -avn /vz/template/centos/5/x86/ root@10.1.4.61:/vz/template/centos/5/x86/<br />
<br />
if you see a ton of output from the dry run command, then clearly there are some differences. You may opt to let the rsync complete (without running in dry run mode) the only downside is you've now used up more space on the dst and also the centos template will be a mess with old and new data- it will be difficult if not impossible to undo (if someday we wanted to reclaim the space).<br />
<br />
If you choose to merge templates, you should closely inspect the dry run output. You should also take care to exclude anything in the /config directory. For example:<br />
<br />
rsync -av -e ssh --stats --exclude=x86/config /vz/template/ubuntu/10.04/ root@10.1.4.62:/vz/template/ubuntu/10.04/<br />
<br />
Which will avoid this directory and contents:<br />
<pre>[root@virt11 /vz2/private]# ls /vz/template/ubuntu/10.04/x86/config*<br />
app os</pre><br />
<br />
This is important to avoid since the config may differ on the destination and we are really only interested in making sure the pacakges are there, not overwriting a newer config with an older one.<br />
<br />
If the dst system was missing a template, you have 2 choices: <br />
# put the missing template on the dst system. 2 choices here: <br />
## Install the template from rpm (found under backup2: /mnt/data4/vzrpms/distro/) or <br />
## rsync over the template (found under /vz/template) - see above<br />
# put the ve on a system which has all the proper templates<br />
<br />
=== pre-seeding a migration ===<br />
<br />
When migrating a customer (or when doing many) depending on how much data you have to transfer, it can take some time. Further, it can be difficult to gauge when a migration will complete or how long it will take. To help speed up the process and get a better idea about how long it will take you can pre-transfer a customer's data to the destination server. If done correctly, vzmigrate will see the pre-transferred data and pick up where you left off, having much less to transfer (just changed/new files). <br />
<br />
We believe vzmigrate uses rsync to do it's transfer. Therefore not only can you use rsync to do a pre-seed, you can also run rsync to see what is causing a repeatedly-failing vzmigrate to fail. <br />
<br />
There's no magic to a pre-seed, you just need to make sure it's named correctly.<br />
<br />
Given:<br />
<br />
source: /vz1/private/1234<br />
<br />
and you want to migrate to /vz2 on the target system, your rsync would look like:<br />
<br />
rsync -av /vz1/private/1234/ root@x.x.x.x:/vz2/private/1234.migrated/<br />
<br />
After running that successful rsync, the ensuing migrateonline (or migrate) will take much less time to complete- depending on the # of files to be analyzed and the # of changed files. In any case, it'll be much much faster than had you just started the migration from scratch.<br />
<br />
Further, as we discuss elsewhere in this topic, a failed migration can be moved from <tt>/vz/private/1234</tt> to <tt>/vz/private/1234.migrated</tt> on the destination if you want to restart a failed migration. This should '''only''' be done if the migration failed and the CT is not running on the destination HN.<br />
<br />
=== migrateonline intructions: src >=3.x -> dst>=3.x ===<br />
<br />
A script called [[#migrateonline|migrateonline]] was written to handle this kind of move. It is basically a wrapper for <tt>vzmigrate</tt> – vzmigrate is a util to seamlessly- as no no reboot of the ve necessary- move a ve from one host to another. This wrapper was initially written cause vz virtuozzo version 2.6.0 has a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables, causing problems when the ve was started up on the dst system. [[#migrate|migrate]] mitigates that. Since it makes multiple ssh connections to the dst virt, it’s a good idea to put the pub key for the src virt in the authorized_keys file on the dst virt. In addition, migrateonline emails ve owners when their migration starts and stops. For this to happen they need to put email addresses (on a single line, space delimited) in a file on their system: /migrate_notify. If the optional target dir is not specified, the ve will be moved to the same private/root location as it was on the src virt. Note: <tt>migrate</tt> is equivalent to <tt>migrateonline</tt>, but will <tt>migrate</tt> a ve AND restart it in the process.<br />
<br />
<pre>[root@virt12 sbin]# migrateonline<br />
usage: /usr/local/sbin/migrateonline <ip of node migrating to> <veid> [target dir: vz | vz1 | vz2]<br />
[root@virt12 sbin]# migrateonline 10.1.4.64 1212 vz<br />
starting to migrate 1212 at Sat Mar 26 22:40:38 PST 2005<br />
Turning off offline_management<br />
Saved parameters for VE 1212<br />
migrating with no start on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
clearing the arp cache<br />
now going to 10.1.4.64 and clear cache and starting<br />
starting it<br />
Delete port redirection<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
finished migrating 1212 at Sat Mar 26 22 22:52:01 PST 2005<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine.<br />
<br />
If they had backups, use the mvbackups command to move their backups to the new server:<br />
<br />
mvbackups 1212 virt14 vz<br />
<br />
Rename the ve<br />
<br />
[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/migrated-1212<br />
<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/old-1212-migrated-20120404-noarchive<br />
<br />
Update the customer’s systems in mgmt to reflect the new path and server.<br />
<br />
<br />
IF migrateonline does not work, you can try again using simply migrate- this will result in a brief reboot for the ve.<br />
Before you try again, make sure of a few things:<br />
<br />
Depending on where in the migration died, there may be partial data on the dst system in 1 of 2 places:<br />
(given the example above)<br />
<br />
/vz/private/1212<br />
<br />
or <br />
<br />
/vz/private/1212.migrated<br />
<br />
before you run migrate again, you'll want to rename so that all data is in <br />
1212.migrated:<br />
<br />
mv /vz/private/1212 /vz/private/1212.migrated<br />
<br />
this way, it will pick up where it left off and transfer only new files.<br />
<br />
Likewise, if you want to speed up a migration, you can pre-seed the dst as follows:<br />
<br />
[root@virt12 sbin]# rsync -avSH /vz/private/1212/ root@10.1.4.64:/vz/private/1212.migrated/<br />
<br />
then when you run migrate or migrateonline, it will only need to move the changed files- the migration will complete quickly<br />
<br />
=== migrateonline/migrate failures (migrate manually) ===<br />
<br />
Lets say for whatever reason the migration fails. If it fails with [[#migrateonline|migrateonline]], you should try [[#migrate|migrate]] (which will reboot the customer, so notify them ahead of time).<br />
<br />
You may want to run a [[#pre-seeding_a_migration|pre-seed]] rsync to see if you can find the problem. On older virts, we've seen this problem due to a large logfile (which you can find and encourage the customer to remove/compress):<br />
for f in `find / -size +1048576k`; do ls -lh $f; done<br />
<br />
You may also see migration failing due to quota issues.<br />
<br />
You can try to resolve by copying any quota file into the file you need:<br />
<br />
cp /var/vzquota/quota.1 /var/vzquota/quota.xxx<br />
<br />
If it complains about quota running you should then be able to stop it<br />
<br />
vzquota off xxxx<br />
<br />
If all else fails, migrate to a new VEID<br />
i.e. 1234 becomes 12341<br />
<br />
<br />
If the rsync or [[#migrate|migrate]] fails, you can always move someone manually:<br />
<br />
1. stop ve: <br><br />
v stop 1234<br />
<br />
2. copy over data<br><br />
rsync -avSH /vz/private/1234/ root@1.1.1.1:/vzX/private/1234/<br />
<br />
NOTE: if you've previously seeded the data (run rsync while the VE was up/running), and this is a subsequent rsync, make sure the last rsync you do (while the VE is not running, has the --delete option in the rsync)<br />
<br />
3. copy over conf<br><br />
scp /vzconf/1234.conf root@1.1.1.1:/vzconf<br />
<br />
4. on dst, edit the conf to reflect the right vzX dir<br><br />
vi /vzconf/1234.conf<br />
<br />
5. on src remove the IPs<br><br />
ipdel 1234 2.2.2.2 3.3.3.3<br />
<br />
6. on dst add IPs <br><br />
ipadd 1234 2.2.2.2 3.3.3.3<br />
<br />
7. on dst, start ve: <br><br />
v start 1324<br />
<br />
8. cancel, then archive ve on src per above instrs.<br />
<br />
=== migrate src=2.6.0 -> dst>=2.6.0, or mass-migration with customer notify ===<br />
<br />
A script called <tt>migrate</tt> was written to handle this kind of move. It is basically a wrapper for vzmigrate – vzmigrate is a util to seamlessly move a ve from one host to another. This wrapper was initially written cause vz virtuozzo version 2.6.0 has a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables, causing problems when the ve was started up on the dst system. migrate mitigates that. Since it makes multiple ssh connections to the dst virt, it’s a good idea to put the pub key for the src virt in the authorized_keys file on the dst virt. In addition, migrate emails ve owners when their migration starts and stops. For this to happen they need to put email addresses (on a single line, space delimited) in a file on their system: /migrate_notify. If the optional target dir is not specified, the ve will be moved to the same private/root location as it was on the src virt. Note: migrateonline is equivalent to migrate, but will migrate a ve from one 2.6 '''kernel''' machine to another 2.6 kernel machine without restarting the ve.<br />
<br />
<pre>[root@virt12 sbin]# migrate<br />
usage: /usr/local/sbin/migrate <ip of node migrating to> <veid> [target dir: vz | vz1 | vz2]<br />
[root@virt12 sbin]# migrate 10.1.4.64 1212 vz<br />
starting to migrate 1212 at Sat Mar 26 22:40:38 PST 2005<br />
Turning off offline_management<br />
Saved parameters for VE 1212<br />
migrating with no start on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
clearing the arp cache<br />
now going to 10.1.4.64 and clear cache and starting<br />
starting it<br />
Delete port redirection<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
finished migrating 1212 at Sat Mar 26 22 22:52:01 PST 2005<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine (backup2 or mail).<br />
Cancel the ve (first we have to rename things which migrate changed so cancelve will find them):<br />
<br />
[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/1212.conf<br />
<br />
On 2.6.1 you’ll also have to move the private area:<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/1212<br />
<br />
<pre>[root@virt12 sbin]# cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, [[#cancelve|cancelve]] would offer to remove them. You want to say '''no''' to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
To move everyone off a system, you’d do:<br />
for f in `vl`; do migrate <ip> $f; done<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
=== vzmigrate: src=2.6.1 -> dst>=2.6.0 ===<br />
<br />
This version of vzmigrate works properly with regard to handling ips. It will not notify ve owners of moves as in the above example. Other than that it’s essentially the same.<br />
<br />
<pre>[root@virt12 sbin]# vzmigrate 10.1.4.64 -r no 1212:1212:/vz/private/1212:/vz/root/1212<br />
migrating on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine (backup2 or mail).<br />
Cancel the ve (first we have to rename things which vzmigrate changed so cancelve will find them):<br />
<br />
<pre>[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/1212.conf<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/1212<br />
<br />
[root@virt12 sbin]# cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, <tt>cancelve</tt> would offer to remove them. You want to say no to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
=== src=2.5.x ===<br />
<br />
First, go to the private dir:<br />
<br />
cd /vz1/private/<br />
<br />
Stop the VE - make sure it stops totally cleanly.<br />
<br />
vzctl stop 1212<br />
<br />
Then you’d use vemove - a script written to copy over the config, create tarballs of the ve’s data on the destination virt, and cancel the ve on the source system (in this example we’re going to put a ve that was in /vz1/private on the src virt, in /vz/private on the dst virt):<br />
<br />
<pre>[root@virt12 sbin]# vemove<br />
ERROR: Usage: vemove veid target_ip target_path_dir<br />
[root@virt12 sbin]# vemove 1212 10.1.4.64 /vz/private/1212<br />
tar cfpP - 1212 --ignore-failed-read | (ssh -2 -c arcfour 10.1.4.64 "split - -b 1024m /vz/private/1212.tar" )<br />
scp /vzconf/1212.conf 10.1.4.64:/vzconf<br />
cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, cancelve would offer to remove them. You want to say no to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
When you are done, go to /vz/private on the dst virt you will have files like this:<br />
<br />
<pre>1212.taraa<br />
1212.tarab<br />
1212.tarac</pre><br />
<br />
Each one 1024m (or less, for the last one) in size.<br />
<br />
on the dst server and run:<br />
<br />
cat 1212.tar?? | tar xpPBf -<br />
<br />
and after 20 mins or so it will be totally untarred. Now since the conf<br />
file is already there, you can go ahead and start the system.<br />
<br />
vzctl start 1212<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
NOTE: you MUST tar the system up using the virtuozzo version of tar that<br />
is on all the virt systems, and further you MUST untar the tarball with<br />
the virtuozzo tar, using these options: `<tt>tar xpPBf -</tt>`<br />
<br />
If you tar up an entire VE and move it to a non-virtuozzo machine, that is<br />
ok, and you can untar it there with normal tar commands, but do not untar<br />
it and then repack it with a normal tar and expect it to work - you need<br />
to use virtuozzo tar commands on virtuozzo tarballs to make it work.<br />
<br />
The backups are sort of an exception, since we are just (usually)<br />
restoring user data that was created after we gave them the system, and<br />
therefore has nothing to do with magic symlinks or vz-rpms, etc.<br />
<br />
== Moving a VE on the same virt ==<br />
<br />
Easy way:<br><br />
Scenario 1: ve 123 is to be renamed 1231 and moved from vz1 to vz<br />
<br />
vzmlocal 123:1231:/vz/private/1231:/vz/root/1231<br />
<br />
Scenario 2: ve 123 is to be moved vz1 to vz<br />
<br />
vzmlocal 123:123:/vz/private/123:/vz/root/123<br />
<br />
vzmlocal will reboot the ve at the end of the move<br />
<br />
Manual/old way:<br />
<br />
1) <tt>vzctl stop 123</tt><br><br />
2) <tt>mv /vz1/private/123 /vz/private/.</tt><br><br />
(or cp -a if you want to copy)<br />
3) in <tt>/etc/sysconfig/vz-scripts/123.conf</tt> change value<br><br />
of '<tt>VE_PRIVATE</tt>' variable to point to a new private area location<br />
4) <tt>vzctl start 123</tt><br><br />
5) update backups if needed: <tt>mvbackups 123 virtX virt1 vz</tt><br><br />
6) update management scerens<br><br />
<br />
Notes: a) absolute path to private area is stored in quota file <tt>/var/vzquota/quota.123</tt> - so during first startup quota will be recalculated.<br><br />
b) if you're going to write some script to do a job, you MUST be sure that $VEID won't be expanded to '' in ve config file - ie. you need to escape '$'. Otherwise you might have:<br />
<br />
VE_PRIVATE="/vz/private/"<br />
<br />
in config, and 'vzctl destroy' for this VE ID '''will remove everything under /vz/private/ directory'''.<br />
<br />
== Adding a veth device to a VE ==<br />
<br />
Not totally sure what this is, but a customer asked for it and here's what we did (as instructed by vz support):<br />
<br />
<pre>v set 99 --netif_add eth99 --save<br />
ipdel 99 69.55.230.58<br />
v set 99 --ifname eth99 --ipadd 69.55.230.58 --save<br />
v set 99 --ifname eth99 --gateway 69.55.230.1 --save<br />
<br />
vznetcfg net new veth_net<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
net99 active eth0 veth77.77,veth99.99<br />
veth_net active<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
br0 bridge veth_net<br />
br99 bridge net99<br />
veth99.99 veth net99<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic net99 69.55.227.70/24<br />
<br />
vznetcfg br attach br0 eth0<br />
<br />
(will remove 99 from orig net and move to veth_net)<br />
vznetcfg net addif veth_net veth99.99<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
net99 active<br />
veth_net active eth0 veth77.77,veth99.99<br />
<br />
(delete the old crap)<br />
vznetcfg net del net99<br />
<br />
<br />
Then, to add another device in<br />
<br />
v set 77 --netif_add eth77 --save<br />
ipdel 77 69.55.230.78<br />
v set 77 --ifname eth77 --ipadd 69.55.230.78 --save<br />
v set 77 --ifname eth77 --gateway 69.55.230.1 --save<br />
v set 77 --save --ifname eth77 --network veth_net<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
veth77.77 veth<br />
br0 bridge veth_net<br />
veth99.99 veth veth_net<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic veth_net 69.55.227.70/24<br />
<br />
vznetcfg net addif veth_net veth77.77<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
veth77.77 veth veth_net<br />
br0 bridge veth_net<br />
veth99.99 veth veth_net<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic veth_net 69.55.227.70/24<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
veth_net active eth0 veth77.77,veth99.99<br />
<br />
another example<br />
<br />
v set 1182 --netif_add eth1182 --save<br />
ipdel 1182 69.55.236.217<br />
v set 1182 --ifname eth1182 --ipadd 69.55.236.217 --save<br />
v set 1182 --ifname eth1182 --gateway 69.55.236.1 --save<br />
vznetcfg net addif veth_net veth1182.1182<br />
v set 1182 --save --ifname eth1182 --network veth_net<br />
<br />
<br />
unused/not working commands:<br />
ifconfig veth99.0 0<br />
vznetcfg net list<br />
vznetcfg br new br99 net99<br />
vznetcfg br attach br99 eth0<br />
vznetcfg br show<br />
vznetcfg if list<br />
vznetcfg net addif net99 veth99.99<br />
vznetcfg net addif eth0 net99<br />
<br />
---------<br />
<br />
vznetcfg br new br1182 net1182<br />
<br />
vznetcfg br attach br99 eth0<br />
vznetcfg if list<br />
vznetcfg net addif net99 veth99.99<br />
vznetcfg net addif eth0 net99<br />
<br />
<br />
<br />
vznetcfg net addif eth0 net1182<br />
<br />
<br />
# vznetcfg net addif veth99.0 net99<br />
--- 8< ---<br />
<br />
vznetcfg net new net<br />
# vznetcfg net addif eth0 net99<br />
<br />
<br />
# vzctl set 99 --save --netif_add eth0 (at this stage veth99.0 interface have to appear<br />
on node)<br />
# vzctl set 99 --save --ifname eth0 --ipadd 69.55.230.58 (and probably few more arguments<br />
here - see 'man vzctl')<br />
# vznetcfg net addif veth99.0 net99</pre><br />
<br />
<br />
== Assigning/remove ip from a VE ==<br />
<br />
1. Add or remove ips:<br />
ipdel 1234 1.1.1.1 2.2.2.2<br />
ipadd 1234 1.1.1.1 2.2.2.2<br />
<br />
2. update Mgmt screens<br />
<br />
3. offer to update any DNS we do for them<br />
<br />
4. check to see if we had rules for old IP in firwall<br />
<br />
== Enabling tun device for a ve ==<br />
Note, there’s a command for this: [[#addtun|addtun]]<br />
<br />
For posterity:<br />
Make sure the tun.o module is already loaded before Virtuozzo is started: <br />
lsmod <br />
Allow the VPS to use the TUN/TAP device: <br />
vzctl set 101 --devices c:10:200:rw --save <br />
Create the corresponding device inside the VPS and set the proper permissions: <br />
vzctl exec 101 mkdir -p /dev/net <br />
vzctl exec 101 mknod /dev/net/tun c 10 200 <br />
vzctl exec 101 chmod 600 /dev/net/tun<br />
<br />
== Remaking a system (on same virt) ==<br />
<br />
1. [[#cancelve|cancelve]] (or v destroy x - ONLY if you're POSITIVE no data needs to be saved)<br />
<br />
2. [[#vemake|vemake]] using same veid<br />
<br />
3. [[#mvbackups|mvbackups]] or [[#vb|vb]] (if new mount point)<br />
<br />
4. update mgmt with new dir/ip <br />
<br />
5. update firewall if changed ip<br />
<br />
== Re-initialize quota for a VE ==<br />
<br />
There’s a commamd for this now: [[#clearquota|clearquota]]<br />
<br />
For posterity:<br />
<br />
vzctl stop 1<br />
vzquota drop 1<br />
vzctl start 1<br />
<br />
<br />
== Traffic accounting on linux ==<br />
<br />
DEPRECATED - all tracking is done via bwdb now. This is how we used to track traffic.<br />
<br />
TODO: update for diff versions of vz<br />
<br />
Unlike FreeBSD, where we have to add firewall count rules to the system to count the traffic, on virtuozzo counts the traffic for us. You an see the current traffic stats by running `vznetstat`:<br />
<br />
<pre># vznetstat<br />
VEID Net.Class Output(bytes) Input(bytes)<br />
-----------------------------------------------<br />
24218 1 484M 39M<br />
24245 1 463M 143M<br />
2451 1 2224M 265M<br />
2454 1 2616M 385M<br />
4149 1 125M 68M<br />
418 1 1560M 34M<br />
472 1 1219M 315M<br />
726 1 628M 317M<br />
763 1 223M 82M<br />
771 1 4234M 437M<br />
-----------------------------------------------<br />
Total: 13780M 2090M</pre><br />
<br />
As you can see the VEID is on a line with the in and out bytes. So, we simply run a cron job:<br />
<br />
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /root/vztrafdump.sh<br />
<br />
Just like we do on FreeBSD - this one goes through all the VEs in /vz/private and greps the line from vznetstat that matches them and dumps it in /jc_traffic_dump on their system. Then it does it again for all the VEs in /vz1/private. It is important to note that vznetstat runs only once, and the grepping is done from a temporary file that contains that output - we do this because running vznetstat once for each VE that we read out of /vz/private and /vz1/private would take way too long and be too intensive.<br />
<br />
You do not need to do anything to facilitate this other than make sure that that cron job is running - the vznetstat counters are always running, and any new VEs that are added to the system will be accounted for automatically.<br />
<br />
Traffic resetting no longer works with vz 2.6, so we disable the vztrafdump.sh on those virts.<br />
<br />
== Watchdog script ==<br />
<br />
On some of the older virts, we have a watchdog running that kills procs that are deemed bad per the following:<br />
<br />
/root/watchdog from quar1<br />
<br />
<pre>if echo $line | awk '{print $(NF-3)}' | grep [5-9]...<br />
then<br />
# 50-90%<br />
if echo $line | awk '{print $(NF-1)}' | grep "...:.."<br />
then<br />
# running for > 99min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
if echo $line | awk '{print $(NF-1)}' | grep "....m"<br />
then<br />
# running for > 1000min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
fi<br />
<br />
if echo $line | awk '{print $(NF-3)}' | grep [1-9]...<br />
then<br />
# running for 10-90 percent<br />
if echo $line | awk '{print $NF}' | egrep 'cfusion|counter|vchkpw'<br />
then<br />
<br />
if echo $line | awk '{print $(NF-1)}' | grep "[2-9]:.."<br />
then<br />
# between 2-9min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
elif echo $line | awk '{print $(NF-1)}' | grep "[0-9][0-9]:.."<br />
then<br />
# up to 99min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
fi<br />
fi</pre><br />
<br />
<br />
== Misc Linux Items ==<br />
<br />
We are overselling hard drive space ... when you configure a linux system with a certain amount of disk space (the default is 4gigs) you do not actually use up 4gigs of space on the system. The diskspace setting for a user is simply a cap, and they only use up as much space on the actual disk drive as they are actually using.<br />
<br />
When you create a new linux system, even though there are some 300 RPMs or so installed, if you run `df -k` you will see that the entire 4gig partition is empty - no space is being used. This is because the files in their system are "magic symlinks" to the template for their OS that is in /vz/template - however, any changes to any of those files will "disconnect" them and they will immediately begin using space in their system. Further, any new files uploaded (even if those new files overwrite existing files) will take up space on the partition.<br />
<br />
<br />
if you see this:<br />
<br />
<pre>[root@virt8 root]# vzctl stop 160 ; vzctl start 160<br />
VE is not running<br />
Starting VE ...<br />
VE is unmounted<br />
VE is mounted<br />
Adding IP address(es): 69.55.226.83 69.55.226.84<br />
bash ERROR: Can't change file /etc/sysconfig/network<br />
Deleting IP address(es): 69.55.226.83 69.55.226.84<br />
VE is unmounted<br />
[root@virt8 root]#</pre><br />
<br />
it probably means they no longer have /bin/bash - copy one in for them<br />
<br />
ALSO: another possibility is that they have removed the `ed` RPM from their system - it needs to be reinstalled into their system. But since their system is down, this is tricky ...<br />
<br />
VE startup scripts used by 'vzctl' want package 'ed' to be available inside VE. So if package 'ed' will be enabled in OS template config and OS template itself VE #827 is based on - this error should be fixed.<br />
<br />
yes, it is possible to add RPM to VE while it not running.<br />
Try to do following:<br />
<br />
<pre># cd /vz/template/<OS_template_with_ed_package>/<br />
# vzctl mount 827<br />
# rpm -Uvh --root /vz/root/827 --veid 827 ed-0.2-25.i386.vz.rpm</pre><br />
<br />
Usually theres an error, but its ok<br />
<br />
Note: replace 'ed-0.2-25.i386.vz.rpm' in last command with actual<br />
version of 'ed' package you have.<br />
<br />
----<br />
<br />
So how do I know what template the user has ? cat their conf file and it is listed in there. For example, if the conf file has:<br />
<br />
<pre>[root@virt12 sbin]# vc 1103<br />
…snip…<br />
OSTEMPLATE="debian-3.0/20030822"<br />
TEMPLATES="mod_perl-deb30/20030707 mod_ssl-deb30/20030703 mysql-deb30/20030707 proftpd-deb30/20030703 webmin-deb30/20030823 "<br />
…</pre><br />
<br />
then they are on debian 3.0, all of their system RPMs are in /vz/template/debian-3.0, and they are using version 20030822 of that debian 3.0 template. Also, they’ve also got additional packages installed (mod_perl, mod_ssl, etc). Those are also found under /vz/template<br />
<br />
----<br />
<br />
Edits needed to run java:<br />
<br />
When we first created the VEs, the default setting for privvmpages was 93000:94000 ... which was high enough that most people never had problems ... however, you can;t run java or jdk or tomcat or anything java related with that setting. We have found that by setting privvmpages to 610000:615000 that java runs just fine. That is now the default setting. It is exceedingly rare that anyone needs it higher than that, although we have seen it once or twice.<br />
<br />
Any problems with java at all - the first thing you need to do is see if the failcnt has raised for privvmpages.<br />
<br />
<pre># vzctl start 160<br />
Starting VE ...<br />
vzquota : (error) Quota on syscall for 160: Device or resource busy<br />
Running vzquota on failed for VE 160 [3]</pre><br />
<br />
This is because my pwd is _in_ their private directory - you can't start it until you move out<br />
<br />
People seem to have trouble with php if they are clueless newbies. Here are two common problems/solutions:<br />
<br />
no... but i figured it out myself. problem was the php.ini file that came<br />
vanilla with the account was not configured to work with apache (the<br />
ENGINE directive was set to off).<br />
<br />
everything else seems fine now.<br />
<br />
----<br />
<br />
the problem was in the php.ini file. I noticed that is wasnt showing<br />
the code when it was in an html file so I looked at the php.ini file<br />
and had to change it so it recognized <? tags aswell as <?php tags.<br />
<br />
Also, make sure added to httpd.conf<br />
AddType application/x-httpd-php .php<br />
<br />
----<br />
<br />
You can set the time zone:<br />
<br />
You can change the timezone by doing this:<br />
<br />
ln -sf /usr/share/zoneinfo/<zone> /etc/localtime<br />
<br />
where <zone> is the zone you want in the /usr/share/zoneinfo/ directory.<br />
<br />
----<br />
<br />
Failing shm_open calls:<br />
<br />
first, please check if /dev/shm is mounted inside VE.<br />
'cat /proc/mounts' command should show something like this:<br />
tmpfs /dev/shm tmpfs rw 0 0<br />
<br />
If /dev/shm is not mounted you have 2 ways to solve issue:<br />
1. execute following command inside VE (doesn't require VE reboot):<br />
mount -t tmpfs none /dev/shm<br />
2. add following string to /etc/fstab inside VE and reboot it:<br />
tmpfs /dev/shm tmpfs defaults 0 0<br />
<br />
----<br />
<br />
You can have a mounted but not running ve<br />
Just:<br />
vzctl mount <veid><br />
<br />
----<br />
<br />
When a debian sys can’t get on the network, and you try:<br />
<br />
<pre>vzctl set 1046 --ipadd 69.55.227.117<br />
Adding IP address(es): 69.55.227.117<br />
Failed to bring up lo.<br />
Failed to bring up venet0.</pre><br />
<br />
They probably removed iproute package, which must be the one from swsoft. To restore:<br />
<pre># dpkg -i --veid=1046 --admindir=/vz1/private/1046/root/var/lib/dpkg --instdir=/vz1/private/1046/root/ /vz/template/debian-3.0/iproute_20010824-8_i386.vz.deb<br />
(Reading database ... 16007 files and directories currently installed.)<br />
Preparing to replace iproute 20010824-8 (using .../iproute_20010824-8_i386.vz.deb) ...<br />
Unpacking replacement iproute ...<br />
Setting up iproute (20010824-8) ...</pre><br />
<br />
Then restart their ve<br />
<br />
----<br />
<br />
in a ve i do:<br />
<br />
<pre>cd /<br />
du -h .</pre><br />
<br />
and get: 483M .<br />
<br />
i do:<br />
<br />
<pre>bash-2.05a# df -h<br />
Filesystem Size Used Avail Use% Mounted on<br />
/dev/vzfs 4.0G 2.3G 1.7G 56% /</pre><br />
<br />
how can this be?<br />
<br />
Is it possible that quota file was corrupted somehow? Please try to: <br />
<pre>vzctl stop <VEID><br />
vzquota drop <VEID><br />
vzquota init <VEID><br />
vzctl start <VEID></pre><br />
<br />
----<br />
<br />
How to stop vz from starting after reboot:<br />
<br />
VIRTUOZZO=no <br />
in <br />
/etc/sysconfig/vz<br />
<br />
To start: <br />
service vz start<br />
(after setting VIRTUOZZO=yes in /etc/sysconfig/vz)<br />
<br />
service vz restart will do some kind of 'soft reboot' -- restart all<br />
VPSes and reload modules without rebooting the node<br />
<br />
if you need to shut down all VPSes really really fast, run killall -9 init<br />
<br />
----<br />
<br />
Postfix tip:<br />
<br />
You may want to tweak settings: default_process_limit=10<br />
<br />
= Virtuozzo VPS Management Tools =<br />
<br />
== vm ==<br />
<br />
TODO<br />
<br />
== cancelve ==<br />
cancelve <veid><br />
this will:<br />
* stop a ve<br />
* check for backups (offer to remove them from the backup server <br />
and the backup.config)<br />
* rename the private dir<br />
* check for PTR, provide the commands to reset to default<br />
* and rename the ve’s config<br />
* remind you to remove firewall rules<br />
* remind you to remove DNS entries<br />
<br />
== ipadd ==<br />
ipadd <veid> <ip> [ip] [ip]<br />
add’s ip(s) to a ve<br />
<br />
== ipdel ==<br />
ipdel <veid> <ip> [ip] [ip]<br />
removes ip(s) from a ve<br />
<br />
== vc ==<br />
vc <veid><br />
the equivalent of: <tt>cat /vzconf/<veid>.conf</tt><br />
<br />
== vl ==<br />
vl<br />
will displays a list of ve #’s, 1 per line. (ostensibly to use in a for loop)<br />
<br />
== vp ==<br />
vp <veid><br />
the equivalent of: <tt>vzps auxww –E <veid></tt><br />
<br />
== vpe ==<br />
DEPRECATED<br />
vpe <veid> <br />
this will allow you to do a vp when a ve is running out of control, the equivalent of (deprecated since vp operates outside the VPS): <br />
<pre>vzctl set <veid> --kmemsize 2100000:2200000<br />
vzctl exec <veid> ps auxw<br />
vzctl set <veid> --kmemsize (ve’s orig lvalue):(ve’s orig hvalue)</pre><br />
<br />
== vt ==<br />
vt <veid><br />
the equivalent of: <tt>vztop –E <veid></tt><br />
<br />
== vr ==<br />
vr <veid><br />
the equivalent of: <tt>vzctl stop <veid>; vzctl start <veid></tt><br />
You can run this even if the ve is down - the stop command will just fail<br />
<br />
== vs ==<br />
vs [veid]<br />
displays status (output of <tt>vzctl status <veid></tt>) on each ve configured on the system (as determined by <tt>/vzconf/*.conf</tt>)<br />
If passed an argument, gives the status for just that ve. <br />
A running system looks like:<br />
<tt>VEID 16066 exist mounted running</tt><br />
<br />
A ve which is not running (but does exist) looks like:<br />
<tt>VEID 9990 exist unmounted down</tt><br />
<br />
A ve which is not running and doesn’t exist looks like:<br />
<tt>VEID 421 deleted unmounted down</tt><br />
<br />
== vs2 ==<br />
vs2 [veid]<br />
this is similar to vs in that it displays status (output of <tt>vzctl status <veid></tt>) on each ve,<br />
but the difference is it’s list comes from doing an ls on the data dirs. This was meant to catch <br />
the rare case where a ve configured but exists. <br />
<br />
== vw ==<br />
vw [veid]<br />
displays the output of ‘<tt>w</tt>’ (the equivalent of <tt>vzctl exec <veid> w</tt>) for each configured ve (as determined by <tt>/vzconf/*.conf</tt>). Useful for determing which ve is contributing to a heavily-loaded system.<br />
If passed an argument, gives ‘<tt>w</tt>‘ output for just that ve. <br />
Ex:<br />
<pre>[root@virt2 etc]# vw<br />
134<br />
10:52pm up 79 days, 6:14, 0 users, load average: 0.02, 0.02, 0.00<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT<br />
16027<br />
2:52pm up 7 days, 19:54, 0 users, load average: 0.00, 0.00, 0.00<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT<br />
16055<br />
2:52pm up 79 days, 6:38, 0 users, load average: 0.00, 0.04, 0.07<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT</pre><br />
<br />
== vwe ==<br />
vwe [constraint]<br />
just like <tt>vw</tt>, but takes a constraint as an argument, only show’s ve’s with loads >= the constraint provided. If no constraint is provided, 1 is used by default<br />
<br />
== vzs ==<br />
vzs [veid]<br />
displays the beancounter status for all ve’s, or a particular ve if an argument is passed<br />
<br />
== ve ==<br />
ve <veid><br />
the equivalent of: <tt>vzctl enter <veid></tt><br />
<br />
== vx ==<br />
vx <veid> <command><br />
the equivalent of: <tt>/usr/sbin/vzctl exec <veid> <command></tt><br />
<br />
== dprocs ==<br />
dprocs [count]<br />
a script which outputs a continuous report (or a certain number of reports if an option is passed) of processes stuck in the D state and which VPS’s those procs belong to.<br />
<br />
== setmem ==<br />
setmem <VEID> <256|512|768|1024|2048><br />
adjusts the memory resources for the VE<br />
<br />
== afacheck.sh ==<br />
afacheck.sh<br />
displays the health/status of containers and mirrors on an adaptec card (currently quar1, tempvirt1-2, virt9, virt10)- all other are LSI<br />
<br />
== backup ==<br />
backup<br />
backup script called nightly to update virt scripts and do customer backups<br />
<br />
== checkload.pl ==<br />
checkload.pl<br />
this was intended to be setup as a cronjob to watch processes on a virt when the load <br />
rises above a certain level. Not currently in use.<br />
<br />
== findbackuppigs.pl ==<br />
findbackuppigs.pl<br />
looks for files larger than 50MB which customers have asked us to backup. Emails matches<br />
to linux@johncompanies.com<br />
<br />
== gatherlinux.pl ==<br />
gatherlinux.pl<br />
gathers up data about ve’s configured and writes to a file. Used for audits against the db<br />
<br />
== gb ==<br />
gb <search><br />
greps backup.config for the given search parameter<br />
<br />
== gbg ==<br />
gbg <search><br />
greps backup.config for the given search parameter and presents just the directories (for clean pasting)<br />
<br />
== linuxtrafficgather.pl ==<br />
linuxtrafficgather.pl [yy-mm]<br />
sends a traffic usage summary by ve to support@johncomapnies.com and payments@johncompanies.com.<br />
Optional arguments are year and month (must be in the past). If not passed, assumes last month. Relies on <br />
traffic logs created by netstatreset and netstatbackup<br />
<br />
== linuxtrafficwatch.pl ==<br />
linuxtrafficwatch.pl<br />
checks traffic usage and emails support@johncomapnies.com when a ve reaches the warning level (35G) and<br />
the limit (40G). Works on virtuozzo versions <= 2.6.x. We really aren’t using this anymore now that we have netflow.<br />
<br />
== linuxtrafficwatch2.pl ==<br />
linuxtrafficwatch2.pl<br />
checks traffic usage and emails support@johncomapnies.com when a ve reaches the warning level (35G) and<br />
the limit (40G). Works on virtuozzo version 2.6.x. We really aren’t using this anymore now that we have netflow.<br />
<br />
== load.pl ==<br />
load.pl<br />
feeds info to load mrtg - executed by inetd.<br />
<br />
== mb (linux) ==<br />
mb <mount|umount><br />
(nfs) mounts and umounts dirs to backup2. Shortcuts are mbm and mbu to mount and unmount. <br />
<br />
== migrate ==<br />
migrate <ip of node migrating to> <veid> <target_veid> [target dir: vz | vz1 | vz2]<br />
this is basically a wrapper for <tt>vzmigrate</tt> – vzmigrate is a util to seamlessly move a ve from one host to another. This wrapper was written cause vz virtuozzo version 2.6 had a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables. This script mitigates that. Since it makes multiple ssh connections to the target host, it’s a good idea to put the pub key for the src system in the authorized_keys file on the target host. In addition, it emails ve owners when their migration starts and stops (if they place email addresses in a file on their system: /migrate_notify). To move everyone off a system, you’d do:<br />
for f in `vl`; do migrate <ip> $f; done<br />
<br />
== migrateonline ==<br />
migrateonline <ip of node migrating to> <veid> <target_veid> [target dir: vz | vz1 | vz2]<br />
this is the same as migrate but will migrate a ve in <tt>–online</tt> mode which means it won’t be shut down at the end of the migration. This only works when migrating ve’s between 2 machines running a 2.6 kernel (currently tempvirt1-2. virt16-19, virt12). If you get an error that the machine you’re trying to migrate to has a different CPU or features, etc, then you have to edit the file and add the –f switch to the vzmigrate line- you can basically ignore this kind of warning (but never ignore a warning about missing templates on the destination node). NOTE: This edit (if made to migrateonline) will be overwritten by the base script during each night’s backup.<br />
<br />
== netstatbackup ==<br />
DEPRECATED<br />
netstatbackup <br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.5.x. <br />
<br />
== netstatbackup2 ==<br />
DEPRECATED<br />
netstatbackup2<br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.6.x. <br />
<br />
== netstatreset ==<br />
DEPRECATED<br />
netstatreset<br />
writes traffic count data to a logfile and resets counters to 0. Works on virtuozzo versions 2.5.x <br />
<br />
== netstatreset2 ==<br />
DEPRECATED<br />
netstatreset2<br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.6.x. <br />
<br />
== orphanedbackupwatchlinux ==<br />
orphanedbackupwatchlinux <br />
looks for directories on backup2 which aren’t configured in backup.config and offers to <br />
delete them<br />
<br />
== rsync.backup (linux) ==<br />
rsync.backup<br />
does customer backups (relies on backup.config)<br />
<br />
== startvirt.pl ==<br />
startvirt.pl<br />
forks off start ve commands – keeps 6 running at a time. This is not to be used on systems where fastboot is enabled as it circumvents the benefit of the fastboot. The script will occasionally not exit gracefully and will continue to use up CPU, so it should be watched. Also, don’t exit from the script till you’re sure all ve’s are started – if you do you need to start them manually and may have to free up locks. On some systems, the startvirt script doesn’t exit cleanly and you have to ^C out of it. Be careful though- doing so can leave some VE’s in an odd bootup state and you may need to ‘vr’ them manually. You should check to see which ve’s aren’t running and/or confirm all have started when ^C’ing out of startvirt.<br />
<br />
== taskdone (linux) ==<br />
taskdone<br />
when called will email support@johncompanies.com with the hostname of the machine from which it was <br />
executed as the subject<br />
<br />
== vb (linux) ==<br />
vb<br />
the equivalent of: <tt>vi /usr/local/sbin/backup.config</tt><br />
<br />
== vemakeXX ==<br />
DEPRECATED<br />
vemakerh9 <br />
ve create script for RH9 (see vemake)<br />
<br />
vemakedebian30 <br />
ve create script for debian 3.0 (Woody) (see vemake)<br />
<br />
vemakedebian31 <br />
ve create script for debian 3.1 (Sarge) (see vemake)<br />
<br />
vemakedebian40 <br />
ve create script for debian 4.0 (Etch) (see vemake)<br />
<br />
vemakefedora, vemakefedora2, vemakefedora4, vemakefedora5, vemakefedora6, vemakefedora7<br />
ve create script for fedora core 1, 2, 4, 5, 6, 7 respectively (see vemake)<br />
<br />
vemakecentos3, vemakecentos4<br />
ve create script for centos 3, 4 respectively (see vemake)<br />
<br />
vemakesuse, vemakesuse93, vemakesuse100<br />
ve create script for suse 9.2, 9.3, 10.0 respectively (see vemake)<br />
<br />
vemakeubuntu5, vemakeubuntu606, vemakeubuntu606 vemakeubuntu704<br />
ve create script for ubuntu 5.10, 6.06, 6.10, 7.04 respectively (see vemake)<br />
<br />
== vemove ==<br />
DEPRECATED<br />
vemove <veid> <target_ip> </vz/private/123><br />
this script simplifies the old way of moving ve’s from one system to another - in short moving a ve to or from a virt running virtuozzo < 2.6.x<br />
It’s the equivalent of:<br />
<tt>tar cfpP - <veid> --ignore-failed-read | (ssh -2 -c arcfour <target_ip> "split - -b 1024m </vz/private/123>.tar" )</tt>This should only be used if migrate/vzmigrate can’t be used. <br />
<br />
== vim.watchdog ==<br />
vim.watchdog <br />
looks for and kills procs matching “vi|vim|nano|pine|elm” that are running for a long time & consuming lots of cpu. Works on virtuozzo versions 2.5.x<br />
<br />
== vim.watchdog2 ==<br />
vim.watchdog2<br />
looks for and kills procs matching “vi|vim|nano|pine|elm” that are running for a long time & consuming lots of cpu.<br />
Works on virtuozzo versions 2.6.x.<br />
<br />
== vzmigrate ==<br />
vzmigrate <target_ip> -r no <veid>:[dst veid]:[dst /vzX/private/veid]:[dst /vzX/root/veid]<br />
(this is the raw command “wrapped” by migrate/migrateonline) this will seamlessly move a ve from one host to another. The ve will run for the duration of the migration till the very end when it’s shut down, ip moved and started up on the target system. The filesystem on the src will remain. This should be watched – occasionally the move will timeout and leave the system shut down. If target private and root aren’t specified it just puts it in /vz. Only works when both systems are running virtuozzo 2.6.x<br />
<br />
== vztrafdump.sh ==<br />
DEPRECATED<br />
vztrafdump.sh<br />
writes traffic usage info by ve to a file called jc_traffic_dump in each ve’s / dir. Works on virtuozzo versions <= 2.5.x. <br />
<br />
== vztrafdump2.sh ==<br />
DEPRECATED<br />
vztrafdump2.sh<br />
writes traffic usage info by ve to a file called jc_traffic_dump in each ve’s / dir. Works on virtuozzo versions 2.6.x. <br />
<br />
== addtun ==<br />
addtun <veid><br />
Add’s tun device to ve.<br />
<br />
== bwcap ==<br />
bwcap <veid> <kbps><br />
Ex: <tt>bwcap 1234 512</tt><br />
Caps a VE’s bandwidth to the amount given<br />
<br />
== setdisk ==<br />
setdisk <veid> <diskspace in GB><br />
Ex: <tt>setdisk 1234 5</tt><br />
Gives a VE’s a given amount of disk space<br />
<br />
== vdf ==<br />
vdf <veid> <br />
the equivalent of: <tt>vzctl exec <veid> df –h</tt><br />
<br />
== vdff ==<br />
vdff<br />
runs a (condensed) vdf for all ve’s in your pwd (must be run from /vz/privateN)<br />
<br />
== mvbackups ==<br />
mvbackups <veid> <target_machine> (virt1) <target_dir> (vz1)<br />
moves backups from one location to another on the backup server, and provides you with option to remove entries from current backup.config, and simple paste command to add the config to backup.config on the target server<br />
<br />
== checkquota ==<br />
checkquota<br />
for all the ve’s in the cwd (run from /vz/private, /vz1/private, etc) reports what vz quota says they’re using and what the actual usage is (as reported by du)<br />
<br />
== clearquota ==<br />
clearquota <veid><br />
Recalculates a ve’s quota, prints out the usage before and after. The equivalent of:<br />
<tt>vdf <veid>; v stop <veid>; vzquota drop <veid>; v start <veid>; vdf <veid></tt><br />
<br />
== dprocs ==<br />
dprocs<br />
Sometimes the server’s have a large number of processes get stuck in the D state- this script shows (every 3 secs) which VE’s have D procs, which procs<br />
are stuck and a running average of the top “offenders”<br />
<br />
== vzstat ==<br />
vstat<br />
sort of like top for VZ. sort VEs by CPU usage by pressing 'o' and then 'c' keys<br />
<br />
== stopvirt ==<br />
stopvirt<br />
will stop VEs as fast as it can, 6 at a time. May not exit when complete so you should watch [[#vzstat|vzstat]] in another window.</div>Supporthttps://wiki.jcihosting.com/index.php?title=VPS_Management&diff=1033VPS Management2013-03-11T23:46:18Z<p>Support: /* Making new customer VE (vm) */</p>
<hr />
<div>= Common Problems =<br />
== Login to any machine without a password ==<br />
<br />
This is possible via the use of ssh keys. The process is thus:<br />
<br />
1. place the public key for your user (root@mail) in the /root/.ssh/authorized_keys file on the server you wish to login to<br />
cat /root/.ssh/id_dsa.pub<br />
(paste that into authorized_keys on the target server). If the file doesn't exist, create it.<br />
<br />
2. enable root login (usually only applies to FreeBSD). Edit the /etc/ssh/sshd_config on the target server and change:<br />
<tt>#PermitRootLogin no</tt><br />
to<br />
<tt>PermitRootLogin yes</tt><br />
<br />
3. Restart the sshd on the target machine. First, find the sshd process: <br />
jailps <hostname> | grep sshd <br />
or <br />
vp <VEID> | grep sshd<br />
<br />
Look for the process resembling:<br />
root 17296 0.0 0.0 5280 1036 ? Ss 2011 4:27 /usr/sbin/sshd <br />
(this is the sshd)<br />
<br />
Not:<br />
root 6270 0.5 0.0 6808 2536 ? Ss 14:33 0:00 sshd: root [priv]<br />
(this is an sshd child- someone already ssh'd in as root)<br />
<br />
Restart the sshd: <br />
kill -1 <PID><br />
<br />
Ex:<br />
kill -1 17296<br />
<br />
You may now ssh in.<br />
<br />
Once you're done, IF you enabled root login, you should repeat steps 2 and 3 to disable root logins.<br />
<br />
== Letting someone in who has locked themselves out (killed sshd, lost pwd) ==<br />
<br />
There are two ways people frequently lock themselves out - either they forget a password, or they kill off sshd somehow.<br />
<br />
These are actually both fairly easy to solve. First, let's say someone kills off their sshd, or somehow mangles /etc/ssh/sshd_config such that it no longer lets them in.<br />
<br />
Their email may be very short, or it may have all sorts of details about how you should fix sshd_config to let them in ... just ignore all of this. They can fix their own mangled sshd. Fixing this is very simple. First, edit the /etc/inetd.conf on their system and uncomment the telnet line:<br />
<br />
telnet stream tcp nowait root /usr/libexec/telnetd telnetd<br />
#telnet stream tcp6 nowait root /usr/libexec/telnetd telnetd<br />
<br />
(just leave the tcp6 version of telnet commented)<br />
<br />
Then, use jailps to list the processes on their system, and find their inetd process. Then simply:<br />
<br />
kill -HUP (pid)<br />
<br />
where (pid) is the PID of their inetd process. Now they have telnet running on their system and they can log in and do whatever they need to do.<br />
<br />
The only complications that could occur are:<br />
<br />
a) their firewall config on our firewall has port 23 blocked, in which case you will need to open that - will be covered in a different lesson.<br />
<br />
b) they are not running inetd, so you can't HUP it. If this happens, edit their /etc/rc.conf, add the inetd_enable="YES" line, and then kill<br />
their jail with /tmp/jailkill.pl - then restart their jail with the jail line from their quad/safe file. Easy.<br />
<br />
If they have forgotten a password,<br />
<br />
On 6.x+ you can reset their password with:<br />
jexec <jailID from jls> passwd root<br />
<br />
Note: the default password for 6.x jails is 8ico2987, for 4.x it is p455agfa<br />
<br />
On 4.x, you need to cd to their etc directory<br />
... for instance:<br />
<br />
cd /mnt/data2/198.78.65.136-col00261-DIR/etc<br />
<br />
and run:<br />
<br />
vipw -d .<br />
<br />
Then paste in these two lines (theres a paste with these):<br />
<br />
root:$1$krszPxhk$xkCepSnz3mIikT3vCtJCt0:0:0::0:0:Charlie &:/root:/bin/csh<br />
user:$1$Mx9p5Npk$QdMU6c8YQqp2FW2M3irEh/:1001:1001::0:0:User &:/home/user:/bin/sh<br />
<br />
overwriting the lines they already have for "user" and "root" - then just tell them that both user and root have been reset to the default password of p455agfa.<br />
<br />
For linux, just passwd inside shell or <br />
vzctl set <veid> --userpasswd root:p455agfa –save<br />
<br />
Starting in 2009 we began giving out randomized passwords for FreeBSD and Linux as the default password. That is stored with each system in Mgmt. You should look for and reset the password to that password in the event of a reset and refer the customer to use their original password from their welcome email- this way we don’t have to send the password again via email (in clear text).<br />
<br />
<br />
== sendmail can’t be contacted from ext ip (only locally) ==<br />
<br />
By default redhat puts this line in sendmail.mc:<br />
<br />
DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')<br />
<br />
which makes it only answer on localhost. Comment it out like:<br />
<br />
dnl DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')<br />
<br />
and then rebuild sendmail.cf with:<br />
<br />
m4 /etc/mail/sendmail.mc > /etc/sendmail.cf<br />
<br />
== virt doesn’t properly let go of ve’s ip(s) when moved to another system ==<br />
<br />
On virtuozzo 2.6 systems, it's been observed that when moving ips from one virt to another that sometimes the routing table will not get updated to reflect the removal of the ip addresses.<br />
<br />
A recent example was a customer that was moving to a new ve on a new virt and the ip addresses were traded between the two ve's. After the trade the two systems were not able to talk to each other. When looking at the routing table for the old system all the ip addresses were still in the routing table as being local, like so:<br />
<br />
<pre>netstat -rn | grep 69.55.225.149<br />
69.55.225.149 0.0.0.0 255.255.255.255 UH 40 0 0 venet0</pre><br />
<br />
This was preventing traffic to the other system from being routed properly.<br />
The solution is to manually delete the route:<br />
<br />
route delete 69.55.225.149 gw 0.0.0.0<br />
<br />
Supposedly, this was fixed in 2.6.1<br />
<br />
== sshd on FreeBSD 6.2 segfaults ==<br />
<br />
First try to reinstall ssh<br />
<br />
<pre>cd /usr/src/secure<br />
cd lib/libssh<br />
make depend && make all install<br />
cd ../../usr.sbin/sshd<br />
make depend && make all install<br />
cd ../../usr.bin/ssh<br />
make depend && make all install</pre><br />
<br />
Failing that, find the library that’s messed up:<br />
<br />
<pre>ldd /usr/sbin/sshd<br />
libssh.so.3 => /usr/lib/libssh.so.3 (0x280a3000) <br />
libutil.so.5 => /lib/libutil.so.5 (0x280d8000) <br />
libz.so.3 => /lib/libz.so.3 (0x280e4000) <br />
libwrap.so.4 => /usr/lib/libwrap.so.4 (0x280f5000) <br />
libpam.so.3 => /usr/lib/libpam.so.3 (0x280fc000) <br />
libbsm.so.1 => /usr/lib/libbsm.so.1 (0x28103000) <br />
libgssapi.so.8 => /usr/lib/libgssapi.so.8 (0x28112000) <br />
libkrb5.so.8 => /usr/lib/libkrb5.so.8 (0x28120000) <br />
libasn1.so.8 => /usr/lib/libasn1.so.8 (0x28154000) <br />
libcom_err.so.3 => /usr/lib/libcom_err.so.3 (0x28175000) <br />
libroken.so.8 => /usr/lib/libroken.so.8 (0x28177000) <br />
libcrypto.so.4 => /lib/libcrypto.so.4 (0x28183000) <br />
libcrypt.so.3 => /lib/libcrypt.so.3 (0x28276000) <br />
libc.so.6 => /lib/libc.so.6 (0x2828e000) <br />
libmd.so.3 => /lib/libmd.so.3 (0x28373000)</pre><br />
<br />
md5 them and compare to other jail hosts or jails running on host<br />
<br />
for libcrypto reinstall:<br />
<pre>/usr/src/crypto<br />
make depend && make all install</pre><br />
<br />
= FreeBSD VPS =<br />
<br />
== Starting jails: Quad/Safe Files ==<br />
<br />
FreeBSD customer systems do not start up automatically at boot time. When one of our freebsd machines boots up, it boots up, and does nothing else. To start jails, we put the commands to start each jail into a shell script(s) and run the script(s). Jail startup is something that needs to be actively monitored, which is why we don’t just run the script automatically. More on monitoring later.<br />
<br />
NOTE: >=7.x we have moved to 1 quad file: <tt>quad1</tt>. Startups are not done by running each quad, but rather [[#startalljails|startalljails]] which relies on the contents of <tt>quad1</tt>. The specifics of this are lower in this article. What follows here applies for pre 7.x systems.<br />
<br />
There are eight files in <tt>/usr/local/jail/rc.d</tt>:<br />
<br />
<pre>jail3# ls /usr/local/jail/rc.d/<br />
quad1 quad2 quad3 quad4 safe1 safe2 safe3 safe4<br />
jail3#</pre><br />
<br />
four quad files and four safe files.<br />
<br />
Each file contains an even number of system startup blocks (total number of jails divided by 4)<br />
<br />
The reason for this is, if we make one large script to startup all the systems at boot time, it will take too long - the first system in the script will start up right after system boot, which is great, but the last system may not start for another 20 minutes.<br />
<br />
Since there is no way to parralelize this during the startup procedure, we simply open four terminals (in screen window 9) and run each script, one in each terminal. This way they all run simultaneously, and the very last system in each startup script gets started in 1/4th the time it would if there was one large file<br />
<br />
The files are generally organized so that quad/safe 1&2 have only jails from disk 1, and quad/safe 3&4 have jails from disk 2. This helps ensure that only 2 fscks on any disk are going on at once. Further, they are balanced so that all quad/safe’s finish executing around the same time. We do this by making sure each quad/safe has a similar number of jails and represents a similar number of inodes (see js).<br />
<br />
The other, very important reason we do it this way, and this is the reason there are quad files and safe files, is that in the event of a system crash, every single vn-backed filesystem that was mounted at the time of system crash needs to be fsck'd. However, fsck'ing takes time, so if we shut the system down gracefully, we don't want to fsck.<br />
<br />
Therefore, we have two sets of scripts - the four quad scripts are identical to the four safe scripts except for the fact that the quad scripts contain fsck commands for each filesystem.<br />
<br />
So, if you shut a system down gracefully, start four terminals and run safe1 in window one, and safe2 in window 2, and so on.<br />
<br />
If you crash, start four terminals (or go to screen window 9) and run quad1 in window one, and quad2 in window 2, and so on.<br />
<br />
Here is a snip of (a 4.x version) quad2 from jail17:<br />
<br />
<pre>vnconfig /dev/vn16 /mnt/data2/69.55.228.7-col00820<br />
fsck -y /dev/vn16<br />
mount /dev/vn16c /mnt/data2/69.55.228.7-col00820-DIR<br />
chmod 0666 /mnt/data2/69.55.228.7-col00820-DIR/dev/null<br />
jail /mnt/data2/69.55.228.7-col00820-DIR mail1.phimail.com 69.55.228.7 /bin/sh /etc/rc<br />
<br />
# moved to data2 col00368<br />
#vnconfig /dev/vn28 /mnt/data2/69.55.236.132-col00368<br />
#fsck -y /dev/vn28<br />
#mount /dev/vn28c /mnt/data2/69.55.236.132-col00368-DIR<br />
#chmod 0666 /mnt/data2/69.55.236.132-col00368-DIR/dev/null<br />
#jail /mnt/data2/69.55.236.132-col00368-DIR limehouse.org 69.55.236.132 /bin/sh /etc/rc<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
vnconfig /dev/vn22 /mnt/data2/69.55.228.13-col01063<br />
fsck -y /dev/vn22<br />
mount /dev/vn22c /mnt/data2/69.55.228.13-col01063-DIR<br />
chmod 0666 /mnt/data2/69.55.228.13-col01063-DIR/dev/null<br />
jail /mnt/data2/69.55.228.13-col01063-DIR www.widestream.com.au 69.55.228.13 /bin/sh /etc/rc<br />
<br />
# cancelled col00106<br />
#vnconfig /dev/vn15 /mnt/data2/69.55.238.5-col00106<br />
#fsck -y /dev/vn15<br />
#mount /dev/vn15c /mnt/data2/69.55.238.5-col00106-DIR<br />
#chmod 0666 /mnt/data2/69.55.238.5-col00106-DIR/dev/null<br />
#jail /mnt/data2/69.55.238.5-col00106-DIR mail.azebu.net 69.55.238.5 /bin/sh /etc/rc</pre><br />
<br />
As you can see, two of the systems specified are commented out - presumably those customers cancelled, or were moved to new servers.<br />
<br />
As you can see, the vnconfig line is the simpler command line, not the longer one that was used when it was first configured. As you can see, all that is done is, vnconfig the filesystem, then fsck it, then mount it. The fourth command is the `jail` command used to start the system – but that will be covered later.<br />
<br />
Here is the safe2 file from jail17:<br />
<br />
<pre>vnconfig /dev/vn16 /mnt/data2/69.55.228.7-col00820<br />
mount /dev/vn16c /mnt/data2/69.55.228.7-col00820-DIR<br />
chmod 0666 /mnt/data2/69.55.228.7-col00820-DIR/dev/null<br />
jail /mnt/data2/69.55.228.7-col00820-DIR mail1.phimail.com 69.55.228.7 /bin/sh /etc/rc<br />
<br />
# moved to data2 col00368<br />
#vnconfig /dev/vn28 /mnt/data2/69.55.236.132-col00368<br />
#mount /dev/vn28c /mnt/data2/69.55.236.132-col00368-DIR<br />
#chmod 0666 /mnt/data2/69.55.236.132-col00368-DIR/dev/null<br />
#jail /mnt/data2/69.55.236.132-col00368-DIR limehouse.org 69.55.236.132 /bin/sh /etc/rc<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
vnconfig /dev/vn22 /mnt/data2/69.55.228.13-col01063<br />
mount /dev/vn22c /mnt/data2/69.55.228.13-col01063-DIR<br />
chmod 0666 /mnt/data2/69.55.228.13-col01063-DIR/dev/null<br />
jail /mnt/data2/69.55.228.13-col01063-DIR www.widestream.com.au 69.55.228.13 /bin/sh /etc/rc<br />
<br />
# cancelled col00106<br />
#vnconfig /dev/vn15 /mnt/data2/69.55.238.5-col00106<br />
#mount /dev/vn15c /mnt/data2/69.55.238.5-col00106-DIR<br />
#chmod 0666 /mnt/data2/69.55.238.5-col00106-DIR/dev/null<br />
#jail /mnt/data2/69.55.238.5-col00106-DIR mail.azebu.net 69.55.238.5 /bin/sh /etc/rc</pre><br />
<br />
As you can see, it is exactly the same, but it does not have the fsck lines.<br />
<br />
Take a look at the last entry - note that the file is named:<br />
<br />
/mnt/data2/69.55.238.5-col00106<br />
<br />
and the mount point is named:<br />
<br />
/mnt/data2/69.55.238.5-col00106-DIR<br />
<br />
This is the general format on all the FreeBSD systems. The file is always named:<br />
<br />
IP-custnumber<br />
<br />
and the directory is named:<br />
<br />
IP-custnumber-DIR<br />
<br />
If you run safe when you need a fsck, the mount will fail and jail will fail:<br />
<br />
# mount /dev/vn1c /mnt/data2/jails/65.248.2.131-ns1.kozubik.com-DIR<br />
mount: /dev/vn1c: Operation not permitted<br />
<br />
No reboot needed, just run the quad script<br />
<br />
Starting with 6.x jails, we added block delimiters to the quad/safe files, the block looks like:<br />
<br />
<pre>echo '## begin ##: nuie.solaris.mu'<br />
fsck -y /dev/concat/v30v31a<br />
mount /dev/concat/v30v31a /mnt/data1/69.55.228.218-col01441-DIR<br />
mount_devfs devfs /mnt/data1/69.55.228.218-col01441-DIR/dev<br />
devfs -m /mnt/data1/69.55.228.218-col01441-DIR/dev rule -s 3 applyset<br />
jail /mnt/data1/69.55.228.218-col01441-DIR nuie.solaris.mu 69.55.228.218 /bin/sh /etc/rc<br />
echo '## end ##: nuie.solaris.mu'</pre><br />
<br />
These are more than just informative when running quad/safe’s, the echo lines MUST be present for certain tools to work properly. So it’s important that any updates to the hostname also be updated on the 2 echo lines. For example, if you try to startjail a jail with a hostname which is on the jail line but not the echo lines, the command will return with host not found.<br />
<br />
=== FreeBSD 7.x+ notes ===<br />
<br />
Starting with the release of FreeBSD 7.x, we are doing jail startups in a slightly different way. First, thereis only 1 file: <tt>/usr/local/jail/rc.d/quad1</tt><br />
There are no other quads or corresponding safe files. The reason for this is twofold, 1. We can pass –C to fsck which will tell is to skip the fsck if the fs is clean (no more need for safe files), 2. We have a new startup script which can be launched multiple times, running in parallel to start jails, where quad1 is the master jail file. <br />
Quad1 could still be run as a shell script, but it would take a very long time for it to run completely so it’s not advisable; or you should break it down into smaller chunks (like quad1, quad2, quad3, etc)<br />
<br />
Here is a snip of (a 7.x version) quad1 from jail2:<br />
<br />
<pre>echo '## begin ##: projects.tw.com'<br />
mdconfig -a -t vnode -f /mnt/data1/69.55.230.46-col01213 -u 50<br />
fsck -Cy /dev/md50c<br />
mount /dev/md50c /mnt/data1/69.55.230.46-col01213-DIR<br />
mount -t devfs devfs /mnt/data1/69.55.230.46-col01213-DIR/dev<br />
devfs -m /mnt/data1/69.55.230.46-col01213-DIR/dev rule -s 3 applyset<br />
jail /mnt/data1/69.55.230.46-col01213-DIR projects.tw.com 69.55.230.46 /bin/sh /etc/rc<br />
echo '## end ##: projects.tw.com'</pre><br />
<br />
Cancelled jails are no longer commented out and stored in quad1, rather they’re moved to <tt>/usr/local/jail/rc.d/deprecated</tt> <br />
<br />
To start these jails, start the 4 ssh sessions as you would for a normal crash and then instead of running quad1-4, instead run startalljails in each window. IMPORTANT- before running startalljails you should make sure you ran preboot once as it will clear out all the lockfiles and enable startalljails to work properly.<br />
<br />
== Problems with the quad/safe files ==<br />
<br />
When you run the quad/safe files, there are two problems that can occur - either a particular system will hang during initialization, OR a system will spit out output to the screen, impeding your ability to do anything. Or both.<br />
<br />
First off, when you start a jail, you see output like this:<br />
<br />
<pre>Skipping disk checks ...<br />
adjkerntz[25285]: sysctl(put_wallclock): Operation not permitted<br />
Doing initial network setup:.<br />
ifconfig: ioctl (SIOCDIFADDR): permission denied<br />
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384<br />
Additional routing options: TCP keepalive=YESsysctl:<br />
net.inet.tcp.always_keepalive: Operation not permitted.<br />
Routing daemons:.<br />
Additional daemons: syslogd.<br />
Doing additional network setup:.<br />
Starting final network daemons:.<br />
ELF ldconfig path: /usr/lib /usr/lib/compat /usr/X11R6/lib /usr/local/lib<br />
a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout /usr/X11R6/lib/aout<br />
Starting standard daemons: inetd cron sshd sendmail sendmail-clientmqueue.<br />
Initial rc.i386 initialization:.<br />
Configuring syscons: blanktime.<br />
Additional ABI support:.<br />
Local package initialization:.<br />
Additional TCP options:.</pre><br />
<br />
Now, let's look at this line, near the end:<br />
<br />
Local package initialization:.<br />
<br />
This is where a list of daemons that are set to start at boot time willshow up. You might see something like:<br />
<br />
Local package initialization: mysqld apache sendmail sendmail-clientmqueue<br />
<br />
Or something like this:<br />
<br />
Local package initialization: postgres postfix apache<br />
<br />
The problem is that many systems (about 4-5 per machine) will hang on that line. Basically it will get to some of the way through the total daemons to be started:<br />
<br />
Local package initialization: mysqld apache<br />
<br />
and will just sit there. Forever.<br />
<br />
Fortunately, pressing ctrl-c will break out of it. Not only will it break out of it, but it will also continue on that same line and start the other daemons:<br />
<br />
Local package initialization: mysqld apache ^c sendmail-clientmqueue<br />
<br />
and then continue on to finish the startup, and then move to the next system to be started.<br />
<br />
So what does this mean? It means that if a machine crashes, and you start four screen-windows to run four quads or four safes, you need to periodically cycle between them and see if any systems are stuck at that point, causing their quad/safe file to hang. A good rule of thumb is, if you see a system at that point in the startup, give it another 100 seconds - if it is still at the exact same spot, hit ctrl-c. Its also a good idea to go back into the quad file (just before the first command in the jail startup block) and note that this jail tends to need a control-c or more time as follows:<br />
<pre>echo '### NOTE ### slow sendmail'<br />
echo '### NOTE ###: ^C @ Starting sendmail.'</pre><br />
<br />
'''NEVER''' hit ctrl-c repeatedly if you don't get an immediate response - that will cause the following jail’s startup commands to be aborted.<br />
<br />
A second problem that can occur is that a jail - maybe the first one in that particular quad/safe, maybe the last one, or maybe one in the middle, will start spitting out status or error messages from one of its init scripts. This is not a problem - basically, hit enter a few times and see if you get a prompt - if you do get a prompt, that means that the quad/safe script has already completed. Therefore it is safe to log out (and log out of the user that you su'd from) and then log back in (if necessary).<br />
<br />
The tricky thing is, if a system in the middle starts flooding with messages, and you hit enter a few times and don't get a prompt. Are you not getting a prompt because some subsequent system is hanging at the initialization, as we discussede above ? Or are you not getting a prompt because that quad file is currently running an fsck ? Usually you can tell by scrolling back in screen’s history to see what it was doing before you started getting the messages.<br />
<br />
If you don’t get clues from history, you have to use your judgement - instead of giving it 100 seconds to respond, perhaps give it 2-3 mins ... if you still get no response (no prompt) when you hit enter, hit ctrl-c. However, be aware that you might still be hitting ctrl-c in the middle of an fsck. This means you will get an error like "filesystem still marked dirty" and then the vnconfig for it will fail and so will the jail command, and the next system in the quad file will then start starting up.<br />
<br />
If this happens, just wait until the end of all the quad files have finished, and start that system manually.<br />
<br />
If things really get weird, like a screen flooded with errors, and you can't get a prompt, and ctrl-c does nothing, then you need to just eventually (give it ten mins or so) just kill that window with ctrl-p, then k, and then log in again and manually check which systems are now running and which aren't, and manually start up any that are not.<br />
<br />
Don't EVER risk running a particular quad/safe file a second time.<br />
If the quad/safe script gets executed twice, reboot the machine immediately.<br />
<br />
So, for all the above reasons, anytime a machine crashes and you run all the quads or all the safes, '''always''' check every jail afterwards to make sure it is running - even if you have no hangs or complications at all.<br />
Run this command:<br />
<br />
<tt>[[#jailpsall|jailpsall]]</tt> <br />
<br />
Note: [[#postboot|postboot]] also populates ipfw counts, so it '''should not be run multiple times''', use <tt>jailpsall</tt> for subsequent extensive ps’ing<br />
<br />
And make sure they all show as running. If one does not show as running, check its /etc/rc.conf file first to see if maybe it is using a different hostname first before starting it manually.<br />
<br />
One thing we have implemented to alleviate these startup hangs and noisy jails, is to put jail start blocks that are slow or hangy at the bottom of the safe/quad file. Further, for each bad jail we note in each quad/safe just before the start block something like:<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
<br />
That way we’ll be prepared to ^C when we see that message appear during the quad/safe startup process. If you observe a new, undocumented hang, '''after''' the quad/safe has finished, place a line similar to the above in the quad file, move the jail start block to the end of the file, then run [[#buildsafe|buildsafe]]<br />
<br />
<br />
== Recovering from a crash (FreeBSD) ==<br />
<br />
=== Diagnose whether you have a crash ===<br />
The most important thing is to get the machine and all jails back up as soon as possible. Note the time, you’ll need to create a crash log entry (Mgmt. -> Reference -> CrashLog). The first thing to do is head over to the [[Screen#Screen_Organization|serial console screen]] and see if there’s any kernel error messages output. Try to copy any messages (or just a sample of repeating messages) you see into the notes section of the crash log. If there are no messages, the machine may just be really busy- wait a bit (5-10min) to see if it comes back. If it's still pinging, odds are its very busy. Note, if you see messages about swap space exhausted, the server is obviously out of memory, however it may recover briefly enough for you to get a jtop in to see who's lauched a ton of procs (most likely) and then issue a quick jailkill to get it back under control.<br />
<br />
If it doesn't come back, or the messages indicate a fatal error, you will need to proceed with a power cycle (ctrl+alt+del will not work).<br />
<br />
=== Power cycle the server ===<br />
If this machine is not a Dell 2950 with a [[DRAC/RMM#DRAC|DRAC card]] (i.e. if you can’t ssh into the DRAC card (as root, using the standard root pass) and issue <br />
racadm serveraction hardreset<br />
then you will need someone at the data center power the macine off, wait 30 sec, then turn it back on. Make sure to re-attach via console:<br />
tip jailX<br />
immediately after power down. <br />
<br />
=== (Re)attach to the console ===<br />
Stay on the console the entire time during boot. As the BIOS posts- look out for the RAID card output- does everything look healthy? The output may be scrambled, look for "DEGRADED" or "FAILED". Once the OS starts booting you will be disconnected (dropped back to the shell on the console server) a couple times during the boot up. The reason you want to quickly re-attach is two-fold: 1. If you don’t reattach quickly then you won’t get any console output, 2. you want to be attached before the server ''potentially'' starts (an extensive) fsck. If you attach after the fsck begins, you’ll have seen no indication it started an fsck and the server will appear frozen during startup- no output, no response. <br />
<br />
IMPORTANT NOTE: on some older FreeBSD systems, there will be no output to the video (KVM) console as it boots up. The console output is redirected to the serial port ... so if a jail crashes, and you attach a kvm, the output during the bootup procedure will not be shown on the screen. However, when the bootup is done, you will get a login prompt on the screen and will be able to log in as normal. <tt>/boot/loader.conf</tt> is where serial console redirect output lives, so comment that if you want to catch output on kvm.<br />
On newer systems it sends most output to both locations. <br />
<br />
=== Assess the heath of the server ===<br />
Once the server boots up fully, you should be able to ssh in. Look around- make sure all the mounts are there and reporting the correct size/usage (i.e. /mnt/data1 /mnt/data2 /mnt/data3 - look in /etc/fstab to determine which mount points should be there), check to see if RAID mirrors are healthy. See [[RAID_Cards#Common_CLI_commands_.28megacli.29|megacli]], [[#aaccheck|aaccheck]]<br />
<br />
Before you start the jails, you need to run [[#preboot|preboot]]. This will do some assurance checks to make sure things are prepped to start the jails. Any issues that come out of preboot need to be addressed before starting jails.<br />
<br />
=== Start jails ===<br />
[[#Starting_jails:_Quad.2FSafe_Files|More on starting jails]]<br />
Customer jails (the VPSs) do not start up automatically at boot time. When a FreeBSD machines boots up, it boots up, and does nothing else. To start jails, we put the commands to start each jail into a shell script(s) and run the script(s). Jail startup is something that needs to be actively monitored, which is why we don’t just run the script automatically. <br />
<br />
In order to start jails, we run the quad files: quad1 quad2 quad3 and quad4 (on new systems there is only quad1). If the machine was cleanly rebooted- which wouldn't be the case if this was a crash, you may run the safe files (safe1 safe2 safe3 safe4) in lieu of quads. <br />
<br />
Open up 4 logins to the server (use the windows in [[Screen#Screen_Organization|a9]])<br />
In each of the 4 windows you will:<br />
<br />
If there is a [[#startalljails|startalljails]] script (and only quad1), run that command in each of the 4 windows. It will parse through the quad1 file and start each jail. Follow the instructions [[#Problems_with_the_quad.2Fsafe_files|here]] for monitoring startup. Note that you can be a little more lenient with jails that take awhile to start- startalljails will work around the slow jails and start the rest. As long as there aren't 4 jails which are "hung" during startup, the rest will get started eventually.<br />
-or-<br />
If there is no startalljails script, there will be multiple quad files. In each of the 4 windows, start each of the quads. i.e. start quad1 in window1, quad2 in window2 and so on. DO NOT start any quad twice. It will crash the server. If you accidentally do this, just jailkill all the jails which are in the quad and run the quad again. Follow the instructions here for monitoring quad startup.<br />
<br />
Note the time the last jail boots- this is what you will enter in the crash log.<br />
<br />
Save the crash log.<br />
<br />
=== Check to make sure all jails have started ===<br />
There's a simple script which will make sure all jails have started, and enter the ipfw counter rules: [[#postboot|postboot]] <br />
Run postboot, which will do a jailps on each jail it finds (excluding commented out jails) in the quad file(s). We're looking for 2 things:<br />
# systems spawning out of control or too many procs<br />
# jails which haven't started<br />
On 7.x and newer systems it will print out the problems (which jails haven't started) at the conclusion of postboot. <br />
On older systems you will need to watch closely to see if/when there's a problem, namely:<br />
<br />
[hostname] doesnt exist on this server<br />
<br />
When you get this message, it means one of 2 things:<br />
1. the jail really didn't start:<br />
When a jail doesn't start it usually boils down to a problem in the quad file. Perhaps the path name is wrong (data1 vs data2) or the name of the vn/mdfile is wrong. Once this is corrected, you will need to run the commands from the quad file manually, or you may use <tt>startjail <hostname></tt><br />
<br />
2. the customer has changed their hostname (and not told us) so their jail ''is'' running, just under a different hostname:<br />
On systems with jls, this is easy to rectify. First, get the customer info: <tt>g <hostname></tt><br />
Then look for the customer in jls: <tt>jls | grep <col0XXXX></tt><br />
From there you will see their new hostname- you should update that hostname in the quad file: don't forget to edit it on the <tt>## begin ##</tt> and <tt>## end ##</tt> lines, and in mgmt. <br />
On older systems without jls, this will be harder, you will need to look further to see their hostname- perhaps its in their /etc/rc.conf<br />
<br />
<br />
Once all jails are started, do some spot checks- try to ssh or browse to some customers, just to make sure things are really ok.<br />
<br />
== Adding disk to a 7.x/8.x jail ==<br />
or<br />
== Moving customer to a different drive (md) ==<br />
<br />
NOTE: this doesn’t apply to mx2 which uses gvinum. Use same procedure as 6.x<br />
NOTE: if you unmount before mdconfig, re-mdconfig (attach) then unmount then mdconfig -u again <br />
<br />
<br />
(parts to change/customize are <tt><span style="color:red">red</span></tt>)<br />
<br />
<br />
If someone wants more disk space, there’s a paste for it, send it to them (explains about downtime, etc).<br />
<br />
1. Figure out the space avail from <tt>js</tt>. Ideally, you want to put the customers new space on a different partition (and create the new md on the new partition). <br />
<br />
<br />
2. make a mental note of how much space they're currently using<br />
<br />
<br />
3. <tt>jailkill <hostname></tt> <br />
<br />
<br />
4. Umount it (including their devfs) but leave the md config’d (so if you use stopjail, you will have to re-mdconfig it)<br />
<br />
<br />
5. <tt>g <customerID></tt> to get the info (IP/cust#) needed to feed the new mdfile and mount name, and to see the current md device. <br />
<br />
<br />
6a. When there's enough room to place new system on an alternate, or the same drive:<br />
USE CAUTION not to overwrite (touch, mdconfig) existing md!!<br><br />
<tt>touch /mnt/data<span style="color:red">3/69.55.234.66-col01334</span><br><br />
mdconfig -a -t vnode -s 10g -f /mnt/data3/69.55.234.66-col01334 -u 97<br><br />
newfs /dev/md97</tt><br />
<br />
<br />
Optional- if new space is on a different drive, move the mount point directory AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
<tt>mount /dev/md<span style="color:red">97</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<br />
confirm you are mounted to /dev/md<span style="color:red">97</span> and space is correct:<br><br />
<tt>df .</tt> <br />
<br />
<br />
do the dump and pipe directly to restore:<br><br />
<tt>dump -0a -f - /dev/md<span style="color:red">1</span> | restore -r -f - <br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use <tt>df</tt> to confirm the restored data size matches the original usage figure<br />
<br />
<br />
md-unconfig old system:<br><br />
<tt>mdconfig -d -u <span style="color:red">1</span></tt><br />
<br />
<br />
archive old mdfile. <br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.237.26-col00241</span> /mnt/data<span style="color:red">1/old-col00241-mdfile-noarchive-20091211</span></tt><br />
<br />
<br />
edit quad (vq1) to point to new (/mnt/data<span style="color:red">3</span>) location AND new md number (md<span style="color:red">97</span>)<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
6b. When there's not enough room on an alternate partition, or on the same drive...but there is enough room if you were to remove the existing customer's space:<br />
<br />
<br />
mount backup nfs mounts:<br><br />
<tt>mbm</tt><br> <br />
(run <tt>df</tt> to confirm backup mounts are mounted)<br />
<br />
<br />
dump the customer to backup2 or backup1<br><br />
<tt>dump -0a -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> /dev/md<span style="color:red">1</span></tt><br><br />
(when complete WITHOUT errors, <tt>du</tt> the dump file to confirm it matches size, roughly, with usage)<br />
<br />
<br />
unconfigure and remove old mdfile<br><br />
<tt>mdconfig -d -u <span style="color:red">1</span><br><br />
rm /mnt/data<span style="color:red">1/69.55.237.26-col00241</span></tt><br><br />
(there should now be enough space to recreate your bigger system. If not, run sync a couple times)<br />
<br />
<br />
create the new system (ok to reuse old mdfile and md#):<br><br />
<tt>touch /mnt/data<span style="color:red">1/69.55.234.66-col01334</span><br><br />
mdconfig -a -t vnode -s <span style="color:red">10</span>g -f /mnt/data<span style="color:red">1/69.55.234.66-col01334</span> -u <span style="color:red">1</span><br><br />
newfs /dev/md<span style="color:red">1</span><br><br />
mount /dev/md<span style="color:red">1</span> /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> <br><br />
cd /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<br />
confirm you are mounted to /dev/md<span style="color:red">1</span> and space is correct:<br><br />
<tt>df .</tt> <br />
<br />
<br />
do the restore from the dumpfile on the backup server:<br><br />
<tt>restore -r -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> .<br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
umount nfs:<br><br />
<tt>mbu</tt><br />
<br />
<br />
If md# changed (or mount point), edit quad (<tt>vq1</tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new md number (md<span style="color:red">1</span>)<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
7. update disk (and dir if applicable) in mgmt screen<br />
<br />
<br />
8. update backup list AND move backups, if applicatble<br />
<br />
Ex: <tt>mvbackups <span style="color:red">69.55.237.26-col00241</span> jail<span style="color:red">9</span> data<span style="color:red">3</span></tt><br />
<br />
<br />
9. Optional: archive old mdfile<br />
<br />
<tt>mbm<br><br />
gzip -c old-col01588-mdfile-noarchive-20120329 > /deprecated/old-col01588-mdfile-noarchive-20120329.gz<br><br />
mbu<br><br />
rm old-col01588-mdfile-noarchive-20120329</tt><br />
<br />
== Adding disk to a 6.x jail (gvinum/gconcat) ==<br />
or<br />
== Moving customer to a different drive (gvinum/gconcat) ==<br />
<br />
(parts to change are <span style="color:red">highlighted</span>)<br />
<br />
<br />
If someone wants more disk space, there’s a paste for it, send it to them (explains about downtime, etc).<br />
<br />
<br />
1. Figure out the space avail from [[#js|js]]. Ideally, you want to put the customers new space on a different partition (and create the new md on the new partition). <br />
<br />
<br />
2. make a mental note of how much space they're currently using<br />
<br />
<br />
3. <tt>[[#stopjail|stopjail]] <hostname></tt> <br />
<br />
<br />
4. <tt>[[#g|g]] <customerID></tt> to get the info (IP/cust#) needed to feed the new mount name and existing volume/device. <br />
<br />
<br />
5a. When there's enough room to place new system on an alternate, or the same drive (using only UNUSED - including if it's in use by the system in question - gvinum volumes):<br />
<br />
<br />
Configure the new device:<br><br />
A. for a 2G system (single gvinum volume):<br><br />
<tt>bsdlabel -r -w /dev/gvinum/v<span style="color:red">123</span><br><br />
newfs /dev/gvinum/v<span style="color:red">123</span>a</tt><br><br />
<br />
-or- <br />
<br />
<br />
B. for a >2G system (create a gconcat volume):<br><br />
try to grab a contiguous block of gvinum volumes. gconcat volumes MAY NOT span drives (i.e. you cannot use a gvinum volume from data3 and a volume from data2 in the same gconcat volume). <br />
<br />
<tt>gconcat label <span style="color:red">v82-v84 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84</span><br><br />
bsdlabel -r -w /dev/concat/<span style="color:red">v82-v84</span><br><br />
newfs /dev/concat/<span style="color:red">v82-v84</span>a</tt><br />
<br />
<br />
Other valid gconcat examples:<br><br />
<tt>gconcat label v82-v84v109v112 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84 /dev/gvinum/v109 /dev/gvinum/v112<br><br />
gconcat label v82v83 /dev/gvinum/v82 /dev/gvinum/v83</tt> <br><br />
Note, long names will truncate: v144v145v148-v115 will truncate to v144v145v148-v1 (so you will refer to it as v144v145v148-v1 thereafter)<br />
<br />
<br />
Optional- if new volume is on a different drive, move the mount point directory (get the drive from js output) AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
<br />
confirm you are mounted to the device (<tt>/dev/gvinum/v<span style="color:red">123</span>a</tt> OR <tt>/dev/concat/<span style="color:red">v82-v84</span></tt>) and space is correct:<br><br />
A. <tt>mount /dev/gvinum/v<span style="color:red">123</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br><br />
-or-<br><br />
B. <tt>mount /dev/concat/<span style="color:red">v82-v84</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<tt>cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
df .</tt> <br />
<br />
<br />
do the dump and pipe directly to restore:<br><br />
<tt>dump -0a -f - /dev/gvinum/v<span style="color:red">1</span> | restore -r -f - <br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
edit quad (<tt>vq<span style="color:red">1</span></tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new volume (<tt>/dev/gvinum/v<span style="color:red">123</span>a</tt> or <tt>/dev/concat/<span style="color:red">v82-v84</span>a</tt>) , run <tt>buildsafe</tt><br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
5b. When there's not enough room on an alternate partition, or on the same drive...but there is enough room if you were to remove the existing customer's space (i.e. if you want/need to reuse the existing gvinum volumes and add on more):<br />
<br />
<br />
mount backup nfs mounts:<br><br />
<tt>mbm</tt> <br><br />
(run df to confirm backup mounts are mounted)<br />
<br />
<br />
dump the customer to backup2 or backup1<br><br />
<tt>dump -0a -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> /dev/<span style="color:red">gconcat/v106-v107</span></tt><br><br />
(when complete WITHOUT errors, du the dump file to confirm it matches size, roughly, with usage)<br />
<br />
<br />
unconfigure the old gconcat volume<br><br />
list member gvinum volumes:<br />
<br />
<tt>gconcat list <span style="color:red">v106v107</span></tt><br />
<br />
<br />
Output will resemble:<br><br />
<pre>Geom name: v106v107<br />
State: UP<br />
Status: Total=2, Online=2<br />
Type: AUTOMATIC<br />
ID: 3530663882<br />
Providers:<br />
1. Name: concat/v106v107<br />
Mediasize: 4294966272 (4.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e2<br />
Consumers:<br />
1. Name: gvinum/sd/v106.p0.s0<br />
Mediasize: 2147483648 (2.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e3<br />
Start: 0<br />
End: 2147483136<br />
2. Name: gvinum/sd/v107.p0.s0<br />
Mediasize: 2147483648 (2.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e3<br />
Start: 2147483136<br />
End: 4294966272</pre><br />
<br />
stop volume and clear members<br><br />
<tt>gconcat stop <span style="color:red">v106v107</span><br><br />
gconcat clear <span style="color:red">gvinum/sd/v106.p0.s0 gvinum/sd/v107.p0.s0</span></tt><br />
<br />
<br />
create new device- and its ok to reuse old/former members<br><br />
try to grab a contiguous block of gvinum volumes. gconcat volumes MAY NOT span drives (i.e. you cannot use a gvinum volume from data3 and a volume from data2 in the same gconcat volume). <br><br />
<tt>gconcat label <span style="color:red">v82-v84v106v107 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84 /dev/gvinum/v106 /dev/gvinum/v107</span> <br><br />
bsdlabel -r -w /dev/concat/<span style="color:red">v82-v84v106v107</span><br><br />
newfs /dev/concat/<span style="color:red">v82-v84v106v107</span>a</tt><br />
<br />
<br />
Optional- if new volume is on a different drive, move the mount point directory (get the drive from js output) AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
confirm you are mounted to the device (/dev/concat/<span style="color:red">v82-v84v106v107</span>) and space is correct:<br><br />
<tt>mount /dev/concat/<span style="color:red">v82-v84v106v107</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<tt>cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
df .</tt> <br />
<br />
do the restore from the dumpfile on the backup server:<br><br />
<tt>restore -r -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> .<br><br />
rm restoresymtable</tt><br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
edit quad (<tt>vq<span style="color:red">1</span></tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new volume (<tt>/dev/concat/<span style="color:red">v82-v84v106v107</span>a</tt>), run buildsafe<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
TODO: clean up/clear old gvin/gconcat vol<br />
<br />
<br />
6. update disk (and dir if applicable) in mgmt screen<br />
<br />
<br />
7. update backup list AND move backups, if applicatble<br />
<br />
<br />
Ex: <tt>mvbackups <span style="color:red">69.55.237.26-col00241</span> jail<span style="color:red">9</span> data<span style="color:red">3</span></tt><br />
<br />
<br />
<br />
DEPRECATED - steps to tack on a new gvin to existing gconcat- leads to corrupted fs<br />
bsdlabel -e /dev/concat/v82-v84<br />
<br />
To figure out new size of the c partition, multiply 4194304 by the # of 2G gvinum volumes and subtract the # of 2G volumes:<br />
10G: 4194304 * 5 – 5 = 20971515<br />
8G: 4194304 * 4 – 4 = 16777212<br />
6G: 4194304 * 3 – 3 = 12582909<br />
4G: 4194304 * 2 – 2 = 8388606<br />
<br />
To figure out the new size of the a partition, subtract 16 from the c partition:<br />
10G: 20971515 – 16 = 20971499<br />
8G: 16777212 – 16 = 16777196<br />
6G: 12582909 – 16 = 12582893<br />
4G: 8388606 – 16 = 8388590<br />
<br />
Orig:<br />
8 partitions:<br />
# size offset fstype [fsize bsize bps/cpg]<br />
a: 8388590 16 4.2BSD 2048 16384 28552<br />
c: 8388606 0 unused 0 0 # "raw" part, don't edit<br />
<br />
New:<br />
8 partitions:<br />
# size offset fstype [fsize bsize bps/cpg]<br />
a: 12582893 16 4.2BSD 2048 16384 28552<br />
c: 12582909 0 unused 0 0 # "raw" part, don't edit<br />
<br />
sync; sync<br />
<br />
growfs /dev/concat/v82-v84a<br />
<br />
fsck –fy /dev/concat/v82-v84a<br />
<br />
sync<br />
<br />
fsck –fy /dev/concat/v82-v84a<br />
<br />
(keep running fsck’s till NO errors)<br />
<br />
== Problems un-mounting - and with mount_null’s ==<br />
<br />
If you cannot unmount a filesystem, beacuse it says the filesystem is busy, it is because of three things:<br />
<br />
a) the jail is still running<br />
<br />
b) you are actually in that directory, even though the jail is stopped<br />
<br />
c) there are still dev, null_mount or linprocfs mount points mounted inside that directory.<br />
<br />
d) when trying to umount null_mounts that are really long and you get an error like “No such file or directory”, it’s an OS bug where the dir is truncated. No known fix<br />
<br />
e) there are still files open somewhere inside the dir. Use <tt>fstat | grep <cid></tt> to find the process that has files open<br />
<br />
f) Starting with 6.x, the jail mechanism does a poor job of keeping track of processes running in a jail and if it thinks there are still procs running, it will refuse to umount the disk. If this is happening you should see a low number in the #REF column when you run jls. In this case you ''can'' safely <tt>umount –f</tt> the mount. <br />
<br />
Please note -if you forcibly unmount a (4.x) filesystem that has null_mounts<br />
still mounted in it, the system '''will crash''' within 10-15 mins.<br />
<br />
== Misc jail Items ==<br />
<br />
We are overselling hard drive space on jail2, jail8, jail9, a couple jails on jail17, jail4, jail12 and jail18.<br />
Even though the vn file shows 4G size, it doesn’t actually occupy that amount of space on the disk. So be careful not to fill up drives where we’re overselling – use oversellcheck to confirm you’re not oversold by more than 10G.<br />
There are other truncated jails, they are generally noted in a the file on the root system: /root/truncated<br />
<br />
The act of moving a truncated vn to another system un-does the truncating- the truncated vn is filled with 0’s and it occupies physical disk space for which it’s configured. So, you should use dumpremote to preserve the truncation.<br />
<br />
= FreeBSD VPS Management Tools =<br />
<br />
These files are located in /usr/local/jail/rc.d and /usr/local/jail/bin<br />
<br />
== jailmake ==<br />
<br />
Applies to 7.x+ <br />
On older systems syntax differs, run jailmake once to see.<br />
<br />
Note: this procedure differs on mx2 which is 7.x but still uses gvinum<br />
<br />
# run js to figure out which md’s are in use, which disk has enough space, IP to put it on<br />
# use col00xxx for both hostnames if they don’t give you a hostname<br />
# copy over dir, ip and password to pending customer screen<br />
<br />
<tt>Usage: jailmake IP[,IP] CID disk[1|2|3] md# hostname shorthost ipfw# email [size in GB]</tt><br />
<br />
Ex: <br />
<br />
Jail2# jailmake 69.55.234.66 col01334 3 97 vps.bsd.it vps 1334 fb@bsd.it<br />
<br />
== jailps ==<br />
jailps [hostname]<br />
DEPRECATED FOR jps: displays processes belonging to/running inside a jail. The command<br />
takes one (optional) argument – the hostname of the jail you wish to query. If you don’t <br />
supply an argument, all processes on the machine are listed and grouped by jail. <br />
<br />
== jps ==<br />
jps [hostname]<br />
displays processes belonging to/running inside a jail. The command<br />
takes one (optional) argument – the hostname or ID of the jail you wish to query. <br />
<br />
== jailkill ==<br />
jailkill <hostname><br />
stops all process running in a jail.<br />
<br />
You can also run:<br />
jailkill <JID><br />
<br />
=== problems ===<br />
Occasionally you will hit an issue where jail will not kill off:<br />
<br />
<pre>jail9# jailkill www.domain.com<br />
www.domain.com .. killed: none<br />
jail9#</pre><br />
Because no processes are running under that hostname. You cannot use jailps.pl either:<br />
<br />
<pre>jail9# jailps www.domain.com<br />
www.domain.com doesn’t exist on this server<br />
jail9#</pre><br />
<br />
The reasons for this are usually:<br />
* the jail is no longer running<br />
<br />
* the jail's hostname has changed<br />
In this case, <br />
<br />
>=6.x: run a <tt>jls|grep <jail's IP></tt> to find the correct hostname, then update the quad file, then kill the jail.<br />
<br />
<6.x: the first step is to cat their /etc/rc.conf file to see if you can tell what they set the new hostname to. This very often works. For example:<br />
<br />
cat /mnt/data2/198.78.65.136-col00261-DIR/etc/rc.conf<br />
<br />
But maybe they set the hostname with the hostname command, and the original hostname is still in /etc/rc.conf.<br />
<br />
The welcome email clearly states that they should tell us if they change their hostname, so there is no problem in just emailing them and asking them what they set the new hostname to.<br />
<br />
Once you know the new hostname OR if a customer simply emails to inform you that they have set the hostname to something different, you need to edit the quad and safe files that their system is in to input the new hostname.<br />
<br />
However, if push comes to shove and you cannot find out the hostname from them or from their system, then you need to start doing some detective work.<br />
<br />
The easiest thing to do is run jailps looking for a hostname similar to their original hostname. Or you could get into the /bin/sh shell by running:<br />
<br />
/bin/sh<br />
<br />
and then looking at every hostname of every process:<br />
<br />
for f in `ls /proc` ; do cat /proc/$f/status ; done<br />
<br />
and scanning for a hostname that is either similar to their original hostname, or that you don't see in any of the quad safe files.<br />
<br />
This is very brute force though, and it is possible that catting every file in /proc is dangerous - I don't recommend it. A better thing would be to identify any processes that you know belong to this system – perhaps the reason you are trying to find this system is because they are running something bad - and just catting the status from only that PID.<br />
<br />
Somewhere there’s a jail where there may be 2 systems named www. Look at /etc/rc.conf and make sure they’re both really www. If they are, jailkill www, jailps www to make sure not running. Then immediately restart the other one, as the fqdn (as found from a rev nslookup)<br />
<br />
* on >=6.x the hostname may not yet be hashed:<br />
<pre>jail9 /# jls<br />
JID Hostname Path IP Address(es)<br />
1 bitnet.dgate.org /mnt/data1/69.55.232.50-col02094-DIR 69.55.232.50<br />
2 ns3.hctc.net /mnt/data1/69.55.234.52-col01925-DIR 69.55.234.52<br />
3 bsd1 /mnt/data1/69.55.232.44-col00155-DIR 69.55.232.44<br />
4 let2.bbag.org /mnt/data1/69.55.230.92-col00202-DIR 69.55.230.92<br />
5 post.org /mnt/data2/69.55.232.51-col02095-DIR 69.55.232.51 ...<br />
6 ns2 /mnt/data1/69.55.232.47-col01506-DIR 69.55.232.47 ...<br />
7 arlen.server.net /mnt/data1/69.55.232.52-col01171-DIR 69.55.232.52<br />
8 deskfood.com /mnt/data1/69.55.232.71-col00419-DIR 69.55.232.71<br />
9 mirage.confluentforms.com /mnt/data1/69.55.232.54-col02105-DIR 69.55.232.54 ...<br />
10 beachmember.com /mnt/data1/69.55.232.59-col02107-DIR 69.55.232.59<br />
11 www.agottem.com /mnt/data1/69.55.232.60-col02109-DIR 69.55.232.60<br />
12 sdhobbit.myglance.org /mnt/data1/69.55.236.82-col01708-DIR 69.55.236.82<br />
13 ns1.jnielsen.net /mnt/data1/69.55.234.48-col00204-DIR 69.55.234.48 ...<br />
14 ymt.rollingegg.net /mnt/data2/69.55.236.71-col01678-DIR 69.55.236.71<br />
15 verse.unixlore.net /mnt/data1/69.55.232.58-col02131-DIR 69.55.232.58<br />
16 smcc-mail.org /mnt/data2/69.55.232.68-col02144-DIR 69.55.232.68<br />
17 kasoutsuki.w4jdh.net /mnt/data2/69.55.232.46-col02147-DIR 69.55.232.46<br />
18 dili.thium.net /mnt/data2/69.55.232.80-col01901-DIR 69.55.232.80<br />
20 www.tekmarsis.com /mnt/data2/69.55.232.66-col02155-DIR 69.55.232.66<br />
21 vps.yoxel.net /mnt/data2/69.55.236.67-col01673-DIR 69.55.236.67<br />
22 smitty.twitalertz.com /mnt/data2/69.55.232.84-col02153-DIR 69.55.232.84<br />
23 deliver4.klatha.com /mnt/data2/69.55.232.67-col02160-DIR 69.55.232.67<br />
24 nideffer.com /mnt/data2/69.55.232.65-col00412-DIR 69.55.232.65<br />
25 usa.hanyuan.com /mnt/data2/69.55.232.57-col02163-DIR 69.55.232.57<br />
26 daifuku.ppbh.com /mnt/data2/69.55.236.91-col01720-DIR 69.55.236.91<br />
27 collins.greencape.net /mnt/data2/69.55.232.83-col01294-DIR 69.55.232.83<br />
28 ragebox.com /mnt/data2/69.55.230.104-col01278-DIR 69.55.230.104<br />
29 outside.mt.net /mnt/data2/69.55.232.72-col02166-DIR 69.55.232.72<br />
30 vps.payneful.ca /mnt/data2/69.55.234.98-col01999-DIR 69.55.234.98<br />
31 higgins /mnt/data2/69.55.232.87-col02165-DIR 69.55.232.87 ...<br />
32 ozymandius /mnt/data2/69.55.228.96-col01233-DIR 69.55.228.96<br />
33 trusted.realtors.org /mnt/data2/69.55.238.72-col02170-DIR 69.55.238.72<br />
34 jc1.flanderous.com /mnt/data2/69.55.239.22-col01504-DIR 69.55.239.22<br />
36 guppylog.com /mnt/data2/69.55.238.73-col00036-DIR 69.55.238.73<br />
40 haliohost.com /mnt/data2/69.55.234.41-col01916-DIR 69.55.234.41 ...<br />
41 satyr.jorge.cc /mnt/data1/69.55.232.70-col01963-DIR 69.55.232.70<br />
jail9 /# jailkill satyr.jorge.cc<br />
ERROR: jail_: jail "satyr,jorge,cc" not found<br />
</pre><br />
<br />
Note how it's saying <tt>satyr,jorge,cc</tt> is not found, and not <tt>satyr.jorge.cc</tt>. <br />
<br />
The jail subsystem tracks things using comma-delimited hostnames. That is created every few hours:<br />
<br />
jail9 /# crontab -l<br />
0 0,6,12,18 * * * /usr/local/jail/bin/sync_jail_names<br />
<br />
So if we run this manually:<br />
jail9 /# /usr/local/jail/bin/sync_jail_names<br />
<br />
Then kill the jail:<br />
jail9 /# jailkill satyr.jorge.cc<br />
successfully killed: satyr,jorge,cc</pre><br />
<br />
It worked.<br />
<br />
<br />
If you ever see this when trying to kill a jail:<br />
<br />
<pre># jailkill e-scribe.com<br />
killing JID: 6 hostname: e-scribe.com<br />
3 procs running<br />
3 procs running<br />
3 procs running<br />
3 procs running<br />
...</pre><br />
<br />
<tt>[[#jailkill|jailkill]]</tt> probably got lost trying to kill off the jail. Just ctrl-c the jailkill process, then run a jailps on the hostname, and <tt>kill -9</tt> any process which is still running. Keep running jailps and <tt>kill -9</tt> till all processes are gone.<br />
<br />
== jailpsall ==<br />
jailpsall<br />
will run a jailps on all jails configured in the quad files (this is different from<br />
jailps with no arguments as it won’t help you find a “hidden” system)<br />
<br />
== jailpsw ==<br />
jailpsw<br />
will run a jailps with an extra -w to provide wider output<br />
<br />
== jt (>=7.x) ==<br />
jt<br />
displays the top 20 processes on the server (the top 20 processes from top) and <br />
which jail owns them. This is very helpful for determining who is doing what when<br />
the server is very busy.<br />
<br />
== jtop (>=7.x) ==<br />
jtop<br />
a wrapper for top displaying processes on the server and which jail owns them. Constantly updates, like top. <br />
<br />
== jtop (<7.x) ==<br />
jtop<br />
displays the top 20 processes on the server (the top 20 processes from top) and <br />
which jail owns them. This is very helpful for determining who is doing what when<br />
the server is very busy.<br />
<br />
== stopjail ==<br />
stopjail <hostname> [1]<br />
this will jailkill, umount and vnconfig –u a jail. If passed an optional 2nd<br />
argument, it will not exit before umounting and un-vnconfig’ing in the event<br />
jailkill returns no processes killed. This is useful if you just want to umount<br />
and vnconfig –u a jail you’ve already killed. It is intelligent in that it won’t <br />
try to umount or vnconfig –u if it’s not necessary.<br />
<br />
== startjail ==<br />
startjail <hostname><br />
this will start vnconfig, mount (including linprocfs and null-mounts), and start a jail.<br />
Essentially, it reads the jail’s relevant block from the right quad file and executes it.<br />
It is intelligent in that it won’t try to mount or vnconfig if it’s not necessary.<br />
<br />
== jpid ==<br />
jpid <pid><br />
displays information about a process – including which jail owns it.<br />
It’s the equivalent of running cat /proc/<pid>/status<br />
<br />
== canceljail ==<br />
canceljail <hostname> [1]<br />
this will stop a jail (the equivalent of stopjail), check for backups (offer to remove them <br />
from the backup server and the backup.config), rename the vnfile, remove the dir, and <br />
edit quad/safe. If passed an optional 2nd argument, it will not exit upon failing to kill<br />
and processes owned by the jail. This is useful if you just want to cancel a jail which <br />
is already stopped.<br />
<br />
== jls ==<br />
jls [-v]<br />
Lists all jails running:<br />
<pre>JID #REF IP Address Hostname Path<br />
101 135 69.55.224.148 mail.pc9.org /mnt/data2/69.55.224.148-col01034-DIR</pre><br />
<br />
#REF is the number of references or procs(?) running<br />
<br />
Running with -v will give you all IPs assigned to each jail (7.2 up)<br />
<pre>JID #REF Hostname Path IP Address(es)<br />
101 139 mail.pc9.org /mnt/data2/69.55.224.148-col01034-DIR 69.55.224.14869.55.234.85</pre><br />
<br />
== startalljails ==<br />
startalljails<br />
7.2+ only. This will parse through quad1 and start all jails. It utilizes lockfiles so it won’t try to start a jail more than once- therefore multiple instances can be running in parallel without fear of starting a jail twice. If a jail startup gets stuck, you can ^C without fear of killing the script. IMPORTANT- before running startalljails you should make sure you ran preboot once as it will clear out all the lockfiles and enable startalljails to work properly.<br />
<br />
== aaccheck.sh ==<br />
aaccheck.sh<br />
displayes the output of container list and task list from aaccli<br />
<br />
== backup ==<br />
backup<br />
backup script called nightly to update jail scripts and do customer backups<br />
<br />
== buildsafe ==<br />
buildsafe<br />
creates safe files based on quads (automatically removing the fsck’s). This will destructively overwrite safe files<br />
<br />
== checkload.pl ==<br />
checkload.pl<br />
this was intended to be setup as a cronjob to watch processes on a jail when the load <br />
rises above a certain level. Not currently in use.<br />
<br />
== checkprio.pl ==<br />
checkprio.pl<br />
will look for any process (other than the current shell’s csh, sh, sshd procs) with a non-normal priority and normalize it<br />
<br />
== diskusagemon == <br />
diskusagemon <mount point> <1k blocks><br />
watches a mount point’s disk use, when it reaches the level specified in the 2nd argument,<br />
it exits. This is useful when doing a restore and you want to be paged as it’s nearing completion.<br />
Best used as: <tt>diskusagemon /asd/asd 1234; pagexxx</tt><br />
<br />
== dumprestore ==<br />
dumprestore <dumpfile><br />
this is a perl expect script which automatically enters ‘1’ and ‘y’. It seems to cause restore to fail<br />
to set owner permissions on large restores.<br />
<br />
== g ==<br />
g <search><br />
greps the quad/safe files for the given search parameter<br />
<br />
== gather.pl ==<br />
gather.pl<br />
gathers up data about jails configured and writes to a file. Used for audits against the db<br />
<br />
== gb ==<br />
gb <search><br />
greps backup.config for the given search parameter<br />
<br />
== gbg ==<br />
gbg <search><br />
greps backup.config for the given search parameter and presents just the directories (for clean pasting)<br />
<br />
== ipfwbackup ==<br />
ipfwbackup<br />
writes ipfw traffic count data to a logfile<br />
<br />
== ipfwreset ==<br />
ipfwreset<br />
writes ipfw traffic count data to a logfile and resets counters to 0<br />
<br />
== js ==<br />
js<br />
output varies by OS version, but generally provides information about the base jail:<br />
- which vn’s are in use<br />
- disk usage<br />
- info about the contents of quads<br />
- the # of inodes represented by the jails contained in the group (133.2 in the example below), and how many jails per data mount, as well as subtotals<br />
- ips bound to the base machine but not in use by a jail<br />
- free gvinum volumes, or unused vn’s or used md’s<br />
<br />
<pre>/usr/local/jail/rc.d/quad1:<br />
/mnt/data1 133.2 (1)<br />
/mnt/data2 1040.5 (7)<br />
total 1173.7 (8)<br />
/usr/local/jail/rc.d/quad2:<br />
/mnt/data1 983.4 (6)<br />
total 983.4 (6)<br />
/usr/local/jail/rc.d/quad3:<br />
/mnt/data1 693.4 (4)<br />
/mnt/data2 371.6 (3)<br />
total 1065 (7)<br />
/usr/local/jail/rc.d/quad4:<br />
/mnt/data1 466.6 (3)<br />
/mnt/data2 882.2 (5)<br />
total 1348.8 (8)<br />
/mnt/data1: 2276.6 (14)<br />
/mnt/data2: 2294.3 (15)<br />
<br />
Available IPs:<br />
69.55.230.11 69.55.230.13 69.55.228.200<br />
<br />
Available volumes:<br />
v78 /mnt/data2 2G<br />
v79 /mnt/data2 2G<br />
v80 /mnt/data2 2G</pre> <br />
<br />
== load.pl ==<br />
load.pl<br />
feeds info to load mrtg - executed by inetd.<br />
<br />
== makevirginjail ==<br />
makevirginjail<br />
Only on some systems, makes an empty jail (doesn't do restore step)<br />
<br />
== mb == <br />
mb <mount|umount><br />
(nfs) mounts and umounts dirs to backup2. Shortcuts are mbm and mbu to mount and unmount. <br />
<br />
== notify.sh ==<br />
notify.sh<br />
emails reboot@johncompanies.com – intended to be called at boot time to alert us to a machine which panics and reboots and isn’t caught by bb or castle.<br />
<br />
== orphanedbackupwatch ==<br />
orphanedbackupwatch<br />
looks for directories on backup2 which aren’t configured in backup.config and offers to delete them<br />
<br />
== postboot ==<br />
postboot<br />
to be run after a machine reboot and quad/safe’s are done executing. It will:<br />
* do chmod 666 on each jail’s /dev/null<br />
* add ipfw counts<br />
* run jailpsall (so you can see if a configured jail isn’t running)<br />
<br />
== preboot ==<br />
preboot<br />
to be run before running quad/safe – checks for misconfigurations: <br />
* a jail configured in a quad but not a safe<br />
* a jail is listed more than once in a quad<br />
* the ip assigned to a jail isn’t configured on the machine<br />
* alias numbering skips in the rc.conf (resulting in the above)<br />
* orphaned vnfile's that aren't mentioned in a quad/safe<br />
* ip mismatches between dir/vnfile name and the jail’s ip<br />
* dir/vnfiles's in quad/safe that don’t exist <br />
<br />
== quadanalyze.pl ==<br />
quadanalyze.pl<br />
called by js, produces the info (seen above with js explanation) about the contents of quad (inode count, # of jails, etc.)<br />
<br />
== rsync.backup ==<br />
rsync.backup<br />
does customer backups (relies on backup.config)<br />
<br />
== taskdone ==<br />
taskdone<br />
when called will email support@johncompanies.com with the hostname of the machine from which it was executed as the subject<br />
<br />
== topten ==<br />
topten<br />
summarizes the top 10 traffic users (called by ipfwreset)<br />
<br />
== trafficgather.pl ==<br />
trafficgather.pl [yy-mm]<br />
sends a traffic usage summary by jail to support@johncomapnies.com and payments@johncompanies.com. Optional arguments are year and month (must be in the past). If not passed, assumes last month. Relies on traffic logs created by ipfwreset and ipfwbackup<br />
<br />
== trafficwatch.pl ==<br />
trafficwatch.pl<br />
checks traffic usage and emails support@johncomapnies.com when a jail reaches the warning level (35G) and the limit (40G). We really aren’t using this anymore now that we have netflow.<br />
<br />
== trafstats ==<br />
trafstats<br />
writes ipfw traffic usage info by jail to a file called jc_traffic_dump in each jail’s / dir<br />
<br />
== truncate_jailmake ==<br />
truncate_jailmake<br />
a version of jailmake which creates truncated vnfiles.<br />
<br />
== vb ==<br />
vb<br />
the equivalent of: <tt>vi /usr/local/jail/bin/backup.config</tt><br />
<br />
== vs (freebsd) ==<br />
vs<n><br />
the equivalent of: <tt>vi /usr/local/jail/rc.d/safe<n></tt><br />
<br />
vq<n><br />
the equivalent of: vi /usr/local/jail/rc.d/quad<n><br />
<br />
== dumpremote ==<br />
dumpremote <user@machine> </remote/location/file-dump> <vnX><br />
ex: dumpremote user@10.1.4.117 /mnt/data3/remote.echoditto.com-dump 7<br />
this will dump a vn filesystem to a remote machine and location<br />
<br />
== oversellcheck ==<br />
oversellcheck<br />
displays how much a disk is oversold or undersold taking into account truncated vn files. Only for use on 4.x systems<br />
<br />
== mvbackups (freebsd) ==<br />
mvbackups <dir> (1.1.1.1-col00001-DIR) <target_machine> (jail1) <target_dir> (data1)<br />
moves backups from one location to another on the backup server, and provides you with option to remove entries from current backup.config, and simple paste command to add the config to backup.config on the target server<br />
<br />
== jailnice ==<br />
jailnice <hostname><br />
applies <tt>renice 19 [PID]</tt> and <tt>rtprio 31 –[PID]</tt> to each process in the given jail<br />
<br />
== dumpremoterestore ==<br />
dumpremoterestore <device> <ip of target machine> <dir on target machine><br />
ex: dumpremoterestore /dev/vn51 10.1.4.118 /mnt/data2/69.55.239.45-col00688-DIR<br />
dumps a device and restores it to a directory on a remote machine. Requires that you enable root ssh on the <br />
remote machine.<br />
<br />
== psj ==<br />
psj<br />
shows just the procs running on the base system – a ps auxw but without jail’d procs present<br />
<br />
== perc5iraidchk ==<br />
perc5iraidchk<br />
checks for degraded arrays on Dell 2950 systems with Perc5/6 controllers<br />
<br />
== perc4eraidchk ==<br />
perc4eraidchk<br />
checks for degraded arrays on Dell 2850 systems with Perc4e/Di controllers<br />
<br />
= Making new customer VE (vm) =<br />
<br />
This applies only to new virts >= 4.x<br />
<br />
grab ip from ipmap (if opened from the pending cust screen it should take you to the right block). You can also run vzlist -a to see what block is in use, generally. Try to find an IP that's in the same block of class C IP's already on the box.<br />
<br />
1. confirm ip you want to use isn’t in use via Mgmt. -> IP Map in management screens<br />
<br />
2. put CT on whichever partition has more space<br />
<br />
3. vm CID IP hostname /vz[1|2] email[,email] template ( <LB|LBP|LS|LM|LMP|LP> [disk] | <gb_disk/mb_ram/gb_burst> ) <br />
vm col00009 69.55.230.238 centos.testdave.com /vz1 dsmith@n2.net centos-6-x86_64 LM<br />
<br />
4. copy veid, dir, ip and password to pending customer screen. activate customer<br />
<br />
= Virtuozzo VPS =<br />
<br />
Note: We use VEID (Virtual Environment ID) and CTID (Container ID) interchangably. Similarly, VE and CT. They mean the same thing.<br />
VZPP = VirtuoZzo Power Panel (the control panel for each CT)<br />
<br />
All linux systems exist in /vz, /vz1 or /vz2 - since each linux machine holds roughly 60-90 customers, there will be roughly 30-45 in each partition.<br />
<br />
The actual filesystem of the system in question is in:<br />
<br />
/vz(1-2)/private/(VEID)<br />
<br />
Where VEID is the identifier for that system - an all-numeric string larger than 100.<br />
<br />
The actual mounted and running systems are in the corresponding:<br />
<br />
/vz(1-2)/root/(VEID)<br />
<br />
But we rarely interact with any system from this mount point.<br />
<br />
You should never need to touch the root portion of their system – however you can traverse their filesystem by going to <tt>/vz(1-2)/private/(VEID)/root</tt> (<tt>/vz(1-2)/private/(VEID)/fs/root</tt> on 4.x systems) the root of their filesystem is in that directory, and their entire system is underneath that.<br />
<br />
Every VE has a startup script in <tt>/etc/sysconfig/vz-scripts</tt> (which is symlinked as <tt>/vzconf</tt> on all systems) - the VE startup script is simply named <tt>(VEID).conf</tt> - it contains all the system parameters for that VE:<br />
<br />
<pre># Configuration file generated by vzsplit for 60 VE<br />
# on HN with total amount of physical mem 2011 Mb<br />
<br />
VERSION="2"<br />
CLASSID="2"<br />
<br />
ONBOOT="yes"<br />
<br />
KMEMSIZE="8100000:8200000"<br />
LOCKEDPAGES="322:322"<br />
PRIVVMPAGES="610000:615000"<br />
SHMPAGES="33000:34500"<br />
NUMPROC="410:415"<br />
PHYSPAGES="0:2147483647"<br />
VMGUARPAGES="13019:2147483647"<br />
OOMGUARPAGES="13019:2147483647"<br />
NUMTCPSOCK="1210:1215"<br />
NUMFLOCK="107:117"<br />
NUMPTY="19:19"<br />
NUMSIGINFO="274:274"<br />
TCPSNDBUF="1800000:1900000"<br />
TCPRCVBUF="1800000:1900000"<br />
OTHERSOCKBUF="900000:950000"<br />
DGRAMRCVBUF="200000:200000"<br />
NUMOTHERSOCK="650:660"<br />
DCACHE="786432:818029"<br />
NUMFILE="7500:7600"<br />
AVNUMPROC="51:51"<br />
IPTENTRIES="155:155"<br />
DISKSPACE="4194304:4613734"<br />
DISKINODES="400000:420000"<br />
CPUUNITS="1412"<br />
QUOTAUGIDLIMIT="2000"<br />
VE_ROOT="/vz1/root/636"<br />
VE_PRIVATE="/vz1/private/636"<br />
NAMESERVER="69.55.225.225 69.55.230.3"<br />
OSTEMPLATE="vzredhat-7.3/20030305"<br />
VE_TYPE="regular"<br />
IP_ADDRESS="69.55.225.229"<br />
HOSTNAME="textengine.net"</pre><br />
<br />
<br />
As you can see, the hostname is set here, the disk space is set here, the number of inodes, the number of files that can be open, the number of tcp sockets, etc. - all are set here.<br />
<br />
In fact, everything that can be set on this customer system is set in this conf file.<br />
<br />
<br />
All interaction with the customer system is done with the VEID. You start the system by running:<br />
<br />
vzctl start 999<br />
<br />
You stop it by running:<br />
<br />
vzctl stop 999<br />
<br />
You execute commands in it by running:<br />
<br />
vzctl exec 999 df -k<br />
<br />
You enter into it, via a root-shell backdoor with:<br />
<br />
vzctl enter 999<br />
<br />
and you set parameters for the system, while it is still running, with:<br />
<br />
vzctl set 999 --diskspace 6100000:6200000 --save<br />
<br />
<tt>vzctl</tt> is the most commonly used command - we have aliased <tt>v</tt> to <tt>vzctl</tt> since we use it so often. We’ll continue to use <tt>vzctl</tt> in our examples, but feel free to use just <tt>v</tt>.<br />
<br />
Let's say the user wants more diskspace. You can cat their conf file and see:<br />
<br />
DISKSPACE="4194304:4613734"<br />
<br />
So right now they have 4gigs of space. You can then change it to 6 with:<br />
<br />
vzctl set 999 --diskspace 6100000:6200000 --save<br />
<br />
IMPORTANT: all issuances of the vzctl set command need to end with <tt>–save</tt> - if they don't, the setting will be set, but it will not be saved to the conf file, and they will not have those settings next time they boot.<br />
<br />
All of the tunables in the conf file can be set with the vzctl set command. Note that in the conf file, and on the vzctl set command line, we always issue two numbers seperated by a colon - that is because we are setting the hard and soft limits. Always set the hard limit slightly above the soft limit, as you see it is in the conf file for all those settings.<br />
<br />
There are also things you can set with `<tt>vzctl set</tt>` that are not in the conf file as settings, per se. For instance, you can add IPs:<br />
<br />
vzctl set 999 --ipadd 10.10.10.10 --save<br />
<br />
or multiple IPs:<br />
<br />
vzctl set 999 --ipadd 10.10.10.10 --ipadd 10.10.20.30 --save<br />
<br />
or change the hostname:<br />
<br />
vzctl set 999 --hostname www.example.com --save<br />
<br />
You can even set the nameservers:<br />
<br />
vzctl set 999 --nameserver 198.78.66.4 --nameserver 198.78.70.180 --save<br />
<br />
Although you probably will never do that.<br />
<br />
You can disable a VPS from being started (by VZPP or reboot) (4.x):<br />
<br />
vzctl set 999 --disabled yes --save <br />
<br />
You can disable a VPS from being started (by VZPP or reboot) (<=3.x):<br />
<br />
vzctl set 999 --onboot=no --save <br />
<br />
You can disable a VPS from using his control panel:<br />
<br />
vzctl set 999 --offline_management=no --save <br />
<br />
You can suspend a VPS, so it can be resumed in the same state it was in when it was stopped (4.x):<br />
<br />
vzctl suspend 999<br />
<br />
and to resume it:<br />
<br />
vzctl resume 999<br />
<br />
to see who owns process:<br />
vzpid <PID><br />
<br />
to mount up an unmounted ve:<br />
vzctl mount 827<br />
<br />
<br />
To see network stats for CT's:<br />
<pre># vznetstat<br />
VEID Net.Class Output(bytes) Input(bytes)<br />
-----------------------------------------------<br />
24218 1 484M 39M<br />
24245 1 463M 143M<br />
2451 1 2224M 265M<br />
2454 1 2616M 385M<br />
4149 1 125M 68M<br />
418 1 1560M 34M<br />
472 1 1219M 315M<br />
726 1 628M 317M<br />
763 1 223M 82M<br />
771 1 4234M 437M<br />
-----------------------------------------------<br />
Total: 13780M 2090M</pre><br />
<br />
<br />
One thing that sometimes comes up on older systems that we created with smaller defaults is that the system would run out of inodes. The user will email and say they cannot create any more files or grow any files larger, but they will also say that they are not out of diskspace ... they are running:<br />
<br />
df -k<br />
<br />
and seeing how much space is free - and they are not out of space. They are most likely out of inodes - which they would see by running:<br />
<br />
df -i<br />
<br />
So, the first thing you should do is enter their system with:<br />
<br />
vzctl enter 999<br />
<br />
and run: <tt>df -i</tt><br />
<br />
to confirm your theory. Then exit their system. Then simply cat their conf file and see what their inodes are set to (probably 200000:200000, since that was the old default on the older systems) and run:<br />
<br />
vzctl set 999 --diskinodes 400000:400000 --save<br />
<br />
If they are not out of inodes, then a good possibility is that they have maxed out their numfile configuration variable, which controls how many files they can have in their system. The current default is 7500 (which nobody has ever hit), but the old default was as low as 2000, so you would run something like:<br />
<br />
vzctl set 999 --numfile 7500:7500 --save<br />
<br />
----<br />
<br />
You cannot start or stop a VE if your pwd is its private (/vz/private/999) or root (/vz/root/999) directories, or anywhere below them.<br />
<br />
<br />
== Recovering from a crash (linux) ==<br />
<br />
=== Diagnose whether you have a crash ===<br />
The most important thing is to get the machine and all ve’s back up as soon as possible. Note the time, you’ll need to create a crash log entry (Mgmt. -> Reference -> CrashLog). The first thing to do is head over to the [[Screen#Screen_Organization|serial console screen]] and see if there’s any kernel error messages output. Try to copy any messages (or just a sample of repeating messages) you see into the notes section of the crash log – these will also likely need to be sent to virtuozzo for interpretation. If the messages are spewing too fast, hit ^O + H to start a screen log dump which you can ob1182.pts-38.bb serve after the machine is rebooted. Additionally, if the machine is responsive, you can get a trace to send to virtuozzo by hooking up a kvm and entering these 3 sequences:<br />
<pre>alt+print screen+m<br />
alt+print screen+p<br />
alt+print screen+t</pre><br />
<br />
If there are no messages, the machine may just be really busy- wait a bit (5-10min) to see if it comes back. If it's still pinging, odds are its very busy. If it doesn't come back, or the messages indicate a fatal error, you will need to proceed with a power cycle (ctrl+alt+del will not work).<br />
<br />
=== Power cycle the server ===<br />
If this machine is not a Dell 2950 with a [[DRAC/RMM#DRAC|DRAC card]] (i.e. if you can’t ssh into the DRAC card and issue racadm serveraction hardreset, then you will need someone at the data center to power the macine off, wait 30 sec, then turn it back on. Make sure to re-attach via console (<tt>tip virtxx</tt>) immediately after power down. <br />
<br />
=== (Re)attach to the console ===<br />
Stay on the console the entire time during boot. As the BIOS posts- look out for the RAID card output- does everything look healthy? The output may be scrambled, look for "DEGRADED" or "FAILED". Once the OS starts booting you will be disconnected (dropped back to the shell on the console server) a couple times during the boot up. The reason you want to quickly re-attach is two-fold: 1. If you don’t reattach quickly then you won’t get any console output, 2. you want to be attached before the server ''potentially'' starts (an extensive) fsck. If you attach after the fsck begins, you’ll have seen no indication it started an fsck and the server will appear frozen during startup- no output, no response. <br />
<br />
=== Start containers/VE's/VPSs ===<br />
When the machine begins to start VE’s, it’s safe to leave the console and login via ssh. All virts should be set to auto start all the VEs after a crash. Further, most (newer) virts are set to “fastboot” it’s VE’s (to find out, do:<br />
grep -i fast /etc/sysconfig/vz <br />
and look for <tt>VZFASTBOOT=yes</tt>). If this was set prior to the machine’s crash (setting it after the machine boots will not have any effect until the vz service is restarted) it will start each ve as fast as possible, in serial, then go thru each VE (serially), shutting it down running a vzquota (disk usage) check, then bringing it back up. The benefit is that all VE’s are brought up quickly (within 15min or so depending on the #), the downside is a customer watching closely will notice 2 outages – 1st the machine crash, 2nd their quota check (which will be a much shorter downtime- on the order of a few minutes). <br />
<br />
Where “fastboot” is not set to yes (i.e on quar1), vz will start them consecutively, checking the quotas one at a time, and the 60th VE may not start until an hour or two later - this is not acceptable.<br />
<br />
The good news is, if you run vzctl start for a VE that is already started, you will simply get an error: <tt>VE is already started</tt>. Further, if you attempt to vzctl start a VE that is in the process of being started, you will simply get an error: unable to lock VE. So, there is no danger in simply running scripts to start smaller sets of VEs. If the system is not autostarting, then there is no issue, and even if it does, when it conflicts, one process (yours or the autostart) will lose, and just move on to the next one.<br />
<br />
A script has been written to assist with ve starts: [[#startvirt.pl|startvirt.pl]] which will start 6 ve’s at once until there are no more left. If startvirt.pl is used on a system where “fastboot” was on, it will circumvent the fastboot for ve’s started by startvirt.pl – they will go through the complete quota check before starting- therefore this is not advisable when a system has crashed. When a system is booted cleanly, and there's no need for vzquota checks, then startvirt.pl is safe and advisable to run.<br />
<br />
=== Make sure all containers are running ===<br />
You can quickly get a feel for how many ve’s are started by running:<br />
<br />
<pre>[root@virt4 log]# vs<br />
VEID 16066 exist mounted running<br />
VEID 16067 exist mounted running<br />
VEID 4102 exist mounted running<br />
VEID 4112 exist mounted running<br />
VEID 4116 exist mounted running<br />
VEID 4122 exist mounted running<br />
VEID 4123 exist mounted running<br />
VEID 4124 exist mounted running<br />
VEID 4132 exist mounted running<br />
VEID 4148 exist mounted running<br />
VEID 4151 exist mounted running<br />
VEID 4155 exist mounted running<br />
VEID 42 exist mounted running<br />
VEID 432 exist mounted running<br />
VEID 434 exist mounted running<br />
VEID 442 exist mounted running<br />
VEID 450 exist mounted running<br />
VEID 452 exist mounted running<br />
VEID 453 exist mounted running<br />
VEID 454 exist mounted running<br />
VEID 462 exist mounted running<br />
VEID 463 exist mounted running<br />
VEID 464 exist mounted running<br />
VEID 465 exist mounted running<br />
VEID 477 exist mounted running<br />
VEID 484 exist mounted running<br />
VEID 486 exist mounted running<br />
VEID 490 exist mounted running</pre><br />
<br />
So to see how many ve’s have started:<br />
<pre>[root@virt11 root]# vs | grep running | wc -l<br />
39</pre><br />
<br />
And to see how many haven’t:<br />
<pre>[root@virt11 root]# vs | grep down | wc -l<br />
0</pre><br />
<br />
And how many we should have running:<br />
<pre>[root@virt11 root]# vs | wc -l<br />
39</pre><br />
<br />
Another tool you can use to see which ve’s have started, among other things is [[#vzstat|vzstat]]. It will give you CPU, memory, and other stats on each ve and the overall system. It’s a good thing to watch as ve’s are starting (note the VENum parameter, it will tell you how many have started):<br />
<br />
<pre>4:37pm, up 3 days, 5:31, 1 user, load average: 1.57, 1.68, 1.79<br />
VENum 40, procs 1705: running 2, sleeping 1694, unint 0, zombie 9, stopped 0<br />
CPU [ OK ]: VEs 57%, VE0 0%, user 8%, sys 7%, idle 85%, lat(ms) 412/2<br />
Mem [ OK ]: total 6057MB, free 9MB/54MB (low/high), lat(ms) 0/0<br />
Swap [ OK ]: tot 6142MB, free 4953MB, in 0.000MB/s, out 0.000MB/s<br />
Net [ OK ]: tot: in 0.043MB/s 402pkt/s, out 0.382MB/s 4116pkt/s<br />
Disks [ OK ]: in 0.002MB/s, out 0.000MB/s<br />
<br />
VEID ST %VM %KM PROC CPU SOCK FCNT MLAT IP<br />
1 OK 1.0/17 0.0/0.4 0/32/256 0.0/0.5 39/1256 0 9 69.55.227.152<br />
21 OK 1.3/39 0.1/0.2 0/46/410 0.2/2.8 23/1860 0 6 69.55.239.60<br />
133 OK 3.1/39 0.1/0.3 1/34/410 6.3/2.8 98/1860 0 0 69.55.227.147<br />
263 OK 2.3/39 0.1/0.2 0/56/410 0.3/2.8 34/1860 0 1 69.55.237.74<br />
456 OK 17/39 0.1/0.2 0/100/410 0.1/2.8 48/1860 0 11 69.55.236.65<br />
476 OK 0.6/39 0.0/0.2 0/33/410 0.1/2.8 96/1860 0 10 69.55.227.151<br />
524 OK 1.8/39 0.1/0.2 0/33/410 0.0/2.8 28/1860 0 0 69.55.227.153<br />
594 OK 3.1/39 0.1/0.2 0/45/410 0.0/2.8 87/1860 0 1 69.55.239.40<br />
670 OK 7.7/39 0.2/0.3 0/98/410 0.0/2.8 64/1860 0 216 69.55.225.136<br />
691 OK 2.0/39 0.1/0.2 0/31/410 0.0/0.7 25/1860 0 1 69.55.234.96<br />
744 OK 0.1/17 0.0/0.5 0/10/410 0.0/0.7 7/1860 0 6 69.55.224.253<br />
755 OK 1.1/39 0.0/0.2 0/27/410 0.0/2.8 33/1860 0 0 192.168.1.4<br />
835 OK 1.1/39 0.0/0.2 0/19/410 0.0/2.8 5/1860 0 0 69.55.227.134<br />
856 OK 0.3/39 0.0/0.2 0/13/410 0.0/2.8 16/1860 0 0 69.55.227.137<br />
936 OK 3.2/52 0.2/0.4 0/75/410 0.2/0.7 69/1910 0 8 69.55.224.181<br />
1020 OK 3.9/39 0.1/0.2 0/60/410 0.1/0.7 55/1860 0 8 69.55.227.52<br />
1027 OK 0.3/39 0.0/0.2 0/14/410 0.0/2.8 17/1860 0 0 69.55.227.83<br />
1029 OK 1.9/39 0.1/0.2 0/48/410 0.2/2.8 25/1860 0 5 69.55.227.85<br />
1032 OK 12/39 0.1/0.4 0/80/410 0.0/2.8 41/1860 0 8 69.55.227.90</pre><br />
<br />
When you are all done, you will want to make sure that all the VEs really did get started, run vs one more time.<br />
<br />
Note the time all ve’s are back up and enter that into and save the crash log entry.<br />
<br />
Occasionally, a ve will not start automatically. The most common reason for a ve not to come up normally is the ve was at it’s disk limit before the crash, and will not start since they’re over the limit. To overcome this, set the disk space to current usage level (the system will give this to you when it fails to start), start the ve, then re-set the disk space back to the prior level. Lastly, contact the customer to let them know they’re out of disk (or allocate more disk if they're entitled to more).<br />
<br />
== Hitting performance barriers and fixing them ==<br />
<br />
There are multiple modes virtuozzo offers to allocate resources to a ve. We utilize 2: SLM and UBC parameters<br />
On our 4.x systems, we use all SLM – it’s simpler to manage and understand. There are a few systems on virt19/18 that may also use SLM. Everything else uses UBC. <br />
You can tell a SLM ve by:<br />
<br />
SLMMODE="all"<br />
<br />
in their conf file. <br />
<br />
TODO: detail SLM modes and parameters.<br />
<br />
If someone is in SLM mode and they hit memory resource limits, they simply need to upgrade to more memory.<br />
<br />
The following applies to everyone else (UBC).<br />
<br />
Customers will often email and say that they are getting out of memory errors - a common one is "cannot fork" ... basically, anytime you see something odd like this, it means they are hitting one of their limits that is in place in their conf file.<br />
<br />
The conf file, however, simply shows their limits - how do we know what they are currently at ?<br />
<br />
The answer is a file called v - this file contains the current status (and peaks) of their performance settings, and also counts how many times they have hit the barrier. The output of the file looks like this:<br />
<br />
<pre>764: kmemsize 384113 898185 8100000 8200000 0<br />
lockedpages 0 0 322 322 0<br />
privvmpages 1292 7108 610000 615000 0<br />
shmpages 270 528 33000 34500 0<br />
dummy 0 0 0 0 0<br />
numproc 8 23 410 415 0<br />
physpages 48 5624 0 2147483647 0<br />
vmguarpages 0 0 13019 2147483647 0<br />
oomguarpages 641 6389 13019 2147483647 0<br />
numtcpsock 3 21 1210 1215 0<br />
numflock 1 3 107 117 0<br />
numpty 0 2 19 19 0<br />
numsiginfo 0 4 274 274 0<br />
tcpsndbuf 0 80928 1800000 1900000 0 <br />
tcprcvbuf 0 108976 1800000 1900000 0<br />
othersockbuf 2224 37568 900000 950000 0<br />
dgramrcvbuf 0 4272 200000 200000 0<br />
numothersock 3 9 650 660 0<br />
dcachesize 53922 100320 786432 818029 0<br />
numfile 161 382 7500 7600 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
numiptent 4 4 155 155 0</pre><br />
<br />
The first column is the name of the counter in question - the same names we saw in the systems conf file. The second column is the _current_ value of that counter, the third column is the max that that counter has ever risen to, the fourth column is the soft limit, and the fifth column is the hard limit (which is the same as the numbers in that systems conf file).<br />
<br />
The sixth number is the failcount - how many times the current usage has risen to hit the barrier. It will increase as soon as the current usage hits the soft limit.<br />
<br />
The problem with /proc/user_beancounters is that it actually contains that set of data for every running VE - so you can't just cat /proc/user_beancounters - it is too long and you get info for every other running system.<br />
<br />
You can vzctl enter the system and run:<br />
<br />
vzctl enter 9999<br />
cat /proc/user_beancounters<br />
<br />
inside their system, and you will just see the stats for their particular system, but entering their system every time you want to see it is combersome.<br />
<br />
So, I wrote a simple script called "vzs" which simply greps for the VEID, and spits out the next 20 or so lines (however many lines there are in the output, I forget) after it. For instance:<br />
<br />
<pre># vzs 765:<br />
765: kmemsize 2007936 2562780 8100000 8200000 0<br />
lockedpages 0 8 322 322 0<br />
privvmpages 26925 71126 610000 615000 0<br />
shmpages 16654 16750 33000 34500 0<br />
dummy 0 0 0 0 0<br />
numproc 41 57 410 415 0<br />
physpages 1794 49160 0 2147483647 0<br />
vmguarpages 0 0 13019 2147483647 0<br />
oomguarpages 4780 51270 13019 2147483647 0<br />
numtcpsock 23 37 1210 1215 0<br />
numflock 17 39 107 117 0<br />
numpty 1 3 19 19 0<br />
numsiginfo 0 6 274 274 0<br />
tcpsndbuf 22240 333600 1800000 1900000 0<br />
tcprcvbuf 0 222656 1800000 1900000 0<br />
othersockbuf 104528 414944 900000 950000 0<br />
dgramrcvbuf 0 4448 200000 200000 0<br />
numothersock 73 105 650 660 0<br />
dcachesize 247038 309111 786432 818029 0<br />
numfile 904 1231 7500 7600 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
numiptent 4 4 155 155 0</pre><br />
<br />
That showed us just the portion of /proc/user_beancounters for system 765.<br />
<br />
When you run the vzs command, always add a : after the VEID.<br />
<br />
So, if a customer complains about some out of memory errors, or no more files, or no more ptys, or just has an unspecific complain about processes dying, etc., the very first thing you need to do is check their beancounters with vzs. Usually you will spot an item that has a high failcount and needs to be upped.<br />
<br />
At that point you could simply up the counter with `vzctl set`. Generally pick a number 10-20% higher than the old one, and make the hard limit slightly larger than the the soft limit. However our systems now come in several levels and those levels have more/different memory allocations. If someone is complaining about something other than a memory limit (pty, numiptent, numflock), it’s generally safe to increase it, at least to the same level as what’s in the /vzconf/4unlimited file on the newest virt. If someone is hitting a memory limit, first make sure they are given what they deserve:<br />
<br />
(refer to mgmt -> payments -> packages)<br />
<br />
To set those levels, you use the [[#setmem|setmem]] command. <br />
<br />
The alternate (DEPRECATED) method would be to use one of 3 commands:<br />
256 <veid><br />
300 <veid><br />
384 <veid><br />
512 <veid><br />
<br />
If the levels were not right (you’d run vzs <veid> before and after to see the effect) tell the customer they’ve been adjusted and be done with it. If the levels were right, tell the customer they must upgrade to a higher package, tell them how to see level (control panel) and that they can reboot their system to escape this lockup contidion.<br />
<br />
Customers can also complain that their site is totally unreachable, or complain that it is down ... if the underlying machine is up, and all seems well, you may notice in the beancounters that network-specific counters are failing - such as numtcpsock, tcpsndbuf or tcprcvbuf. This will keep them from talking on the network and make it seem like their system is down. Again, just up the limits and things should be fine.<br />
<br />
On virts 1-4, you should first look at the default settings for that item on a later virt, such as virt 8 - we have increased the defaults a lot since the early machines. So, if you are going to up a counter on virt2, instead of upping it by 10-20%, instead up it to the new default that you see on virt8.<br />
<br />
== Moving a VE to another virt (migrate/migrateonline) ==<br />
<br />
This will take a while to complete - and it is best to do this at night when the load is light on both machines.<br />
<br />
There are different methods for this, depending on which version of virtuozzo is installed on the src. and dst. virt. <br />
To check which version is running: <br />
[root@virt12 private]# cat /etc/virtuozzo-release<br />
Virtuozzo release 2.6.0<br />
<br />
Ok, let's say that the VE is 1212, and vital stats are:<br />
<pre>[root@virt12 sbin]# vc 1212<br />
VE_ROOT="/vz1/root/1212"<br />
VE_PRIVATE="/vz1/private/1212"<br />
OSTEMPLATE="fedora-core-2/20040903"<br />
IP_ADDRESS="69.55.229.84"<br />
TEMPLATES="devel-fc2/20040903 php-fc2/20040813 mysql-fc2/20040812 postgresql-fc2/20040813 mod_perl-fc2/20040812 mod_ssl-fc2/20040811 jre-fc2/20040823 jdk-fc2/20040823 mailman-fc2/20040823 analog-fc2/20040824 proftpd-fc2/20040818 tomcat-fc2/20040823 usermin-fc2/20040909 webmin-fc2/20040909 uw-imap-fc2/20040830 phpBB-fc2/20040831 spamassassin-fc2/20040910 PostNuke-fc2/20040824 sl-webalizer-fc2/20040<br />
818"<br />
<br />
[root@virt12 sbin]# vzctl exec 1212 df -h<br />
Filesystem Size Used Avail Use% Mounted on<br />
/dev/vzfs 4.0G 405M 3.7G 10% /</pre><br />
<br />
From this you can see that he’s using (and will minimally need free on the dst server) ~400MB, and he’s running on a Fedora 2 template, version 20040903. He’s also got a bunch of other templates installed. It’s is '''vital''' that '''all''' these templates exist on the dst system. To confirm that, on the dst system run:<br />
<br />
For < 3.0:<br />
<br />
<pre>[root@virt14 private]# vzpkgls | grep fc2<br />
devel-fc2 20040903<br />
PostNuke-fc2 20040824<br />
analog-fc2 20040824<br />
awstats-fc2 20040824<br />
bbClone-fc2 20040824<br />
jdk-fc2 20040823<br />
jre-fc2 20040823<br />
mailman-fc2 20040823<br />
mod_frontpage-fc2 20040816<br />
mod_perl-fc2 20040812<br />
mod_ssl-fc2 20040811<br />
mysql-fc2 20040812<br />
openwebmail-fc2 20040817<br />
php-fc2 20040813<br />
phpBB-fc2 20040831<br />
postgresql-fc2 20040813<br />
proftpd-fc2 20040818<br />
sl-webalizer-fc2 20040818<br />
spamassassin-fc2 20040910<br />
tomcat-fc2 20040823<br />
usermin-fc2 20040909<br />
uw-imap-fc2 20040830<br />
webmin-fc2 20040909<br />
[root@virt14 private]# vzpkgls | grep fedora<br />
fedora-core-1 20040121 20040818<br />
fedora-core-devel-1 20040121 20040818<br />
fedora-core-2 20040903<br />
[root@virt14 private]#</pre><br />
<br />
For these older systems, you can simply match up the date on the template. <br />
<br />
For >= 3.0:<br />
<br />
<pre>[root@virt19 /vz2/private]# vzpkg list<br />
centos-5-x86 2008-01-07 22:05:57<br />
centos-5-x86 devel<br />
centos-5-x86 jre<br />
centos-5-x86 jsdk<br />
centos-5-x86 mod_perl<br />
centos-5-x86 mod_ssl<br />
centos-5-x86 mysql<br />
centos-5-x86 php<br />
centos-5-x86 plesk9<br />
centos-5-x86 plesk9-antivirus<br />
centos-5-x86 plesk9-api<br />
centos-5-x86 plesk9-atmail<br />
centos-5-x86 plesk9-backup<br />
centos-5-x86 plesk9-horde<br />
centos-5-x86 plesk9-mailman<br />
centos-5-x86 plesk9-mod-bw<br />
centos-5-x86 plesk9-postfix<br />
centos-5-x86 plesk9-ppwse<br />
centos-5-x86 plesk9-psa-firewall<br />
centos-5-x86 plesk9-psa-vpn<br />
centos-5-x86 plesk9-psa-fileserver<br />
centos-5-x86 plesk9-qmail<br />
centos-5-x86 plesk9-sb-publish<br />
centos-5-x86 plesk9-vault<br />
centos-5-x86 plesk9-vault-most-popular<br />
centos-5-x86 plesk9-watchdog</pre><br />
<br />
On these newer systems, it's difficult to tell whether the template on the dst matches exactly the src. Just cause a centos-5-x86 is listed on both servers doesn't mean all the same packages are there on the dst. To truly know, you must perform a sample rsync:<br />
<br />
rsync -avn /vz/template/centos/5/x86/ root@10.1.4.61:/vz/template/centos/5/x86/<br />
<br />
if you see a ton of output from the dry run command, then clearly there are some differences. You may opt to let the rsync complete (without running in dry run mode) the only downside is you've now used up more space on the dst and also the centos template will be a mess with old and new data- it will be difficult if not impossible to undo (if someday we wanted to reclaim the space).<br />
<br />
If you choose to merge templates, you should closely inspect the dry run output. You should also take care to exclude anything in the /config directory. For example:<br />
<br />
rsync -av -e ssh --stats --exclude=x86/config /vz/template/ubuntu/10.04/ root@10.1.4.62:/vz/template/ubuntu/10.04/<br />
<br />
Which will avoid this directory and contents:<br />
<pre>[root@virt11 /vz2/private]# ls /vz/template/ubuntu/10.04/x86/config*<br />
app os</pre><br />
<br />
This is important to avoid since the config may differ on the destination and we are really only interested in making sure the pacakges are there, not overwriting a newer config with an older one.<br />
<br />
If the dst system was missing a template, you have 2 choices: <br />
# put the missing template on the dst system. 2 choices here: <br />
## Install the template from rpm (found under backup2: /mnt/data4/vzrpms/distro/) or <br />
## rsync over the template (found under /vz/template) - see above<br />
# put the ve on a system which has all the proper templates<br />
<br />
=== pre-seeding a migration ===<br />
<br />
When migrating a customer (or when doing many) depending on how much data you have to transfer, it can take some time. Further, it can be difficult to gauge when a migration will complete or how long it will take. To help speed up the process and get a better idea about how long it will take you can pre-transfer a customer's data to the destination server. If done correctly, vzmigrate will see the pre-transferred data and pick up where you left off, having much less to transfer (just changed/new files). <br />
<br />
We believe vzmigrate uses rsync to do it's transfer. Therefore not only can you use rsync to do a pre-seed, you can also run rsync to see what is causing a repeatedly-failing vzmigrate to fail. <br />
<br />
There's no magic to a pre-seed, you just need to make sure it's named correctly.<br />
<br />
Given:<br />
<br />
source: /vz1/private/1234<br />
<br />
and you want to migrate to /vz2 on the target system, your rsync would look like:<br />
<br />
rsync -av /vz1/private/1234/ root@x.x.x.x:/vz2/private/1234.migrated/<br />
<br />
After running that successful rsync, the ensuing migrateonline (or migrate) will take much less time to complete- depending on the # of files to be analyzed and the # of changed files. In any case, it'll be much much faster than had you just started the migration from scratch.<br />
<br />
Further, as we discuss elsewhere in this topic, a failed migration can be moved from <tt>/vz/private/1234</tt> to <tt>/vz/private/1234.migrated</tt> on the destination if you want to restart a failed migration. This should '''only''' be done if the migration failed and the CT is not running on the destination HN.<br />
<br />
=== migrateonline intructions: src >=3.x -> dst>=3.x ===<br />
<br />
A script called [[#migrateonline|migrateonline]] was written to handle this kind of move. It is basically a wrapper for <tt>vzmigrate</tt> – vzmigrate is a util to seamlessly- as no no reboot of the ve necessary- move a ve from one host to another. This wrapper was initially written cause vz virtuozzo version 2.6.0 has a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables, causing problems when the ve was started up on the dst system. [[#migrate|migrate]] mitigates that. Since it makes multiple ssh connections to the dst virt, it’s a good idea to put the pub key for the src virt in the authorized_keys file on the dst virt. In addition, migrateonline emails ve owners when their migration starts and stops. For this to happen they need to put email addresses (on a single line, space delimited) in a file on their system: /migrate_notify. If the optional target dir is not specified, the ve will be moved to the same private/root location as it was on the src virt. Note: <tt>migrate</tt> is equivalent to <tt>migrateonline</tt>, but will <tt>migrate</tt> a ve AND restart it in the process.<br />
<br />
<pre>[root@virt12 sbin]# migrateonline<br />
usage: /usr/local/sbin/migrateonline <ip of node migrating to> <veid> [target dir: vz | vz1 | vz2]<br />
[root@virt12 sbin]# migrateonline 10.1.4.64 1212 vz<br />
starting to migrate 1212 at Sat Mar 26 22:40:38 PST 2005<br />
Turning off offline_management<br />
Saved parameters for VE 1212<br />
migrating with no start on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
clearing the arp cache<br />
now going to 10.1.4.64 and clear cache and starting<br />
starting it<br />
Delete port redirection<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
finished migrating 1212 at Sat Mar 26 22 22:52:01 PST 2005<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine.<br />
<br />
If they had backups, use the mvbackups command to move their backups to the new server:<br />
<br />
mvbackups 1212 virt14 vz<br />
<br />
Rename the ve<br />
<br />
[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/migrated-1212<br />
<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/old-1212-migrated-20120404-noarchive<br />
<br />
Update the customer’s systems in mgmt to reflect the new path and server.<br />
<br />
<br />
IF migrateonline does not work, you can try again using simply migrate- this will result in a brief reboot for the ve.<br />
Before you try again, make sure of a few things:<br />
<br />
Depending on where in the migration died, there may be partial data on the dst system in 1 of 2 places:<br />
(given the example above)<br />
<br />
/vz/private/1212<br />
<br />
or <br />
<br />
/vz/private/1212.migrated<br />
<br />
before you run migrate again, you'll want to rename so that all data is in <br />
1212.migrated:<br />
<br />
mv /vz/private/1212 /vz/private/1212.migrated<br />
<br />
this way, it will pick up where it left off and transfer only new files.<br />
<br />
Likewise, if you want to speed up a migration, you can pre-seed the dst as follows:<br />
<br />
[root@virt12 sbin]# rsync -avSH /vz/private/1212/ root@10.1.4.64:/vz/private/1212.migrated/<br />
<br />
then when you run migrate or migrateonline, it will only need to move the changed files- the migration will complete quickly<br />
<br />
=== migrateonline/migrate failures (migrate manually) ===<br />
<br />
Lets say for whatever reason the migration fails. If it fails with [[#migrateonline|migrateonline]], you should try [[#migrate|migrate]] (which will reboot the customer, so notify them ahead of time).<br />
<br />
You may want to run a [[#pre-seeding_a_migration|pre-seed]] rsync to see if you can find the problem. On older virts, we've seen this problem due to a large logfile (which you can find and encourage the customer to remove/compress):<br />
for f in `find / -size +1048576k`; do ls -lh $f; done<br />
<br />
You may also see migration failing due to quota issues.<br />
<br />
You can try to resolve by copying any quota file into the file you need:<br />
<br />
cp /var/vzquota/quota.1 /var/vzquota/quota.xxx<br />
<br />
If it complains about quota running you should then be able to stop it<br />
<br />
vzquota off xxxx<br />
<br />
If all else fails, migrate to a new VEID<br />
i.e. 1234 becomes 12341<br />
<br />
<br />
If the rsync or [[#migrate|migrate]] fails, you can always move someone manually:<br />
<br />
1. stop ve: <br><br />
v stop 1234<br />
<br />
2. copy over data<br><br />
rsync -avSH /vz/private/1234/ root@1.1.1.1:/vzX/private/1234/<br />
<br />
NOTE: if you've previously seeded the data (run rsync while the VE was up/running), and this is a subsequent rsync, make sure the last rsync you do (while the VE is not running, has the --delete option in the rsync)<br />
<br />
3. copy over conf<br><br />
scp /vzconf/1234.conf root@1.1.1.1:/vzconf<br />
<br />
4. on dst, edit the conf to reflect the right vzX dir<br><br />
vi /vzconf/1234.conf<br />
<br />
5. on src remove the IPs<br><br />
ipdel 1234 2.2.2.2 3.3.3.3<br />
<br />
6. on dst add IPs <br><br />
ipadd 1234 2.2.2.2 3.3.3.3<br />
<br />
7. on dst, start ve: <br><br />
v start 1324<br />
<br />
8. cancel, then archive ve on src per above instrs.<br />
<br />
=== migrate src=2.6.0 -> dst>=2.6.0, or mass-migration with customer notify ===<br />
<br />
A script called <tt>migrate</tt> was written to handle this kind of move. It is basically a wrapper for vzmigrate – vzmigrate is a util to seamlessly move a ve from one host to another. This wrapper was initially written cause vz virtuozzo version 2.6.0 has a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables, causing problems when the ve was started up on the dst system. migrate mitigates that. Since it makes multiple ssh connections to the dst virt, it’s a good idea to put the pub key for the src virt in the authorized_keys file on the dst virt. In addition, migrate emails ve owners when their migration starts and stops. For this to happen they need to put email addresses (on a single line, space delimited) in a file on their system: /migrate_notify. If the optional target dir is not specified, the ve will be moved to the same private/root location as it was on the src virt. Note: migrateonline is equivalent to migrate, but will migrate a ve from one 2.6 '''kernel''' machine to another 2.6 kernel machine without restarting the ve.<br />
<br />
<pre>[root@virt12 sbin]# migrate<br />
usage: /usr/local/sbin/migrate <ip of node migrating to> <veid> [target dir: vz | vz1 | vz2]<br />
[root@virt12 sbin]# migrate 10.1.4.64 1212 vz<br />
starting to migrate 1212 at Sat Mar 26 22:40:38 PST 2005<br />
Turning off offline_management<br />
Saved parameters for VE 1212<br />
migrating with no start on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
clearing the arp cache<br />
now going to 10.1.4.64 and clear cache and starting<br />
starting it<br />
Delete port redirection<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
finished migrating 1212 at Sat Mar 26 22 22:52:01 PST 2005<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine (backup2 or mail).<br />
Cancel the ve (first we have to rename things which migrate changed so cancelve will find them):<br />
<br />
[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/1212.conf<br />
<br />
On 2.6.1 you’ll also have to move the private area:<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/1212<br />
<br />
<pre>[root@virt12 sbin]# cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, [[#cancelve|cancelve]] would offer to remove them. You want to say '''no''' to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
To move everyone off a system, you’d do:<br />
for f in `vl`; do migrate <ip> $f; done<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
=== vzmigrate: src=2.6.1 -> dst>=2.6.0 ===<br />
<br />
This version of vzmigrate works properly with regard to handling ips. It will not notify ve owners of moves as in the above example. Other than that it’s essentially the same.<br />
<br />
<pre>[root@virt12 sbin]# vzmigrate 10.1.4.64 -r no 1212:1212:/vz/private/1212:/vz/root/1212<br />
migrating on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine (backup2 or mail).<br />
Cancel the ve (first we have to rename things which vzmigrate changed so cancelve will find them):<br />
<br />
<pre>[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/1212.conf<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/1212<br />
<br />
[root@virt12 sbin]# cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, <tt>cancelve</tt> would offer to remove them. You want to say no to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
=== src=2.5.x ===<br />
<br />
First, go to the private dir:<br />
<br />
cd /vz1/private/<br />
<br />
Stop the VE - make sure it stops totally cleanly.<br />
<br />
vzctl stop 1212<br />
<br />
Then you’d use vemove - a script written to copy over the config, create tarballs of the ve’s data on the destination virt, and cancel the ve on the source system (in this example we’re going to put a ve that was in /vz1/private on the src virt, in /vz/private on the dst virt):<br />
<br />
<pre>[root@virt12 sbin]# vemove<br />
ERROR: Usage: vemove veid target_ip target_path_dir<br />
[root@virt12 sbin]# vemove 1212 10.1.4.64 /vz/private/1212<br />
tar cfpP - 1212 --ignore-failed-read | (ssh -2 -c arcfour 10.1.4.64 "split - -b 1024m /vz/private/1212.tar" )<br />
scp /vzconf/1212.conf 10.1.4.64:/vzconf<br />
cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, cancelve would offer to remove them. You want to say no to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
When you are done, go to /vz/private on the dst virt you will have files like this:<br />
<br />
<pre>1212.taraa<br />
1212.tarab<br />
1212.tarac</pre><br />
<br />
Each one 1024m (or less, for the last one) in size.<br />
<br />
on the dst server and run:<br />
<br />
cat 1212.tar?? | tar xpPBf -<br />
<br />
and after 20 mins or so it will be totally untarred. Now since the conf<br />
file is already there, you can go ahead and start the system.<br />
<br />
vzctl start 1212<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
NOTE: you MUST tar the system up using the virtuozzo version of tar that<br />
is on all the virt systems, and further you MUST untar the tarball with<br />
the virtuozzo tar, using these options: `<tt>tar xpPBf -</tt>`<br />
<br />
If you tar up an entire VE and move it to a non-virtuozzo machine, that is<br />
ok, and you can untar it there with normal tar commands, but do not untar<br />
it and then repack it with a normal tar and expect it to work - you need<br />
to use virtuozzo tar commands on virtuozzo tarballs to make it work.<br />
<br />
The backups are sort of an exception, since we are just (usually)<br />
restoring user data that was created after we gave them the system, and<br />
therefore has nothing to do with magic symlinks or vz-rpms, etc.<br />
<br />
== Moving a VE on the same virt ==<br />
<br />
Easy way:<br><br />
Scenario 1: ve 123 is to be renamed 1231 and moved from vz1 to vz<br />
<br />
vzmlocal 123:1231:/vz/private/1231:/vz/root/1231<br />
<br />
Scenario 2: ve 123 is to be moved vz1 to vz<br />
<br />
vzmlocal 123:123:/vz/private/123:/vz/root/123<br />
<br />
vzmlocal will reboot the ve at the end of the move<br />
<br />
Manual/old way:<br />
<br />
1) <tt>vzctl stop 123</tt><br><br />
2) <tt>mv /vz1/private/123 /vz/private/.</tt><br><br />
(or cp -a if you want to copy)<br />
3) in <tt>/etc/sysconfig/vz-scripts/123.conf</tt> change value<br><br />
of '<tt>VE_PRIVATE</tt>' variable to point to a new private area location<br />
4) <tt>vzctl start 123</tt><br><br />
5) update backups if needed: <tt>mvbackups 123 virtX virt1 vz</tt><br><br />
6) update management scerens<br><br />
<br />
Notes: a) absolute path to private area is stored in quota file <tt>/var/vzquota/quota.123</tt> - so during first startup quota will be recalculated.<br><br />
b) if you're going to write some script to do a job, you MUST be sure that $VEID won't be expanded to '' in ve config file - ie. you need to escape '$'. Otherwise you might have:<br />
<br />
VE_PRIVATE="/vz/private/"<br />
<br />
in config, and 'vzctl destroy' for this VE ID '''will remove everything under /vz/private/ directory'''.<br />
<br />
== Adding a veth device to a VE ==<br />
<br />
Not totally sure what this is, but a customer asked for it and here's what we did (as instructed by vz support):<br />
<br />
<pre>v set 99 --netif_add eth99 --save<br />
ipdel 99 69.55.230.58<br />
v set 99 --ifname eth99 --ipadd 69.55.230.58 --save<br />
v set 99 --ifname eth99 --gateway 69.55.230.1 --save<br />
<br />
vznetcfg net new veth_net<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
net99 active eth0 veth77.77,veth99.99<br />
veth_net active<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
br0 bridge veth_net<br />
br99 bridge net99<br />
veth99.99 veth net99<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic net99 69.55.227.70/24<br />
<br />
vznetcfg br attach br0 eth0<br />
<br />
(will remove 99 from orig net and move to veth_net)<br />
vznetcfg net addif veth_net veth99.99<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
net99 active<br />
veth_net active eth0 veth77.77,veth99.99<br />
<br />
(delete the old crap)<br />
vznetcfg net del net99<br />
<br />
<br />
Then, to add another device in<br />
<br />
v set 77 --netif_add eth77 --save<br />
ipdel 77 69.55.230.78<br />
v set 77 --ifname eth77 --ipadd 69.55.230.78 --save<br />
v set 77 --ifname eth77 --gateway 69.55.230.1 --save<br />
v set 77 --save --ifname eth77 --network veth_net<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
veth77.77 veth<br />
br0 bridge veth_net<br />
veth99.99 veth veth_net<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic veth_net 69.55.227.70/24<br />
<br />
vznetcfg net addif veth_net veth77.77<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
veth77.77 veth veth_net<br />
br0 bridge veth_net<br />
veth99.99 veth veth_net<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic veth_net 69.55.227.70/24<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
veth_net active eth0 veth77.77,veth99.99<br />
<br />
another example<br />
<br />
v set 1182 --netif_add eth1182 --save<br />
ipdel 1182 69.55.236.217<br />
v set 1182 --ifname eth1182 --ipadd 69.55.236.217 --save<br />
v set 1182 --ifname eth1182 --gateway 69.55.236.1 --save<br />
vznetcfg net addif veth_net veth1182.1182<br />
v set 1182 --save --ifname eth1182 --network veth_net<br />
<br />
<br />
unused/not working commands:<br />
ifconfig veth99.0 0<br />
vznetcfg net list<br />
vznetcfg br new br99 net99<br />
vznetcfg br attach br99 eth0<br />
vznetcfg br show<br />
vznetcfg if list<br />
vznetcfg net addif net99 veth99.99<br />
vznetcfg net addif eth0 net99<br />
<br />
---------<br />
<br />
vznetcfg br new br1182 net1182<br />
<br />
vznetcfg br attach br99 eth0<br />
vznetcfg if list<br />
vznetcfg net addif net99 veth99.99<br />
vznetcfg net addif eth0 net99<br />
<br />
<br />
<br />
vznetcfg net addif eth0 net1182<br />
<br />
<br />
# vznetcfg net addif veth99.0 net99<br />
--- 8< ---<br />
<br />
vznetcfg net new net<br />
# vznetcfg net addif eth0 net99<br />
<br />
<br />
# vzctl set 99 --save --netif_add eth0 (at this stage veth99.0 interface have to appear<br />
on node)<br />
# vzctl set 99 --save --ifname eth0 --ipadd 69.55.230.58 (and probably few more arguments<br />
here - see 'man vzctl')<br />
# vznetcfg net addif veth99.0 net99</pre><br />
<br />
<br />
== Assigning/remove ip from a VE ==<br />
<br />
1. Add or remove ips:<br />
ipdel 1234 1.1.1.1 2.2.2.2<br />
ipadd 1234 1.1.1.1 2.2.2.2<br />
<br />
2. update Mgmt screens<br />
<br />
3. offer to update any DNS we do for them<br />
<br />
4. check to see if we had rules for old IP in firwall<br />
<br />
== Enabling tun device for a ve ==<br />
Note, there’s a command for this: [[#addtun|addtun]]<br />
<br />
For posterity:<br />
Make sure the tun.o module is already loaded before Virtuozzo is started: <br />
lsmod <br />
Allow the VPS to use the TUN/TAP device: <br />
vzctl set 101 --devices c:10:200:rw --save <br />
Create the corresponding device inside the VPS and set the proper permissions: <br />
vzctl exec 101 mkdir -p /dev/net <br />
vzctl exec 101 mknod /dev/net/tun c 10 200 <br />
vzctl exec 101 chmod 600 /dev/net/tun<br />
<br />
== Remaking a system (on same virt) ==<br />
<br />
1. [[#cancelve|cancelve]] (or v destroy x - ONLY if you're POSITIVE no data needs to be saved)<br />
<br />
2. [[#vemake|vemake]] using same veid<br />
<br />
3. [[#mvbackups|mvbackups]] or [[#vb|vb]] (if new mount point)<br />
<br />
4. update mgmt with new dir/ip <br />
<br />
5. update firewall if changed ip<br />
<br />
== Re-initialize quota for a VE ==<br />
<br />
There’s a commamd for this now: [[#clearquota|clearquota]]<br />
<br />
For posterity:<br />
<br />
vzctl stop 1<br />
vzquota drop 1<br />
vzctl start 1<br />
<br />
<br />
== Traffic accounting on linux ==<br />
<br />
DEPRECATED - all tracking is done via bwdb now. This is how we used to track traffic.<br />
<br />
TODO: update for diff versions of vz<br />
<br />
Unlike FreeBSD, where we have to add firewall count rules to the system to count the traffic, on virtuozzo counts the traffic for us. You an see the current traffic stats by running `vznetstat`:<br />
<br />
<pre># vznetstat<br />
VEID Net.Class Output(bytes) Input(bytes)<br />
-----------------------------------------------<br />
24218 1 484M 39M<br />
24245 1 463M 143M<br />
2451 1 2224M 265M<br />
2454 1 2616M 385M<br />
4149 1 125M 68M<br />
418 1 1560M 34M<br />
472 1 1219M 315M<br />
726 1 628M 317M<br />
763 1 223M 82M<br />
771 1 4234M 437M<br />
-----------------------------------------------<br />
Total: 13780M 2090M</pre><br />
<br />
As you can see the VEID is on a line with the in and out bytes. So, we simply run a cron job:<br />
<br />
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /root/vztrafdump.sh<br />
<br />
Just like we do on FreeBSD - this one goes through all the VEs in /vz/private and greps the line from vznetstat that matches them and dumps it in /jc_traffic_dump on their system. Then it does it again for all the VEs in /vz1/private. It is important to note that vznetstat runs only once, and the grepping is done from a temporary file that contains that output - we do this because running vznetstat once for each VE that we read out of /vz/private and /vz1/private would take way too long and be too intensive.<br />
<br />
You do not need to do anything to facilitate this other than make sure that that cron job is running - the vznetstat counters are always running, and any new VEs that are added to the system will be accounted for automatically.<br />
<br />
Traffic resetting no longer works with vz 2.6, so we disable the vztrafdump.sh on those virts.<br />
<br />
== Watchdog script ==<br />
<br />
On some of the older virts, we have a watchdog running that kills procs that are deemed bad per the following:<br />
<br />
/root/watchdog from quar1<br />
<br />
<pre>if echo $line | awk '{print $(NF-3)}' | grep [5-9]...<br />
then<br />
# 50-90%<br />
if echo $line | awk '{print $(NF-1)}' | grep "...:.."<br />
then<br />
# running for > 99min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
if echo $line | awk '{print $(NF-1)}' | grep "....m"<br />
then<br />
# running for > 1000min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
fi<br />
<br />
if echo $line | awk '{print $(NF-3)}' | grep [1-9]...<br />
then<br />
# running for 10-90 percent<br />
if echo $line | awk '{print $NF}' | egrep 'cfusion|counter|vchkpw'<br />
then<br />
<br />
if echo $line | awk '{print $(NF-1)}' | grep "[2-9]:.."<br />
then<br />
# between 2-9min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
elif echo $line | awk '{print $(NF-1)}' | grep "[0-9][0-9]:.."<br />
then<br />
# up to 99min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
fi<br />
fi</pre><br />
<br />
<br />
== Misc Linux Items ==<br />
<br />
We are overselling hard drive space ... when you configure a linux system with a certain amount of disk space (the default is 4gigs) you do not actually use up 4gigs of space on the system. The diskspace setting for a user is simply a cap, and they only use up as much space on the actual disk drive as they are actually using.<br />
<br />
When you create a new linux system, even though there are some 300 RPMs or so installed, if you run `df -k` you will see that the entire 4gig partition is empty - no space is being used. This is because the files in their system are "magic symlinks" to the template for their OS that is in /vz/template - however, any changes to any of those files will "disconnect" them and they will immediately begin using space in their system. Further, any new files uploaded (even if those new files overwrite existing files) will take up space on the partition.<br />
<br />
<br />
if you see this:<br />
<br />
<pre>[root@virt8 root]# vzctl stop 160 ; vzctl start 160<br />
VE is not running<br />
Starting VE ...<br />
VE is unmounted<br />
VE is mounted<br />
Adding IP address(es): 69.55.226.83 69.55.226.84<br />
bash ERROR: Can't change file /etc/sysconfig/network<br />
Deleting IP address(es): 69.55.226.83 69.55.226.84<br />
VE is unmounted<br />
[root@virt8 root]#</pre><br />
<br />
it probably means they no longer have /bin/bash - copy one in for them<br />
<br />
ALSO: another possibility is that they have removed the `ed` RPM from their system - it needs to be reinstalled into their system. But since their system is down, this is tricky ...<br />
<br />
VE startup scripts used by 'vzctl' want package 'ed' to be available inside VE. So if package 'ed' will be enabled in OS template config and OS template itself VE #827 is based on - this error should be fixed.<br />
<br />
yes, it is possible to add RPM to VE while it not running.<br />
Try to do following:<br />
<br />
<pre># cd /vz/template/<OS_template_with_ed_package>/<br />
# vzctl mount 827<br />
# rpm -Uvh --root /vz/root/827 --veid 827 ed-0.2-25.i386.vz.rpm</pre><br />
<br />
Usually theres an error, but its ok<br />
<br />
Note: replace 'ed-0.2-25.i386.vz.rpm' in last command with actual<br />
version of 'ed' package you have.<br />
<br />
----<br />
<br />
So how do I know what template the user has ? cat their conf file and it is listed in there. For example, if the conf file has:<br />
<br />
<pre>[root@virt12 sbin]# vc 1103<br />
…snip…<br />
OSTEMPLATE="debian-3.0/20030822"<br />
TEMPLATES="mod_perl-deb30/20030707 mod_ssl-deb30/20030703 mysql-deb30/20030707 proftpd-deb30/20030703 webmin-deb30/20030823 "<br />
…</pre><br />
<br />
then they are on debian 3.0, all of their system RPMs are in /vz/template/debian-3.0, and they are using version 20030822 of that debian 3.0 template. Also, they’ve also got additional packages installed (mod_perl, mod_ssl, etc). Those are also found under /vz/template<br />
<br />
----<br />
<br />
Edits needed to run java:<br />
<br />
When we first created the VEs, the default setting for privvmpages was 93000:94000 ... which was high enough that most people never had problems ... however, you can;t run java or jdk or tomcat or anything java related with that setting. We have found that by setting privvmpages to 610000:615000 that java runs just fine. That is now the default setting. It is exceedingly rare that anyone needs it higher than that, although we have seen it once or twice.<br />
<br />
Any problems with java at all - the first thing you need to do is see if the failcnt has raised for privvmpages.<br />
<br />
<pre># vzctl start 160<br />
Starting VE ...<br />
vzquota : (error) Quota on syscall for 160: Device or resource busy<br />
Running vzquota on failed for VE 160 [3]</pre><br />
<br />
This is because my pwd is _in_ their private directory - you can't start it until you move out<br />
<br />
People seem to have trouble with php if they are clueless newbies. Here are two common problems/solutions:<br />
<br />
no... but i figured it out myself. problem was the php.ini file that came<br />
vanilla with the account was not configured to work with apache (the<br />
ENGINE directive was set to off).<br />
<br />
everything else seems fine now.<br />
<br />
----<br />
<br />
the problem was in the php.ini file. I noticed that is wasnt showing<br />
the code when it was in an html file so I looked at the php.ini file<br />
and had to change it so it recognized <? tags aswell as <?php tags.<br />
<br />
Also, make sure added to httpd.conf<br />
AddType application/x-httpd-php .php<br />
<br />
----<br />
<br />
You can set the time zone:<br />
<br />
You can change the timezone by doing this:<br />
<br />
ln -sf /usr/share/zoneinfo/<zone> /etc/localtime<br />
<br />
where <zone> is the zone you want in the /usr/share/zoneinfo/ directory.<br />
<br />
----<br />
<br />
Failing shm_open calls:<br />
<br />
first, please check if /dev/shm is mounted inside VE.<br />
'cat /proc/mounts' command should show something like this:<br />
tmpfs /dev/shm tmpfs rw 0 0<br />
<br />
If /dev/shm is not mounted you have 2 ways to solve issue:<br />
1. execute following command inside VE (doesn't require VE reboot):<br />
mount -t tmpfs none /dev/shm<br />
2. add following string to /etc/fstab inside VE and reboot it:<br />
tmpfs /dev/shm tmpfs defaults 0 0<br />
<br />
----<br />
<br />
You can have a mounted but not running ve<br />
Just:<br />
vzctl mount <veid><br />
<br />
----<br />
<br />
When a debian sys can’t get on the network, and you try:<br />
<br />
<pre>vzctl set 1046 --ipadd 69.55.227.117<br />
Adding IP address(es): 69.55.227.117<br />
Failed to bring up lo.<br />
Failed to bring up venet0.</pre><br />
<br />
They probably removed iproute package, which must be the one from swsoft. To restore:<br />
<pre># dpkg -i --veid=1046 --admindir=/vz1/private/1046/root/var/lib/dpkg --instdir=/vz1/private/1046/root/ /vz/template/debian-3.0/iproute_20010824-8_i386.vz.deb<br />
(Reading database ... 16007 files and directories currently installed.)<br />
Preparing to replace iproute 20010824-8 (using .../iproute_20010824-8_i386.vz.deb) ...<br />
Unpacking replacement iproute ...<br />
Setting up iproute (20010824-8) ...</pre><br />
<br />
Then restart their ve<br />
<br />
----<br />
<br />
in a ve i do:<br />
<br />
<pre>cd /<br />
du -h .</pre><br />
<br />
and get: 483M .<br />
<br />
i do:<br />
<br />
<pre>bash-2.05a# df -h<br />
Filesystem Size Used Avail Use% Mounted on<br />
/dev/vzfs 4.0G 2.3G 1.7G 56% /</pre><br />
<br />
how can this be?<br />
<br />
Is it possible that quota file was corrupted somehow? Please try to: <br />
<pre>vzctl stop <VEID><br />
vzquota drop <VEID><br />
vzquota init <VEID><br />
vzctl start <VEID></pre><br />
<br />
----<br />
<br />
How to stop vz from starting after reboot:<br />
<br />
VIRTUOZZO=no <br />
in <br />
/etc/sysconfig/vz<br />
<br />
To start: <br />
service vz start<br />
(after setting VIRTUOZZO=yes in /etc/sysconfig/vz)<br />
<br />
service vz restart will do some kind of 'soft reboot' -- restart all<br />
VPSes and reload modules without rebooting the node<br />
<br />
if you need to shut down all VPSes really really fast, run killall -9 init<br />
<br />
----<br />
<br />
Postfix tip:<br />
<br />
You may want to tweak settings: default_process_limit=10<br />
<br />
= Virtuozzo VPS Management Tools =<br />
<br />
== vm ==<br />
<br />
TODO<br />
<br />
== cancelve ==<br />
cancelve <veid><br />
this will:<br />
* stop a ve<br />
* check for backups (offer to remove them from the backup server <br />
and the backup.config)<br />
* rename the private dir<br />
* check for PTR, provide the commands to reset to default<br />
* and rename the ve’s config<br />
* remind you to remove firewall rules<br />
* remind you to remove DNS entries<br />
<br />
== ipadd ==<br />
ipadd <veid> <ip> [ip] [ip]<br />
add’s ip(s) to a ve<br />
<br />
== ipdel ==<br />
ipdel <veid> <ip> [ip] [ip]<br />
removes ip(s) from a ve<br />
<br />
== vc ==<br />
vc <veid><br />
the equivalent of: <tt>cat /vzconf/<veid>.conf</tt><br />
<br />
== vl ==<br />
vl<br />
will displays a list of ve #’s, 1 per line. (ostensibly to use in a for loop)<br />
<br />
== vp ==<br />
vp <veid><br />
the equivalent of: <tt>vzps auxww –E <veid></tt><br />
<br />
== vpe ==<br />
DEPRECATED<br />
vpe <veid> <br />
this will allow you to do a vp when a ve is running out of control, the equivalent of (deprecated since vp operates outside the VPS): <br />
<pre>vzctl set <veid> --kmemsize 2100000:2200000<br />
vzctl exec <veid> ps auxw<br />
vzctl set <veid> --kmemsize (ve’s orig lvalue):(ve’s orig hvalue)</pre><br />
<br />
== vt ==<br />
vt <veid><br />
the equivalent of: <tt>vztop –E <veid></tt><br />
<br />
== vr ==<br />
vr <veid><br />
the equivalent of: <tt>vzctl stop <veid>; vzctl start <veid></tt><br />
You can run this even if the ve is down - the stop command will just fail<br />
<br />
== vs ==<br />
vs [veid]<br />
displays status (output of <tt>vzctl status <veid></tt>) on each ve configured on the system (as determined by <tt>/vzconf/*.conf</tt>)<br />
If passed an argument, gives the status for just that ve. <br />
A running system looks like:<br />
<tt>VEID 16066 exist mounted running</tt><br />
<br />
A ve which is not running (but does exist) looks like:<br />
<tt>VEID 9990 exist unmounted down</tt><br />
<br />
A ve which is not running and doesn’t exist looks like:<br />
<tt>VEID 421 deleted unmounted down</tt><br />
<br />
== vs2 ==<br />
vs2 [veid]<br />
this is similar to vs in that it displays status (output of <tt>vzctl status <veid></tt>) on each ve,<br />
but the difference is it’s list comes from doing an ls on the data dirs. This was meant to catch <br />
the rare case where a ve configured but exists. <br />
<br />
== vw ==<br />
vw [veid]<br />
displays the output of ‘<tt>w</tt>’ (the equivalent of <tt>vzctl exec <veid> w</tt>) for each configured ve (as determined by <tt>/vzconf/*.conf</tt>). Useful for determing which ve is contributing to a heavily-loaded system.<br />
If passed an argument, gives ‘<tt>w</tt>‘ output for just that ve. <br />
Ex:<br />
<pre>[root@virt2 etc]# vw<br />
134<br />
10:52pm up 79 days, 6:14, 0 users, load average: 0.02, 0.02, 0.00<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT<br />
16027<br />
2:52pm up 7 days, 19:54, 0 users, load average: 0.00, 0.00, 0.00<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT<br />
16055<br />
2:52pm up 79 days, 6:38, 0 users, load average: 0.00, 0.04, 0.07<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT</pre><br />
<br />
== vwe ==<br />
vwe [constraint]<br />
just like <tt>vw</tt>, but takes a constraint as an argument, only show’s ve’s with loads >= the constraint provided. If no constraint is provided, 1 is used by default<br />
<br />
== vzs ==<br />
vzs [veid]<br />
displays the beancounter status for all ve’s, or a particular ve if an argument is passed<br />
<br />
== ve ==<br />
ve <veid><br />
the equivalent of: <tt>vzctl enter <veid></tt><br />
<br />
== vx ==<br />
vx <veid> <command><br />
the equivalent of: <tt>/usr/sbin/vzctl exec <veid> <command></tt><br />
<br />
== dprocs ==<br />
dprocs [count]<br />
a script which outputs a continuous report (or a certain number of reports if an option is passed) of processes stuck in the D state and which VPS’s those procs belong to.<br />
<br />
== setmem ==<br />
setmem <VEID> <256|512|768|1024|2048><br />
adjusts the memory resources for the VE<br />
<br />
== afacheck.sh ==<br />
afacheck.sh<br />
displays the health/status of containers and mirrors on an adaptec card (currently quar1, tempvirt1-2, virt9, virt10)- all other are LSI<br />
<br />
== backup ==<br />
backup<br />
backup script called nightly to update virt scripts and do customer backups<br />
<br />
== checkload.pl ==<br />
checkload.pl<br />
this was intended to be setup as a cronjob to watch processes on a virt when the load <br />
rises above a certain level. Not currently in use.<br />
<br />
== findbackuppigs.pl ==<br />
findbackuppigs.pl<br />
looks for files larger than 50MB which customers have asked us to backup. Emails matches<br />
to linux@johncompanies.com<br />
<br />
== gatherlinux.pl ==<br />
gatherlinux.pl<br />
gathers up data about ve’s configured and writes to a file. Used for audits against the db<br />
<br />
== gb ==<br />
gb <search><br />
greps backup.config for the given search parameter<br />
<br />
== gbg ==<br />
gbg <search><br />
greps backup.config for the given search parameter and presents just the directories (for clean pasting)<br />
<br />
== linuxtrafficgather.pl ==<br />
linuxtrafficgather.pl [yy-mm]<br />
sends a traffic usage summary by ve to support@johncomapnies.com and payments@johncompanies.com.<br />
Optional arguments are year and month (must be in the past). If not passed, assumes last month. Relies on <br />
traffic logs created by netstatreset and netstatbackup<br />
<br />
== linuxtrafficwatch.pl ==<br />
linuxtrafficwatch.pl<br />
checks traffic usage and emails support@johncomapnies.com when a ve reaches the warning level (35G) and<br />
the limit (40G). Works on virtuozzo versions <= 2.6.x. We really aren’t using this anymore now that we have netflow.<br />
<br />
== linuxtrafficwatch2.pl ==<br />
linuxtrafficwatch2.pl<br />
checks traffic usage and emails support@johncomapnies.com when a ve reaches the warning level (35G) and<br />
the limit (40G). Works on virtuozzo version 2.6.x. We really aren’t using this anymore now that we have netflow.<br />
<br />
== load.pl ==<br />
load.pl<br />
feeds info to load mrtg - executed by inetd.<br />
<br />
== mb (linux) ==<br />
mb <mount|umount><br />
(nfs) mounts and umounts dirs to backup2. Shortcuts are mbm and mbu to mount and unmount. <br />
<br />
== migrate ==<br />
migrate <ip of node migrating to> <veid> <target_veid> [target dir: vz | vz1 | vz2]<br />
this is basically a wrapper for <tt>vzmigrate</tt> – vzmigrate is a util to seamlessly move a ve from one host to another. This wrapper was written cause vz virtuozzo version 2.6 had a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables. This script mitigates that. Since it makes multiple ssh connections to the target host, it’s a good idea to put the pub key for the src system in the authorized_keys file on the target host. In addition, it emails ve owners when their migration starts and stops (if they place email addresses in a file on their system: /migrate_notify). To move everyone off a system, you’d do:<br />
for f in `vl`; do migrate <ip> $f; done<br />
<br />
== migrateonline ==<br />
migrateonline <ip of node migrating to> <veid> <target_veid> [target dir: vz | vz1 | vz2]<br />
this is the same as migrate but will migrate a ve in <tt>–online</tt> mode which means it won’t be shut down at the end of the migration. This only works when migrating ve’s between 2 machines running a 2.6 kernel (currently tempvirt1-2. virt16-19, virt12). If you get an error that the machine you’re trying to migrate to has a different CPU or features, etc, then you have to edit the file and add the –f switch to the vzmigrate line- you can basically ignore this kind of warning (but never ignore a warning about missing templates on the destination node). NOTE: This edit (if made to migrateonline) will be overwritten by the base script during each night’s backup.<br />
<br />
== netstatbackup ==<br />
DEPRECATED<br />
netstatbackup <br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.5.x. <br />
<br />
== netstatbackup2 ==<br />
DEPRECATED<br />
netstatbackup2<br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.6.x. <br />
<br />
== netstatreset ==<br />
DEPRECATED<br />
netstatreset<br />
writes traffic count data to a logfile and resets counters to 0. Works on virtuozzo versions 2.5.x <br />
<br />
== netstatreset2 ==<br />
DEPRECATED<br />
netstatreset2<br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.6.x. <br />
<br />
== orphanedbackupwatchlinux ==<br />
orphanedbackupwatchlinux <br />
looks for directories on backup2 which aren’t configured in backup.config and offers to <br />
delete them<br />
<br />
== rsync.backup (linux) ==<br />
rsync.backup<br />
does customer backups (relies on backup.config)<br />
<br />
== startvirt.pl ==<br />
startvirt.pl<br />
forks off start ve commands – keeps 6 running at a time. This is not to be used on systems where fastboot is enabled as it circumvents the benefit of the fastboot. The script will occasionally not exit gracefully and will continue to use up CPU, so it should be watched. Also, don’t exit from the script till you’re sure all ve’s are started – if you do you need to start them manually and may have to free up locks. On some systems, the startvirt script doesn’t exit cleanly and you have to ^C out of it. Be careful though- doing so can leave some VE’s in an odd bootup state and you may need to ‘vr’ them manually. You should check to see which ve’s aren’t running and/or confirm all have started when ^C’ing out of startvirt.<br />
<br />
== taskdone (linux) ==<br />
taskdone<br />
when called will email support@johncompanies.com with the hostname of the machine from which it was <br />
executed as the subject<br />
<br />
== vb (linux) ==<br />
vb<br />
the equivalent of: <tt>vi /usr/local/sbin/backup.config</tt><br />
<br />
== vemakeXX ==<br />
DEPRECATED<br />
vemakerh9 <br />
ve create script for RH9 (see vemake)<br />
<br />
vemakedebian30 <br />
ve create script for debian 3.0 (Woody) (see vemake)<br />
<br />
vemakedebian31 <br />
ve create script for debian 3.1 (Sarge) (see vemake)<br />
<br />
vemakedebian40 <br />
ve create script for debian 4.0 (Etch) (see vemake)<br />
<br />
vemakefedora, vemakefedora2, vemakefedora4, vemakefedora5, vemakefedora6, vemakefedora7<br />
ve create script for fedora core 1, 2, 4, 5, 6, 7 respectively (see vemake)<br />
<br />
vemakecentos3, vemakecentos4<br />
ve create script for centos 3, 4 respectively (see vemake)<br />
<br />
vemakesuse, vemakesuse93, vemakesuse100<br />
ve create script for suse 9.2, 9.3, 10.0 respectively (see vemake)<br />
<br />
vemakeubuntu5, vemakeubuntu606, vemakeubuntu606 vemakeubuntu704<br />
ve create script for ubuntu 5.10, 6.06, 6.10, 7.04 respectively (see vemake)<br />
<br />
== vemove ==<br />
DEPRECATED<br />
vemove <veid> <target_ip> </vz/private/123><br />
this script simplifies the old way of moving ve’s from one system to another - in short moving a ve to or from a virt running virtuozzo < 2.6.x<br />
It’s the equivalent of:<br />
<tt>tar cfpP - <veid> --ignore-failed-read | (ssh -2 -c arcfour <target_ip> "split - -b 1024m </vz/private/123>.tar" )</tt>This should only be used if migrate/vzmigrate can’t be used. <br />
<br />
== vim.watchdog ==<br />
vim.watchdog <br />
looks for and kills procs matching “vi|vim|nano|pine|elm” that are running for a long time & consuming lots of cpu. Works on virtuozzo versions 2.5.x<br />
<br />
== vim.watchdog2 ==<br />
vim.watchdog2<br />
looks for and kills procs matching “vi|vim|nano|pine|elm” that are running for a long time & consuming lots of cpu.<br />
Works on virtuozzo versions 2.6.x.<br />
<br />
== vzmigrate ==<br />
vzmigrate <target_ip> -r no <veid>:[dst veid]:[dst /vzX/private/veid]:[dst /vzX/root/veid]<br />
(this is the raw command “wrapped” by migrate/migrateonline) this will seamlessly move a ve from one host to another. The ve will run for the duration of the migration till the very end when it’s shut down, ip moved and started up on the target system. The filesystem on the src will remain. This should be watched – occasionally the move will timeout and leave the system shut down. If target private and root aren’t specified it just puts it in /vz. Only works when both systems are running virtuozzo 2.6.x<br />
<br />
== vztrafdump.sh ==<br />
DEPRECATED<br />
vztrafdump.sh<br />
writes traffic usage info by ve to a file called jc_traffic_dump in each ve’s / dir. Works on virtuozzo versions <= 2.5.x. <br />
<br />
== vztrafdump2.sh ==<br />
DEPRECATED<br />
vztrafdump2.sh<br />
writes traffic usage info by ve to a file called jc_traffic_dump in each ve’s / dir. Works on virtuozzo versions 2.6.x. <br />
<br />
== addtun ==<br />
addtun <veid><br />
Add’s tun device to ve.<br />
<br />
== bwcap ==<br />
bwcap <veid> <kbps><br />
Ex: <tt>bwcap 1234 512</tt><br />
Caps a VE’s bandwidth to the amount given<br />
<br />
== setdisk ==<br />
setdisk <veid> <diskspace in GB><br />
Ex: <tt>setdisk 1234 5</tt><br />
Gives a VE’s a given amount of disk space<br />
<br />
== vdf ==<br />
vdf <veid> <br />
the equivalent of: <tt>vzctl exec <veid> df –h</tt><br />
<br />
== vdff ==<br />
vdff<br />
runs a (condensed) vdf for all ve’s in your pwd (must be run from /vz/privateN)<br />
<br />
== mvbackups ==<br />
mvbackups <veid> <target_machine> (virt1) <target_dir> (vz1)<br />
moves backups from one location to another on the backup server, and provides you with option to remove entries from current backup.config, and simple paste command to add the config to backup.config on the target server<br />
<br />
== checkquota ==<br />
checkquota<br />
for all the ve’s in the cwd (run from /vz/private, /vz1/private, etc) reports what vz quota says they’re using and what the actual usage is (as reported by du)<br />
<br />
== clearquota ==<br />
clearquota <veid><br />
Recalculates a ve’s quota, prints out the usage before and after. The equivalent of:<br />
<tt>vdf <veid>; v stop <veid>; vzquota drop <veid>; v start <veid>; vdf <veid></tt><br />
<br />
== dprocs ==<br />
dprocs<br />
Sometimes the server’s have a large number of processes get stuck in the D state- this script shows (every 3 secs) which VE’s have D procs, which procs<br />
are stuck and a running average of the top “offenders”<br />
<br />
== vzstat ==<br />
vstat<br />
sort of like top for VZ. sort VEs by CPU usage by pressing 'o' and then 'c' keys<br />
<br />
== stopvirt ==<br />
stopvirt<br />
will stop VEs as fast as it can, 6 at a time. May not exit when complete so you should watch [[#vzstat|vzstat]] in another window.</div>Supporthttps://wiki.jcihosting.com/index.php?title=VPS_Management&diff=1032VPS Management2013-03-11T23:44:32Z<p>Support: /* Virtuozzo VPS */</p>
<hr />
<div>= Common Problems =<br />
== Login to any machine without a password ==<br />
<br />
This is possible via the use of ssh keys. The process is thus:<br />
<br />
1. place the public key for your user (root@mail) in the /root/.ssh/authorized_keys file on the server you wish to login to<br />
cat /root/.ssh/id_dsa.pub<br />
(paste that into authorized_keys on the target server). If the file doesn't exist, create it.<br />
<br />
2. enable root login (usually only applies to FreeBSD). Edit the /etc/ssh/sshd_config on the target server and change:<br />
<tt>#PermitRootLogin no</tt><br />
to<br />
<tt>PermitRootLogin yes</tt><br />
<br />
3. Restart the sshd on the target machine. First, find the sshd process: <br />
jailps <hostname> | grep sshd <br />
or <br />
vp <VEID> | grep sshd<br />
<br />
Look for the process resembling:<br />
root 17296 0.0 0.0 5280 1036 ? Ss 2011 4:27 /usr/sbin/sshd <br />
(this is the sshd)<br />
<br />
Not:<br />
root 6270 0.5 0.0 6808 2536 ? Ss 14:33 0:00 sshd: root [priv]<br />
(this is an sshd child- someone already ssh'd in as root)<br />
<br />
Restart the sshd: <br />
kill -1 <PID><br />
<br />
Ex:<br />
kill -1 17296<br />
<br />
You may now ssh in.<br />
<br />
Once you're done, IF you enabled root login, you should repeat steps 2 and 3 to disable root logins.<br />
<br />
== Letting someone in who has locked themselves out (killed sshd, lost pwd) ==<br />
<br />
There are two ways people frequently lock themselves out - either they forget a password, or they kill off sshd somehow.<br />
<br />
These are actually both fairly easy to solve. First, let's say someone kills off their sshd, or somehow mangles /etc/ssh/sshd_config such that it no longer lets them in.<br />
<br />
Their email may be very short, or it may have all sorts of details about how you should fix sshd_config to let them in ... just ignore all of this. They can fix their own mangled sshd. Fixing this is very simple. First, edit the /etc/inetd.conf on their system and uncomment the telnet line:<br />
<br />
telnet stream tcp nowait root /usr/libexec/telnetd telnetd<br />
#telnet stream tcp6 nowait root /usr/libexec/telnetd telnetd<br />
<br />
(just leave the tcp6 version of telnet commented)<br />
<br />
Then, use jailps to list the processes on their system, and find their inetd process. Then simply:<br />
<br />
kill -HUP (pid)<br />
<br />
where (pid) is the PID of their inetd process. Now they have telnet running on their system and they can log in and do whatever they need to do.<br />
<br />
The only complications that could occur are:<br />
<br />
a) their firewall config on our firewall has port 23 blocked, in which case you will need to open that - will be covered in a different lesson.<br />
<br />
b) they are not running inetd, so you can't HUP it. If this happens, edit their /etc/rc.conf, add the inetd_enable="YES" line, and then kill<br />
their jail with /tmp/jailkill.pl - then restart their jail with the jail line from their quad/safe file. Easy.<br />
<br />
If they have forgotten a password,<br />
<br />
On 6.x+ you can reset their password with:<br />
jexec <jailID from jls> passwd root<br />
<br />
Note: the default password for 6.x jails is 8ico2987, for 4.x it is p455agfa<br />
<br />
On 4.x, you need to cd to their etc directory<br />
... for instance:<br />
<br />
cd /mnt/data2/198.78.65.136-col00261-DIR/etc<br />
<br />
and run:<br />
<br />
vipw -d .<br />
<br />
Then paste in these two lines (theres a paste with these):<br />
<br />
root:$1$krszPxhk$xkCepSnz3mIikT3vCtJCt0:0:0::0:0:Charlie &:/root:/bin/csh<br />
user:$1$Mx9p5Npk$QdMU6c8YQqp2FW2M3irEh/:1001:1001::0:0:User &:/home/user:/bin/sh<br />
<br />
overwriting the lines they already have for "user" and "root" - then just tell them that both user and root have been reset to the default password of p455agfa.<br />
<br />
For linux, just passwd inside shell or <br />
vzctl set <veid> --userpasswd root:p455agfa –save<br />
<br />
Starting in 2009 we began giving out randomized passwords for FreeBSD and Linux as the default password. That is stored with each system in Mgmt. You should look for and reset the password to that password in the event of a reset and refer the customer to use their original password from their welcome email- this way we don’t have to send the password again via email (in clear text).<br />
<br />
<br />
== sendmail can’t be contacted from ext ip (only locally) ==<br />
<br />
By default redhat puts this line in sendmail.mc:<br />
<br />
DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')<br />
<br />
which makes it only answer on localhost. Comment it out like:<br />
<br />
dnl DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')<br />
<br />
and then rebuild sendmail.cf with:<br />
<br />
m4 /etc/mail/sendmail.mc > /etc/sendmail.cf<br />
<br />
== virt doesn’t properly let go of ve’s ip(s) when moved to another system ==<br />
<br />
On virtuozzo 2.6 systems, it's been observed that when moving ips from one virt to another that sometimes the routing table will not get updated to reflect the removal of the ip addresses.<br />
<br />
A recent example was a customer that was moving to a new ve on a new virt and the ip addresses were traded between the two ve's. After the trade the two systems were not able to talk to each other. When looking at the routing table for the old system all the ip addresses were still in the routing table as being local, like so:<br />
<br />
<pre>netstat -rn | grep 69.55.225.149<br />
69.55.225.149 0.0.0.0 255.255.255.255 UH 40 0 0 venet0</pre><br />
<br />
This was preventing traffic to the other system from being routed properly.<br />
The solution is to manually delete the route:<br />
<br />
route delete 69.55.225.149 gw 0.0.0.0<br />
<br />
Supposedly, this was fixed in 2.6.1<br />
<br />
== sshd on FreeBSD 6.2 segfaults ==<br />
<br />
First try to reinstall ssh<br />
<br />
<pre>cd /usr/src/secure<br />
cd lib/libssh<br />
make depend && make all install<br />
cd ../../usr.sbin/sshd<br />
make depend && make all install<br />
cd ../../usr.bin/ssh<br />
make depend && make all install</pre><br />
<br />
Failing that, find the library that’s messed up:<br />
<br />
<pre>ldd /usr/sbin/sshd<br />
libssh.so.3 => /usr/lib/libssh.so.3 (0x280a3000) <br />
libutil.so.5 => /lib/libutil.so.5 (0x280d8000) <br />
libz.so.3 => /lib/libz.so.3 (0x280e4000) <br />
libwrap.so.4 => /usr/lib/libwrap.so.4 (0x280f5000) <br />
libpam.so.3 => /usr/lib/libpam.so.3 (0x280fc000) <br />
libbsm.so.1 => /usr/lib/libbsm.so.1 (0x28103000) <br />
libgssapi.so.8 => /usr/lib/libgssapi.so.8 (0x28112000) <br />
libkrb5.so.8 => /usr/lib/libkrb5.so.8 (0x28120000) <br />
libasn1.so.8 => /usr/lib/libasn1.so.8 (0x28154000) <br />
libcom_err.so.3 => /usr/lib/libcom_err.so.3 (0x28175000) <br />
libroken.so.8 => /usr/lib/libroken.so.8 (0x28177000) <br />
libcrypto.so.4 => /lib/libcrypto.so.4 (0x28183000) <br />
libcrypt.so.3 => /lib/libcrypt.so.3 (0x28276000) <br />
libc.so.6 => /lib/libc.so.6 (0x2828e000) <br />
libmd.so.3 => /lib/libmd.so.3 (0x28373000)</pre><br />
<br />
md5 them and compare to other jail hosts or jails running on host<br />
<br />
for libcrypto reinstall:<br />
<pre>/usr/src/crypto<br />
make depend && make all install</pre><br />
<br />
= FreeBSD VPS =<br />
<br />
== Starting jails: Quad/Safe Files ==<br />
<br />
FreeBSD customer systems do not start up automatically at boot time. When one of our freebsd machines boots up, it boots up, and does nothing else. To start jails, we put the commands to start each jail into a shell script(s) and run the script(s). Jail startup is something that needs to be actively monitored, which is why we don’t just run the script automatically. More on monitoring later.<br />
<br />
NOTE: >=7.x we have moved to 1 quad file: <tt>quad1</tt>. Startups are not done by running each quad, but rather [[#startalljails|startalljails]] which relies on the contents of <tt>quad1</tt>. The specifics of this are lower in this article. What follows here applies for pre 7.x systems.<br />
<br />
There are eight files in <tt>/usr/local/jail/rc.d</tt>:<br />
<br />
<pre>jail3# ls /usr/local/jail/rc.d/<br />
quad1 quad2 quad3 quad4 safe1 safe2 safe3 safe4<br />
jail3#</pre><br />
<br />
four quad files and four safe files.<br />
<br />
Each file contains an even number of system startup blocks (total number of jails divided by 4)<br />
<br />
The reason for this is, if we make one large script to startup all the systems at boot time, it will take too long - the first system in the script will start up right after system boot, which is great, but the last system may not start for another 20 minutes.<br />
<br />
Since there is no way to parralelize this during the startup procedure, we simply open four terminals (in screen window 9) and run each script, one in each terminal. This way they all run simultaneously, and the very last system in each startup script gets started in 1/4th the time it would if there was one large file<br />
<br />
The files are generally organized so that quad/safe 1&2 have only jails from disk 1, and quad/safe 3&4 have jails from disk 2. This helps ensure that only 2 fscks on any disk are going on at once. Further, they are balanced so that all quad/safe’s finish executing around the same time. We do this by making sure each quad/safe has a similar number of jails and represents a similar number of inodes (see js).<br />
<br />
The other, very important reason we do it this way, and this is the reason there are quad files and safe files, is that in the event of a system crash, every single vn-backed filesystem that was mounted at the time of system crash needs to be fsck'd. However, fsck'ing takes time, so if we shut the system down gracefully, we don't want to fsck.<br />
<br />
Therefore, we have two sets of scripts - the four quad scripts are identical to the four safe scripts except for the fact that the quad scripts contain fsck commands for each filesystem.<br />
<br />
So, if you shut a system down gracefully, start four terminals and run safe1 in window one, and safe2 in window 2, and so on.<br />
<br />
If you crash, start four terminals (or go to screen window 9) and run quad1 in window one, and quad2 in window 2, and so on.<br />
<br />
Here is a snip of (a 4.x version) quad2 from jail17:<br />
<br />
<pre>vnconfig /dev/vn16 /mnt/data2/69.55.228.7-col00820<br />
fsck -y /dev/vn16<br />
mount /dev/vn16c /mnt/data2/69.55.228.7-col00820-DIR<br />
chmod 0666 /mnt/data2/69.55.228.7-col00820-DIR/dev/null<br />
jail /mnt/data2/69.55.228.7-col00820-DIR mail1.phimail.com 69.55.228.7 /bin/sh /etc/rc<br />
<br />
# moved to data2 col00368<br />
#vnconfig /dev/vn28 /mnt/data2/69.55.236.132-col00368<br />
#fsck -y /dev/vn28<br />
#mount /dev/vn28c /mnt/data2/69.55.236.132-col00368-DIR<br />
#chmod 0666 /mnt/data2/69.55.236.132-col00368-DIR/dev/null<br />
#jail /mnt/data2/69.55.236.132-col00368-DIR limehouse.org 69.55.236.132 /bin/sh /etc/rc<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
vnconfig /dev/vn22 /mnt/data2/69.55.228.13-col01063<br />
fsck -y /dev/vn22<br />
mount /dev/vn22c /mnt/data2/69.55.228.13-col01063-DIR<br />
chmod 0666 /mnt/data2/69.55.228.13-col01063-DIR/dev/null<br />
jail /mnt/data2/69.55.228.13-col01063-DIR www.widestream.com.au 69.55.228.13 /bin/sh /etc/rc<br />
<br />
# cancelled col00106<br />
#vnconfig /dev/vn15 /mnt/data2/69.55.238.5-col00106<br />
#fsck -y /dev/vn15<br />
#mount /dev/vn15c /mnt/data2/69.55.238.5-col00106-DIR<br />
#chmod 0666 /mnt/data2/69.55.238.5-col00106-DIR/dev/null<br />
#jail /mnt/data2/69.55.238.5-col00106-DIR mail.azebu.net 69.55.238.5 /bin/sh /etc/rc</pre><br />
<br />
As you can see, two of the systems specified are commented out - presumably those customers cancelled, or were moved to new servers.<br />
<br />
As you can see, the vnconfig line is the simpler command line, not the longer one that was used when it was first configured. As you can see, all that is done is, vnconfig the filesystem, then fsck it, then mount it. The fourth command is the `jail` command used to start the system – but that will be covered later.<br />
<br />
Here is the safe2 file from jail17:<br />
<br />
<pre>vnconfig /dev/vn16 /mnt/data2/69.55.228.7-col00820<br />
mount /dev/vn16c /mnt/data2/69.55.228.7-col00820-DIR<br />
chmod 0666 /mnt/data2/69.55.228.7-col00820-DIR/dev/null<br />
jail /mnt/data2/69.55.228.7-col00820-DIR mail1.phimail.com 69.55.228.7 /bin/sh /etc/rc<br />
<br />
# moved to data2 col00368<br />
#vnconfig /dev/vn28 /mnt/data2/69.55.236.132-col00368<br />
#mount /dev/vn28c /mnt/data2/69.55.236.132-col00368-DIR<br />
#chmod 0666 /mnt/data2/69.55.236.132-col00368-DIR/dev/null<br />
#jail /mnt/data2/69.55.236.132-col00368-DIR limehouse.org 69.55.236.132 /bin/sh /etc/rc<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
vnconfig /dev/vn22 /mnt/data2/69.55.228.13-col01063<br />
mount /dev/vn22c /mnt/data2/69.55.228.13-col01063-DIR<br />
chmod 0666 /mnt/data2/69.55.228.13-col01063-DIR/dev/null<br />
jail /mnt/data2/69.55.228.13-col01063-DIR www.widestream.com.au 69.55.228.13 /bin/sh /etc/rc<br />
<br />
# cancelled col00106<br />
#vnconfig /dev/vn15 /mnt/data2/69.55.238.5-col00106<br />
#mount /dev/vn15c /mnt/data2/69.55.238.5-col00106-DIR<br />
#chmod 0666 /mnt/data2/69.55.238.5-col00106-DIR/dev/null<br />
#jail /mnt/data2/69.55.238.5-col00106-DIR mail.azebu.net 69.55.238.5 /bin/sh /etc/rc</pre><br />
<br />
As you can see, it is exactly the same, but it does not have the fsck lines.<br />
<br />
Take a look at the last entry - note that the file is named:<br />
<br />
/mnt/data2/69.55.238.5-col00106<br />
<br />
and the mount point is named:<br />
<br />
/mnt/data2/69.55.238.5-col00106-DIR<br />
<br />
This is the general format on all the FreeBSD systems. The file is always named:<br />
<br />
IP-custnumber<br />
<br />
and the directory is named:<br />
<br />
IP-custnumber-DIR<br />
<br />
If you run safe when you need a fsck, the mount will fail and jail will fail:<br />
<br />
# mount /dev/vn1c /mnt/data2/jails/65.248.2.131-ns1.kozubik.com-DIR<br />
mount: /dev/vn1c: Operation not permitted<br />
<br />
No reboot needed, just run the quad script<br />
<br />
Starting with 6.x jails, we added block delimiters to the quad/safe files, the block looks like:<br />
<br />
<pre>echo '## begin ##: nuie.solaris.mu'<br />
fsck -y /dev/concat/v30v31a<br />
mount /dev/concat/v30v31a /mnt/data1/69.55.228.218-col01441-DIR<br />
mount_devfs devfs /mnt/data1/69.55.228.218-col01441-DIR/dev<br />
devfs -m /mnt/data1/69.55.228.218-col01441-DIR/dev rule -s 3 applyset<br />
jail /mnt/data1/69.55.228.218-col01441-DIR nuie.solaris.mu 69.55.228.218 /bin/sh /etc/rc<br />
echo '## end ##: nuie.solaris.mu'</pre><br />
<br />
These are more than just informative when running quad/safe’s, the echo lines MUST be present for certain tools to work properly. So it’s important that any updates to the hostname also be updated on the 2 echo lines. For example, if you try to startjail a jail with a hostname which is on the jail line but not the echo lines, the command will return with host not found.<br />
<br />
=== FreeBSD 7.x+ notes ===<br />
<br />
Starting with the release of FreeBSD 7.x, we are doing jail startups in a slightly different way. First, thereis only 1 file: <tt>/usr/local/jail/rc.d/quad1</tt><br />
There are no other quads or corresponding safe files. The reason for this is twofold, 1. We can pass –C to fsck which will tell is to skip the fsck if the fs is clean (no more need for safe files), 2. We have a new startup script which can be launched multiple times, running in parallel to start jails, where quad1 is the master jail file. <br />
Quad1 could still be run as a shell script, but it would take a very long time for it to run completely so it’s not advisable; or you should break it down into smaller chunks (like quad1, quad2, quad3, etc)<br />
<br />
Here is a snip of (a 7.x version) quad1 from jail2:<br />
<br />
<pre>echo '## begin ##: projects.tw.com'<br />
mdconfig -a -t vnode -f /mnt/data1/69.55.230.46-col01213 -u 50<br />
fsck -Cy /dev/md50c<br />
mount /dev/md50c /mnt/data1/69.55.230.46-col01213-DIR<br />
mount -t devfs devfs /mnt/data1/69.55.230.46-col01213-DIR/dev<br />
devfs -m /mnt/data1/69.55.230.46-col01213-DIR/dev rule -s 3 applyset<br />
jail /mnt/data1/69.55.230.46-col01213-DIR projects.tw.com 69.55.230.46 /bin/sh /etc/rc<br />
echo '## end ##: projects.tw.com'</pre><br />
<br />
Cancelled jails are no longer commented out and stored in quad1, rather they’re moved to <tt>/usr/local/jail/rc.d/deprecated</tt> <br />
<br />
To start these jails, start the 4 ssh sessions as you would for a normal crash and then instead of running quad1-4, instead run startalljails in each window. IMPORTANT- before running startalljails you should make sure you ran preboot once as it will clear out all the lockfiles and enable startalljails to work properly.<br />
<br />
== Problems with the quad/safe files ==<br />
<br />
When you run the quad/safe files, there are two problems that can occur - either a particular system will hang during initialization, OR a system will spit out output to the screen, impeding your ability to do anything. Or both.<br />
<br />
First off, when you start a jail, you see output like this:<br />
<br />
<pre>Skipping disk checks ...<br />
adjkerntz[25285]: sysctl(put_wallclock): Operation not permitted<br />
Doing initial network setup:.<br />
ifconfig: ioctl (SIOCDIFADDR): permission denied<br />
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384<br />
Additional routing options: TCP keepalive=YESsysctl:<br />
net.inet.tcp.always_keepalive: Operation not permitted.<br />
Routing daemons:.<br />
Additional daemons: syslogd.<br />
Doing additional network setup:.<br />
Starting final network daemons:.<br />
ELF ldconfig path: /usr/lib /usr/lib/compat /usr/X11R6/lib /usr/local/lib<br />
a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout /usr/X11R6/lib/aout<br />
Starting standard daemons: inetd cron sshd sendmail sendmail-clientmqueue.<br />
Initial rc.i386 initialization:.<br />
Configuring syscons: blanktime.<br />
Additional ABI support:.<br />
Local package initialization:.<br />
Additional TCP options:.</pre><br />
<br />
Now, let's look at this line, near the end:<br />
<br />
Local package initialization:.<br />
<br />
This is where a list of daemons that are set to start at boot time willshow up. You might see something like:<br />
<br />
Local package initialization: mysqld apache sendmail sendmail-clientmqueue<br />
<br />
Or something like this:<br />
<br />
Local package initialization: postgres postfix apache<br />
<br />
The problem is that many systems (about 4-5 per machine) will hang on that line. Basically it will get to some of the way through the total daemons to be started:<br />
<br />
Local package initialization: mysqld apache<br />
<br />
and will just sit there. Forever.<br />
<br />
Fortunately, pressing ctrl-c will break out of it. Not only will it break out of it, but it will also continue on that same line and start the other daemons:<br />
<br />
Local package initialization: mysqld apache ^c sendmail-clientmqueue<br />
<br />
and then continue on to finish the startup, and then move to the next system to be started.<br />
<br />
So what does this mean? It means that if a machine crashes, and you start four screen-windows to run four quads or four safes, you need to periodically cycle between them and see if any systems are stuck at that point, causing their quad/safe file to hang. A good rule of thumb is, if you see a system at that point in the startup, give it another 100 seconds - if it is still at the exact same spot, hit ctrl-c. Its also a good idea to go back into the quad file (just before the first command in the jail startup block) and note that this jail tends to need a control-c or more time as follows:<br />
<pre>echo '### NOTE ### slow sendmail'<br />
echo '### NOTE ###: ^C @ Starting sendmail.'</pre><br />
<br />
'''NEVER''' hit ctrl-c repeatedly if you don't get an immediate response - that will cause the following jail’s startup commands to be aborted.<br />
<br />
A second problem that can occur is that a jail - maybe the first one in that particular quad/safe, maybe the last one, or maybe one in the middle, will start spitting out status or error messages from one of its init scripts. This is not a problem - basically, hit enter a few times and see if you get a prompt - if you do get a prompt, that means that the quad/safe script has already completed. Therefore it is safe to log out (and log out of the user that you su'd from) and then log back in (if necessary).<br />
<br />
The tricky thing is, if a system in the middle starts flooding with messages, and you hit enter a few times and don't get a prompt. Are you not getting a prompt because some subsequent system is hanging at the initialization, as we discussede above ? Or are you not getting a prompt because that quad file is currently running an fsck ? Usually you can tell by scrolling back in screen’s history to see what it was doing before you started getting the messages.<br />
<br />
If you don’t get clues from history, you have to use your judgement - instead of giving it 100 seconds to respond, perhaps give it 2-3 mins ... if you still get no response (no prompt) when you hit enter, hit ctrl-c. However, be aware that you might still be hitting ctrl-c in the middle of an fsck. This means you will get an error like "filesystem still marked dirty" and then the vnconfig for it will fail and so will the jail command, and the next system in the quad file will then start starting up.<br />
<br />
If this happens, just wait until the end of all the quad files have finished, and start that system manually.<br />
<br />
If things really get weird, like a screen flooded with errors, and you can't get a prompt, and ctrl-c does nothing, then you need to just eventually (give it ten mins or so) just kill that window with ctrl-p, then k, and then log in again and manually check which systems are now running and which aren't, and manually start up any that are not.<br />
<br />
Don't EVER risk running a particular quad/safe file a second time.<br />
If the quad/safe script gets executed twice, reboot the machine immediately.<br />
<br />
So, for all the above reasons, anytime a machine crashes and you run all the quads or all the safes, '''always''' check every jail afterwards to make sure it is running - even if you have no hangs or complications at all.<br />
Run this command:<br />
<br />
<tt>[[#jailpsall|jailpsall]]</tt> <br />
<br />
Note: [[#postboot|postboot]] also populates ipfw counts, so it '''should not be run multiple times''', use <tt>jailpsall</tt> for subsequent extensive ps’ing<br />
<br />
And make sure they all show as running. If one does not show as running, check its /etc/rc.conf file first to see if maybe it is using a different hostname first before starting it manually.<br />
<br />
One thing we have implemented to alleviate these startup hangs and noisy jails, is to put jail start blocks that are slow or hangy at the bottom of the safe/quad file. Further, for each bad jail we note in each quad/safe just before the start block something like:<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
<br />
That way we’ll be prepared to ^C when we see that message appear during the quad/safe startup process. If you observe a new, undocumented hang, '''after''' the quad/safe has finished, place a line similar to the above in the quad file, move the jail start block to the end of the file, then run [[#buildsafe|buildsafe]]<br />
<br />
<br />
== Recovering from a crash (FreeBSD) ==<br />
<br />
=== Diagnose whether you have a crash ===<br />
The most important thing is to get the machine and all jails back up as soon as possible. Note the time, you’ll need to create a crash log entry (Mgmt. -> Reference -> CrashLog). The first thing to do is head over to the [[Screen#Screen_Organization|serial console screen]] and see if there’s any kernel error messages output. Try to copy any messages (or just a sample of repeating messages) you see into the notes section of the crash log. If there are no messages, the machine may just be really busy- wait a bit (5-10min) to see if it comes back. If it's still pinging, odds are its very busy. Note, if you see messages about swap space exhausted, the server is obviously out of memory, however it may recover briefly enough for you to get a jtop in to see who's lauched a ton of procs (most likely) and then issue a quick jailkill to get it back under control.<br />
<br />
If it doesn't come back, or the messages indicate a fatal error, you will need to proceed with a power cycle (ctrl+alt+del will not work).<br />
<br />
=== Power cycle the server ===<br />
If this machine is not a Dell 2950 with a [[DRAC/RMM#DRAC|DRAC card]] (i.e. if you can’t ssh into the DRAC card (as root, using the standard root pass) and issue <br />
racadm serveraction hardreset<br />
then you will need someone at the data center power the macine off, wait 30 sec, then turn it back on. Make sure to re-attach via console:<br />
tip jailX<br />
immediately after power down. <br />
<br />
=== (Re)attach to the console ===<br />
Stay on the console the entire time during boot. As the BIOS posts- look out for the RAID card output- does everything look healthy? The output may be scrambled, look for "DEGRADED" or "FAILED". Once the OS starts booting you will be disconnected (dropped back to the shell on the console server) a couple times during the boot up. The reason you want to quickly re-attach is two-fold: 1. If you don’t reattach quickly then you won’t get any console output, 2. you want to be attached before the server ''potentially'' starts (an extensive) fsck. If you attach after the fsck begins, you’ll have seen no indication it started an fsck and the server will appear frozen during startup- no output, no response. <br />
<br />
IMPORTANT NOTE: on some older FreeBSD systems, there will be no output to the video (KVM) console as it boots up. The console output is redirected to the serial port ... so if a jail crashes, and you attach a kvm, the output during the bootup procedure will not be shown on the screen. However, when the bootup is done, you will get a login prompt on the screen and will be able to log in as normal. <tt>/boot/loader.conf</tt> is where serial console redirect output lives, so comment that if you want to catch output on kvm.<br />
On newer systems it sends most output to both locations. <br />
<br />
=== Assess the heath of the server ===<br />
Once the server boots up fully, you should be able to ssh in. Look around- make sure all the mounts are there and reporting the correct size/usage (i.e. /mnt/data1 /mnt/data2 /mnt/data3 - look in /etc/fstab to determine which mount points should be there), check to see if RAID mirrors are healthy. See [[RAID_Cards#Common_CLI_commands_.28megacli.29|megacli]], [[#aaccheck|aaccheck]]<br />
<br />
Before you start the jails, you need to run [[#preboot|preboot]]. This will do some assurance checks to make sure things are prepped to start the jails. Any issues that come out of preboot need to be addressed before starting jails.<br />
<br />
=== Start jails ===<br />
[[#Starting_jails:_Quad.2FSafe_Files|More on starting jails]]<br />
Customer jails (the VPSs) do not start up automatically at boot time. When a FreeBSD machines boots up, it boots up, and does nothing else. To start jails, we put the commands to start each jail into a shell script(s) and run the script(s). Jail startup is something that needs to be actively monitored, which is why we don’t just run the script automatically. <br />
<br />
In order to start jails, we run the quad files: quad1 quad2 quad3 and quad4 (on new systems there is only quad1). If the machine was cleanly rebooted- which wouldn't be the case if this was a crash, you may run the safe files (safe1 safe2 safe3 safe4) in lieu of quads. <br />
<br />
Open up 4 logins to the server (use the windows in [[Screen#Screen_Organization|a9]])<br />
In each of the 4 windows you will:<br />
<br />
If there is a [[#startalljails|startalljails]] script (and only quad1), run that command in each of the 4 windows. It will parse through the quad1 file and start each jail. Follow the instructions [[#Problems_with_the_quad.2Fsafe_files|here]] for monitoring startup. Note that you can be a little more lenient with jails that take awhile to start- startalljails will work around the slow jails and start the rest. As long as there aren't 4 jails which are "hung" during startup, the rest will get started eventually.<br />
-or-<br />
If there is no startalljails script, there will be multiple quad files. In each of the 4 windows, start each of the quads. i.e. start quad1 in window1, quad2 in window2 and so on. DO NOT start any quad twice. It will crash the server. If you accidentally do this, just jailkill all the jails which are in the quad and run the quad again. Follow the instructions here for monitoring quad startup.<br />
<br />
Note the time the last jail boots- this is what you will enter in the crash log.<br />
<br />
Save the crash log.<br />
<br />
=== Check to make sure all jails have started ===<br />
There's a simple script which will make sure all jails have started, and enter the ipfw counter rules: [[#postboot|postboot]] <br />
Run postboot, which will do a jailps on each jail it finds (excluding commented out jails) in the quad file(s). We're looking for 2 things:<br />
# systems spawning out of control or too many procs<br />
# jails which haven't started<br />
On 7.x and newer systems it will print out the problems (which jails haven't started) at the conclusion of postboot. <br />
On older systems you will need to watch closely to see if/when there's a problem, namely:<br />
<br />
[hostname] doesnt exist on this server<br />
<br />
When you get this message, it means one of 2 things:<br />
1. the jail really didn't start:<br />
When a jail doesn't start it usually boils down to a problem in the quad file. Perhaps the path name is wrong (data1 vs data2) or the name of the vn/mdfile is wrong. Once this is corrected, you will need to run the commands from the quad file manually, or you may use <tt>startjail <hostname></tt><br />
<br />
2. the customer has changed their hostname (and not told us) so their jail ''is'' running, just under a different hostname:<br />
On systems with jls, this is easy to rectify. First, get the customer info: <tt>g <hostname></tt><br />
Then look for the customer in jls: <tt>jls | grep <col0XXXX></tt><br />
From there you will see their new hostname- you should update that hostname in the quad file: don't forget to edit it on the <tt>## begin ##</tt> and <tt>## end ##</tt> lines, and in mgmt. <br />
On older systems without jls, this will be harder, you will need to look further to see their hostname- perhaps its in their /etc/rc.conf<br />
<br />
<br />
Once all jails are started, do some spot checks- try to ssh or browse to some customers, just to make sure things are really ok.<br />
<br />
== Adding disk to a 7.x/8.x jail ==<br />
or<br />
== Moving customer to a different drive (md) ==<br />
<br />
NOTE: this doesn’t apply to mx2 which uses gvinum. Use same procedure as 6.x<br />
NOTE: if you unmount before mdconfig, re-mdconfig (attach) then unmount then mdconfig -u again <br />
<br />
<br />
(parts to change/customize are <tt><span style="color:red">red</span></tt>)<br />
<br />
<br />
If someone wants more disk space, there’s a paste for it, send it to them (explains about downtime, etc).<br />
<br />
1. Figure out the space avail from <tt>js</tt>. Ideally, you want to put the customers new space on a different partition (and create the new md on the new partition). <br />
<br />
<br />
2. make a mental note of how much space they're currently using<br />
<br />
<br />
3. <tt>jailkill <hostname></tt> <br />
<br />
<br />
4. Umount it (including their devfs) but leave the md config’d (so if you use stopjail, you will have to re-mdconfig it)<br />
<br />
<br />
5. <tt>g <customerID></tt> to get the info (IP/cust#) needed to feed the new mdfile and mount name, and to see the current md device. <br />
<br />
<br />
6a. When there's enough room to place new system on an alternate, or the same drive:<br />
USE CAUTION not to overwrite (touch, mdconfig) existing md!!<br><br />
<tt>touch /mnt/data<span style="color:red">3/69.55.234.66-col01334</span><br><br />
mdconfig -a -t vnode -s 10g -f /mnt/data3/69.55.234.66-col01334 -u 97<br><br />
newfs /dev/md97</tt><br />
<br />
<br />
Optional- if new space is on a different drive, move the mount point directory AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
<tt>mount /dev/md<span style="color:red">97</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<br />
confirm you are mounted to /dev/md<span style="color:red">97</span> and space is correct:<br><br />
<tt>df .</tt> <br />
<br />
<br />
do the dump and pipe directly to restore:<br><br />
<tt>dump -0a -f - /dev/md<span style="color:red">1</span> | restore -r -f - <br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use <tt>df</tt> to confirm the restored data size matches the original usage figure<br />
<br />
<br />
md-unconfig old system:<br><br />
<tt>mdconfig -d -u <span style="color:red">1</span></tt><br />
<br />
<br />
archive old mdfile. <br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.237.26-col00241</span> /mnt/data<span style="color:red">1/old-col00241-mdfile-noarchive-20091211</span></tt><br />
<br />
<br />
edit quad (vq1) to point to new (/mnt/data<span style="color:red">3</span>) location AND new md number (md<span style="color:red">97</span>)<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
6b. When there's not enough room on an alternate partition, or on the same drive...but there is enough room if you were to remove the existing customer's space:<br />
<br />
<br />
mount backup nfs mounts:<br><br />
<tt>mbm</tt><br> <br />
(run <tt>df</tt> to confirm backup mounts are mounted)<br />
<br />
<br />
dump the customer to backup2 or backup1<br><br />
<tt>dump -0a -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> /dev/md<span style="color:red">1</span></tt><br><br />
(when complete WITHOUT errors, <tt>du</tt> the dump file to confirm it matches size, roughly, with usage)<br />
<br />
<br />
unconfigure and remove old mdfile<br><br />
<tt>mdconfig -d -u <span style="color:red">1</span><br><br />
rm /mnt/data<span style="color:red">1/69.55.237.26-col00241</span></tt><br><br />
(there should now be enough space to recreate your bigger system. If not, run sync a couple times)<br />
<br />
<br />
create the new system (ok to reuse old mdfile and md#):<br><br />
<tt>touch /mnt/data<span style="color:red">1/69.55.234.66-col01334</span><br><br />
mdconfig -a -t vnode -s <span style="color:red">10</span>g -f /mnt/data<span style="color:red">1/69.55.234.66-col01334</span> -u <span style="color:red">1</span><br><br />
newfs /dev/md<span style="color:red">1</span><br><br />
mount /dev/md<span style="color:red">1</span> /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> <br><br />
cd /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<br />
confirm you are mounted to /dev/md<span style="color:red">1</span> and space is correct:<br><br />
<tt>df .</tt> <br />
<br />
<br />
do the restore from the dumpfile on the backup server:<br><br />
<tt>restore -r -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> .<br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
umount nfs:<br><br />
<tt>mbu</tt><br />
<br />
<br />
If md# changed (or mount point), edit quad (<tt>vq1</tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new md number (md<span style="color:red">1</span>)<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
7. update disk (and dir if applicable) in mgmt screen<br />
<br />
<br />
8. update backup list AND move backups, if applicatble<br />
<br />
Ex: <tt>mvbackups <span style="color:red">69.55.237.26-col00241</span> jail<span style="color:red">9</span> data<span style="color:red">3</span></tt><br />
<br />
<br />
9. Optional: archive old mdfile<br />
<br />
<tt>mbm<br><br />
gzip -c old-col01588-mdfile-noarchive-20120329 > /deprecated/old-col01588-mdfile-noarchive-20120329.gz<br><br />
mbu<br><br />
rm old-col01588-mdfile-noarchive-20120329</tt><br />
<br />
== Adding disk to a 6.x jail (gvinum/gconcat) ==<br />
or<br />
== Moving customer to a different drive (gvinum/gconcat) ==<br />
<br />
(parts to change are <span style="color:red">highlighted</span>)<br />
<br />
<br />
If someone wants more disk space, there’s a paste for it, send it to them (explains about downtime, etc).<br />
<br />
<br />
1. Figure out the space avail from [[#js|js]]. Ideally, you want to put the customers new space on a different partition (and create the new md on the new partition). <br />
<br />
<br />
2. make a mental note of how much space they're currently using<br />
<br />
<br />
3. <tt>[[#stopjail|stopjail]] <hostname></tt> <br />
<br />
<br />
4. <tt>[[#g|g]] <customerID></tt> to get the info (IP/cust#) needed to feed the new mount name and existing volume/device. <br />
<br />
<br />
5a. When there's enough room to place new system on an alternate, or the same drive (using only UNUSED - including if it's in use by the system in question - gvinum volumes):<br />
<br />
<br />
Configure the new device:<br><br />
A. for a 2G system (single gvinum volume):<br><br />
<tt>bsdlabel -r -w /dev/gvinum/v<span style="color:red">123</span><br><br />
newfs /dev/gvinum/v<span style="color:red">123</span>a</tt><br><br />
<br />
-or- <br />
<br />
<br />
B. for a >2G system (create a gconcat volume):<br><br />
try to grab a contiguous block of gvinum volumes. gconcat volumes MAY NOT span drives (i.e. you cannot use a gvinum volume from data3 and a volume from data2 in the same gconcat volume). <br />
<br />
<tt>gconcat label <span style="color:red">v82-v84 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84</span><br><br />
bsdlabel -r -w /dev/concat/<span style="color:red">v82-v84</span><br><br />
newfs /dev/concat/<span style="color:red">v82-v84</span>a</tt><br />
<br />
<br />
Other valid gconcat examples:<br><br />
<tt>gconcat label v82-v84v109v112 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84 /dev/gvinum/v109 /dev/gvinum/v112<br><br />
gconcat label v82v83 /dev/gvinum/v82 /dev/gvinum/v83</tt> <br><br />
Note, long names will truncate: v144v145v148-v115 will truncate to v144v145v148-v1 (so you will refer to it as v144v145v148-v1 thereafter)<br />
<br />
<br />
Optional- if new volume is on a different drive, move the mount point directory (get the drive from js output) AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
<br />
confirm you are mounted to the device (<tt>/dev/gvinum/v<span style="color:red">123</span>a</tt> OR <tt>/dev/concat/<span style="color:red">v82-v84</span></tt>) and space is correct:<br><br />
A. <tt>mount /dev/gvinum/v<span style="color:red">123</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br><br />
-or-<br><br />
B. <tt>mount /dev/concat/<span style="color:red">v82-v84</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<tt>cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
df .</tt> <br />
<br />
<br />
do the dump and pipe directly to restore:<br><br />
<tt>dump -0a -f - /dev/gvinum/v<span style="color:red">1</span> | restore -r -f - <br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
edit quad (<tt>vq<span style="color:red">1</span></tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new volume (<tt>/dev/gvinum/v<span style="color:red">123</span>a</tt> or <tt>/dev/concat/<span style="color:red">v82-v84</span>a</tt>) , run <tt>buildsafe</tt><br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
5b. When there's not enough room on an alternate partition, or on the same drive...but there is enough room if you were to remove the existing customer's space (i.e. if you want/need to reuse the existing gvinum volumes and add on more):<br />
<br />
<br />
mount backup nfs mounts:<br><br />
<tt>mbm</tt> <br><br />
(run df to confirm backup mounts are mounted)<br />
<br />
<br />
dump the customer to backup2 or backup1<br><br />
<tt>dump -0a -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> /dev/<span style="color:red">gconcat/v106-v107</span></tt><br><br />
(when complete WITHOUT errors, du the dump file to confirm it matches size, roughly, with usage)<br />
<br />
<br />
unconfigure the old gconcat volume<br><br />
list member gvinum volumes:<br />
<br />
<tt>gconcat list <span style="color:red">v106v107</span></tt><br />
<br />
<br />
Output will resemble:<br><br />
<pre>Geom name: v106v107<br />
State: UP<br />
Status: Total=2, Online=2<br />
Type: AUTOMATIC<br />
ID: 3530663882<br />
Providers:<br />
1. Name: concat/v106v107<br />
Mediasize: 4294966272 (4.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e2<br />
Consumers:<br />
1. Name: gvinum/sd/v106.p0.s0<br />
Mediasize: 2147483648 (2.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e3<br />
Start: 0<br />
End: 2147483136<br />
2. Name: gvinum/sd/v107.p0.s0<br />
Mediasize: 2147483648 (2.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e3<br />
Start: 2147483136<br />
End: 4294966272</pre><br />
<br />
stop volume and clear members<br><br />
<tt>gconcat stop <span style="color:red">v106v107</span><br><br />
gconcat clear <span style="color:red">gvinum/sd/v106.p0.s0 gvinum/sd/v107.p0.s0</span></tt><br />
<br />
<br />
create new device- and its ok to reuse old/former members<br><br />
try to grab a contiguous block of gvinum volumes. gconcat volumes MAY NOT span drives (i.e. you cannot use a gvinum volume from data3 and a volume from data2 in the same gconcat volume). <br><br />
<tt>gconcat label <span style="color:red">v82-v84v106v107 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84 /dev/gvinum/v106 /dev/gvinum/v107</span> <br><br />
bsdlabel -r -w /dev/concat/<span style="color:red">v82-v84v106v107</span><br><br />
newfs /dev/concat/<span style="color:red">v82-v84v106v107</span>a</tt><br />
<br />
<br />
Optional- if new volume is on a different drive, move the mount point directory (get the drive from js output) AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
confirm you are mounted to the device (/dev/concat/<span style="color:red">v82-v84v106v107</span>) and space is correct:<br><br />
<tt>mount /dev/concat/<span style="color:red">v82-v84v106v107</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<tt>cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
df .</tt> <br />
<br />
do the restore from the dumpfile on the backup server:<br><br />
<tt>restore -r -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> .<br><br />
rm restoresymtable</tt><br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
edit quad (<tt>vq<span style="color:red">1</span></tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new volume (<tt>/dev/concat/<span style="color:red">v82-v84v106v107</span>a</tt>), run buildsafe<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
TODO: clean up/clear old gvin/gconcat vol<br />
<br />
<br />
6. update disk (and dir if applicable) in mgmt screen<br />
<br />
<br />
7. update backup list AND move backups, if applicatble<br />
<br />
<br />
Ex: <tt>mvbackups <span style="color:red">69.55.237.26-col00241</span> jail<span style="color:red">9</span> data<span style="color:red">3</span></tt><br />
<br />
<br />
<br />
DEPRECATED - steps to tack on a new gvin to existing gconcat- leads to corrupted fs<br />
bsdlabel -e /dev/concat/v82-v84<br />
<br />
To figure out new size of the c partition, multiply 4194304 by the # of 2G gvinum volumes and subtract the # of 2G volumes:<br />
10G: 4194304 * 5 – 5 = 20971515<br />
8G: 4194304 * 4 – 4 = 16777212<br />
6G: 4194304 * 3 – 3 = 12582909<br />
4G: 4194304 * 2 – 2 = 8388606<br />
<br />
To figure out the new size of the a partition, subtract 16 from the c partition:<br />
10G: 20971515 – 16 = 20971499<br />
8G: 16777212 – 16 = 16777196<br />
6G: 12582909 – 16 = 12582893<br />
4G: 8388606 – 16 = 8388590<br />
<br />
Orig:<br />
8 partitions:<br />
# size offset fstype [fsize bsize bps/cpg]<br />
a: 8388590 16 4.2BSD 2048 16384 28552<br />
c: 8388606 0 unused 0 0 # "raw" part, don't edit<br />
<br />
New:<br />
8 partitions:<br />
# size offset fstype [fsize bsize bps/cpg]<br />
a: 12582893 16 4.2BSD 2048 16384 28552<br />
c: 12582909 0 unused 0 0 # "raw" part, don't edit<br />
<br />
sync; sync<br />
<br />
growfs /dev/concat/v82-v84a<br />
<br />
fsck –fy /dev/concat/v82-v84a<br />
<br />
sync<br />
<br />
fsck –fy /dev/concat/v82-v84a<br />
<br />
(keep running fsck’s till NO errors)<br />
<br />
== Problems un-mounting - and with mount_null’s ==<br />
<br />
If you cannot unmount a filesystem, beacuse it says the filesystem is busy, it is because of three things:<br />
<br />
a) the jail is still running<br />
<br />
b) you are actually in that directory, even though the jail is stopped<br />
<br />
c) there are still dev, null_mount or linprocfs mount points mounted inside that directory.<br />
<br />
d) when trying to umount null_mounts that are really long and you get an error like “No such file or directory”, it’s an OS bug where the dir is truncated. No known fix<br />
<br />
e) there are still files open somewhere inside the dir. Use <tt>fstat | grep <cid></tt> to find the process that has files open<br />
<br />
f) Starting with 6.x, the jail mechanism does a poor job of keeping track of processes running in a jail and if it thinks there are still procs running, it will refuse to umount the disk. If this is happening you should see a low number in the #REF column when you run jls. In this case you ''can'' safely <tt>umount –f</tt> the mount. <br />
<br />
Please note -if you forcibly unmount a (4.x) filesystem that has null_mounts<br />
still mounted in it, the system '''will crash''' within 10-15 mins.<br />
<br />
== Misc jail Items ==<br />
<br />
We are overselling hard drive space on jail2, jail8, jail9, a couple jails on jail17, jail4, jail12 and jail18.<br />
Even though the vn file shows 4G size, it doesn’t actually occupy that amount of space on the disk. So be careful not to fill up drives where we’re overselling – use oversellcheck to confirm you’re not oversold by more than 10G.<br />
There are other truncated jails, they are generally noted in a the file on the root system: /root/truncated<br />
<br />
The act of moving a truncated vn to another system un-does the truncating- the truncated vn is filled with 0’s and it occupies physical disk space for which it’s configured. So, you should use dumpremote to preserve the truncation.<br />
<br />
= FreeBSD VPS Management Tools =<br />
<br />
These files are located in /usr/local/jail/rc.d and /usr/local/jail/bin<br />
<br />
== jailmake ==<br />
<br />
Applies to 7.x+ <br />
On older systems syntax differs, run jailmake once to see.<br />
<br />
Note: this procedure differs on mx2 which is 7.x but still uses gvinum<br />
<br />
# run js to figure out which md’s are in use, which disk has enough space, IP to put it on<br />
# use col00xxx for both hostnames if they don’t give you a hostname<br />
# copy over dir, ip and password to pending customer screen<br />
<br />
<tt>Usage: jailmake IP[,IP] CID disk[1|2|3] md# hostname shorthost ipfw# email [size in GB]</tt><br />
<br />
Ex: <br />
<br />
Jail2# jailmake 69.55.234.66 col01334 3 97 vps.bsd.it vps 1334 fb@bsd.it<br />
<br />
== jailps ==<br />
jailps [hostname]<br />
DEPRECATED FOR jps: displays processes belonging to/running inside a jail. The command<br />
takes one (optional) argument – the hostname of the jail you wish to query. If you don’t <br />
supply an argument, all processes on the machine are listed and grouped by jail. <br />
<br />
== jps ==<br />
jps [hostname]<br />
displays processes belonging to/running inside a jail. The command<br />
takes one (optional) argument – the hostname or ID of the jail you wish to query. <br />
<br />
== jailkill ==<br />
jailkill <hostname><br />
stops all process running in a jail.<br />
<br />
You can also run:<br />
jailkill <JID><br />
<br />
=== problems ===<br />
Occasionally you will hit an issue where jail will not kill off:<br />
<br />
<pre>jail9# jailkill www.domain.com<br />
www.domain.com .. killed: none<br />
jail9#</pre><br />
Because no processes are running under that hostname. You cannot use jailps.pl either:<br />
<br />
<pre>jail9# jailps www.domain.com<br />
www.domain.com doesn’t exist on this server<br />
jail9#</pre><br />
<br />
The reasons for this are usually:<br />
* the jail is no longer running<br />
<br />
* the jail's hostname has changed<br />
In this case, <br />
<br />
>=6.x: run a <tt>jls|grep <jail's IP></tt> to find the correct hostname, then update the quad file, then kill the jail.<br />
<br />
<6.x: the first step is to cat their /etc/rc.conf file to see if you can tell what they set the new hostname to. This very often works. For example:<br />
<br />
cat /mnt/data2/198.78.65.136-col00261-DIR/etc/rc.conf<br />
<br />
But maybe they set the hostname with the hostname command, and the original hostname is still in /etc/rc.conf.<br />
<br />
The welcome email clearly states that they should tell us if they change their hostname, so there is no problem in just emailing them and asking them what they set the new hostname to.<br />
<br />
Once you know the new hostname OR if a customer simply emails to inform you that they have set the hostname to something different, you need to edit the quad and safe files that their system is in to input the new hostname.<br />
<br />
However, if push comes to shove and you cannot find out the hostname from them or from their system, then you need to start doing some detective work.<br />
<br />
The easiest thing to do is run jailps looking for a hostname similar to their original hostname. Or you could get into the /bin/sh shell by running:<br />
<br />
/bin/sh<br />
<br />
and then looking at every hostname of every process:<br />
<br />
for f in `ls /proc` ; do cat /proc/$f/status ; done<br />
<br />
and scanning for a hostname that is either similar to their original hostname, or that you don't see in any of the quad safe files.<br />
<br />
This is very brute force though, and it is possible that catting every file in /proc is dangerous - I don't recommend it. A better thing would be to identify any processes that you know belong to this system – perhaps the reason you are trying to find this system is because they are running something bad - and just catting the status from only that PID.<br />
<br />
Somewhere there’s a jail where there may be 2 systems named www. Look at /etc/rc.conf and make sure they’re both really www. If they are, jailkill www, jailps www to make sure not running. Then immediately restart the other one, as the fqdn (as found from a rev nslookup)<br />
<br />
* on >=6.x the hostname may not yet be hashed:<br />
<pre>jail9 /# jls<br />
JID Hostname Path IP Address(es)<br />
1 bitnet.dgate.org /mnt/data1/69.55.232.50-col02094-DIR 69.55.232.50<br />
2 ns3.hctc.net /mnt/data1/69.55.234.52-col01925-DIR 69.55.234.52<br />
3 bsd1 /mnt/data1/69.55.232.44-col00155-DIR 69.55.232.44<br />
4 let2.bbag.org /mnt/data1/69.55.230.92-col00202-DIR 69.55.230.92<br />
5 post.org /mnt/data2/69.55.232.51-col02095-DIR 69.55.232.51 ...<br />
6 ns2 /mnt/data1/69.55.232.47-col01506-DIR 69.55.232.47 ...<br />
7 arlen.server.net /mnt/data1/69.55.232.52-col01171-DIR 69.55.232.52<br />
8 deskfood.com /mnt/data1/69.55.232.71-col00419-DIR 69.55.232.71<br />
9 mirage.confluentforms.com /mnt/data1/69.55.232.54-col02105-DIR 69.55.232.54 ...<br />
10 beachmember.com /mnt/data1/69.55.232.59-col02107-DIR 69.55.232.59<br />
11 www.agottem.com /mnt/data1/69.55.232.60-col02109-DIR 69.55.232.60<br />
12 sdhobbit.myglance.org /mnt/data1/69.55.236.82-col01708-DIR 69.55.236.82<br />
13 ns1.jnielsen.net /mnt/data1/69.55.234.48-col00204-DIR 69.55.234.48 ...<br />
14 ymt.rollingegg.net /mnt/data2/69.55.236.71-col01678-DIR 69.55.236.71<br />
15 verse.unixlore.net /mnt/data1/69.55.232.58-col02131-DIR 69.55.232.58<br />
16 smcc-mail.org /mnt/data2/69.55.232.68-col02144-DIR 69.55.232.68<br />
17 kasoutsuki.w4jdh.net /mnt/data2/69.55.232.46-col02147-DIR 69.55.232.46<br />
18 dili.thium.net /mnt/data2/69.55.232.80-col01901-DIR 69.55.232.80<br />
20 www.tekmarsis.com /mnt/data2/69.55.232.66-col02155-DIR 69.55.232.66<br />
21 vps.yoxel.net /mnt/data2/69.55.236.67-col01673-DIR 69.55.236.67<br />
22 smitty.twitalertz.com /mnt/data2/69.55.232.84-col02153-DIR 69.55.232.84<br />
23 deliver4.klatha.com /mnt/data2/69.55.232.67-col02160-DIR 69.55.232.67<br />
24 nideffer.com /mnt/data2/69.55.232.65-col00412-DIR 69.55.232.65<br />
25 usa.hanyuan.com /mnt/data2/69.55.232.57-col02163-DIR 69.55.232.57<br />
26 daifuku.ppbh.com /mnt/data2/69.55.236.91-col01720-DIR 69.55.236.91<br />
27 collins.greencape.net /mnt/data2/69.55.232.83-col01294-DIR 69.55.232.83<br />
28 ragebox.com /mnt/data2/69.55.230.104-col01278-DIR 69.55.230.104<br />
29 outside.mt.net /mnt/data2/69.55.232.72-col02166-DIR 69.55.232.72<br />
30 vps.payneful.ca /mnt/data2/69.55.234.98-col01999-DIR 69.55.234.98<br />
31 higgins /mnt/data2/69.55.232.87-col02165-DIR 69.55.232.87 ...<br />
32 ozymandius /mnt/data2/69.55.228.96-col01233-DIR 69.55.228.96<br />
33 trusted.realtors.org /mnt/data2/69.55.238.72-col02170-DIR 69.55.238.72<br />
34 jc1.flanderous.com /mnt/data2/69.55.239.22-col01504-DIR 69.55.239.22<br />
36 guppylog.com /mnt/data2/69.55.238.73-col00036-DIR 69.55.238.73<br />
40 haliohost.com /mnt/data2/69.55.234.41-col01916-DIR 69.55.234.41 ...<br />
41 satyr.jorge.cc /mnt/data1/69.55.232.70-col01963-DIR 69.55.232.70<br />
jail9 /# jailkill satyr.jorge.cc<br />
ERROR: jail_: jail "satyr,jorge,cc" not found<br />
</pre><br />
<br />
Note how it's saying <tt>satyr,jorge,cc</tt> is not found, and not <tt>satyr.jorge.cc</tt>. <br />
<br />
The jail subsystem tracks things using comma-delimited hostnames. That is created every few hours:<br />
<br />
jail9 /# crontab -l<br />
0 0,6,12,18 * * * /usr/local/jail/bin/sync_jail_names<br />
<br />
So if we run this manually:<br />
jail9 /# /usr/local/jail/bin/sync_jail_names<br />
<br />
Then kill the jail:<br />
jail9 /# jailkill satyr.jorge.cc<br />
successfully killed: satyr,jorge,cc</pre><br />
<br />
It worked.<br />
<br />
<br />
If you ever see this when trying to kill a jail:<br />
<br />
<pre># jailkill e-scribe.com<br />
killing JID: 6 hostname: e-scribe.com<br />
3 procs running<br />
3 procs running<br />
3 procs running<br />
3 procs running<br />
...</pre><br />
<br />
<tt>[[#jailkill|jailkill]]</tt> probably got lost trying to kill off the jail. Just ctrl-c the jailkill process, then run a jailps on the hostname, and <tt>kill -9</tt> any process which is still running. Keep running jailps and <tt>kill -9</tt> till all processes are gone.<br />
<br />
== jailpsall ==<br />
jailpsall<br />
will run a jailps on all jails configured in the quad files (this is different from<br />
jailps with no arguments as it won’t help you find a “hidden” system)<br />
<br />
== jailpsw ==<br />
jailpsw<br />
will run a jailps with an extra -w to provide wider output<br />
<br />
== jt (>=7.x) ==<br />
jt<br />
displays the top 20 processes on the server (the top 20 processes from top) and <br />
which jail owns them. This is very helpful for determining who is doing what when<br />
the server is very busy.<br />
<br />
== jtop (>=7.x) ==<br />
jtop<br />
a wrapper for top displaying processes on the server and which jail owns them. Constantly updates, like top. <br />
<br />
== jtop (<7.x) ==<br />
jtop<br />
displays the top 20 processes on the server (the top 20 processes from top) and <br />
which jail owns them. This is very helpful for determining who is doing what when<br />
the server is very busy.<br />
<br />
== stopjail ==<br />
stopjail <hostname> [1]<br />
this will jailkill, umount and vnconfig –u a jail. If passed an optional 2nd<br />
argument, it will not exit before umounting and un-vnconfig’ing in the event<br />
jailkill returns no processes killed. This is useful if you just want to umount<br />
and vnconfig –u a jail you’ve already killed. It is intelligent in that it won’t <br />
try to umount or vnconfig –u if it’s not necessary.<br />
<br />
== startjail ==<br />
startjail <hostname><br />
this will start vnconfig, mount (including linprocfs and null-mounts), and start a jail.<br />
Essentially, it reads the jail’s relevant block from the right quad file and executes it.<br />
It is intelligent in that it won’t try to mount or vnconfig if it’s not necessary.<br />
<br />
== jpid ==<br />
jpid <pid><br />
displays information about a process – including which jail owns it.<br />
It’s the equivalent of running cat /proc/<pid>/status<br />
<br />
== canceljail ==<br />
canceljail <hostname> [1]<br />
this will stop a jail (the equivalent of stopjail), check for backups (offer to remove them <br />
from the backup server and the backup.config), rename the vnfile, remove the dir, and <br />
edit quad/safe. If passed an optional 2nd argument, it will not exit upon failing to kill<br />
and processes owned by the jail. This is useful if you just want to cancel a jail which <br />
is already stopped.<br />
<br />
== jls ==<br />
jls [-v]<br />
Lists all jails running:<br />
<pre>JID #REF IP Address Hostname Path<br />
101 135 69.55.224.148 mail.pc9.org /mnt/data2/69.55.224.148-col01034-DIR</pre><br />
<br />
#REF is the number of references or procs(?) running<br />
<br />
Running with -v will give you all IPs assigned to each jail (7.2 up)<br />
<pre>JID #REF Hostname Path IP Address(es)<br />
101 139 mail.pc9.org /mnt/data2/69.55.224.148-col01034-DIR 69.55.224.14869.55.234.85</pre><br />
<br />
== startalljails ==<br />
startalljails<br />
7.2+ only. This will parse through quad1 and start all jails. It utilizes lockfiles so it won’t try to start a jail more than once- therefore multiple instances can be running in parallel without fear of starting a jail twice. If a jail startup gets stuck, you can ^C without fear of killing the script. IMPORTANT- before running startalljails you should make sure you ran preboot once as it will clear out all the lockfiles and enable startalljails to work properly.<br />
<br />
== aaccheck.sh ==<br />
aaccheck.sh<br />
displayes the output of container list and task list from aaccli<br />
<br />
== backup ==<br />
backup<br />
backup script called nightly to update jail scripts and do customer backups<br />
<br />
== buildsafe ==<br />
buildsafe<br />
creates safe files based on quads (automatically removing the fsck’s). This will destructively overwrite safe files<br />
<br />
== checkload.pl ==<br />
checkload.pl<br />
this was intended to be setup as a cronjob to watch processes on a jail when the load <br />
rises above a certain level. Not currently in use.<br />
<br />
== checkprio.pl ==<br />
checkprio.pl<br />
will look for any process (other than the current shell’s csh, sh, sshd procs) with a non-normal priority and normalize it<br />
<br />
== diskusagemon == <br />
diskusagemon <mount point> <1k blocks><br />
watches a mount point’s disk use, when it reaches the level specified in the 2nd argument,<br />
it exits. This is useful when doing a restore and you want to be paged as it’s nearing completion.<br />
Best used as: <tt>diskusagemon /asd/asd 1234; pagexxx</tt><br />
<br />
== dumprestore ==<br />
dumprestore <dumpfile><br />
this is a perl expect script which automatically enters ‘1’ and ‘y’. It seems to cause restore to fail<br />
to set owner permissions on large restores.<br />
<br />
== g ==<br />
g <search><br />
greps the quad/safe files for the given search parameter<br />
<br />
== gather.pl ==<br />
gather.pl<br />
gathers up data about jails configured and writes to a file. Used for audits against the db<br />
<br />
== gb ==<br />
gb <search><br />
greps backup.config for the given search parameter<br />
<br />
== gbg ==<br />
gbg <search><br />
greps backup.config for the given search parameter and presents just the directories (for clean pasting)<br />
<br />
== ipfwbackup ==<br />
ipfwbackup<br />
writes ipfw traffic count data to a logfile<br />
<br />
== ipfwreset ==<br />
ipfwreset<br />
writes ipfw traffic count data to a logfile and resets counters to 0<br />
<br />
== js ==<br />
js<br />
output varies by OS version, but generally provides information about the base jail:<br />
- which vn’s are in use<br />
- disk usage<br />
- info about the contents of quads<br />
- the # of inodes represented by the jails contained in the group (133.2 in the example below), and how many jails per data mount, as well as subtotals<br />
- ips bound to the base machine but not in use by a jail<br />
- free gvinum volumes, or unused vn’s or used md’s<br />
<br />
<pre>/usr/local/jail/rc.d/quad1:<br />
/mnt/data1 133.2 (1)<br />
/mnt/data2 1040.5 (7)<br />
total 1173.7 (8)<br />
/usr/local/jail/rc.d/quad2:<br />
/mnt/data1 983.4 (6)<br />
total 983.4 (6)<br />
/usr/local/jail/rc.d/quad3:<br />
/mnt/data1 693.4 (4)<br />
/mnt/data2 371.6 (3)<br />
total 1065 (7)<br />
/usr/local/jail/rc.d/quad4:<br />
/mnt/data1 466.6 (3)<br />
/mnt/data2 882.2 (5)<br />
total 1348.8 (8)<br />
/mnt/data1: 2276.6 (14)<br />
/mnt/data2: 2294.3 (15)<br />
<br />
Available IPs:<br />
69.55.230.11 69.55.230.13 69.55.228.200<br />
<br />
Available volumes:<br />
v78 /mnt/data2 2G<br />
v79 /mnt/data2 2G<br />
v80 /mnt/data2 2G</pre> <br />
<br />
== load.pl ==<br />
load.pl<br />
feeds info to load mrtg - executed by inetd.<br />
<br />
== makevirginjail ==<br />
makevirginjail<br />
Only on some systems, makes an empty jail (doesn't do restore step)<br />
<br />
== mb == <br />
mb <mount|umount><br />
(nfs) mounts and umounts dirs to backup2. Shortcuts are mbm and mbu to mount and unmount. <br />
<br />
== notify.sh ==<br />
notify.sh<br />
emails reboot@johncompanies.com – intended to be called at boot time to alert us to a machine which panics and reboots and isn’t caught by bb or castle.<br />
<br />
== orphanedbackupwatch ==<br />
orphanedbackupwatch<br />
looks for directories on backup2 which aren’t configured in backup.config and offers to delete them<br />
<br />
== postboot ==<br />
postboot<br />
to be run after a machine reboot and quad/safe’s are done executing. It will:<br />
* do chmod 666 on each jail’s /dev/null<br />
* add ipfw counts<br />
* run jailpsall (so you can see if a configured jail isn’t running)<br />
<br />
== preboot ==<br />
preboot<br />
to be run before running quad/safe – checks for misconfigurations: <br />
* a jail configured in a quad but not a safe<br />
* a jail is listed more than once in a quad<br />
* the ip assigned to a jail isn’t configured on the machine<br />
* alias numbering skips in the rc.conf (resulting in the above)<br />
* orphaned vnfile's that aren't mentioned in a quad/safe<br />
* ip mismatches between dir/vnfile name and the jail’s ip<br />
* dir/vnfiles's in quad/safe that don’t exist <br />
<br />
== quadanalyze.pl ==<br />
quadanalyze.pl<br />
called by js, produces the info (seen above with js explanation) about the contents of quad (inode count, # of jails, etc.)<br />
<br />
== rsync.backup ==<br />
rsync.backup<br />
does customer backups (relies on backup.config)<br />
<br />
== taskdone ==<br />
taskdone<br />
when called will email support@johncompanies.com with the hostname of the machine from which it was executed as the subject<br />
<br />
== topten ==<br />
topten<br />
summarizes the top 10 traffic users (called by ipfwreset)<br />
<br />
== trafficgather.pl ==<br />
trafficgather.pl [yy-mm]<br />
sends a traffic usage summary by jail to support@johncomapnies.com and payments@johncompanies.com. Optional arguments are year and month (must be in the past). If not passed, assumes last month. Relies on traffic logs created by ipfwreset and ipfwbackup<br />
<br />
== trafficwatch.pl ==<br />
trafficwatch.pl<br />
checks traffic usage and emails support@johncomapnies.com when a jail reaches the warning level (35G) and the limit (40G). We really aren’t using this anymore now that we have netflow.<br />
<br />
== trafstats ==<br />
trafstats<br />
writes ipfw traffic usage info by jail to a file called jc_traffic_dump in each jail’s / dir<br />
<br />
== truncate_jailmake ==<br />
truncate_jailmake<br />
a version of jailmake which creates truncated vnfiles.<br />
<br />
== vb ==<br />
vb<br />
the equivalent of: <tt>vi /usr/local/jail/bin/backup.config</tt><br />
<br />
== vs (freebsd) ==<br />
vs<n><br />
the equivalent of: <tt>vi /usr/local/jail/rc.d/safe<n></tt><br />
<br />
vq<n><br />
the equivalent of: vi /usr/local/jail/rc.d/quad<n><br />
<br />
== dumpremote ==<br />
dumpremote <user@machine> </remote/location/file-dump> <vnX><br />
ex: dumpremote user@10.1.4.117 /mnt/data3/remote.echoditto.com-dump 7<br />
this will dump a vn filesystem to a remote machine and location<br />
<br />
== oversellcheck ==<br />
oversellcheck<br />
displays how much a disk is oversold or undersold taking into account truncated vn files. Only for use on 4.x systems<br />
<br />
== mvbackups (freebsd) ==<br />
mvbackups <dir> (1.1.1.1-col00001-DIR) <target_machine> (jail1) <target_dir> (data1)<br />
moves backups from one location to another on the backup server, and provides you with option to remove entries from current backup.config, and simple paste command to add the config to backup.config on the target server<br />
<br />
== jailnice ==<br />
jailnice <hostname><br />
applies <tt>renice 19 [PID]</tt> and <tt>rtprio 31 –[PID]</tt> to each process in the given jail<br />
<br />
== dumpremoterestore ==<br />
dumpremoterestore <device> <ip of target machine> <dir on target machine><br />
ex: dumpremoterestore /dev/vn51 10.1.4.118 /mnt/data2/69.55.239.45-col00688-DIR<br />
dumps a device and restores it to a directory on a remote machine. Requires that you enable root ssh on the <br />
remote machine.<br />
<br />
== psj ==<br />
psj<br />
shows just the procs running on the base system – a ps auxw but without jail’d procs present<br />
<br />
== perc5iraidchk ==<br />
perc5iraidchk<br />
checks for degraded arrays on Dell 2950 systems with Perc5/6 controllers<br />
<br />
== perc4eraidchk ==<br />
perc4eraidchk<br />
checks for degraded arrays on Dell 2850 systems with Perc4e/Di controllers<br />
<br />
= Making new customer VE (vm) =<br />
<br />
This applies only to new virts >= 4.x<br />
<br />
grab ip from ipmap (if opened from the pending cust screen it should take you to the right block). You can also run vzlist -a to see what block is in use, generally. <br />
# confirm ip you want to use isn’t in use via Mgmt. -> IP Map in management screens<br />
# put CT on opposite on whichever partition has more space<br />
# vm <CID> <IP> <hostname> <mountpoint> <email> <template> <256|384|512|768> [disk] <br />
vm col01959 69.55.236.214 gemini /vz1 finbar@arrivell.com ubuntu-9.04 256<br />
# copy veid, dir, ip and password to pending customer screen<br />
<br />
<br />
= Virtuozzo VPS =<br />
<br />
Note: We use VEID (Virtual Environment ID) and CTID (Container ID) interchangably. Similarly, VE and CT. They mean the same thing.<br />
VZPP = VirtuoZzo Power Panel (the control panel for each CT)<br />
<br />
All linux systems exist in /vz, /vz1 or /vz2 - since each linux machine holds roughly 60-90 customers, there will be roughly 30-45 in each partition.<br />
<br />
The actual filesystem of the system in question is in:<br />
<br />
/vz(1-2)/private/(VEID)<br />
<br />
Where VEID is the identifier for that system - an all-numeric string larger than 100.<br />
<br />
The actual mounted and running systems are in the corresponding:<br />
<br />
/vz(1-2)/root/(VEID)<br />
<br />
But we rarely interact with any system from this mount point.<br />
<br />
You should never need to touch the root portion of their system – however you can traverse their filesystem by going to <tt>/vz(1-2)/private/(VEID)/root</tt> (<tt>/vz(1-2)/private/(VEID)/fs/root</tt> on 4.x systems) the root of their filesystem is in that directory, and their entire system is underneath that.<br />
<br />
Every VE has a startup script in <tt>/etc/sysconfig/vz-scripts</tt> (which is symlinked as <tt>/vzconf</tt> on all systems) - the VE startup script is simply named <tt>(VEID).conf</tt> - it contains all the system parameters for that VE:<br />
<br />
<pre># Configuration file generated by vzsplit for 60 VE<br />
# on HN with total amount of physical mem 2011 Mb<br />
<br />
VERSION="2"<br />
CLASSID="2"<br />
<br />
ONBOOT="yes"<br />
<br />
KMEMSIZE="8100000:8200000"<br />
LOCKEDPAGES="322:322"<br />
PRIVVMPAGES="610000:615000"<br />
SHMPAGES="33000:34500"<br />
NUMPROC="410:415"<br />
PHYSPAGES="0:2147483647"<br />
VMGUARPAGES="13019:2147483647"<br />
OOMGUARPAGES="13019:2147483647"<br />
NUMTCPSOCK="1210:1215"<br />
NUMFLOCK="107:117"<br />
NUMPTY="19:19"<br />
NUMSIGINFO="274:274"<br />
TCPSNDBUF="1800000:1900000"<br />
TCPRCVBUF="1800000:1900000"<br />
OTHERSOCKBUF="900000:950000"<br />
DGRAMRCVBUF="200000:200000"<br />
NUMOTHERSOCK="650:660"<br />
DCACHE="786432:818029"<br />
NUMFILE="7500:7600"<br />
AVNUMPROC="51:51"<br />
IPTENTRIES="155:155"<br />
DISKSPACE="4194304:4613734"<br />
DISKINODES="400000:420000"<br />
CPUUNITS="1412"<br />
QUOTAUGIDLIMIT="2000"<br />
VE_ROOT="/vz1/root/636"<br />
VE_PRIVATE="/vz1/private/636"<br />
NAMESERVER="69.55.225.225 69.55.230.3"<br />
OSTEMPLATE="vzredhat-7.3/20030305"<br />
VE_TYPE="regular"<br />
IP_ADDRESS="69.55.225.229"<br />
HOSTNAME="textengine.net"</pre><br />
<br />
<br />
As you can see, the hostname is set here, the disk space is set here, the number of inodes, the number of files that can be open, the number of tcp sockets, etc. - all are set here.<br />
<br />
In fact, everything that can be set on this customer system is set in this conf file.<br />
<br />
<br />
All interaction with the customer system is done with the VEID. You start the system by running:<br />
<br />
vzctl start 999<br />
<br />
You stop it by running:<br />
<br />
vzctl stop 999<br />
<br />
You execute commands in it by running:<br />
<br />
vzctl exec 999 df -k<br />
<br />
You enter into it, via a root-shell backdoor with:<br />
<br />
vzctl enter 999<br />
<br />
and you set parameters for the system, while it is still running, with:<br />
<br />
vzctl set 999 --diskspace 6100000:6200000 --save<br />
<br />
<tt>vzctl</tt> is the most commonly used command - we have aliased <tt>v</tt> to <tt>vzctl</tt> since we use it so often. We’ll continue to use <tt>vzctl</tt> in our examples, but feel free to use just <tt>v</tt>.<br />
<br />
Let's say the user wants more diskspace. You can cat their conf file and see:<br />
<br />
DISKSPACE="4194304:4613734"<br />
<br />
So right now they have 4gigs of space. You can then change it to 6 with:<br />
<br />
vzctl set 999 --diskspace 6100000:6200000 --save<br />
<br />
IMPORTANT: all issuances of the vzctl set command need to end with <tt>–save</tt> - if they don't, the setting will be set, but it will not be saved to the conf file, and they will not have those settings next time they boot.<br />
<br />
All of the tunables in the conf file can be set with the vzctl set command. Note that in the conf file, and on the vzctl set command line, we always issue two numbers seperated by a colon - that is because we are setting the hard and soft limits. Always set the hard limit slightly above the soft limit, as you see it is in the conf file for all those settings.<br />
<br />
There are also things you can set with `<tt>vzctl set</tt>` that are not in the conf file as settings, per se. For instance, you can add IPs:<br />
<br />
vzctl set 999 --ipadd 10.10.10.10 --save<br />
<br />
or multiple IPs:<br />
<br />
vzctl set 999 --ipadd 10.10.10.10 --ipadd 10.10.20.30 --save<br />
<br />
or change the hostname:<br />
<br />
vzctl set 999 --hostname www.example.com --save<br />
<br />
You can even set the nameservers:<br />
<br />
vzctl set 999 --nameserver 198.78.66.4 --nameserver 198.78.70.180 --save<br />
<br />
Although you probably will never do that.<br />
<br />
You can disable a VPS from being started (by VZPP or reboot) (4.x):<br />
<br />
vzctl set 999 --disabled yes --save <br />
<br />
You can disable a VPS from being started (by VZPP or reboot) (<=3.x):<br />
<br />
vzctl set 999 --onboot=no --save <br />
<br />
You can disable a VPS from using his control panel:<br />
<br />
vzctl set 999 --offline_management=no --save <br />
<br />
You can suspend a VPS, so it can be resumed in the same state it was in when it was stopped (4.x):<br />
<br />
vzctl suspend 999<br />
<br />
and to resume it:<br />
<br />
vzctl resume 999<br />
<br />
to see who owns process:<br />
vzpid <PID><br />
<br />
to mount up an unmounted ve:<br />
vzctl mount 827<br />
<br />
<br />
To see network stats for CT's:<br />
<pre># vznetstat<br />
VEID Net.Class Output(bytes) Input(bytes)<br />
-----------------------------------------------<br />
24218 1 484M 39M<br />
24245 1 463M 143M<br />
2451 1 2224M 265M<br />
2454 1 2616M 385M<br />
4149 1 125M 68M<br />
418 1 1560M 34M<br />
472 1 1219M 315M<br />
726 1 628M 317M<br />
763 1 223M 82M<br />
771 1 4234M 437M<br />
-----------------------------------------------<br />
Total: 13780M 2090M</pre><br />
<br />
<br />
One thing that sometimes comes up on older systems that we created with smaller defaults is that the system would run out of inodes. The user will email and say they cannot create any more files or grow any files larger, but they will also say that they are not out of diskspace ... they are running:<br />
<br />
df -k<br />
<br />
and seeing how much space is free - and they are not out of space. They are most likely out of inodes - which they would see by running:<br />
<br />
df -i<br />
<br />
So, the first thing you should do is enter their system with:<br />
<br />
vzctl enter 999<br />
<br />
and run: <tt>df -i</tt><br />
<br />
to confirm your theory. Then exit their system. Then simply cat their conf file and see what their inodes are set to (probably 200000:200000, since that was the old default on the older systems) and run:<br />
<br />
vzctl set 999 --diskinodes 400000:400000 --save<br />
<br />
If they are not out of inodes, then a good possibility is that they have maxed out their numfile configuration variable, which controls how many files they can have in their system. The current default is 7500 (which nobody has ever hit), but the old default was as low as 2000, so you would run something like:<br />
<br />
vzctl set 999 --numfile 7500:7500 --save<br />
<br />
----<br />
<br />
You cannot start or stop a VE if your pwd is its private (/vz/private/999) or root (/vz/root/999) directories, or anywhere below them.<br />
<br />
<br />
== Recovering from a crash (linux) ==<br />
<br />
=== Diagnose whether you have a crash ===<br />
The most important thing is to get the machine and all ve’s back up as soon as possible. Note the time, you’ll need to create a crash log entry (Mgmt. -> Reference -> CrashLog). The first thing to do is head over to the [[Screen#Screen_Organization|serial console screen]] and see if there’s any kernel error messages output. Try to copy any messages (or just a sample of repeating messages) you see into the notes section of the crash log – these will also likely need to be sent to virtuozzo for interpretation. If the messages are spewing too fast, hit ^O + H to start a screen log dump which you can ob1182.pts-38.bb serve after the machine is rebooted. Additionally, if the machine is responsive, you can get a trace to send to virtuozzo by hooking up a kvm and entering these 3 sequences:<br />
<pre>alt+print screen+m<br />
alt+print screen+p<br />
alt+print screen+t</pre><br />
<br />
If there are no messages, the machine may just be really busy- wait a bit (5-10min) to see if it comes back. If it's still pinging, odds are its very busy. If it doesn't come back, or the messages indicate a fatal error, you will need to proceed with a power cycle (ctrl+alt+del will not work).<br />
<br />
=== Power cycle the server ===<br />
If this machine is not a Dell 2950 with a [[DRAC/RMM#DRAC|DRAC card]] (i.e. if you can’t ssh into the DRAC card and issue racadm serveraction hardreset, then you will need someone at the data center to power the macine off, wait 30 sec, then turn it back on. Make sure to re-attach via console (<tt>tip virtxx</tt>) immediately after power down. <br />
<br />
=== (Re)attach to the console ===<br />
Stay on the console the entire time during boot. As the BIOS posts- look out for the RAID card output- does everything look healthy? The output may be scrambled, look for "DEGRADED" or "FAILED". Once the OS starts booting you will be disconnected (dropped back to the shell on the console server) a couple times during the boot up. The reason you want to quickly re-attach is two-fold: 1. If you don’t reattach quickly then you won’t get any console output, 2. you want to be attached before the server ''potentially'' starts (an extensive) fsck. If you attach after the fsck begins, you’ll have seen no indication it started an fsck and the server will appear frozen during startup- no output, no response. <br />
<br />
=== Start containers/VE's/VPSs ===<br />
When the machine begins to start VE’s, it’s safe to leave the console and login via ssh. All virts should be set to auto start all the VEs after a crash. Further, most (newer) virts are set to “fastboot” it’s VE’s (to find out, do:<br />
grep -i fast /etc/sysconfig/vz <br />
and look for <tt>VZFASTBOOT=yes</tt>). If this was set prior to the machine’s crash (setting it after the machine boots will not have any effect until the vz service is restarted) it will start each ve as fast as possible, in serial, then go thru each VE (serially), shutting it down running a vzquota (disk usage) check, then bringing it back up. The benefit is that all VE’s are brought up quickly (within 15min or so depending on the #), the downside is a customer watching closely will notice 2 outages – 1st the machine crash, 2nd their quota check (which will be a much shorter downtime- on the order of a few minutes). <br />
<br />
Where “fastboot” is not set to yes (i.e on quar1), vz will start them consecutively, checking the quotas one at a time, and the 60th VE may not start until an hour or two later - this is not acceptable.<br />
<br />
The good news is, if you run vzctl start for a VE that is already started, you will simply get an error: <tt>VE is already started</tt>. Further, if you attempt to vzctl start a VE that is in the process of being started, you will simply get an error: unable to lock VE. So, there is no danger in simply running scripts to start smaller sets of VEs. If the system is not autostarting, then there is no issue, and even if it does, when it conflicts, one process (yours or the autostart) will lose, and just move on to the next one.<br />
<br />
A script has been written to assist with ve starts: [[#startvirt.pl|startvirt.pl]] which will start 6 ve’s at once until there are no more left. If startvirt.pl is used on a system where “fastboot” was on, it will circumvent the fastboot for ve’s started by startvirt.pl – they will go through the complete quota check before starting- therefore this is not advisable when a system has crashed. When a system is booted cleanly, and there's no need for vzquota checks, then startvirt.pl is safe and advisable to run.<br />
<br />
=== Make sure all containers are running ===<br />
You can quickly get a feel for how many ve’s are started by running:<br />
<br />
<pre>[root@virt4 log]# vs<br />
VEID 16066 exist mounted running<br />
VEID 16067 exist mounted running<br />
VEID 4102 exist mounted running<br />
VEID 4112 exist mounted running<br />
VEID 4116 exist mounted running<br />
VEID 4122 exist mounted running<br />
VEID 4123 exist mounted running<br />
VEID 4124 exist mounted running<br />
VEID 4132 exist mounted running<br />
VEID 4148 exist mounted running<br />
VEID 4151 exist mounted running<br />
VEID 4155 exist mounted running<br />
VEID 42 exist mounted running<br />
VEID 432 exist mounted running<br />
VEID 434 exist mounted running<br />
VEID 442 exist mounted running<br />
VEID 450 exist mounted running<br />
VEID 452 exist mounted running<br />
VEID 453 exist mounted running<br />
VEID 454 exist mounted running<br />
VEID 462 exist mounted running<br />
VEID 463 exist mounted running<br />
VEID 464 exist mounted running<br />
VEID 465 exist mounted running<br />
VEID 477 exist mounted running<br />
VEID 484 exist mounted running<br />
VEID 486 exist mounted running<br />
VEID 490 exist mounted running</pre><br />
<br />
So to see how many ve’s have started:<br />
<pre>[root@virt11 root]# vs | grep running | wc -l<br />
39</pre><br />
<br />
And to see how many haven’t:<br />
<pre>[root@virt11 root]# vs | grep down | wc -l<br />
0</pre><br />
<br />
And how many we should have running:<br />
<pre>[root@virt11 root]# vs | wc -l<br />
39</pre><br />
<br />
Another tool you can use to see which ve’s have started, among other things is [[#vzstat|vzstat]]. It will give you CPU, memory, and other stats on each ve and the overall system. It’s a good thing to watch as ve’s are starting (note the VENum parameter, it will tell you how many have started):<br />
<br />
<pre>4:37pm, up 3 days, 5:31, 1 user, load average: 1.57, 1.68, 1.79<br />
VENum 40, procs 1705: running 2, sleeping 1694, unint 0, zombie 9, stopped 0<br />
CPU [ OK ]: VEs 57%, VE0 0%, user 8%, sys 7%, idle 85%, lat(ms) 412/2<br />
Mem [ OK ]: total 6057MB, free 9MB/54MB (low/high), lat(ms) 0/0<br />
Swap [ OK ]: tot 6142MB, free 4953MB, in 0.000MB/s, out 0.000MB/s<br />
Net [ OK ]: tot: in 0.043MB/s 402pkt/s, out 0.382MB/s 4116pkt/s<br />
Disks [ OK ]: in 0.002MB/s, out 0.000MB/s<br />
<br />
VEID ST %VM %KM PROC CPU SOCK FCNT MLAT IP<br />
1 OK 1.0/17 0.0/0.4 0/32/256 0.0/0.5 39/1256 0 9 69.55.227.152<br />
21 OK 1.3/39 0.1/0.2 0/46/410 0.2/2.8 23/1860 0 6 69.55.239.60<br />
133 OK 3.1/39 0.1/0.3 1/34/410 6.3/2.8 98/1860 0 0 69.55.227.147<br />
263 OK 2.3/39 0.1/0.2 0/56/410 0.3/2.8 34/1860 0 1 69.55.237.74<br />
456 OK 17/39 0.1/0.2 0/100/410 0.1/2.8 48/1860 0 11 69.55.236.65<br />
476 OK 0.6/39 0.0/0.2 0/33/410 0.1/2.8 96/1860 0 10 69.55.227.151<br />
524 OK 1.8/39 0.1/0.2 0/33/410 0.0/2.8 28/1860 0 0 69.55.227.153<br />
594 OK 3.1/39 0.1/0.2 0/45/410 0.0/2.8 87/1860 0 1 69.55.239.40<br />
670 OK 7.7/39 0.2/0.3 0/98/410 0.0/2.8 64/1860 0 216 69.55.225.136<br />
691 OK 2.0/39 0.1/0.2 0/31/410 0.0/0.7 25/1860 0 1 69.55.234.96<br />
744 OK 0.1/17 0.0/0.5 0/10/410 0.0/0.7 7/1860 0 6 69.55.224.253<br />
755 OK 1.1/39 0.0/0.2 0/27/410 0.0/2.8 33/1860 0 0 192.168.1.4<br />
835 OK 1.1/39 0.0/0.2 0/19/410 0.0/2.8 5/1860 0 0 69.55.227.134<br />
856 OK 0.3/39 0.0/0.2 0/13/410 0.0/2.8 16/1860 0 0 69.55.227.137<br />
936 OK 3.2/52 0.2/0.4 0/75/410 0.2/0.7 69/1910 0 8 69.55.224.181<br />
1020 OK 3.9/39 0.1/0.2 0/60/410 0.1/0.7 55/1860 0 8 69.55.227.52<br />
1027 OK 0.3/39 0.0/0.2 0/14/410 0.0/2.8 17/1860 0 0 69.55.227.83<br />
1029 OK 1.9/39 0.1/0.2 0/48/410 0.2/2.8 25/1860 0 5 69.55.227.85<br />
1032 OK 12/39 0.1/0.4 0/80/410 0.0/2.8 41/1860 0 8 69.55.227.90</pre><br />
<br />
When you are all done, you will want to make sure that all the VEs really did get started, run vs one more time.<br />
<br />
Note the time all ve’s are back up and enter that into and save the crash log entry.<br />
<br />
Occasionally, a ve will not start automatically. The most common reason for a ve not to come up normally is the ve was at it’s disk limit before the crash, and will not start since they’re over the limit. To overcome this, set the disk space to current usage level (the system will give this to you when it fails to start), start the ve, then re-set the disk space back to the prior level. Lastly, contact the customer to let them know they’re out of disk (or allocate more disk if they're entitled to more).<br />
<br />
== Hitting performance barriers and fixing them ==<br />
<br />
There are multiple modes virtuozzo offers to allocate resources to a ve. We utilize 2: SLM and UBC parameters<br />
On our 4.x systems, we use all SLM – it’s simpler to manage and understand. There are a few systems on virt19/18 that may also use SLM. Everything else uses UBC. <br />
You can tell a SLM ve by:<br />
<br />
SLMMODE="all"<br />
<br />
in their conf file. <br />
<br />
TODO: detail SLM modes and parameters.<br />
<br />
If someone is in SLM mode and they hit memory resource limits, they simply need to upgrade to more memory.<br />
<br />
The following applies to everyone else (UBC).<br />
<br />
Customers will often email and say that they are getting out of memory errors - a common one is "cannot fork" ... basically, anytime you see something odd like this, it means they are hitting one of their limits that is in place in their conf file.<br />
<br />
The conf file, however, simply shows their limits - how do we know what they are currently at ?<br />
<br />
The answer is a file called v - this file contains the current status (and peaks) of their performance settings, and also counts how many times they have hit the barrier. The output of the file looks like this:<br />
<br />
<pre>764: kmemsize 384113 898185 8100000 8200000 0<br />
lockedpages 0 0 322 322 0<br />
privvmpages 1292 7108 610000 615000 0<br />
shmpages 270 528 33000 34500 0<br />
dummy 0 0 0 0 0<br />
numproc 8 23 410 415 0<br />
physpages 48 5624 0 2147483647 0<br />
vmguarpages 0 0 13019 2147483647 0<br />
oomguarpages 641 6389 13019 2147483647 0<br />
numtcpsock 3 21 1210 1215 0<br />
numflock 1 3 107 117 0<br />
numpty 0 2 19 19 0<br />
numsiginfo 0 4 274 274 0<br />
tcpsndbuf 0 80928 1800000 1900000 0 <br />
tcprcvbuf 0 108976 1800000 1900000 0<br />
othersockbuf 2224 37568 900000 950000 0<br />
dgramrcvbuf 0 4272 200000 200000 0<br />
numothersock 3 9 650 660 0<br />
dcachesize 53922 100320 786432 818029 0<br />
numfile 161 382 7500 7600 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
numiptent 4 4 155 155 0</pre><br />
<br />
The first column is the name of the counter in question - the same names we saw in the systems conf file. The second column is the _current_ value of that counter, the third column is the max that that counter has ever risen to, the fourth column is the soft limit, and the fifth column is the hard limit (which is the same as the numbers in that systems conf file).<br />
<br />
The sixth number is the failcount - how many times the current usage has risen to hit the barrier. It will increase as soon as the current usage hits the soft limit.<br />
<br />
The problem with /proc/user_beancounters is that it actually contains that set of data for every running VE - so you can't just cat /proc/user_beancounters - it is too long and you get info for every other running system.<br />
<br />
You can vzctl enter the system and run:<br />
<br />
vzctl enter 9999<br />
cat /proc/user_beancounters<br />
<br />
inside their system, and you will just see the stats for their particular system, but entering their system every time you want to see it is combersome.<br />
<br />
So, I wrote a simple script called "vzs" which simply greps for the VEID, and spits out the next 20 or so lines (however many lines there are in the output, I forget) after it. For instance:<br />
<br />
<pre># vzs 765:<br />
765: kmemsize 2007936 2562780 8100000 8200000 0<br />
lockedpages 0 8 322 322 0<br />
privvmpages 26925 71126 610000 615000 0<br />
shmpages 16654 16750 33000 34500 0<br />
dummy 0 0 0 0 0<br />
numproc 41 57 410 415 0<br />
physpages 1794 49160 0 2147483647 0<br />
vmguarpages 0 0 13019 2147483647 0<br />
oomguarpages 4780 51270 13019 2147483647 0<br />
numtcpsock 23 37 1210 1215 0<br />
numflock 17 39 107 117 0<br />
numpty 1 3 19 19 0<br />
numsiginfo 0 6 274 274 0<br />
tcpsndbuf 22240 333600 1800000 1900000 0<br />
tcprcvbuf 0 222656 1800000 1900000 0<br />
othersockbuf 104528 414944 900000 950000 0<br />
dgramrcvbuf 0 4448 200000 200000 0<br />
numothersock 73 105 650 660 0<br />
dcachesize 247038 309111 786432 818029 0<br />
numfile 904 1231 7500 7600 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
numiptent 4 4 155 155 0</pre><br />
<br />
That showed us just the portion of /proc/user_beancounters for system 765.<br />
<br />
When you run the vzs command, always add a : after the VEID.<br />
<br />
So, if a customer complains about some out of memory errors, or no more files, or no more ptys, or just has an unspecific complain about processes dying, etc., the very first thing you need to do is check their beancounters with vzs. Usually you will spot an item that has a high failcount and needs to be upped.<br />
<br />
At that point you could simply up the counter with `vzctl set`. Generally pick a number 10-20% higher than the old one, and make the hard limit slightly larger than the the soft limit. However our systems now come in several levels and those levels have more/different memory allocations. If someone is complaining about something other than a memory limit (pty, numiptent, numflock), it’s generally safe to increase it, at least to the same level as what’s in the /vzconf/4unlimited file on the newest virt. If someone is hitting a memory limit, first make sure they are given what they deserve:<br />
<br />
(refer to mgmt -> payments -> packages)<br />
<br />
To set those levels, you use the [[#setmem|setmem]] command. <br />
<br />
The alternate (DEPRECATED) method would be to use one of 3 commands:<br />
256 <veid><br />
300 <veid><br />
384 <veid><br />
512 <veid><br />
<br />
If the levels were not right (you’d run vzs <veid> before and after to see the effect) tell the customer they’ve been adjusted and be done with it. If the levels were right, tell the customer they must upgrade to a higher package, tell them how to see level (control panel) and that they can reboot their system to escape this lockup contidion.<br />
<br />
Customers can also complain that their site is totally unreachable, or complain that it is down ... if the underlying machine is up, and all seems well, you may notice in the beancounters that network-specific counters are failing - such as numtcpsock, tcpsndbuf or tcprcvbuf. This will keep them from talking on the network and make it seem like their system is down. Again, just up the limits and things should be fine.<br />
<br />
On virts 1-4, you should first look at the default settings for that item on a later virt, such as virt 8 - we have increased the defaults a lot since the early machines. So, if you are going to up a counter on virt2, instead of upping it by 10-20%, instead up it to the new default that you see on virt8.<br />
<br />
== Moving a VE to another virt (migrate/migrateonline) ==<br />
<br />
This will take a while to complete - and it is best to do this at night when the load is light on both machines.<br />
<br />
There are different methods for this, depending on which version of virtuozzo is installed on the src. and dst. virt. <br />
To check which version is running: <br />
[root@virt12 private]# cat /etc/virtuozzo-release<br />
Virtuozzo release 2.6.0<br />
<br />
Ok, let's say that the VE is 1212, and vital stats are:<br />
<pre>[root@virt12 sbin]# vc 1212<br />
VE_ROOT="/vz1/root/1212"<br />
VE_PRIVATE="/vz1/private/1212"<br />
OSTEMPLATE="fedora-core-2/20040903"<br />
IP_ADDRESS="69.55.229.84"<br />
TEMPLATES="devel-fc2/20040903 php-fc2/20040813 mysql-fc2/20040812 postgresql-fc2/20040813 mod_perl-fc2/20040812 mod_ssl-fc2/20040811 jre-fc2/20040823 jdk-fc2/20040823 mailman-fc2/20040823 analog-fc2/20040824 proftpd-fc2/20040818 tomcat-fc2/20040823 usermin-fc2/20040909 webmin-fc2/20040909 uw-imap-fc2/20040830 phpBB-fc2/20040831 spamassassin-fc2/20040910 PostNuke-fc2/20040824 sl-webalizer-fc2/20040<br />
818"<br />
<br />
[root@virt12 sbin]# vzctl exec 1212 df -h<br />
Filesystem Size Used Avail Use% Mounted on<br />
/dev/vzfs 4.0G 405M 3.7G 10% /</pre><br />
<br />
From this you can see that he’s using (and will minimally need free on the dst server) ~400MB, and he’s running on a Fedora 2 template, version 20040903. He’s also got a bunch of other templates installed. It’s is '''vital''' that '''all''' these templates exist on the dst system. To confirm that, on the dst system run:<br />
<br />
For < 3.0:<br />
<br />
<pre>[root@virt14 private]# vzpkgls | grep fc2<br />
devel-fc2 20040903<br />
PostNuke-fc2 20040824<br />
analog-fc2 20040824<br />
awstats-fc2 20040824<br />
bbClone-fc2 20040824<br />
jdk-fc2 20040823<br />
jre-fc2 20040823<br />
mailman-fc2 20040823<br />
mod_frontpage-fc2 20040816<br />
mod_perl-fc2 20040812<br />
mod_ssl-fc2 20040811<br />
mysql-fc2 20040812<br />
openwebmail-fc2 20040817<br />
php-fc2 20040813<br />
phpBB-fc2 20040831<br />
postgresql-fc2 20040813<br />
proftpd-fc2 20040818<br />
sl-webalizer-fc2 20040818<br />
spamassassin-fc2 20040910<br />
tomcat-fc2 20040823<br />
usermin-fc2 20040909<br />
uw-imap-fc2 20040830<br />
webmin-fc2 20040909<br />
[root@virt14 private]# vzpkgls | grep fedora<br />
fedora-core-1 20040121 20040818<br />
fedora-core-devel-1 20040121 20040818<br />
fedora-core-2 20040903<br />
[root@virt14 private]#</pre><br />
<br />
For these older systems, you can simply match up the date on the template. <br />
<br />
For >= 3.0:<br />
<br />
<pre>[root@virt19 /vz2/private]# vzpkg list<br />
centos-5-x86 2008-01-07 22:05:57<br />
centos-5-x86 devel<br />
centos-5-x86 jre<br />
centos-5-x86 jsdk<br />
centos-5-x86 mod_perl<br />
centos-5-x86 mod_ssl<br />
centos-5-x86 mysql<br />
centos-5-x86 php<br />
centos-5-x86 plesk9<br />
centos-5-x86 plesk9-antivirus<br />
centos-5-x86 plesk9-api<br />
centos-5-x86 plesk9-atmail<br />
centos-5-x86 plesk9-backup<br />
centos-5-x86 plesk9-horde<br />
centos-5-x86 plesk9-mailman<br />
centos-5-x86 plesk9-mod-bw<br />
centos-5-x86 plesk9-postfix<br />
centos-5-x86 plesk9-ppwse<br />
centos-5-x86 plesk9-psa-firewall<br />
centos-5-x86 plesk9-psa-vpn<br />
centos-5-x86 plesk9-psa-fileserver<br />
centos-5-x86 plesk9-qmail<br />
centos-5-x86 plesk9-sb-publish<br />
centos-5-x86 plesk9-vault<br />
centos-5-x86 plesk9-vault-most-popular<br />
centos-5-x86 plesk9-watchdog</pre><br />
<br />
On these newer systems, it's difficult to tell whether the template on the dst matches exactly the src. Just cause a centos-5-x86 is listed on both servers doesn't mean all the same packages are there on the dst. To truly know, you must perform a sample rsync:<br />
<br />
rsync -avn /vz/template/centos/5/x86/ root@10.1.4.61:/vz/template/centos/5/x86/<br />
<br />
if you see a ton of output from the dry run command, then clearly there are some differences. You may opt to let the rsync complete (without running in dry run mode) the only downside is you've now used up more space on the dst and also the centos template will be a mess with old and new data- it will be difficult if not impossible to undo (if someday we wanted to reclaim the space).<br />
<br />
If you choose to merge templates, you should closely inspect the dry run output. You should also take care to exclude anything in the /config directory. For example:<br />
<br />
rsync -av -e ssh --stats --exclude=x86/config /vz/template/ubuntu/10.04/ root@10.1.4.62:/vz/template/ubuntu/10.04/<br />
<br />
Which will avoid this directory and contents:<br />
<pre>[root@virt11 /vz2/private]# ls /vz/template/ubuntu/10.04/x86/config*<br />
app os</pre><br />
<br />
This is important to avoid since the config may differ on the destination and we are really only interested in making sure the pacakges are there, not overwriting a newer config with an older one.<br />
<br />
If the dst system was missing a template, you have 2 choices: <br />
# put the missing template on the dst system. 2 choices here: <br />
## Install the template from rpm (found under backup2: /mnt/data4/vzrpms/distro/) or <br />
## rsync over the template (found under /vz/template) - see above<br />
# put the ve on a system which has all the proper templates<br />
<br />
=== pre-seeding a migration ===<br />
<br />
When migrating a customer (or when doing many) depending on how much data you have to transfer, it can take some time. Further, it can be difficult to gauge when a migration will complete or how long it will take. To help speed up the process and get a better idea about how long it will take you can pre-transfer a customer's data to the destination server. If done correctly, vzmigrate will see the pre-transferred data and pick up where you left off, having much less to transfer (just changed/new files). <br />
<br />
We believe vzmigrate uses rsync to do it's transfer. Therefore not only can you use rsync to do a pre-seed, you can also run rsync to see what is causing a repeatedly-failing vzmigrate to fail. <br />
<br />
There's no magic to a pre-seed, you just need to make sure it's named correctly.<br />
<br />
Given:<br />
<br />
source: /vz1/private/1234<br />
<br />
and you want to migrate to /vz2 on the target system, your rsync would look like:<br />
<br />
rsync -av /vz1/private/1234/ root@x.x.x.x:/vz2/private/1234.migrated/<br />
<br />
After running that successful rsync, the ensuing migrateonline (or migrate) will take much less time to complete- depending on the # of files to be analyzed and the # of changed files. In any case, it'll be much much faster than had you just started the migration from scratch.<br />
<br />
Further, as we discuss elsewhere in this topic, a failed migration can be moved from <tt>/vz/private/1234</tt> to <tt>/vz/private/1234.migrated</tt> on the destination if you want to restart a failed migration. This should '''only''' be done if the migration failed and the CT is not running on the destination HN.<br />
<br />
=== migrateonline intructions: src >=3.x -> dst>=3.x ===<br />
<br />
A script called [[#migrateonline|migrateonline]] was written to handle this kind of move. It is basically a wrapper for <tt>vzmigrate</tt> – vzmigrate is a util to seamlessly- as no no reboot of the ve necessary- move a ve from one host to another. This wrapper was initially written cause vz virtuozzo version 2.6.0 has a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables, causing problems when the ve was started up on the dst system. [[#migrate|migrate]] mitigates that. Since it makes multiple ssh connections to the dst virt, it’s a good idea to put the pub key for the src virt in the authorized_keys file on the dst virt. In addition, migrateonline emails ve owners when their migration starts and stops. For this to happen they need to put email addresses (on a single line, space delimited) in a file on their system: /migrate_notify. If the optional target dir is not specified, the ve will be moved to the same private/root location as it was on the src virt. Note: <tt>migrate</tt> is equivalent to <tt>migrateonline</tt>, but will <tt>migrate</tt> a ve AND restart it in the process.<br />
<br />
<pre>[root@virt12 sbin]# migrateonline<br />
usage: /usr/local/sbin/migrateonline <ip of node migrating to> <veid> [target dir: vz | vz1 | vz2]<br />
[root@virt12 sbin]# migrateonline 10.1.4.64 1212 vz<br />
starting to migrate 1212 at Sat Mar 26 22:40:38 PST 2005<br />
Turning off offline_management<br />
Saved parameters for VE 1212<br />
migrating with no start on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
clearing the arp cache<br />
now going to 10.1.4.64 and clear cache and starting<br />
starting it<br />
Delete port redirection<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
finished migrating 1212 at Sat Mar 26 22 22:52:01 PST 2005<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine.<br />
<br />
If they had backups, use the mvbackups command to move their backups to the new server:<br />
<br />
mvbackups 1212 virt14 vz<br />
<br />
Rename the ve<br />
<br />
[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/migrated-1212<br />
<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/old-1212-migrated-20120404-noarchive<br />
<br />
Update the customer’s systems in mgmt to reflect the new path and server.<br />
<br />
<br />
IF migrateonline does not work, you can try again using simply migrate- this will result in a brief reboot for the ve.<br />
Before you try again, make sure of a few things:<br />
<br />
Depending on where in the migration died, there may be partial data on the dst system in 1 of 2 places:<br />
(given the example above)<br />
<br />
/vz/private/1212<br />
<br />
or <br />
<br />
/vz/private/1212.migrated<br />
<br />
before you run migrate again, you'll want to rename so that all data is in <br />
1212.migrated:<br />
<br />
mv /vz/private/1212 /vz/private/1212.migrated<br />
<br />
this way, it will pick up where it left off and transfer only new files.<br />
<br />
Likewise, if you want to speed up a migration, you can pre-seed the dst as follows:<br />
<br />
[root@virt12 sbin]# rsync -avSH /vz/private/1212/ root@10.1.4.64:/vz/private/1212.migrated/<br />
<br />
then when you run migrate or migrateonline, it will only need to move the changed files- the migration will complete quickly<br />
<br />
=== migrateonline/migrate failures (migrate manually) ===<br />
<br />
Lets say for whatever reason the migration fails. If it fails with [[#migrateonline|migrateonline]], you should try [[#migrate|migrate]] (which will reboot the customer, so notify them ahead of time).<br />
<br />
You may want to run a [[#pre-seeding_a_migration|pre-seed]] rsync to see if you can find the problem. On older virts, we've seen this problem due to a large logfile (which you can find and encourage the customer to remove/compress):<br />
for f in `find / -size +1048576k`; do ls -lh $f; done<br />
<br />
You may also see migration failing due to quota issues.<br />
<br />
You can try to resolve by copying any quota file into the file you need:<br />
<br />
cp /var/vzquota/quota.1 /var/vzquota/quota.xxx<br />
<br />
If it complains about quota running you should then be able to stop it<br />
<br />
vzquota off xxxx<br />
<br />
If all else fails, migrate to a new VEID<br />
i.e. 1234 becomes 12341<br />
<br />
<br />
If the rsync or [[#migrate|migrate]] fails, you can always move someone manually:<br />
<br />
1. stop ve: <br><br />
v stop 1234<br />
<br />
2. copy over data<br><br />
rsync -avSH /vz/private/1234/ root@1.1.1.1:/vzX/private/1234/<br />
<br />
NOTE: if you've previously seeded the data (run rsync while the VE was up/running), and this is a subsequent rsync, make sure the last rsync you do (while the VE is not running, has the --delete option in the rsync)<br />
<br />
3. copy over conf<br><br />
scp /vzconf/1234.conf root@1.1.1.1:/vzconf<br />
<br />
4. on dst, edit the conf to reflect the right vzX dir<br><br />
vi /vzconf/1234.conf<br />
<br />
5. on src remove the IPs<br><br />
ipdel 1234 2.2.2.2 3.3.3.3<br />
<br />
6. on dst add IPs <br><br />
ipadd 1234 2.2.2.2 3.3.3.3<br />
<br />
7. on dst, start ve: <br><br />
v start 1324<br />
<br />
8. cancel, then archive ve on src per above instrs.<br />
<br />
=== migrate src=2.6.0 -> dst>=2.6.0, or mass-migration with customer notify ===<br />
<br />
A script called <tt>migrate</tt> was written to handle this kind of move. It is basically a wrapper for vzmigrate – vzmigrate is a util to seamlessly move a ve from one host to another. This wrapper was initially written cause vz virtuozzo version 2.6.0 has a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables, causing problems when the ve was started up on the dst system. migrate mitigates that. Since it makes multiple ssh connections to the dst virt, it’s a good idea to put the pub key for the src virt in the authorized_keys file on the dst virt. In addition, migrate emails ve owners when their migration starts and stops. For this to happen they need to put email addresses (on a single line, space delimited) in a file on their system: /migrate_notify. If the optional target dir is not specified, the ve will be moved to the same private/root location as it was on the src virt. Note: migrateonline is equivalent to migrate, but will migrate a ve from one 2.6 '''kernel''' machine to another 2.6 kernel machine without restarting the ve.<br />
<br />
<pre>[root@virt12 sbin]# migrate<br />
usage: /usr/local/sbin/migrate <ip of node migrating to> <veid> [target dir: vz | vz1 | vz2]<br />
[root@virt12 sbin]# migrate 10.1.4.64 1212 vz<br />
starting to migrate 1212 at Sat Mar 26 22:40:38 PST 2005<br />
Turning off offline_management<br />
Saved parameters for VE 1212<br />
migrating with no start on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
clearing the arp cache<br />
now going to 10.1.4.64 and clear cache and starting<br />
starting it<br />
Delete port redirection<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
finished migrating 1212 at Sat Mar 26 22 22:52:01 PST 2005<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine (backup2 or mail).<br />
Cancel the ve (first we have to rename things which migrate changed so cancelve will find them):<br />
<br />
[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/1212.conf<br />
<br />
On 2.6.1 you’ll also have to move the private area:<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/1212<br />
<br />
<pre>[root@virt12 sbin]# cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, [[#cancelve|cancelve]] would offer to remove them. You want to say '''no''' to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
To move everyone off a system, you’d do:<br />
for f in `vl`; do migrate <ip> $f; done<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
=== vzmigrate: src=2.6.1 -> dst>=2.6.0 ===<br />
<br />
This version of vzmigrate works properly with regard to handling ips. It will not notify ve owners of moves as in the above example. Other than that it’s essentially the same.<br />
<br />
<pre>[root@virt12 sbin]# vzmigrate 10.1.4.64 -r no 1212:1212:/vz/private/1212:/vz/root/1212<br />
migrating on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine (backup2 or mail).<br />
Cancel the ve (first we have to rename things which vzmigrate changed so cancelve will find them):<br />
<br />
<pre>[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/1212.conf<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/1212<br />
<br />
[root@virt12 sbin]# cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, <tt>cancelve</tt> would offer to remove them. You want to say no to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
=== src=2.5.x ===<br />
<br />
First, go to the private dir:<br />
<br />
cd /vz1/private/<br />
<br />
Stop the VE - make sure it stops totally cleanly.<br />
<br />
vzctl stop 1212<br />
<br />
Then you’d use vemove - a script written to copy over the config, create tarballs of the ve’s data on the destination virt, and cancel the ve on the source system (in this example we’re going to put a ve that was in /vz1/private on the src virt, in /vz/private on the dst virt):<br />
<br />
<pre>[root@virt12 sbin]# vemove<br />
ERROR: Usage: vemove veid target_ip target_path_dir<br />
[root@virt12 sbin]# vemove 1212 10.1.4.64 /vz/private/1212<br />
tar cfpP - 1212 --ignore-failed-read | (ssh -2 -c arcfour 10.1.4.64 "split - -b 1024m /vz/private/1212.tar" )<br />
scp /vzconf/1212.conf 10.1.4.64:/vzconf<br />
cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, cancelve would offer to remove them. You want to say no to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
When you are done, go to /vz/private on the dst virt you will have files like this:<br />
<br />
<pre>1212.taraa<br />
1212.tarab<br />
1212.tarac</pre><br />
<br />
Each one 1024m (or less, for the last one) in size.<br />
<br />
on the dst server and run:<br />
<br />
cat 1212.tar?? | tar xpPBf -<br />
<br />
and after 20 mins or so it will be totally untarred. Now since the conf<br />
file is already there, you can go ahead and start the system.<br />
<br />
vzctl start 1212<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
NOTE: you MUST tar the system up using the virtuozzo version of tar that<br />
is on all the virt systems, and further you MUST untar the tarball with<br />
the virtuozzo tar, using these options: `<tt>tar xpPBf -</tt>`<br />
<br />
If you tar up an entire VE and move it to a non-virtuozzo machine, that is<br />
ok, and you can untar it there with normal tar commands, but do not untar<br />
it and then repack it with a normal tar and expect it to work - you need<br />
to use virtuozzo tar commands on virtuozzo tarballs to make it work.<br />
<br />
The backups are sort of an exception, since we are just (usually)<br />
restoring user data that was created after we gave them the system, and<br />
therefore has nothing to do with magic symlinks or vz-rpms, etc.<br />
<br />
== Moving a VE on the same virt ==<br />
<br />
Easy way:<br><br />
Scenario 1: ve 123 is to be renamed 1231 and moved from vz1 to vz<br />
<br />
vzmlocal 123:1231:/vz/private/1231:/vz/root/1231<br />
<br />
Scenario 2: ve 123 is to be moved vz1 to vz<br />
<br />
vzmlocal 123:123:/vz/private/123:/vz/root/123<br />
<br />
vzmlocal will reboot the ve at the end of the move<br />
<br />
Manual/old way:<br />
<br />
1) <tt>vzctl stop 123</tt><br><br />
2) <tt>mv /vz1/private/123 /vz/private/.</tt><br><br />
(or cp -a if you want to copy)<br />
3) in <tt>/etc/sysconfig/vz-scripts/123.conf</tt> change value<br><br />
of '<tt>VE_PRIVATE</tt>' variable to point to a new private area location<br />
4) <tt>vzctl start 123</tt><br><br />
5) update backups if needed: <tt>mvbackups 123 virtX virt1 vz</tt><br><br />
6) update management scerens<br><br />
<br />
Notes: a) absolute path to private area is stored in quota file <tt>/var/vzquota/quota.123</tt> - so during first startup quota will be recalculated.<br><br />
b) if you're going to write some script to do a job, you MUST be sure that $VEID won't be expanded to '' in ve config file - ie. you need to escape '$'. Otherwise you might have:<br />
<br />
VE_PRIVATE="/vz/private/"<br />
<br />
in config, and 'vzctl destroy' for this VE ID '''will remove everything under /vz/private/ directory'''.<br />
<br />
== Adding a veth device to a VE ==<br />
<br />
Not totally sure what this is, but a customer asked for it and here's what we did (as instructed by vz support):<br />
<br />
<pre>v set 99 --netif_add eth99 --save<br />
ipdel 99 69.55.230.58<br />
v set 99 --ifname eth99 --ipadd 69.55.230.58 --save<br />
v set 99 --ifname eth99 --gateway 69.55.230.1 --save<br />
<br />
vznetcfg net new veth_net<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
net99 active eth0 veth77.77,veth99.99<br />
veth_net active<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
br0 bridge veth_net<br />
br99 bridge net99<br />
veth99.99 veth net99<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic net99 69.55.227.70/24<br />
<br />
vznetcfg br attach br0 eth0<br />
<br />
(will remove 99 from orig net and move to veth_net)<br />
vznetcfg net addif veth_net veth99.99<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
net99 active<br />
veth_net active eth0 veth77.77,veth99.99<br />
<br />
(delete the old crap)<br />
vznetcfg net del net99<br />
<br />
<br />
Then, to add another device in<br />
<br />
v set 77 --netif_add eth77 --save<br />
ipdel 77 69.55.230.78<br />
v set 77 --ifname eth77 --ipadd 69.55.230.78 --save<br />
v set 77 --ifname eth77 --gateway 69.55.230.1 --save<br />
v set 77 --save --ifname eth77 --network veth_net<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
veth77.77 veth<br />
br0 bridge veth_net<br />
veth99.99 veth veth_net<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic veth_net 69.55.227.70/24<br />
<br />
vznetcfg net addif veth_net veth77.77<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
veth77.77 veth veth_net<br />
br0 bridge veth_net<br />
veth99.99 veth veth_net<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic veth_net 69.55.227.70/24<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
veth_net active eth0 veth77.77,veth99.99<br />
<br />
another example<br />
<br />
v set 1182 --netif_add eth1182 --save<br />
ipdel 1182 69.55.236.217<br />
v set 1182 --ifname eth1182 --ipadd 69.55.236.217 --save<br />
v set 1182 --ifname eth1182 --gateway 69.55.236.1 --save<br />
vznetcfg net addif veth_net veth1182.1182<br />
v set 1182 --save --ifname eth1182 --network veth_net<br />
<br />
<br />
unused/not working commands:<br />
ifconfig veth99.0 0<br />
vznetcfg net list<br />
vznetcfg br new br99 net99<br />
vznetcfg br attach br99 eth0<br />
vznetcfg br show<br />
vznetcfg if list<br />
vznetcfg net addif net99 veth99.99<br />
vznetcfg net addif eth0 net99<br />
<br />
---------<br />
<br />
vznetcfg br new br1182 net1182<br />
<br />
vznetcfg br attach br99 eth0<br />
vznetcfg if list<br />
vznetcfg net addif net99 veth99.99<br />
vznetcfg net addif eth0 net99<br />
<br />
<br />
<br />
vznetcfg net addif eth0 net1182<br />
<br />
<br />
# vznetcfg net addif veth99.0 net99<br />
--- 8< ---<br />
<br />
vznetcfg net new net<br />
# vznetcfg net addif eth0 net99<br />
<br />
<br />
# vzctl set 99 --save --netif_add eth0 (at this stage veth99.0 interface have to appear<br />
on node)<br />
# vzctl set 99 --save --ifname eth0 --ipadd 69.55.230.58 (and probably few more arguments<br />
here - see 'man vzctl')<br />
# vznetcfg net addif veth99.0 net99</pre><br />
<br />
<br />
== Assigning/remove ip from a VE ==<br />
<br />
1. Add or remove ips:<br />
ipdel 1234 1.1.1.1 2.2.2.2<br />
ipadd 1234 1.1.1.1 2.2.2.2<br />
<br />
2. update Mgmt screens<br />
<br />
3. offer to update any DNS we do for them<br />
<br />
4. check to see if we had rules for old IP in firwall<br />
<br />
== Enabling tun device for a ve ==<br />
Note, there’s a command for this: [[#addtun|addtun]]<br />
<br />
For posterity:<br />
Make sure the tun.o module is already loaded before Virtuozzo is started: <br />
lsmod <br />
Allow the VPS to use the TUN/TAP device: <br />
vzctl set 101 --devices c:10:200:rw --save <br />
Create the corresponding device inside the VPS and set the proper permissions: <br />
vzctl exec 101 mkdir -p /dev/net <br />
vzctl exec 101 mknod /dev/net/tun c 10 200 <br />
vzctl exec 101 chmod 600 /dev/net/tun<br />
<br />
== Remaking a system (on same virt) ==<br />
<br />
1. [[#cancelve|cancelve]] (or v destroy x - ONLY if you're POSITIVE no data needs to be saved)<br />
<br />
2. [[#vemake|vemake]] using same veid<br />
<br />
3. [[#mvbackups|mvbackups]] or [[#vb|vb]] (if new mount point)<br />
<br />
4. update mgmt with new dir/ip <br />
<br />
5. update firewall if changed ip<br />
<br />
== Re-initialize quota for a VE ==<br />
<br />
There’s a commamd for this now: [[#clearquota|clearquota]]<br />
<br />
For posterity:<br />
<br />
vzctl stop 1<br />
vzquota drop 1<br />
vzctl start 1<br />
<br />
<br />
== Traffic accounting on linux ==<br />
<br />
DEPRECATED - all tracking is done via bwdb now. This is how we used to track traffic.<br />
<br />
TODO: update for diff versions of vz<br />
<br />
Unlike FreeBSD, where we have to add firewall count rules to the system to count the traffic, on virtuozzo counts the traffic for us. You an see the current traffic stats by running `vznetstat`:<br />
<br />
<pre># vznetstat<br />
VEID Net.Class Output(bytes) Input(bytes)<br />
-----------------------------------------------<br />
24218 1 484M 39M<br />
24245 1 463M 143M<br />
2451 1 2224M 265M<br />
2454 1 2616M 385M<br />
4149 1 125M 68M<br />
418 1 1560M 34M<br />
472 1 1219M 315M<br />
726 1 628M 317M<br />
763 1 223M 82M<br />
771 1 4234M 437M<br />
-----------------------------------------------<br />
Total: 13780M 2090M</pre><br />
<br />
As you can see the VEID is on a line with the in and out bytes. So, we simply run a cron job:<br />
<br />
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /root/vztrafdump.sh<br />
<br />
Just like we do on FreeBSD - this one goes through all the VEs in /vz/private and greps the line from vznetstat that matches them and dumps it in /jc_traffic_dump on their system. Then it does it again for all the VEs in /vz1/private. It is important to note that vznetstat runs only once, and the grepping is done from a temporary file that contains that output - we do this because running vznetstat once for each VE that we read out of /vz/private and /vz1/private would take way too long and be too intensive.<br />
<br />
You do not need to do anything to facilitate this other than make sure that that cron job is running - the vznetstat counters are always running, and any new VEs that are added to the system will be accounted for automatically.<br />
<br />
Traffic resetting no longer works with vz 2.6, so we disable the vztrafdump.sh on those virts.<br />
<br />
== Watchdog script ==<br />
<br />
On some of the older virts, we have a watchdog running that kills procs that are deemed bad per the following:<br />
<br />
/root/watchdog from quar1<br />
<br />
<pre>if echo $line | awk '{print $(NF-3)}' | grep [5-9]...<br />
then<br />
# 50-90%<br />
if echo $line | awk '{print $(NF-1)}' | grep "...:.."<br />
then<br />
# running for > 99min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
if echo $line | awk '{print $(NF-1)}' | grep "....m"<br />
then<br />
# running for > 1000min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
fi<br />
<br />
if echo $line | awk '{print $(NF-3)}' | grep [1-9]...<br />
then<br />
# running for 10-90 percent<br />
if echo $line | awk '{print $NF}' | egrep 'cfusion|counter|vchkpw'<br />
then<br />
<br />
if echo $line | awk '{print $(NF-1)}' | grep "[2-9]:.."<br />
then<br />
# between 2-9min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
elif echo $line | awk '{print $(NF-1)}' | grep "[0-9][0-9]:.."<br />
then<br />
# up to 99min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
fi<br />
fi</pre><br />
<br />
<br />
== Misc Linux Items ==<br />
<br />
We are overselling hard drive space ... when you configure a linux system with a certain amount of disk space (the default is 4gigs) you do not actually use up 4gigs of space on the system. The diskspace setting for a user is simply a cap, and they only use up as much space on the actual disk drive as they are actually using.<br />
<br />
When you create a new linux system, even though there are some 300 RPMs or so installed, if you run `df -k` you will see that the entire 4gig partition is empty - no space is being used. This is because the files in their system are "magic symlinks" to the template for their OS that is in /vz/template - however, any changes to any of those files will "disconnect" them and they will immediately begin using space in their system. Further, any new files uploaded (even if those new files overwrite existing files) will take up space on the partition.<br />
<br />
<br />
if you see this:<br />
<br />
<pre>[root@virt8 root]# vzctl stop 160 ; vzctl start 160<br />
VE is not running<br />
Starting VE ...<br />
VE is unmounted<br />
VE is mounted<br />
Adding IP address(es): 69.55.226.83 69.55.226.84<br />
bash ERROR: Can't change file /etc/sysconfig/network<br />
Deleting IP address(es): 69.55.226.83 69.55.226.84<br />
VE is unmounted<br />
[root@virt8 root]#</pre><br />
<br />
it probably means they no longer have /bin/bash - copy one in for them<br />
<br />
ALSO: another possibility is that they have removed the `ed` RPM from their system - it needs to be reinstalled into their system. But since their system is down, this is tricky ...<br />
<br />
VE startup scripts used by 'vzctl' want package 'ed' to be available inside VE. So if package 'ed' will be enabled in OS template config and OS template itself VE #827 is based on - this error should be fixed.<br />
<br />
yes, it is possible to add RPM to VE while it not running.<br />
Try to do following:<br />
<br />
<pre># cd /vz/template/<OS_template_with_ed_package>/<br />
# vzctl mount 827<br />
# rpm -Uvh --root /vz/root/827 --veid 827 ed-0.2-25.i386.vz.rpm</pre><br />
<br />
Usually theres an error, but its ok<br />
<br />
Note: replace 'ed-0.2-25.i386.vz.rpm' in last command with actual<br />
version of 'ed' package you have.<br />
<br />
----<br />
<br />
So how do I know what template the user has ? cat their conf file and it is listed in there. For example, if the conf file has:<br />
<br />
<pre>[root@virt12 sbin]# vc 1103<br />
…snip…<br />
OSTEMPLATE="debian-3.0/20030822"<br />
TEMPLATES="mod_perl-deb30/20030707 mod_ssl-deb30/20030703 mysql-deb30/20030707 proftpd-deb30/20030703 webmin-deb30/20030823 "<br />
…</pre><br />
<br />
then they are on debian 3.0, all of their system RPMs are in /vz/template/debian-3.0, and they are using version 20030822 of that debian 3.0 template. Also, they’ve also got additional packages installed (mod_perl, mod_ssl, etc). Those are also found under /vz/template<br />
<br />
----<br />
<br />
Edits needed to run java:<br />
<br />
When we first created the VEs, the default setting for privvmpages was 93000:94000 ... which was high enough that most people never had problems ... however, you can;t run java or jdk or tomcat or anything java related with that setting. We have found that by setting privvmpages to 610000:615000 that java runs just fine. That is now the default setting. It is exceedingly rare that anyone needs it higher than that, although we have seen it once or twice.<br />
<br />
Any problems with java at all - the first thing you need to do is see if the failcnt has raised for privvmpages.<br />
<br />
<pre># vzctl start 160<br />
Starting VE ...<br />
vzquota : (error) Quota on syscall for 160: Device or resource busy<br />
Running vzquota on failed for VE 160 [3]</pre><br />
<br />
This is because my pwd is _in_ their private directory - you can't start it until you move out<br />
<br />
People seem to have trouble with php if they are clueless newbies. Here are two common problems/solutions:<br />
<br />
no... but i figured it out myself. problem was the php.ini file that came<br />
vanilla with the account was not configured to work with apache (the<br />
ENGINE directive was set to off).<br />
<br />
everything else seems fine now.<br />
<br />
----<br />
<br />
the problem was in the php.ini file. I noticed that is wasnt showing<br />
the code when it was in an html file so I looked at the php.ini file<br />
and had to change it so it recognized <? tags aswell as <?php tags.<br />
<br />
Also, make sure added to httpd.conf<br />
AddType application/x-httpd-php .php<br />
<br />
----<br />
<br />
You can set the time zone:<br />
<br />
You can change the timezone by doing this:<br />
<br />
ln -sf /usr/share/zoneinfo/<zone> /etc/localtime<br />
<br />
where <zone> is the zone you want in the /usr/share/zoneinfo/ directory.<br />
<br />
----<br />
<br />
Failing shm_open calls:<br />
<br />
first, please check if /dev/shm is mounted inside VE.<br />
'cat /proc/mounts' command should show something like this:<br />
tmpfs /dev/shm tmpfs rw 0 0<br />
<br />
If /dev/shm is not mounted you have 2 ways to solve issue:<br />
1. execute following command inside VE (doesn't require VE reboot):<br />
mount -t tmpfs none /dev/shm<br />
2. add following string to /etc/fstab inside VE and reboot it:<br />
tmpfs /dev/shm tmpfs defaults 0 0<br />
<br />
----<br />
<br />
You can have a mounted but not running ve<br />
Just:<br />
vzctl mount <veid><br />
<br />
----<br />
<br />
When a debian sys can’t get on the network, and you try:<br />
<br />
<pre>vzctl set 1046 --ipadd 69.55.227.117<br />
Adding IP address(es): 69.55.227.117<br />
Failed to bring up lo.<br />
Failed to bring up venet0.</pre><br />
<br />
They probably removed iproute package, which must be the one from swsoft. To restore:<br />
<pre># dpkg -i --veid=1046 --admindir=/vz1/private/1046/root/var/lib/dpkg --instdir=/vz1/private/1046/root/ /vz/template/debian-3.0/iproute_20010824-8_i386.vz.deb<br />
(Reading database ... 16007 files and directories currently installed.)<br />
Preparing to replace iproute 20010824-8 (using .../iproute_20010824-8_i386.vz.deb) ...<br />
Unpacking replacement iproute ...<br />
Setting up iproute (20010824-8) ...</pre><br />
<br />
Then restart their ve<br />
<br />
----<br />
<br />
in a ve i do:<br />
<br />
<pre>cd /<br />
du -h .</pre><br />
<br />
and get: 483M .<br />
<br />
i do:<br />
<br />
<pre>bash-2.05a# df -h<br />
Filesystem Size Used Avail Use% Mounted on<br />
/dev/vzfs 4.0G 2.3G 1.7G 56% /</pre><br />
<br />
how can this be?<br />
<br />
Is it possible that quota file was corrupted somehow? Please try to: <br />
<pre>vzctl stop <VEID><br />
vzquota drop <VEID><br />
vzquota init <VEID><br />
vzctl start <VEID></pre><br />
<br />
----<br />
<br />
How to stop vz from starting after reboot:<br />
<br />
VIRTUOZZO=no <br />
in <br />
/etc/sysconfig/vz<br />
<br />
To start: <br />
service vz start<br />
(after setting VIRTUOZZO=yes in /etc/sysconfig/vz)<br />
<br />
service vz restart will do some kind of 'soft reboot' -- restart all<br />
VPSes and reload modules without rebooting the node<br />
<br />
if you need to shut down all VPSes really really fast, run killall -9 init<br />
<br />
----<br />
<br />
Postfix tip:<br />
<br />
You may want to tweak settings: default_process_limit=10<br />
<br />
= Virtuozzo VPS Management Tools =<br />
<br />
== vm ==<br />
<br />
TODO<br />
<br />
== cancelve ==<br />
cancelve <veid><br />
this will:<br />
* stop a ve<br />
* check for backups (offer to remove them from the backup server <br />
and the backup.config)<br />
* rename the private dir<br />
* check for PTR, provide the commands to reset to default<br />
* and rename the ve’s config<br />
* remind you to remove firewall rules<br />
* remind you to remove DNS entries<br />
<br />
== ipadd ==<br />
ipadd <veid> <ip> [ip] [ip]<br />
add’s ip(s) to a ve<br />
<br />
== ipdel ==<br />
ipdel <veid> <ip> [ip] [ip]<br />
removes ip(s) from a ve<br />
<br />
== vc ==<br />
vc <veid><br />
the equivalent of: <tt>cat /vzconf/<veid>.conf</tt><br />
<br />
== vl ==<br />
vl<br />
will displays a list of ve #’s, 1 per line. (ostensibly to use in a for loop)<br />
<br />
== vp ==<br />
vp <veid><br />
the equivalent of: <tt>vzps auxww –E <veid></tt><br />
<br />
== vpe ==<br />
DEPRECATED<br />
vpe <veid> <br />
this will allow you to do a vp when a ve is running out of control, the equivalent of (deprecated since vp operates outside the VPS): <br />
<pre>vzctl set <veid> --kmemsize 2100000:2200000<br />
vzctl exec <veid> ps auxw<br />
vzctl set <veid> --kmemsize (ve’s orig lvalue):(ve’s orig hvalue)</pre><br />
<br />
== vt ==<br />
vt <veid><br />
the equivalent of: <tt>vztop –E <veid></tt><br />
<br />
== vr ==<br />
vr <veid><br />
the equivalent of: <tt>vzctl stop <veid>; vzctl start <veid></tt><br />
You can run this even if the ve is down - the stop command will just fail<br />
<br />
== vs ==<br />
vs [veid]<br />
displays status (output of <tt>vzctl status <veid></tt>) on each ve configured on the system (as determined by <tt>/vzconf/*.conf</tt>)<br />
If passed an argument, gives the status for just that ve. <br />
A running system looks like:<br />
<tt>VEID 16066 exist mounted running</tt><br />
<br />
A ve which is not running (but does exist) looks like:<br />
<tt>VEID 9990 exist unmounted down</tt><br />
<br />
A ve which is not running and doesn’t exist looks like:<br />
<tt>VEID 421 deleted unmounted down</tt><br />
<br />
== vs2 ==<br />
vs2 [veid]<br />
this is similar to vs in that it displays status (output of <tt>vzctl status <veid></tt>) on each ve,<br />
but the difference is it’s list comes from doing an ls on the data dirs. This was meant to catch <br />
the rare case where a ve configured but exists. <br />
<br />
== vw ==<br />
vw [veid]<br />
displays the output of ‘<tt>w</tt>’ (the equivalent of <tt>vzctl exec <veid> w</tt>) for each configured ve (as determined by <tt>/vzconf/*.conf</tt>). Useful for determing which ve is contributing to a heavily-loaded system.<br />
If passed an argument, gives ‘<tt>w</tt>‘ output for just that ve. <br />
Ex:<br />
<pre>[root@virt2 etc]# vw<br />
134<br />
10:52pm up 79 days, 6:14, 0 users, load average: 0.02, 0.02, 0.00<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT<br />
16027<br />
2:52pm up 7 days, 19:54, 0 users, load average: 0.00, 0.00, 0.00<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT<br />
16055<br />
2:52pm up 79 days, 6:38, 0 users, load average: 0.00, 0.04, 0.07<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT</pre><br />
<br />
== vwe ==<br />
vwe [constraint]<br />
just like <tt>vw</tt>, but takes a constraint as an argument, only show’s ve’s with loads >= the constraint provided. If no constraint is provided, 1 is used by default<br />
<br />
== vzs ==<br />
vzs [veid]<br />
displays the beancounter status for all ve’s, or a particular ve if an argument is passed<br />
<br />
== ve ==<br />
ve <veid><br />
the equivalent of: <tt>vzctl enter <veid></tt><br />
<br />
== vx ==<br />
vx <veid> <command><br />
the equivalent of: <tt>/usr/sbin/vzctl exec <veid> <command></tt><br />
<br />
== dprocs ==<br />
dprocs [count]<br />
a script which outputs a continuous report (or a certain number of reports if an option is passed) of processes stuck in the D state and which VPS’s those procs belong to.<br />
<br />
== setmem ==<br />
setmem <VEID> <256|512|768|1024|2048><br />
adjusts the memory resources for the VE<br />
<br />
== afacheck.sh ==<br />
afacheck.sh<br />
displays the health/status of containers and mirrors on an adaptec card (currently quar1, tempvirt1-2, virt9, virt10)- all other are LSI<br />
<br />
== backup ==<br />
backup<br />
backup script called nightly to update virt scripts and do customer backups<br />
<br />
== checkload.pl ==<br />
checkload.pl<br />
this was intended to be setup as a cronjob to watch processes on a virt when the load <br />
rises above a certain level. Not currently in use.<br />
<br />
== findbackuppigs.pl ==<br />
findbackuppigs.pl<br />
looks for files larger than 50MB which customers have asked us to backup. Emails matches<br />
to linux@johncompanies.com<br />
<br />
== gatherlinux.pl ==<br />
gatherlinux.pl<br />
gathers up data about ve’s configured and writes to a file. Used for audits against the db<br />
<br />
== gb ==<br />
gb <search><br />
greps backup.config for the given search parameter<br />
<br />
== gbg ==<br />
gbg <search><br />
greps backup.config for the given search parameter and presents just the directories (for clean pasting)<br />
<br />
== linuxtrafficgather.pl ==<br />
linuxtrafficgather.pl [yy-mm]<br />
sends a traffic usage summary by ve to support@johncomapnies.com and payments@johncompanies.com.<br />
Optional arguments are year and month (must be in the past). If not passed, assumes last month. Relies on <br />
traffic logs created by netstatreset and netstatbackup<br />
<br />
== linuxtrafficwatch.pl ==<br />
linuxtrafficwatch.pl<br />
checks traffic usage and emails support@johncomapnies.com when a ve reaches the warning level (35G) and<br />
the limit (40G). Works on virtuozzo versions <= 2.6.x. We really aren’t using this anymore now that we have netflow.<br />
<br />
== linuxtrafficwatch2.pl ==<br />
linuxtrafficwatch2.pl<br />
checks traffic usage and emails support@johncomapnies.com when a ve reaches the warning level (35G) and<br />
the limit (40G). Works on virtuozzo version 2.6.x. We really aren’t using this anymore now that we have netflow.<br />
<br />
== load.pl ==<br />
load.pl<br />
feeds info to load mrtg - executed by inetd.<br />
<br />
== mb (linux) ==<br />
mb <mount|umount><br />
(nfs) mounts and umounts dirs to backup2. Shortcuts are mbm and mbu to mount and unmount. <br />
<br />
== migrate ==<br />
migrate <ip of node migrating to> <veid> <target_veid> [target dir: vz | vz1 | vz2]<br />
this is basically a wrapper for <tt>vzmigrate</tt> – vzmigrate is a util to seamlessly move a ve from one host to another. This wrapper was written cause vz virtuozzo version 2.6 had a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables. This script mitigates that. Since it makes multiple ssh connections to the target host, it’s a good idea to put the pub key for the src system in the authorized_keys file on the target host. In addition, it emails ve owners when their migration starts and stops (if they place email addresses in a file on their system: /migrate_notify). To move everyone off a system, you’d do:<br />
for f in `vl`; do migrate <ip> $f; done<br />
<br />
== migrateonline ==<br />
migrateonline <ip of node migrating to> <veid> <target_veid> [target dir: vz | vz1 | vz2]<br />
this is the same as migrate but will migrate a ve in <tt>–online</tt> mode which means it won’t be shut down at the end of the migration. This only works when migrating ve’s between 2 machines running a 2.6 kernel (currently tempvirt1-2. virt16-19, virt12). If you get an error that the machine you’re trying to migrate to has a different CPU or features, etc, then you have to edit the file and add the –f switch to the vzmigrate line- you can basically ignore this kind of warning (but never ignore a warning about missing templates on the destination node). NOTE: This edit (if made to migrateonline) will be overwritten by the base script during each night’s backup.<br />
<br />
== netstatbackup ==<br />
DEPRECATED<br />
netstatbackup <br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.5.x. <br />
<br />
== netstatbackup2 ==<br />
DEPRECATED<br />
netstatbackup2<br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.6.x. <br />
<br />
== netstatreset ==<br />
DEPRECATED<br />
netstatreset<br />
writes traffic count data to a logfile and resets counters to 0. Works on virtuozzo versions 2.5.x <br />
<br />
== netstatreset2 ==<br />
DEPRECATED<br />
netstatreset2<br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.6.x. <br />
<br />
== orphanedbackupwatchlinux ==<br />
orphanedbackupwatchlinux <br />
looks for directories on backup2 which aren’t configured in backup.config and offers to <br />
delete them<br />
<br />
== rsync.backup (linux) ==<br />
rsync.backup<br />
does customer backups (relies on backup.config)<br />
<br />
== startvirt.pl ==<br />
startvirt.pl<br />
forks off start ve commands – keeps 6 running at a time. This is not to be used on systems where fastboot is enabled as it circumvents the benefit of the fastboot. The script will occasionally not exit gracefully and will continue to use up CPU, so it should be watched. Also, don’t exit from the script till you’re sure all ve’s are started – if you do you need to start them manually and may have to free up locks. On some systems, the startvirt script doesn’t exit cleanly and you have to ^C out of it. Be careful though- doing so can leave some VE’s in an odd bootup state and you may need to ‘vr’ them manually. You should check to see which ve’s aren’t running and/or confirm all have started when ^C’ing out of startvirt.<br />
<br />
== taskdone (linux) ==<br />
taskdone<br />
when called will email support@johncompanies.com with the hostname of the machine from which it was <br />
executed as the subject<br />
<br />
== vb (linux) ==<br />
vb<br />
the equivalent of: <tt>vi /usr/local/sbin/backup.config</tt><br />
<br />
== vemakeXX ==<br />
DEPRECATED<br />
vemakerh9 <br />
ve create script for RH9 (see vemake)<br />
<br />
vemakedebian30 <br />
ve create script for debian 3.0 (Woody) (see vemake)<br />
<br />
vemakedebian31 <br />
ve create script for debian 3.1 (Sarge) (see vemake)<br />
<br />
vemakedebian40 <br />
ve create script for debian 4.0 (Etch) (see vemake)<br />
<br />
vemakefedora, vemakefedora2, vemakefedora4, vemakefedora5, vemakefedora6, vemakefedora7<br />
ve create script for fedora core 1, 2, 4, 5, 6, 7 respectively (see vemake)<br />
<br />
vemakecentos3, vemakecentos4<br />
ve create script for centos 3, 4 respectively (see vemake)<br />
<br />
vemakesuse, vemakesuse93, vemakesuse100<br />
ve create script for suse 9.2, 9.3, 10.0 respectively (see vemake)<br />
<br />
vemakeubuntu5, vemakeubuntu606, vemakeubuntu606 vemakeubuntu704<br />
ve create script for ubuntu 5.10, 6.06, 6.10, 7.04 respectively (see vemake)<br />
<br />
== vemove ==<br />
DEPRECATED<br />
vemove <veid> <target_ip> </vz/private/123><br />
this script simplifies the old way of moving ve’s from one system to another - in short moving a ve to or from a virt running virtuozzo < 2.6.x<br />
It’s the equivalent of:<br />
<tt>tar cfpP - <veid> --ignore-failed-read | (ssh -2 -c arcfour <target_ip> "split - -b 1024m </vz/private/123>.tar" )</tt>This should only be used if migrate/vzmigrate can’t be used. <br />
<br />
== vim.watchdog ==<br />
vim.watchdog <br />
looks for and kills procs matching “vi|vim|nano|pine|elm” that are running for a long time & consuming lots of cpu. Works on virtuozzo versions 2.5.x<br />
<br />
== vim.watchdog2 ==<br />
vim.watchdog2<br />
looks for and kills procs matching “vi|vim|nano|pine|elm” that are running for a long time & consuming lots of cpu.<br />
Works on virtuozzo versions 2.6.x.<br />
<br />
== vzmigrate ==<br />
vzmigrate <target_ip> -r no <veid>:[dst veid]:[dst /vzX/private/veid]:[dst /vzX/root/veid]<br />
(this is the raw command “wrapped” by migrate/migrateonline) this will seamlessly move a ve from one host to another. The ve will run for the duration of the migration till the very end when it’s shut down, ip moved and started up on the target system. The filesystem on the src will remain. This should be watched – occasionally the move will timeout and leave the system shut down. If target private and root aren’t specified it just puts it in /vz. Only works when both systems are running virtuozzo 2.6.x<br />
<br />
== vztrafdump.sh ==<br />
DEPRECATED<br />
vztrafdump.sh<br />
writes traffic usage info by ve to a file called jc_traffic_dump in each ve’s / dir. Works on virtuozzo versions <= 2.5.x. <br />
<br />
== vztrafdump2.sh ==<br />
DEPRECATED<br />
vztrafdump2.sh<br />
writes traffic usage info by ve to a file called jc_traffic_dump in each ve’s / dir. Works on virtuozzo versions 2.6.x. <br />
<br />
== addtun ==<br />
addtun <veid><br />
Add’s tun device to ve.<br />
<br />
== bwcap ==<br />
bwcap <veid> <kbps><br />
Ex: <tt>bwcap 1234 512</tt><br />
Caps a VE’s bandwidth to the amount given<br />
<br />
== setdisk ==<br />
setdisk <veid> <diskspace in GB><br />
Ex: <tt>setdisk 1234 5</tt><br />
Gives a VE’s a given amount of disk space<br />
<br />
== vdf ==<br />
vdf <veid> <br />
the equivalent of: <tt>vzctl exec <veid> df –h</tt><br />
<br />
== vdff ==<br />
vdff<br />
runs a (condensed) vdf for all ve’s in your pwd (must be run from /vz/privateN)<br />
<br />
== mvbackups ==<br />
mvbackups <veid> <target_machine> (virt1) <target_dir> (vz1)<br />
moves backups from one location to another on the backup server, and provides you with option to remove entries from current backup.config, and simple paste command to add the config to backup.config on the target server<br />
<br />
== checkquota ==<br />
checkquota<br />
for all the ve’s in the cwd (run from /vz/private, /vz1/private, etc) reports what vz quota says they’re using and what the actual usage is (as reported by du)<br />
<br />
== clearquota ==<br />
clearquota <veid><br />
Recalculates a ve’s quota, prints out the usage before and after. The equivalent of:<br />
<tt>vdf <veid>; v stop <veid>; vzquota drop <veid>; v start <veid>; vdf <veid></tt><br />
<br />
== dprocs ==<br />
dprocs<br />
Sometimes the server’s have a large number of processes get stuck in the D state- this script shows (every 3 secs) which VE’s have D procs, which procs<br />
are stuck and a running average of the top “offenders”<br />
<br />
== vzstat ==<br />
vstat<br />
sort of like top for VZ. sort VEs by CPU usage by pressing 'o' and then 'c' keys<br />
<br />
== stopvirt ==<br />
stopvirt<br />
will stop VEs as fast as it can, 6 at a time. May not exit when complete so you should watch [[#vzstat|vzstat]] in another window.</div>Supporthttps://wiki.jcihosting.com/index.php?title=Virtuozzo_/_Linux_Reference&diff=1031Virtuozzo / Linux Reference2013-03-11T23:43:01Z<p>Support: /* Moving virt (drives) to new hardware */</p>
<hr />
<div>= Updating VZ =<br />
<br />
Update Virtuozzo to version 4.0.0<br />
Apply minor updates for Virtuozzo 3.0.0<br />
<br />
...<br />
When you get to the last screen, you will be given an option to <br />
<br />
( ) Automatically reboot after update<br />
(*) Reboot later manually<br />
<br />
make sure you choose to reboot later.<br />
<br />
= Migrating Templates =<br />
One nice feature VZ offers is the ability to run a virtual environment (VE or CT as it's now called) based on a particular ditribution on a host which may or may not share the same distribution. For instance, you could run a CT running Ubuntu while the host machine (Hardware Node or HN) is running CentOS. This is accomplished via the use of OS templates. As long as the HN contains the OS template on which a particular CT relies, it will run there. As such, if you want to move a CT from one HN to another, you will need to make sure the OS template exists there.<br />
<br />
Modern versions of VZ (>= 4.x) will check for and automatically move the OS template over to the host as you're (vz)migrating the CT over to a new HN. However we like to make sure the template is in place prior to moving the CT. <br />
<br />
The first step is to see if the template exists on the host. To see OS templates installed:<br />
<br />
2.x (shows OS and application templates):<br />
<pre># vzpkgls<br />
mod_ssl-rh9 20040624<br />
mysql-rh9 20030814 20050412<br />
openwebmail-rh9 20050427<br />
php-rh9 20050506<br />
phpBB-rh9 20041229<br />
postgresql-rh9 20050719<br />
sl-webalizer-rh9 20040119<br />
spamassassin-rh9 20040127<br />
tomcat-rh9 20040524<br />
usermin-rh9 20040116</pre><br />
<br />
>= 3.x (shows just OS templates, run without -O to see application as well):<br />
<pre># vzpkg list -O<br />
ubuntu-10.04-x86 2010-06-01 11:39:05<br />
ubuntu-11.04-x86 2011-06-22 14:23:59<br />
ubuntu-9.10-x86 2010-03-11 17:39:51<br />
centos-5-x86 2010-03-11 17:12:27<br />
debian-5.0-x86 2010-03-11 17:33:34</pre><br />
<br />
<br />
All template files are located:<br />
<br />
<pre># ls /vz/template/<br />
ColdFusion-fc1 jre-fc1 openwebmail-fc1 sl-webalizer-fc1<br />
ColdFusion-fc2 jre-fc2 openwebmail-fc2 sl-webalizer-fc2<br />
ColdFusion-rh9 jre-rh9 openwebmail-rh9 sl-webalizer-rh9<br />
PostNuke-fc1 jre-suse92 openwebmail-suse92 spamassassin-fc1<br />
PostNuke-fc2 jrun php-deb31 spamassassin-fc2<br />
PostNuke-rh9 mailman-deb31 php-fc1 spamassassin-rh9<br />
analog-fc1 mailman-fc1 php-fc2 spamassassin-suse92<br />
analog-fc2 mailman-fc2 php-rh9 suse-9.2<br />
analog-rh9 mailman-rh9 php-suse92 tomcat-fc1<br />
awstats-fc1 mailman-suse92 phpBB-fc1 tomcat-fc2<br />
awstats-rh9 mod_frontpage-fc1 phpBB-fc2 tomcat-rh9<br />
bbClone-fc1 mod_frontpage-fc2 phpBB-rh9 tomcat-suse92<br />
bbClone-fc2 mod_frontpage-rh9 phpmyadmin-deb31 urchin-fc1<br />
bbClone-rh9 mod_frontpage-suse92 postgresql-deb31 usermin-fc1<br />
conf mod_perl-deb31 postgresql-fc1 usermin-fc2<br />
debian-3.1 mod_perl-fc1 postgresql-fc2 usermin-rh9<br />
devel-deb31 mod_perl-fc2 postgresql-rh9 uw-imap-fc1<br />
devel-fc2 mod_perl-rh9 postgresql-suse92 uw-imap-fc2<br />
devel-suse92 mod_perl-suse92 proftpd-deb31 uw-imap-rh9<br />
fedora-core-1 mod_ssl-fc1 proftpd-fc1 vzredhat-7.3<br />
fedora-core-2 mod_ssl-fc2 proftpd-fc2 vzredhat-8.0<br />
jdk-deb31 mod_ssl-rh9 proftpd-rh9 webalizer-suse92<br />
jdk-fc1 mysql-deb31 proftpd-suse92 webmin-fc1<br />
jdk-fc2 mysql-fc1 redhat-7.2 webmin-fc2<br />
jdk-rh9 mysql-fc2 redhat-9 webmin-rh9<br />
jdk-suse92 mysql-rh9 redhat-as3-minimal<br />
jre-deb31 mysql-suse92 redhat-devel-9<br />
</pre><br />
<br />
What we see here are the base OS templates: <tt>redhat-9, fedora-core-1</tt> and supporting application templates: <tt>postgresql-rh9, mailman-rh9, jdk-fc1, mod_ssl-fc1</tt>. So if you wanted to copy over all of redhat 9 you'd need to copy the contents of:<br />
<pre># ls -d /vz/template/*rh9*<br />
/vz/template/ColdFusion-rh9 /vz/template/mod_frontpage-rh9 /vz/template/proftpd-rh9<br />
/vz/template/PostNuke-rh9 /vz/template/mod_perl-rh9 /vz/template/sl-webalizer-rh9<br />
/vz/template/analog-rh9 /vz/template/mod_ssl-rh9 /vz/template/spamassassin-rh9<br />
/vz/template/awstats-rh9 /vz/template/mysql-rh9 /vz/template/tomcat-rh9<br />
/vz/template/bbClone-rh9 /vz/template/openwebmail-rh9 /vz/template/usermin-rh9<br />
/vz/template/jdk-rh9 /vz/template/php-rh9 /vz/template/uw-imap-rh9<br />
/vz/template/jre-rh9 /vz/template/phpBB-rh9 /vz/template/webmin-rh9<br />
/vz/template/mailman-rh9 /vz/template/postgresql-rh9<br />
<br />
# ls -d /vz/template/*redhat-9*<br />
/vz/template/redhat-9<br />
</pre><br />
<br />
To get a template from one HN to another, it simply needs to be rsync'd over to the target HN. It's rsync'd over in this fashion:<br />
<br />
rsync -av -e ssh /vz/template/redhat-9 root@10.1.4.65:/vz/template<br />
rsync -av -e ssh /vz/template/*rh9 root@10.1.4.65:/vz/template<br />
<br />
Newer (>= 3.x) VZ employs "EZ" templates which are organized a little differently:<br />
<br />
<pre># ls /vz/template/<br />
cache CmdTool.log debian redhat-as3-minimal-20080630.tar.gz vc<br />
centos conf redhat-as3-minimal ubuntu<br />
<br />
# ls /vz/template/ubuntu/<br />
10.04 11.04 9.10<br />
# ls /vz/template/ubuntu/10.04/<br />
x86<br />
# ls /vz/template/ubuntu/10.04/x86/<br />
aap_1.091-1_all<br />
aap-doc_1.091-1_all<br />
adduser_3.112ubuntu1_all<br />
anacron_2.3-13.1ubuntu11_i386<br />
apache2_2.2.14-5ubuntu8.2_i386<br />
apache2_2.2.14-5ubuntu8.4_i386<br />
apache2_2.2.14-5ubuntu8.6_i386<br />
apache2_2.2.14-5ubuntu8.7_i386<br />
apache2_2.2.14-5ubuntu8.8_i386<br />
apache2_2.2.14-5ubuntu8.9_i386<br />
<...snip><br />
<br />
# ls /vz/template/cache/<br />
centos-5-x86.tar.gz suse-11.3-x86.tar.gz ubuntu-9.10-x86.tar.gz<br />
debian-5.0-x86.tar.gz ubuntu-10.04-x86.tar.gz<br />
fedora-core-14-x86.tar.gz ubuntu-11.04-x86.tar.gz<br />
</pre><br />
<br />
So what we see is the entire OS and all applications are in 1 directory, and organized by distribution and release version. So to move Ubuntu 10.04:<br />
<br />
rsync -av -e ssh /vz/template/ubuntu/10.04 root@10.1.4.69:/vz/template/ubuntu<br />
<br />
and you also need to move the cache:<br />
<br />
rsync -av -e ssh /vz/template/cache/ubuntu-10.04-x86.tar.gz root@10.1.4.69:/vz/template/cache/<br />
<br />
Once the transfer is complete, you will be able to run <tt>vzpkgls</tt> or <tt>vzpkg list -O</tt> and see the template(s) listed on the target HN.<br />
<br />
So far, this all supposes template doesn't exist on the target- very simple, just copy it over. If the template exists already on the target HN, this requires closer inspection. With older templates, simply moving over the templates would be the end of it- you see the version listed when you do a <tt>vzplgls</tt>:<br />
<br />
mysql-rh9 20030814 20050412<br />
<br />
this means there are 2 versions of the mysql package for RH9: 20030814 and 20050412<br />
As an aside, you can't copy over just 1 of them, but if you rsync'd over the <tt>mysql-rh9</tt> directory, you'd see 20030814 and 20050412 on the target HN. As long as the versions match up, you're good to go.<br />
<br />
With EZ templates, the versions are not as easy to decipher. When you look at the <tt>vzpkg list</tt> output, that simply tells you when a cache was created. A cache is simply a snapshot of all the data needed for the latest versions of the OS and it's applications at the time the cache was created. It is a date, and thus tells you nothing about the versions within. So for instance, if you had a cache for Ubuntu 10.04 from 2007 and you ran <tt>vzpkg create cache ubuntu-10.04-x86</tt> it would re-download the latest version of apache, mysql and so on. And any ''subsequent'' ubuntu 10.04 CT's created thereafter would use those newer versions, whereas older CT's based on 10.04 would use older templates. You can see how this works by looking at the files under:<br />
<br />
<pre># ls /vz/template/ubuntu/10.04/x86/<br />
aap_1.091-1_all<br />
aap-doc_1.091-1_all<br />
adduser_3.112ubuntu1_all<br />
anacron_2.3-13.1ubuntu11_i386<br />
apache2_2.2.14-5ubuntu8.2_i386<br />
apache2_2.2.14-5ubuntu8.4_i386<br />
apache2_2.2.14-5ubuntu8.6_i386<br />
apache2_2.2.14-5ubuntu8.7_i386<br />
apache2_2.2.14-5ubuntu8.8_i386<br />
apache2_2.2.14-5ubuntu8.9_i386<br />
<...snip><br />
</pre><br />
<br />
You can see that this template has in it 6 different versions of apache2. This means that this template was merged with other 10.04 templates and/or it has been cached a few times, each time a new apache was available. The unfortunate thing is it's almost impossible to know which CT's are using which version of apache so it's hard to try to prune this down and remove unwanted/unneeded applications.<br />
<br />
So, if you see another HN has Ubuntu 10.04 on it, you need to see/confirm that it has all the exact files your CT needs. You can run a test rsync to see what ''would'' be transferred (what's missing):<br />
<br />
rsync -avn --stats -e ssh /vz/template/ubuntu/10.04 root@10.1.4.69:/vz/template/ubuntu<br />
<br />
Based on the amount of data you get back from the rsync, you will be able to decide whether or not to remove the -n and allow the full transfer to go through. The downside to doing the transfer is the situation described above- you've permanently joined these templates and now you may have 10 versions of apache on the target HN. There's one other potential pitfall to this kind of merging- it will also potentially copy over shared files, for which there is no particular version, most of which are in <tt>/vz/template/ubuntu/10.04/x86/config</tt>: <br />
<br />
cat /vz/template/ubuntu/10.04/x86/config/os/default/repositories<br />
/vz/template/ubuntu/10.04/x86/config/os/default/repositories<br />
<br />
Here are some specific things to look out for. Case: copying centos-5 on virt19 to virt12. We dumped the rsync output to a file so we could inspect:<br />
<br />
[root@virt19 /vz/template]# rsync -an -e ssh /vz/template/centos/ root@10.1.4.62:/vz/template/centos/ > v12_c5<br />
<br />
(note, that can be dangerous, better to add on full path <tt>/vz/template/centos/5</tt>)<br />
<br />
So in the output file we see things like:<br />
<br />
<pre><br />
x86/base0/cachecookie<br />
x86/base0/primary.xml.gz <br />
x86/base0/primary.xml.gz.sqlite <br />
x86/base0/repomd.xml <br />
x86/base1/cachecookie <br />
x86/base1/filelists.xml.gz <br />
x86/base1/filelists.xml.gz.sqlite <br />
x86/base1/primary.xml.gz <br />
x86/base1/primary.xml.gz.sqlite <br />
x86/base1/repomd.xml<br />
x86/base2/cachecookie<br />
</pre><br />
<br />
Which make us nervous cause those are not versioned files, i.e.:<br />
5/x86/MAKEDEV-3.23-1.2.i386/usr/sbin/mksock<br />
<br />
So those files will replace existing files on v12 rather than the case of the latter where it's likely being added. So caution should be given and deference to whichever version is newer (which rsync should take care of, but that will depend on the date each file was created and perhaps not the actual data within).<br />
<br />
If we look further in the file we see:<br />
<br />
x86/psa-appvault-moodle-1.8-8202920080409011254.noarch/usr/local/psa/var/cgitory/moodle-1.<br />
9/htdocs/lib/editor/htmlarea/plugins/TableOperations/img/cell-split.gif<br />
<br />
and other files looking like:<br />
5/x86/plesk-base-9.0.1-cos5.build90090127.18.i586/etc/sw/keys/info<br />
<br />
Which means plesk is here. We don't need plesk on v12 (the CT moving over doesn't use it) so we rerun the rsync and exclude it:<br />
<br />
[root@virt19 /vz/template]# rsync -an -e ssh --exclude="plesk*" --exclude="psa*" /vz/template/centos/ root@10.1.4.62:/vz/template/centos/ > v12_c5<br />
<br />
and now it's much smaller:<br />
<br />
[root@virt19 /vz/template]# cat v12_c5|wc -l<br />
97104<br />
<br />
Before running with excludes it was:<br />
[root@virt19 /vz/template]# cat v12_c5|wc -l <br />
238238<br />
<br />
So again, look at what you're doing. Generally, combining templates is fine however.<br />
<br />
If you're happy with the merge, run the rsync again without the "-n" to let it actually do the transfer.<br />
<br />
Once done, don't forget to add the new template to the HN in Management -> Reference -> Templates<br />
<br />
= getting help =<br />
<br />
= vzup2date =<br />
TODO <br />
= Adding new OS/application templates =<br />
<br />
Virtuozzo will lag a bit behind the initial release of an OS, perhaps by a month or more. Further, a newly-released OS may be available, but there may not be many/any application templates available initially. It pays to wait a bit.<br />
<br />
A template is easily installed via [[#vzup2date|vzup2date]]:<br />
<br />
vzup2date -z<br />
<br />
You will be given a list of OS's to install. Select an OS, then customize that selection by checking/unchecking the application templates we do/don't want to include/offer.<br />
<br />
Generally, here are the app templates we include/offer:<br />
<pre>cyrus-imap<br />
devel<br />
imp<br />
jre<br />
jsdk<br />
mailman<br />
mod_perl<br />
mod_ssl<br />
mysql<br />
php<br />
phpmyadmin<br />
phppgadmin<br />
postgresql<br />
proftpd<br />
spamassassin<br />
squirrelmail<br />
tomcat<br />
vsftpd<br />
</pre><br />
<br />
We do not (any longer) offer webalizer or analog.<br />
<br />
After selecting and installing the OS and apps, you should ensure it's installed:<br />
<br />
vzpkg list -O<br />
<br />
Your new OS (ex. fedora-core-13-x86_64) will be listed, without a corresponding cache date:<br />
<pre><br />
[root@virt13 /vzconf]# vzpkg list -O<br />
centos-6-x86 2011-07-22 13:24:41<br />
centos-6-x86_64 2012-03-09 20:42:22<br />
centos-5-x86 2009-07-27 16:58:24<br />
centos-5-x86_64 2012-03-09 22:38:53<br />
ubuntu-11.10-x86_64 2012-03-01 23:13:33<br />
ubuntu-9.04-x86 2009-06-02 11:32:39<br />
ubuntu-12.04-x86_64 2012-05-29 13:50:50<br />
ubuntu-8.04-x86 2008-09-17 21:27:20<br />
ubuntu-8.10-x86 2009-03-13 13:58:57<br />
ubuntu-11.04-x86 2011-12-05 11:15:46<br />
ubuntu-7.04-x86 2008-09-24 16:42:50<br />
redhat-el6-x86_64<br />
fedora-core-13-x86_64 <br />
fedora-core-12-x86 2010-02-06 13:24:32<br />
fedora-core-15-x86 2011-12-05 11:36:43<br />
fedora-core-11-x86 2009-10-01 16:30:59<br />
fedora-core-14-x86<br />
fedora-core-16-x86_64 2012-03-13 11:40:23<br />
fedora-core-9-x86 2008-11-29 10:52:18<br />
debian-6.0-x86 2011-07-29 16:00:35<br />
debian-6.0-x86_64 2012-03-12 19:53:33<br />
debian-5.0-x86 2009-03-12 12:39:11<br />
</pre><br />
<br />
You can also confirm the apps installed:<br />
<pre>[root@virt13 /vzconf]# vzpkg list fedora-core-13-x86_64<br />
fedora-core-13-x86_64 2013-03-08 11:39:44<br />
fedora-core-13-x86_64 devel<br />
fedora-core-13-x86_64 php<br />
fedora-core-13-x86_64 proftpd<br />
fedora-core-13-x86_64 postgresql<br />
fedora-core-13-x86_64 phpmyadmin<br />
fedora-core-13-x86_64 mysql<br />
</pre><br />
<br />
Before you can use the OS you must cache it. To create the cache run:<br />
vzpkg cache [OS]<br />
<br />
ex: <tt>vzpkg cache fedora-core-13-x86_64</tt><br />
<br />
Now it's ready to use. In order to be able to use it with the [[VPS_Management#vm|vm]] command, you must create a file:<br />
jctmpl-[OS]<br />
<br />
ex: <tt>jctmpl-fedora-core-13-x86_64</tt><br />
<br />
In it, on line 1 we place the OS, followed by the app names, ex:<br />
<pre>more jctmpl-fedora-core-13-x86_64<br />
fedora-core-13-x86_64<br />
postgresql<br />
jsdk<br />
mod_ssl<br />
phpmyadmin<br />
mod_perl<br />
tomcat<br />
devel<br />
php<br />
mailman<br />
cyrus-imap<br />
squirrelmail<br />
proftpd<br />
mysql<br />
phppgadmin<br />
spamassassin<br />
</pre><br />
<br />
The packages will be installed in that order, so any pre-req's should be handled there.<br />
<br />
Now, when you run [[VPS_Management#vm|vm]] it will show you the newly-added OS and you may create a CT using it.<br />
<br />
The last few tasks are related to mgmt:<br />
<br />
1. add to mgmt->reference->templates (use the existing templates as an example). Once added, the new template will be included with the list of OS's available for the given HN.<br />
2. if this is going to be a new OS offered for ordering, it needs to be added to the order form and webpage.<br />
<br />
a. to get a list of applications and versions in the new OS, you should create a test CT, enter it and run <tt>dpkg -l|sort</tt> (or <tt>rpm -aq|sort</tt> in centos/fedora) and enter that list in the following places:<br><br />
<tt>/usr/local/www/jc_pub/[osname].html</tt><br><br />
<tt>/usr/local/www/signup/html/[osname].html</tt><br><br />
Note, you will have to move the most recent OS data from <tt>/usr/local/www/signup/html/[osname].html</tt> into the corresponding <tt>/usr/local/www/signup/html/[osname]_old.html</tt> updating the version links in both.<br />
<br />
b. to add the new OS to the order form, you will need to update <tt>/usr/local/www/signup/html/step1.html</tt> in several places (for regular and OSS orders) as well as updating the templateID which you can get from mgmt->reference->templates (or [[Management_System_/_Public_Website_/_Signup_/_Account_Manager#jc:ref_templates|jc:ref_templates]])<br />
<br />
<br />
<br />
Old method to obtain the templates:<br />
<br />
Go to http://www.parallels.com/en/virtuozzo/templates/catalog/<br />
<br />
Download the RPM's and install them<br />
<br />
vzpkg list<br />
to see the new packages/os templates<br />
<br />
vzpkg create cache "OS_EZ_template_name"<br />
to create a cache<br />
<br />
<br />
= Update OS template =<br />
<br />
Installation and update of the new version of OS template will not affect the existing containers. It will affect only the new containers, created after this update operation. In terms of fixing "vzctl enter" error and SSH connectivity to Ubuntu based containers, it is necessary to recreate the template and not to update it, so the commands to run are:<br />
<br />
<pre>rpm -Uhv ubuntu-8.04-x86-ez-4.0.0-9.swsoft.noarch.rpm<br />
vzpkg clean ubuntu-8.04-x86<br />
vzpkg remove cache ubuntu-8.04-x86<br />
vzpkg create cache ubuntu-8.04-x86</pre><br />
<br />
<br />
= Installing new SSL cert on VZCP =<br />
<br />
<pre>openssl req -days 1999 -new -x509 -nodes -out server.crt -keyout server.key<br />
<br />
US<br />
CA<br />
San Diego<br />
johncompanies.com<br />
johncompanies.com<br />
virt15ohncompanies.com<br />
linux@johncompanies.com<br />
<br />
cp -p /etc/httpd/conf/ssl.crt/server.crt /etc/httpd/conf/ssl.crt/server.crt.bak<br />
cp -p /etc/httpd/conf/ssl.key/server.key /etc/httpd/conf/ssl.key/server.key.bak<br />
cp server.crt /etc/httpd/conf/ssl.crt/server.crt<br />
cp server.key /etc/httpd/conf/ssl.key/server.key<br />
/etc/init.d/vzcp restart</pre><br />
<br />
= Moving virt (drives) to new hardware =<br />
<br />
If you have a case where a virt is dead (due to hardware failure), you can move the drives to a new hardware setup.<br />
Here are the steps and things to look for:<br />
<br />
Move the drives to the new box. Most likely, the drives will be recognized and there will be no problems booting up. On Dell PERC 5/6 cards you will need to import the new "foreign" volume via the raid BIOS.<br />
<br />
If the hardware (motherboard) is different, it’s possible the nic’s won’t be recognized. Look at /etc/modules.conf the ethernet drivers are either:<br />
<br />
<pre>alias eth0 tg3<br />
alias eth1 tg3</pre><br />
(supermicro)<br />
<br />
Or<br />
<br />
<pre>alias eth0 e100<br />
alias eth1 e1000</pre><br />
(intel)<br />
<br />
Or<br />
<br />
<pre>alias eth0 bnx2<br />
alias eth1 bnx2</pre><br />
(dell 29500)<br />
<br />
Also, you may see eth0 and eth1 swapped so if after confirming the eth devices are present (may require a reboot), then you may still need to swap the green/yellow network cables in the back to get them on the right port.<br />
<br />
VZ will not run (for long) on the new hardware, so you will need to get the license reissued. You may create/download a temporary license via their website for the interim.<br />
<br />
= Updating kernel on virt =<br />
<br />
We now use [[#vzup2date|vzup2date]] to update systems and kernels. What follows is what we used to do when we had to download kernels and install manually on old systems:<br />
<br />
<br />
1. install kernel and modules<br />
<pre># rpm -ivh vzkernel-enterprise-2.4.20-021stab028.12.777.i686.rpm \<br />
vzmodules-enterprise-2.4.20-021stab028.12.swsoft.i686.rpm<br />
vzkernel-smp ##################################################<br />
vzmodules-smp ##################################################</pre><br />
<br />
DO NOT USE the "rpm -Uhv" command to install the kernel, otherwise all the previously installed kernels may be removed from your system. <br />
<br />
2. update lilo/grub<br />
<br />
Lilo:<br />
<br />
vi /etc/lilo.conf<br />
<br />
rename old Virtuozzo kernel label from "linux+virtuozzo" to "virtuozzo_old"<br />
and add a new entry pointing to the newly installed virtuozzo kernel image.<br />
For example:<br />
<br />
<pre>prompt<br />
timeout=50<br />
default=linux+virtuozzo<br />
append="debug panic=5 console=tty console=ttyS0,38400"<br />
boot=/dev/sda<br />
map=/boot/map<br />
install=/boot/boot.b<br />
message=/boot/message<br />
linear<br />
<br />
image=/boot/vmlinuz-2.4.18-3smp<br />
label=linux<br />
initrd=/boot/initrd-2.4.18-3smp.img<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.18-3<br />
label=linux-up<br />
initrd=/boot/initrd-2.4.18-3.img<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-020stab009.5.777-enterprise<br />
label=2.4.20-9.5.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-020stab009.8.777-enterprise<br />
label=2.4.20-9.8.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-020stab009.21.777-enterprise<br />
label=2.4.20-9.21.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-020stab009.24.777-enterprise<br />
label=2.4.20-9.24.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-021stab028.3.777-enterprise<br />
label=2.4.20-28.3.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-021stab028.5.777-enterprise<br />
label=old_linux+vz<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-021stab028.12.777-enterprise<br />
label=linux+virtuozzo<br />
read-only<br />
root=/dev/sda1</pre><br />
<br />
With this configuration, the new kernel is the default kernel to use at boot. The original kernel entry has been retained with the label "old_linux+vz ".<br />
<br />
Grub:<br />
<br />
vi /boot/grub /menu.lst<br />
<br />
Add new entry at the top.<br />
For example:<br />
<br />
<pre>serial --unit=0 --speed=38400<br />
terminal --timeout=10 serial console<br />
default=0<br />
timeout=10<br />
splashimage=(hd0,0)/boot/grub/splash.xpm.gz<br />
title Virtuozzo 2.6.1 (2.4.20-021stab028.12.777-enterprise)<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-22.4.20-021stab028.12.777-enterprise root=/dev/sda1 console=tty0 \ console=ttyS0,38400<br />
title Virtuozzo 2.6.1 (2.4.20-021stab028.3.777-enterprise)<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.20-021stab028.3.777-enterprise root=/dev/sda1 console=tty0 \ console=ttyS0,38400<br />
title Red Hat Linux (2.4.18-3smp)<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.18-3smp ro root=/dev/sda1<br />
initrd /boot/initrd-2.4.18-3smp.img<br />
title Red Hat Linux-up (2.4.18-3)<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.18-3 ro root=/dev/sda1<br />
initrd /boot/initrd-2.4.18-3.img<br />
title Linux with Virtuozzo 2.5<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.20-020stab009.3.777-smp ro root=/dev/sda1<br />
title vz-enterpriseo<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.20-020stab009.21.777-enterprise ro root=/dev/sda1<br />
title vz-enterprise<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.20-020stab009.24.777-enterprise ro root=/dev/sda1 \ console=tty0 console=ttyS0,38400</pre><br />
<br />
3. run the lilo (if using lilo)<br />
<br />
<pre># lilo<br />
Added linux<br />
Added linux-up<br />
Added virtuozzo_old<br />
Added linux+virtuozzo *</pre><br />
<br />
4. reboot the server<br />
shutdown -r 23:05<br />
<br />
when it comes time for the shutdown, run [[VPS_Management#stopvirt|stopvirt]] to get things shut down faster<br />
<br />
<br />
= Remaking service VE (<=3.x only) =<br />
<br />
Occasionally the control panel for VEs (aka VZPP) will stop working and you just need to reinstall the service VE (which runs the control panels for all VEs on the HN). Here's how that's done :<br />
<br />
<pre>vzctl stop 1<br />
ipdel 1 69.55.234.152<br />
vzmlocal 1:2<br />
vzctl set 2 --save --onboot no<br />
vzsveinstall -t redhat-as3-minimal -d /root/Rel261/HW/RPMS -s 69.55.234.152 --skip-client</pre><br />
<br />
on 3.0:<br />
<br />
<pre>vzsveinstall -t redhat-as3-minimal -d /vz/Rel300/HW/RPMS -s 69.55.229.151 --skip-client<br />
<br />
vzctl stop 1<br />
vzctl mount 1<br />
vzctl mount 2<br />
/bin/cp -aRf /vz/root/2/home/vzagent0 /vz/root/1/home<br />
/bin/cp -aRf /vz/root/2/var/lib/mysql/VZADB /vz/root/1/var/lib/mysql<br />
/bin/cp -af /vz/root/2/etc/shadow /vz/root/1/etc/<br />
/bin/cp -aRf /vz/root/2/var/vzcp/static/vz/skins/ /vz/root/1/var/vzcp/static/vz<br />
/bin/cp -af /vz/root/2/etc/vzcp/pp/menu.xml /vz/root/1/etc/vzcp/pp/<br />
/bin/cp -af /vz/root/2/etc/vzcp/pp/dashboard.xml /vz/root/1/etc/vzcp/pp/<br />
<br />
vzctl umount 2<br />
vzctl umount 1<br />
vzctl start 1</pre></div>Supporthttps://wiki.jcihosting.com/index.php?title=Virtuozzo_/_Linux_Reference&diff=1030Virtuozzo / Linux Reference2013-03-11T23:41:36Z<p>Support: /* Adding new OS/application templates */</p>
<hr />
<div>= Updating VZ =<br />
<br />
Update Virtuozzo to version 4.0.0<br />
Apply minor updates for Virtuozzo 3.0.0<br />
<br />
...<br />
When you get to the last screen, you will be given an option to <br />
<br />
( ) Automatically reboot after update<br />
(*) Reboot later manually<br />
<br />
make sure you choose to reboot later.<br />
<br />
= Migrating Templates =<br />
One nice feature VZ offers is the ability to run a virtual environment (VE or CT as it's now called) based on a particular ditribution on a host which may or may not share the same distribution. For instance, you could run a CT running Ubuntu while the host machine (Hardware Node or HN) is running CentOS. This is accomplished via the use of OS templates. As long as the HN contains the OS template on which a particular CT relies, it will run there. As such, if you want to move a CT from one HN to another, you will need to make sure the OS template exists there.<br />
<br />
Modern versions of VZ (>= 4.x) will check for and automatically move the OS template over to the host as you're (vz)migrating the CT over to a new HN. However we like to make sure the template is in place prior to moving the CT. <br />
<br />
The first step is to see if the template exists on the host. To see OS templates installed:<br />
<br />
2.x (shows OS and application templates):<br />
<pre># vzpkgls<br />
mod_ssl-rh9 20040624<br />
mysql-rh9 20030814 20050412<br />
openwebmail-rh9 20050427<br />
php-rh9 20050506<br />
phpBB-rh9 20041229<br />
postgresql-rh9 20050719<br />
sl-webalizer-rh9 20040119<br />
spamassassin-rh9 20040127<br />
tomcat-rh9 20040524<br />
usermin-rh9 20040116</pre><br />
<br />
>= 3.x (shows just OS templates, run without -O to see application as well):<br />
<pre># vzpkg list -O<br />
ubuntu-10.04-x86 2010-06-01 11:39:05<br />
ubuntu-11.04-x86 2011-06-22 14:23:59<br />
ubuntu-9.10-x86 2010-03-11 17:39:51<br />
centos-5-x86 2010-03-11 17:12:27<br />
debian-5.0-x86 2010-03-11 17:33:34</pre><br />
<br />
<br />
All template files are located:<br />
<br />
<pre># ls /vz/template/<br />
ColdFusion-fc1 jre-fc1 openwebmail-fc1 sl-webalizer-fc1<br />
ColdFusion-fc2 jre-fc2 openwebmail-fc2 sl-webalizer-fc2<br />
ColdFusion-rh9 jre-rh9 openwebmail-rh9 sl-webalizer-rh9<br />
PostNuke-fc1 jre-suse92 openwebmail-suse92 spamassassin-fc1<br />
PostNuke-fc2 jrun php-deb31 spamassassin-fc2<br />
PostNuke-rh9 mailman-deb31 php-fc1 spamassassin-rh9<br />
analog-fc1 mailman-fc1 php-fc2 spamassassin-suse92<br />
analog-fc2 mailman-fc2 php-rh9 suse-9.2<br />
analog-rh9 mailman-rh9 php-suse92 tomcat-fc1<br />
awstats-fc1 mailman-suse92 phpBB-fc1 tomcat-fc2<br />
awstats-rh9 mod_frontpage-fc1 phpBB-fc2 tomcat-rh9<br />
bbClone-fc1 mod_frontpage-fc2 phpBB-rh9 tomcat-suse92<br />
bbClone-fc2 mod_frontpage-rh9 phpmyadmin-deb31 urchin-fc1<br />
bbClone-rh9 mod_frontpage-suse92 postgresql-deb31 usermin-fc1<br />
conf mod_perl-deb31 postgresql-fc1 usermin-fc2<br />
debian-3.1 mod_perl-fc1 postgresql-fc2 usermin-rh9<br />
devel-deb31 mod_perl-fc2 postgresql-rh9 uw-imap-fc1<br />
devel-fc2 mod_perl-rh9 postgresql-suse92 uw-imap-fc2<br />
devel-suse92 mod_perl-suse92 proftpd-deb31 uw-imap-rh9<br />
fedora-core-1 mod_ssl-fc1 proftpd-fc1 vzredhat-7.3<br />
fedora-core-2 mod_ssl-fc2 proftpd-fc2 vzredhat-8.0<br />
jdk-deb31 mod_ssl-rh9 proftpd-rh9 webalizer-suse92<br />
jdk-fc1 mysql-deb31 proftpd-suse92 webmin-fc1<br />
jdk-fc2 mysql-fc1 redhat-7.2 webmin-fc2<br />
jdk-rh9 mysql-fc2 redhat-9 webmin-rh9<br />
jdk-suse92 mysql-rh9 redhat-as3-minimal<br />
jre-deb31 mysql-suse92 redhat-devel-9<br />
</pre><br />
<br />
What we see here are the base OS templates: <tt>redhat-9, fedora-core-1</tt> and supporting application templates: <tt>postgresql-rh9, mailman-rh9, jdk-fc1, mod_ssl-fc1</tt>. So if you wanted to copy over all of redhat 9 you'd need to copy the contents of:<br />
<pre># ls -d /vz/template/*rh9*<br />
/vz/template/ColdFusion-rh9 /vz/template/mod_frontpage-rh9 /vz/template/proftpd-rh9<br />
/vz/template/PostNuke-rh9 /vz/template/mod_perl-rh9 /vz/template/sl-webalizer-rh9<br />
/vz/template/analog-rh9 /vz/template/mod_ssl-rh9 /vz/template/spamassassin-rh9<br />
/vz/template/awstats-rh9 /vz/template/mysql-rh9 /vz/template/tomcat-rh9<br />
/vz/template/bbClone-rh9 /vz/template/openwebmail-rh9 /vz/template/usermin-rh9<br />
/vz/template/jdk-rh9 /vz/template/php-rh9 /vz/template/uw-imap-rh9<br />
/vz/template/jre-rh9 /vz/template/phpBB-rh9 /vz/template/webmin-rh9<br />
/vz/template/mailman-rh9 /vz/template/postgresql-rh9<br />
<br />
# ls -d /vz/template/*redhat-9*<br />
/vz/template/redhat-9<br />
</pre><br />
<br />
To get a template from one HN to another, it simply needs to be rsync'd over to the target HN. It's rsync'd over in this fashion:<br />
<br />
rsync -av -e ssh /vz/template/redhat-9 root@10.1.4.65:/vz/template<br />
rsync -av -e ssh /vz/template/*rh9 root@10.1.4.65:/vz/template<br />
<br />
Newer (>= 3.x) VZ employs "EZ" templates which are organized a little differently:<br />
<br />
<pre># ls /vz/template/<br />
cache CmdTool.log debian redhat-as3-minimal-20080630.tar.gz vc<br />
centos conf redhat-as3-minimal ubuntu<br />
<br />
# ls /vz/template/ubuntu/<br />
10.04 11.04 9.10<br />
# ls /vz/template/ubuntu/10.04/<br />
x86<br />
# ls /vz/template/ubuntu/10.04/x86/<br />
aap_1.091-1_all<br />
aap-doc_1.091-1_all<br />
adduser_3.112ubuntu1_all<br />
anacron_2.3-13.1ubuntu11_i386<br />
apache2_2.2.14-5ubuntu8.2_i386<br />
apache2_2.2.14-5ubuntu8.4_i386<br />
apache2_2.2.14-5ubuntu8.6_i386<br />
apache2_2.2.14-5ubuntu8.7_i386<br />
apache2_2.2.14-5ubuntu8.8_i386<br />
apache2_2.2.14-5ubuntu8.9_i386<br />
<...snip><br />
<br />
# ls /vz/template/cache/<br />
centos-5-x86.tar.gz suse-11.3-x86.tar.gz ubuntu-9.10-x86.tar.gz<br />
debian-5.0-x86.tar.gz ubuntu-10.04-x86.tar.gz<br />
fedora-core-14-x86.tar.gz ubuntu-11.04-x86.tar.gz<br />
</pre><br />
<br />
So what we see is the entire OS and all applications are in 1 directory, and organized by distribution and release version. So to move Ubuntu 10.04:<br />
<br />
rsync -av -e ssh /vz/template/ubuntu/10.04 root@10.1.4.69:/vz/template/ubuntu<br />
<br />
and you also need to move the cache:<br />
<br />
rsync -av -e ssh /vz/template/cache/ubuntu-10.04-x86.tar.gz root@10.1.4.69:/vz/template/cache/<br />
<br />
Once the transfer is complete, you will be able to run <tt>vzpkgls</tt> or <tt>vzpkg list -O</tt> and see the template(s) listed on the target HN.<br />
<br />
So far, this all supposes template doesn't exist on the target- very simple, just copy it over. If the template exists already on the target HN, this requires closer inspection. With older templates, simply moving over the templates would be the end of it- you see the version listed when you do a <tt>vzplgls</tt>:<br />
<br />
mysql-rh9 20030814 20050412<br />
<br />
this means there are 2 versions of the mysql package for RH9: 20030814 and 20050412<br />
As an aside, you can't copy over just 1 of them, but if you rsync'd over the <tt>mysql-rh9</tt> directory, you'd see 20030814 and 20050412 on the target HN. As long as the versions match up, you're good to go.<br />
<br />
With EZ templates, the versions are not as easy to decipher. When you look at the <tt>vzpkg list</tt> output, that simply tells you when a cache was created. A cache is simply a snapshot of all the data needed for the latest versions of the OS and it's applications at the time the cache was created. It is a date, and thus tells you nothing about the versions within. So for instance, if you had a cache for Ubuntu 10.04 from 2007 and you ran <tt>vzpkg create cache ubuntu-10.04-x86</tt> it would re-download the latest version of apache, mysql and so on. And any ''subsequent'' ubuntu 10.04 CT's created thereafter would use those newer versions, whereas older CT's based on 10.04 would use older templates. You can see how this works by looking at the files under:<br />
<br />
<pre># ls /vz/template/ubuntu/10.04/x86/<br />
aap_1.091-1_all<br />
aap-doc_1.091-1_all<br />
adduser_3.112ubuntu1_all<br />
anacron_2.3-13.1ubuntu11_i386<br />
apache2_2.2.14-5ubuntu8.2_i386<br />
apache2_2.2.14-5ubuntu8.4_i386<br />
apache2_2.2.14-5ubuntu8.6_i386<br />
apache2_2.2.14-5ubuntu8.7_i386<br />
apache2_2.2.14-5ubuntu8.8_i386<br />
apache2_2.2.14-5ubuntu8.9_i386<br />
<...snip><br />
</pre><br />
<br />
You can see that this template has in it 6 different versions of apache2. This means that this template was merged with other 10.04 templates and/or it has been cached a few times, each time a new apache was available. The unfortunate thing is it's almost impossible to know which CT's are using which version of apache so it's hard to try to prune this down and remove unwanted/unneeded applications.<br />
<br />
So, if you see another HN has Ubuntu 10.04 on it, you need to see/confirm that it has all the exact files your CT needs. You can run a test rsync to see what ''would'' be transferred (what's missing):<br />
<br />
rsync -avn --stats -e ssh /vz/template/ubuntu/10.04 root@10.1.4.69:/vz/template/ubuntu<br />
<br />
Based on the amount of data you get back from the rsync, you will be able to decide whether or not to remove the -n and allow the full transfer to go through. The downside to doing the transfer is the situation described above- you've permanently joined these templates and now you may have 10 versions of apache on the target HN. There's one other potential pitfall to this kind of merging- it will also potentially copy over shared files, for which there is no particular version, most of which are in <tt>/vz/template/ubuntu/10.04/x86/config</tt>: <br />
<br />
cat /vz/template/ubuntu/10.04/x86/config/os/default/repositories<br />
/vz/template/ubuntu/10.04/x86/config/os/default/repositories<br />
<br />
Here are some specific things to look out for. Case: copying centos-5 on virt19 to virt12. We dumped the rsync output to a file so we could inspect:<br />
<br />
[root@virt19 /vz/template]# rsync -an -e ssh /vz/template/centos/ root@10.1.4.62:/vz/template/centos/ > v12_c5<br />
<br />
(note, that can be dangerous, better to add on full path <tt>/vz/template/centos/5</tt>)<br />
<br />
So in the output file we see things like:<br />
<br />
<pre><br />
x86/base0/cachecookie<br />
x86/base0/primary.xml.gz <br />
x86/base0/primary.xml.gz.sqlite <br />
x86/base0/repomd.xml <br />
x86/base1/cachecookie <br />
x86/base1/filelists.xml.gz <br />
x86/base1/filelists.xml.gz.sqlite <br />
x86/base1/primary.xml.gz <br />
x86/base1/primary.xml.gz.sqlite <br />
x86/base1/repomd.xml<br />
x86/base2/cachecookie<br />
</pre><br />
<br />
Which make us nervous cause those are not versioned files, i.e.:<br />
5/x86/MAKEDEV-3.23-1.2.i386/usr/sbin/mksock<br />
<br />
So those files will replace existing files on v12 rather than the case of the latter where it's likely being added. So caution should be given and deference to whichever version is newer (which rsync should take care of, but that will depend on the date each file was created and perhaps not the actual data within).<br />
<br />
If we look further in the file we see:<br />
<br />
x86/psa-appvault-moodle-1.8-8202920080409011254.noarch/usr/local/psa/var/cgitory/moodle-1.<br />
9/htdocs/lib/editor/htmlarea/plugins/TableOperations/img/cell-split.gif<br />
<br />
and other files looking like:<br />
5/x86/plesk-base-9.0.1-cos5.build90090127.18.i586/etc/sw/keys/info<br />
<br />
Which means plesk is here. We don't need plesk on v12 (the CT moving over doesn't use it) so we rerun the rsync and exclude it:<br />
<br />
[root@virt19 /vz/template]# rsync -an -e ssh --exclude="plesk*" --exclude="psa*" /vz/template/centos/ root@10.1.4.62:/vz/template/centos/ > v12_c5<br />
<br />
and now it's much smaller:<br />
<br />
[root@virt19 /vz/template]# cat v12_c5|wc -l<br />
97104<br />
<br />
Before running with excludes it was:<br />
[root@virt19 /vz/template]# cat v12_c5|wc -l <br />
238238<br />
<br />
So again, look at what you're doing. Generally, combining templates is fine however.<br />
<br />
If you're happy with the merge, run the rsync again without the "-n" to let it actually do the transfer.<br />
<br />
Once done, don't forget to add the new template to the HN in Management -> Reference -> Templates<br />
<br />
= getting help =<br />
<br />
= vzup2date =<br />
TODO <br />
= Adding new OS/application templates =<br />
<br />
Virtuozzo will lag a bit behind the initial release of an OS, perhaps by a month or more. Further, a newly-released OS may be available, but there may not be many/any application templates available initially. It pays to wait a bit.<br />
<br />
A template is easily installed via [[#vzup2date|vzup2date]]:<br />
<br />
vzup2date -z<br />
<br />
You will be given a list of OS's to install. Select an OS, then customize that selection by checking/unchecking the application templates we do/don't want to include/offer.<br />
<br />
Generally, here are the app templates we include/offer:<br />
<pre>cyrus-imap<br />
devel<br />
imp<br />
jre<br />
jsdk<br />
mailman<br />
mod_perl<br />
mod_ssl<br />
mysql<br />
php<br />
phpmyadmin<br />
phppgadmin<br />
postgresql<br />
proftpd<br />
spamassassin<br />
squirrelmail<br />
tomcat<br />
vsftpd<br />
</pre><br />
<br />
We do not (any longer) offer webalizer or analog.<br />
<br />
After selecting and installing the OS and apps, you should ensure it's installed:<br />
<br />
vzpkg list -O<br />
<br />
Your new OS (ex. fedora-core-13-x86_64) will be listed, without a corresponding cache date:<br />
<pre><br />
[root@virt13 /vzconf]# vzpkg list -O<br />
centos-6-x86 2011-07-22 13:24:41<br />
centos-6-x86_64 2012-03-09 20:42:22<br />
centos-5-x86 2009-07-27 16:58:24<br />
centos-5-x86_64 2012-03-09 22:38:53<br />
ubuntu-11.10-x86_64 2012-03-01 23:13:33<br />
ubuntu-9.04-x86 2009-06-02 11:32:39<br />
ubuntu-12.04-x86_64 2012-05-29 13:50:50<br />
ubuntu-8.04-x86 2008-09-17 21:27:20<br />
ubuntu-8.10-x86 2009-03-13 13:58:57<br />
ubuntu-11.04-x86 2011-12-05 11:15:46<br />
ubuntu-7.04-x86 2008-09-24 16:42:50<br />
redhat-el6-x86_64<br />
fedora-core-13-x86_64 <br />
fedora-core-12-x86 2010-02-06 13:24:32<br />
fedora-core-15-x86 2011-12-05 11:36:43<br />
fedora-core-11-x86 2009-10-01 16:30:59<br />
fedora-core-14-x86<br />
fedora-core-16-x86_64 2012-03-13 11:40:23<br />
fedora-core-9-x86 2008-11-29 10:52:18<br />
debian-6.0-x86 2011-07-29 16:00:35<br />
debian-6.0-x86_64 2012-03-12 19:53:33<br />
debian-5.0-x86 2009-03-12 12:39:11<br />
</pre><br />
<br />
You can also confirm the apps installed:<br />
<pre>[root@virt13 /vzconf]# vzpkg list fedora-core-13-x86_64<br />
fedora-core-13-x86_64 2013-03-08 11:39:44<br />
fedora-core-13-x86_64 devel<br />
fedora-core-13-x86_64 php<br />
fedora-core-13-x86_64 proftpd<br />
fedora-core-13-x86_64 postgresql<br />
fedora-core-13-x86_64 phpmyadmin<br />
fedora-core-13-x86_64 mysql<br />
</pre><br />
<br />
Before you can use the OS you must cache it. To create the cache run:<br />
vzpkg cache [OS]<br />
<br />
ex: <tt>vzpkg cache fedora-core-13-x86_64</tt><br />
<br />
Now it's ready to use. In order to be able to use it with the [[VPS_Management#vm|vm]] command, you must create a file:<br />
jctmpl-[OS]<br />
<br />
ex: <tt>jctmpl-fedora-core-13-x86_64</tt><br />
<br />
In it, on line 1 we place the OS, followed by the app names, ex:<br />
<pre>more jctmpl-fedora-core-13-x86_64<br />
fedora-core-13-x86_64<br />
postgresql<br />
jsdk<br />
mod_ssl<br />
phpmyadmin<br />
mod_perl<br />
tomcat<br />
devel<br />
php<br />
mailman<br />
cyrus-imap<br />
squirrelmail<br />
proftpd<br />
mysql<br />
phppgadmin<br />
spamassassin<br />
</pre><br />
<br />
The packages will be installed in that order, so any pre-req's should be handled there.<br />
<br />
Now, when you run [[VPS_Management#vm|vm]] it will show you the newly-added OS and you may create a CT using it.<br />
<br />
The last few tasks are related to mgmt:<br />
<br />
1. add to mgmt->reference->templates (use the existing templates as an example). Once added, the new template will be included with the list of OS's available for the given HN.<br />
2. if this is going to be a new OS offered for ordering, it needs to be added to the order form and webpage.<br />
<br />
a. to get a list of applications and versions in the new OS, you should create a test CT, enter it and run <tt>dpkg -l|sort</tt> (or <tt>rpm -aq|sort</tt> in centos/fedora) and enter that list in the following places:<br><br />
<tt>/usr/local/www/jc_pub/[osname].html</tt><br><br />
<tt>/usr/local/www/signup/html/[osname].html</tt><br><br />
Note, you will have to move the most recent OS data from <tt>/usr/local/www/signup/html/[osname].html</tt> into the corresponding <tt>/usr/local/www/signup/html/[osname]_old.html</tt> updating the version links in both.<br />
<br />
b. to add the new OS to the order form, you will need to update <tt>/usr/local/www/signup/html/step1.html</tt> in several places (for regular and OSS orders) as well as updating the templateID which you can get from mgmt->reference->templates (or [[Management_System_/_Public_Website_/_Signup_/_Account_Manager#jc:ref_templates|jc:ref_templates]])<br />
<br />
<br />
<br />
Old method to obtain the templates:<br />
<br />
Go to http://www.parallels.com/en/virtuozzo/templates/catalog/<br />
<br />
Download the RPM's and install them<br />
<br />
vzpkg list<br />
to see the new packages/os templates<br />
<br />
vzpkg create cache "OS_EZ_template_name"<br />
to create a cache<br />
<br />
<br />
= Update OS template =<br />
<br />
Installation and update of the new version of OS template will not affect the existing containers. It will affect only the new containers, created after this update operation. In terms of fixing "vzctl enter" error and SSH connectivity to Ubuntu based containers, it is necessary to recreate the template and not to update it, so the commands to run are:<br />
<br />
<pre>rpm -Uhv ubuntu-8.04-x86-ez-4.0.0-9.swsoft.noarch.rpm<br />
vzpkg clean ubuntu-8.04-x86<br />
vzpkg remove cache ubuntu-8.04-x86<br />
vzpkg create cache ubuntu-8.04-x86</pre><br />
<br />
<br />
= Installing new SSL cert on VZCP =<br />
<br />
<pre>openssl req -days 1999 -new -x509 -nodes -out server.crt -keyout server.key<br />
<br />
US<br />
CA<br />
San Diego<br />
johncompanies.com<br />
johncompanies.com<br />
virt15ohncompanies.com<br />
linux@johncompanies.com<br />
<br />
cp -p /etc/httpd/conf/ssl.crt/server.crt /etc/httpd/conf/ssl.crt/server.crt.bak<br />
cp -p /etc/httpd/conf/ssl.key/server.key /etc/httpd/conf/ssl.key/server.key.bak<br />
cp server.crt /etc/httpd/conf/ssl.crt/server.crt<br />
cp server.key /etc/httpd/conf/ssl.key/server.key<br />
/etc/init.d/vzcp restart</pre><br />
<br />
= Moving virt (drives) to new hardware =<br />
<br />
If you have a case where a virt is dead (due to hardware failure), you can move the drives to a new hardware setup.<br />
Here are the steps and things to look for:<br />
<br />
Move the drives to the new box. Most likely, the drives will be recognized and there will be no problems booting up.<br />
<br />
If the hardware (motherboard) is different, it’s possible the nic’s won’t be recognized. Look at /etc/modules.conf the ethernet drivers are either:<br />
<br />
<pre>alias eth0 tg3<br />
alias eth1 tg3</pre><br />
(supermicro)<br />
<br />
Or<br />
<br />
<pre>alias eth0 e100<br />
alias eth1 e1000</pre><br />
(intel)<br />
<br />
Or<br />
<br />
<pre>alias eth0 bnx2<br />
alias eth1 bnx2</pre><br />
(dell 29500)<br />
<br />
Also, you may see eth0 and eth1 swapped so if after confirming the eth devices are present (may require a reboot), then you may still need to swap the green/yellow network cables in the back to get them on the right port.<br />
<br />
VZ will not run (for long) on the new hardware, so you will need to get the license reissued. You may create/download a temporary license via their website for the interim.<br />
<br />
= Updating kernel on virt =<br />
<br />
We now use [[#vzup2date|vzup2date]] to update systems and kernels. What follows is what we used to do when we had to download kernels and install manually on old systems:<br />
<br />
<br />
1. install kernel and modules<br />
<pre># rpm -ivh vzkernel-enterprise-2.4.20-021stab028.12.777.i686.rpm \<br />
vzmodules-enterprise-2.4.20-021stab028.12.swsoft.i686.rpm<br />
vzkernel-smp ##################################################<br />
vzmodules-smp ##################################################</pre><br />
<br />
DO NOT USE the "rpm -Uhv" command to install the kernel, otherwise all the previously installed kernels may be removed from your system. <br />
<br />
2. update lilo/grub<br />
<br />
Lilo:<br />
<br />
vi /etc/lilo.conf<br />
<br />
rename old Virtuozzo kernel label from "linux+virtuozzo" to "virtuozzo_old"<br />
and add a new entry pointing to the newly installed virtuozzo kernel image.<br />
For example:<br />
<br />
<pre>prompt<br />
timeout=50<br />
default=linux+virtuozzo<br />
append="debug panic=5 console=tty console=ttyS0,38400"<br />
boot=/dev/sda<br />
map=/boot/map<br />
install=/boot/boot.b<br />
message=/boot/message<br />
linear<br />
<br />
image=/boot/vmlinuz-2.4.18-3smp<br />
label=linux<br />
initrd=/boot/initrd-2.4.18-3smp.img<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.18-3<br />
label=linux-up<br />
initrd=/boot/initrd-2.4.18-3.img<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-020stab009.5.777-enterprise<br />
label=2.4.20-9.5.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-020stab009.8.777-enterprise<br />
label=2.4.20-9.8.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-020stab009.21.777-enterprise<br />
label=2.4.20-9.21.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-020stab009.24.777-enterprise<br />
label=2.4.20-9.24.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-021stab028.3.777-enterprise<br />
label=2.4.20-28.3.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-021stab028.5.777-enterprise<br />
label=old_linux+vz<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-021stab028.12.777-enterprise<br />
label=linux+virtuozzo<br />
read-only<br />
root=/dev/sda1</pre><br />
<br />
With this configuration, the new kernel is the default kernel to use at boot. The original kernel entry has been retained with the label "old_linux+vz ".<br />
<br />
Grub:<br />
<br />
vi /boot/grub /menu.lst<br />
<br />
Add new entry at the top.<br />
For example:<br />
<br />
<pre>serial --unit=0 --speed=38400<br />
terminal --timeout=10 serial console<br />
default=0<br />
timeout=10<br />
splashimage=(hd0,0)/boot/grub/splash.xpm.gz<br />
title Virtuozzo 2.6.1 (2.4.20-021stab028.12.777-enterprise)<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-22.4.20-021stab028.12.777-enterprise root=/dev/sda1 console=tty0 \ console=ttyS0,38400<br />
title Virtuozzo 2.6.1 (2.4.20-021stab028.3.777-enterprise)<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.20-021stab028.3.777-enterprise root=/dev/sda1 console=tty0 \ console=ttyS0,38400<br />
title Red Hat Linux (2.4.18-3smp)<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.18-3smp ro root=/dev/sda1<br />
initrd /boot/initrd-2.4.18-3smp.img<br />
title Red Hat Linux-up (2.4.18-3)<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.18-3 ro root=/dev/sda1<br />
initrd /boot/initrd-2.4.18-3.img<br />
title Linux with Virtuozzo 2.5<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.20-020stab009.3.777-smp ro root=/dev/sda1<br />
title vz-enterpriseo<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.20-020stab009.21.777-enterprise ro root=/dev/sda1<br />
title vz-enterprise<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.20-020stab009.24.777-enterprise ro root=/dev/sda1 \ console=tty0 console=ttyS0,38400</pre><br />
<br />
3. run the lilo (if using lilo)<br />
<br />
<pre># lilo<br />
Added linux<br />
Added linux-up<br />
Added virtuozzo_old<br />
Added linux+virtuozzo *</pre><br />
<br />
4. reboot the server<br />
shutdown -r 23:05<br />
<br />
when it comes time for the shutdown, run [[VPS_Management#stopvirt|stopvirt]] to get things shut down faster<br />
<br />
<br />
= Remaking service VE (<=3.x only) =<br />
<br />
Occasionally the control panel for VEs (aka VZPP) will stop working and you just need to reinstall the service VE (which runs the control panels for all VEs on the HN). Here's how that's done :<br />
<br />
<pre>vzctl stop 1<br />
ipdel 1 69.55.234.152<br />
vzmlocal 1:2<br />
vzctl set 2 --save --onboot no<br />
vzsveinstall -t redhat-as3-minimal -d /root/Rel261/HW/RPMS -s 69.55.234.152 --skip-client</pre><br />
<br />
on 3.0:<br />
<br />
<pre>vzsveinstall -t redhat-as3-minimal -d /vz/Rel300/HW/RPMS -s 69.55.229.151 --skip-client<br />
<br />
vzctl stop 1<br />
vzctl mount 1<br />
vzctl mount 2<br />
/bin/cp -aRf /vz/root/2/home/vzagent0 /vz/root/1/home<br />
/bin/cp -aRf /vz/root/2/var/lib/mysql/VZADB /vz/root/1/var/lib/mysql<br />
/bin/cp -af /vz/root/2/etc/shadow /vz/root/1/etc/<br />
/bin/cp -aRf /vz/root/2/var/vzcp/static/vz/skins/ /vz/root/1/var/vzcp/static/vz<br />
/bin/cp -af /vz/root/2/etc/vzcp/pp/menu.xml /vz/root/1/etc/vzcp/pp/<br />
/bin/cp -af /vz/root/2/etc/vzcp/pp/dashboard.xml /vz/root/1/etc/vzcp/pp/<br />
<br />
vzctl umount 2<br />
vzctl umount 1<br />
vzctl start 1</pre></div>Supporthttps://wiki.jcihosting.com/index.php?title=Virtuozzo_/_Linux_Reference&diff=1029Virtuozzo / Linux Reference2013-03-11T23:37:56Z<p>Support: /* Adding new OS/application templates */</p>
<hr />
<div>= Updating VZ =<br />
<br />
Update Virtuozzo to version 4.0.0<br />
Apply minor updates for Virtuozzo 3.0.0<br />
<br />
...<br />
When you get to the last screen, you will be given an option to <br />
<br />
( ) Automatically reboot after update<br />
(*) Reboot later manually<br />
<br />
make sure you choose to reboot later.<br />
<br />
= Migrating Templates =<br />
One nice feature VZ offers is the ability to run a virtual environment (VE or CT as it's now called) based on a particular ditribution on a host which may or may not share the same distribution. For instance, you could run a CT running Ubuntu while the host machine (Hardware Node or HN) is running CentOS. This is accomplished via the use of OS templates. As long as the HN contains the OS template on which a particular CT relies, it will run there. As such, if you want to move a CT from one HN to another, you will need to make sure the OS template exists there.<br />
<br />
Modern versions of VZ (>= 4.x) will check for and automatically move the OS template over to the host as you're (vz)migrating the CT over to a new HN. However we like to make sure the template is in place prior to moving the CT. <br />
<br />
The first step is to see if the template exists on the host. To see OS templates installed:<br />
<br />
2.x (shows OS and application templates):<br />
<pre># vzpkgls<br />
mod_ssl-rh9 20040624<br />
mysql-rh9 20030814 20050412<br />
openwebmail-rh9 20050427<br />
php-rh9 20050506<br />
phpBB-rh9 20041229<br />
postgresql-rh9 20050719<br />
sl-webalizer-rh9 20040119<br />
spamassassin-rh9 20040127<br />
tomcat-rh9 20040524<br />
usermin-rh9 20040116</pre><br />
<br />
>= 3.x (shows just OS templates, run without -O to see application as well):<br />
<pre># vzpkg list -O<br />
ubuntu-10.04-x86 2010-06-01 11:39:05<br />
ubuntu-11.04-x86 2011-06-22 14:23:59<br />
ubuntu-9.10-x86 2010-03-11 17:39:51<br />
centos-5-x86 2010-03-11 17:12:27<br />
debian-5.0-x86 2010-03-11 17:33:34</pre><br />
<br />
<br />
All template files are located:<br />
<br />
<pre># ls /vz/template/<br />
ColdFusion-fc1 jre-fc1 openwebmail-fc1 sl-webalizer-fc1<br />
ColdFusion-fc2 jre-fc2 openwebmail-fc2 sl-webalizer-fc2<br />
ColdFusion-rh9 jre-rh9 openwebmail-rh9 sl-webalizer-rh9<br />
PostNuke-fc1 jre-suse92 openwebmail-suse92 spamassassin-fc1<br />
PostNuke-fc2 jrun php-deb31 spamassassin-fc2<br />
PostNuke-rh9 mailman-deb31 php-fc1 spamassassin-rh9<br />
analog-fc1 mailman-fc1 php-fc2 spamassassin-suse92<br />
analog-fc2 mailman-fc2 php-rh9 suse-9.2<br />
analog-rh9 mailman-rh9 php-suse92 tomcat-fc1<br />
awstats-fc1 mailman-suse92 phpBB-fc1 tomcat-fc2<br />
awstats-rh9 mod_frontpage-fc1 phpBB-fc2 tomcat-rh9<br />
bbClone-fc1 mod_frontpage-fc2 phpBB-rh9 tomcat-suse92<br />
bbClone-fc2 mod_frontpage-rh9 phpmyadmin-deb31 urchin-fc1<br />
bbClone-rh9 mod_frontpage-suse92 postgresql-deb31 usermin-fc1<br />
conf mod_perl-deb31 postgresql-fc1 usermin-fc2<br />
debian-3.1 mod_perl-fc1 postgresql-fc2 usermin-rh9<br />
devel-deb31 mod_perl-fc2 postgresql-rh9 uw-imap-fc1<br />
devel-fc2 mod_perl-rh9 postgresql-suse92 uw-imap-fc2<br />
devel-suse92 mod_perl-suse92 proftpd-deb31 uw-imap-rh9<br />
fedora-core-1 mod_ssl-fc1 proftpd-fc1 vzredhat-7.3<br />
fedora-core-2 mod_ssl-fc2 proftpd-fc2 vzredhat-8.0<br />
jdk-deb31 mod_ssl-rh9 proftpd-rh9 webalizer-suse92<br />
jdk-fc1 mysql-deb31 proftpd-suse92 webmin-fc1<br />
jdk-fc2 mysql-fc1 redhat-7.2 webmin-fc2<br />
jdk-rh9 mysql-fc2 redhat-9 webmin-rh9<br />
jdk-suse92 mysql-rh9 redhat-as3-minimal<br />
jre-deb31 mysql-suse92 redhat-devel-9<br />
</pre><br />
<br />
What we see here are the base OS templates: <tt>redhat-9, fedora-core-1</tt> and supporting application templates: <tt>postgresql-rh9, mailman-rh9, jdk-fc1, mod_ssl-fc1</tt>. So if you wanted to copy over all of redhat 9 you'd need to copy the contents of:<br />
<pre># ls -d /vz/template/*rh9*<br />
/vz/template/ColdFusion-rh9 /vz/template/mod_frontpage-rh9 /vz/template/proftpd-rh9<br />
/vz/template/PostNuke-rh9 /vz/template/mod_perl-rh9 /vz/template/sl-webalizer-rh9<br />
/vz/template/analog-rh9 /vz/template/mod_ssl-rh9 /vz/template/spamassassin-rh9<br />
/vz/template/awstats-rh9 /vz/template/mysql-rh9 /vz/template/tomcat-rh9<br />
/vz/template/bbClone-rh9 /vz/template/openwebmail-rh9 /vz/template/usermin-rh9<br />
/vz/template/jdk-rh9 /vz/template/php-rh9 /vz/template/uw-imap-rh9<br />
/vz/template/jre-rh9 /vz/template/phpBB-rh9 /vz/template/webmin-rh9<br />
/vz/template/mailman-rh9 /vz/template/postgresql-rh9<br />
<br />
# ls -d /vz/template/*redhat-9*<br />
/vz/template/redhat-9<br />
</pre><br />
<br />
To get a template from one HN to another, it simply needs to be rsync'd over to the target HN. It's rsync'd over in this fashion:<br />
<br />
rsync -av -e ssh /vz/template/redhat-9 root@10.1.4.65:/vz/template<br />
rsync -av -e ssh /vz/template/*rh9 root@10.1.4.65:/vz/template<br />
<br />
Newer (>= 3.x) VZ employs "EZ" templates which are organized a little differently:<br />
<br />
<pre># ls /vz/template/<br />
cache CmdTool.log debian redhat-as3-minimal-20080630.tar.gz vc<br />
centos conf redhat-as3-minimal ubuntu<br />
<br />
# ls /vz/template/ubuntu/<br />
10.04 11.04 9.10<br />
# ls /vz/template/ubuntu/10.04/<br />
x86<br />
# ls /vz/template/ubuntu/10.04/x86/<br />
aap_1.091-1_all<br />
aap-doc_1.091-1_all<br />
adduser_3.112ubuntu1_all<br />
anacron_2.3-13.1ubuntu11_i386<br />
apache2_2.2.14-5ubuntu8.2_i386<br />
apache2_2.2.14-5ubuntu8.4_i386<br />
apache2_2.2.14-5ubuntu8.6_i386<br />
apache2_2.2.14-5ubuntu8.7_i386<br />
apache2_2.2.14-5ubuntu8.8_i386<br />
apache2_2.2.14-5ubuntu8.9_i386<br />
<...snip><br />
<br />
# ls /vz/template/cache/<br />
centos-5-x86.tar.gz suse-11.3-x86.tar.gz ubuntu-9.10-x86.tar.gz<br />
debian-5.0-x86.tar.gz ubuntu-10.04-x86.tar.gz<br />
fedora-core-14-x86.tar.gz ubuntu-11.04-x86.tar.gz<br />
</pre><br />
<br />
So what we see is the entire OS and all applications are in 1 directory, and organized by distribution and release version. So to move Ubuntu 10.04:<br />
<br />
rsync -av -e ssh /vz/template/ubuntu/10.04 root@10.1.4.69:/vz/template/ubuntu<br />
<br />
and you also need to move the cache:<br />
<br />
rsync -av -e ssh /vz/template/cache/ubuntu-10.04-x86.tar.gz root@10.1.4.69:/vz/template/cache/<br />
<br />
Once the transfer is complete, you will be able to run <tt>vzpkgls</tt> or <tt>vzpkg list -O</tt> and see the template(s) listed on the target HN.<br />
<br />
So far, this all supposes template doesn't exist on the target- very simple, just copy it over. If the template exists already on the target HN, this requires closer inspection. With older templates, simply moving over the templates would be the end of it- you see the version listed when you do a <tt>vzplgls</tt>:<br />
<br />
mysql-rh9 20030814 20050412<br />
<br />
this means there are 2 versions of the mysql package for RH9: 20030814 and 20050412<br />
As an aside, you can't copy over just 1 of them, but if you rsync'd over the <tt>mysql-rh9</tt> directory, you'd see 20030814 and 20050412 on the target HN. As long as the versions match up, you're good to go.<br />
<br />
With EZ templates, the versions are not as easy to decipher. When you look at the <tt>vzpkg list</tt> output, that simply tells you when a cache was created. A cache is simply a snapshot of all the data needed for the latest versions of the OS and it's applications at the time the cache was created. It is a date, and thus tells you nothing about the versions within. So for instance, if you had a cache for Ubuntu 10.04 from 2007 and you ran <tt>vzpkg create cache ubuntu-10.04-x86</tt> it would re-download the latest version of apache, mysql and so on. And any ''subsequent'' ubuntu 10.04 CT's created thereafter would use those newer versions, whereas older CT's based on 10.04 would use older templates. You can see how this works by looking at the files under:<br />
<br />
<pre># ls /vz/template/ubuntu/10.04/x86/<br />
aap_1.091-1_all<br />
aap-doc_1.091-1_all<br />
adduser_3.112ubuntu1_all<br />
anacron_2.3-13.1ubuntu11_i386<br />
apache2_2.2.14-5ubuntu8.2_i386<br />
apache2_2.2.14-5ubuntu8.4_i386<br />
apache2_2.2.14-5ubuntu8.6_i386<br />
apache2_2.2.14-5ubuntu8.7_i386<br />
apache2_2.2.14-5ubuntu8.8_i386<br />
apache2_2.2.14-5ubuntu8.9_i386<br />
<...snip><br />
</pre><br />
<br />
You can see that this template has in it 6 different versions of apache2. This means that this template was merged with other 10.04 templates and/or it has been cached a few times, each time a new apache was available. The unfortunate thing is it's almost impossible to know which CT's are using which version of apache so it's hard to try to prune this down and remove unwanted/unneeded applications.<br />
<br />
So, if you see another HN has Ubuntu 10.04 on it, you need to see/confirm that it has all the exact files your CT needs. You can run a test rsync to see what ''would'' be transferred (what's missing):<br />
<br />
rsync -avn --stats -e ssh /vz/template/ubuntu/10.04 root@10.1.4.69:/vz/template/ubuntu<br />
<br />
Based on the amount of data you get back from the rsync, you will be able to decide whether or not to remove the -n and allow the full transfer to go through. The downside to doing the transfer is the situation described above- you've permanently joined these templates and now you may have 10 versions of apache on the target HN. There's one other potential pitfall to this kind of merging- it will also potentially copy over shared files, for which there is no particular version, most of which are in <tt>/vz/template/ubuntu/10.04/x86/config</tt>: <br />
<br />
cat /vz/template/ubuntu/10.04/x86/config/os/default/repositories<br />
/vz/template/ubuntu/10.04/x86/config/os/default/repositories<br />
<br />
Here are some specific things to look out for. Case: copying centos-5 on virt19 to virt12. We dumped the rsync output to a file so we could inspect:<br />
<br />
[root@virt19 /vz/template]# rsync -an -e ssh /vz/template/centos/ root@10.1.4.62:/vz/template/centos/ > v12_c5<br />
<br />
(note, that can be dangerous, better to add on full path <tt>/vz/template/centos/5</tt>)<br />
<br />
So in the output file we see things like:<br />
<br />
<pre><br />
x86/base0/cachecookie<br />
x86/base0/primary.xml.gz <br />
x86/base0/primary.xml.gz.sqlite <br />
x86/base0/repomd.xml <br />
x86/base1/cachecookie <br />
x86/base1/filelists.xml.gz <br />
x86/base1/filelists.xml.gz.sqlite <br />
x86/base1/primary.xml.gz <br />
x86/base1/primary.xml.gz.sqlite <br />
x86/base1/repomd.xml<br />
x86/base2/cachecookie<br />
</pre><br />
<br />
Which make us nervous cause those are not versioned files, i.e.:<br />
5/x86/MAKEDEV-3.23-1.2.i386/usr/sbin/mksock<br />
<br />
So those files will replace existing files on v12 rather than the case of the latter where it's likely being added. So caution should be given and deference to whichever version is newer (which rsync should take care of, but that will depend on the date each file was created and perhaps not the actual data within).<br />
<br />
If we look further in the file we see:<br />
<br />
x86/psa-appvault-moodle-1.8-8202920080409011254.noarch/usr/local/psa/var/cgitory/moodle-1.<br />
9/htdocs/lib/editor/htmlarea/plugins/TableOperations/img/cell-split.gif<br />
<br />
and other files looking like:<br />
5/x86/plesk-base-9.0.1-cos5.build90090127.18.i586/etc/sw/keys/info<br />
<br />
Which means plesk is here. We don't need plesk on v12 (the CT moving over doesn't use it) so we rerun the rsync and exclude it:<br />
<br />
[root@virt19 /vz/template]# rsync -an -e ssh --exclude="plesk*" --exclude="psa*" /vz/template/centos/ root@10.1.4.62:/vz/template/centos/ > v12_c5<br />
<br />
and now it's much smaller:<br />
<br />
[root@virt19 /vz/template]# cat v12_c5|wc -l<br />
97104<br />
<br />
Before running with excludes it was:<br />
[root@virt19 /vz/template]# cat v12_c5|wc -l <br />
238238<br />
<br />
So again, look at what you're doing. Generally, combining templates is fine however.<br />
<br />
If you're happy with the merge, run the rsync again without the "-n" to let it actually do the transfer.<br />
<br />
Once done, don't forget to add the new template to the HN in Management -> Reference -> Templates<br />
<br />
= getting help =<br />
<br />
= vzup2date =<br />
TODO <br />
= Adding new OS/application templates =<br />
<br />
Virtuozzo will lag a bit behind the initial release of an OS, perhaps by a month or more. Further, a newly-released OS may be available, but there may not be many/any application templates available initially. It pays to wait a bit.<br />
<br />
A template is easily installed via [[#vzup2date|vzup2date]]:<br />
<br />
vzup2date -z<br />
<br />
You will be given a list of OS's to install. Select an OS, then customize that selection by checking/unchecking the application templates we do/don't want to include/offer.<br />
<br />
Generally, here are the app templates we include/offer:<br />
<pre>cyrus-imap<br />
devel<br />
imp<br />
jre<br />
jsdk<br />
mailman<br />
mod_perl<br />
mod_ssl<br />
mysql<br />
php<br />
phpmyadmin<br />
phppgadmin<br />
postgresql<br />
proftpd<br />
spamassassin<br />
squirrelmail<br />
tomcat<br />
vsftpd<br />
</pre><br />
<br />
We do not (any longer) offer webalizer or analog.<br />
<br />
After selecting and installing the OS and apps, you should ensure it's installed:<br />
<br />
vzpkg list -O<br />
<br />
Your new OS (ex. fedora-core-13-x86_64) will be listed, without a corresponding cache date:<br />
<pre><br />
[root@virt13 /vzconf]# vzpkg list -O<br />
centos-6-x86 2011-07-22 13:24:41<br />
centos-6-x86_64 2012-03-09 20:42:22<br />
centos-5-x86 2009-07-27 16:58:24<br />
centos-5-x86_64 2012-03-09 22:38:53<br />
ubuntu-11.10-x86_64 2012-03-01 23:13:33<br />
ubuntu-9.04-x86 2009-06-02 11:32:39<br />
ubuntu-12.04-x86_64 2012-05-29 13:50:50<br />
ubuntu-8.04-x86 2008-09-17 21:27:20<br />
ubuntu-8.10-x86 2009-03-13 13:58:57<br />
ubuntu-11.04-x86 2011-12-05 11:15:46<br />
ubuntu-7.04-x86 2008-09-24 16:42:50<br />
redhat-el6-x86_64<br />
fedora-core-13-x86_64 <br />
fedora-core-12-x86 2010-02-06 13:24:32<br />
fedora-core-15-x86 2011-12-05 11:36:43<br />
fedora-core-11-x86 2009-10-01 16:30:59<br />
fedora-core-14-x86<br />
fedora-core-16-x86_64 2012-03-13 11:40:23<br />
fedora-core-9-x86 2008-11-29 10:52:18<br />
debian-6.0-x86 2011-07-29 16:00:35<br />
debian-6.0-x86_64 2012-03-12 19:53:33<br />
debian-5.0-x86 2009-03-12 12:39:11<br />
</pre><br />
<br />
You can also confirm the apps installed:<br />
<pre>[root@virt13 /vzconf]# vzpkg list fedora-core-13-x86_64<br />
fedora-core-13-x86_64 2013-03-08 11:39:44<br />
fedora-core-13-x86_64 devel<br />
fedora-core-13-x86_64 php<br />
fedora-core-13-x86_64 proftpd<br />
fedora-core-13-x86_64 postgresql<br />
fedora-core-13-x86_64 phpmyadmin<br />
fedora-core-13-x86_64 mysql<br />
</pre><br />
<br />
Before you can use the OS you must cache it. To create the cache run:<br />
vzpkg cache [OS]<br />
<br />
ex: <tt>vzpkg cache fedora-core-13-x86_64</tt><br />
<br />
Now it's ready to use. In order to be able to use it with the [[VPS_Management#vm|vm]] command, you must create a file:<br />
jctmpl-[OS]<br />
<br />
ex: <tt>jctmpl-fedora-core-13-x86_64</tt><br />
<br />
In it, on line 1 we place the OS, followed by the app names, ex:<br />
<pre>more jctmpl-fedora-core-13-x86_64<br />
fedora-core-13-x86_64<br />
postgresql<br />
jsdk<br />
mod_ssl<br />
phpmyadmin<br />
mod_perl<br />
tomcat<br />
devel<br />
php<br />
mailman<br />
cyrus-imap<br />
squirrelmail<br />
proftpd<br />
mysql<br />
phppgadmin<br />
spamassassin<br />
</pre><br />
<br />
The packages will be installed in that order, so any pre-req's should be handled there.<br />
<br />
Now, when you run [[VPS_Management#vm|vm]] it will show you the newly-added OS and you may create a CT using it.<br />
<br />
The last few tasks are related to mgmt:<br />
<br />
1. add to mgmt->reference->templates (use the existing templates as an example). Once added, the new template will be included with the list of OS's available for the given HN.<br />
2. if this is going to be a new OS offered for ordering, it needs to be added to the order form and webpage.<br />
<br />
a. to get a list of applications and versions in the new OS, you should create a test CT, enter it and run <tt>dpkg -l|sort</tt> (or <tt>rpm -aq|sort</tt> in centos/fedora) and enter that list in the following places:<br><br />
<tt>/usr/local/www/jc_pub/[osname].html</tt><br><br />
<tt>/usr/local/www/signup/html/[osname].html</tt><br><br />
Note, you will have to move the most recent OS data from <tt>/usr/local/www/signup/html/[osname].html</tt> into the corresponding <tt>/usr/local/www/signup/html/[osname]_old.html</tt> updating the version links in both.<br />
<br />
b. to add the new OS to the order form, you will need to update <tt>/usr/local/www/signup/html/step1.html</tt> in several places (for regular and OSS orders) as well as updating the templateID which you can get from mgmt->reference->templates (or [[Management_System_/_Public_Website_/_Signup_/_Account_Manager#jc:ref_templates|jc:ref_templates]])<br />
<br />
<br />
<br />
Old method to obtain the templates:<br />
<br />
Go to http://www.parallels.com/en/virtuozzo/templates/catalog/<br />
<br />
Download the RPM's and install them<br />
<br />
vzpkg list<br />
to see the new packages/os templates<br />
<br />
vzpkg create cache "OS_EZ_template_name"<br />
to create a cache<br />
<br />
= Updating kernel on virt =<br />
<br />
We now use [[#vzup2date|vzup2date]] to update systems and kernels. What follows is what we used to do when we had to download kernels and install manually on old systems:<br />
<br />
<br />
1. install kernel and modules<br />
<pre># rpm -ivh vzkernel-enterprise-2.4.20-021stab028.12.777.i686.rpm \<br />
vzmodules-enterprise-2.4.20-021stab028.12.swsoft.i686.rpm<br />
vzkernel-smp ##################################################<br />
vzmodules-smp ##################################################</pre><br />
<br />
DO NOT USE the "rpm -Uhv" command to install the kernel, otherwise all the previously installed kernels may be removed from your system. <br />
<br />
2. update lilo/grub<br />
<br />
Lilo:<br />
<br />
vi /etc/lilo.conf<br />
<br />
rename old Virtuozzo kernel label from "linux+virtuozzo" to "virtuozzo_old"<br />
and add a new entry pointing to the newly installed virtuozzo kernel image.<br />
For example:<br />
<br />
<pre>prompt<br />
timeout=50<br />
default=linux+virtuozzo<br />
append="debug panic=5 console=tty console=ttyS0,38400"<br />
boot=/dev/sda<br />
map=/boot/map<br />
install=/boot/boot.b<br />
message=/boot/message<br />
linear<br />
<br />
image=/boot/vmlinuz-2.4.18-3smp<br />
label=linux<br />
initrd=/boot/initrd-2.4.18-3smp.img<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.18-3<br />
label=linux-up<br />
initrd=/boot/initrd-2.4.18-3.img<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-020stab009.5.777-enterprise<br />
label=2.4.20-9.5.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-020stab009.8.777-enterprise<br />
label=2.4.20-9.8.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-020stab009.21.777-enterprise<br />
label=2.4.20-9.21.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-020stab009.24.777-enterprise<br />
label=2.4.20-9.24.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-021stab028.3.777-enterprise<br />
label=2.4.20-28.3.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-021stab028.5.777-enterprise<br />
label=old_linux+vz<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-021stab028.12.777-enterprise<br />
label=linux+virtuozzo<br />
read-only<br />
root=/dev/sda1</pre><br />
<br />
With this configuration, the new kernel is the default kernel to use at boot. The original kernel entry has been retained with the label "old_linux+vz ".<br />
<br />
Grub:<br />
<br />
vi /boot/grub /menu.lst<br />
<br />
Add new entry at the top.<br />
For example:<br />
<br />
<pre>serial --unit=0 --speed=38400<br />
terminal --timeout=10 serial console<br />
default=0<br />
timeout=10<br />
splashimage=(hd0,0)/boot/grub/splash.xpm.gz<br />
title Virtuozzo 2.6.1 (2.4.20-021stab028.12.777-enterprise)<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-22.4.20-021stab028.12.777-enterprise root=/dev/sda1 console=tty0 \ console=ttyS0,38400<br />
title Virtuozzo 2.6.1 (2.4.20-021stab028.3.777-enterprise)<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.20-021stab028.3.777-enterprise root=/dev/sda1 console=tty0 \ console=ttyS0,38400<br />
title Red Hat Linux (2.4.18-3smp)<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.18-3smp ro root=/dev/sda1<br />
initrd /boot/initrd-2.4.18-3smp.img<br />
title Red Hat Linux-up (2.4.18-3)<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.18-3 ro root=/dev/sda1<br />
initrd /boot/initrd-2.4.18-3.img<br />
title Linux with Virtuozzo 2.5<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.20-020stab009.3.777-smp ro root=/dev/sda1<br />
title vz-enterpriseo<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.20-020stab009.21.777-enterprise ro root=/dev/sda1<br />
title vz-enterprise<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.20-020stab009.24.777-enterprise ro root=/dev/sda1 \ console=tty0 console=ttyS0,38400</pre><br />
<br />
3. run the lilo (if using lilo)<br />
<br />
<pre># lilo<br />
Added linux<br />
Added linux-up<br />
Added virtuozzo_old<br />
Added linux+virtuozzo *</pre><br />
<br />
4. reboot the server<br />
shutdown -r 23:05<br />
<br />
when it comes time for the shutdown, run [[VPS_Management#stopvirt|stopvirt]] to get things shut down faster<br />
<br />
<br />
= Remaking service VE (<=3.x only) =<br />
<br />
Occasionally the control panel for VEs (aka VZPP) will stop working and you just need to reinstall the service VE (which runs the control panels for all VEs on the HN). Here's how that's done :<br />
<br />
<pre>vzctl stop 1<br />
ipdel 1 69.55.234.152<br />
vzmlocal 1:2<br />
vzctl set 2 --save --onboot no<br />
vzsveinstall -t redhat-as3-minimal -d /root/Rel261/HW/RPMS -s 69.55.234.152 --skip-client</pre><br />
<br />
on 3.0:<br />
<br />
<pre>vzsveinstall -t redhat-as3-minimal -d /vz/Rel300/HW/RPMS -s 69.55.229.151 --skip-client<br />
<br />
vzctl stop 1<br />
vzctl mount 1<br />
vzctl mount 2<br />
/bin/cp -aRf /vz/root/2/home/vzagent0 /vz/root/1/home<br />
/bin/cp -aRf /vz/root/2/var/lib/mysql/VZADB /vz/root/1/var/lib/mysql<br />
/bin/cp -af /vz/root/2/etc/shadow /vz/root/1/etc/<br />
/bin/cp -aRf /vz/root/2/var/vzcp/static/vz/skins/ /vz/root/1/var/vzcp/static/vz<br />
/bin/cp -af /vz/root/2/etc/vzcp/pp/menu.xml /vz/root/1/etc/vzcp/pp/<br />
/bin/cp -af /vz/root/2/etc/vzcp/pp/dashboard.xml /vz/root/1/etc/vzcp/pp/<br />
<br />
vzctl umount 2<br />
vzctl umount 1<br />
vzctl start 1</pre></div>Supporthttps://wiki.jcihosting.com/index.php?title=Virtuozzo_/_Linux_Reference&diff=1028Virtuozzo / Linux Reference2013-03-11T23:37:31Z<p>Support: /* Adding new OS/application templates */</p>
<hr />
<div>= Updating VZ =<br />
<br />
Update Virtuozzo to version 4.0.0<br />
Apply minor updates for Virtuozzo 3.0.0<br />
<br />
...<br />
When you get to the last screen, you will be given an option to <br />
<br />
( ) Automatically reboot after update<br />
(*) Reboot later manually<br />
<br />
make sure you choose to reboot later.<br />
<br />
= Migrating Templates =<br />
One nice feature VZ offers is the ability to run a virtual environment (VE or CT as it's now called) based on a particular ditribution on a host which may or may not share the same distribution. For instance, you could run a CT running Ubuntu while the host machine (Hardware Node or HN) is running CentOS. This is accomplished via the use of OS templates. As long as the HN contains the OS template on which a particular CT relies, it will run there. As such, if you want to move a CT from one HN to another, you will need to make sure the OS template exists there.<br />
<br />
Modern versions of VZ (>= 4.x) will check for and automatically move the OS template over to the host as you're (vz)migrating the CT over to a new HN. However we like to make sure the template is in place prior to moving the CT. <br />
<br />
The first step is to see if the template exists on the host. To see OS templates installed:<br />
<br />
2.x (shows OS and application templates):<br />
<pre># vzpkgls<br />
mod_ssl-rh9 20040624<br />
mysql-rh9 20030814 20050412<br />
openwebmail-rh9 20050427<br />
php-rh9 20050506<br />
phpBB-rh9 20041229<br />
postgresql-rh9 20050719<br />
sl-webalizer-rh9 20040119<br />
spamassassin-rh9 20040127<br />
tomcat-rh9 20040524<br />
usermin-rh9 20040116</pre><br />
<br />
>= 3.x (shows just OS templates, run without -O to see application as well):<br />
<pre># vzpkg list -O<br />
ubuntu-10.04-x86 2010-06-01 11:39:05<br />
ubuntu-11.04-x86 2011-06-22 14:23:59<br />
ubuntu-9.10-x86 2010-03-11 17:39:51<br />
centos-5-x86 2010-03-11 17:12:27<br />
debian-5.0-x86 2010-03-11 17:33:34</pre><br />
<br />
<br />
All template files are located:<br />
<br />
<pre># ls /vz/template/<br />
ColdFusion-fc1 jre-fc1 openwebmail-fc1 sl-webalizer-fc1<br />
ColdFusion-fc2 jre-fc2 openwebmail-fc2 sl-webalizer-fc2<br />
ColdFusion-rh9 jre-rh9 openwebmail-rh9 sl-webalizer-rh9<br />
PostNuke-fc1 jre-suse92 openwebmail-suse92 spamassassin-fc1<br />
PostNuke-fc2 jrun php-deb31 spamassassin-fc2<br />
PostNuke-rh9 mailman-deb31 php-fc1 spamassassin-rh9<br />
analog-fc1 mailman-fc1 php-fc2 spamassassin-suse92<br />
analog-fc2 mailman-fc2 php-rh9 suse-9.2<br />
analog-rh9 mailman-rh9 php-suse92 tomcat-fc1<br />
awstats-fc1 mailman-suse92 phpBB-fc1 tomcat-fc2<br />
awstats-rh9 mod_frontpage-fc1 phpBB-fc2 tomcat-rh9<br />
bbClone-fc1 mod_frontpage-fc2 phpBB-rh9 tomcat-suse92<br />
bbClone-fc2 mod_frontpage-rh9 phpmyadmin-deb31 urchin-fc1<br />
bbClone-rh9 mod_frontpage-suse92 postgresql-deb31 usermin-fc1<br />
conf mod_perl-deb31 postgresql-fc1 usermin-fc2<br />
debian-3.1 mod_perl-fc1 postgresql-fc2 usermin-rh9<br />
devel-deb31 mod_perl-fc2 postgresql-rh9 uw-imap-fc1<br />
devel-fc2 mod_perl-rh9 postgresql-suse92 uw-imap-fc2<br />
devel-suse92 mod_perl-suse92 proftpd-deb31 uw-imap-rh9<br />
fedora-core-1 mod_ssl-fc1 proftpd-fc1 vzredhat-7.3<br />
fedora-core-2 mod_ssl-fc2 proftpd-fc2 vzredhat-8.0<br />
jdk-deb31 mod_ssl-rh9 proftpd-rh9 webalizer-suse92<br />
jdk-fc1 mysql-deb31 proftpd-suse92 webmin-fc1<br />
jdk-fc2 mysql-fc1 redhat-7.2 webmin-fc2<br />
jdk-rh9 mysql-fc2 redhat-9 webmin-rh9<br />
jdk-suse92 mysql-rh9 redhat-as3-minimal<br />
jre-deb31 mysql-suse92 redhat-devel-9<br />
</pre><br />
<br />
What we see here are the base OS templates: <tt>redhat-9, fedora-core-1</tt> and supporting application templates: <tt>postgresql-rh9, mailman-rh9, jdk-fc1, mod_ssl-fc1</tt>. So if you wanted to copy over all of redhat 9 you'd need to copy the contents of:<br />
<pre># ls -d /vz/template/*rh9*<br />
/vz/template/ColdFusion-rh9 /vz/template/mod_frontpage-rh9 /vz/template/proftpd-rh9<br />
/vz/template/PostNuke-rh9 /vz/template/mod_perl-rh9 /vz/template/sl-webalizer-rh9<br />
/vz/template/analog-rh9 /vz/template/mod_ssl-rh9 /vz/template/spamassassin-rh9<br />
/vz/template/awstats-rh9 /vz/template/mysql-rh9 /vz/template/tomcat-rh9<br />
/vz/template/bbClone-rh9 /vz/template/openwebmail-rh9 /vz/template/usermin-rh9<br />
/vz/template/jdk-rh9 /vz/template/php-rh9 /vz/template/uw-imap-rh9<br />
/vz/template/jre-rh9 /vz/template/phpBB-rh9 /vz/template/webmin-rh9<br />
/vz/template/mailman-rh9 /vz/template/postgresql-rh9<br />
<br />
# ls -d /vz/template/*redhat-9*<br />
/vz/template/redhat-9<br />
</pre><br />
<br />
To get a template from one HN to another, it simply needs to be rsync'd over to the target HN. It's rsync'd over in this fashion:<br />
<br />
rsync -av -e ssh /vz/template/redhat-9 root@10.1.4.65:/vz/template<br />
rsync -av -e ssh /vz/template/*rh9 root@10.1.4.65:/vz/template<br />
<br />
Newer (>= 3.x) VZ employs "EZ" templates which are organized a little differently:<br />
<br />
<pre># ls /vz/template/<br />
cache CmdTool.log debian redhat-as3-minimal-20080630.tar.gz vc<br />
centos conf redhat-as3-minimal ubuntu<br />
<br />
# ls /vz/template/ubuntu/<br />
10.04 11.04 9.10<br />
# ls /vz/template/ubuntu/10.04/<br />
x86<br />
# ls /vz/template/ubuntu/10.04/x86/<br />
aap_1.091-1_all<br />
aap-doc_1.091-1_all<br />
adduser_3.112ubuntu1_all<br />
anacron_2.3-13.1ubuntu11_i386<br />
apache2_2.2.14-5ubuntu8.2_i386<br />
apache2_2.2.14-5ubuntu8.4_i386<br />
apache2_2.2.14-5ubuntu8.6_i386<br />
apache2_2.2.14-5ubuntu8.7_i386<br />
apache2_2.2.14-5ubuntu8.8_i386<br />
apache2_2.2.14-5ubuntu8.9_i386<br />
<...snip><br />
<br />
# ls /vz/template/cache/<br />
centos-5-x86.tar.gz suse-11.3-x86.tar.gz ubuntu-9.10-x86.tar.gz<br />
debian-5.0-x86.tar.gz ubuntu-10.04-x86.tar.gz<br />
fedora-core-14-x86.tar.gz ubuntu-11.04-x86.tar.gz<br />
</pre><br />
<br />
So what we see is the entire OS and all applications are in 1 directory, and organized by distribution and release version. So to move Ubuntu 10.04:<br />
<br />
rsync -av -e ssh /vz/template/ubuntu/10.04 root@10.1.4.69:/vz/template/ubuntu<br />
<br />
and you also need to move the cache:<br />
<br />
rsync -av -e ssh /vz/template/cache/ubuntu-10.04-x86.tar.gz root@10.1.4.69:/vz/template/cache/<br />
<br />
Once the transfer is complete, you will be able to run <tt>vzpkgls</tt> or <tt>vzpkg list -O</tt> and see the template(s) listed on the target HN.<br />
<br />
So far, this all supposes template doesn't exist on the target- very simple, just copy it over. If the template exists already on the target HN, this requires closer inspection. With older templates, simply moving over the templates would be the end of it- you see the version listed when you do a <tt>vzplgls</tt>:<br />
<br />
mysql-rh9 20030814 20050412<br />
<br />
this means there are 2 versions of the mysql package for RH9: 20030814 and 20050412<br />
As an aside, you can't copy over just 1 of them, but if you rsync'd over the <tt>mysql-rh9</tt> directory, you'd see 20030814 and 20050412 on the target HN. As long as the versions match up, you're good to go.<br />
<br />
With EZ templates, the versions are not as easy to decipher. When you look at the <tt>vzpkg list</tt> output, that simply tells you when a cache was created. A cache is simply a snapshot of all the data needed for the latest versions of the OS and it's applications at the time the cache was created. It is a date, and thus tells you nothing about the versions within. So for instance, if you had a cache for Ubuntu 10.04 from 2007 and you ran <tt>vzpkg create cache ubuntu-10.04-x86</tt> it would re-download the latest version of apache, mysql and so on. And any ''subsequent'' ubuntu 10.04 CT's created thereafter would use those newer versions, whereas older CT's based on 10.04 would use older templates. You can see how this works by looking at the files under:<br />
<br />
<pre># ls /vz/template/ubuntu/10.04/x86/<br />
aap_1.091-1_all<br />
aap-doc_1.091-1_all<br />
adduser_3.112ubuntu1_all<br />
anacron_2.3-13.1ubuntu11_i386<br />
apache2_2.2.14-5ubuntu8.2_i386<br />
apache2_2.2.14-5ubuntu8.4_i386<br />
apache2_2.2.14-5ubuntu8.6_i386<br />
apache2_2.2.14-5ubuntu8.7_i386<br />
apache2_2.2.14-5ubuntu8.8_i386<br />
apache2_2.2.14-5ubuntu8.9_i386<br />
<...snip><br />
</pre><br />
<br />
You can see that this template has in it 6 different versions of apache2. This means that this template was merged with other 10.04 templates and/or it has been cached a few times, each time a new apache was available. The unfortunate thing is it's almost impossible to know which CT's are using which version of apache so it's hard to try to prune this down and remove unwanted/unneeded applications.<br />
<br />
So, if you see another HN has Ubuntu 10.04 on it, you need to see/confirm that it has all the exact files your CT needs. You can run a test rsync to see what ''would'' be transferred (what's missing):<br />
<br />
rsync -avn --stats -e ssh /vz/template/ubuntu/10.04 root@10.1.4.69:/vz/template/ubuntu<br />
<br />
Based on the amount of data you get back from the rsync, you will be able to decide whether or not to remove the -n and allow the full transfer to go through. The downside to doing the transfer is the situation described above- you've permanently joined these templates and now you may have 10 versions of apache on the target HN. There's one other potential pitfall to this kind of merging- it will also potentially copy over shared files, for which there is no particular version, most of which are in <tt>/vz/template/ubuntu/10.04/x86/config</tt>: <br />
<br />
cat /vz/template/ubuntu/10.04/x86/config/os/default/repositories<br />
/vz/template/ubuntu/10.04/x86/config/os/default/repositories<br />
<br />
Here are some specific things to look out for. Case: copying centos-5 on virt19 to virt12. We dumped the rsync output to a file so we could inspect:<br />
<br />
[root@virt19 /vz/template]# rsync -an -e ssh /vz/template/centos/ root@10.1.4.62:/vz/template/centos/ > v12_c5<br />
<br />
(note, that can be dangerous, better to add on full path <tt>/vz/template/centos/5</tt>)<br />
<br />
So in the output file we see things like:<br />
<br />
<pre><br />
x86/base0/cachecookie<br />
x86/base0/primary.xml.gz <br />
x86/base0/primary.xml.gz.sqlite <br />
x86/base0/repomd.xml <br />
x86/base1/cachecookie <br />
x86/base1/filelists.xml.gz <br />
x86/base1/filelists.xml.gz.sqlite <br />
x86/base1/primary.xml.gz <br />
x86/base1/primary.xml.gz.sqlite <br />
x86/base1/repomd.xml<br />
x86/base2/cachecookie<br />
</pre><br />
<br />
Which make us nervous cause those are not versioned files, i.e.:<br />
5/x86/MAKEDEV-3.23-1.2.i386/usr/sbin/mksock<br />
<br />
So those files will replace existing files on v12 rather than the case of the latter where it's likely being added. So caution should be given and deference to whichever version is newer (which rsync should take care of, but that will depend on the date each file was created and perhaps not the actual data within).<br />
<br />
If we look further in the file we see:<br />
<br />
x86/psa-appvault-moodle-1.8-8202920080409011254.noarch/usr/local/psa/var/cgitory/moodle-1.<br />
9/htdocs/lib/editor/htmlarea/plugins/TableOperations/img/cell-split.gif<br />
<br />
and other files looking like:<br />
5/x86/plesk-base-9.0.1-cos5.build90090127.18.i586/etc/sw/keys/info<br />
<br />
Which means plesk is here. We don't need plesk on v12 (the CT moving over doesn't use it) so we rerun the rsync and exclude it:<br />
<br />
[root@virt19 /vz/template]# rsync -an -e ssh --exclude="plesk*" --exclude="psa*" /vz/template/centos/ root@10.1.4.62:/vz/template/centos/ > v12_c5<br />
<br />
and now it's much smaller:<br />
<br />
[root@virt19 /vz/template]# cat v12_c5|wc -l<br />
97104<br />
<br />
Before running with excludes it was:<br />
[root@virt19 /vz/template]# cat v12_c5|wc -l <br />
238238<br />
<br />
So again, look at what you're doing. Generally, combining templates is fine however.<br />
<br />
If you're happy with the merge, run the rsync again without the "-n" to let it actually do the transfer.<br />
<br />
Once done, don't forget to add the new template to the HN in Management -> Reference -> Templates<br />
<br />
= getting help =<br />
<br />
= vzup2date =<br />
TODO <br />
= Adding new OS/application templates =<br />
<br />
Virtuozzo will lag a bit behind the initial release of an OS, perhaps by a month or more. Further, a newly-released OS may be available, but there may not be many/any application templates available initially. It pays to wait a bit.<br />
<br />
A template is easily installed via [[#vzup2date|vzup2date]]:<br />
<br />
vzup2date -z<br />
<br />
You will be given a list of OS's to install. Select an OS, then customize that selection by checking/unchecking the application templates we do/don't want to include/offer.<br />
<br />
Generally, here are the app templates we include/offer:<br />
<pre>cyrus-imap<br />
devel<br />
imp<br />
jre<br />
jsdk<br />
mailman<br />
mod_perl<br />
mod_ssl<br />
mysql<br />
php<br />
phpmyadmin<br />
phppgadmin<br />
postgresql<br />
proftpd<br />
spamassassin<br />
squirrelmail<br />
tomcat<br />
vsftpd<br />
</pre><br />
<br />
We do not (any longer) offer webalizer or analog.<br />
<br />
After selecting and installing the OS and apps, you should ensure it's installed:<br />
<br />
vzpkg list -O<br />
<br />
Your new OS (ex. fedora-core-13-x86_64) will be listed, without a corresponding cache date:<br />
<pre><br />
[root@virt13 /vzconf]# vzpkg list -O<br />
centos-6-x86 2011-07-22 13:24:41<br />
centos-6-x86_64 2012-03-09 20:42:22<br />
centos-5-x86 2009-07-27 16:58:24<br />
centos-5-x86_64 2012-03-09 22:38:53<br />
ubuntu-11.10-x86_64 2012-03-01 23:13:33<br />
ubuntu-9.04-x86 2009-06-02 11:32:39<br />
ubuntu-12.04-x86_64 2012-05-29 13:50:50<br />
ubuntu-8.04-x86 2008-09-17 21:27:20<br />
ubuntu-8.10-x86 2009-03-13 13:58:57<br />
ubuntu-11.04-x86 2011-12-05 11:15:46<br />
ubuntu-7.04-x86 2008-09-24 16:42:50<br />
redhat-el6-x86_64<br />
fedora-core-13-x86_64 <br />
fedora-core-12-x86 2010-02-06 13:24:32<br />
fedora-core-15-x86 2011-12-05 11:36:43<br />
fedora-core-11-x86 2009-10-01 16:30:59<br />
fedora-core-14-x86<br />
fedora-core-16-x86_64 2012-03-13 11:40:23<br />
fedora-core-9-x86 2008-11-29 10:52:18<br />
debian-6.0-x86 2011-07-29 16:00:35<br />
debian-6.0-x86_64 2012-03-12 19:53:33<br />
debian-5.0-x86 2009-03-12 12:39:11<br />
</pre><br />
<br />
You can also confirm the apps installed:<br />
<pre>[root@virt13 /vzconf]# vzpkg list fedora-core-13-x86_64<br />
fedora-core-13-x86_64 2013-03-08 11:39:44<br />
fedora-core-13-x86_64 devel<br />
fedora-core-13-x86_64 php<br />
fedora-core-13-x86_64 proftpd<br />
fedora-core-13-x86_64 postgresql<br />
fedora-core-13-x86_64 phpmyadmin<br />
fedora-core-13-x86_64 mysql<br />
</pre><br />
<br />
Before you can use the OS you must cache it. To create the cache run:<br />
vzpkg cache [OS]<br />
<br />
ex: <tt>vzpkg cache fedora-core-13-x86_64</tt><br />
<br />
Now it's ready to use. In order to be able to use it with the [[VPS_Management#vm|vm]] command, you must create a file:<br />
jctmpl-[OS]<br />
<br />
ex: <tt>jctmpl-fedora-core-13-x86_64</tt><br />
<br />
In it, on line 1 we place the OS, followed by the app names, ex:<br />
<pre>more jctmpl-fedora-core-13-x86_64<br />
fedora-core-13-x86_64<br />
postgresql<br />
jsdk<br />
mod_ssl<br />
phpmyadmin<br />
mod_perl<br />
tomcat<br />
devel<br />
php<br />
mailman<br />
cyrus-imap<br />
squirrelmail<br />
proftpd<br />
mysql<br />
phppgadmin<br />
spamassassin<br />
</pre><br />
<br />
The packages will be installed in that order, so any pre-req's should be handled there.<br />
<br />
Now, when you run [[VPS_Management#vm|vm]] it will show you the newly-added OS and you may create a CT using it.<br />
<br />
The last few tasks are related to mgmt:<br />
<br />
1. add to mgmt->reference->templates (use the existing templates as an example). Once added, the new template will be included with the list of OS's available for the given HN.<br />
2. if this is going to be a new OS offered for ordering, it needs to be added to the order form and webpage.<br />
<br />
a. to get a list of applications and versions in the new OS, you should create a test CT, enter it and run <tt>dpkg -l|sort</tt> (or <tt>rpm -aq|sort</tt> in centos/fedora) and enter that list in the following places:<br><br />
<tt>/usr/local/www/jc_pub/[osname].html</tt><br><br />
<tt>/usr/local/www/signup/html/[osname].html</tt><br><br />
Note, you will have to move the most recent OS data from <tt>/usr/local/www/signup/html/[osname].html</tt> into the corresponding <tt>/usr/local/www/signup/html/[osname]_old.html</tt> updating the version links in both.<br />
<br />
b. to add the new OS to the order form, you will need to update <tt>/usr/local/www/signup/html/step1.html</tt> in several places (for regular and OSS orders) as well as updating the templateID which you can get from mgmt->reference->templates (or [[Management_System_/_Public_Website_/_Signup_/_Account_Manager#jc:ref_templates|jc:ref_templates]])<br />
<br />
<br />
<br />
Old method:<br />
<br />
Go to http://www.parallels.com/en/virtuozzo/templates/catalog/<br />
<br />
Download the RPM's and install them<br />
<br />
vzpkg list<br />
to see the new packages/os templates<br />
<br />
vzpkg create cache "OS_EZ_template_name"<br />
to create a cache<br />
<br />
= Updating kernel on virt =<br />
<br />
We now use [[#vzup2date|vzup2date]] to update systems and kernels. What follows is what we used to do when we had to download kernels and install manually on old systems:<br />
<br />
<br />
1. install kernel and modules<br />
<pre># rpm -ivh vzkernel-enterprise-2.4.20-021stab028.12.777.i686.rpm \<br />
vzmodules-enterprise-2.4.20-021stab028.12.swsoft.i686.rpm<br />
vzkernel-smp ##################################################<br />
vzmodules-smp ##################################################</pre><br />
<br />
DO NOT USE the "rpm -Uhv" command to install the kernel, otherwise all the previously installed kernels may be removed from your system. <br />
<br />
2. update lilo/grub<br />
<br />
Lilo:<br />
<br />
vi /etc/lilo.conf<br />
<br />
rename old Virtuozzo kernel label from "linux+virtuozzo" to "virtuozzo_old"<br />
and add a new entry pointing to the newly installed virtuozzo kernel image.<br />
For example:<br />
<br />
<pre>prompt<br />
timeout=50<br />
default=linux+virtuozzo<br />
append="debug panic=5 console=tty console=ttyS0,38400"<br />
boot=/dev/sda<br />
map=/boot/map<br />
install=/boot/boot.b<br />
message=/boot/message<br />
linear<br />
<br />
image=/boot/vmlinuz-2.4.18-3smp<br />
label=linux<br />
initrd=/boot/initrd-2.4.18-3smp.img<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.18-3<br />
label=linux-up<br />
initrd=/boot/initrd-2.4.18-3.img<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-020stab009.5.777-enterprise<br />
label=2.4.20-9.5.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-020stab009.8.777-enterprise<br />
label=2.4.20-9.8.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-020stab009.21.777-enterprise<br />
label=2.4.20-9.21.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-020stab009.24.777-enterprise<br />
label=2.4.20-9.24.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-021stab028.3.777-enterprise<br />
label=2.4.20-28.3.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-021stab028.5.777-enterprise<br />
label=old_linux+vz<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-021stab028.12.777-enterprise<br />
label=linux+virtuozzo<br />
read-only<br />
root=/dev/sda1</pre><br />
<br />
With this configuration, the new kernel is the default kernel to use at boot. The original kernel entry has been retained with the label "old_linux+vz ".<br />
<br />
Grub:<br />
<br />
vi /boot/grub /menu.lst<br />
<br />
Add new entry at the top.<br />
For example:<br />
<br />
<pre>serial --unit=0 --speed=38400<br />
terminal --timeout=10 serial console<br />
default=0<br />
timeout=10<br />
splashimage=(hd0,0)/boot/grub/splash.xpm.gz<br />
title Virtuozzo 2.6.1 (2.4.20-021stab028.12.777-enterprise)<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-22.4.20-021stab028.12.777-enterprise root=/dev/sda1 console=tty0 \ console=ttyS0,38400<br />
title Virtuozzo 2.6.1 (2.4.20-021stab028.3.777-enterprise)<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.20-021stab028.3.777-enterprise root=/dev/sda1 console=tty0 \ console=ttyS0,38400<br />
title Red Hat Linux (2.4.18-3smp)<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.18-3smp ro root=/dev/sda1<br />
initrd /boot/initrd-2.4.18-3smp.img<br />
title Red Hat Linux-up (2.4.18-3)<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.18-3 ro root=/dev/sda1<br />
initrd /boot/initrd-2.4.18-3.img<br />
title Linux with Virtuozzo 2.5<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.20-020stab009.3.777-smp ro root=/dev/sda1<br />
title vz-enterpriseo<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.20-020stab009.21.777-enterprise ro root=/dev/sda1<br />
title vz-enterprise<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.20-020stab009.24.777-enterprise ro root=/dev/sda1 \ console=tty0 console=ttyS0,38400</pre><br />
<br />
3. run the lilo (if using lilo)<br />
<br />
<pre># lilo<br />
Added linux<br />
Added linux-up<br />
Added virtuozzo_old<br />
Added linux+virtuozzo *</pre><br />
<br />
4. reboot the server<br />
shutdown -r 23:05<br />
<br />
when it comes time for the shutdown, run [[VPS_Management#stopvirt|stopvirt]] to get things shut down faster<br />
<br />
<br />
= Remaking service VE (<=3.x only) =<br />
<br />
Occasionally the control panel for VEs (aka VZPP) will stop working and you just need to reinstall the service VE (which runs the control panels for all VEs on the HN). Here's how that's done :<br />
<br />
<pre>vzctl stop 1<br />
ipdel 1 69.55.234.152<br />
vzmlocal 1:2<br />
vzctl set 2 --save --onboot no<br />
vzsveinstall -t redhat-as3-minimal -d /root/Rel261/HW/RPMS -s 69.55.234.152 --skip-client</pre><br />
<br />
on 3.0:<br />
<br />
<pre>vzsveinstall -t redhat-as3-minimal -d /vz/Rel300/HW/RPMS -s 69.55.229.151 --skip-client<br />
<br />
vzctl stop 1<br />
vzctl mount 1<br />
vzctl mount 2<br />
/bin/cp -aRf /vz/root/2/home/vzagent0 /vz/root/1/home<br />
/bin/cp -aRf /vz/root/2/var/lib/mysql/VZADB /vz/root/1/var/lib/mysql<br />
/bin/cp -af /vz/root/2/etc/shadow /vz/root/1/etc/<br />
/bin/cp -aRf /vz/root/2/var/vzcp/static/vz/skins/ /vz/root/1/var/vzcp/static/vz<br />
/bin/cp -af /vz/root/2/etc/vzcp/pp/menu.xml /vz/root/1/etc/vzcp/pp/<br />
/bin/cp -af /vz/root/2/etc/vzcp/pp/dashboard.xml /vz/root/1/etc/vzcp/pp/<br />
<br />
vzctl umount 2<br />
vzctl umount 1<br />
vzctl start 1</pre></div>Supporthttps://wiki.jcihosting.com/index.php?title=Virtuozzo_/_Linux_Reference&diff=1027Virtuozzo / Linux Reference2013-03-11T23:37:12Z<p>Support: /* Adding new OS/application templates */</p>
<hr />
<div>= Updating VZ =<br />
<br />
Update Virtuozzo to version 4.0.0<br />
Apply minor updates for Virtuozzo 3.0.0<br />
<br />
...<br />
When you get to the last screen, you will be given an option to <br />
<br />
( ) Automatically reboot after update<br />
(*) Reboot later manually<br />
<br />
make sure you choose to reboot later.<br />
<br />
= Migrating Templates =<br />
One nice feature VZ offers is the ability to run a virtual environment (VE or CT as it's now called) based on a particular ditribution on a host which may or may not share the same distribution. For instance, you could run a CT running Ubuntu while the host machine (Hardware Node or HN) is running CentOS. This is accomplished via the use of OS templates. As long as the HN contains the OS template on which a particular CT relies, it will run there. As such, if you want to move a CT from one HN to another, you will need to make sure the OS template exists there.<br />
<br />
Modern versions of VZ (>= 4.x) will check for and automatically move the OS template over to the host as you're (vz)migrating the CT over to a new HN. However we like to make sure the template is in place prior to moving the CT. <br />
<br />
The first step is to see if the template exists on the host. To see OS templates installed:<br />
<br />
2.x (shows OS and application templates):<br />
<pre># vzpkgls<br />
mod_ssl-rh9 20040624<br />
mysql-rh9 20030814 20050412<br />
openwebmail-rh9 20050427<br />
php-rh9 20050506<br />
phpBB-rh9 20041229<br />
postgresql-rh9 20050719<br />
sl-webalizer-rh9 20040119<br />
spamassassin-rh9 20040127<br />
tomcat-rh9 20040524<br />
usermin-rh9 20040116</pre><br />
<br />
>= 3.x (shows just OS templates, run without -O to see application as well):<br />
<pre># vzpkg list -O<br />
ubuntu-10.04-x86 2010-06-01 11:39:05<br />
ubuntu-11.04-x86 2011-06-22 14:23:59<br />
ubuntu-9.10-x86 2010-03-11 17:39:51<br />
centos-5-x86 2010-03-11 17:12:27<br />
debian-5.0-x86 2010-03-11 17:33:34</pre><br />
<br />
<br />
All template files are located:<br />
<br />
<pre># ls /vz/template/<br />
ColdFusion-fc1 jre-fc1 openwebmail-fc1 sl-webalizer-fc1<br />
ColdFusion-fc2 jre-fc2 openwebmail-fc2 sl-webalizer-fc2<br />
ColdFusion-rh9 jre-rh9 openwebmail-rh9 sl-webalizer-rh9<br />
PostNuke-fc1 jre-suse92 openwebmail-suse92 spamassassin-fc1<br />
PostNuke-fc2 jrun php-deb31 spamassassin-fc2<br />
PostNuke-rh9 mailman-deb31 php-fc1 spamassassin-rh9<br />
analog-fc1 mailman-fc1 php-fc2 spamassassin-suse92<br />
analog-fc2 mailman-fc2 php-rh9 suse-9.2<br />
analog-rh9 mailman-rh9 php-suse92 tomcat-fc1<br />
awstats-fc1 mailman-suse92 phpBB-fc1 tomcat-fc2<br />
awstats-rh9 mod_frontpage-fc1 phpBB-fc2 tomcat-rh9<br />
bbClone-fc1 mod_frontpage-fc2 phpBB-rh9 tomcat-suse92<br />
bbClone-fc2 mod_frontpage-rh9 phpmyadmin-deb31 urchin-fc1<br />
bbClone-rh9 mod_frontpage-suse92 postgresql-deb31 usermin-fc1<br />
conf mod_perl-deb31 postgresql-fc1 usermin-fc2<br />
debian-3.1 mod_perl-fc1 postgresql-fc2 usermin-rh9<br />
devel-deb31 mod_perl-fc2 postgresql-rh9 uw-imap-fc1<br />
devel-fc2 mod_perl-rh9 postgresql-suse92 uw-imap-fc2<br />
devel-suse92 mod_perl-suse92 proftpd-deb31 uw-imap-rh9<br />
fedora-core-1 mod_ssl-fc1 proftpd-fc1 vzredhat-7.3<br />
fedora-core-2 mod_ssl-fc2 proftpd-fc2 vzredhat-8.0<br />
jdk-deb31 mod_ssl-rh9 proftpd-rh9 webalizer-suse92<br />
jdk-fc1 mysql-deb31 proftpd-suse92 webmin-fc1<br />
jdk-fc2 mysql-fc1 redhat-7.2 webmin-fc2<br />
jdk-rh9 mysql-fc2 redhat-9 webmin-rh9<br />
jdk-suse92 mysql-rh9 redhat-as3-minimal<br />
jre-deb31 mysql-suse92 redhat-devel-9<br />
</pre><br />
<br />
What we see here are the base OS templates: <tt>redhat-9, fedora-core-1</tt> and supporting application templates: <tt>postgresql-rh9, mailman-rh9, jdk-fc1, mod_ssl-fc1</tt>. So if you wanted to copy over all of redhat 9 you'd need to copy the contents of:<br />
<pre># ls -d /vz/template/*rh9*<br />
/vz/template/ColdFusion-rh9 /vz/template/mod_frontpage-rh9 /vz/template/proftpd-rh9<br />
/vz/template/PostNuke-rh9 /vz/template/mod_perl-rh9 /vz/template/sl-webalizer-rh9<br />
/vz/template/analog-rh9 /vz/template/mod_ssl-rh9 /vz/template/spamassassin-rh9<br />
/vz/template/awstats-rh9 /vz/template/mysql-rh9 /vz/template/tomcat-rh9<br />
/vz/template/bbClone-rh9 /vz/template/openwebmail-rh9 /vz/template/usermin-rh9<br />
/vz/template/jdk-rh9 /vz/template/php-rh9 /vz/template/uw-imap-rh9<br />
/vz/template/jre-rh9 /vz/template/phpBB-rh9 /vz/template/webmin-rh9<br />
/vz/template/mailman-rh9 /vz/template/postgresql-rh9<br />
<br />
# ls -d /vz/template/*redhat-9*<br />
/vz/template/redhat-9<br />
</pre><br />
<br />
To get a template from one HN to another, it simply needs to be rsync'd over to the target HN. It's rsync'd over in this fashion:<br />
<br />
rsync -av -e ssh /vz/template/redhat-9 root@10.1.4.65:/vz/template<br />
rsync -av -e ssh /vz/template/*rh9 root@10.1.4.65:/vz/template<br />
<br />
Newer (>= 3.x) VZ employs "EZ" templates which are organized a little differently:<br />
<br />
<pre># ls /vz/template/<br />
cache CmdTool.log debian redhat-as3-minimal-20080630.tar.gz vc<br />
centos conf redhat-as3-minimal ubuntu<br />
<br />
# ls /vz/template/ubuntu/<br />
10.04 11.04 9.10<br />
# ls /vz/template/ubuntu/10.04/<br />
x86<br />
# ls /vz/template/ubuntu/10.04/x86/<br />
aap_1.091-1_all<br />
aap-doc_1.091-1_all<br />
adduser_3.112ubuntu1_all<br />
anacron_2.3-13.1ubuntu11_i386<br />
apache2_2.2.14-5ubuntu8.2_i386<br />
apache2_2.2.14-5ubuntu8.4_i386<br />
apache2_2.2.14-5ubuntu8.6_i386<br />
apache2_2.2.14-5ubuntu8.7_i386<br />
apache2_2.2.14-5ubuntu8.8_i386<br />
apache2_2.2.14-5ubuntu8.9_i386<br />
<...snip><br />
<br />
# ls /vz/template/cache/<br />
centos-5-x86.tar.gz suse-11.3-x86.tar.gz ubuntu-9.10-x86.tar.gz<br />
debian-5.0-x86.tar.gz ubuntu-10.04-x86.tar.gz<br />
fedora-core-14-x86.tar.gz ubuntu-11.04-x86.tar.gz<br />
</pre><br />
<br />
So what we see is the entire OS and all applications are in 1 directory, and organized by distribution and release version. So to move Ubuntu 10.04:<br />
<br />
rsync -av -e ssh /vz/template/ubuntu/10.04 root@10.1.4.69:/vz/template/ubuntu<br />
<br />
and you also need to move the cache:<br />
<br />
rsync -av -e ssh /vz/template/cache/ubuntu-10.04-x86.tar.gz root@10.1.4.69:/vz/template/cache/<br />
<br />
Once the transfer is complete, you will be able to run <tt>vzpkgls</tt> or <tt>vzpkg list -O</tt> and see the template(s) listed on the target HN.<br />
<br />
So far, this all supposes template doesn't exist on the target- very simple, just copy it over. If the template exists already on the target HN, this requires closer inspection. With older templates, simply moving over the templates would be the end of it- you see the version listed when you do a <tt>vzplgls</tt>:<br />
<br />
mysql-rh9 20030814 20050412<br />
<br />
this means there are 2 versions of the mysql package for RH9: 20030814 and 20050412<br />
As an aside, you can't copy over just 1 of them, but if you rsync'd over the <tt>mysql-rh9</tt> directory, you'd see 20030814 and 20050412 on the target HN. As long as the versions match up, you're good to go.<br />
<br />
With EZ templates, the versions are not as easy to decipher. When you look at the <tt>vzpkg list</tt> output, that simply tells you when a cache was created. A cache is simply a snapshot of all the data needed for the latest versions of the OS and it's applications at the time the cache was created. It is a date, and thus tells you nothing about the versions within. So for instance, if you had a cache for Ubuntu 10.04 from 2007 and you ran <tt>vzpkg create cache ubuntu-10.04-x86</tt> it would re-download the latest version of apache, mysql and so on. And any ''subsequent'' ubuntu 10.04 CT's created thereafter would use those newer versions, whereas older CT's based on 10.04 would use older templates. You can see how this works by looking at the files under:<br />
<br />
<pre># ls /vz/template/ubuntu/10.04/x86/<br />
aap_1.091-1_all<br />
aap-doc_1.091-1_all<br />
adduser_3.112ubuntu1_all<br />
anacron_2.3-13.1ubuntu11_i386<br />
apache2_2.2.14-5ubuntu8.2_i386<br />
apache2_2.2.14-5ubuntu8.4_i386<br />
apache2_2.2.14-5ubuntu8.6_i386<br />
apache2_2.2.14-5ubuntu8.7_i386<br />
apache2_2.2.14-5ubuntu8.8_i386<br />
apache2_2.2.14-5ubuntu8.9_i386<br />
<...snip><br />
</pre><br />
<br />
You can see that this template has in it 6 different versions of apache2. This means that this template was merged with other 10.04 templates and/or it has been cached a few times, each time a new apache was available. The unfortunate thing is it's almost impossible to know which CT's are using which version of apache so it's hard to try to prune this down and remove unwanted/unneeded applications.<br />
<br />
So, if you see another HN has Ubuntu 10.04 on it, you need to see/confirm that it has all the exact files your CT needs. You can run a test rsync to see what ''would'' be transferred (what's missing):<br />
<br />
rsync -avn --stats -e ssh /vz/template/ubuntu/10.04 root@10.1.4.69:/vz/template/ubuntu<br />
<br />
Based on the amount of data you get back from the rsync, you will be able to decide whether or not to remove the -n and allow the full transfer to go through. The downside to doing the transfer is the situation described above- you've permanently joined these templates and now you may have 10 versions of apache on the target HN. There's one other potential pitfall to this kind of merging- it will also potentially copy over shared files, for which there is no particular version, most of which are in <tt>/vz/template/ubuntu/10.04/x86/config</tt>: <br />
<br />
cat /vz/template/ubuntu/10.04/x86/config/os/default/repositories<br />
/vz/template/ubuntu/10.04/x86/config/os/default/repositories<br />
<br />
Here are some specific things to look out for. Case: copying centos-5 on virt19 to virt12. We dumped the rsync output to a file so we could inspect:<br />
<br />
[root@virt19 /vz/template]# rsync -an -e ssh /vz/template/centos/ root@10.1.4.62:/vz/template/centos/ > v12_c5<br />
<br />
(note, that can be dangerous, better to add on full path <tt>/vz/template/centos/5</tt>)<br />
<br />
So in the output file we see things like:<br />
<br />
<pre><br />
x86/base0/cachecookie<br />
x86/base0/primary.xml.gz <br />
x86/base0/primary.xml.gz.sqlite <br />
x86/base0/repomd.xml <br />
x86/base1/cachecookie <br />
x86/base1/filelists.xml.gz <br />
x86/base1/filelists.xml.gz.sqlite <br />
x86/base1/primary.xml.gz <br />
x86/base1/primary.xml.gz.sqlite <br />
x86/base1/repomd.xml<br />
x86/base2/cachecookie<br />
</pre><br />
<br />
Which make us nervous cause those are not versioned files, i.e.:<br />
5/x86/MAKEDEV-3.23-1.2.i386/usr/sbin/mksock<br />
<br />
So those files will replace existing files on v12 rather than the case of the latter where it's likely being added. So caution should be given and deference to whichever version is newer (which rsync should take care of, but that will depend on the date each file was created and perhaps not the actual data within).<br />
<br />
If we look further in the file we see:<br />
<br />
x86/psa-appvault-moodle-1.8-8202920080409011254.noarch/usr/local/psa/var/cgitory/moodle-1.<br />
9/htdocs/lib/editor/htmlarea/plugins/TableOperations/img/cell-split.gif<br />
<br />
and other files looking like:<br />
5/x86/plesk-base-9.0.1-cos5.build90090127.18.i586/etc/sw/keys/info<br />
<br />
Which means plesk is here. We don't need plesk on v12 (the CT moving over doesn't use it) so we rerun the rsync and exclude it:<br />
<br />
[root@virt19 /vz/template]# rsync -an -e ssh --exclude="plesk*" --exclude="psa*" /vz/template/centos/ root@10.1.4.62:/vz/template/centos/ > v12_c5<br />
<br />
and now it's much smaller:<br />
<br />
[root@virt19 /vz/template]# cat v12_c5|wc -l<br />
97104<br />
<br />
Before running with excludes it was:<br />
[root@virt19 /vz/template]# cat v12_c5|wc -l <br />
238238<br />
<br />
So again, look at what you're doing. Generally, combining templates is fine however.<br />
<br />
If you're happy with the merge, run the rsync again without the "-n" to let it actually do the transfer.<br />
<br />
Once done, don't forget to add the new template to the HN in Management -> Reference -> Templates<br />
<br />
= getting help =<br />
<br />
= vzup2date =<br />
TODO <br />
= Adding new OS/application templates =<br />
<br />
Virtuozzo will lag a bit behind the initial release of an OS, perhaps by a month or more. Further, a newly-released OS may be available, but there may not be many/any application templates available initially. It pays to wait a bit.<br />
<br />
A template is easily installed via [[#vzup2date|vzup2date]]:<br />
<br />
vzup2date -z<br />
<br />
You will be given a list of OS's to install. Select an OS, then customize that selection by checking/unchecking the application templates we do/don't want to include/offer.<br />
<br />
Generally, here are the app templates we include/offer:<br />
<pre>cyrus-imap<br />
devel<br />
imp<br />
jre<br />
jsdk<br />
mailman<br />
mod_perl<br />
mod_ssl<br />
mysql<br />
php<br />
phpmyadmin<br />
phppgadmin<br />
postgresql<br />
proftpd<br />
spamassassin<br />
squirrelmail<br />
tomcat<br />
vsftpd<br />
</pre><br />
<br />
We do not (any longer) offer webalizer or analog.<br />
<br />
After selecting and installing the OS and apps, you should ensure it's installed:<br />
<br />
vzpkg list -O<br />
<br />
Your new OS (ex. fedora-core-13-x86_64) will be listed, without a corresponding cache date:<br />
<pre><br />
[root@virt13 /vzconf]# vzpkg list -O<br />
centos-6-x86 2011-07-22 13:24:41<br />
centos-6-x86_64 2012-03-09 20:42:22<br />
centos-5-x86 2009-07-27 16:58:24<br />
centos-5-x86_64 2012-03-09 22:38:53<br />
ubuntu-11.10-x86_64 2012-03-01 23:13:33<br />
ubuntu-9.04-x86 2009-06-02 11:32:39<br />
ubuntu-12.04-x86_64 2012-05-29 13:50:50<br />
ubuntu-8.04-x86 2008-09-17 21:27:20<br />
ubuntu-8.10-x86 2009-03-13 13:58:57<br />
ubuntu-11.04-x86 2011-12-05 11:15:46<br />
ubuntu-7.04-x86 2008-09-24 16:42:50<br />
redhat-el6-x86_64<br />
fedora-core-13-x86_64 <br />
fedora-core-12-x86 2010-02-06 13:24:32<br />
fedora-core-15-x86 2011-12-05 11:36:43<br />
fedora-core-11-x86 2009-10-01 16:30:59<br />
fedora-core-14-x86<br />
fedora-core-16-x86_64 2012-03-13 11:40:23<br />
fedora-core-9-x86 2008-11-29 10:52:18<br />
debian-6.0-x86 2011-07-29 16:00:35<br />
debian-6.0-x86_64 2012-03-12 19:53:33<br />
debian-5.0-x86 2009-03-12 12:39:11<br />
</pre><br />
<br />
You can also confirm the apps installed:<br />
<pre>[root@virt13 /vzconf]# vzpkg list fedora-core-13-x86_64<br />
fedora-core-13-x86_64 2013-03-08 11:39:44<br />
fedora-core-13-x86_64 devel<br />
fedora-core-13-x86_64 php<br />
fedora-core-13-x86_64 proftpd<br />
fedora-core-13-x86_64 postgresql<br />
fedora-core-13-x86_64 phpmyadmin<br />
fedora-core-13-x86_64 mysql<br />
</pre><br />
<br />
Before you can use the OS you must cache it. To create the cache run:<br />
vzpkg cache [OS]<br />
<br />
ex: <tt>vzpkg cache fedora-core-13-x86_64</tt><br />
<br />
Now it's ready to use. In order to be able to use it with the [[VPS_Management#vm|vm]] command, you must create a file:<br />
jctmpl-[OS]<br />
<br />
ex: <tt>jctmpl-fedora-core-13-x86_64</tt><br />
<br />
In it, on line 1 we place the OS, followed by the app names, ex:<br />
<pre>more jctmpl-fedora-core-13-x86_64<br />
fedora-core-13-x86_64<br />
postgresql<br />
jsdk<br />
mod_ssl<br />
phpmyadmin<br />
mod_perl<br />
tomcat<br />
devel<br />
php<br />
mailman<br />
cyrus-imap<br />
squirrelmail<br />
proftpd<br />
mysql<br />
phppgadmin<br />
spamassassin<br />
</pre><br />
<br />
The packages will be installed in that order, so any pre-req's should be handled there.<br />
<br />
Now, when you run [[VPS_Management#vm|vm]] it will show you the newly-added OS and you may create a CT using it.<br />
<br />
The last few tasks are related to mgmt:<br />
<br />
1. add to mgmt->reference->templates (use the existing templates as an example). Once added, the new template will be included with the list of OS's available for the given HN.<br />
2. if this is going to be a new OS offered for ordering, it needs to be added to the order form and webpage.<br />
a. to get a list of applications and versions in the new OS, you should create a test CT, enter it and run <tt>dpkg -l|sort</tt> (or <tt>rpm -aq|sort</tt> in centos/fedora) and enter that list in the following places:<br><br />
<tt>/usr/local/www/jc_pub/[osname].html</tt><br><br />
<tt>/usr/local/www/signup/html/[osname].html</tt><br><br />
Note, you will have to move the most recent OS data from <tt>/usr/local/www/signup/html/[osname].html</tt> into the corresponding <tt>/usr/local/www/signup/html/[osname]_old.html</tt> updating the version links in both.<br />
<br />
b. to add the new OS to the order form, you will need to update <tt>/usr/local/www/signup/html/step1.html</tt> in several places (for regular and OSS orders) as well as updating the templateID which you can get from mgmt->reference->templates (or [[Management_System_/_Public_Website_/_Signup_/_Account_Manager#jc:ref_templates|jc:ref_templates]])<br />
<br />
<br />
<br />
Old method:<br />
<br />
Go to http://www.parallels.com/en/virtuozzo/templates/catalog/<br />
<br />
Download the RPM's and install them<br />
<br />
vzpkg list<br />
to see the new packages/os templates<br />
<br />
vzpkg create cache "OS_EZ_template_name"<br />
to create a cache<br />
<br />
= Updating kernel on virt =<br />
<br />
We now use [[#vzup2date|vzup2date]] to update systems and kernels. What follows is what we used to do when we had to download kernels and install manually on old systems:<br />
<br />
<br />
1. install kernel and modules<br />
<pre># rpm -ivh vzkernel-enterprise-2.4.20-021stab028.12.777.i686.rpm \<br />
vzmodules-enterprise-2.4.20-021stab028.12.swsoft.i686.rpm<br />
vzkernel-smp ##################################################<br />
vzmodules-smp ##################################################</pre><br />
<br />
DO NOT USE the "rpm -Uhv" command to install the kernel, otherwise all the previously installed kernels may be removed from your system. <br />
<br />
2. update lilo/grub<br />
<br />
Lilo:<br />
<br />
vi /etc/lilo.conf<br />
<br />
rename old Virtuozzo kernel label from "linux+virtuozzo" to "virtuozzo_old"<br />
and add a new entry pointing to the newly installed virtuozzo kernel image.<br />
For example:<br />
<br />
<pre>prompt<br />
timeout=50<br />
default=linux+virtuozzo<br />
append="debug panic=5 console=tty console=ttyS0,38400"<br />
boot=/dev/sda<br />
map=/boot/map<br />
install=/boot/boot.b<br />
message=/boot/message<br />
linear<br />
<br />
image=/boot/vmlinuz-2.4.18-3smp<br />
label=linux<br />
initrd=/boot/initrd-2.4.18-3smp.img<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.18-3<br />
label=linux-up<br />
initrd=/boot/initrd-2.4.18-3.img<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-020stab009.5.777-enterprise<br />
label=2.4.20-9.5.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-020stab009.8.777-enterprise<br />
label=2.4.20-9.8.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-020stab009.21.777-enterprise<br />
label=2.4.20-9.21.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-020stab009.24.777-enterprise<br />
label=2.4.20-9.24.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-021stab028.3.777-enterprise<br />
label=2.4.20-28.3.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-021stab028.5.777-enterprise<br />
label=old_linux+vz<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-021stab028.12.777-enterprise<br />
label=linux+virtuozzo<br />
read-only<br />
root=/dev/sda1</pre><br />
<br />
With this configuration, the new kernel is the default kernel to use at boot. The original kernel entry has been retained with the label "old_linux+vz ".<br />
<br />
Grub:<br />
<br />
vi /boot/grub /menu.lst<br />
<br />
Add new entry at the top.<br />
For example:<br />
<br />
<pre>serial --unit=0 --speed=38400<br />
terminal --timeout=10 serial console<br />
default=0<br />
timeout=10<br />
splashimage=(hd0,0)/boot/grub/splash.xpm.gz<br />
title Virtuozzo 2.6.1 (2.4.20-021stab028.12.777-enterprise)<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-22.4.20-021stab028.12.777-enterprise root=/dev/sda1 console=tty0 \ console=ttyS0,38400<br />
title Virtuozzo 2.6.1 (2.4.20-021stab028.3.777-enterprise)<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.20-021stab028.3.777-enterprise root=/dev/sda1 console=tty0 \ console=ttyS0,38400<br />
title Red Hat Linux (2.4.18-3smp)<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.18-3smp ro root=/dev/sda1<br />
initrd /boot/initrd-2.4.18-3smp.img<br />
title Red Hat Linux-up (2.4.18-3)<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.18-3 ro root=/dev/sda1<br />
initrd /boot/initrd-2.4.18-3.img<br />
title Linux with Virtuozzo 2.5<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.20-020stab009.3.777-smp ro root=/dev/sda1<br />
title vz-enterpriseo<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.20-020stab009.21.777-enterprise ro root=/dev/sda1<br />
title vz-enterprise<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.20-020stab009.24.777-enterprise ro root=/dev/sda1 \ console=tty0 console=ttyS0,38400</pre><br />
<br />
3. run the lilo (if using lilo)<br />
<br />
<pre># lilo<br />
Added linux<br />
Added linux-up<br />
Added virtuozzo_old<br />
Added linux+virtuozzo *</pre><br />
<br />
4. reboot the server<br />
shutdown -r 23:05<br />
<br />
when it comes time for the shutdown, run [[VPS_Management#stopvirt|stopvirt]] to get things shut down faster<br />
<br />
<br />
= Remaking service VE (<=3.x only) =<br />
<br />
Occasionally the control panel for VEs (aka VZPP) will stop working and you just need to reinstall the service VE (which runs the control panels for all VEs on the HN). Here's how that's done :<br />
<br />
<pre>vzctl stop 1<br />
ipdel 1 69.55.234.152<br />
vzmlocal 1:2<br />
vzctl set 2 --save --onboot no<br />
vzsveinstall -t redhat-as3-minimal -d /root/Rel261/HW/RPMS -s 69.55.234.152 --skip-client</pre><br />
<br />
on 3.0:<br />
<br />
<pre>vzsveinstall -t redhat-as3-minimal -d /vz/Rel300/HW/RPMS -s 69.55.229.151 --skip-client<br />
<br />
vzctl stop 1<br />
vzctl mount 1<br />
vzctl mount 2<br />
/bin/cp -aRf /vz/root/2/home/vzagent0 /vz/root/1/home<br />
/bin/cp -aRf /vz/root/2/var/lib/mysql/VZADB /vz/root/1/var/lib/mysql<br />
/bin/cp -af /vz/root/2/etc/shadow /vz/root/1/etc/<br />
/bin/cp -aRf /vz/root/2/var/vzcp/static/vz/skins/ /vz/root/1/var/vzcp/static/vz<br />
/bin/cp -af /vz/root/2/etc/vzcp/pp/menu.xml /vz/root/1/etc/vzcp/pp/<br />
/bin/cp -af /vz/root/2/etc/vzcp/pp/dashboard.xml /vz/root/1/etc/vzcp/pp/<br />
<br />
vzctl umount 2<br />
vzctl umount 1<br />
vzctl start 1</pre></div>Supporthttps://wiki.jcihosting.com/index.php?title=Virtuozzo_/_Linux_Reference&diff=1026Virtuozzo / Linux Reference2013-03-11T23:36:23Z<p>Support: /* adding new templates */</p>
<hr />
<div>= Updating VZ =<br />
<br />
Update Virtuozzo to version 4.0.0<br />
Apply minor updates for Virtuozzo 3.0.0<br />
<br />
...<br />
When you get to the last screen, you will be given an option to <br />
<br />
( ) Automatically reboot after update<br />
(*) Reboot later manually<br />
<br />
make sure you choose to reboot later.<br />
<br />
= Migrating Templates =<br />
One nice feature VZ offers is the ability to run a virtual environment (VE or CT as it's now called) based on a particular ditribution on a host which may or may not share the same distribution. For instance, you could run a CT running Ubuntu while the host machine (Hardware Node or HN) is running CentOS. This is accomplished via the use of OS templates. As long as the HN contains the OS template on which a particular CT relies, it will run there. As such, if you want to move a CT from one HN to another, you will need to make sure the OS template exists there.<br />
<br />
Modern versions of VZ (>= 4.x) will check for and automatically move the OS template over to the host as you're (vz)migrating the CT over to a new HN. However we like to make sure the template is in place prior to moving the CT. <br />
<br />
The first step is to see if the template exists on the host. To see OS templates installed:<br />
<br />
2.x (shows OS and application templates):<br />
<pre># vzpkgls<br />
mod_ssl-rh9 20040624<br />
mysql-rh9 20030814 20050412<br />
openwebmail-rh9 20050427<br />
php-rh9 20050506<br />
phpBB-rh9 20041229<br />
postgresql-rh9 20050719<br />
sl-webalizer-rh9 20040119<br />
spamassassin-rh9 20040127<br />
tomcat-rh9 20040524<br />
usermin-rh9 20040116</pre><br />
<br />
>= 3.x (shows just OS templates, run without -O to see application as well):<br />
<pre># vzpkg list -O<br />
ubuntu-10.04-x86 2010-06-01 11:39:05<br />
ubuntu-11.04-x86 2011-06-22 14:23:59<br />
ubuntu-9.10-x86 2010-03-11 17:39:51<br />
centos-5-x86 2010-03-11 17:12:27<br />
debian-5.0-x86 2010-03-11 17:33:34</pre><br />
<br />
<br />
All template files are located:<br />
<br />
<pre># ls /vz/template/<br />
ColdFusion-fc1 jre-fc1 openwebmail-fc1 sl-webalizer-fc1<br />
ColdFusion-fc2 jre-fc2 openwebmail-fc2 sl-webalizer-fc2<br />
ColdFusion-rh9 jre-rh9 openwebmail-rh9 sl-webalizer-rh9<br />
PostNuke-fc1 jre-suse92 openwebmail-suse92 spamassassin-fc1<br />
PostNuke-fc2 jrun php-deb31 spamassassin-fc2<br />
PostNuke-rh9 mailman-deb31 php-fc1 spamassassin-rh9<br />
analog-fc1 mailman-fc1 php-fc2 spamassassin-suse92<br />
analog-fc2 mailman-fc2 php-rh9 suse-9.2<br />
analog-rh9 mailman-rh9 php-suse92 tomcat-fc1<br />
awstats-fc1 mailman-suse92 phpBB-fc1 tomcat-fc2<br />
awstats-rh9 mod_frontpage-fc1 phpBB-fc2 tomcat-rh9<br />
bbClone-fc1 mod_frontpage-fc2 phpBB-rh9 tomcat-suse92<br />
bbClone-fc2 mod_frontpage-rh9 phpmyadmin-deb31 urchin-fc1<br />
bbClone-rh9 mod_frontpage-suse92 postgresql-deb31 usermin-fc1<br />
conf mod_perl-deb31 postgresql-fc1 usermin-fc2<br />
debian-3.1 mod_perl-fc1 postgresql-fc2 usermin-rh9<br />
devel-deb31 mod_perl-fc2 postgresql-rh9 uw-imap-fc1<br />
devel-fc2 mod_perl-rh9 postgresql-suse92 uw-imap-fc2<br />
devel-suse92 mod_perl-suse92 proftpd-deb31 uw-imap-rh9<br />
fedora-core-1 mod_ssl-fc1 proftpd-fc1 vzredhat-7.3<br />
fedora-core-2 mod_ssl-fc2 proftpd-fc2 vzredhat-8.0<br />
jdk-deb31 mod_ssl-rh9 proftpd-rh9 webalizer-suse92<br />
jdk-fc1 mysql-deb31 proftpd-suse92 webmin-fc1<br />
jdk-fc2 mysql-fc1 redhat-7.2 webmin-fc2<br />
jdk-rh9 mysql-fc2 redhat-9 webmin-rh9<br />
jdk-suse92 mysql-rh9 redhat-as3-minimal<br />
jre-deb31 mysql-suse92 redhat-devel-9<br />
</pre><br />
<br />
What we see here are the base OS templates: <tt>redhat-9, fedora-core-1</tt> and supporting application templates: <tt>postgresql-rh9, mailman-rh9, jdk-fc1, mod_ssl-fc1</tt>. So if you wanted to copy over all of redhat 9 you'd need to copy the contents of:<br />
<pre># ls -d /vz/template/*rh9*<br />
/vz/template/ColdFusion-rh9 /vz/template/mod_frontpage-rh9 /vz/template/proftpd-rh9<br />
/vz/template/PostNuke-rh9 /vz/template/mod_perl-rh9 /vz/template/sl-webalizer-rh9<br />
/vz/template/analog-rh9 /vz/template/mod_ssl-rh9 /vz/template/spamassassin-rh9<br />
/vz/template/awstats-rh9 /vz/template/mysql-rh9 /vz/template/tomcat-rh9<br />
/vz/template/bbClone-rh9 /vz/template/openwebmail-rh9 /vz/template/usermin-rh9<br />
/vz/template/jdk-rh9 /vz/template/php-rh9 /vz/template/uw-imap-rh9<br />
/vz/template/jre-rh9 /vz/template/phpBB-rh9 /vz/template/webmin-rh9<br />
/vz/template/mailman-rh9 /vz/template/postgresql-rh9<br />
<br />
# ls -d /vz/template/*redhat-9*<br />
/vz/template/redhat-9<br />
</pre><br />
<br />
To get a template from one HN to another, it simply needs to be rsync'd over to the target HN. It's rsync'd over in this fashion:<br />
<br />
rsync -av -e ssh /vz/template/redhat-9 root@10.1.4.65:/vz/template<br />
rsync -av -e ssh /vz/template/*rh9 root@10.1.4.65:/vz/template<br />
<br />
Newer (>= 3.x) VZ employs "EZ" templates which are organized a little differently:<br />
<br />
<pre># ls /vz/template/<br />
cache CmdTool.log debian redhat-as3-minimal-20080630.tar.gz vc<br />
centos conf redhat-as3-minimal ubuntu<br />
<br />
# ls /vz/template/ubuntu/<br />
10.04 11.04 9.10<br />
# ls /vz/template/ubuntu/10.04/<br />
x86<br />
# ls /vz/template/ubuntu/10.04/x86/<br />
aap_1.091-1_all<br />
aap-doc_1.091-1_all<br />
adduser_3.112ubuntu1_all<br />
anacron_2.3-13.1ubuntu11_i386<br />
apache2_2.2.14-5ubuntu8.2_i386<br />
apache2_2.2.14-5ubuntu8.4_i386<br />
apache2_2.2.14-5ubuntu8.6_i386<br />
apache2_2.2.14-5ubuntu8.7_i386<br />
apache2_2.2.14-5ubuntu8.8_i386<br />
apache2_2.2.14-5ubuntu8.9_i386<br />
<...snip><br />
<br />
# ls /vz/template/cache/<br />
centos-5-x86.tar.gz suse-11.3-x86.tar.gz ubuntu-9.10-x86.tar.gz<br />
debian-5.0-x86.tar.gz ubuntu-10.04-x86.tar.gz<br />
fedora-core-14-x86.tar.gz ubuntu-11.04-x86.tar.gz<br />
</pre><br />
<br />
So what we see is the entire OS and all applications are in 1 directory, and organized by distribution and release version. So to move Ubuntu 10.04:<br />
<br />
rsync -av -e ssh /vz/template/ubuntu/10.04 root@10.1.4.69:/vz/template/ubuntu<br />
<br />
and you also need to move the cache:<br />
<br />
rsync -av -e ssh /vz/template/cache/ubuntu-10.04-x86.tar.gz root@10.1.4.69:/vz/template/cache/<br />
<br />
Once the transfer is complete, you will be able to run <tt>vzpkgls</tt> or <tt>vzpkg list -O</tt> and see the template(s) listed on the target HN.<br />
<br />
So far, this all supposes template doesn't exist on the target- very simple, just copy it over. If the template exists already on the target HN, this requires closer inspection. With older templates, simply moving over the templates would be the end of it- you see the version listed when you do a <tt>vzplgls</tt>:<br />
<br />
mysql-rh9 20030814 20050412<br />
<br />
this means there are 2 versions of the mysql package for RH9: 20030814 and 20050412<br />
As an aside, you can't copy over just 1 of them, but if you rsync'd over the <tt>mysql-rh9</tt> directory, you'd see 20030814 and 20050412 on the target HN. As long as the versions match up, you're good to go.<br />
<br />
With EZ templates, the versions are not as easy to decipher. When you look at the <tt>vzpkg list</tt> output, that simply tells you when a cache was created. A cache is simply a snapshot of all the data needed for the latest versions of the OS and it's applications at the time the cache was created. It is a date, and thus tells you nothing about the versions within. So for instance, if you had a cache for Ubuntu 10.04 from 2007 and you ran <tt>vzpkg create cache ubuntu-10.04-x86</tt> it would re-download the latest version of apache, mysql and so on. And any ''subsequent'' ubuntu 10.04 CT's created thereafter would use those newer versions, whereas older CT's based on 10.04 would use older templates. You can see how this works by looking at the files under:<br />
<br />
<pre># ls /vz/template/ubuntu/10.04/x86/<br />
aap_1.091-1_all<br />
aap-doc_1.091-1_all<br />
adduser_3.112ubuntu1_all<br />
anacron_2.3-13.1ubuntu11_i386<br />
apache2_2.2.14-5ubuntu8.2_i386<br />
apache2_2.2.14-5ubuntu8.4_i386<br />
apache2_2.2.14-5ubuntu8.6_i386<br />
apache2_2.2.14-5ubuntu8.7_i386<br />
apache2_2.2.14-5ubuntu8.8_i386<br />
apache2_2.2.14-5ubuntu8.9_i386<br />
<...snip><br />
</pre><br />
<br />
You can see that this template has in it 6 different versions of apache2. This means that this template was merged with other 10.04 templates and/or it has been cached a few times, each time a new apache was available. The unfortunate thing is it's almost impossible to know which CT's are using which version of apache so it's hard to try to prune this down and remove unwanted/unneeded applications.<br />
<br />
So, if you see another HN has Ubuntu 10.04 on it, you need to see/confirm that it has all the exact files your CT needs. You can run a test rsync to see what ''would'' be transferred (what's missing):<br />
<br />
rsync -avn --stats -e ssh /vz/template/ubuntu/10.04 root@10.1.4.69:/vz/template/ubuntu<br />
<br />
Based on the amount of data you get back from the rsync, you will be able to decide whether or not to remove the -n and allow the full transfer to go through. The downside to doing the transfer is the situation described above- you've permanently joined these templates and now you may have 10 versions of apache on the target HN. There's one other potential pitfall to this kind of merging- it will also potentially copy over shared files, for which there is no particular version, most of which are in <tt>/vz/template/ubuntu/10.04/x86/config</tt>: <br />
<br />
cat /vz/template/ubuntu/10.04/x86/config/os/default/repositories<br />
/vz/template/ubuntu/10.04/x86/config/os/default/repositories<br />
<br />
Here are some specific things to look out for. Case: copying centos-5 on virt19 to virt12. We dumped the rsync output to a file so we could inspect:<br />
<br />
[root@virt19 /vz/template]# rsync -an -e ssh /vz/template/centos/ root@10.1.4.62:/vz/template/centos/ > v12_c5<br />
<br />
(note, that can be dangerous, better to add on full path <tt>/vz/template/centos/5</tt>)<br />
<br />
So in the output file we see things like:<br />
<br />
<pre><br />
x86/base0/cachecookie<br />
x86/base0/primary.xml.gz <br />
x86/base0/primary.xml.gz.sqlite <br />
x86/base0/repomd.xml <br />
x86/base1/cachecookie <br />
x86/base1/filelists.xml.gz <br />
x86/base1/filelists.xml.gz.sqlite <br />
x86/base1/primary.xml.gz <br />
x86/base1/primary.xml.gz.sqlite <br />
x86/base1/repomd.xml<br />
x86/base2/cachecookie<br />
</pre><br />
<br />
Which make us nervous cause those are not versioned files, i.e.:<br />
5/x86/MAKEDEV-3.23-1.2.i386/usr/sbin/mksock<br />
<br />
So those files will replace existing files on v12 rather than the case of the latter where it's likely being added. So caution should be given and deference to whichever version is newer (which rsync should take care of, but that will depend on the date each file was created and perhaps not the actual data within).<br />
<br />
If we look further in the file we see:<br />
<br />
x86/psa-appvault-moodle-1.8-8202920080409011254.noarch/usr/local/psa/var/cgitory/moodle-1.<br />
9/htdocs/lib/editor/htmlarea/plugins/TableOperations/img/cell-split.gif<br />
<br />
and other files looking like:<br />
5/x86/plesk-base-9.0.1-cos5.build90090127.18.i586/etc/sw/keys/info<br />
<br />
Which means plesk is here. We don't need plesk on v12 (the CT moving over doesn't use it) so we rerun the rsync and exclude it:<br />
<br />
[root@virt19 /vz/template]# rsync -an -e ssh --exclude="plesk*" --exclude="psa*" /vz/template/centos/ root@10.1.4.62:/vz/template/centos/ > v12_c5<br />
<br />
and now it's much smaller:<br />
<br />
[root@virt19 /vz/template]# cat v12_c5|wc -l<br />
97104<br />
<br />
Before running with excludes it was:<br />
[root@virt19 /vz/template]# cat v12_c5|wc -l <br />
238238<br />
<br />
So again, look at what you're doing. Generally, combining templates is fine however.<br />
<br />
If you're happy with the merge, run the rsync again without the "-n" to let it actually do the transfer.<br />
<br />
Once done, don't forget to add the new template to the HN in Management -> Reference -> Templates<br />
<br />
= getting help =<br />
<br />
= vzup2date =<br />
TODO <br />
= Adding new OS/application templates =<br />
<br />
Virtuozzo will lag a bit behind the initial release of an OS, perhaps by a month or more. Further, a newly-released OS may be available, but there may not be many/any application templates available initially. It pays to wait a bit.<br />
<br />
A template is easily installed via [[#vzup2date|vzup2date]]:<br />
<br />
vzup2date -z<br />
<br />
You will be given a list of OS's to install. Select an OS, then customize that selection by checking/unchecking the application templates we do/don't want to include/offer.<br />
<br />
Generally, here are the app templates we include/offer:<br />
<pre>cyrus-imap<br />
devel<br />
imp<br />
jre<br />
jsdk<br />
mailman<br />
mod_perl<br />
mod_ssl<br />
mysql<br />
php<br />
phpmyadmin<br />
phppgadmin<br />
postgresql<br />
proftpd<br />
spamassassin<br />
squirrelmail<br />
tomcat<br />
vsftpd<br />
</pre><br />
<br />
We do not (any longer) offer webalizer or analog.<br />
<br />
After selecting and installing the OS and apps, you should ensure it's installed:<br />
<br />
vzpkg list -O<br />
<br />
Your new OS (ex. fedora-core-13-x86_64) will be listed, without a corresponding cache date:<br />
<pre><br />
[root@virt13 /vzconf]# vzpkg list -O<br />
centos-6-x86 2011-07-22 13:24:41<br />
centos-6-x86_64 2012-03-09 20:42:22<br />
centos-5-x86 2009-07-27 16:58:24<br />
centos-5-x86_64 2012-03-09 22:38:53<br />
ubuntu-11.10-x86_64 2012-03-01 23:13:33<br />
ubuntu-9.04-x86 2009-06-02 11:32:39<br />
ubuntu-12.04-x86_64 2012-05-29 13:50:50<br />
ubuntu-8.04-x86 2008-09-17 21:27:20<br />
ubuntu-8.10-x86 2009-03-13 13:58:57<br />
ubuntu-11.04-x86 2011-12-05 11:15:46<br />
ubuntu-7.04-x86 2008-09-24 16:42:50<br />
redhat-el6-x86_64<br />
fedora-core-13-x86_64 <br />
fedora-core-12-x86 2010-02-06 13:24:32<br />
fedora-core-15-x86 2011-12-05 11:36:43<br />
fedora-core-11-x86 2009-10-01 16:30:59<br />
fedora-core-14-x86<br />
fedora-core-16-x86_64 2012-03-13 11:40:23<br />
fedora-core-9-x86 2008-11-29 10:52:18<br />
debian-6.0-x86 2011-07-29 16:00:35<br />
debian-6.0-x86_64 2012-03-12 19:53:33<br />
debian-5.0-x86 2009-03-12 12:39:11<br />
</pre><br />
<br />
You can also confirm the apps installed:<br />
<pre>[root@virt13 /vzconf]# vzpkg list fedora-core-13-x86_64<br />
fedora-core-13-x86_64 2013-03-08 11:39:44<br />
fedora-core-13-x86_64 devel<br />
fedora-core-13-x86_64 php<br />
fedora-core-13-x86_64 proftpd<br />
fedora-core-13-x86_64 postgresql<br />
fedora-core-13-x86_64 phpmyadmin<br />
fedora-core-13-x86_64 mysql<br />
</pre><br />
<br />
Before you can use the OS you must cache it. To create the cache run:<br />
vzpkg cache [OS]<br />
<br />
ex: <tt>vzpkg cache fedora-core-13-x86_64</tt><br />
<br />
Now it's ready to use. In order to be able to use it with the [[VPS_Management#vm|vm]] command, you must create a file:<br />
jctmpl-[OS]<br />
<br />
ex: <tt>jctmpl-fedora-core-13-x86_64</tt><br />
<br />
In it, on line 1 we place the OS, followed by the app names, ex:<br />
<pre>more jctmpl-fedora-core-13-x86_64<br />
fedora-core-13-x86_64<br />
postgresql<br />
jsdk<br />
mod_ssl<br />
phpmyadmin<br />
mod_perl<br />
tomcat<br />
devel<br />
php<br />
mailman<br />
cyrus-imap<br />
squirrelmail<br />
proftpd<br />
mysql<br />
phppgadmin<br />
spamassassin<br />
</pre><br />
<br />
The packages will be installed in that order, so any pre-req's should be handled there.<br />
<br />
Now, when you run [[VPS_Management#vm|vm]] it will show you the newly-added OS and you may create a CT using it.<br />
<br />
The last few tasks are related to mgmt:<br />
<br />
# add to mgmt->reference->templates (use the existing templates as an example). Once added, the new template will be included with the list of OS's available for the given HN.<br />
# if this is going to be a new OS offered for ordering, it needs to be added to the order form and webpage.<br />
## to get a list of applications and versions in the new OS, you should create a test CT, enter it and run <tt>dpkg -l|sort</tt> (or <tt>rpm -aq|sort</tt> in centos/fedora) and enter that list in the following places:<br><br />
<tt>/usr/local/www/jc_pub/[osname].html</tt><br><br />
<tt>/usr/local/www/signup/html/[osname].html</tt><br><br />
Note, you will have to move the most recent OS data from <tt>/usr/local/www/signup/html/[osname].html</tt> into the corresponding <tt>/usr/local/www/signup/html/[osname]_old.html</tt> updating the version links in both.<br />
## to add the new OS to the order form, you will need to update <tt>/usr/local/www/signup/html/step1.html</tt> in several places (for regular and OSS orders) as well as updating the templateID which you can get from mgmt->reference->templates (or [[Management_System_/_Public_Website_/_Signup_/_Account_Manager#jc:ref_templates|jc:ref_templates]])<br />
<br />
<br />
<br />
Old method:<br />
<br />
Go to http://www.parallels.com/en/virtuozzo/templates/catalog/<br />
<br />
Download the RPM's and install them<br />
<br />
vzpkg list<br />
to see the new packages/os templates<br />
<br />
vzpkg create cache "OS_EZ_template_name"<br />
to create a cache<br />
<br />
= Updating kernel on virt =<br />
<br />
We now use [[#vzup2date|vzup2date]] to update systems and kernels. What follows is what we used to do when we had to download kernels and install manually on old systems:<br />
<br />
<br />
1. install kernel and modules<br />
<pre># rpm -ivh vzkernel-enterprise-2.4.20-021stab028.12.777.i686.rpm \<br />
vzmodules-enterprise-2.4.20-021stab028.12.swsoft.i686.rpm<br />
vzkernel-smp ##################################################<br />
vzmodules-smp ##################################################</pre><br />
<br />
DO NOT USE the "rpm -Uhv" command to install the kernel, otherwise all the previously installed kernels may be removed from your system. <br />
<br />
2. update lilo/grub<br />
<br />
Lilo:<br />
<br />
vi /etc/lilo.conf<br />
<br />
rename old Virtuozzo kernel label from "linux+virtuozzo" to "virtuozzo_old"<br />
and add a new entry pointing to the newly installed virtuozzo kernel image.<br />
For example:<br />
<br />
<pre>prompt<br />
timeout=50<br />
default=linux+virtuozzo<br />
append="debug panic=5 console=tty console=ttyS0,38400"<br />
boot=/dev/sda<br />
map=/boot/map<br />
install=/boot/boot.b<br />
message=/boot/message<br />
linear<br />
<br />
image=/boot/vmlinuz-2.4.18-3smp<br />
label=linux<br />
initrd=/boot/initrd-2.4.18-3smp.img<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.18-3<br />
label=linux-up<br />
initrd=/boot/initrd-2.4.18-3.img<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-020stab009.5.777-enterprise<br />
label=2.4.20-9.5.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-020stab009.8.777-enterprise<br />
label=2.4.20-9.8.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-020stab009.21.777-enterprise<br />
label=2.4.20-9.21.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-020stab009.24.777-enterprise<br />
label=2.4.20-9.24.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-021stab028.3.777-enterprise<br />
label=2.4.20-28.3.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-021stab028.5.777-enterprise<br />
label=old_linux+vz<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-021stab028.12.777-enterprise<br />
label=linux+virtuozzo<br />
read-only<br />
root=/dev/sda1</pre><br />
<br />
With this configuration, the new kernel is the default kernel to use at boot. The original kernel entry has been retained with the label "old_linux+vz ".<br />
<br />
Grub:<br />
<br />
vi /boot/grub /menu.lst<br />
<br />
Add new entry at the top.<br />
For example:<br />
<br />
<pre>serial --unit=0 --speed=38400<br />
terminal --timeout=10 serial console<br />
default=0<br />
timeout=10<br />
splashimage=(hd0,0)/boot/grub/splash.xpm.gz<br />
title Virtuozzo 2.6.1 (2.4.20-021stab028.12.777-enterprise)<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-22.4.20-021stab028.12.777-enterprise root=/dev/sda1 console=tty0 \ console=ttyS0,38400<br />
title Virtuozzo 2.6.1 (2.4.20-021stab028.3.777-enterprise)<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.20-021stab028.3.777-enterprise root=/dev/sda1 console=tty0 \ console=ttyS0,38400<br />
title Red Hat Linux (2.4.18-3smp)<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.18-3smp ro root=/dev/sda1<br />
initrd /boot/initrd-2.4.18-3smp.img<br />
title Red Hat Linux-up (2.4.18-3)<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.18-3 ro root=/dev/sda1<br />
initrd /boot/initrd-2.4.18-3.img<br />
title Linux with Virtuozzo 2.5<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.20-020stab009.3.777-smp ro root=/dev/sda1<br />
title vz-enterpriseo<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.20-020stab009.21.777-enterprise ro root=/dev/sda1<br />
title vz-enterprise<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.20-020stab009.24.777-enterprise ro root=/dev/sda1 \ console=tty0 console=ttyS0,38400</pre><br />
<br />
3. run the lilo (if using lilo)<br />
<br />
<pre># lilo<br />
Added linux<br />
Added linux-up<br />
Added virtuozzo_old<br />
Added linux+virtuozzo *</pre><br />
<br />
4. reboot the server<br />
shutdown -r 23:05<br />
<br />
when it comes time for the shutdown, run [[VPS_Management#stopvirt|stopvirt]] to get things shut down faster<br />
<br />
<br />
= Remaking service VE (<=3.x only) =<br />
<br />
Occasionally the control panel for VEs (aka VZPP) will stop working and you just need to reinstall the service VE (which runs the control panels for all VEs on the HN). Here's how that's done :<br />
<br />
<pre>vzctl stop 1<br />
ipdel 1 69.55.234.152<br />
vzmlocal 1:2<br />
vzctl set 2 --save --onboot no<br />
vzsveinstall -t redhat-as3-minimal -d /root/Rel261/HW/RPMS -s 69.55.234.152 --skip-client</pre><br />
<br />
on 3.0:<br />
<br />
<pre>vzsveinstall -t redhat-as3-minimal -d /vz/Rel300/HW/RPMS -s 69.55.229.151 --skip-client<br />
<br />
vzctl stop 1<br />
vzctl mount 1<br />
vzctl mount 2<br />
/bin/cp -aRf /vz/root/2/home/vzagent0 /vz/root/1/home<br />
/bin/cp -aRf /vz/root/2/var/lib/mysql/VZADB /vz/root/1/var/lib/mysql<br />
/bin/cp -af /vz/root/2/etc/shadow /vz/root/1/etc/<br />
/bin/cp -aRf /vz/root/2/var/vzcp/static/vz/skins/ /vz/root/1/var/vzcp/static/vz<br />
/bin/cp -af /vz/root/2/etc/vzcp/pp/menu.xml /vz/root/1/etc/vzcp/pp/<br />
/bin/cp -af /vz/root/2/etc/vzcp/pp/dashboard.xml /vz/root/1/etc/vzcp/pp/<br />
<br />
vzctl umount 2<br />
vzctl umount 1<br />
vzctl start 1</pre></div>Supporthttps://wiki.jcihosting.com/index.php?title=Management_System_/_Public_Website_/_Signup_/_Account_Manager&diff=1025Management System / Public Website / Signup / Account Manager2013-03-11T23:35:59Z<p>Support: /* jc:billing_history */</p>
<hr />
<div>= Overview =<br />
These systems are all running under the same apache instance on the mail server.<br><br />
<br />
The main ingredients are: apache 1.3.31, mysql, perl, mod_perl and template toolkit.<br />
<br />
Apache is listening on 69.55.230.9 ports 80 and 443<br />
<br />
The webserver can be stopped with:<br />
apachectl stop<br />
<br />
and must be started with:<br />
apachectl startssl<br />
<br />
If you run <tt>apachectl start</tt> none of the ssl-enabled pages will work.<br />
<br />
The database may be stopped and started as follows:<br />
/usr/local/etc/rc.d/mysql-server.sh stop<br />
/usr/local/etc/rc.d/mysql-server.sh start<br />
<br />
* webroot: <tt>/usr/local/www/</tt><br />
* logs: <tt>/var/log/httpd-error.log /var/log/httpd-access.log</tt><br />
* Apache config: <br />
<pre><br />
/usr/local/etc/apache/httpd.conf:<br />
<br />
-SNIP-<br />
<br />
<Directory /usr/local/www><br />
Options none<br />
</Directory><br />
<br />
<br />
<VirtualHost *><br />
SSLDisable<br />
DocumentRoot /usr/local/www/jc_pub<br />
ServerName johncompanies.com<br />
ServerAlias www.johncompanies.com<br />
ServerAlias newwww.johncompanies.com<br />
<br />
<Directory /usr/local/www/jc_pub><br />
Options FollowSymLinks<br />
</Directory><br />
<br />
Redirect /collocation http://www.johncompanies.com<br />
Redirect /colocation http://www.johncompanies.com<br />
<br />
ScriptAlias /cgi-bin/ "/usr/local/www/jc_pub/cgi-bin/"<br />
<Directory /usr/local/www/jc_pub/cgi-bin><br />
AllowOverride None<br />
Options None<br />
Order allow,deny<br />
Allow from all<br />
SetHandler None<br />
</Directory><br />
<br />
</VirtualHost><br />
<br />
<VirtualHost *><br />
ServerName mini.johncompanies.com<br />
SSLDisable<br />
DocumentRoot /usr/local/www/mini<br />
<br />
<Directory /usr/local/www/mini><br />
Options FollowSymLinks<br />
</Directory><br />
<br />
</VirtualHost><br />
<br />
<br />
<VirtualHost _default_:443><br />
<br />
DocumentRoot "/usr/local/www"<br />
<br />
DocumentRoot /usr/local/www<br />
<Directory /usr/local/www><br />
Deny from All<br />
</Directory><br />
<Directory /usr/local/www/mgmt><br />
Allow from All<br />
</Directory><br />
<Directory /usr/local/www/am><br />
Allow from All<br />
</Directory><br />
<Directory /usr/local/www/signup><br />
Allow from All<br />
</Directory><br />
<Directory /usr/local/www/images><br />
Allow from All<br />
</Directory><br />
<Directory /usr/local/www/media><br />
Allow from All<br />
</Directory><br />
<Files favicon.ico><br />
Allow from All<br />
</Files><br />
<br />
SSLEngine on<br />
SSLCipherSuite ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP:+eNULL<br />
<br />
# SSLCertificateFile /usr/local/etc/apache/mail.cert<br />
# SSLCertificateKeyFile /usr/local/etc/apache/mail.key<br />
#SSLCertificateFile /usr/local/etc/apache/ssl.crt/secure.crt<br />
SSLCertificateFile /usr/local/etc/apache/ssl.crt/secure.johncompanies.com.crt<br />
SSLCertificateChainFile /usr/local/etc/apache/ssl.crt/gd_bundle.crt<br />
SSLCertificateKeyFile /usr/local/etc/apache/ssl.key/secure.key<br />
<br />
PerlModule Apache<br />
<br />
# debug stuff<br />
PerlWarn On<br />
PerlTaintCheck Off<br />
PerlModule Apache::StatINC<br />
PerlInitHandler Apache::Reload<br />
# CustomLog /usr/local/apache/logs/access_log.mgmt common<br />
# ErrorLog /usr/local/apache/logs/error_log.mgmt<br />
<br />
PerlModule Apache::DBI<br />
# PerlModule Mgmt<br />
# PerlModule Auth<br />
<br />
PerlSetEnv MGMT_BASE /usr/local/www/mgmt<br />
PerlSetEnv COMMON_BASE /usr/local/www/common<br />
PerlSetEnv SIGNUP_BASE /usr/local/www/signup<br />
PerlSetEnv AM_BASE /usr/local/www/am<br />
PerlSetEnv MGMT_INDEX /mgmt/index.html<br />
PerlSetEnv SIGNUP_INDEX /signup/step1.html<br />
PerlSetEnv AM_INDEX /am/dashboard.html<br />
PerlSetEnv MGMT_LOG_LEVEL debug<br />
PerlSetEnv MGMT_LOG_FILE /usr/local/www/mgmt/Log/mgmt.log<br />
PerlSetEnv SIGNUP_LOG_LEVEL info<br />
PerlSetEnv SIGNUP_LOG_FILE /usr/local/www/signup/Log/signup.log<br />
PerlSetEnv AM_LOG_LEVEL debug<br />
PerlSetEnv AM_LOG_FILE /usr/local/www/am/Log/am.log<br />
PerlSetEnv PP_AUTH_TOKEN Pe3aLk5GdMblAyyLAv5vNYqipcynWdZJKdw1CmcGcIdOz74ujMrDYIov27i<br />
PerlSetEnv PP_URL 'https://www.paypal.com/cgi-bin/webscr'<br />
PerlSetEnv PP_EMAIL 'payments@johncompanies.com'<br />
PerlSetEnv JC_DOMAIN 'secure.johncompanies.com'<br />
PerlSetEnv CC_LOG_FILE /usr/local/www/mgmt/Log/cc.log<br />
PerlSetEnv DEV 0<br />
PerlRequire /usr/local/www/common/conf/startup.pl<br />
<br />
<Directory "/usr/local/www/mgmt"><br />
SetHandler perl-script<br />
PerlHandler Mgmt<br />
PerlSetVar JCMGMTPath /mgmt<br />
PerlSetVar JCMGMTLoginScript /mgmt/login.html<br />
<br />
<Files LOGIN><br />
AuthType Auth<br />
AuthName JCMGMT<br />
SetHandler perl-script<br />
PerlHandler Auth->login<br />
</Files><br />
<br />
<FilesMatch ".*\.html$|.*\.cgi|.*\.png"><br />
AuthType Auth<br />
AuthName JCMGMT<br />
PerlAuthenHandler Auth->authenticate<br />
PerlAuthzHandler Auth->authorize<br />
require valid-user<br />
</FilesMatch><br />
</Directory><br />
<br />
<Directory "/usr/local/www/am"><br />
SetHandler perl-script<br />
PerlHandler AM<br />
PerlSetVar JCAMPath /am<br />
PerlSetVar JCAMLoginScript /am/login.html<br />
#PerlSetVar JCAMExpires +1m<br />
<br />
<Files LOGINAM><br />
AuthType AMAuth<br />
AuthName JCAM<br />
SetHandler perl-script<br />
PerlHandler AMAuth->login<br />
</Files><br />
<br />
<Files PASSRESET><br />
SetHandler perl-script<br />
PerlHandler AMPassR<br />
</Files><br />
<br />
<FilesMatch ".*\.html$|.*\.cgi"><br />
AuthType AMAuth<br />
AuthName JCAM<br />
PerlAuthenHandler AMAuth->authenticate<br />
PerlAuthzHandler AMAuth->authorize<br />
require valid-user<br />
</FilesMatch><br />
</Directory><br />
<br />
<Directory /usr/local/www/mgmt/static><br />
SetHandler None<br />
</Directory><br />
<br />
<Directory /usr/local/www/am/static><br />
SetHandler None<br />
</Directory><br />
<br />
<Directory /usr/local/www/mgmt/bwgraphs><br />
SetHandler None<br />
</Directory><br />
<br />
<Directory /usr/local/www/am/bwgraphs><br />
SetHandler None<br />
</Directory><br />
<br />
<Directory /usr/local/www/images><br />
SetHandler None<br />
</Directory><br />
<br />
<Directory /usr/local/www/media><br />
SetHandler None<br />
</Directory><br />
<Directory /usr/local/www><br />
Options FollowSymlinks<br />
</Directory><br />
<br />
ScriptAlias /mgmt/cgi/ "/usr/local/www/mgmt/cgi/"<br />
<br />
<Directory /usr/local/www/mgmt/cgi><br />
AllowOverride None<br />
Options None<br />
Order allow,deny<br />
Allow from all<br />
SetHandler None<br />
AuthType Auth<br />
AuthName JCMGMT<br />
PerlAuthenHandler Auth->authenticate<br />
PerlAuthzHandler Auth->authorize<br />
require valid-user<br />
</Directory><br />
<br />
<Directory "/usr/local/www/signup"><br />
SetHandler perl-script<br />
PerlHandler Signup<br />
</Directory><br />
<br />
Alias /mgmt/mrtg "/usr/local/www/mgmt/mrtg/data"<br />
<Directory /usr/local/www/mgmt/mrtg/data/><br />
DirectoryIndex index.cgi<br />
SetHandler None<br />
Options ExecCGI<br />
AddHandler cgi-script .cgi<br />
</Directory><br />
<br />
Alias /mgmt/rrd "/usr/local/www/mgmt/mrtg/rrd"<br />
<Directory /usr/local/www/mgmt/mrtg/rrd/><br />
DirectoryIndex index.html<br />
SetHandler None<br />
</Directory><br />
<br />
ScriptAlias /mgmt/bb/cgi-bin/ /usr/home/bb/bbsrc/bb1.9i-btf/web/<br />
Alias /mgmt/bb "/usr/home/bb/bbsrc/bb1.9i-btf/www"<br />
<Directory /usr/home/bb/bbsrc/bb1.9i-btf/www/gifs><br />
SetHandler None<br />
</Directory><br />
<Directory /usr/home/bb/bbsrc/bb1.9i-btf/web><br />
AllowOverride None<br />
Options None<br />
Order allow,deny<br />
Allow from all<br />
PerlSetVar JCMGMTLoginScript /mgmt/login.html<br />
AuthType Auth<br />
AuthName JCMGMT<br />
PerlAuthenHandler Auth->authenticate<br />
PerlAuthzHandler Auth->authorize<br />
require valid-user<br />
</Directory><br />
<Directory /usr/home/bb/bbsrc/bb1.9i-btf/www><br />
PerlSetVar JCMGMTLoginScript /mgmt/login.html<br />
AuthType Auth<br />
AuthName JCMGMT<br />
PerlAuthenHandler Auth->authenticate<br />
PerlAuthzHandler Auth->authorize<br />
require valid-user<br />
</Directory><br />
<br />
Alias /mgmt/awstatsclasses "/usr/local/www/mgmt/awstats/wwwroot/classes/"<br />
Alias /mgmt/awstatscss "/usr/local/www/mgmt/awstats/wwwroot/css/"<br />
Alias /mgmt/awstatsicons "/usr/local/www/mgmt/awstats/wwwroot/icon/"<br />
ScriptAlias /mgmt/awstats/ "/usr/local/www/mgmt/awstats/wwwroot/cgi-bin/"<br />
Alias /mgmt/icon/ "/usr/local/www/mgmt/awstats/wwwroot/icon/"<br />
<br />
<Directory "/usr/local/www/mgmt/awstats/wwwroot/icon"><br />
SetHandler None<br />
Options Indexes MultiViews<br />
AllowOverride None<br />
Order allow,deny<br />
Allow from all<br />
</Directory><br />
<br />
<br />
<Directory "/usr/local/www/mgmt/awstats/wwwroot"><br />
PerlSetVar JCMGMTLoginScript /mgmt/login.html<br />
AuthType Auth<br />
AuthName JCMGMT<br />
PerlAuthenHandler Auth->authenticate<br />
PerlAuthzHandler Auth->authorize<br />
require valid-user<br />
SetHandler None<br />
Options None<br />
AllowOverride None<br />
Order allow,deny<br />
Allow from all<br />
</Directory><br />
</VirtualHost><br />
</pre><br />
<br />
== Template Toolkit ==<br />
<br />
http://www.template-toolkit.org/docs/manual/index.html<br />
<br />
All our dynamic content pages (i.e. anything but jcpub) makes use of the Template Toolkit (TT) framework to create/display our pages. This package allows us to create customied apache handlers via the use of mod_perl. The result is an easy to deploy, easy to develop, flexible and efficient platform.<br />
<br />
All these sites are organized under <tt>/usr/local/www</tt> and consist/rely on several components:<br />
<br />
* common/conf/startup.pl - this contains all the use/require code that brings in all the required libraries and modules required to run our sites and TT. It is run once as apache starts up and applies to all sites.<br />
<br />
* common/conf/Lib - this directory contains all common libraries, primarly those dealing with database access and access to individual tables. <br />
<br />
* mysql:jc - the main table containing everything for mgmt/am/signup sites is contained in the <tt>jc</tt> database. All data is accessed via the mysql user <tt>jc</tt> - this is so we can impose restrictions on what that user may do. That access info is stored in <tt>common/Lib/DBI.pm</tt><br />
The exception to that is the <tt>traffic</tt> database which used to be stored on mail, but has since been exported to run locally on the bwdb server itself. i.e. mgmt/am contact the remote traffic database running elsewhere for that data. That access data is stored in <tt>common/Lib/DBI_BWDB.pm</tt><br />
<br />
Part of TT's efficiency comes from static and natively compiled code/pages. Those pages are stored in:<br />
* mgmt: <tt>/tmp/mgmt_templates/</tt><br />
* am: <tt>/tmp/am_templates</tt><br />
* signup: <tt>/tmp/signup_templates</tt><br />
You could safely delete all of these and apache will just recreate them - though the directory holding the templates must exist, with the right permissions.<br />
<br />
To make editing of pages easier we enable apache modules/directives which direct apache to rebuild these pages when it notices there's a difference (httpd.conf):<br />
PerlModule Apache::StatINC<br />
PerlInitHandler Apache::Reload<br />
<br />
However, any changes made to anything under <tt>Lib</tt> requires an immediate apache restart to take effect and to keep the site running- any changes w/o a restart will break the site.<br />
<br />
Among the directives in httpd.conf you will see a series of variables which are important as they, among other things, direct the mod_perl handlers where certain data is stored:<br />
<br />
This is where all the files for a particular set of pages and unique handler are located:<br />
PerlSetEnv MGMT_BASE /usr/local/www/mgmt<br />
PerlSetEnv COMMON_BASE /usr/local/www/common<br />
PerlSetEnv SIGNUP_BASE /usr/local/www/signup<br />
PerlSetEnv AM_BASE /usr/local/www/am<br />
<br />
This is the default page to present, if none is specified:<br />
PerlSetEnv MGMT_INDEX /mgmt/index.html<br />
PerlSetEnv SIGNUP_INDEX /signup/step1.html<br />
PerlSetEnv AM_INDEX /am/dashboard.html<br />
<br />
Indicates the verbosity and location of the handler logfile (these log files only contain logging which we do internally from our code- these differ from the apache logs which log simple hits and server errors):<br />
PerlSetEnv MGMT_LOG_LEVEL debug<br />
PerlSetEnv MGMT_LOG_FILE /usr/local/www/mgmt/Log/mgmt.log<br />
PerlSetEnv SIGNUP_LOG_LEVEL info<br />
PerlSetEnv SIGNUP_LOG_FILE /usr/local/www/signup/Log/signup.log<br />
PerlSetEnv AM_LOG_LEVEL debug<br />
PerlSetEnv AM_LOG_FILE /usr/local/www/am/Log/am.log<br />
<br />
Variables used for pp API access:<br />
PerlSetEnv PP_AUTH_TOKEN Pe3aLk5GdMblAyyLAv5vNYqipcynWdZJKdw1CmcGcIdOz74ujMrDYIov27i<br />
PerlSetEnv PP_URL 'https://www.paypal.com/cgi-bin/webscr'<br />
PerlSetEnv PP_EMAIL 'payments@johncompanies.com'<br />
<br />
Our base url:<br />
PerlSetEnv JC_DOMAIN 'secure.johncompanies.com'<br />
<br />
Our credit card info logging file:<br />
PerlSetEnv CC_LOG_FILE /usr/local/www/mgmt/Log/cc.log<br />
<br />
Indicates we are/not in a dev environment:<br />
PerlSetEnv DEV 0<br />
<br />
= Public Website (jcpub) =<br />
<br />
* domain: <tt>http://www.johncompanies.com http://johncompanies.com</tt><br />
* webroot: <tt>/usr/local/www/jc_pub</tt><br />
<br />
Our public-facing website is all static, standard HTML. We have some light javascripting on some pages, but by in large it's a very WYSIWYG site setup.<br />
<br><br />
<br><br />
<br />
= Signup (signup) =<br />
== Setup / organization / notes ==<br />
<br />
* domain: <tt>https://secure.johncompanies.com/signup/step1.html</tt><br />
* webroot: <tt>/usr/local/www/signup</tt><br />
<br />
Directory structure:<br />
* am: contains the main handler <tt>Signup.pm</tt> and the ipn activity log <tt>ipn.log</tt><br />
* signup/Lib: contains the log module <tt>SLog.pm</tt><br />
* signup/Log: contains the log <tt>signup.log</tt>, and cardit card activity log <tt>cc.log</tt><br />
* signup/Plugin: contains the plugin module <tt>AMP.pm</tt> and the form fill in module <tt>FillInForm.pm</tt><br />
* signup/html: contains all web pages<br />
<br />
=== Usage - server order ===<br />
<br />
Once a customer leaves our public site and visits any page under https://secure.johncompanies.com/signup/ they are in the signup section of our site. There isn't a strict rule that all these pages are signup-related, for instance there are some pages related to payments that run in the signup namespace. This is for convenience since this codebase doesn't require a login/authentication.<br />
<br />
The signup code primarly deals with the ordering process. Customers entering the signup process via https://secure.johncompanies.com/signup/step1.html will have the ability to order any product. Several links throughout our site pass optional parameters to this page to pre-select the product the customer has selected. For instance, a link to order a linux package looks like https://secure.johncompanies.com/signup/step1.html?svc=linux<br />
<br />
In step 1 the customer is prompted to select the package/service level, their OS, any backup space (a la carte), ownership info and contact info. All customers must provide a billing and admin (technical) contact- both may be the same person. They may also (optionally) provide an alternate contact for the billing and admin contacts. What this means is if we are unable to reach them (perhaps due to some outage on their server, where their primary email is hosted) this is how we'd reach them. They must provide at least 1 non-alternate billing and admin contact (i.e. if they provide just 1 contact and mark it billing and admin, it cannot also be marked alternate. They would need to add another contact and mark it as alternate.)<br />
<br />
To update the packages available or OS choices for colo's, edit:<br><br />
<tt>signup/html/step1.html</tt><br />
<br />
When editing the OS templates for VPSs, make sure the values for the input options matches the existing template_id (jc:ref_templates.template_id). Also remember that there are 2 sets of linux and freebsd order options to edit (1 normal and 1 oss discount).<br />
<br />
<br />
In step 2 the customer is prompted for more info based upon their service selection:<br />
<br />
<br />
any server:<br />
* hostname<br />
* ips - if the package allows add-on ips they are given the option to purchase more<br />
* bandwidth - they are given the option to purchase additional bandwidth<br />
* bandwidth overage - they are prompted to choose from 3 options for when they use all their bandwidth in a given month: 1. stop traffic, 2. bill for additional usage up to an amount they specify, 3. allow unlimited bandwidth and bill accordingly<br />
* payment method (cc, paypal, echeck (paypal), invoice)<br />
* existing customer - we ask if they are an existing customer so we know if this server order is new or a replacement on an existing account or if it is to be set up under a new account<br />
<br />
<br />
vps:<br />
* disk space - if the package allows add-on space they are given the option to purchase more<br />
<br />
<br />
colo: <br />
* time zone to set the server to<br />
* # ips to assign initially, and a reason to assign more<br />
* disk partitioning (this should add up to the amount of the disk included with the system)<br />
* special instructions<br />
* (nfs) backup space - provided the option to purchase extra a la carte<br />
<br />
<br />
In step 3 they are shown the total price for the selected services and given the choice of billing intervals: 1/6/12 months. Discounts and savings are displayed. They are requested to tell us how they found us here. However, if they indicated they were an existing customer on step 2 and they are replacing or adding to an existing account, they are also prompted for info:<br />
* hostname/ip of the server being replaced or the account to which we are adding the server<br />
* selection of a new, permanent ip or a temporary ip<br />
* how to handle the information provided in the signup in regards to existing data on file (replace/merge)<br />
<br />
<br />
In step 4 they are shown a summary of the entire order, and prompted to accept our TOS.<br />
Once they submit this page, their signup is registered regardless of whether they go on to make payment.<br />
<br />
The next step depends on their selected payment option. If they are paying via credit card they are prompted to enter their card info, then shown a payment confirmation screen upon successful payment.<br />
<br />
If they are paying via paypal, they're sent off to paypal to setup payment, then returned to our site to complete the payment and shown a payment confirmation screen upon successful payment.<br />
<br />
Note on payment- they can reload this page and it will not result in a reorder, in fact they cannot go back and accidentally reorder at all. If the customer is ording a colo no payment is taken, rather we do an authorization or setup their paypal payment, but we don't actually take funds until the server is in service.<br />
<br />
=== Usage - colo quote ===<br />
<br />
Another facility provided by the signup code is our colo quote tool. A customer visiting this page https://secure.johncompanies.com/signup/colo_quote.html is able to request a quote on a customized server. The requests end up in the New Signups section of mgmt. Once responded to, the customer is given a special signup link which leads them into the signup process where they may order the customized server.<br />
<br />
In the colo quote tool they are asked to provide:<br />
* processor<br />
* RAM<br />
* raid level<br />
* drive size<br />
* IPs<br />
* bandwidth<br />
* nfs backup space<br />
* OS<br />
* requested delivery date<br />
* OS install option (JC-installed or self-install via IPKVM)<br />
* current customer Y/N<br />
* referral source<br />
* comments/questions<br />
<br />
To update the list of OS's or the hardware options, you'll edit:<br />
<tt>signup/html/colo_quote.html</tt><br />
<br />
As long as the field names don't change, this will not break compatibility with the quote build tool in mgmt.<br />
<br />
= Management System (mgmt) =<br />
== Setup / organization / notes ==<br />
<br />
* domain: <tt>https://secure.johncompanies.com/mgmt</tt><br />
* webroot: <tt>/usr/local/www/mgmt</tt><br />
<br />
Directory structure:<br />
* mgmt: contains the main handler <tt>Mgmt.pm</tt> and the authentication (<tt>Auth.pm</tt>) handler, the bigbrother status file <tt>bb.html</tt>, the credit card gpg key <tt>pubring.gpg</tt><br />
* mgmt/Lib: contains the log module <tt>MLog.pm</tt><br />
* mgmt/Log: contains the log <tt>am.log</tt>, and ats activity log <tt>ats.log</tt><br />
* mgmt/Plugin: contains the plugins required for the various mgmt pages<br />
* mgmt/html: contains all web pages<br />
* mgmt/static: contains static, non-interpreted content like graphics and javascripts<br />
* mgmt/awstats: our stats package. Not controlled by TT, just lives here so you have to authenticate in mgmt to see it<br />
* mgmt/bwgraphs: deprecated? b/w graphing libraries and code<br />
* mgmt/mrtg: mrtg graphs. Not controlled by TT, just lives here so you have to authenticate in mgmt to see it<br />
<br />
All functions located in mgmt are password-protected. Once you're logged in, as long as you keep your browser open (maintain session cookie) you will not need to login again. Users are authenticated via table: <tt>jc.users</tt> To add users or change password, update this table.<br />
<br />
=== menu ===<br />
<br />
==== layout ====<br />
<br />
* Pastes<br />
This link is twofold: first it is a dropdown menu of some frequently-used responses to customer questions. Clicking on any of these items will copy the text of the response into your clipboard- provided you are running in a compatible browser (IE). Clicking on the "Pastes" button itself will open up a new window containing many more pastes. Clicking on the links at the top will also copy in the text, which you may review in the page below. <br />
<br />
Updating: <tt>mgmt/html/pastes.html</tt><br />
<br />
* Load <br />
Opens up a new window showing mrtg-style load figures for all our hardware, and network traffic for several important servers (firewall, backup). Clicking on each graph drills down for more data.<br><br />
See [[#mrtg|mrtg]]<br />
<br />
* BigBrother <br />
Opens up a new window to the Big Brother monitoring system.<br><br />
See [[BigBrother|BigBrother]]<br />
<br />
* New Signups <br />
Opens up a new window to the order processing page.<br><br />
See [[#New_Signups|New Signups]]<br />
<br />
* PayFlow<br />
Opens up a new window to the the PayFlow manager.<br />
See [[Payment_System#PayFlow|PayFlow]]<br />
<br />
* Cabinet map<br />
Opens up a new window to the [[Cabinetmap|cabinet map]] in this wiki<br />
<br />
* Monitoring<br />
Opens up windows to monitor the following:<br><br />
ATS - monitor/manage ATSs at i2b. See [[#ats|ATS control]]<br><br />
Backups - monitor the size and timing of backups running from our servers to backup1. See [[#backupmonitor|backup monitoring]]<br><br />
Colo Monitor - a bb instance that only monitors customers who have requested/receive the service. See [[BigBrother#monitor.johncompanies.com|monitor.johncompanies.com]]<br><br />
ATS-n - the web-based interface for the ATS. See [[ATS|ATS]]<br><br />
Switches - mrtg pages for our switches<br><br />
System Counts - display's how many VPSs are on each system (according to the database- may not be 100% accurate)<br><br />
XX VZPP - VZPP (Virtuozzo Power Panel) for each server. See [[Virtuozzo_Reference|Virtuozzo Reference]]<br><br />
<br />
* Payments<br />
Paypal history - takes you to the paypal page where you can do advanced searches based on email/subid/txnid<br><br />
2Checkout V2 - 2Checkout site<br><br />
Invoice Pmt Entry - allows easy entry of payments for any customer paying via invoice<br><br />
Arrears list - shows customers who have a balance due<br><br />
Pmt notices - shows outstanding payment invoices, primarily for customers paying via paypal<br><br />
Pfp batch - shows items pending processing via payflow. See [[#pfp_batch|pfp_batch]]<br><br />
Payment links - deprecated<br><br />
Payment audit - show(ed) discrepancies between pp sub and billing sub<br><br />
Packages - shows all our current/past packages. See [[#pkgs|packages]]<br><br />
Orphaned R.pmts - shows all paypal subs in the system which are not associated with an account. These should only/all be from people who signed up for payment fraudulently, and all subs should be in cancelled/disabled state. The only real reason to visit this page is in the case of a customer who is adding a server and who's signed up, you would link his paypal sub to his account here.<br><br />
IPN Input - allows you to manually input a paypal IPN into our system<br><br />
Request pmt - broken. ??<br><br />
<br />
* Reference<br />
IPKVM - links to our IPKVMs. See [[IPKVM|IPKVM]]<br><br />
VZ Ticket - submit a trouble ticket for help with VZ. See [[Virtuozzo_reference#getting_help|getting help]]<br><br />
IP Map - display's ip allocation information. See [[#ipmap|ipmap]]<br><br />
Acct Mgr - quick link to the [[#Account_Manager_.28AM.29|Account Manager]]<br><br />
Asset DB - our asset database. See [[#asset_database|asset database]]<br><br />
Bandwidth - bandwidth query page. See [[Bandwidth_Management#Reporting|Reporting]]<br><br />
Web stats - awstats page<br><br />
Domaintools - handy whois resource<br><br />
CrashLog - tool to log the details of a crash. See [[#crash_log|crash log]]<br><br />
DosLog - tool to log the details of a dos attack. See [[#dos_log|dos log]]<br><br />
URL: Linux VPS - takes you right to the linux vps order page<br><br />
URL: FreeBSD VPS - takes you right to the linux freebsd order page<br><br />
URL: Dedicated - takes you right to the dedicated server order page<br><br />
Notices - takes you to the notices tool. See [[#notices|notices]]<br><br />
Templates - template management/tracking. See [[#templates|templates]]<br><br />
VZ Reference - VZ reference/knowledge base at the parallels website<br><br />
VZ Templates - template repository at parallels website. The best method for installing new templates is <tt>vzup2date</tt>. See [[Virtuozzo_Reference#adding_new_templates|adding new templates]]<br><br />
Bugzilla - our bugzilla site<br><br />
redit.com - customer portal at redit. Get realtime bandwidth and cabinet power usage<br><br />
Manual - old/deprecated<br><br />
johncompanies.com - our website<br><br />
Linux welcome - generic welcome email<br><br />
BSD welcome - freebsd welcome email<br><br />
Debian welcome - debian/ubuntu welcome email<br><br />
Spam check - site to check if an ip is blacklisted<br><br />
DEV mgmt - mgmt site on dev server<br><br />
DEV am - AM on dev server<br><br />
DEV jcpub - public site on dev server<br><br />
<br />
* Wiki<br />
This.<br><br />
<br />
==== editing ====<br />
The dropdown menu is driven by a couple files:<br><br />
<tt>mgmt/static/HM_Arrays.js<br><br />
mgmt/static/HM_Loader.js</tt><br />
<br />
to change or add menu items, edit HM_Arrays.js<br><br />
covering every topic would be lengthy but here are some tips:<br />
<br />
* in each section, each entry but the last must end with a comma<br />
* each entry is formatted: <tt>["name", action, 1,0,N]</tt> where N is 0 that means this is a simple button, when N is 1 it is a dropdown and another menu is attached.<br />
* in the case of the first, top-level group:<br />
<pre>["Pastes","JavaScript:win1=window.open('','win1'); win1.location='https://secure.johncompanies.com/mgmt/pastes.html'; void(0)",1,0,1],<br />
["Load","JavaScript:win7=window.open('','win7'); win7.location='https://secure.johncompanies.com/mgmt/mrtg/index.cgi'; void(0)",1,0,0],<br />
["BigBrother","JavaScript:win9=window.open('','win9'); win9.location='https://secure.johncompanies.com/mgmt/bb/'; void(0)",1,0,0],<br />
["New Signups","https://secure.johncompanies.com/mgmt/pending.html",1,0,0],<br />
["PayFlow","JavaScript:winpf=window.open('','winpf'); winpf.location='https://manager.paypal.com/'; void(0)",1,0,0],<br />
["Cabinet map","JavaScript:win25=window.open('','win25'); win25.location='https://secure.johncompanies.com/mgmt/cabinetmap.html'; void(0)",1,0,0],<br />
["Monitoring","",1,0,1],<br />
["Payments","",1,0,1],<br />
["Reference","",1,0,1],<br />
["Wiki","JavaScript:winwiki=window.open('','winwiki'); winwiki.location='https://wiki.johncompanies.com/'; void(0)",1,0,0]<br />
],</pre><br />
<br />
You see that Pastes is the first menu item and that it has a submenu. That submenu is defined in the <tt>HM_Array2_1</tt> array. Likewise, the 7th menu item, Monitoring, has a submenu defined in <tt>HM_Array2_7</tt>. And that furter has nested menus, the first item in the Monitoring dropdown has a nested menu in <tt>HM_Array2_7_1</tt> , and the 13th item in <tt>HM_Array2_7_13</tt><br />
<br />
* when setting a window to open a new window (most menu items) it's important to keep track of the window name if you want each link to open a unique window. i.e. <tt>JavaScript:win10a=window.open('','win10a'...</tt> make the <tt>win10a</tt> name is unique.<br />
<br />
== Searching ==<br />
<br />
== Customer main view page ==<br />
Most interaction in mgmt is done on or from the main customer page. This is the main landing page you'll get if you're searching for a customer. From this page you can see and edit:<br />
<br />
* systems/services <br />
* ownership info<br />
* bandwidth usage info<br />
* contact info<br />
* billing info<br />
<br />
=== Systems ===<br />
This table displays the systems and services to which a customer has subscribed. Depending on the type of system, certain data will appear. Namely:<br />
* hostname<br />
* root directory (VPS only)<br />
* OS (VPS only)<br />
* disk space allocated (VPS only)<br />
* CT id (linux VPS only)<br />
* sysid<br />
* IPs <br />
* dates of start/cancel/stop<br />
* package ID/smount+interval/next bill date<br />
<br />
<br />
Any item black and underlined (this goes for anything in mgmt) is clickable and will copy into your clipboard (IE only). Items in this table with special background coloring indicates:<br />
* grey = cancelled system. To un-stop a system, databases need to be manually edited- so be careful.<br />
* blue = monotired by castle. If you uncheck and save the monitor flag (in the system edit screen), you must also update the tracking sheet (a0, p4). Before setting up a probe with castle, make sure that there are no notes indicating the customer has previously requested not to be monitored ("don't monitor"), or “can't monitor” which means castle just can't monitor them for some reason. <br />
* yellow = pending shutdown (cancel date set)<br />
<br />
To setup the system for (future) cancel, click the "cancel" link. You will be prompted to provide a cancel date. This will indicate the date you intend to shut down the system. Usually this is in the future since the customer pre-pays for service. Also, you want to set at least 1 day in the future to allow the customer to get the "[[System-generated_Notifications#.22IMPORTANT_REMINDER_-_SYSTEM_SHUT_DOWN_TODAY.22|Shutdown today]]" email, clueing them in to the fact that their server is going away. The 2 quick links for 5-days and 10-days will set the date in the future (5 or 10 days), both optimized to allow the customer to receive the shutdown reminder notices. If you want the customer not to receive shutdown reminder notices, you should set that field to no. If you want to un-cancel a system, go into the edit screen and remove/clear the cancel date.<br />
<br />
<br />
When a system has been shut down (or pulled from the rack in the case of a colo), click the "stop" link and set the date it was stopped. If this is the last active system on the account, it will mark the customer cancelled <br />
<br />
<br />
To edit aspects of the system click the "edit" link under the "actions" column.<br />
<br />
==== edit ====<br />
From this screen you may edit most if not all aspects of the system in question. <br />
* machine - pretty simple, what host is the VPS on or is this a (jc owned) managedcolo or (customer owned) colo <br />
* directory - (VPS only) where the VPS is located on the host<br />
* disk - (VPS only) how much they actually have. If you need to set this higher, edit systems.disk<br />
* hostname<br />
* veid - linux (VPS only) aka CT id<br />
* os tempalte - (VPS only) this will auto-populate with the templates/os on the host selected in the machine field. To add templates you need to do so via the [[#Templates|Templates]] page<br />
* password - (VPS only) if this is not set this system was setup with our universal/standard password: p455agfa or 8ico2987<br />
* ats port - (colo only) port and ATS to which this server is connected<br />
* cabinet - (colo only) location of colo<br />
* ips - ips assigned to server and the ability to assign more (space delimited if adding mult). A handy link will open ipmap and clicking ips should copy back the ip to the new ip field.<br />
* dates - start/stop/cancel<br />
* monitored - indicate whether this system is on castle's probe map<br />
* base, disk, ip packages - adjust all aspects of packages. If a field is red that means there's a mismatch between what the customer has and what the package covers. i.e. if they have 50 GB of disk and the package only gives 30, you'd need to add a disk package for 20 GB or adjust the base package to include 50 GB. Anything yellow indicates an altered/custom value. i.e. if the package comes with 6 IPs and you set it to 5, that will be highlighted in yellow. Or if the price was $19 and you make it $17, prices will be yellow'd. Green fields in the Totals row indicate a good matchup for the given attribute (i.e. disk allocated to server is the same as disk provided by package(s)). <br />
<br />
If you are moving a customer to a new machine, that may be done here- the old machine will be marked as stopped and show up as an additional row in the view screen. So really that is a stop and add operation behind the scenes in mgmt.<br />
<br />
==== add ====<br />
<br />
This works very similar to edit, and like it implies will add a new system to the customer.<br />
<br />
=== Owner Info ===<br />
<br />
=== Contact Info ===<br />
<br />
=== Notes ===<br />
<br />
=== Bandwidth ===<br />
<br />
=== Billing History ===<br />
<br />
=== Payment Sources ===<br />
<br />
=== Outstanding Invoices ===<br />
<br />
=== (Legacy) Billing ===<br />
<br />
=== (Legacy) Recurring Payments ===<br />
<br />
== ats ==<br />
<br />
== backupmonitor ==<br />
<br />
== asset database ==<br />
<br />
== crash log ==<br />
<br />
== dos log ==<br />
<br />
== notices ==<br />
<br />
== New Signups ==<br />
<br />
== Templates ==<br />
<br />
== mrtg ==<br />
* domain: <tt>https://secure.johncompanies.com/mgmt/mrtg/index.cgi/ https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p3&path=</tt><br />
* webroot: <tt>/usr/local/www/mgmt/mrtg</tt><br />
All configuration is done via *.cfg files. The main load graph is found in mrtg1.cfg<br />
All other config files are for various switches. Switch config files are rebuilt out of a cron jobs running on mail. This ensures if we change a port name (desc) that the mrtg we look at has the latest info. So if you want to change port naming, please do it in the switch itself. If you have problems getting new devices setup or change existing devices you may need to change permissions on the cfg file as well as the data file in <tt>/usr/local/www/mgmt/mrtg/data</tt>, including removal of the rrd file if necessary.<br />
<br />
== Errors ==<br />
<br />
=== "Lock wait timeout exceeded" ===<br />
<br />
<pre><br />
delete error - Can't delete a2206e24: DBD::mysql::st execute failed: Lock wait timeout exceeded; try restarting transaction [for Statement "DELETE FROM invoice WHERE inv_ref=? "] at /usr/local/lib/perl5/site_perl/5.6.1/DBIx/ContextualFetch.pm line 51. at /usr/local/www/mgmt/Plugin/Billing.pm line 1934<br />
</pre><br />
<br />
This is the result of an unclean submit/commit. Usually from an error or a double click on something that should have been single click. To clear this up, restart the database:<br />
<br />
<pre><br />
mail /usr/local/www/scripts# /usr/local/etc/rc.d/mysql-server.sh stop<br />
mysqldmail /usr/local/www/scripts#<br />
</pre><br />
<br />
It takes a minute to shutdown. I keep running the command until it says it isn't running, then I start it:<br />
<br />
<pre>mail /usr/local/www/scripts# /usr/local/etc/rc.d/mysql-server.sh stop<br />
mysqldmail /usr/local/www/scripts# /usr/local/etc/rc.d/mysql-server.sh stop<br />
mysqldmail /usr/local/www/scripts# /usr/local/etc/rc.d/mysql-server.sh stop<br />
mysql-server isn't running<br />
mail /usr/local/www/scripts# /usr/local/etc/rc.d/mysql-server.sh start<br />
mysqldmail /usr/local/www/scripts#</pre><br />
<br />
= Account Manager (AM) =<br />
== Setup / organization / notes ==<br />
<br />
* domain: <tt>https://secure.johncompanies.com/am</tt><br />
* webroot: <tt>/usr/local/www/am</tt><br />
<br />
Directory structure:<br />
* am: contains the main handler <tt>AM.pm</tt>, the authentication (<tt>AMAuth.pm</tt>), and password reset (<tt>AMPassR.pm</tt>) handlers<br />
* am/Lib: contains the log module <tt>AMLog.pm</tt><br />
* am/Log: contains the log <tt>am.log</tt>, and ats activity log <tt>ats.log</tt><br />
* am/Plugin: contains the plugin module <tt>AMP.pm</tt> and the form fill in module <tt>FillInForm.pm</tt><br />
* am/html: contains all web pages<br />
* am/static: contains static, non-interpreted content like graphics and javascripts<br />
<br />
<br />
The Account Manager was designed and intended to allow customers to do the following:<br />
* see their services and various details about it<br />
** IP<br />
** type service<br />
** cost of service<br />
** link to control panel (virtuozzo CT only)<br />
* to manage their service level<br />
* to power cycle their server (if at i2b)<br />
* to see their bandwidth usage and allocation<br />
* to manage their contact info<br />
* to review their billing history<br />
* to update their billing method info<br />
<br />
It is accessed via our secure website. Customers must login using their CID (colNNNNN) and a password which is created specifically for use with the AM. It is not tied in any way to their server, though some customers may be confused by that or think that changing the password in one location will affect the other.<br />
<br />
JC staff may login to any customer's AM by using a special (all access) password.<br />
<br />
In order to access the AM for any given customer, a few things must be in place:<br />
# they must have a password issued for their AM access. In order to determine if a password has been set you can look at the "AM pass" (jc:customers.am_auth) field in the customer's page in mgmt. If it says "(hidden)" a password has been established. If instead you see 3 links:<br>reset activate<br>activate+email<br>that means the customer has no password and access has not been enabled.<br />
# the "owner email" (jc:customers.owner_email) field must contain the owner's email address. If a customer has been issued the pass to their AM but an owner email has not been selected, they will be prompted to select one from the existing contacts upon logging in for the first time. The selected owner will be emailed. It's for this reason that if you access the AM for a customer with the global pass, you should make sure a owner has been selected before hand (enter the info in mgmt) and don't select it in the AM, lest you want them to get a random email.<br />
<br />
If no password has been established (or it has been forgotten- it's stored as a 1 way hash) it must be reset. The customer may do this themselves via the password reset page (linked off the AM login page). This will require them to have on file and use/match the owner_email field. You can also reset the password via mgmt via 2 methods:<br />
# reset: will just reset their password and show it to you in mgmt. It will also give you handy instructions (commands) on how to place their CID and AM password in a file on their VPS. We prefer to do it this way since we then don't have to decide if the person requesting access is authorized or not. If we place the CID/password in a file on their VPS and make it viewable by root only, then only someone with root access could get to that and if you have root access, you're considered authorized.<br />
# activate+email: this will set/reset their password and send an email to the customer with access instructions and it will instruct them to find the AM access info in the <tt>/root/ampass</tt> on their VPS- which you will create for them as soon as you click this link. If the customer has only colo's, the site will recognize that and email them the instructions/password.<br />
# activate: this will enable AM access. The customer may then use the password reset function to get a password set for access.<br />
<br />
There is lots of contextual/hover help throughout the AM to help customers understand what they're looking at.<br />
<br />
Note - if a(n old) customer with no AM access gets a new VPS, the welcome email will indicate they have AM access established- this isn't true. You will need to at least activate the AM for them and possibly also reset their pass and update the ampass file accordingly.<br />
<br />
== Usage ==<br />
<br />
=== Dashboard ===<br />
This is the main/landing page for the AM. It has several sections:<br />
<br />
* Announcements: This is where we make available any announcement we might have for the customer. If the customer had a billing issue, it would be listed here.<br />
<br />
* Services: lists the services (servers) which the customer has and basic info about it (service type, IP, hostname, OS)<br />
<br />
* Bandwidth: shows condensed usage/allocation, overage and how we deal with overage (per the customer's preference).<br />
<br />
* Contacts: shows the contacts on file for the account and their role.<br />
<br />
=== Services ===<br />
<br />
This page shows in detail what services they have with us. For instance, their package and what it comes with in terms of IP, disk space, bandwidth, price. There is a link for them to change service on this page- this just trigger's an email to support so they can't actually change their service instantly.<br />
<br />
If they have a colo @ i2b, the power controls are available on this page.<br />
The status of the power outlet to which their server is connected is displayed ("Current Status") and if the ATS is responding (see below) 2 radio options are given for controlling the port: "Off" and "On". Pretty self explanatory, "Off" turns the port off and "On" turns it on. In order to power cycle the server, the will need to first select the "Off" option and click the "Submit" button. A window will popup and display the status of the port until the status switches to off (it will auto refresh every 10sec). To power back on, they just need to set the option to "On" and click "Submit".<br />
<br />
If the customer has a virtuozzo VPS, the link to their VZ control panel is provided.<br />
<br />
=== Bandwidth ===<br />
<br />
This page allows the customer to pull up bandwidth usage statistics. The reporting of those statistics is available in several flavors and can be customized to show data based on:<br />
* date range<br />
* server<br />
* IP(s)<br />
* data flowing in a particular direction<br />
* protocol (ICMP, TCP, UDP) <br />
* port(s)<br />
<br />
Further, data can be output as:<br />
* daily bar graph<br />
* 15min granularity bar graph<br />
* pie chart<br />
* table/raw<br />
<br />
The only thing to keep in mind here is that anything but the daily bar graph query will take up a bit more time as all other queries come from much larger tables and thus require more time to seek through the database.<br />
<br />
The graphs generated by this activity actually generate files which live on the server for a period of time, and are removed by cronjob.<br />
<br />
=== Billing ===<br />
<br />
This screen displays the payment method currently active for the customer, and allows them to update that info (in the case of credit card payments). <br />
<br />
It also lists their payment history, oldest to newest, and their outstanding balance.<br />
<br />
It displays their ongoing JC charges.<br />
<br />
If they have a paypal subscription setup it displays that.<br />
<br />
Finally, if they have an outstanding (paypal) invoice, it will display here and give them a link to (pay) it.<br />
<br />
=== Account Info ===<br />
<br />
Allows the customer to see / update:<br />
* account owner info<br />
* contact info<br />
* the account manager password<br />
* bandwidth preferences<br />
<br />
Any modifications to these settings will require the user to request a security token to be emailed to the owner (jc:customers.owner_email) which they are required to provide in order to make updates to this page. Once the customer provides this key and goes on to edit the info it is no longer valid and any subsequent visit to the edit page will require the generation of a new security token. Obviously, this requires the customer to have the ability to retrieve email at the owner's email address on file.<br />
<br />
=== Support ===<br />
<br />
This page allows the customer to quickly send us a message. They are allowed to send it as/from one of the contacts on file (or provide a new one) and they can indicate which server the issue is regarding. The email is cc'd back to the customer for their records. This form is intelligent and will send the email to the right place based on the server/product they select- support@ or linux@<br />
<br />
== Problems ==<br />
=== Power management: Status and control temporarily unavailable ===<br />
<br />
ATS isn't responding. See [[ATS#Rebooting_and_Recovering|Rebooting and Recovering ]]<br />
<br />
=== Power control doesn't work ===<br />
<br />
If after requesting a change in power status, the popup status window never reflects a change to the requested status, that means the ATS isn't responding and the customer should try setting the power again. If that still doesn't work, the ATS is out and needs to be reset. See [[ATS#Rebooting and Recovering|Rebooting and Recovering]]<br />
<br />
= Database =<br />
<br />
The database behind AM/Signup/MGMT runs on mysql on the [[Infrastructure_Machines#mail|mail]] server.<br />
<br />
== jc:billing_history ==<br />
<br />
== jc:ref_templates ==</div>Supporthttps://wiki.jcihosting.com/index.php?title=VPS_Management&diff=1024VPS Management2013-03-11T23:16:16Z<p>Support: /* migrate manually (if migrate/online fails) */</p>
<hr />
<div>= Common Problems =<br />
== Login to any machine without a password ==<br />
<br />
This is possible via the use of ssh keys. The process is thus:<br />
<br />
1. place the public key for your user (root@mail) in the /root/.ssh/authorized_keys file on the server you wish to login to<br />
cat /root/.ssh/id_dsa.pub<br />
(paste that into authorized_keys on the target server). If the file doesn't exist, create it.<br />
<br />
2. enable root login (usually only applies to FreeBSD). Edit the /etc/ssh/sshd_config on the target server and change:<br />
<tt>#PermitRootLogin no</tt><br />
to<br />
<tt>PermitRootLogin yes</tt><br />
<br />
3. Restart the sshd on the target machine. First, find the sshd process: <br />
jailps <hostname> | grep sshd <br />
or <br />
vp <VEID> | grep sshd<br />
<br />
Look for the process resembling:<br />
root 17296 0.0 0.0 5280 1036 ? Ss 2011 4:27 /usr/sbin/sshd <br />
(this is the sshd)<br />
<br />
Not:<br />
root 6270 0.5 0.0 6808 2536 ? Ss 14:33 0:00 sshd: root [priv]<br />
(this is an sshd child- someone already ssh'd in as root)<br />
<br />
Restart the sshd: <br />
kill -1 <PID><br />
<br />
Ex:<br />
kill -1 17296<br />
<br />
You may now ssh in.<br />
<br />
Once you're done, IF you enabled root login, you should repeat steps 2 and 3 to disable root logins.<br />
<br />
== Letting someone in who has locked themselves out (killed sshd, lost pwd) ==<br />
<br />
There are two ways people frequently lock themselves out - either they forget a password, or they kill off sshd somehow.<br />
<br />
These are actually both fairly easy to solve. First, let's say someone kills off their sshd, or somehow mangles /etc/ssh/sshd_config such that it no longer lets them in.<br />
<br />
Their email may be very short, or it may have all sorts of details about how you should fix sshd_config to let them in ... just ignore all of this. They can fix their own mangled sshd. Fixing this is very simple. First, edit the /etc/inetd.conf on their system and uncomment the telnet line:<br />
<br />
telnet stream tcp nowait root /usr/libexec/telnetd telnetd<br />
#telnet stream tcp6 nowait root /usr/libexec/telnetd telnetd<br />
<br />
(just leave the tcp6 version of telnet commented)<br />
<br />
Then, use jailps to list the processes on their system, and find their inetd process. Then simply:<br />
<br />
kill -HUP (pid)<br />
<br />
where (pid) is the PID of their inetd process. Now they have telnet running on their system and they can log in and do whatever they need to do.<br />
<br />
The only complications that could occur are:<br />
<br />
a) their firewall config on our firewall has port 23 blocked, in which case you will need to open that - will be covered in a different lesson.<br />
<br />
b) they are not running inetd, so you can't HUP it. If this happens, edit their /etc/rc.conf, add the inetd_enable="YES" line, and then kill<br />
their jail with /tmp/jailkill.pl - then restart their jail with the jail line from their quad/safe file. Easy.<br />
<br />
If they have forgotten a password,<br />
<br />
On 6.x+ you can reset their password with:<br />
jexec <jailID from jls> passwd root<br />
<br />
Note: the default password for 6.x jails is 8ico2987, for 4.x it is p455agfa<br />
<br />
On 4.x, you need to cd to their etc directory<br />
... for instance:<br />
<br />
cd /mnt/data2/198.78.65.136-col00261-DIR/etc<br />
<br />
and run:<br />
<br />
vipw -d .<br />
<br />
Then paste in these two lines (theres a paste with these):<br />
<br />
root:$1$krszPxhk$xkCepSnz3mIikT3vCtJCt0:0:0::0:0:Charlie &:/root:/bin/csh<br />
user:$1$Mx9p5Npk$QdMU6c8YQqp2FW2M3irEh/:1001:1001::0:0:User &:/home/user:/bin/sh<br />
<br />
overwriting the lines they already have for "user" and "root" - then just tell them that both user and root have been reset to the default password of p455agfa.<br />
<br />
For linux, just passwd inside shell or <br />
vzctl set <veid> --userpasswd root:p455agfa –save<br />
<br />
Starting in 2009 we began giving out randomized passwords for FreeBSD and Linux as the default password. That is stored with each system in Mgmt. You should look for and reset the password to that password in the event of a reset and refer the customer to use their original password from their welcome email- this way we don’t have to send the password again via email (in clear text).<br />
<br />
<br />
== sendmail can’t be contacted from ext ip (only locally) ==<br />
<br />
By default redhat puts this line in sendmail.mc:<br />
<br />
DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')<br />
<br />
which makes it only answer on localhost. Comment it out like:<br />
<br />
dnl DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')<br />
<br />
and then rebuild sendmail.cf with:<br />
<br />
m4 /etc/mail/sendmail.mc > /etc/sendmail.cf<br />
<br />
== virt doesn’t properly let go of ve’s ip(s) when moved to another system ==<br />
<br />
On virtuozzo 2.6 systems, it's been observed that when moving ips from one virt to another that sometimes the routing table will not get updated to reflect the removal of the ip addresses.<br />
<br />
A recent example was a customer that was moving to a new ve on a new virt and the ip addresses were traded between the two ve's. After the trade the two systems were not able to talk to each other. When looking at the routing table for the old system all the ip addresses were still in the routing table as being local, like so:<br />
<br />
<pre>netstat -rn | grep 69.55.225.149<br />
69.55.225.149 0.0.0.0 255.255.255.255 UH 40 0 0 venet0</pre><br />
<br />
This was preventing traffic to the other system from being routed properly.<br />
The solution is to manually delete the route:<br />
<br />
route delete 69.55.225.149 gw 0.0.0.0<br />
<br />
Supposedly, this was fixed in 2.6.1<br />
<br />
== sshd on FreeBSD 6.2 segfaults ==<br />
<br />
First try to reinstall ssh<br />
<br />
<pre>cd /usr/src/secure<br />
cd lib/libssh<br />
make depend && make all install<br />
cd ../../usr.sbin/sshd<br />
make depend && make all install<br />
cd ../../usr.bin/ssh<br />
make depend && make all install</pre><br />
<br />
Failing that, find the library that’s messed up:<br />
<br />
<pre>ldd /usr/sbin/sshd<br />
libssh.so.3 => /usr/lib/libssh.so.3 (0x280a3000) <br />
libutil.so.5 => /lib/libutil.so.5 (0x280d8000) <br />
libz.so.3 => /lib/libz.so.3 (0x280e4000) <br />
libwrap.so.4 => /usr/lib/libwrap.so.4 (0x280f5000) <br />
libpam.so.3 => /usr/lib/libpam.so.3 (0x280fc000) <br />
libbsm.so.1 => /usr/lib/libbsm.so.1 (0x28103000) <br />
libgssapi.so.8 => /usr/lib/libgssapi.so.8 (0x28112000) <br />
libkrb5.so.8 => /usr/lib/libkrb5.so.8 (0x28120000) <br />
libasn1.so.8 => /usr/lib/libasn1.so.8 (0x28154000) <br />
libcom_err.so.3 => /usr/lib/libcom_err.so.3 (0x28175000) <br />
libroken.so.8 => /usr/lib/libroken.so.8 (0x28177000) <br />
libcrypto.so.4 => /lib/libcrypto.so.4 (0x28183000) <br />
libcrypt.so.3 => /lib/libcrypt.so.3 (0x28276000) <br />
libc.so.6 => /lib/libc.so.6 (0x2828e000) <br />
libmd.so.3 => /lib/libmd.so.3 (0x28373000)</pre><br />
<br />
md5 them and compare to other jail hosts or jails running on host<br />
<br />
for libcrypto reinstall:<br />
<pre>/usr/src/crypto<br />
make depend && make all install</pre><br />
<br />
= FreeBSD VPS =<br />
<br />
== Starting jails: Quad/Safe Files ==<br />
<br />
FreeBSD customer systems do not start up automatically at boot time. When one of our freebsd machines boots up, it boots up, and does nothing else. To start jails, we put the commands to start each jail into a shell script(s) and run the script(s). Jail startup is something that needs to be actively monitored, which is why we don’t just run the script automatically. More on monitoring later.<br />
<br />
NOTE: >=7.x we have moved to 1 quad file: <tt>quad1</tt>. Startups are not done by running each quad, but rather [[#startalljails|startalljails]] which relies on the contents of <tt>quad1</tt>. The specifics of this are lower in this article. What follows here applies for pre 7.x systems.<br />
<br />
There are eight files in <tt>/usr/local/jail/rc.d</tt>:<br />
<br />
<pre>jail3# ls /usr/local/jail/rc.d/<br />
quad1 quad2 quad3 quad4 safe1 safe2 safe3 safe4<br />
jail3#</pre><br />
<br />
four quad files and four safe files.<br />
<br />
Each file contains an even number of system startup blocks (total number of jails divided by 4)<br />
<br />
The reason for this is, if we make one large script to startup all the systems at boot time, it will take too long - the first system in the script will start up right after system boot, which is great, but the last system may not start for another 20 minutes.<br />
<br />
Since there is no way to parralelize this during the startup procedure, we simply open four terminals (in screen window 9) and run each script, one in each terminal. This way they all run simultaneously, and the very last system in each startup script gets started in 1/4th the time it would if there was one large file<br />
<br />
The files are generally organized so that quad/safe 1&2 have only jails from disk 1, and quad/safe 3&4 have jails from disk 2. This helps ensure that only 2 fscks on any disk are going on at once. Further, they are balanced so that all quad/safe’s finish executing around the same time. We do this by making sure each quad/safe has a similar number of jails and represents a similar number of inodes (see js).<br />
<br />
The other, very important reason we do it this way, and this is the reason there are quad files and safe files, is that in the event of a system crash, every single vn-backed filesystem that was mounted at the time of system crash needs to be fsck'd. However, fsck'ing takes time, so if we shut the system down gracefully, we don't want to fsck.<br />
<br />
Therefore, we have two sets of scripts - the four quad scripts are identical to the four safe scripts except for the fact that the quad scripts contain fsck commands for each filesystem.<br />
<br />
So, if you shut a system down gracefully, start four terminals and run safe1 in window one, and safe2 in window 2, and so on.<br />
<br />
If you crash, start four terminals (or go to screen window 9) and run quad1 in window one, and quad2 in window 2, and so on.<br />
<br />
Here is a snip of (a 4.x version) quad2 from jail17:<br />
<br />
<pre>vnconfig /dev/vn16 /mnt/data2/69.55.228.7-col00820<br />
fsck -y /dev/vn16<br />
mount /dev/vn16c /mnt/data2/69.55.228.7-col00820-DIR<br />
chmod 0666 /mnt/data2/69.55.228.7-col00820-DIR/dev/null<br />
jail /mnt/data2/69.55.228.7-col00820-DIR mail1.phimail.com 69.55.228.7 /bin/sh /etc/rc<br />
<br />
# moved to data2 col00368<br />
#vnconfig /dev/vn28 /mnt/data2/69.55.236.132-col00368<br />
#fsck -y /dev/vn28<br />
#mount /dev/vn28c /mnt/data2/69.55.236.132-col00368-DIR<br />
#chmod 0666 /mnt/data2/69.55.236.132-col00368-DIR/dev/null<br />
#jail /mnt/data2/69.55.236.132-col00368-DIR limehouse.org 69.55.236.132 /bin/sh /etc/rc<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
vnconfig /dev/vn22 /mnt/data2/69.55.228.13-col01063<br />
fsck -y /dev/vn22<br />
mount /dev/vn22c /mnt/data2/69.55.228.13-col01063-DIR<br />
chmod 0666 /mnt/data2/69.55.228.13-col01063-DIR/dev/null<br />
jail /mnt/data2/69.55.228.13-col01063-DIR www.widestream.com.au 69.55.228.13 /bin/sh /etc/rc<br />
<br />
# cancelled col00106<br />
#vnconfig /dev/vn15 /mnt/data2/69.55.238.5-col00106<br />
#fsck -y /dev/vn15<br />
#mount /dev/vn15c /mnt/data2/69.55.238.5-col00106-DIR<br />
#chmod 0666 /mnt/data2/69.55.238.5-col00106-DIR/dev/null<br />
#jail /mnt/data2/69.55.238.5-col00106-DIR mail.azebu.net 69.55.238.5 /bin/sh /etc/rc</pre><br />
<br />
As you can see, two of the systems specified are commented out - presumably those customers cancelled, or were moved to new servers.<br />
<br />
As you can see, the vnconfig line is the simpler command line, not the longer one that was used when it was first configured. As you can see, all that is done is, vnconfig the filesystem, then fsck it, then mount it. The fourth command is the `jail` command used to start the system – but that will be covered later.<br />
<br />
Here is the safe2 file from jail17:<br />
<br />
<pre>vnconfig /dev/vn16 /mnt/data2/69.55.228.7-col00820<br />
mount /dev/vn16c /mnt/data2/69.55.228.7-col00820-DIR<br />
chmod 0666 /mnt/data2/69.55.228.7-col00820-DIR/dev/null<br />
jail /mnt/data2/69.55.228.7-col00820-DIR mail1.phimail.com 69.55.228.7 /bin/sh /etc/rc<br />
<br />
# moved to data2 col00368<br />
#vnconfig /dev/vn28 /mnt/data2/69.55.236.132-col00368<br />
#mount /dev/vn28c /mnt/data2/69.55.236.132-col00368-DIR<br />
#chmod 0666 /mnt/data2/69.55.236.132-col00368-DIR/dev/null<br />
#jail /mnt/data2/69.55.236.132-col00368-DIR limehouse.org 69.55.236.132 /bin/sh /etc/rc<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
vnconfig /dev/vn22 /mnt/data2/69.55.228.13-col01063<br />
mount /dev/vn22c /mnt/data2/69.55.228.13-col01063-DIR<br />
chmod 0666 /mnt/data2/69.55.228.13-col01063-DIR/dev/null<br />
jail /mnt/data2/69.55.228.13-col01063-DIR www.widestream.com.au 69.55.228.13 /bin/sh /etc/rc<br />
<br />
# cancelled col00106<br />
#vnconfig /dev/vn15 /mnt/data2/69.55.238.5-col00106<br />
#mount /dev/vn15c /mnt/data2/69.55.238.5-col00106-DIR<br />
#chmod 0666 /mnt/data2/69.55.238.5-col00106-DIR/dev/null<br />
#jail /mnt/data2/69.55.238.5-col00106-DIR mail.azebu.net 69.55.238.5 /bin/sh /etc/rc</pre><br />
<br />
As you can see, it is exactly the same, but it does not have the fsck lines.<br />
<br />
Take a look at the last entry - note that the file is named:<br />
<br />
/mnt/data2/69.55.238.5-col00106<br />
<br />
and the mount point is named:<br />
<br />
/mnt/data2/69.55.238.5-col00106-DIR<br />
<br />
This is the general format on all the FreeBSD systems. The file is always named:<br />
<br />
IP-custnumber<br />
<br />
and the directory is named:<br />
<br />
IP-custnumber-DIR<br />
<br />
If you run safe when you need a fsck, the mount will fail and jail will fail:<br />
<br />
# mount /dev/vn1c /mnt/data2/jails/65.248.2.131-ns1.kozubik.com-DIR<br />
mount: /dev/vn1c: Operation not permitted<br />
<br />
No reboot needed, just run the quad script<br />
<br />
Starting with 6.x jails, we added block delimiters to the quad/safe files, the block looks like:<br />
<br />
<pre>echo '## begin ##: nuie.solaris.mu'<br />
fsck -y /dev/concat/v30v31a<br />
mount /dev/concat/v30v31a /mnt/data1/69.55.228.218-col01441-DIR<br />
mount_devfs devfs /mnt/data1/69.55.228.218-col01441-DIR/dev<br />
devfs -m /mnt/data1/69.55.228.218-col01441-DIR/dev rule -s 3 applyset<br />
jail /mnt/data1/69.55.228.218-col01441-DIR nuie.solaris.mu 69.55.228.218 /bin/sh /etc/rc<br />
echo '## end ##: nuie.solaris.mu'</pre><br />
<br />
These are more than just informative when running quad/safe’s, the echo lines MUST be present for certain tools to work properly. So it’s important that any updates to the hostname also be updated on the 2 echo lines. For example, if you try to startjail a jail with a hostname which is on the jail line but not the echo lines, the command will return with host not found.<br />
<br />
=== FreeBSD 7.x+ notes ===<br />
<br />
Starting with the release of FreeBSD 7.x, we are doing jail startups in a slightly different way. First, thereis only 1 file: <tt>/usr/local/jail/rc.d/quad1</tt><br />
There are no other quads or corresponding safe files. The reason for this is twofold, 1. We can pass –C to fsck which will tell is to skip the fsck if the fs is clean (no more need for safe files), 2. We have a new startup script which can be launched multiple times, running in parallel to start jails, where quad1 is the master jail file. <br />
Quad1 could still be run as a shell script, but it would take a very long time for it to run completely so it’s not advisable; or you should break it down into smaller chunks (like quad1, quad2, quad3, etc)<br />
<br />
Here is a snip of (a 7.x version) quad1 from jail2:<br />
<br />
<pre>echo '## begin ##: projects.tw.com'<br />
mdconfig -a -t vnode -f /mnt/data1/69.55.230.46-col01213 -u 50<br />
fsck -Cy /dev/md50c<br />
mount /dev/md50c /mnt/data1/69.55.230.46-col01213-DIR<br />
mount -t devfs devfs /mnt/data1/69.55.230.46-col01213-DIR/dev<br />
devfs -m /mnt/data1/69.55.230.46-col01213-DIR/dev rule -s 3 applyset<br />
jail /mnt/data1/69.55.230.46-col01213-DIR projects.tw.com 69.55.230.46 /bin/sh /etc/rc<br />
echo '## end ##: projects.tw.com'</pre><br />
<br />
Cancelled jails are no longer commented out and stored in quad1, rather they’re moved to <tt>/usr/local/jail/rc.d/deprecated</tt> <br />
<br />
To start these jails, start the 4 ssh sessions as you would for a normal crash and then instead of running quad1-4, instead run startalljails in each window. IMPORTANT- before running startalljails you should make sure you ran preboot once as it will clear out all the lockfiles and enable startalljails to work properly.<br />
<br />
== Problems with the quad/safe files ==<br />
<br />
When you run the quad/safe files, there are two problems that can occur - either a particular system will hang during initialization, OR a system will spit out output to the screen, impeding your ability to do anything. Or both.<br />
<br />
First off, when you start a jail, you see output like this:<br />
<br />
<pre>Skipping disk checks ...<br />
adjkerntz[25285]: sysctl(put_wallclock): Operation not permitted<br />
Doing initial network setup:.<br />
ifconfig: ioctl (SIOCDIFADDR): permission denied<br />
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384<br />
Additional routing options: TCP keepalive=YESsysctl:<br />
net.inet.tcp.always_keepalive: Operation not permitted.<br />
Routing daemons:.<br />
Additional daemons: syslogd.<br />
Doing additional network setup:.<br />
Starting final network daemons:.<br />
ELF ldconfig path: /usr/lib /usr/lib/compat /usr/X11R6/lib /usr/local/lib<br />
a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout /usr/X11R6/lib/aout<br />
Starting standard daemons: inetd cron sshd sendmail sendmail-clientmqueue.<br />
Initial rc.i386 initialization:.<br />
Configuring syscons: blanktime.<br />
Additional ABI support:.<br />
Local package initialization:.<br />
Additional TCP options:.</pre><br />
<br />
Now, let's look at this line, near the end:<br />
<br />
Local package initialization:.<br />
<br />
This is where a list of daemons that are set to start at boot time willshow up. You might see something like:<br />
<br />
Local package initialization: mysqld apache sendmail sendmail-clientmqueue<br />
<br />
Or something like this:<br />
<br />
Local package initialization: postgres postfix apache<br />
<br />
The problem is that many systems (about 4-5 per machine) will hang on that line. Basically it will get to some of the way through the total daemons to be started:<br />
<br />
Local package initialization: mysqld apache<br />
<br />
and will just sit there. Forever.<br />
<br />
Fortunately, pressing ctrl-c will break out of it. Not only will it break out of it, but it will also continue on that same line and start the other daemons:<br />
<br />
Local package initialization: mysqld apache ^c sendmail-clientmqueue<br />
<br />
and then continue on to finish the startup, and then move to the next system to be started.<br />
<br />
So what does this mean? It means that if a machine crashes, and you start four screen-windows to run four quads or four safes, you need to periodically cycle between them and see if any systems are stuck at that point, causing their quad/safe file to hang. A good rule of thumb is, if you see a system at that point in the startup, give it another 100 seconds - if it is still at the exact same spot, hit ctrl-c. Its also a good idea to go back into the quad file (just before the first command in the jail startup block) and note that this jail tends to need a control-c or more time as follows:<br />
<pre>echo '### NOTE ### slow sendmail'<br />
echo '### NOTE ###: ^C @ Starting sendmail.'</pre><br />
<br />
'''NEVER''' hit ctrl-c repeatedly if you don't get an immediate response - that will cause the following jail’s startup commands to be aborted.<br />
<br />
A second problem that can occur is that a jail - maybe the first one in that particular quad/safe, maybe the last one, or maybe one in the middle, will start spitting out status or error messages from one of its init scripts. This is not a problem - basically, hit enter a few times and see if you get a prompt - if you do get a prompt, that means that the quad/safe script has already completed. Therefore it is safe to log out (and log out of the user that you su'd from) and then log back in (if necessary).<br />
<br />
The tricky thing is, if a system in the middle starts flooding with messages, and you hit enter a few times and don't get a prompt. Are you not getting a prompt because some subsequent system is hanging at the initialization, as we discussede above ? Or are you not getting a prompt because that quad file is currently running an fsck ? Usually you can tell by scrolling back in screen’s history to see what it was doing before you started getting the messages.<br />
<br />
If you don’t get clues from history, you have to use your judgement - instead of giving it 100 seconds to respond, perhaps give it 2-3 mins ... if you still get no response (no prompt) when you hit enter, hit ctrl-c. However, be aware that you might still be hitting ctrl-c in the middle of an fsck. This means you will get an error like "filesystem still marked dirty" and then the vnconfig for it will fail and so will the jail command, and the next system in the quad file will then start starting up.<br />
<br />
If this happens, just wait until the end of all the quad files have finished, and start that system manually.<br />
<br />
If things really get weird, like a screen flooded with errors, and you can't get a prompt, and ctrl-c does nothing, then you need to just eventually (give it ten mins or so) just kill that window with ctrl-p, then k, and then log in again and manually check which systems are now running and which aren't, and manually start up any that are not.<br />
<br />
Don't EVER risk running a particular quad/safe file a second time.<br />
If the quad/safe script gets executed twice, reboot the machine immediately.<br />
<br />
So, for all the above reasons, anytime a machine crashes and you run all the quads or all the safes, '''always''' check every jail afterwards to make sure it is running - even if you have no hangs or complications at all.<br />
Run this command:<br />
<br />
<tt>[[#jailpsall|jailpsall]]</tt> <br />
<br />
Note: [[#postboot|postboot]] also populates ipfw counts, so it '''should not be run multiple times''', use <tt>jailpsall</tt> for subsequent extensive ps’ing<br />
<br />
And make sure they all show as running. If one does not show as running, check its /etc/rc.conf file first to see if maybe it is using a different hostname first before starting it manually.<br />
<br />
One thing we have implemented to alleviate these startup hangs and noisy jails, is to put jail start blocks that are slow or hangy at the bottom of the safe/quad file. Further, for each bad jail we note in each quad/safe just before the start block something like:<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
<br />
That way we’ll be prepared to ^C when we see that message appear during the quad/safe startup process. If you observe a new, undocumented hang, '''after''' the quad/safe has finished, place a line similar to the above in the quad file, move the jail start block to the end of the file, then run [[#buildsafe|buildsafe]]<br />
<br />
<br />
== Recovering from a crash (FreeBSD) ==<br />
<br />
=== Diagnose whether you have a crash ===<br />
The most important thing is to get the machine and all jails back up as soon as possible. Note the time, you’ll need to create a crash log entry (Mgmt. -> Reference -> CrashLog). The first thing to do is head over to the [[Screen#Screen_Organization|serial console screen]] and see if there’s any kernel error messages output. Try to copy any messages (or just a sample of repeating messages) you see into the notes section of the crash log. If there are no messages, the machine may just be really busy- wait a bit (5-10min) to see if it comes back. If it's still pinging, odds are its very busy. Note, if you see messages about swap space exhausted, the server is obviously out of memory, however it may recover briefly enough for you to get a jtop in to see who's lauched a ton of procs (most likely) and then issue a quick jailkill to get it back under control.<br />
<br />
If it doesn't come back, or the messages indicate a fatal error, you will need to proceed with a power cycle (ctrl+alt+del will not work).<br />
<br />
=== Power cycle the server ===<br />
If this machine is not a Dell 2950 with a [[DRAC/RMM#DRAC|DRAC card]] (i.e. if you can’t ssh into the DRAC card (as root, using the standard root pass) and issue <br />
racadm serveraction hardreset<br />
then you will need someone at the data center power the macine off, wait 30 sec, then turn it back on. Make sure to re-attach via console:<br />
tip jailX<br />
immediately after power down. <br />
<br />
=== (Re)attach to the console ===<br />
Stay on the console the entire time during boot. As the BIOS posts- look out for the RAID card output- does everything look healthy? The output may be scrambled, look for "DEGRADED" or "FAILED". Once the OS starts booting you will be disconnected (dropped back to the shell on the console server) a couple times during the boot up. The reason you want to quickly re-attach is two-fold: 1. If you don’t reattach quickly then you won’t get any console output, 2. you want to be attached before the server ''potentially'' starts (an extensive) fsck. If you attach after the fsck begins, you’ll have seen no indication it started an fsck and the server will appear frozen during startup- no output, no response. <br />
<br />
IMPORTANT NOTE: on some older FreeBSD systems, there will be no output to the video (KVM) console as it boots up. The console output is redirected to the serial port ... so if a jail crashes, and you attach a kvm, the output during the bootup procedure will not be shown on the screen. However, when the bootup is done, you will get a login prompt on the screen and will be able to log in as normal. <tt>/boot/loader.conf</tt> is where serial console redirect output lives, so comment that if you want to catch output on kvm.<br />
On newer systems it sends most output to both locations. <br />
<br />
=== Assess the heath of the server ===<br />
Once the server boots up fully, you should be able to ssh in. Look around- make sure all the mounts are there and reporting the correct size/usage (i.e. /mnt/data1 /mnt/data2 /mnt/data3 - look in /etc/fstab to determine which mount points should be there), check to see if RAID mirrors are healthy. See [[RAID_Cards#Common_CLI_commands_.28megacli.29|megacli]], [[#aaccheck|aaccheck]]<br />
<br />
Before you start the jails, you need to run [[#preboot|preboot]]. This will do some assurance checks to make sure things are prepped to start the jails. Any issues that come out of preboot need to be addressed before starting jails.<br />
<br />
=== Start jails ===<br />
[[#Starting_jails:_Quad.2FSafe_Files|More on starting jails]]<br />
Customer jails (the VPSs) do not start up automatically at boot time. When a FreeBSD machines boots up, it boots up, and does nothing else. To start jails, we put the commands to start each jail into a shell script(s) and run the script(s). Jail startup is something that needs to be actively monitored, which is why we don’t just run the script automatically. <br />
<br />
In order to start jails, we run the quad files: quad1 quad2 quad3 and quad4 (on new systems there is only quad1). If the machine was cleanly rebooted- which wouldn't be the case if this was a crash, you may run the safe files (safe1 safe2 safe3 safe4) in lieu of quads. <br />
<br />
Open up 4 logins to the server (use the windows in [[Screen#Screen_Organization|a9]])<br />
In each of the 4 windows you will:<br />
<br />
If there is a [[#startalljails|startalljails]] script (and only quad1), run that command in each of the 4 windows. It will parse through the quad1 file and start each jail. Follow the instructions [[#Problems_with_the_quad.2Fsafe_files|here]] for monitoring startup. Note that you can be a little more lenient with jails that take awhile to start- startalljails will work around the slow jails and start the rest. As long as there aren't 4 jails which are "hung" during startup, the rest will get started eventually.<br />
-or-<br />
If there is no startalljails script, there will be multiple quad files. In each of the 4 windows, start each of the quads. i.e. start quad1 in window1, quad2 in window2 and so on. DO NOT start any quad twice. It will crash the server. If you accidentally do this, just jailkill all the jails which are in the quad and run the quad again. Follow the instructions here for monitoring quad startup.<br />
<br />
Note the time the last jail boots- this is what you will enter in the crash log.<br />
<br />
Save the crash log.<br />
<br />
=== Check to make sure all jails have started ===<br />
There's a simple script which will make sure all jails have started, and enter the ipfw counter rules: [[#postboot|postboot]] <br />
Run postboot, which will do a jailps on each jail it finds (excluding commented out jails) in the quad file(s). We're looking for 2 things:<br />
# systems spawning out of control or too many procs<br />
# jails which haven't started<br />
On 7.x and newer systems it will print out the problems (which jails haven't started) at the conclusion of postboot. <br />
On older systems you will need to watch closely to see if/when there's a problem, namely:<br />
<br />
[hostname] doesnt exist on this server<br />
<br />
When you get this message, it means one of 2 things:<br />
1. the jail really didn't start:<br />
When a jail doesn't start it usually boils down to a problem in the quad file. Perhaps the path name is wrong (data1 vs data2) or the name of the vn/mdfile is wrong. Once this is corrected, you will need to run the commands from the quad file manually, or you may use <tt>startjail <hostname></tt><br />
<br />
2. the customer has changed their hostname (and not told us) so their jail ''is'' running, just under a different hostname:<br />
On systems with jls, this is easy to rectify. First, get the customer info: <tt>g <hostname></tt><br />
Then look for the customer in jls: <tt>jls | grep <col0XXXX></tt><br />
From there you will see their new hostname- you should update that hostname in the quad file: don't forget to edit it on the <tt>## begin ##</tt> and <tt>## end ##</tt> lines, and in mgmt. <br />
On older systems without jls, this will be harder, you will need to look further to see their hostname- perhaps its in their /etc/rc.conf<br />
<br />
<br />
Once all jails are started, do some spot checks- try to ssh or browse to some customers, just to make sure things are really ok.<br />
<br />
== Adding disk to a 7.x/8.x jail ==<br />
or<br />
== Moving customer to a different drive (md) ==<br />
<br />
NOTE: this doesn’t apply to mx2 which uses gvinum. Use same procedure as 6.x<br />
NOTE: if you unmount before mdconfig, re-mdconfig (attach) then unmount then mdconfig -u again <br />
<br />
<br />
(parts to change/customize are <tt><span style="color:red">red</span></tt>)<br />
<br />
<br />
If someone wants more disk space, there’s a paste for it, send it to them (explains about downtime, etc).<br />
<br />
1. Figure out the space avail from <tt>js</tt>. Ideally, you want to put the customers new space on a different partition (and create the new md on the new partition). <br />
<br />
<br />
2. make a mental note of how much space they're currently using<br />
<br />
<br />
3. <tt>jailkill <hostname></tt> <br />
<br />
<br />
4. Umount it (including their devfs) but leave the md config’d (so if you use stopjail, you will have to re-mdconfig it)<br />
<br />
<br />
5. <tt>g <customerID></tt> to get the info (IP/cust#) needed to feed the new mdfile and mount name, and to see the current md device. <br />
<br />
<br />
6a. When there's enough room to place new system on an alternate, or the same drive:<br />
USE CAUTION not to overwrite (touch, mdconfig) existing md!!<br><br />
<tt>touch /mnt/data<span style="color:red">3/69.55.234.66-col01334</span><br><br />
mdconfig -a -t vnode -s 10g -f /mnt/data3/69.55.234.66-col01334 -u 97<br><br />
newfs /dev/md97</tt><br />
<br />
<br />
Optional- if new space is on a different drive, move the mount point directory AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
<tt>mount /dev/md<span style="color:red">97</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<br />
confirm you are mounted to /dev/md<span style="color:red">97</span> and space is correct:<br><br />
<tt>df .</tt> <br />
<br />
<br />
do the dump and pipe directly to restore:<br><br />
<tt>dump -0a -f - /dev/md<span style="color:red">1</span> | restore -r -f - <br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use <tt>df</tt> to confirm the restored data size matches the original usage figure<br />
<br />
<br />
md-unconfig old system:<br><br />
<tt>mdconfig -d -u <span style="color:red">1</span></tt><br />
<br />
<br />
archive old mdfile. <br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.237.26-col00241</span> /mnt/data<span style="color:red">1/old-col00241-mdfile-noarchive-20091211</span></tt><br />
<br />
<br />
edit quad (vq1) to point to new (/mnt/data<span style="color:red">3</span>) location AND new md number (md<span style="color:red">97</span>)<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
6b. When there's not enough room on an alternate partition, or on the same drive...but there is enough room if you were to remove the existing customer's space:<br />
<br />
<br />
mount backup nfs mounts:<br><br />
<tt>mbm</tt><br> <br />
(run <tt>df</tt> to confirm backup mounts are mounted)<br />
<br />
<br />
dump the customer to backup2 or backup1<br><br />
<tt>dump -0a -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> /dev/md<span style="color:red">1</span></tt><br><br />
(when complete WITHOUT errors, <tt>du</tt> the dump file to confirm it matches size, roughly, with usage)<br />
<br />
<br />
unconfigure and remove old mdfile<br><br />
<tt>mdconfig -d -u <span style="color:red">1</span><br><br />
rm /mnt/data<span style="color:red">1/69.55.237.26-col00241</span></tt><br><br />
(there should now be enough space to recreate your bigger system. If not, run sync a couple times)<br />
<br />
<br />
create the new system (ok to reuse old mdfile and md#):<br><br />
<tt>touch /mnt/data<span style="color:red">1/69.55.234.66-col01334</span><br><br />
mdconfig -a -t vnode -s <span style="color:red">10</span>g -f /mnt/data<span style="color:red">1/69.55.234.66-col01334</span> -u <span style="color:red">1</span><br><br />
newfs /dev/md<span style="color:red">1</span><br><br />
mount /dev/md<span style="color:red">1</span> /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> <br><br />
cd /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<br />
confirm you are mounted to /dev/md<span style="color:red">1</span> and space is correct:<br><br />
<tt>df .</tt> <br />
<br />
<br />
do the restore from the dumpfile on the backup server:<br><br />
<tt>restore -r -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> .<br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
umount nfs:<br><br />
<tt>mbu</tt><br />
<br />
<br />
If md# changed (or mount point), edit quad (<tt>vq1</tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new md number (md<span style="color:red">1</span>)<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
7. update disk (and dir if applicable) in mgmt screen<br />
<br />
<br />
8. update backup list AND move backups, if applicatble<br />
<br />
Ex: <tt>mvbackups <span style="color:red">69.55.237.26-col00241</span> jail<span style="color:red">9</span> data<span style="color:red">3</span></tt><br />
<br />
<br />
9. Optional: archive old mdfile<br />
<br />
<tt>mbm<br><br />
gzip -c old-col01588-mdfile-noarchive-20120329 > /deprecated/old-col01588-mdfile-noarchive-20120329.gz<br><br />
mbu<br><br />
rm old-col01588-mdfile-noarchive-20120329</tt><br />
<br />
== Adding disk to a 6.x jail (gvinum/gconcat) ==<br />
or<br />
== Moving customer to a different drive (gvinum/gconcat) ==<br />
<br />
(parts to change are <span style="color:red">highlighted</span>)<br />
<br />
<br />
If someone wants more disk space, there’s a paste for it, send it to them (explains about downtime, etc).<br />
<br />
<br />
1. Figure out the space avail from [[#js|js]]. Ideally, you want to put the customers new space on a different partition (and create the new md on the new partition). <br />
<br />
<br />
2. make a mental note of how much space they're currently using<br />
<br />
<br />
3. <tt>[[#stopjail|stopjail]] <hostname></tt> <br />
<br />
<br />
4. <tt>[[#g|g]] <customerID></tt> to get the info (IP/cust#) needed to feed the new mount name and existing volume/device. <br />
<br />
<br />
5a. When there's enough room to place new system on an alternate, or the same drive (using only UNUSED - including if it's in use by the system in question - gvinum volumes):<br />
<br />
<br />
Configure the new device:<br><br />
A. for a 2G system (single gvinum volume):<br><br />
<tt>bsdlabel -r -w /dev/gvinum/v<span style="color:red">123</span><br><br />
newfs /dev/gvinum/v<span style="color:red">123</span>a</tt><br><br />
<br />
-or- <br />
<br />
<br />
B. for a >2G system (create a gconcat volume):<br><br />
try to grab a contiguous block of gvinum volumes. gconcat volumes MAY NOT span drives (i.e. you cannot use a gvinum volume from data3 and a volume from data2 in the same gconcat volume). <br />
<br />
<tt>gconcat label <span style="color:red">v82-v84 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84</span><br><br />
bsdlabel -r -w /dev/concat/<span style="color:red">v82-v84</span><br><br />
newfs /dev/concat/<span style="color:red">v82-v84</span>a</tt><br />
<br />
<br />
Other valid gconcat examples:<br><br />
<tt>gconcat label v82-v84v109v112 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84 /dev/gvinum/v109 /dev/gvinum/v112<br><br />
gconcat label v82v83 /dev/gvinum/v82 /dev/gvinum/v83</tt> <br><br />
Note, long names will truncate: v144v145v148-v115 will truncate to v144v145v148-v1 (so you will refer to it as v144v145v148-v1 thereafter)<br />
<br />
<br />
Optional- if new volume is on a different drive, move the mount point directory (get the drive from js output) AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
<br />
confirm you are mounted to the device (<tt>/dev/gvinum/v<span style="color:red">123</span>a</tt> OR <tt>/dev/concat/<span style="color:red">v82-v84</span></tt>) and space is correct:<br><br />
A. <tt>mount /dev/gvinum/v<span style="color:red">123</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br><br />
-or-<br><br />
B. <tt>mount /dev/concat/<span style="color:red">v82-v84</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<tt>cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
df .</tt> <br />
<br />
<br />
do the dump and pipe directly to restore:<br><br />
<tt>dump -0a -f - /dev/gvinum/v<span style="color:red">1</span> | restore -r -f - <br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
edit quad (<tt>vq<span style="color:red">1</span></tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new volume (<tt>/dev/gvinum/v<span style="color:red">123</span>a</tt> or <tt>/dev/concat/<span style="color:red">v82-v84</span>a</tt>) , run <tt>buildsafe</tt><br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
5b. When there's not enough room on an alternate partition, or on the same drive...but there is enough room if you were to remove the existing customer's space (i.e. if you want/need to reuse the existing gvinum volumes and add on more):<br />
<br />
<br />
mount backup nfs mounts:<br><br />
<tt>mbm</tt> <br><br />
(run df to confirm backup mounts are mounted)<br />
<br />
<br />
dump the customer to backup2 or backup1<br><br />
<tt>dump -0a -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> /dev/<span style="color:red">gconcat/v106-v107</span></tt><br><br />
(when complete WITHOUT errors, du the dump file to confirm it matches size, roughly, with usage)<br />
<br />
<br />
unconfigure the old gconcat volume<br><br />
list member gvinum volumes:<br />
<br />
<tt>gconcat list <span style="color:red">v106v107</span></tt><br />
<br />
<br />
Output will resemble:<br><br />
<pre>Geom name: v106v107<br />
State: UP<br />
Status: Total=2, Online=2<br />
Type: AUTOMATIC<br />
ID: 3530663882<br />
Providers:<br />
1. Name: concat/v106v107<br />
Mediasize: 4294966272 (4.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e2<br />
Consumers:<br />
1. Name: gvinum/sd/v106.p0.s0<br />
Mediasize: 2147483648 (2.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e3<br />
Start: 0<br />
End: 2147483136<br />
2. Name: gvinum/sd/v107.p0.s0<br />
Mediasize: 2147483648 (2.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e3<br />
Start: 2147483136<br />
End: 4294966272</pre><br />
<br />
stop volume and clear members<br><br />
<tt>gconcat stop <span style="color:red">v106v107</span><br><br />
gconcat clear <span style="color:red">gvinum/sd/v106.p0.s0 gvinum/sd/v107.p0.s0</span></tt><br />
<br />
<br />
create new device- and its ok to reuse old/former members<br><br />
try to grab a contiguous block of gvinum volumes. gconcat volumes MAY NOT span drives (i.e. you cannot use a gvinum volume from data3 and a volume from data2 in the same gconcat volume). <br><br />
<tt>gconcat label <span style="color:red">v82-v84v106v107 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84 /dev/gvinum/v106 /dev/gvinum/v107</span> <br><br />
bsdlabel -r -w /dev/concat/<span style="color:red">v82-v84v106v107</span><br><br />
newfs /dev/concat/<span style="color:red">v82-v84v106v107</span>a</tt><br />
<br />
<br />
Optional- if new volume is on a different drive, move the mount point directory (get the drive from js output) AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
confirm you are mounted to the device (/dev/concat/<span style="color:red">v82-v84v106v107</span>) and space is correct:<br><br />
<tt>mount /dev/concat/<span style="color:red">v82-v84v106v107</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<tt>cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
df .</tt> <br />
<br />
do the restore from the dumpfile on the backup server:<br><br />
<tt>restore -r -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> .<br><br />
rm restoresymtable</tt><br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
edit quad (<tt>vq<span style="color:red">1</span></tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new volume (<tt>/dev/concat/<span style="color:red">v82-v84v106v107</span>a</tt>), run buildsafe<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
TODO: clean up/clear old gvin/gconcat vol<br />
<br />
<br />
6. update disk (and dir if applicable) in mgmt screen<br />
<br />
<br />
7. update backup list AND move backups, if applicatble<br />
<br />
<br />
Ex: <tt>mvbackups <span style="color:red">69.55.237.26-col00241</span> jail<span style="color:red">9</span> data<span style="color:red">3</span></tt><br />
<br />
<br />
<br />
DEPRECATED - steps to tack on a new gvin to existing gconcat- leads to corrupted fs<br />
bsdlabel -e /dev/concat/v82-v84<br />
<br />
To figure out new size of the c partition, multiply 4194304 by the # of 2G gvinum volumes and subtract the # of 2G volumes:<br />
10G: 4194304 * 5 – 5 = 20971515<br />
8G: 4194304 * 4 – 4 = 16777212<br />
6G: 4194304 * 3 – 3 = 12582909<br />
4G: 4194304 * 2 – 2 = 8388606<br />
<br />
To figure out the new size of the a partition, subtract 16 from the c partition:<br />
10G: 20971515 – 16 = 20971499<br />
8G: 16777212 – 16 = 16777196<br />
6G: 12582909 – 16 = 12582893<br />
4G: 8388606 – 16 = 8388590<br />
<br />
Orig:<br />
8 partitions:<br />
# size offset fstype [fsize bsize bps/cpg]<br />
a: 8388590 16 4.2BSD 2048 16384 28552<br />
c: 8388606 0 unused 0 0 # "raw" part, don't edit<br />
<br />
New:<br />
8 partitions:<br />
# size offset fstype [fsize bsize bps/cpg]<br />
a: 12582893 16 4.2BSD 2048 16384 28552<br />
c: 12582909 0 unused 0 0 # "raw" part, don't edit<br />
<br />
sync; sync<br />
<br />
growfs /dev/concat/v82-v84a<br />
<br />
fsck –fy /dev/concat/v82-v84a<br />
<br />
sync<br />
<br />
fsck –fy /dev/concat/v82-v84a<br />
<br />
(keep running fsck’s till NO errors)<br />
<br />
== Problems un-mounting - and with mount_null’s ==<br />
<br />
If you cannot unmount a filesystem, beacuse it says the filesystem is busy, it is because of three things:<br />
<br />
a) the jail is still running<br />
<br />
b) you are actually in that directory, even though the jail is stopped<br />
<br />
c) there are still dev, null_mount or linprocfs mount points mounted inside that directory.<br />
<br />
d) when trying to umount null_mounts that are really long and you get an error like “No such file or directory”, it’s an OS bug where the dir is truncated. No known fix<br />
<br />
e) there are still files open somewhere inside the dir. Use <tt>fstat | grep <cid></tt> to find the process that has files open<br />
<br />
f) Starting with 6.x, the jail mechanism does a poor job of keeping track of processes running in a jail and if it thinks there are still procs running, it will refuse to umount the disk. If this is happening you should see a low number in the #REF column when you run jls. In this case you ''can'' safely <tt>umount –f</tt> the mount. <br />
<br />
Please note -if you forcibly unmount a (4.x) filesystem that has null_mounts<br />
still mounted in it, the system '''will crash''' within 10-15 mins.<br />
<br />
== Misc jail Items ==<br />
<br />
We are overselling hard drive space on jail2, jail8, jail9, a couple jails on jail17, jail4, jail12 and jail18.<br />
Even though the vn file shows 4G size, it doesn’t actually occupy that amount of space on the disk. So be careful not to fill up drives where we’re overselling – use oversellcheck to confirm you’re not oversold by more than 10G.<br />
There are other truncated jails, they are generally noted in a the file on the root system: /root/truncated<br />
<br />
The act of moving a truncated vn to another system un-does the truncating- the truncated vn is filled with 0’s and it occupies physical disk space for which it’s configured. So, you should use dumpremote to preserve the truncation.<br />
<br />
= FreeBSD VPS Management Tools =<br />
<br />
These files are located in /usr/local/jail/rc.d and /usr/local/jail/bin<br />
<br />
== jailmake ==<br />
<br />
Applies to 7.x+ <br />
On older systems syntax differs, run jailmake once to see.<br />
<br />
Note: this procedure differs on mx2 which is 7.x but still uses gvinum<br />
<br />
# run js to figure out which md’s are in use, which disk has enough space, IP to put it on<br />
# use col00xxx for both hostnames if they don’t give you a hostname<br />
# copy over dir, ip and password to pending customer screen<br />
<br />
<tt>Usage: jailmake IP[,IP] CID disk[1|2|3] md# hostname shorthost ipfw# email [size in GB]</tt><br />
<br />
Ex: <br />
<br />
Jail2# jailmake 69.55.234.66 col01334 3 97 vps.bsd.it vps 1334 fb@bsd.it<br />
<br />
== jailps ==<br />
jailps [hostname]<br />
DEPRECATED FOR jps: displays processes belonging to/running inside a jail. The command<br />
takes one (optional) argument – the hostname of the jail you wish to query. If you don’t <br />
supply an argument, all processes on the machine are listed and grouped by jail. <br />
<br />
== jps ==<br />
jps [hostname]<br />
displays processes belonging to/running inside a jail. The command<br />
takes one (optional) argument – the hostname or ID of the jail you wish to query. <br />
<br />
== jailkill ==<br />
jailkill <hostname><br />
stops all process running in a jail.<br />
<br />
You can also run:<br />
jailkill <JID><br />
<br />
=== problems ===<br />
Occasionally you will hit an issue where jail will not kill off:<br />
<br />
<pre>jail9# jailkill www.domain.com<br />
www.domain.com .. killed: none<br />
jail9#</pre><br />
Because no processes are running under that hostname. You cannot use jailps.pl either:<br />
<br />
<pre>jail9# jailps www.domain.com<br />
www.domain.com doesn’t exist on this server<br />
jail9#</pre><br />
<br />
The reasons for this are usually:<br />
* the jail is no longer running<br />
<br />
* the jail's hostname has changed<br />
In this case, <br />
<br />
>=6.x: run a <tt>jls|grep <jail's IP></tt> to find the correct hostname, then update the quad file, then kill the jail.<br />
<br />
<6.x: the first step is to cat their /etc/rc.conf file to see if you can tell what they set the new hostname to. This very often works. For example:<br />
<br />
cat /mnt/data2/198.78.65.136-col00261-DIR/etc/rc.conf<br />
<br />
But maybe they set the hostname with the hostname command, and the original hostname is still in /etc/rc.conf.<br />
<br />
The welcome email clearly states that they should tell us if they change their hostname, so there is no problem in just emailing them and asking them what they set the new hostname to.<br />
<br />
Once you know the new hostname OR if a customer simply emails to inform you that they have set the hostname to something different, you need to edit the quad and safe files that their system is in to input the new hostname.<br />
<br />
However, if push comes to shove and you cannot find out the hostname from them or from their system, then you need to start doing some detective work.<br />
<br />
The easiest thing to do is run jailps looking for a hostname similar to their original hostname. Or you could get into the /bin/sh shell by running:<br />
<br />
/bin/sh<br />
<br />
and then looking at every hostname of every process:<br />
<br />
for f in `ls /proc` ; do cat /proc/$f/status ; done<br />
<br />
and scanning for a hostname that is either similar to their original hostname, or that you don't see in any of the quad safe files.<br />
<br />
This is very brute force though, and it is possible that catting every file in /proc is dangerous - I don't recommend it. A better thing would be to identify any processes that you know belong to this system – perhaps the reason you are trying to find this system is because they are running something bad - and just catting the status from only that PID.<br />
<br />
Somewhere there’s a jail where there may be 2 systems named www. Look at /etc/rc.conf and make sure they’re both really www. If they are, jailkill www, jailps www to make sure not running. Then immediately restart the other one, as the fqdn (as found from a rev nslookup)<br />
<br />
* on >=6.x the hostname may not yet be hashed:<br />
<pre>jail9 /# jls<br />
JID Hostname Path IP Address(es)<br />
1 bitnet.dgate.org /mnt/data1/69.55.232.50-col02094-DIR 69.55.232.50<br />
2 ns3.hctc.net /mnt/data1/69.55.234.52-col01925-DIR 69.55.234.52<br />
3 bsd1 /mnt/data1/69.55.232.44-col00155-DIR 69.55.232.44<br />
4 let2.bbag.org /mnt/data1/69.55.230.92-col00202-DIR 69.55.230.92<br />
5 post.org /mnt/data2/69.55.232.51-col02095-DIR 69.55.232.51 ...<br />
6 ns2 /mnt/data1/69.55.232.47-col01506-DIR 69.55.232.47 ...<br />
7 arlen.server.net /mnt/data1/69.55.232.52-col01171-DIR 69.55.232.52<br />
8 deskfood.com /mnt/data1/69.55.232.71-col00419-DIR 69.55.232.71<br />
9 mirage.confluentforms.com /mnt/data1/69.55.232.54-col02105-DIR 69.55.232.54 ...<br />
10 beachmember.com /mnt/data1/69.55.232.59-col02107-DIR 69.55.232.59<br />
11 www.agottem.com /mnt/data1/69.55.232.60-col02109-DIR 69.55.232.60<br />
12 sdhobbit.myglance.org /mnt/data1/69.55.236.82-col01708-DIR 69.55.236.82<br />
13 ns1.jnielsen.net /mnt/data1/69.55.234.48-col00204-DIR 69.55.234.48 ...<br />
14 ymt.rollingegg.net /mnt/data2/69.55.236.71-col01678-DIR 69.55.236.71<br />
15 verse.unixlore.net /mnt/data1/69.55.232.58-col02131-DIR 69.55.232.58<br />
16 smcc-mail.org /mnt/data2/69.55.232.68-col02144-DIR 69.55.232.68<br />
17 kasoutsuki.w4jdh.net /mnt/data2/69.55.232.46-col02147-DIR 69.55.232.46<br />
18 dili.thium.net /mnt/data2/69.55.232.80-col01901-DIR 69.55.232.80<br />
20 www.tekmarsis.com /mnt/data2/69.55.232.66-col02155-DIR 69.55.232.66<br />
21 vps.yoxel.net /mnt/data2/69.55.236.67-col01673-DIR 69.55.236.67<br />
22 smitty.twitalertz.com /mnt/data2/69.55.232.84-col02153-DIR 69.55.232.84<br />
23 deliver4.klatha.com /mnt/data2/69.55.232.67-col02160-DIR 69.55.232.67<br />
24 nideffer.com /mnt/data2/69.55.232.65-col00412-DIR 69.55.232.65<br />
25 usa.hanyuan.com /mnt/data2/69.55.232.57-col02163-DIR 69.55.232.57<br />
26 daifuku.ppbh.com /mnt/data2/69.55.236.91-col01720-DIR 69.55.236.91<br />
27 collins.greencape.net /mnt/data2/69.55.232.83-col01294-DIR 69.55.232.83<br />
28 ragebox.com /mnt/data2/69.55.230.104-col01278-DIR 69.55.230.104<br />
29 outside.mt.net /mnt/data2/69.55.232.72-col02166-DIR 69.55.232.72<br />
30 vps.payneful.ca /mnt/data2/69.55.234.98-col01999-DIR 69.55.234.98<br />
31 higgins /mnt/data2/69.55.232.87-col02165-DIR 69.55.232.87 ...<br />
32 ozymandius /mnt/data2/69.55.228.96-col01233-DIR 69.55.228.96<br />
33 trusted.realtors.org /mnt/data2/69.55.238.72-col02170-DIR 69.55.238.72<br />
34 jc1.flanderous.com /mnt/data2/69.55.239.22-col01504-DIR 69.55.239.22<br />
36 guppylog.com /mnt/data2/69.55.238.73-col00036-DIR 69.55.238.73<br />
40 haliohost.com /mnt/data2/69.55.234.41-col01916-DIR 69.55.234.41 ...<br />
41 satyr.jorge.cc /mnt/data1/69.55.232.70-col01963-DIR 69.55.232.70<br />
jail9 /# jailkill satyr.jorge.cc<br />
ERROR: jail_: jail "satyr,jorge,cc" not found<br />
</pre><br />
<br />
Note how it's saying <tt>satyr,jorge,cc</tt> is not found, and not <tt>satyr.jorge.cc</tt>. <br />
<br />
The jail subsystem tracks things using comma-delimited hostnames. That is created every few hours:<br />
<br />
jail9 /# crontab -l<br />
0 0,6,12,18 * * * /usr/local/jail/bin/sync_jail_names<br />
<br />
So if we run this manually:<br />
jail9 /# /usr/local/jail/bin/sync_jail_names<br />
<br />
Then kill the jail:<br />
jail9 /# jailkill satyr.jorge.cc<br />
successfully killed: satyr,jorge,cc</pre><br />
<br />
It worked.<br />
<br />
<br />
If you ever see this when trying to kill a jail:<br />
<br />
<pre># jailkill e-scribe.com<br />
killing JID: 6 hostname: e-scribe.com<br />
3 procs running<br />
3 procs running<br />
3 procs running<br />
3 procs running<br />
...</pre><br />
<br />
<tt>[[#jailkill|jailkill]]</tt> probably got lost trying to kill off the jail. Just ctrl-c the jailkill process, then run a jailps on the hostname, and <tt>kill -9</tt> any process which is still running. Keep running jailps and <tt>kill -9</tt> till all processes are gone.<br />
<br />
== jailpsall ==<br />
jailpsall<br />
will run a jailps on all jails configured in the quad files (this is different from<br />
jailps with no arguments as it won’t help you find a “hidden” system)<br />
<br />
== jailpsw ==<br />
jailpsw<br />
will run a jailps with an extra -w to provide wider output<br />
<br />
== jt (>=7.x) ==<br />
jt<br />
displays the top 20 processes on the server (the top 20 processes from top) and <br />
which jail owns them. This is very helpful for determining who is doing what when<br />
the server is very busy.<br />
<br />
== jtop (>=7.x) ==<br />
jtop<br />
a wrapper for top displaying processes on the server and which jail owns them. Constantly updates, like top. <br />
<br />
== jtop (<7.x) ==<br />
jtop<br />
displays the top 20 processes on the server (the top 20 processes from top) and <br />
which jail owns them. This is very helpful for determining who is doing what when<br />
the server is very busy.<br />
<br />
== stopjail ==<br />
stopjail <hostname> [1]<br />
this will jailkill, umount and vnconfig –u a jail. If passed an optional 2nd<br />
argument, it will not exit before umounting and un-vnconfig’ing in the event<br />
jailkill returns no processes killed. This is useful if you just want to umount<br />
and vnconfig –u a jail you’ve already killed. It is intelligent in that it won’t <br />
try to umount or vnconfig –u if it’s not necessary.<br />
<br />
== startjail ==<br />
startjail <hostname><br />
this will start vnconfig, mount (including linprocfs and null-mounts), and start a jail.<br />
Essentially, it reads the jail’s relevant block from the right quad file and executes it.<br />
It is intelligent in that it won’t try to mount or vnconfig if it’s not necessary.<br />
<br />
== jpid ==<br />
jpid <pid><br />
displays information about a process – including which jail owns it.<br />
It’s the equivalent of running cat /proc/<pid>/status<br />
<br />
== canceljail ==<br />
canceljail <hostname> [1]<br />
this will stop a jail (the equivalent of stopjail), check for backups (offer to remove them <br />
from the backup server and the backup.config), rename the vnfile, remove the dir, and <br />
edit quad/safe. If passed an optional 2nd argument, it will not exit upon failing to kill<br />
and processes owned by the jail. This is useful if you just want to cancel a jail which <br />
is already stopped.<br />
<br />
== jls ==<br />
jls [-v]<br />
Lists all jails running:<br />
<pre>JID #REF IP Address Hostname Path<br />
101 135 69.55.224.148 mail.pc9.org /mnt/data2/69.55.224.148-col01034-DIR</pre><br />
<br />
#REF is the number of references or procs(?) running<br />
<br />
Running with -v will give you all IPs assigned to each jail (7.2 up)<br />
<pre>JID #REF Hostname Path IP Address(es)<br />
101 139 mail.pc9.org /mnt/data2/69.55.224.148-col01034-DIR 69.55.224.14869.55.234.85</pre><br />
<br />
== startalljails ==<br />
startalljails<br />
7.2+ only. This will parse through quad1 and start all jails. It utilizes lockfiles so it won’t try to start a jail more than once- therefore multiple instances can be running in parallel without fear of starting a jail twice. If a jail startup gets stuck, you can ^C without fear of killing the script. IMPORTANT- before running startalljails you should make sure you ran preboot once as it will clear out all the lockfiles and enable startalljails to work properly.<br />
<br />
== aaccheck.sh ==<br />
aaccheck.sh<br />
displayes the output of container list and task list from aaccli<br />
<br />
== backup ==<br />
backup<br />
backup script called nightly to update jail scripts and do customer backups<br />
<br />
== buildsafe ==<br />
buildsafe<br />
creates safe files based on quads (automatically removing the fsck’s). This will destructively overwrite safe files<br />
<br />
== checkload.pl ==<br />
checkload.pl<br />
this was intended to be setup as a cronjob to watch processes on a jail when the load <br />
rises above a certain level. Not currently in use.<br />
<br />
== checkprio.pl ==<br />
checkprio.pl<br />
will look for any process (other than the current shell’s csh, sh, sshd procs) with a non-normal priority and normalize it<br />
<br />
== diskusagemon == <br />
diskusagemon <mount point> <1k blocks><br />
watches a mount point’s disk use, when it reaches the level specified in the 2nd argument,<br />
it exits. This is useful when doing a restore and you want to be paged as it’s nearing completion.<br />
Best used as: <tt>diskusagemon /asd/asd 1234; pagexxx</tt><br />
<br />
== dumprestore ==<br />
dumprestore <dumpfile><br />
this is a perl expect script which automatically enters ‘1’ and ‘y’. It seems to cause restore to fail<br />
to set owner permissions on large restores.<br />
<br />
== g ==<br />
g <search><br />
greps the quad/safe files for the given search parameter<br />
<br />
== gather.pl ==<br />
gather.pl<br />
gathers up data about jails configured and writes to a file. Used for audits against the db<br />
<br />
== gb ==<br />
gb <search><br />
greps backup.config for the given search parameter<br />
<br />
== gbg ==<br />
gbg <search><br />
greps backup.config for the given search parameter and presents just the directories (for clean pasting)<br />
<br />
== ipfwbackup ==<br />
ipfwbackup<br />
writes ipfw traffic count data to a logfile<br />
<br />
== ipfwreset ==<br />
ipfwreset<br />
writes ipfw traffic count data to a logfile and resets counters to 0<br />
<br />
== js ==<br />
js<br />
output varies by OS version, but generally provides information about the base jail:<br />
- which vn’s are in use<br />
- disk usage<br />
- info about the contents of quads<br />
- the # of inodes represented by the jails contained in the group (133.2 in the example below), and how many jails per data mount, as well as subtotals<br />
- ips bound to the base machine but not in use by a jail<br />
- free gvinum volumes, or unused vn’s or used md’s<br />
<br />
<pre>/usr/local/jail/rc.d/quad1:<br />
/mnt/data1 133.2 (1)<br />
/mnt/data2 1040.5 (7)<br />
total 1173.7 (8)<br />
/usr/local/jail/rc.d/quad2:<br />
/mnt/data1 983.4 (6)<br />
total 983.4 (6)<br />
/usr/local/jail/rc.d/quad3:<br />
/mnt/data1 693.4 (4)<br />
/mnt/data2 371.6 (3)<br />
total 1065 (7)<br />
/usr/local/jail/rc.d/quad4:<br />
/mnt/data1 466.6 (3)<br />
/mnt/data2 882.2 (5)<br />
total 1348.8 (8)<br />
/mnt/data1: 2276.6 (14)<br />
/mnt/data2: 2294.3 (15)<br />
<br />
Available IPs:<br />
69.55.230.11 69.55.230.13 69.55.228.200<br />
<br />
Available volumes:<br />
v78 /mnt/data2 2G<br />
v79 /mnt/data2 2G<br />
v80 /mnt/data2 2G</pre> <br />
<br />
== load.pl ==<br />
load.pl<br />
feeds info to load mrtg - executed by inetd.<br />
<br />
== makevirginjail ==<br />
makevirginjail<br />
Only on some systems, makes an empty jail (doesn't do restore step)<br />
<br />
== mb == <br />
mb <mount|umount><br />
(nfs) mounts and umounts dirs to backup2. Shortcuts are mbm and mbu to mount and unmount. <br />
<br />
== notify.sh ==<br />
notify.sh<br />
emails reboot@johncompanies.com – intended to be called at boot time to alert us to a machine which panics and reboots and isn’t caught by bb or castle.<br />
<br />
== orphanedbackupwatch ==<br />
orphanedbackupwatch<br />
looks for directories on backup2 which aren’t configured in backup.config and offers to delete them<br />
<br />
== postboot ==<br />
postboot<br />
to be run after a machine reboot and quad/safe’s are done executing. It will:<br />
* do chmod 666 on each jail’s /dev/null<br />
* add ipfw counts<br />
* run jailpsall (so you can see if a configured jail isn’t running)<br />
<br />
== preboot ==<br />
preboot<br />
to be run before running quad/safe – checks for misconfigurations: <br />
* a jail configured in a quad but not a safe<br />
* a jail is listed more than once in a quad<br />
* the ip assigned to a jail isn’t configured on the machine<br />
* alias numbering skips in the rc.conf (resulting in the above)<br />
* orphaned vnfile's that aren't mentioned in a quad/safe<br />
* ip mismatches between dir/vnfile name and the jail’s ip<br />
* dir/vnfiles's in quad/safe that don’t exist <br />
<br />
== quadanalyze.pl ==<br />
quadanalyze.pl<br />
called by js, produces the info (seen above with js explanation) about the contents of quad (inode count, # of jails, etc.)<br />
<br />
== rsync.backup ==<br />
rsync.backup<br />
does customer backups (relies on backup.config)<br />
<br />
== taskdone ==<br />
taskdone<br />
when called will email support@johncompanies.com with the hostname of the machine from which it was executed as the subject<br />
<br />
== topten ==<br />
topten<br />
summarizes the top 10 traffic users (called by ipfwreset)<br />
<br />
== trafficgather.pl ==<br />
trafficgather.pl [yy-mm]<br />
sends a traffic usage summary by jail to support@johncomapnies.com and payments@johncompanies.com. Optional arguments are year and month (must be in the past). If not passed, assumes last month. Relies on traffic logs created by ipfwreset and ipfwbackup<br />
<br />
== trafficwatch.pl ==<br />
trafficwatch.pl<br />
checks traffic usage and emails support@johncomapnies.com when a jail reaches the warning level (35G) and the limit (40G). We really aren’t using this anymore now that we have netflow.<br />
<br />
== trafstats ==<br />
trafstats<br />
writes ipfw traffic usage info by jail to a file called jc_traffic_dump in each jail’s / dir<br />
<br />
== truncate_jailmake ==<br />
truncate_jailmake<br />
a version of jailmake which creates truncated vnfiles.<br />
<br />
== vb ==<br />
vb<br />
the equivalent of: <tt>vi /usr/local/jail/bin/backup.config</tt><br />
<br />
== vs (freebsd) ==<br />
vs<n><br />
the equivalent of: <tt>vi /usr/local/jail/rc.d/safe<n></tt><br />
<br />
vq<n><br />
the equivalent of: vi /usr/local/jail/rc.d/quad<n><br />
<br />
== dumpremote ==<br />
dumpremote <user@machine> </remote/location/file-dump> <vnX><br />
ex: dumpremote user@10.1.4.117 /mnt/data3/remote.echoditto.com-dump 7<br />
this will dump a vn filesystem to a remote machine and location<br />
<br />
== oversellcheck ==<br />
oversellcheck<br />
displays how much a disk is oversold or undersold taking into account truncated vn files. Only for use on 4.x systems<br />
<br />
== mvbackups (freebsd) ==<br />
mvbackups <dir> (1.1.1.1-col00001-DIR) <target_machine> (jail1) <target_dir> (data1)<br />
moves backups from one location to another on the backup server, and provides you with option to remove entries from current backup.config, and simple paste command to add the config to backup.config on the target server<br />
<br />
== jailnice ==<br />
jailnice <hostname><br />
applies <tt>renice 19 [PID]</tt> and <tt>rtprio 31 –[PID]</tt> to each process in the given jail<br />
<br />
== dumpremoterestore ==<br />
dumpremoterestore <device> <ip of target machine> <dir on target machine><br />
ex: dumpremoterestore /dev/vn51 10.1.4.118 /mnt/data2/69.55.239.45-col00688-DIR<br />
dumps a device and restores it to a directory on a remote machine. Requires that you enable root ssh on the <br />
remote machine.<br />
<br />
== psj ==<br />
psj<br />
shows just the procs running on the base system – a ps auxw but without jail’d procs present<br />
<br />
== perc5iraidchk ==<br />
perc5iraidchk<br />
checks for degraded arrays on Dell 2950 systems with Perc5/6 controllers<br />
<br />
== perc4eraidchk ==<br />
perc4eraidchk<br />
checks for degraded arrays on Dell 2850 systems with Perc4e/Di controllers<br />
<br />
= Virtuozzo VPS =<br />
<br />
Note: We use VEID (Virtual Environment ID) and CTID (Container ID) interchangably. Similarly, VE and CT. They mean the same thing.<br />
VZPP = VirtuoZzo Power Panel (the control panel for each CT)<br />
<br />
All linux systems exist in /vz, /vz1 or /vz2 - since each linux machine holds roughly 60-90 customers, there will be roughly 30-45 in each partition.<br />
<br />
The actual filesystem of the system in question is in:<br />
<br />
/vz(1-2)/private/(VEID)<br />
<br />
Where VEID is the identifier for that system - an all-numeric string larger than 100.<br />
<br />
The actual mounted and running systems are in the corresponding:<br />
<br />
/vz(1-2)/root/(VEID)<br />
<br />
But we rarely interact with any system from this mount point.<br />
<br />
You should never need to touch the root portion of their system – however you can traverse their filesystem by going to <tt>/vz(1-2)/private/(VEID)/root</tt> (<tt>/vz(1-2)/private/(VEID)/fs/root</tt> on 4.x systems) the root of their filesystem is in that directory, and their entire system is underneath that.<br />
<br />
Every VE has a startup script in <tt>/etc/sysconfig/vz-scripts</tt> (which is symlinked as <tt>/vzconf</tt> on all systems) - the VE startup script is simply named <tt>(VEID).conf</tt> - it contains all the system parameters for that VE:<br />
<br />
<pre># Configuration file generated by vzsplit for 60 VE<br />
# on HN with total amount of physical mem 2011 Mb<br />
<br />
VERSION="2"<br />
CLASSID="2"<br />
<br />
ONBOOT="yes"<br />
<br />
KMEMSIZE="8100000:8200000"<br />
LOCKEDPAGES="322:322"<br />
PRIVVMPAGES="610000:615000"<br />
SHMPAGES="33000:34500"<br />
NUMPROC="410:415"<br />
PHYSPAGES="0:2147483647"<br />
VMGUARPAGES="13019:2147483647"<br />
OOMGUARPAGES="13019:2147483647"<br />
NUMTCPSOCK="1210:1215"<br />
NUMFLOCK="107:117"<br />
NUMPTY="19:19"<br />
NUMSIGINFO="274:274"<br />
TCPSNDBUF="1800000:1900000"<br />
TCPRCVBUF="1800000:1900000"<br />
OTHERSOCKBUF="900000:950000"<br />
DGRAMRCVBUF="200000:200000"<br />
NUMOTHERSOCK="650:660"<br />
DCACHE="786432:818029"<br />
NUMFILE="7500:7600"<br />
AVNUMPROC="51:51"<br />
IPTENTRIES="155:155"<br />
DISKSPACE="4194304:4613734"<br />
DISKINODES="400000:420000"<br />
CPUUNITS="1412"<br />
QUOTAUGIDLIMIT="2000"<br />
VE_ROOT="/vz1/root/636"<br />
VE_PRIVATE="/vz1/private/636"<br />
NAMESERVER="69.55.225.225 69.55.230.3"<br />
OSTEMPLATE="vzredhat-7.3/20030305"<br />
VE_TYPE="regular"<br />
IP_ADDRESS="69.55.225.229"<br />
HOSTNAME="textengine.net"</pre><br />
<br />
<br />
As you can see, the hostname is set here, the disk space is set here, the number of inodes, the number of files that can be open, the number of tcp sockets, etc. - all are set here.<br />
<br />
In fact, everything that can be set on this customer system is set in this conf file.<br />
<br />
<br />
All interaction with the customer system is done with the VEID. You start the system by running:<br />
<br />
vzctl start 999<br />
<br />
You stop it by running:<br />
<br />
vzctl stop 999<br />
<br />
You execute commands in it by running:<br />
<br />
vzctl exec 999 df -k<br />
<br />
You enter into it, via a root-shell backdoor with:<br />
<br />
vzctl enter 999<br />
<br />
and you set parameters for the system, while it is still running, with:<br />
<br />
vzctl set 999 --diskspace 6100000:6200000 --save<br />
<br />
<tt>vzctl</tt> is the most commonly used command - we have aliased <tt>v</tt> to <tt>vzctl</tt> since we use it so often. We’ll continue to use <tt>vzctl</tt> in our examples, but feel free to use just <tt>v</tt>.<br />
<br />
Let's say the user wants more diskspace. You can cat their conf file and see:<br />
<br />
DISKSPACE="4194304:4613734"<br />
<br />
So right now they have 4gigs of space. You can then change it to 6 with:<br />
<br />
vzctl set 999 --diskspace 6100000:6200000 --save<br />
<br />
IMPORTANT: all issuances of the vzctl set command need to end with <tt>–save</tt> - if they don't, the setting will be set, but it will not be saved to the conf file, and they will not have those settings next time they boot.<br />
<br />
All of the tunables in the conf file can be set with the vzctl set command. Note that in the conf file, and on the vzctl set command line, we always issue two numbers seperated by a colon - that is because we are setting the hard and soft limits. Always set the hard limit slightly above the soft limit, as you see it is in the conf file for all those settings.<br />
<br />
There are also things you can set with `<tt>vzctl set</tt>` that are not in the conf file as settings, per se. For instance, you can add IPs:<br />
<br />
vzctl set 999 --ipadd 10.10.10.10 --save<br />
<br />
or multiple IPs:<br />
<br />
vzctl set 999 --ipadd 10.10.10.10 --ipadd 10.10.20.30 --save<br />
<br />
or change the hostname:<br />
<br />
vzctl set 999 --hostname www.example.com --save<br />
<br />
You can even set the nameservers:<br />
<br />
vzctl set 999 --nameserver 198.78.66.4 --nameserver 198.78.70.180 --save<br />
<br />
Although you probably will never do that.<br />
<br />
You can disable a VPS from being started (by VZPP or reboot) (4.x):<br />
<br />
vzctl set 999 --disabled yes --save <br />
<br />
You can disable a VPS from being started (by VZPP or reboot) (<=3.x):<br />
<br />
vzctl set 999 --onboot=no --save <br />
<br />
You can disable a VPS from using his control panel:<br />
<br />
vzctl set 999 --offline_management=no --save <br />
<br />
You can suspend a VPS, so it can be resumed in the same state it was in when it was stopped (4.x):<br />
<br />
vzctl suspend 999<br />
<br />
and to resume it:<br />
<br />
vzctl resume 999<br />
<br />
to see who owns process:<br />
vzpid <PID><br />
<br />
to mount up an unmounted ve:<br />
vzctl mount 827<br />
<br />
<br />
To see network stats for CT's:<br />
<pre># vznetstat<br />
VEID Net.Class Output(bytes) Input(bytes)<br />
-----------------------------------------------<br />
24218 1 484M 39M<br />
24245 1 463M 143M<br />
2451 1 2224M 265M<br />
2454 1 2616M 385M<br />
4149 1 125M 68M<br />
418 1 1560M 34M<br />
472 1 1219M 315M<br />
726 1 628M 317M<br />
763 1 223M 82M<br />
771 1 4234M 437M<br />
-----------------------------------------------<br />
Total: 13780M 2090M</pre><br />
<br />
<br />
One thing that sometimes comes up on older systems that we created with smaller defaults is that the system would run out of inodes. The user will email and say they cannot create any more files or grow any files larger, but they will also say that they are not out of diskspace ... they are running:<br />
<br />
df -k<br />
<br />
and seeing how much space is free - and they are not out of space. They are most likely out of inodes - which they would see by running:<br />
<br />
df -i<br />
<br />
So, the first thing you should do is enter their system with:<br />
<br />
vzctl enter 999<br />
<br />
and run: <tt>df -i</tt><br />
<br />
to confirm your theory. Then exit their system. Then simply cat their conf file and see what their inodes are set to (probably 200000:200000, since that was the old default on the older systems) and run:<br />
<br />
vzctl set 999 --diskinodes 400000:400000 --save<br />
<br />
If they are not out of inodes, then a good possibility is that they have maxed out their numfile configuration variable, which controls how many files they can have in their system. The current default is 7500 (which nobody has ever hit), but the old default was as low as 2000, so you would run something like:<br />
<br />
vzctl set 999 --numfile 7500:7500 --save<br />
<br />
----<br />
<br />
You cannot start or stop a VE if your pwd is its private (/vz/private/999) or root (/vz/root/999) directories, or anywhere below them.<br />
<br />
<br />
== Recovering from a crash (linux) ==<br />
<br />
=== Diagnose whether you have a crash ===<br />
The most important thing is to get the machine and all ve’s back up as soon as possible. Note the time, you’ll need to create a crash log entry (Mgmt. -> Reference -> CrashLog). The first thing to do is head over to the [[Screen#Screen_Organization|serial console screen]] and see if there’s any kernel error messages output. Try to copy any messages (or just a sample of repeating messages) you see into the notes section of the crash log – these will also likely need to be sent to virtuozzo for interpretation. If the messages are spewing too fast, hit ^O + H to start a screen log dump which you can ob1182.pts-38.bb serve after the machine is rebooted. Additionally, if the machine is responsive, you can get a trace to send to virtuozzo by hooking up a kvm and entering these 3 sequences:<br />
<pre>alt+print screen+m<br />
alt+print screen+p<br />
alt+print screen+t</pre><br />
<br />
If there are no messages, the machine may just be really busy- wait a bit (5-10min) to see if it comes back. If it's still pinging, odds are its very busy. If it doesn't come back, or the messages indicate a fatal error, you will need to proceed with a power cycle (ctrl+alt+del will not work).<br />
<br />
=== Power cycle the server ===<br />
If this machine is not a Dell 2950 with a [[DRAC/RMM#DRAC|DRAC card]] (i.e. if you can’t ssh into the DRAC card and issue racadm serveraction hardreset, then you will need someone at the data center to power the macine off, wait 30 sec, then turn it back on. Make sure to re-attach via console (<tt>tip virtxx</tt>) immediately after power down. <br />
<br />
=== (Re)attach to the console ===<br />
Stay on the console the entire time during boot. As the BIOS posts- look out for the RAID card output- does everything look healthy? The output may be scrambled, look for "DEGRADED" or "FAILED". Once the OS starts booting you will be disconnected (dropped back to the shell on the console server) a couple times during the boot up. The reason you want to quickly re-attach is two-fold: 1. If you don’t reattach quickly then you won’t get any console output, 2. you want to be attached before the server ''potentially'' starts (an extensive) fsck. If you attach after the fsck begins, you’ll have seen no indication it started an fsck and the server will appear frozen during startup- no output, no response. <br />
<br />
=== Start containers/VE's/VPSs ===<br />
When the machine begins to start VE’s, it’s safe to leave the console and login via ssh. All virts should be set to auto start all the VEs after a crash. Further, most (newer) virts are set to “fastboot” it’s VE’s (to find out, do:<br />
grep -i fast /etc/sysconfig/vz <br />
and look for <tt>VZFASTBOOT=yes</tt>). If this was set prior to the machine’s crash (setting it after the machine boots will not have any effect until the vz service is restarted) it will start each ve as fast as possible, in serial, then go thru each VE (serially), shutting it down running a vzquota (disk usage) check, then bringing it back up. The benefit is that all VE’s are brought up quickly (within 15min or so depending on the #), the downside is a customer watching closely will notice 2 outages – 1st the machine crash, 2nd their quota check (which will be a much shorter downtime- on the order of a few minutes). <br />
<br />
Where “fastboot” is not set to yes (i.e on quar1), vz will start them consecutively, checking the quotas one at a time, and the 60th VE may not start until an hour or two later - this is not acceptable.<br />
<br />
The good news is, if you run vzctl start for a VE that is already started, you will simply get an error: <tt>VE is already started</tt>. Further, if you attempt to vzctl start a VE that is in the process of being started, you will simply get an error: unable to lock VE. So, there is no danger in simply running scripts to start smaller sets of VEs. If the system is not autostarting, then there is no issue, and even if it does, when it conflicts, one process (yours or the autostart) will lose, and just move on to the next one.<br />
<br />
A script has been written to assist with ve starts: [[#startvirt.pl|startvirt.pl]] which will start 6 ve’s at once until there are no more left. If startvirt.pl is used on a system where “fastboot” was on, it will circumvent the fastboot for ve’s started by startvirt.pl – they will go through the complete quota check before starting- therefore this is not advisable when a system has crashed. When a system is booted cleanly, and there's no need for vzquota checks, then startvirt.pl is safe and advisable to run.<br />
<br />
=== Make sure all containers are running ===<br />
You can quickly get a feel for how many ve’s are started by running:<br />
<br />
<pre>[root@virt4 log]# vs<br />
VEID 16066 exist mounted running<br />
VEID 16067 exist mounted running<br />
VEID 4102 exist mounted running<br />
VEID 4112 exist mounted running<br />
VEID 4116 exist mounted running<br />
VEID 4122 exist mounted running<br />
VEID 4123 exist mounted running<br />
VEID 4124 exist mounted running<br />
VEID 4132 exist mounted running<br />
VEID 4148 exist mounted running<br />
VEID 4151 exist mounted running<br />
VEID 4155 exist mounted running<br />
VEID 42 exist mounted running<br />
VEID 432 exist mounted running<br />
VEID 434 exist mounted running<br />
VEID 442 exist mounted running<br />
VEID 450 exist mounted running<br />
VEID 452 exist mounted running<br />
VEID 453 exist mounted running<br />
VEID 454 exist mounted running<br />
VEID 462 exist mounted running<br />
VEID 463 exist mounted running<br />
VEID 464 exist mounted running<br />
VEID 465 exist mounted running<br />
VEID 477 exist mounted running<br />
VEID 484 exist mounted running<br />
VEID 486 exist mounted running<br />
VEID 490 exist mounted running</pre><br />
<br />
So to see how many ve’s have started:<br />
<pre>[root@virt11 root]# vs | grep running | wc -l<br />
39</pre><br />
<br />
And to see how many haven’t:<br />
<pre>[root@virt11 root]# vs | grep down | wc -l<br />
0</pre><br />
<br />
And how many we should have running:<br />
<pre>[root@virt11 root]# vs | wc -l<br />
39</pre><br />
<br />
Another tool you can use to see which ve’s have started, among other things is [[#vzstat|vzstat]]. It will give you CPU, memory, and other stats on each ve and the overall system. It’s a good thing to watch as ve’s are starting (note the VENum parameter, it will tell you how many have started):<br />
<br />
<pre>4:37pm, up 3 days, 5:31, 1 user, load average: 1.57, 1.68, 1.79<br />
VENum 40, procs 1705: running 2, sleeping 1694, unint 0, zombie 9, stopped 0<br />
CPU [ OK ]: VEs 57%, VE0 0%, user 8%, sys 7%, idle 85%, lat(ms) 412/2<br />
Mem [ OK ]: total 6057MB, free 9MB/54MB (low/high), lat(ms) 0/0<br />
Swap [ OK ]: tot 6142MB, free 4953MB, in 0.000MB/s, out 0.000MB/s<br />
Net [ OK ]: tot: in 0.043MB/s 402pkt/s, out 0.382MB/s 4116pkt/s<br />
Disks [ OK ]: in 0.002MB/s, out 0.000MB/s<br />
<br />
VEID ST %VM %KM PROC CPU SOCK FCNT MLAT IP<br />
1 OK 1.0/17 0.0/0.4 0/32/256 0.0/0.5 39/1256 0 9 69.55.227.152<br />
21 OK 1.3/39 0.1/0.2 0/46/410 0.2/2.8 23/1860 0 6 69.55.239.60<br />
133 OK 3.1/39 0.1/0.3 1/34/410 6.3/2.8 98/1860 0 0 69.55.227.147<br />
263 OK 2.3/39 0.1/0.2 0/56/410 0.3/2.8 34/1860 0 1 69.55.237.74<br />
456 OK 17/39 0.1/0.2 0/100/410 0.1/2.8 48/1860 0 11 69.55.236.65<br />
476 OK 0.6/39 0.0/0.2 0/33/410 0.1/2.8 96/1860 0 10 69.55.227.151<br />
524 OK 1.8/39 0.1/0.2 0/33/410 0.0/2.8 28/1860 0 0 69.55.227.153<br />
594 OK 3.1/39 0.1/0.2 0/45/410 0.0/2.8 87/1860 0 1 69.55.239.40<br />
670 OK 7.7/39 0.2/0.3 0/98/410 0.0/2.8 64/1860 0 216 69.55.225.136<br />
691 OK 2.0/39 0.1/0.2 0/31/410 0.0/0.7 25/1860 0 1 69.55.234.96<br />
744 OK 0.1/17 0.0/0.5 0/10/410 0.0/0.7 7/1860 0 6 69.55.224.253<br />
755 OK 1.1/39 0.0/0.2 0/27/410 0.0/2.8 33/1860 0 0 192.168.1.4<br />
835 OK 1.1/39 0.0/0.2 0/19/410 0.0/2.8 5/1860 0 0 69.55.227.134<br />
856 OK 0.3/39 0.0/0.2 0/13/410 0.0/2.8 16/1860 0 0 69.55.227.137<br />
936 OK 3.2/52 0.2/0.4 0/75/410 0.2/0.7 69/1910 0 8 69.55.224.181<br />
1020 OK 3.9/39 0.1/0.2 0/60/410 0.1/0.7 55/1860 0 8 69.55.227.52<br />
1027 OK 0.3/39 0.0/0.2 0/14/410 0.0/2.8 17/1860 0 0 69.55.227.83<br />
1029 OK 1.9/39 0.1/0.2 0/48/410 0.2/2.8 25/1860 0 5 69.55.227.85<br />
1032 OK 12/39 0.1/0.4 0/80/410 0.0/2.8 41/1860 0 8 69.55.227.90</pre><br />
<br />
When you are all done, you will want to make sure that all the VEs really did get started, run vs one more time.<br />
<br />
Note the time all ve’s are back up and enter that into and save the crash log entry.<br />
<br />
Occasionally, a ve will not start automatically. The most common reason for a ve not to come up normally is the ve was at it’s disk limit before the crash, and will not start since they’re over the limit. To overcome this, set the disk space to current usage level (the system will give this to you when it fails to start), start the ve, then re-set the disk space back to the prior level. Lastly, contact the customer to let them know they’re out of disk (or allocate more disk if they're entitled to more).<br />
<br />
== Hitting performance barriers and fixing them ==<br />
<br />
There are multiple modes virtuozzo offers to allocate resources to a ve. We utilize 2: SLM and UBC parameters<br />
On our 4.x systems, we use all SLM – it’s simpler to manage and understand. There are a few systems on virt19/18 that may also use SLM. Everything else uses UBC. <br />
You can tell a SLM ve by:<br />
<br />
SLMMODE="all"<br />
<br />
in their conf file. <br />
<br />
TODO: detail SLM modes and parameters.<br />
<br />
If someone is in SLM mode and they hit memory resource limits, they simply need to upgrade to more memory.<br />
<br />
The following applies to everyone else (UBC).<br />
<br />
Customers will often email and say that they are getting out of memory errors - a common one is "cannot fork" ... basically, anytime you see something odd like this, it means they are hitting one of their limits that is in place in their conf file.<br />
<br />
The conf file, however, simply shows their limits - how do we know what they are currently at ?<br />
<br />
The answer is a file called v - this file contains the current status (and peaks) of their performance settings, and also counts how many times they have hit the barrier. The output of the file looks like this:<br />
<br />
<pre>764: kmemsize 384113 898185 8100000 8200000 0<br />
lockedpages 0 0 322 322 0<br />
privvmpages 1292 7108 610000 615000 0<br />
shmpages 270 528 33000 34500 0<br />
dummy 0 0 0 0 0<br />
numproc 8 23 410 415 0<br />
physpages 48 5624 0 2147483647 0<br />
vmguarpages 0 0 13019 2147483647 0<br />
oomguarpages 641 6389 13019 2147483647 0<br />
numtcpsock 3 21 1210 1215 0<br />
numflock 1 3 107 117 0<br />
numpty 0 2 19 19 0<br />
numsiginfo 0 4 274 274 0<br />
tcpsndbuf 0 80928 1800000 1900000 0 <br />
tcprcvbuf 0 108976 1800000 1900000 0<br />
othersockbuf 2224 37568 900000 950000 0<br />
dgramrcvbuf 0 4272 200000 200000 0<br />
numothersock 3 9 650 660 0<br />
dcachesize 53922 100320 786432 818029 0<br />
numfile 161 382 7500 7600 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
numiptent 4 4 155 155 0</pre><br />
<br />
The first column is the name of the counter in question - the same names we saw in the systems conf file. The second column is the _current_ value of that counter, the third column is the max that that counter has ever risen to, the fourth column is the soft limit, and the fifth column is the hard limit (which is the same as the numbers in that systems conf file).<br />
<br />
The sixth number is the failcount - how many times the current usage has risen to hit the barrier. It will increase as soon as the current usage hits the soft limit.<br />
<br />
The problem with /proc/user_beancounters is that it actually contains that set of data for every running VE - so you can't just cat /proc/user_beancounters - it is too long and you get info for every other running system.<br />
<br />
You can vzctl enter the system and run:<br />
<br />
vzctl enter 9999<br />
cat /proc/user_beancounters<br />
<br />
inside their system, and you will just see the stats for their particular system, but entering their system every time you want to see it is combersome.<br />
<br />
So, I wrote a simple script called "vzs" which simply greps for the VEID, and spits out the next 20 or so lines (however many lines there are in the output, I forget) after it. For instance:<br />
<br />
<pre># vzs 765:<br />
765: kmemsize 2007936 2562780 8100000 8200000 0<br />
lockedpages 0 8 322 322 0<br />
privvmpages 26925 71126 610000 615000 0<br />
shmpages 16654 16750 33000 34500 0<br />
dummy 0 0 0 0 0<br />
numproc 41 57 410 415 0<br />
physpages 1794 49160 0 2147483647 0<br />
vmguarpages 0 0 13019 2147483647 0<br />
oomguarpages 4780 51270 13019 2147483647 0<br />
numtcpsock 23 37 1210 1215 0<br />
numflock 17 39 107 117 0<br />
numpty 1 3 19 19 0<br />
numsiginfo 0 6 274 274 0<br />
tcpsndbuf 22240 333600 1800000 1900000 0<br />
tcprcvbuf 0 222656 1800000 1900000 0<br />
othersockbuf 104528 414944 900000 950000 0<br />
dgramrcvbuf 0 4448 200000 200000 0<br />
numothersock 73 105 650 660 0<br />
dcachesize 247038 309111 786432 818029 0<br />
numfile 904 1231 7500 7600 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
numiptent 4 4 155 155 0</pre><br />
<br />
That showed us just the portion of /proc/user_beancounters for system 765.<br />
<br />
When you run the vzs command, always add a : after the VEID.<br />
<br />
So, if a customer complains about some out of memory errors, or no more files, or no more ptys, or just has an unspecific complain about processes dying, etc., the very first thing you need to do is check their beancounters with vzs. Usually you will spot an item that has a high failcount and needs to be upped.<br />
<br />
At that point you could simply up the counter with `vzctl set`. Generally pick a number 10-20% higher than the old one, and make the hard limit slightly larger than the the soft limit. However our systems now come in several levels and those levels have more/different memory allocations. If someone is complaining about something other than a memory limit (pty, numiptent, numflock), it’s generally safe to increase it, at least to the same level as what’s in the /vzconf/4unlimited file on the newest virt. If someone is hitting a memory limit, first make sure they are given what they deserve:<br />
<br />
(refer to mgmt -> payments -> packages)<br />
<br />
To set those levels, you use the [[#setmem|setmem]] command. <br />
<br />
The alternate (DEPRECATED) method would be to use one of 3 commands:<br />
256 <veid><br />
300 <veid><br />
384 <veid><br />
512 <veid><br />
<br />
If the levels were not right (you’d run vzs <veid> before and after to see the effect) tell the customer they’ve been adjusted and be done with it. If the levels were right, tell the customer they must upgrade to a higher package, tell them how to see level (control panel) and that they can reboot their system to escape this lockup contidion.<br />
<br />
Customers can also complain that their site is totally unreachable, or complain that it is down ... if the underlying machine is up, and all seems well, you may notice in the beancounters that network-specific counters are failing - such as numtcpsock, tcpsndbuf or tcprcvbuf. This will keep them from talking on the network and make it seem like their system is down. Again, just up the limits and things should be fine.<br />
<br />
On virts 1-4, you should first look at the default settings for that item on a later virt, such as virt 8 - we have increased the defaults a lot since the early machines. So, if you are going to up a counter on virt2, instead of upping it by 10-20%, instead up it to the new default that you see on virt8.<br />
<br />
== Moving a VE to another virt (migrate/migrateonline) ==<br />
<br />
This will take a while to complete - and it is best to do this at night when the load is light on both machines.<br />
<br />
There are different methods for this, depending on which version of virtuozzo is installed on the src. and dst. virt. <br />
To check which version is running: <br />
[root@virt12 private]# cat /etc/virtuozzo-release<br />
Virtuozzo release 2.6.0<br />
<br />
Ok, let's say that the VE is 1212, and vital stats are:<br />
<pre>[root@virt12 sbin]# vc 1212<br />
VE_ROOT="/vz1/root/1212"<br />
VE_PRIVATE="/vz1/private/1212"<br />
OSTEMPLATE="fedora-core-2/20040903"<br />
IP_ADDRESS="69.55.229.84"<br />
TEMPLATES="devel-fc2/20040903 php-fc2/20040813 mysql-fc2/20040812 postgresql-fc2/20040813 mod_perl-fc2/20040812 mod_ssl-fc2/20040811 jre-fc2/20040823 jdk-fc2/20040823 mailman-fc2/20040823 analog-fc2/20040824 proftpd-fc2/20040818 tomcat-fc2/20040823 usermin-fc2/20040909 webmin-fc2/20040909 uw-imap-fc2/20040830 phpBB-fc2/20040831 spamassassin-fc2/20040910 PostNuke-fc2/20040824 sl-webalizer-fc2/20040<br />
818"<br />
<br />
[root@virt12 sbin]# vzctl exec 1212 df -h<br />
Filesystem Size Used Avail Use% Mounted on<br />
/dev/vzfs 4.0G 405M 3.7G 10% /</pre><br />
<br />
From this you can see that he’s using (and will minimally need free on the dst server) ~400MB, and he’s running on a Fedora 2 template, version 20040903. He’s also got a bunch of other templates installed. It’s is '''vital''' that '''all''' these templates exist on the dst system. To confirm that, on the dst system run:<br />
<br />
For < 3.0:<br />
<br />
<pre>[root@virt14 private]# vzpkgls | grep fc2<br />
devel-fc2 20040903<br />
PostNuke-fc2 20040824<br />
analog-fc2 20040824<br />
awstats-fc2 20040824<br />
bbClone-fc2 20040824<br />
jdk-fc2 20040823<br />
jre-fc2 20040823<br />
mailman-fc2 20040823<br />
mod_frontpage-fc2 20040816<br />
mod_perl-fc2 20040812<br />
mod_ssl-fc2 20040811<br />
mysql-fc2 20040812<br />
openwebmail-fc2 20040817<br />
php-fc2 20040813<br />
phpBB-fc2 20040831<br />
postgresql-fc2 20040813<br />
proftpd-fc2 20040818<br />
sl-webalizer-fc2 20040818<br />
spamassassin-fc2 20040910<br />
tomcat-fc2 20040823<br />
usermin-fc2 20040909<br />
uw-imap-fc2 20040830<br />
webmin-fc2 20040909<br />
[root@virt14 private]# vzpkgls | grep fedora<br />
fedora-core-1 20040121 20040818<br />
fedora-core-devel-1 20040121 20040818<br />
fedora-core-2 20040903<br />
[root@virt14 private]#</pre><br />
<br />
For these older systems, you can simply match up the date on the template. <br />
<br />
For >= 3.0:<br />
<br />
<pre>[root@virt19 /vz2/private]# vzpkg list<br />
centos-5-x86 2008-01-07 22:05:57<br />
centos-5-x86 devel<br />
centos-5-x86 jre<br />
centos-5-x86 jsdk<br />
centos-5-x86 mod_perl<br />
centos-5-x86 mod_ssl<br />
centos-5-x86 mysql<br />
centos-5-x86 php<br />
centos-5-x86 plesk9<br />
centos-5-x86 plesk9-antivirus<br />
centos-5-x86 plesk9-api<br />
centos-5-x86 plesk9-atmail<br />
centos-5-x86 plesk9-backup<br />
centos-5-x86 plesk9-horde<br />
centos-5-x86 plesk9-mailman<br />
centos-5-x86 plesk9-mod-bw<br />
centos-5-x86 plesk9-postfix<br />
centos-5-x86 plesk9-ppwse<br />
centos-5-x86 plesk9-psa-firewall<br />
centos-5-x86 plesk9-psa-vpn<br />
centos-5-x86 plesk9-psa-fileserver<br />
centos-5-x86 plesk9-qmail<br />
centos-5-x86 plesk9-sb-publish<br />
centos-5-x86 plesk9-vault<br />
centos-5-x86 plesk9-vault-most-popular<br />
centos-5-x86 plesk9-watchdog</pre><br />
<br />
On these newer systems, it's difficult to tell whether the template on the dst matches exactly the src. Just cause a centos-5-x86 is listed on both servers doesn't mean all the same packages are there on the dst. To truly know, you must perform a sample rsync:<br />
<br />
rsync -avn /vz/template/centos/5/x86/ root@10.1.4.61:/vz/template/centos/5/x86/<br />
<br />
if you see a ton of output from the dry run command, then clearly there are some differences. You may opt to let the rsync complete (without running in dry run mode) the only downside is you've now used up more space on the dst and also the centos template will be a mess with old and new data- it will be difficult if not impossible to undo (if someday we wanted to reclaim the space).<br />
<br />
If you choose to merge templates, you should closely inspect the dry run output. You should also take care to exclude anything in the /config directory. For example:<br />
<br />
rsync -av -e ssh --stats --exclude=x86/config /vz/template/ubuntu/10.04/ root@10.1.4.62:/vz/template/ubuntu/10.04/<br />
<br />
Which will avoid this directory and contents:<br />
<pre>[root@virt11 /vz2/private]# ls /vz/template/ubuntu/10.04/x86/config*<br />
app os</pre><br />
<br />
This is important to avoid since the config may differ on the destination and we are really only interested in making sure the pacakges are there, not overwriting a newer config with an older one.<br />
<br />
If the dst system was missing a template, you have 2 choices: <br />
# put the missing template on the dst system. 2 choices here: <br />
## Install the template from rpm (found under backup2: /mnt/data4/vzrpms/distro/) or <br />
## rsync over the template (found under /vz/template) - see above<br />
# put the ve on a system which has all the proper templates<br />
<br />
=== pre-seeding a migration ===<br />
<br />
When migrating a customer (or when doing many) depending on how much data you have to transfer, it can take some time. Further, it can be difficult to gauge when a migration will complete or how long it will take. To help speed up the process and get a better idea about how long it will take you can pre-transfer a customer's data to the destination server. If done correctly, vzmigrate will see the pre-transferred data and pick up where you left off, having much less to transfer (just changed/new files). <br />
<br />
We believe vzmigrate uses rsync to do it's transfer. Therefore not only can you use rsync to do a pre-seed, you can also run rsync to see what is causing a repeatedly-failing vzmigrate to fail. <br />
<br />
There's no magic to a pre-seed, you just need to make sure it's named correctly.<br />
<br />
Given:<br />
<br />
source: /vz1/private/1234<br />
<br />
and you want to migrate to /vz2 on the target system, your rsync would look like:<br />
<br />
rsync -av /vz1/private/1234/ root@x.x.x.x:/vz2/private/1234.migrated/<br />
<br />
After running that successful rsync, the ensuing migrateonline (or migrate) will take much less time to complete- depending on the # of files to be analyzed and the # of changed files. In any case, it'll be much much faster than had you just started the migration from scratch.<br />
<br />
Further, as we discuss elsewhere in this topic, a failed migration can be moved from <tt>/vz/private/1234</tt> to <tt>/vz/private/1234.migrated</tt> on the destination if you want to restart a failed migration. This should '''only''' be done if the migration failed and the CT is not running on the destination HN.<br />
<br />
=== migrateonline intructions: src >=3.x -> dst>=3.x ===<br />
<br />
A script called [[#migrateonline|migrateonline]] was written to handle this kind of move. It is basically a wrapper for <tt>vzmigrate</tt> – vzmigrate is a util to seamlessly- as no no reboot of the ve necessary- move a ve from one host to another. This wrapper was initially written cause vz virtuozzo version 2.6.0 has a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables, causing problems when the ve was started up on the dst system. [[#migrate|migrate]] mitigates that. Since it makes multiple ssh connections to the dst virt, it’s a good idea to put the pub key for the src virt in the authorized_keys file on the dst virt. In addition, migrateonline emails ve owners when their migration starts and stops. For this to happen they need to put email addresses (on a single line, space delimited) in a file on their system: /migrate_notify. If the optional target dir is not specified, the ve will be moved to the same private/root location as it was on the src virt. Note: <tt>migrate</tt> is equivalent to <tt>migrateonline</tt>, but will <tt>migrate</tt> a ve AND restart it in the process.<br />
<br />
<pre>[root@virt12 sbin]# migrateonline<br />
usage: /usr/local/sbin/migrateonline <ip of node migrating to> <veid> [target dir: vz | vz1 | vz2]<br />
[root@virt12 sbin]# migrateonline 10.1.4.64 1212 vz<br />
starting to migrate 1212 at Sat Mar 26 22:40:38 PST 2005<br />
Turning off offline_management<br />
Saved parameters for VE 1212<br />
migrating with no start on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
clearing the arp cache<br />
now going to 10.1.4.64 and clear cache and starting<br />
starting it<br />
Delete port redirection<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
finished migrating 1212 at Sat Mar 26 22 22:52:01 PST 2005<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine.<br />
<br />
If they had backups, use the mvbackups command to move their backups to the new server:<br />
<br />
mvbackups 1212 virt14 vz<br />
<br />
Rename the ve<br />
<br />
[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/migrated-1212<br />
<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/old-1212-migrated-20120404-noarchive<br />
<br />
Update the customer’s systems in mgmt to reflect the new path and server.<br />
<br />
<br />
IF migrateonline does not work, you can try again using simply migrate- this will result in a brief reboot for the ve.<br />
Before you try again, make sure of a few things:<br />
<br />
Depending on where in the migration died, there may be partial data on the dst system in 1 of 2 places:<br />
(given the example above)<br />
<br />
/vz/private/1212<br />
<br />
or <br />
<br />
/vz/private/1212.migrated<br />
<br />
before you run migrate again, you'll want to rename so that all data is in <br />
1212.migrated:<br />
<br />
mv /vz/private/1212 /vz/private/1212.migrated<br />
<br />
this way, it will pick up where it left off and transfer only new files.<br />
<br />
Likewise, if you want to speed up a migration, you can pre-seed the dst as follows:<br />
<br />
[root@virt12 sbin]# rsync -avSH /vz/private/1212/ root@10.1.4.64:/vz/private/1212.migrated/<br />
<br />
then when you run migrate or migrateonline, it will only need to move the changed files- the migration will complete quickly<br />
<br />
=== migrateonline/migrate failures (migrate manually) ===<br />
<br />
Lets say for whatever reason the migration fails. If it fails with [[#migrateonline|migrateonline]], you should try [[#migrate|migrate]] (which will reboot the customer, so notify them ahead of time).<br />
<br />
You may want to run a [[#pre-seeding_a_migration|pre-seed]] rsync to see if you can find the problem. On older virts, we've seen this problem due to a large logfile (which you can find and encourage the customer to remove/compress):<br />
for f in `find / -size +1048576k`; do ls -lh $f; done<br />
<br />
You may also see migration failing due to quota issues.<br />
<br />
You can try to resolve by copying any quota file into the file you need:<br />
<br />
cp /var/vzquota/quota.1 /var/vzquota/quota.xxx<br />
<br />
If it complains about quota running you should then be able to stop it<br />
<br />
vzquota off xxxx<br />
<br />
If all else fails, migrate to a new VEID<br />
i.e. 1234 becomes 12341<br />
<br />
<br />
If the rsync or [[#migrate|migrate]] fails, you can always move someone manually:<br />
<br />
1. stop ve: <br><br />
v stop 1234<br />
<br />
2. copy over data<br><br />
rsync -avSH /vz/private/1234/ root@1.1.1.1:/vzX/private/1234/<br />
<br />
NOTE: if you've previously seeded the data (run rsync while the VE was up/running), and this is a subsequent rsync, make sure the last rsync you do (while the VE is not running, has the --delete option in the rsync)<br />
<br />
3. copy over conf<br><br />
scp /vzconf/1234.conf root@1.1.1.1:/vzconf<br />
<br />
4. on dst, edit the conf to reflect the right vzX dir<br><br />
vi /vzconf/1234.conf<br />
<br />
5. on src remove the IPs<br><br />
ipdel 1234 2.2.2.2 3.3.3.3<br />
<br />
6. on dst add IPs <br><br />
ipadd 1234 2.2.2.2 3.3.3.3<br />
<br />
7. on dst, start ve: <br><br />
v start 1324<br />
<br />
8. cancel, then archive ve on src per above instrs.<br />
<br />
=== migrate src=2.6.0 -> dst>=2.6.0, or mass-migration with customer notify ===<br />
<br />
A script called <tt>migrate</tt> was written to handle this kind of move. It is basically a wrapper for vzmigrate – vzmigrate is a util to seamlessly move a ve from one host to another. This wrapper was initially written cause vz virtuozzo version 2.6.0 has a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables, causing problems when the ve was started up on the dst system. migrate mitigates that. Since it makes multiple ssh connections to the dst virt, it’s a good idea to put the pub key for the src virt in the authorized_keys file on the dst virt. In addition, migrate emails ve owners when their migration starts and stops. For this to happen they need to put email addresses (on a single line, space delimited) in a file on their system: /migrate_notify. If the optional target dir is not specified, the ve will be moved to the same private/root location as it was on the src virt. Note: migrateonline is equivalent to migrate, but will migrate a ve from one 2.6 '''kernel''' machine to another 2.6 kernel machine without restarting the ve.<br />
<br />
<pre>[root@virt12 sbin]# migrate<br />
usage: /usr/local/sbin/migrate <ip of node migrating to> <veid> [target dir: vz | vz1 | vz2]<br />
[root@virt12 sbin]# migrate 10.1.4.64 1212 vz<br />
starting to migrate 1212 at Sat Mar 26 22:40:38 PST 2005<br />
Turning off offline_management<br />
Saved parameters for VE 1212<br />
migrating with no start on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
clearing the arp cache<br />
now going to 10.1.4.64 and clear cache and starting<br />
starting it<br />
Delete port redirection<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
finished migrating 1212 at Sat Mar 26 22 22:52:01 PST 2005<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine (backup2 or mail).<br />
Cancel the ve (first we have to rename things which migrate changed so cancelve will find them):<br />
<br />
[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/1212.conf<br />
<br />
On 2.6.1 you’ll also have to move the private area:<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/1212<br />
<br />
<pre>[root@virt12 sbin]# cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, [[#cancelve|cancelve]] would offer to remove them. You want to say '''no''' to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
To move everyone off a system, you’d do:<br />
for f in `vl`; do migrate <ip> $f; done<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
=== vzmigrate: src=2.6.1 -> dst>=2.6.0 ===<br />
<br />
This version of vzmigrate works properly with regard to handling ips. It will not notify ve owners of moves as in the above example. Other than that it’s essentially the same.<br />
<br />
<pre>[root@virt12 sbin]# vzmigrate 10.1.4.64 -r no 1212:1212:/vz/private/1212:/vz/root/1212<br />
migrating on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine (backup2 or mail).<br />
Cancel the ve (first we have to rename things which vzmigrate changed so cancelve will find them):<br />
<br />
<pre>[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/1212.conf<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/1212<br />
<br />
[root@virt12 sbin]# cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, <tt>cancelve</tt> would offer to remove them. You want to say no to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
=== src=2.5.x ===<br />
<br />
First, go to the private dir:<br />
<br />
cd /vz1/private/<br />
<br />
Stop the VE - make sure it stops totally cleanly.<br />
<br />
vzctl stop 1212<br />
<br />
Then you’d use vemove - a script written to copy over the config, create tarballs of the ve’s data on the destination virt, and cancel the ve on the source system (in this example we’re going to put a ve that was in /vz1/private on the src virt, in /vz/private on the dst virt):<br />
<br />
<pre>[root@virt12 sbin]# vemove<br />
ERROR: Usage: vemove veid target_ip target_path_dir<br />
[root@virt12 sbin]# vemove 1212 10.1.4.64 /vz/private/1212<br />
tar cfpP - 1212 --ignore-failed-read | (ssh -2 -c arcfour 10.1.4.64 "split - -b 1024m /vz/private/1212.tar" )<br />
scp /vzconf/1212.conf 10.1.4.64:/vzconf<br />
cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, cancelve would offer to remove them. You want to say no to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
When you are done, go to /vz/private on the dst virt you will have files like this:<br />
<br />
<pre>1212.taraa<br />
1212.tarab<br />
1212.tarac</pre><br />
<br />
Each one 1024m (or less, for the last one) in size.<br />
<br />
on the dst server and run:<br />
<br />
cat 1212.tar?? | tar xpPBf -<br />
<br />
and after 20 mins or so it will be totally untarred. Now since the conf<br />
file is already there, you can go ahead and start the system.<br />
<br />
vzctl start 1212<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
NOTE: you MUST tar the system up using the virtuozzo version of tar that<br />
is on all the virt systems, and further you MUST untar the tarball with<br />
the virtuozzo tar, using these options: `<tt>tar xpPBf -</tt>`<br />
<br />
If you tar up an entire VE and move it to a non-virtuozzo machine, that is<br />
ok, and you can untar it there with normal tar commands, but do not untar<br />
it and then repack it with a normal tar and expect it to work - you need<br />
to use virtuozzo tar commands on virtuozzo tarballs to make it work.<br />
<br />
The backups are sort of an exception, since we are just (usually)<br />
restoring user data that was created after we gave them the system, and<br />
therefore has nothing to do with magic symlinks or vz-rpms, etc.<br />
<br />
== Moving a VE on the same virt ==<br />
<br />
Easy way:<br><br />
Scenario 1: ve 123 is to be renamed 1231 and moved from vz1 to vz<br />
<br />
vzmlocal 123:1231:/vz/private/1231:/vz/root/1231<br />
<br />
Scenario 2: ve 123 is to be moved vz1 to vz<br />
<br />
vzmlocal 123:123:/vz/private/123:/vz/root/123<br />
<br />
vzmlocal will reboot the ve at the end of the move<br />
<br />
Manual/old way:<br />
<br />
1) <tt>vzctl stop 123</tt><br><br />
2) <tt>mv /vz1/private/123 /vz/private/.</tt><br><br />
(or cp -a if you want to copy)<br />
3) in <tt>/etc/sysconfig/vz-scripts/123.conf</tt> change value<br><br />
of '<tt>VE_PRIVATE</tt>' variable to point to a new private area location<br />
4) <tt>vzctl start 123</tt><br><br />
5) update backups if needed: <tt>mvbackups 123 virtX virt1 vz</tt><br><br />
6) update management scerens<br><br />
<br />
Notes: a) absolute path to private area is stored in quota file <tt>/var/vzquota/quota.123</tt> - so during first startup quota will be recalculated.<br><br />
b) if you're going to write some script to do a job, you MUST be sure that $VEID won't be expanded to '' in ve config file - ie. you need to escape '$'. Otherwise you might have:<br />
<br />
VE_PRIVATE="/vz/private/"<br />
<br />
in config, and 'vzctl destroy' for this VE ID '''will remove everything under /vz/private/ directory'''.<br />
<br />
== Adding a veth device to a VE ==<br />
<br />
Not totally sure what this is, but a customer asked for it and here's what we did (as instructed by vz support):<br />
<br />
<pre>v set 99 --netif_add eth99 --save<br />
ipdel 99 69.55.230.58<br />
v set 99 --ifname eth99 --ipadd 69.55.230.58 --save<br />
v set 99 --ifname eth99 --gateway 69.55.230.1 --save<br />
<br />
vznetcfg net new veth_net<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
net99 active eth0 veth77.77,veth99.99<br />
veth_net active<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
br0 bridge veth_net<br />
br99 bridge net99<br />
veth99.99 veth net99<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic net99 69.55.227.70/24<br />
<br />
vznetcfg br attach br0 eth0<br />
<br />
(will remove 99 from orig net and move to veth_net)<br />
vznetcfg net addif veth_net veth99.99<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
net99 active<br />
veth_net active eth0 veth77.77,veth99.99<br />
<br />
(delete the old crap)<br />
vznetcfg net del net99<br />
<br />
<br />
Then, to add another device in<br />
<br />
v set 77 --netif_add eth77 --save<br />
ipdel 77 69.55.230.78<br />
v set 77 --ifname eth77 --ipadd 69.55.230.78 --save<br />
v set 77 --ifname eth77 --gateway 69.55.230.1 --save<br />
v set 77 --save --ifname eth77 --network veth_net<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
veth77.77 veth<br />
br0 bridge veth_net<br />
veth99.99 veth veth_net<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic veth_net 69.55.227.70/24<br />
<br />
vznetcfg net addif veth_net veth77.77<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
veth77.77 veth veth_net<br />
br0 bridge veth_net<br />
veth99.99 veth veth_net<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic veth_net 69.55.227.70/24<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
veth_net active eth0 veth77.77,veth99.99<br />
<br />
another example<br />
<br />
v set 1182 --netif_add eth1182 --save<br />
ipdel 1182 69.55.236.217<br />
v set 1182 --ifname eth1182 --ipadd 69.55.236.217 --save<br />
v set 1182 --ifname eth1182 --gateway 69.55.236.1 --save<br />
vznetcfg net addif veth_net veth1182.1182<br />
v set 1182 --save --ifname eth1182 --network veth_net<br />
<br />
<br />
unused/not working commands:<br />
ifconfig veth99.0 0<br />
vznetcfg net list<br />
vznetcfg br new br99 net99<br />
vznetcfg br attach br99 eth0<br />
vznetcfg br show<br />
vznetcfg if list<br />
vznetcfg net addif net99 veth99.99<br />
vznetcfg net addif eth0 net99<br />
<br />
---------<br />
<br />
vznetcfg br new br1182 net1182<br />
<br />
vznetcfg br attach br99 eth0<br />
vznetcfg if list<br />
vznetcfg net addif net99 veth99.99<br />
vznetcfg net addif eth0 net99<br />
<br />
<br />
<br />
vznetcfg net addif eth0 net1182<br />
<br />
<br />
# vznetcfg net addif veth99.0 net99<br />
--- 8< ---<br />
<br />
vznetcfg net new net<br />
# vznetcfg net addif eth0 net99<br />
<br />
<br />
# vzctl set 99 --save --netif_add eth0 (at this stage veth99.0 interface have to appear<br />
on node)<br />
# vzctl set 99 --save --ifname eth0 --ipadd 69.55.230.58 (and probably few more arguments<br />
here - see 'man vzctl')<br />
# vznetcfg net addif veth99.0 net99</pre><br />
<br />
<br />
== Assigning/remove ip from a VE ==<br />
<br />
1. Add or remove ips:<br />
ipdel 1234 1.1.1.1 2.2.2.2<br />
ipadd 1234 1.1.1.1 2.2.2.2<br />
<br />
2. update Mgmt screens<br />
<br />
3. offer to update any DNS we do for them<br />
<br />
4. check to see if we had rules for old IP in firwall<br />
<br />
== Enabling tun device for a ve ==<br />
Note, there’s a command for this: [[#addtun|addtun]]<br />
<br />
For posterity:<br />
Make sure the tun.o module is already loaded before Virtuozzo is started: <br />
lsmod <br />
Allow the VPS to use the TUN/TAP device: <br />
vzctl set 101 --devices c:10:200:rw --save <br />
Create the corresponding device inside the VPS and set the proper permissions: <br />
vzctl exec 101 mkdir -p /dev/net <br />
vzctl exec 101 mknod /dev/net/tun c 10 200 <br />
vzctl exec 101 chmod 600 /dev/net/tun<br />
<br />
== Remaking a system (on same virt) ==<br />
<br />
1. [[#cancelve|cancelve]] (or v destroy x - ONLY if you're POSITIVE no data needs to be saved)<br />
<br />
2. [[#vemake|vemake]] using same veid<br />
<br />
3. [[#mvbackups|mvbackups]] or [[#vb|vb]] (if new mount point)<br />
<br />
4. update mgmt with new dir/ip <br />
<br />
5. update firewall if changed ip<br />
<br />
== Re-initialize quota for a VE ==<br />
<br />
There’s a commamd for this now: [[#clearquota|clearquota]]<br />
<br />
For posterity:<br />
<br />
vzctl stop 1<br />
vzquota drop 1<br />
vzctl start 1<br />
<br />
<br />
== Traffic accounting on linux ==<br />
<br />
DEPRECATED - all tracking is done via bwdb now. This is how we used to track traffic.<br />
<br />
TODO: update for diff versions of vz<br />
<br />
Unlike FreeBSD, where we have to add firewall count rules to the system to count the traffic, on virtuozzo counts the traffic for us. You an see the current traffic stats by running `vznetstat`:<br />
<br />
<pre># vznetstat<br />
VEID Net.Class Output(bytes) Input(bytes)<br />
-----------------------------------------------<br />
24218 1 484M 39M<br />
24245 1 463M 143M<br />
2451 1 2224M 265M<br />
2454 1 2616M 385M<br />
4149 1 125M 68M<br />
418 1 1560M 34M<br />
472 1 1219M 315M<br />
726 1 628M 317M<br />
763 1 223M 82M<br />
771 1 4234M 437M<br />
-----------------------------------------------<br />
Total: 13780M 2090M</pre><br />
<br />
As you can see the VEID is on a line with the in and out bytes. So, we simply run a cron job:<br />
<br />
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /root/vztrafdump.sh<br />
<br />
Just like we do on FreeBSD - this one goes through all the VEs in /vz/private and greps the line from vznetstat that matches them and dumps it in /jc_traffic_dump on their system. Then it does it again for all the VEs in /vz1/private. It is important to note that vznetstat runs only once, and the grepping is done from a temporary file that contains that output - we do this because running vznetstat once for each VE that we read out of /vz/private and /vz1/private would take way too long and be too intensive.<br />
<br />
You do not need to do anything to facilitate this other than make sure that that cron job is running - the vznetstat counters are always running, and any new VEs that are added to the system will be accounted for automatically.<br />
<br />
Traffic resetting no longer works with vz 2.6, so we disable the vztrafdump.sh on those virts.<br />
<br />
== Watchdog script ==<br />
<br />
On some of the older virts, we have a watchdog running that kills procs that are deemed bad per the following:<br />
<br />
/root/watchdog from quar1<br />
<br />
<pre>if echo $line | awk '{print $(NF-3)}' | grep [5-9]...<br />
then<br />
# 50-90%<br />
if echo $line | awk '{print $(NF-1)}' | grep "...:.."<br />
then<br />
# running for > 99min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
if echo $line | awk '{print $(NF-1)}' | grep "....m"<br />
then<br />
# running for > 1000min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
fi<br />
<br />
if echo $line | awk '{print $(NF-3)}' | grep [1-9]...<br />
then<br />
# running for 10-90 percent<br />
if echo $line | awk '{print $NF}' | egrep 'cfusion|counter|vchkpw'<br />
then<br />
<br />
if echo $line | awk '{print $(NF-1)}' | grep "[2-9]:.."<br />
then<br />
# between 2-9min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
elif echo $line | awk '{print $(NF-1)}' | grep "[0-9][0-9]:.."<br />
then<br />
# up to 99min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
fi<br />
fi</pre><br />
<br />
<br />
== Misc Linux Items ==<br />
<br />
We are overselling hard drive space ... when you configure a linux system with a certain amount of disk space (the default is 4gigs) you do not actually use up 4gigs of space on the system. The diskspace setting for a user is simply a cap, and they only use up as much space on the actual disk drive as they are actually using.<br />
<br />
When you create a new linux system, even though there are some 300 RPMs or so installed, if you run `df -k` you will see that the entire 4gig partition is empty - no space is being used. This is because the files in their system are "magic symlinks" to the template for their OS that is in /vz/template - however, any changes to any of those files will "disconnect" them and they will immediately begin using space in their system. Further, any new files uploaded (even if those new files overwrite existing files) will take up space on the partition.<br />
<br />
<br />
if you see this:<br />
<br />
<pre>[root@virt8 root]# vzctl stop 160 ; vzctl start 160<br />
VE is not running<br />
Starting VE ...<br />
VE is unmounted<br />
VE is mounted<br />
Adding IP address(es): 69.55.226.83 69.55.226.84<br />
bash ERROR: Can't change file /etc/sysconfig/network<br />
Deleting IP address(es): 69.55.226.83 69.55.226.84<br />
VE is unmounted<br />
[root@virt8 root]#</pre><br />
<br />
it probably means they no longer have /bin/bash - copy one in for them<br />
<br />
ALSO: another possibility is that they have removed the `ed` RPM from their system - it needs to be reinstalled into their system. But since their system is down, this is tricky ...<br />
<br />
VE startup scripts used by 'vzctl' want package 'ed' to be available inside VE. So if package 'ed' will be enabled in OS template config and OS template itself VE #827 is based on - this error should be fixed.<br />
<br />
yes, it is possible to add RPM to VE while it not running.<br />
Try to do following:<br />
<br />
<pre># cd /vz/template/<OS_template_with_ed_package>/<br />
# vzctl mount 827<br />
# rpm -Uvh --root /vz/root/827 --veid 827 ed-0.2-25.i386.vz.rpm</pre><br />
<br />
Usually theres an error, but its ok<br />
<br />
Note: replace 'ed-0.2-25.i386.vz.rpm' in last command with actual<br />
version of 'ed' package you have.<br />
<br />
----<br />
<br />
So how do I know what template the user has ? cat their conf file and it is listed in there. For example, if the conf file has:<br />
<br />
<pre>[root@virt12 sbin]# vc 1103<br />
…snip…<br />
OSTEMPLATE="debian-3.0/20030822"<br />
TEMPLATES="mod_perl-deb30/20030707 mod_ssl-deb30/20030703 mysql-deb30/20030707 proftpd-deb30/20030703 webmin-deb30/20030823 "<br />
…</pre><br />
<br />
then they are on debian 3.0, all of their system RPMs are in /vz/template/debian-3.0, and they are using version 20030822 of that debian 3.0 template. Also, they’ve also got additional packages installed (mod_perl, mod_ssl, etc). Those are also found under /vz/template<br />
<br />
----<br />
<br />
Edits needed to run java:<br />
<br />
When we first created the VEs, the default setting for privvmpages was 93000:94000 ... which was high enough that most people never had problems ... however, you can;t run java or jdk or tomcat or anything java related with that setting. We have found that by setting privvmpages to 610000:615000 that java runs just fine. That is now the default setting. It is exceedingly rare that anyone needs it higher than that, although we have seen it once or twice.<br />
<br />
Any problems with java at all - the first thing you need to do is see if the failcnt has raised for privvmpages.<br />
<br />
<pre># vzctl start 160<br />
Starting VE ...<br />
vzquota : (error) Quota on syscall for 160: Device or resource busy<br />
Running vzquota on failed for VE 160 [3]</pre><br />
<br />
This is because my pwd is _in_ their private directory - you can't start it until you move out<br />
<br />
People seem to have trouble with php if they are clueless newbies. Here are two common problems/solutions:<br />
<br />
no... but i figured it out myself. problem was the php.ini file that came<br />
vanilla with the account was not configured to work with apache (the<br />
ENGINE directive was set to off).<br />
<br />
everything else seems fine now.<br />
<br />
----<br />
<br />
the problem was in the php.ini file. I noticed that is wasnt showing<br />
the code when it was in an html file so I looked at the php.ini file<br />
and had to change it so it recognized <? tags aswell as <?php tags.<br />
<br />
Also, make sure added to httpd.conf<br />
AddType application/x-httpd-php .php<br />
<br />
----<br />
<br />
You can set the time zone:<br />
<br />
You can change the timezone by doing this:<br />
<br />
ln -sf /usr/share/zoneinfo/<zone> /etc/localtime<br />
<br />
where <zone> is the zone you want in the /usr/share/zoneinfo/ directory.<br />
<br />
----<br />
<br />
Failing shm_open calls:<br />
<br />
first, please check if /dev/shm is mounted inside VE.<br />
'cat /proc/mounts' command should show something like this:<br />
tmpfs /dev/shm tmpfs rw 0 0<br />
<br />
If /dev/shm is not mounted you have 2 ways to solve issue:<br />
1. execute following command inside VE (doesn't require VE reboot):<br />
mount -t tmpfs none /dev/shm<br />
2. add following string to /etc/fstab inside VE and reboot it:<br />
tmpfs /dev/shm tmpfs defaults 0 0<br />
<br />
----<br />
<br />
You can have a mounted but not running ve<br />
Just:<br />
vzctl mount <veid><br />
<br />
----<br />
<br />
When a debian sys can’t get on the network, and you try:<br />
<br />
<pre>vzctl set 1046 --ipadd 69.55.227.117<br />
Adding IP address(es): 69.55.227.117<br />
Failed to bring up lo.<br />
Failed to bring up venet0.</pre><br />
<br />
They probably removed iproute package, which must be the one from swsoft. To restore:<br />
<pre># dpkg -i --veid=1046 --admindir=/vz1/private/1046/root/var/lib/dpkg --instdir=/vz1/private/1046/root/ /vz/template/debian-3.0/iproute_20010824-8_i386.vz.deb<br />
(Reading database ... 16007 files and directories currently installed.)<br />
Preparing to replace iproute 20010824-8 (using .../iproute_20010824-8_i386.vz.deb) ...<br />
Unpacking replacement iproute ...<br />
Setting up iproute (20010824-8) ...</pre><br />
<br />
Then restart their ve<br />
<br />
----<br />
<br />
in a ve i do:<br />
<br />
<pre>cd /<br />
du -h .</pre><br />
<br />
and get: 483M .<br />
<br />
i do:<br />
<br />
<pre>bash-2.05a# df -h<br />
Filesystem Size Used Avail Use% Mounted on<br />
/dev/vzfs 4.0G 2.3G 1.7G 56% /</pre><br />
<br />
how can this be?<br />
<br />
Is it possible that quota file was corrupted somehow? Please try to: <br />
<pre>vzctl stop <VEID><br />
vzquota drop <VEID><br />
vzquota init <VEID><br />
vzctl start <VEID></pre><br />
<br />
----<br />
<br />
How to stop vz from starting after reboot:<br />
<br />
VIRTUOZZO=no <br />
in <br />
/etc/sysconfig/vz<br />
<br />
To start: <br />
service vz start<br />
(after setting VIRTUOZZO=yes in /etc/sysconfig/vz)<br />
<br />
service vz restart will do some kind of 'soft reboot' -- restart all<br />
VPSes and reload modules without rebooting the node<br />
<br />
if you need to shut down all VPSes really really fast, run killall -9 init<br />
<br />
----<br />
<br />
Postfix tip:<br />
<br />
You may want to tweak settings: default_process_limit=10<br />
<br />
= Virtuozzo VPS Management Tools =<br />
<br />
== vm ==<br />
<br />
TODO<br />
<br />
== cancelve ==<br />
cancelve <veid><br />
this will:<br />
* stop a ve<br />
* check for backups (offer to remove them from the backup server <br />
and the backup.config)<br />
* rename the private dir<br />
* check for PTR, provide the commands to reset to default<br />
* and rename the ve’s config<br />
* remind you to remove firewall rules<br />
* remind you to remove DNS entries<br />
<br />
== ipadd ==<br />
ipadd <veid> <ip> [ip] [ip]<br />
add’s ip(s) to a ve<br />
<br />
== ipdel ==<br />
ipdel <veid> <ip> [ip] [ip]<br />
removes ip(s) from a ve<br />
<br />
== vc ==<br />
vc <veid><br />
the equivalent of: <tt>cat /vzconf/<veid>.conf</tt><br />
<br />
== vl ==<br />
vl<br />
will displays a list of ve #’s, 1 per line. (ostensibly to use in a for loop)<br />
<br />
== vp ==<br />
vp <veid><br />
the equivalent of: <tt>vzps auxww –E <veid></tt><br />
<br />
== vpe ==<br />
DEPRECATED<br />
vpe <veid> <br />
this will allow you to do a vp when a ve is running out of control, the equivalent of (deprecated since vp operates outside the VPS): <br />
<pre>vzctl set <veid> --kmemsize 2100000:2200000<br />
vzctl exec <veid> ps auxw<br />
vzctl set <veid> --kmemsize (ve’s orig lvalue):(ve’s orig hvalue)</pre><br />
<br />
== vt ==<br />
vt <veid><br />
the equivalent of: <tt>vztop –E <veid></tt><br />
<br />
== vr ==<br />
vr <veid><br />
the equivalent of: <tt>vzctl stop <veid>; vzctl start <veid></tt><br />
You can run this even if the ve is down - the stop command will just fail<br />
<br />
== vs ==<br />
vs [veid]<br />
displays status (output of <tt>vzctl status <veid></tt>) on each ve configured on the system (as determined by <tt>/vzconf/*.conf</tt>)<br />
If passed an argument, gives the status for just that ve. <br />
A running system looks like:<br />
<tt>VEID 16066 exist mounted running</tt><br />
<br />
A ve which is not running (but does exist) looks like:<br />
<tt>VEID 9990 exist unmounted down</tt><br />
<br />
A ve which is not running and doesn’t exist looks like:<br />
<tt>VEID 421 deleted unmounted down</tt><br />
<br />
== vs2 ==<br />
vs2 [veid]<br />
this is similar to vs in that it displays status (output of <tt>vzctl status <veid></tt>) on each ve,<br />
but the difference is it’s list comes from doing an ls on the data dirs. This was meant to catch <br />
the rare case where a ve configured but exists. <br />
<br />
== vw ==<br />
vw [veid]<br />
displays the output of ‘<tt>w</tt>’ (the equivalent of <tt>vzctl exec <veid> w</tt>) for each configured ve (as determined by <tt>/vzconf/*.conf</tt>). Useful for determing which ve is contributing to a heavily-loaded system.<br />
If passed an argument, gives ‘<tt>w</tt>‘ output for just that ve. <br />
Ex:<br />
<pre>[root@virt2 etc]# vw<br />
134<br />
10:52pm up 79 days, 6:14, 0 users, load average: 0.02, 0.02, 0.00<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT<br />
16027<br />
2:52pm up 7 days, 19:54, 0 users, load average: 0.00, 0.00, 0.00<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT<br />
16055<br />
2:52pm up 79 days, 6:38, 0 users, load average: 0.00, 0.04, 0.07<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT</pre><br />
<br />
== vwe ==<br />
vwe [constraint]<br />
just like <tt>vw</tt>, but takes a constraint as an argument, only show’s ve’s with loads >= the constraint provided. If no constraint is provided, 1 is used by default<br />
<br />
== vzs ==<br />
vzs [veid]<br />
displays the beancounter status for all ve’s, or a particular ve if an argument is passed<br />
<br />
== ve ==<br />
ve <veid><br />
the equivalent of: <tt>vzctl enter <veid></tt><br />
<br />
== vx ==<br />
vx <veid> <command><br />
the equivalent of: <tt>/usr/sbin/vzctl exec <veid> <command></tt><br />
<br />
== dprocs ==<br />
dprocs [count]<br />
a script which outputs a continuous report (or a certain number of reports if an option is passed) of processes stuck in the D state and which VPS’s those procs belong to.<br />
<br />
== setmem ==<br />
setmem <VEID> <256|512|768|1024|2048><br />
adjusts the memory resources for the VE<br />
<br />
== afacheck.sh ==<br />
afacheck.sh<br />
displays the health/status of containers and mirrors on an adaptec card (currently quar1, tempvirt1-2, virt9, virt10)- all other are LSI<br />
<br />
== backup ==<br />
backup<br />
backup script called nightly to update virt scripts and do customer backups<br />
<br />
== checkload.pl ==<br />
checkload.pl<br />
this was intended to be setup as a cronjob to watch processes on a virt when the load <br />
rises above a certain level. Not currently in use.<br />
<br />
== findbackuppigs.pl ==<br />
findbackuppigs.pl<br />
looks for files larger than 50MB which customers have asked us to backup. Emails matches<br />
to linux@johncompanies.com<br />
<br />
== gatherlinux.pl ==<br />
gatherlinux.pl<br />
gathers up data about ve’s configured and writes to a file. Used for audits against the db<br />
<br />
== gb ==<br />
gb <search><br />
greps backup.config for the given search parameter<br />
<br />
== gbg ==<br />
gbg <search><br />
greps backup.config for the given search parameter and presents just the directories (for clean pasting)<br />
<br />
== linuxtrafficgather.pl ==<br />
linuxtrafficgather.pl [yy-mm]<br />
sends a traffic usage summary by ve to support@johncomapnies.com and payments@johncompanies.com.<br />
Optional arguments are year and month (must be in the past). If not passed, assumes last month. Relies on <br />
traffic logs created by netstatreset and netstatbackup<br />
<br />
== linuxtrafficwatch.pl ==<br />
linuxtrafficwatch.pl<br />
checks traffic usage and emails support@johncomapnies.com when a ve reaches the warning level (35G) and<br />
the limit (40G). Works on virtuozzo versions <= 2.6.x. We really aren’t using this anymore now that we have netflow.<br />
<br />
== linuxtrafficwatch2.pl ==<br />
linuxtrafficwatch2.pl<br />
checks traffic usage and emails support@johncomapnies.com when a ve reaches the warning level (35G) and<br />
the limit (40G). Works on virtuozzo version 2.6.x. We really aren’t using this anymore now that we have netflow.<br />
<br />
== load.pl ==<br />
load.pl<br />
feeds info to load mrtg - executed by inetd.<br />
<br />
== mb (linux) ==<br />
mb <mount|umount><br />
(nfs) mounts and umounts dirs to backup2. Shortcuts are mbm and mbu to mount and unmount. <br />
<br />
== migrate ==<br />
migrate <ip of node migrating to> <veid> <target_veid> [target dir: vz | vz1 | vz2]<br />
this is basically a wrapper for <tt>vzmigrate</tt> – vzmigrate is a util to seamlessly move a ve from one host to another. This wrapper was written cause vz virtuozzo version 2.6 had a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables. This script mitigates that. Since it makes multiple ssh connections to the target host, it’s a good idea to put the pub key for the src system in the authorized_keys file on the target host. In addition, it emails ve owners when their migration starts and stops (if they place email addresses in a file on their system: /migrate_notify). To move everyone off a system, you’d do:<br />
for f in `vl`; do migrate <ip> $f; done<br />
<br />
== migrateonline ==<br />
migrateonline <ip of node migrating to> <veid> <target_veid> [target dir: vz | vz1 | vz2]<br />
this is the same as migrate but will migrate a ve in <tt>–online</tt> mode which means it won’t be shut down at the end of the migration. This only works when migrating ve’s between 2 machines running a 2.6 kernel (currently tempvirt1-2. virt16-19, virt12). If you get an error that the machine you’re trying to migrate to has a different CPU or features, etc, then you have to edit the file and add the –f switch to the vzmigrate line- you can basically ignore this kind of warning (but never ignore a warning about missing templates on the destination node). NOTE: This edit (if made to migrateonline) will be overwritten by the base script during each night’s backup.<br />
<br />
== netstatbackup ==<br />
DEPRECATED<br />
netstatbackup <br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.5.x. <br />
<br />
== netstatbackup2 ==<br />
DEPRECATED<br />
netstatbackup2<br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.6.x. <br />
<br />
== netstatreset ==<br />
DEPRECATED<br />
netstatreset<br />
writes traffic count data to a logfile and resets counters to 0. Works on virtuozzo versions 2.5.x <br />
<br />
== netstatreset2 ==<br />
DEPRECATED<br />
netstatreset2<br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.6.x. <br />
<br />
== orphanedbackupwatchlinux ==<br />
orphanedbackupwatchlinux <br />
looks for directories on backup2 which aren’t configured in backup.config and offers to <br />
delete them<br />
<br />
== rsync.backup (linux) ==<br />
rsync.backup<br />
does customer backups (relies on backup.config)<br />
<br />
== startvirt.pl ==<br />
startvirt.pl<br />
forks off start ve commands – keeps 6 running at a time. This is not to be used on systems where fastboot is enabled as it circumvents the benefit of the fastboot. The script will occasionally not exit gracefully and will continue to use up CPU, so it should be watched. Also, don’t exit from the script till you’re sure all ve’s are started – if you do you need to start them manually and may have to free up locks. On some systems, the startvirt script doesn’t exit cleanly and you have to ^C out of it. Be careful though- doing so can leave some VE’s in an odd bootup state and you may need to ‘vr’ them manually. You should check to see which ve’s aren’t running and/or confirm all have started when ^C’ing out of startvirt.<br />
<br />
== taskdone (linux) ==<br />
taskdone<br />
when called will email support@johncompanies.com with the hostname of the machine from which it was <br />
executed as the subject<br />
<br />
== vb (linux) ==<br />
vb<br />
the equivalent of: <tt>vi /usr/local/sbin/backup.config</tt><br />
<br />
== vemakeXX ==<br />
DEPRECATED<br />
vemakerh9 <br />
ve create script for RH9 (see vemake)<br />
<br />
vemakedebian30 <br />
ve create script for debian 3.0 (Woody) (see vemake)<br />
<br />
vemakedebian31 <br />
ve create script for debian 3.1 (Sarge) (see vemake)<br />
<br />
vemakedebian40 <br />
ve create script for debian 4.0 (Etch) (see vemake)<br />
<br />
vemakefedora, vemakefedora2, vemakefedora4, vemakefedora5, vemakefedora6, vemakefedora7<br />
ve create script for fedora core 1, 2, 4, 5, 6, 7 respectively (see vemake)<br />
<br />
vemakecentos3, vemakecentos4<br />
ve create script for centos 3, 4 respectively (see vemake)<br />
<br />
vemakesuse, vemakesuse93, vemakesuse100<br />
ve create script for suse 9.2, 9.3, 10.0 respectively (see vemake)<br />
<br />
vemakeubuntu5, vemakeubuntu606, vemakeubuntu606 vemakeubuntu704<br />
ve create script for ubuntu 5.10, 6.06, 6.10, 7.04 respectively (see vemake)<br />
<br />
== vemove ==<br />
DEPRECATED<br />
vemove <veid> <target_ip> </vz/private/123><br />
this script simplifies the old way of moving ve’s from one system to another - in short moving a ve to or from a virt running virtuozzo < 2.6.x<br />
It’s the equivalent of:<br />
<tt>tar cfpP - <veid> --ignore-failed-read | (ssh -2 -c arcfour <target_ip> "split - -b 1024m </vz/private/123>.tar" )</tt>This should only be used if migrate/vzmigrate can’t be used. <br />
<br />
== vim.watchdog ==<br />
vim.watchdog <br />
looks for and kills procs matching “vi|vim|nano|pine|elm” that are running for a long time & consuming lots of cpu. Works on virtuozzo versions 2.5.x<br />
<br />
== vim.watchdog2 ==<br />
vim.watchdog2<br />
looks for and kills procs matching “vi|vim|nano|pine|elm” that are running for a long time & consuming lots of cpu.<br />
Works on virtuozzo versions 2.6.x.<br />
<br />
== vzmigrate ==<br />
vzmigrate <target_ip> -r no <veid>:[dst veid]:[dst /vzX/private/veid]:[dst /vzX/root/veid]<br />
(this is the raw command “wrapped” by migrate/migrateonline) this will seamlessly move a ve from one host to another. The ve will run for the duration of the migration till the very end when it’s shut down, ip moved and started up on the target system. The filesystem on the src will remain. This should be watched – occasionally the move will timeout and leave the system shut down. If target private and root aren’t specified it just puts it in /vz. Only works when both systems are running virtuozzo 2.6.x<br />
<br />
== vztrafdump.sh ==<br />
DEPRECATED<br />
vztrafdump.sh<br />
writes traffic usage info by ve to a file called jc_traffic_dump in each ve’s / dir. Works on virtuozzo versions <= 2.5.x. <br />
<br />
== vztrafdump2.sh ==<br />
DEPRECATED<br />
vztrafdump2.sh<br />
writes traffic usage info by ve to a file called jc_traffic_dump in each ve’s / dir. Works on virtuozzo versions 2.6.x. <br />
<br />
== addtun ==<br />
addtun <veid><br />
Add’s tun device to ve.<br />
<br />
== bwcap ==<br />
bwcap <veid> <kbps><br />
Ex: <tt>bwcap 1234 512</tt><br />
Caps a VE’s bandwidth to the amount given<br />
<br />
== setdisk ==<br />
setdisk <veid> <diskspace in GB><br />
Ex: <tt>setdisk 1234 5</tt><br />
Gives a VE’s a given amount of disk space<br />
<br />
== vdf ==<br />
vdf <veid> <br />
the equivalent of: <tt>vzctl exec <veid> df –h</tt><br />
<br />
== vdff ==<br />
vdff<br />
runs a (condensed) vdf for all ve’s in your pwd (must be run from /vz/privateN)<br />
<br />
== mvbackups ==<br />
mvbackups <veid> <target_machine> (virt1) <target_dir> (vz1)<br />
moves backups from one location to another on the backup server, and provides you with option to remove entries from current backup.config, and simple paste command to add the config to backup.config on the target server<br />
<br />
== checkquota ==<br />
checkquota<br />
for all the ve’s in the cwd (run from /vz/private, /vz1/private, etc) reports what vz quota says they’re using and what the actual usage is (as reported by du)<br />
<br />
== clearquota ==<br />
clearquota <veid><br />
Recalculates a ve’s quota, prints out the usage before and after. The equivalent of:<br />
<tt>vdf <veid>; v stop <veid>; vzquota drop <veid>; v start <veid>; vdf <veid></tt><br />
<br />
== dprocs ==<br />
dprocs<br />
Sometimes the server’s have a large number of processes get stuck in the D state- this script shows (every 3 secs) which VE’s have D procs, which procs<br />
are stuck and a running average of the top “offenders”<br />
<br />
== vzstat ==<br />
vstat<br />
sort of like top for VZ. sort VEs by CPU usage by pressing 'o' and then 'c' keys<br />
<br />
== stopvirt ==<br />
stopvirt<br />
will stop VEs as fast as it can, 6 at a time. May not exit when complete so you should watch [[#vzstat|vzstat]] in another window.</div>Supporthttps://wiki.jcihosting.com/index.php?title=VPS_Management&diff=1023VPS Management2013-03-11T23:14:35Z<p>Support: /* migrate manually (if migrate/online fails) */</p>
<hr />
<div>= Common Problems =<br />
== Login to any machine without a password ==<br />
<br />
This is possible via the use of ssh keys. The process is thus:<br />
<br />
1. place the public key for your user (root@mail) in the /root/.ssh/authorized_keys file on the server you wish to login to<br />
cat /root/.ssh/id_dsa.pub<br />
(paste that into authorized_keys on the target server). If the file doesn't exist, create it.<br />
<br />
2. enable root login (usually only applies to FreeBSD). Edit the /etc/ssh/sshd_config on the target server and change:<br />
<tt>#PermitRootLogin no</tt><br />
to<br />
<tt>PermitRootLogin yes</tt><br />
<br />
3. Restart the sshd on the target machine. First, find the sshd process: <br />
jailps <hostname> | grep sshd <br />
or <br />
vp <VEID> | grep sshd<br />
<br />
Look for the process resembling:<br />
root 17296 0.0 0.0 5280 1036 ? Ss 2011 4:27 /usr/sbin/sshd <br />
(this is the sshd)<br />
<br />
Not:<br />
root 6270 0.5 0.0 6808 2536 ? Ss 14:33 0:00 sshd: root [priv]<br />
(this is an sshd child- someone already ssh'd in as root)<br />
<br />
Restart the sshd: <br />
kill -1 <PID><br />
<br />
Ex:<br />
kill -1 17296<br />
<br />
You may now ssh in.<br />
<br />
Once you're done, IF you enabled root login, you should repeat steps 2 and 3 to disable root logins.<br />
<br />
== Letting someone in who has locked themselves out (killed sshd, lost pwd) ==<br />
<br />
There are two ways people frequently lock themselves out - either they forget a password, or they kill off sshd somehow.<br />
<br />
These are actually both fairly easy to solve. First, let's say someone kills off their sshd, or somehow mangles /etc/ssh/sshd_config such that it no longer lets them in.<br />
<br />
Their email may be very short, or it may have all sorts of details about how you should fix sshd_config to let them in ... just ignore all of this. They can fix their own mangled sshd. Fixing this is very simple. First, edit the /etc/inetd.conf on their system and uncomment the telnet line:<br />
<br />
telnet stream tcp nowait root /usr/libexec/telnetd telnetd<br />
#telnet stream tcp6 nowait root /usr/libexec/telnetd telnetd<br />
<br />
(just leave the tcp6 version of telnet commented)<br />
<br />
Then, use jailps to list the processes on their system, and find their inetd process. Then simply:<br />
<br />
kill -HUP (pid)<br />
<br />
where (pid) is the PID of their inetd process. Now they have telnet running on their system and they can log in and do whatever they need to do.<br />
<br />
The only complications that could occur are:<br />
<br />
a) their firewall config on our firewall has port 23 blocked, in which case you will need to open that - will be covered in a different lesson.<br />
<br />
b) they are not running inetd, so you can't HUP it. If this happens, edit their /etc/rc.conf, add the inetd_enable="YES" line, and then kill<br />
their jail with /tmp/jailkill.pl - then restart their jail with the jail line from their quad/safe file. Easy.<br />
<br />
If they have forgotten a password,<br />
<br />
On 6.x+ you can reset their password with:<br />
jexec <jailID from jls> passwd root<br />
<br />
Note: the default password for 6.x jails is 8ico2987, for 4.x it is p455agfa<br />
<br />
On 4.x, you need to cd to their etc directory<br />
... for instance:<br />
<br />
cd /mnt/data2/198.78.65.136-col00261-DIR/etc<br />
<br />
and run:<br />
<br />
vipw -d .<br />
<br />
Then paste in these two lines (theres a paste with these):<br />
<br />
root:$1$krszPxhk$xkCepSnz3mIikT3vCtJCt0:0:0::0:0:Charlie &:/root:/bin/csh<br />
user:$1$Mx9p5Npk$QdMU6c8YQqp2FW2M3irEh/:1001:1001::0:0:User &:/home/user:/bin/sh<br />
<br />
overwriting the lines they already have for "user" and "root" - then just tell them that both user and root have been reset to the default password of p455agfa.<br />
<br />
For linux, just passwd inside shell or <br />
vzctl set <veid> --userpasswd root:p455agfa –save<br />
<br />
Starting in 2009 we began giving out randomized passwords for FreeBSD and Linux as the default password. That is stored with each system in Mgmt. You should look for and reset the password to that password in the event of a reset and refer the customer to use their original password from their welcome email- this way we don’t have to send the password again via email (in clear text).<br />
<br />
<br />
== sendmail can’t be contacted from ext ip (only locally) ==<br />
<br />
By default redhat puts this line in sendmail.mc:<br />
<br />
DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')<br />
<br />
which makes it only answer on localhost. Comment it out like:<br />
<br />
dnl DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')<br />
<br />
and then rebuild sendmail.cf with:<br />
<br />
m4 /etc/mail/sendmail.mc > /etc/sendmail.cf<br />
<br />
== virt doesn’t properly let go of ve’s ip(s) when moved to another system ==<br />
<br />
On virtuozzo 2.6 systems, it's been observed that when moving ips from one virt to another that sometimes the routing table will not get updated to reflect the removal of the ip addresses.<br />
<br />
A recent example was a customer that was moving to a new ve on a new virt and the ip addresses were traded between the two ve's. After the trade the two systems were not able to talk to each other. When looking at the routing table for the old system all the ip addresses were still in the routing table as being local, like so:<br />
<br />
<pre>netstat -rn | grep 69.55.225.149<br />
69.55.225.149 0.0.0.0 255.255.255.255 UH 40 0 0 venet0</pre><br />
<br />
This was preventing traffic to the other system from being routed properly.<br />
The solution is to manually delete the route:<br />
<br />
route delete 69.55.225.149 gw 0.0.0.0<br />
<br />
Supposedly, this was fixed in 2.6.1<br />
<br />
== sshd on FreeBSD 6.2 segfaults ==<br />
<br />
First try to reinstall ssh<br />
<br />
<pre>cd /usr/src/secure<br />
cd lib/libssh<br />
make depend && make all install<br />
cd ../../usr.sbin/sshd<br />
make depend && make all install<br />
cd ../../usr.bin/ssh<br />
make depend && make all install</pre><br />
<br />
Failing that, find the library that’s messed up:<br />
<br />
<pre>ldd /usr/sbin/sshd<br />
libssh.so.3 => /usr/lib/libssh.so.3 (0x280a3000) <br />
libutil.so.5 => /lib/libutil.so.5 (0x280d8000) <br />
libz.so.3 => /lib/libz.so.3 (0x280e4000) <br />
libwrap.so.4 => /usr/lib/libwrap.so.4 (0x280f5000) <br />
libpam.so.3 => /usr/lib/libpam.so.3 (0x280fc000) <br />
libbsm.so.1 => /usr/lib/libbsm.so.1 (0x28103000) <br />
libgssapi.so.8 => /usr/lib/libgssapi.so.8 (0x28112000) <br />
libkrb5.so.8 => /usr/lib/libkrb5.so.8 (0x28120000) <br />
libasn1.so.8 => /usr/lib/libasn1.so.8 (0x28154000) <br />
libcom_err.so.3 => /usr/lib/libcom_err.so.3 (0x28175000) <br />
libroken.so.8 => /usr/lib/libroken.so.8 (0x28177000) <br />
libcrypto.so.4 => /lib/libcrypto.so.4 (0x28183000) <br />
libcrypt.so.3 => /lib/libcrypt.so.3 (0x28276000) <br />
libc.so.6 => /lib/libc.so.6 (0x2828e000) <br />
libmd.so.3 => /lib/libmd.so.3 (0x28373000)</pre><br />
<br />
md5 them and compare to other jail hosts or jails running on host<br />
<br />
for libcrypto reinstall:<br />
<pre>/usr/src/crypto<br />
make depend && make all install</pre><br />
<br />
= FreeBSD VPS =<br />
<br />
== Starting jails: Quad/Safe Files ==<br />
<br />
FreeBSD customer systems do not start up automatically at boot time. When one of our freebsd machines boots up, it boots up, and does nothing else. To start jails, we put the commands to start each jail into a shell script(s) and run the script(s). Jail startup is something that needs to be actively monitored, which is why we don’t just run the script automatically. More on monitoring later.<br />
<br />
NOTE: >=7.x we have moved to 1 quad file: <tt>quad1</tt>. Startups are not done by running each quad, but rather [[#startalljails|startalljails]] which relies on the contents of <tt>quad1</tt>. The specifics of this are lower in this article. What follows here applies for pre 7.x systems.<br />
<br />
There are eight files in <tt>/usr/local/jail/rc.d</tt>:<br />
<br />
<pre>jail3# ls /usr/local/jail/rc.d/<br />
quad1 quad2 quad3 quad4 safe1 safe2 safe3 safe4<br />
jail3#</pre><br />
<br />
four quad files and four safe files.<br />
<br />
Each file contains an even number of system startup blocks (total number of jails divided by 4)<br />
<br />
The reason for this is, if we make one large script to startup all the systems at boot time, it will take too long - the first system in the script will start up right after system boot, which is great, but the last system may not start for another 20 minutes.<br />
<br />
Since there is no way to parralelize this during the startup procedure, we simply open four terminals (in screen window 9) and run each script, one in each terminal. This way they all run simultaneously, and the very last system in each startup script gets started in 1/4th the time it would if there was one large file<br />
<br />
The files are generally organized so that quad/safe 1&2 have only jails from disk 1, and quad/safe 3&4 have jails from disk 2. This helps ensure that only 2 fscks on any disk are going on at once. Further, they are balanced so that all quad/safe’s finish executing around the same time. We do this by making sure each quad/safe has a similar number of jails and represents a similar number of inodes (see js).<br />
<br />
The other, very important reason we do it this way, and this is the reason there are quad files and safe files, is that in the event of a system crash, every single vn-backed filesystem that was mounted at the time of system crash needs to be fsck'd. However, fsck'ing takes time, so if we shut the system down gracefully, we don't want to fsck.<br />
<br />
Therefore, we have two sets of scripts - the four quad scripts are identical to the four safe scripts except for the fact that the quad scripts contain fsck commands for each filesystem.<br />
<br />
So, if you shut a system down gracefully, start four terminals and run safe1 in window one, and safe2 in window 2, and so on.<br />
<br />
If you crash, start four terminals (or go to screen window 9) and run quad1 in window one, and quad2 in window 2, and so on.<br />
<br />
Here is a snip of (a 4.x version) quad2 from jail17:<br />
<br />
<pre>vnconfig /dev/vn16 /mnt/data2/69.55.228.7-col00820<br />
fsck -y /dev/vn16<br />
mount /dev/vn16c /mnt/data2/69.55.228.7-col00820-DIR<br />
chmod 0666 /mnt/data2/69.55.228.7-col00820-DIR/dev/null<br />
jail /mnt/data2/69.55.228.7-col00820-DIR mail1.phimail.com 69.55.228.7 /bin/sh /etc/rc<br />
<br />
# moved to data2 col00368<br />
#vnconfig /dev/vn28 /mnt/data2/69.55.236.132-col00368<br />
#fsck -y /dev/vn28<br />
#mount /dev/vn28c /mnt/data2/69.55.236.132-col00368-DIR<br />
#chmod 0666 /mnt/data2/69.55.236.132-col00368-DIR/dev/null<br />
#jail /mnt/data2/69.55.236.132-col00368-DIR limehouse.org 69.55.236.132 /bin/sh /etc/rc<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
vnconfig /dev/vn22 /mnt/data2/69.55.228.13-col01063<br />
fsck -y /dev/vn22<br />
mount /dev/vn22c /mnt/data2/69.55.228.13-col01063-DIR<br />
chmod 0666 /mnt/data2/69.55.228.13-col01063-DIR/dev/null<br />
jail /mnt/data2/69.55.228.13-col01063-DIR www.widestream.com.au 69.55.228.13 /bin/sh /etc/rc<br />
<br />
# cancelled col00106<br />
#vnconfig /dev/vn15 /mnt/data2/69.55.238.5-col00106<br />
#fsck -y /dev/vn15<br />
#mount /dev/vn15c /mnt/data2/69.55.238.5-col00106-DIR<br />
#chmod 0666 /mnt/data2/69.55.238.5-col00106-DIR/dev/null<br />
#jail /mnt/data2/69.55.238.5-col00106-DIR mail.azebu.net 69.55.238.5 /bin/sh /etc/rc</pre><br />
<br />
As you can see, two of the systems specified are commented out - presumably those customers cancelled, or were moved to new servers.<br />
<br />
As you can see, the vnconfig line is the simpler command line, not the longer one that was used when it was first configured. As you can see, all that is done is, vnconfig the filesystem, then fsck it, then mount it. The fourth command is the `jail` command used to start the system – but that will be covered later.<br />
<br />
Here is the safe2 file from jail17:<br />
<br />
<pre>vnconfig /dev/vn16 /mnt/data2/69.55.228.7-col00820<br />
mount /dev/vn16c /mnt/data2/69.55.228.7-col00820-DIR<br />
chmod 0666 /mnt/data2/69.55.228.7-col00820-DIR/dev/null<br />
jail /mnt/data2/69.55.228.7-col00820-DIR mail1.phimail.com 69.55.228.7 /bin/sh /etc/rc<br />
<br />
# moved to data2 col00368<br />
#vnconfig /dev/vn28 /mnt/data2/69.55.236.132-col00368<br />
#mount /dev/vn28c /mnt/data2/69.55.236.132-col00368-DIR<br />
#chmod 0666 /mnt/data2/69.55.236.132-col00368-DIR/dev/null<br />
#jail /mnt/data2/69.55.236.132-col00368-DIR limehouse.org 69.55.236.132 /bin/sh /etc/rc<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
vnconfig /dev/vn22 /mnt/data2/69.55.228.13-col01063<br />
mount /dev/vn22c /mnt/data2/69.55.228.13-col01063-DIR<br />
chmod 0666 /mnt/data2/69.55.228.13-col01063-DIR/dev/null<br />
jail /mnt/data2/69.55.228.13-col01063-DIR www.widestream.com.au 69.55.228.13 /bin/sh /etc/rc<br />
<br />
# cancelled col00106<br />
#vnconfig /dev/vn15 /mnt/data2/69.55.238.5-col00106<br />
#mount /dev/vn15c /mnt/data2/69.55.238.5-col00106-DIR<br />
#chmod 0666 /mnt/data2/69.55.238.5-col00106-DIR/dev/null<br />
#jail /mnt/data2/69.55.238.5-col00106-DIR mail.azebu.net 69.55.238.5 /bin/sh /etc/rc</pre><br />
<br />
As you can see, it is exactly the same, but it does not have the fsck lines.<br />
<br />
Take a look at the last entry - note that the file is named:<br />
<br />
/mnt/data2/69.55.238.5-col00106<br />
<br />
and the mount point is named:<br />
<br />
/mnt/data2/69.55.238.5-col00106-DIR<br />
<br />
This is the general format on all the FreeBSD systems. The file is always named:<br />
<br />
IP-custnumber<br />
<br />
and the directory is named:<br />
<br />
IP-custnumber-DIR<br />
<br />
If you run safe when you need a fsck, the mount will fail and jail will fail:<br />
<br />
# mount /dev/vn1c /mnt/data2/jails/65.248.2.131-ns1.kozubik.com-DIR<br />
mount: /dev/vn1c: Operation not permitted<br />
<br />
No reboot needed, just run the quad script<br />
<br />
Starting with 6.x jails, we added block delimiters to the quad/safe files, the block looks like:<br />
<br />
<pre>echo '## begin ##: nuie.solaris.mu'<br />
fsck -y /dev/concat/v30v31a<br />
mount /dev/concat/v30v31a /mnt/data1/69.55.228.218-col01441-DIR<br />
mount_devfs devfs /mnt/data1/69.55.228.218-col01441-DIR/dev<br />
devfs -m /mnt/data1/69.55.228.218-col01441-DIR/dev rule -s 3 applyset<br />
jail /mnt/data1/69.55.228.218-col01441-DIR nuie.solaris.mu 69.55.228.218 /bin/sh /etc/rc<br />
echo '## end ##: nuie.solaris.mu'</pre><br />
<br />
These are more than just informative when running quad/safe’s, the echo lines MUST be present for certain tools to work properly. So it’s important that any updates to the hostname also be updated on the 2 echo lines. For example, if you try to startjail a jail with a hostname which is on the jail line but not the echo lines, the command will return with host not found.<br />
<br />
=== FreeBSD 7.x+ notes ===<br />
<br />
Starting with the release of FreeBSD 7.x, we are doing jail startups in a slightly different way. First, thereis only 1 file: <tt>/usr/local/jail/rc.d/quad1</tt><br />
There are no other quads or corresponding safe files. The reason for this is twofold, 1. We can pass –C to fsck which will tell is to skip the fsck if the fs is clean (no more need for safe files), 2. We have a new startup script which can be launched multiple times, running in parallel to start jails, where quad1 is the master jail file. <br />
Quad1 could still be run as a shell script, but it would take a very long time for it to run completely so it’s not advisable; or you should break it down into smaller chunks (like quad1, quad2, quad3, etc)<br />
<br />
Here is a snip of (a 7.x version) quad1 from jail2:<br />
<br />
<pre>echo '## begin ##: projects.tw.com'<br />
mdconfig -a -t vnode -f /mnt/data1/69.55.230.46-col01213 -u 50<br />
fsck -Cy /dev/md50c<br />
mount /dev/md50c /mnt/data1/69.55.230.46-col01213-DIR<br />
mount -t devfs devfs /mnt/data1/69.55.230.46-col01213-DIR/dev<br />
devfs -m /mnt/data1/69.55.230.46-col01213-DIR/dev rule -s 3 applyset<br />
jail /mnt/data1/69.55.230.46-col01213-DIR projects.tw.com 69.55.230.46 /bin/sh /etc/rc<br />
echo '## end ##: projects.tw.com'</pre><br />
<br />
Cancelled jails are no longer commented out and stored in quad1, rather they’re moved to <tt>/usr/local/jail/rc.d/deprecated</tt> <br />
<br />
To start these jails, start the 4 ssh sessions as you would for a normal crash and then instead of running quad1-4, instead run startalljails in each window. IMPORTANT- before running startalljails you should make sure you ran preboot once as it will clear out all the lockfiles and enable startalljails to work properly.<br />
<br />
== Problems with the quad/safe files ==<br />
<br />
When you run the quad/safe files, there are two problems that can occur - either a particular system will hang during initialization, OR a system will spit out output to the screen, impeding your ability to do anything. Or both.<br />
<br />
First off, when you start a jail, you see output like this:<br />
<br />
<pre>Skipping disk checks ...<br />
adjkerntz[25285]: sysctl(put_wallclock): Operation not permitted<br />
Doing initial network setup:.<br />
ifconfig: ioctl (SIOCDIFADDR): permission denied<br />
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384<br />
Additional routing options: TCP keepalive=YESsysctl:<br />
net.inet.tcp.always_keepalive: Operation not permitted.<br />
Routing daemons:.<br />
Additional daemons: syslogd.<br />
Doing additional network setup:.<br />
Starting final network daemons:.<br />
ELF ldconfig path: /usr/lib /usr/lib/compat /usr/X11R6/lib /usr/local/lib<br />
a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout /usr/X11R6/lib/aout<br />
Starting standard daemons: inetd cron sshd sendmail sendmail-clientmqueue.<br />
Initial rc.i386 initialization:.<br />
Configuring syscons: blanktime.<br />
Additional ABI support:.<br />
Local package initialization:.<br />
Additional TCP options:.</pre><br />
<br />
Now, let's look at this line, near the end:<br />
<br />
Local package initialization:.<br />
<br />
This is where a list of daemons that are set to start at boot time willshow up. You might see something like:<br />
<br />
Local package initialization: mysqld apache sendmail sendmail-clientmqueue<br />
<br />
Or something like this:<br />
<br />
Local package initialization: postgres postfix apache<br />
<br />
The problem is that many systems (about 4-5 per machine) will hang on that line. Basically it will get to some of the way through the total daemons to be started:<br />
<br />
Local package initialization: mysqld apache<br />
<br />
and will just sit there. Forever.<br />
<br />
Fortunately, pressing ctrl-c will break out of it. Not only will it break out of it, but it will also continue on that same line and start the other daemons:<br />
<br />
Local package initialization: mysqld apache ^c sendmail-clientmqueue<br />
<br />
and then continue on to finish the startup, and then move to the next system to be started.<br />
<br />
So what does this mean? It means that if a machine crashes, and you start four screen-windows to run four quads or four safes, you need to periodically cycle between them and see if any systems are stuck at that point, causing their quad/safe file to hang. A good rule of thumb is, if you see a system at that point in the startup, give it another 100 seconds - if it is still at the exact same spot, hit ctrl-c. Its also a good idea to go back into the quad file (just before the first command in the jail startup block) and note that this jail tends to need a control-c or more time as follows:<br />
<pre>echo '### NOTE ### slow sendmail'<br />
echo '### NOTE ###: ^C @ Starting sendmail.'</pre><br />
<br />
'''NEVER''' hit ctrl-c repeatedly if you don't get an immediate response - that will cause the following jail’s startup commands to be aborted.<br />
<br />
A second problem that can occur is that a jail - maybe the first one in that particular quad/safe, maybe the last one, or maybe one in the middle, will start spitting out status or error messages from one of its init scripts. This is not a problem - basically, hit enter a few times and see if you get a prompt - if you do get a prompt, that means that the quad/safe script has already completed. Therefore it is safe to log out (and log out of the user that you su'd from) and then log back in (if necessary).<br />
<br />
The tricky thing is, if a system in the middle starts flooding with messages, and you hit enter a few times and don't get a prompt. Are you not getting a prompt because some subsequent system is hanging at the initialization, as we discussede above ? Or are you not getting a prompt because that quad file is currently running an fsck ? Usually you can tell by scrolling back in screen’s history to see what it was doing before you started getting the messages.<br />
<br />
If you don’t get clues from history, you have to use your judgement - instead of giving it 100 seconds to respond, perhaps give it 2-3 mins ... if you still get no response (no prompt) when you hit enter, hit ctrl-c. However, be aware that you might still be hitting ctrl-c in the middle of an fsck. This means you will get an error like "filesystem still marked dirty" and then the vnconfig for it will fail and so will the jail command, and the next system in the quad file will then start starting up.<br />
<br />
If this happens, just wait until the end of all the quad files have finished, and start that system manually.<br />
<br />
If things really get weird, like a screen flooded with errors, and you can't get a prompt, and ctrl-c does nothing, then you need to just eventually (give it ten mins or so) just kill that window with ctrl-p, then k, and then log in again and manually check which systems are now running and which aren't, and manually start up any that are not.<br />
<br />
Don't EVER risk running a particular quad/safe file a second time.<br />
If the quad/safe script gets executed twice, reboot the machine immediately.<br />
<br />
So, for all the above reasons, anytime a machine crashes and you run all the quads or all the safes, '''always''' check every jail afterwards to make sure it is running - even if you have no hangs or complications at all.<br />
Run this command:<br />
<br />
<tt>[[#jailpsall|jailpsall]]</tt> <br />
<br />
Note: [[#postboot|postboot]] also populates ipfw counts, so it '''should not be run multiple times''', use <tt>jailpsall</tt> for subsequent extensive ps’ing<br />
<br />
And make sure they all show as running. If one does not show as running, check its /etc/rc.conf file first to see if maybe it is using a different hostname first before starting it manually.<br />
<br />
One thing we have implemented to alleviate these startup hangs and noisy jails, is to put jail start blocks that are slow or hangy at the bottom of the safe/quad file. Further, for each bad jail we note in each quad/safe just before the start block something like:<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
<br />
That way we’ll be prepared to ^C when we see that message appear during the quad/safe startup process. If you observe a new, undocumented hang, '''after''' the quad/safe has finished, place a line similar to the above in the quad file, move the jail start block to the end of the file, then run [[#buildsafe|buildsafe]]<br />
<br />
<br />
== Recovering from a crash (FreeBSD) ==<br />
<br />
=== Diagnose whether you have a crash ===<br />
The most important thing is to get the machine and all jails back up as soon as possible. Note the time, you’ll need to create a crash log entry (Mgmt. -> Reference -> CrashLog). The first thing to do is head over to the [[Screen#Screen_Organization|serial console screen]] and see if there’s any kernel error messages output. Try to copy any messages (or just a sample of repeating messages) you see into the notes section of the crash log. If there are no messages, the machine may just be really busy- wait a bit (5-10min) to see if it comes back. If it's still pinging, odds are its very busy. Note, if you see messages about swap space exhausted, the server is obviously out of memory, however it may recover briefly enough for you to get a jtop in to see who's lauched a ton of procs (most likely) and then issue a quick jailkill to get it back under control.<br />
<br />
If it doesn't come back, or the messages indicate a fatal error, you will need to proceed with a power cycle (ctrl+alt+del will not work).<br />
<br />
=== Power cycle the server ===<br />
If this machine is not a Dell 2950 with a [[DRAC/RMM#DRAC|DRAC card]] (i.e. if you can’t ssh into the DRAC card (as root, using the standard root pass) and issue <br />
racadm serveraction hardreset<br />
then you will need someone at the data center power the macine off, wait 30 sec, then turn it back on. Make sure to re-attach via console:<br />
tip jailX<br />
immediately after power down. <br />
<br />
=== (Re)attach to the console ===<br />
Stay on the console the entire time during boot. As the BIOS posts- look out for the RAID card output- does everything look healthy? The output may be scrambled, look for "DEGRADED" or "FAILED". Once the OS starts booting you will be disconnected (dropped back to the shell on the console server) a couple times during the boot up. The reason you want to quickly re-attach is two-fold: 1. If you don’t reattach quickly then you won’t get any console output, 2. you want to be attached before the server ''potentially'' starts (an extensive) fsck. If you attach after the fsck begins, you’ll have seen no indication it started an fsck and the server will appear frozen during startup- no output, no response. <br />
<br />
IMPORTANT NOTE: on some older FreeBSD systems, there will be no output to the video (KVM) console as it boots up. The console output is redirected to the serial port ... so if a jail crashes, and you attach a kvm, the output during the bootup procedure will not be shown on the screen. However, when the bootup is done, you will get a login prompt on the screen and will be able to log in as normal. <tt>/boot/loader.conf</tt> is where serial console redirect output lives, so comment that if you want to catch output on kvm.<br />
On newer systems it sends most output to both locations. <br />
<br />
=== Assess the heath of the server ===<br />
Once the server boots up fully, you should be able to ssh in. Look around- make sure all the mounts are there and reporting the correct size/usage (i.e. /mnt/data1 /mnt/data2 /mnt/data3 - look in /etc/fstab to determine which mount points should be there), check to see if RAID mirrors are healthy. See [[RAID_Cards#Common_CLI_commands_.28megacli.29|megacli]], [[#aaccheck|aaccheck]]<br />
<br />
Before you start the jails, you need to run [[#preboot|preboot]]. This will do some assurance checks to make sure things are prepped to start the jails. Any issues that come out of preboot need to be addressed before starting jails.<br />
<br />
=== Start jails ===<br />
[[#Starting_jails:_Quad.2FSafe_Files|More on starting jails]]<br />
Customer jails (the VPSs) do not start up automatically at boot time. When a FreeBSD machines boots up, it boots up, and does nothing else. To start jails, we put the commands to start each jail into a shell script(s) and run the script(s). Jail startup is something that needs to be actively monitored, which is why we don’t just run the script automatically. <br />
<br />
In order to start jails, we run the quad files: quad1 quad2 quad3 and quad4 (on new systems there is only quad1). If the machine was cleanly rebooted- which wouldn't be the case if this was a crash, you may run the safe files (safe1 safe2 safe3 safe4) in lieu of quads. <br />
<br />
Open up 4 logins to the server (use the windows in [[Screen#Screen_Organization|a9]])<br />
In each of the 4 windows you will:<br />
<br />
If there is a [[#startalljails|startalljails]] script (and only quad1), run that command in each of the 4 windows. It will parse through the quad1 file and start each jail. Follow the instructions [[#Problems_with_the_quad.2Fsafe_files|here]] for monitoring startup. Note that you can be a little more lenient with jails that take awhile to start- startalljails will work around the slow jails and start the rest. As long as there aren't 4 jails which are "hung" during startup, the rest will get started eventually.<br />
-or-<br />
If there is no startalljails script, there will be multiple quad files. In each of the 4 windows, start each of the quads. i.e. start quad1 in window1, quad2 in window2 and so on. DO NOT start any quad twice. It will crash the server. If you accidentally do this, just jailkill all the jails which are in the quad and run the quad again. Follow the instructions here for monitoring quad startup.<br />
<br />
Note the time the last jail boots- this is what you will enter in the crash log.<br />
<br />
Save the crash log.<br />
<br />
=== Check to make sure all jails have started ===<br />
There's a simple script which will make sure all jails have started, and enter the ipfw counter rules: [[#postboot|postboot]] <br />
Run postboot, which will do a jailps on each jail it finds (excluding commented out jails) in the quad file(s). We're looking for 2 things:<br />
# systems spawning out of control or too many procs<br />
# jails which haven't started<br />
On 7.x and newer systems it will print out the problems (which jails haven't started) at the conclusion of postboot. <br />
On older systems you will need to watch closely to see if/when there's a problem, namely:<br />
<br />
[hostname] doesnt exist on this server<br />
<br />
When you get this message, it means one of 2 things:<br />
1. the jail really didn't start:<br />
When a jail doesn't start it usually boils down to a problem in the quad file. Perhaps the path name is wrong (data1 vs data2) or the name of the vn/mdfile is wrong. Once this is corrected, you will need to run the commands from the quad file manually, or you may use <tt>startjail <hostname></tt><br />
<br />
2. the customer has changed their hostname (and not told us) so their jail ''is'' running, just under a different hostname:<br />
On systems with jls, this is easy to rectify. First, get the customer info: <tt>g <hostname></tt><br />
Then look for the customer in jls: <tt>jls | grep <col0XXXX></tt><br />
From there you will see their new hostname- you should update that hostname in the quad file: don't forget to edit it on the <tt>## begin ##</tt> and <tt>## end ##</tt> lines, and in mgmt. <br />
On older systems without jls, this will be harder, you will need to look further to see their hostname- perhaps its in their /etc/rc.conf<br />
<br />
<br />
Once all jails are started, do some spot checks- try to ssh or browse to some customers, just to make sure things are really ok.<br />
<br />
== Adding disk to a 7.x/8.x jail ==<br />
or<br />
== Moving customer to a different drive (md) ==<br />
<br />
NOTE: this doesn’t apply to mx2 which uses gvinum. Use same procedure as 6.x<br />
NOTE: if you unmount before mdconfig, re-mdconfig (attach) then unmount then mdconfig -u again <br />
<br />
<br />
(parts to change/customize are <tt><span style="color:red">red</span></tt>)<br />
<br />
<br />
If someone wants more disk space, there’s a paste for it, send it to them (explains about downtime, etc).<br />
<br />
1. Figure out the space avail from <tt>js</tt>. Ideally, you want to put the customers new space on a different partition (and create the new md on the new partition). <br />
<br />
<br />
2. make a mental note of how much space they're currently using<br />
<br />
<br />
3. <tt>jailkill <hostname></tt> <br />
<br />
<br />
4. Umount it (including their devfs) but leave the md config’d (so if you use stopjail, you will have to re-mdconfig it)<br />
<br />
<br />
5. <tt>g <customerID></tt> to get the info (IP/cust#) needed to feed the new mdfile and mount name, and to see the current md device. <br />
<br />
<br />
6a. When there's enough room to place new system on an alternate, or the same drive:<br />
USE CAUTION not to overwrite (touch, mdconfig) existing md!!<br><br />
<tt>touch /mnt/data<span style="color:red">3/69.55.234.66-col01334</span><br><br />
mdconfig -a -t vnode -s 10g -f /mnt/data3/69.55.234.66-col01334 -u 97<br><br />
newfs /dev/md97</tt><br />
<br />
<br />
Optional- if new space is on a different drive, move the mount point directory AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
<tt>mount /dev/md<span style="color:red">97</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<br />
confirm you are mounted to /dev/md<span style="color:red">97</span> and space is correct:<br><br />
<tt>df .</tt> <br />
<br />
<br />
do the dump and pipe directly to restore:<br><br />
<tt>dump -0a -f - /dev/md<span style="color:red">1</span> | restore -r -f - <br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use <tt>df</tt> to confirm the restored data size matches the original usage figure<br />
<br />
<br />
md-unconfig old system:<br><br />
<tt>mdconfig -d -u <span style="color:red">1</span></tt><br />
<br />
<br />
archive old mdfile. <br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.237.26-col00241</span> /mnt/data<span style="color:red">1/old-col00241-mdfile-noarchive-20091211</span></tt><br />
<br />
<br />
edit quad (vq1) to point to new (/mnt/data<span style="color:red">3</span>) location AND new md number (md<span style="color:red">97</span>)<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
6b. When there's not enough room on an alternate partition, or on the same drive...but there is enough room if you were to remove the existing customer's space:<br />
<br />
<br />
mount backup nfs mounts:<br><br />
<tt>mbm</tt><br> <br />
(run <tt>df</tt> to confirm backup mounts are mounted)<br />
<br />
<br />
dump the customer to backup2 or backup1<br><br />
<tt>dump -0a -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> /dev/md<span style="color:red">1</span></tt><br><br />
(when complete WITHOUT errors, <tt>du</tt> the dump file to confirm it matches size, roughly, with usage)<br />
<br />
<br />
unconfigure and remove old mdfile<br><br />
<tt>mdconfig -d -u <span style="color:red">1</span><br><br />
rm /mnt/data<span style="color:red">1/69.55.237.26-col00241</span></tt><br><br />
(there should now be enough space to recreate your bigger system. If not, run sync a couple times)<br />
<br />
<br />
create the new system (ok to reuse old mdfile and md#):<br><br />
<tt>touch /mnt/data<span style="color:red">1/69.55.234.66-col01334</span><br><br />
mdconfig -a -t vnode -s <span style="color:red">10</span>g -f /mnt/data<span style="color:red">1/69.55.234.66-col01334</span> -u <span style="color:red">1</span><br><br />
newfs /dev/md<span style="color:red">1</span><br><br />
mount /dev/md<span style="color:red">1</span> /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> <br><br />
cd /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<br />
confirm you are mounted to /dev/md<span style="color:red">1</span> and space is correct:<br><br />
<tt>df .</tt> <br />
<br />
<br />
do the restore from the dumpfile on the backup server:<br><br />
<tt>restore -r -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> .<br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
umount nfs:<br><br />
<tt>mbu</tt><br />
<br />
<br />
If md# changed (or mount point), edit quad (<tt>vq1</tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new md number (md<span style="color:red">1</span>)<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
7. update disk (and dir if applicable) in mgmt screen<br />
<br />
<br />
8. update backup list AND move backups, if applicatble<br />
<br />
Ex: <tt>mvbackups <span style="color:red">69.55.237.26-col00241</span> jail<span style="color:red">9</span> data<span style="color:red">3</span></tt><br />
<br />
<br />
9. Optional: archive old mdfile<br />
<br />
<tt>mbm<br><br />
gzip -c old-col01588-mdfile-noarchive-20120329 > /deprecated/old-col01588-mdfile-noarchive-20120329.gz<br><br />
mbu<br><br />
rm old-col01588-mdfile-noarchive-20120329</tt><br />
<br />
== Adding disk to a 6.x jail (gvinum/gconcat) ==<br />
or<br />
== Moving customer to a different drive (gvinum/gconcat) ==<br />
<br />
(parts to change are <span style="color:red">highlighted</span>)<br />
<br />
<br />
If someone wants more disk space, there’s a paste for it, send it to them (explains about downtime, etc).<br />
<br />
<br />
1. Figure out the space avail from [[#js|js]]. Ideally, you want to put the customers new space on a different partition (and create the new md on the new partition). <br />
<br />
<br />
2. make a mental note of how much space they're currently using<br />
<br />
<br />
3. <tt>[[#stopjail|stopjail]] <hostname></tt> <br />
<br />
<br />
4. <tt>[[#g|g]] <customerID></tt> to get the info (IP/cust#) needed to feed the new mount name and existing volume/device. <br />
<br />
<br />
5a. When there's enough room to place new system on an alternate, or the same drive (using only UNUSED - including if it's in use by the system in question - gvinum volumes):<br />
<br />
<br />
Configure the new device:<br><br />
A. for a 2G system (single gvinum volume):<br><br />
<tt>bsdlabel -r -w /dev/gvinum/v<span style="color:red">123</span><br><br />
newfs /dev/gvinum/v<span style="color:red">123</span>a</tt><br><br />
<br />
-or- <br />
<br />
<br />
B. for a >2G system (create a gconcat volume):<br><br />
try to grab a contiguous block of gvinum volumes. gconcat volumes MAY NOT span drives (i.e. you cannot use a gvinum volume from data3 and a volume from data2 in the same gconcat volume). <br />
<br />
<tt>gconcat label <span style="color:red">v82-v84 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84</span><br><br />
bsdlabel -r -w /dev/concat/<span style="color:red">v82-v84</span><br><br />
newfs /dev/concat/<span style="color:red">v82-v84</span>a</tt><br />
<br />
<br />
Other valid gconcat examples:<br><br />
<tt>gconcat label v82-v84v109v112 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84 /dev/gvinum/v109 /dev/gvinum/v112<br><br />
gconcat label v82v83 /dev/gvinum/v82 /dev/gvinum/v83</tt> <br><br />
Note, long names will truncate: v144v145v148-v115 will truncate to v144v145v148-v1 (so you will refer to it as v144v145v148-v1 thereafter)<br />
<br />
<br />
Optional- if new volume is on a different drive, move the mount point directory (get the drive from js output) AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
<br />
confirm you are mounted to the device (<tt>/dev/gvinum/v<span style="color:red">123</span>a</tt> OR <tt>/dev/concat/<span style="color:red">v82-v84</span></tt>) and space is correct:<br><br />
A. <tt>mount /dev/gvinum/v<span style="color:red">123</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br><br />
-or-<br><br />
B. <tt>mount /dev/concat/<span style="color:red">v82-v84</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<tt>cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
df .</tt> <br />
<br />
<br />
do the dump and pipe directly to restore:<br><br />
<tt>dump -0a -f - /dev/gvinum/v<span style="color:red">1</span> | restore -r -f - <br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
edit quad (<tt>vq<span style="color:red">1</span></tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new volume (<tt>/dev/gvinum/v<span style="color:red">123</span>a</tt> or <tt>/dev/concat/<span style="color:red">v82-v84</span>a</tt>) , run <tt>buildsafe</tt><br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
5b. When there's not enough room on an alternate partition, or on the same drive...but there is enough room if you were to remove the existing customer's space (i.e. if you want/need to reuse the existing gvinum volumes and add on more):<br />
<br />
<br />
mount backup nfs mounts:<br><br />
<tt>mbm</tt> <br><br />
(run df to confirm backup mounts are mounted)<br />
<br />
<br />
dump the customer to backup2 or backup1<br><br />
<tt>dump -0a -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> /dev/<span style="color:red">gconcat/v106-v107</span></tt><br><br />
(when complete WITHOUT errors, du the dump file to confirm it matches size, roughly, with usage)<br />
<br />
<br />
unconfigure the old gconcat volume<br><br />
list member gvinum volumes:<br />
<br />
<tt>gconcat list <span style="color:red">v106v107</span></tt><br />
<br />
<br />
Output will resemble:<br><br />
<pre>Geom name: v106v107<br />
State: UP<br />
Status: Total=2, Online=2<br />
Type: AUTOMATIC<br />
ID: 3530663882<br />
Providers:<br />
1. Name: concat/v106v107<br />
Mediasize: 4294966272 (4.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e2<br />
Consumers:<br />
1. Name: gvinum/sd/v106.p0.s0<br />
Mediasize: 2147483648 (2.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e3<br />
Start: 0<br />
End: 2147483136<br />
2. Name: gvinum/sd/v107.p0.s0<br />
Mediasize: 2147483648 (2.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e3<br />
Start: 2147483136<br />
End: 4294966272</pre><br />
<br />
stop volume and clear members<br><br />
<tt>gconcat stop <span style="color:red">v106v107</span><br><br />
gconcat clear <span style="color:red">gvinum/sd/v106.p0.s0 gvinum/sd/v107.p0.s0</span></tt><br />
<br />
<br />
create new device- and its ok to reuse old/former members<br><br />
try to grab a contiguous block of gvinum volumes. gconcat volumes MAY NOT span drives (i.e. you cannot use a gvinum volume from data3 and a volume from data2 in the same gconcat volume). <br><br />
<tt>gconcat label <span style="color:red">v82-v84v106v107 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84 /dev/gvinum/v106 /dev/gvinum/v107</span> <br><br />
bsdlabel -r -w /dev/concat/<span style="color:red">v82-v84v106v107</span><br><br />
newfs /dev/concat/<span style="color:red">v82-v84v106v107</span>a</tt><br />
<br />
<br />
Optional- if new volume is on a different drive, move the mount point directory (get the drive from js output) AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
confirm you are mounted to the device (/dev/concat/<span style="color:red">v82-v84v106v107</span>) and space is correct:<br><br />
<tt>mount /dev/concat/<span style="color:red">v82-v84v106v107</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<tt>cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
df .</tt> <br />
<br />
do the restore from the dumpfile on the backup server:<br><br />
<tt>restore -r -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> .<br><br />
rm restoresymtable</tt><br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
edit quad (<tt>vq<span style="color:red">1</span></tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new volume (<tt>/dev/concat/<span style="color:red">v82-v84v106v107</span>a</tt>), run buildsafe<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
TODO: clean up/clear old gvin/gconcat vol<br />
<br />
<br />
6. update disk (and dir if applicable) in mgmt screen<br />
<br />
<br />
7. update backup list AND move backups, if applicatble<br />
<br />
<br />
Ex: <tt>mvbackups <span style="color:red">69.55.237.26-col00241</span> jail<span style="color:red">9</span> data<span style="color:red">3</span></tt><br />
<br />
<br />
<br />
DEPRECATED - steps to tack on a new gvin to existing gconcat- leads to corrupted fs<br />
bsdlabel -e /dev/concat/v82-v84<br />
<br />
To figure out new size of the c partition, multiply 4194304 by the # of 2G gvinum volumes and subtract the # of 2G volumes:<br />
10G: 4194304 * 5 – 5 = 20971515<br />
8G: 4194304 * 4 – 4 = 16777212<br />
6G: 4194304 * 3 – 3 = 12582909<br />
4G: 4194304 * 2 – 2 = 8388606<br />
<br />
To figure out the new size of the a partition, subtract 16 from the c partition:<br />
10G: 20971515 – 16 = 20971499<br />
8G: 16777212 – 16 = 16777196<br />
6G: 12582909 – 16 = 12582893<br />
4G: 8388606 – 16 = 8388590<br />
<br />
Orig:<br />
8 partitions:<br />
# size offset fstype [fsize bsize bps/cpg]<br />
a: 8388590 16 4.2BSD 2048 16384 28552<br />
c: 8388606 0 unused 0 0 # "raw" part, don't edit<br />
<br />
New:<br />
8 partitions:<br />
# size offset fstype [fsize bsize bps/cpg]<br />
a: 12582893 16 4.2BSD 2048 16384 28552<br />
c: 12582909 0 unused 0 0 # "raw" part, don't edit<br />
<br />
sync; sync<br />
<br />
growfs /dev/concat/v82-v84a<br />
<br />
fsck –fy /dev/concat/v82-v84a<br />
<br />
sync<br />
<br />
fsck –fy /dev/concat/v82-v84a<br />
<br />
(keep running fsck’s till NO errors)<br />
<br />
== Problems un-mounting - and with mount_null’s ==<br />
<br />
If you cannot unmount a filesystem, beacuse it says the filesystem is busy, it is because of three things:<br />
<br />
a) the jail is still running<br />
<br />
b) you are actually in that directory, even though the jail is stopped<br />
<br />
c) there are still dev, null_mount or linprocfs mount points mounted inside that directory.<br />
<br />
d) when trying to umount null_mounts that are really long and you get an error like “No such file or directory”, it’s an OS bug where the dir is truncated. No known fix<br />
<br />
e) there are still files open somewhere inside the dir. Use <tt>fstat | grep <cid></tt> to find the process that has files open<br />
<br />
f) Starting with 6.x, the jail mechanism does a poor job of keeping track of processes running in a jail and if it thinks there are still procs running, it will refuse to umount the disk. If this is happening you should see a low number in the #REF column when you run jls. In this case you ''can'' safely <tt>umount –f</tt> the mount. <br />
<br />
Please note -if you forcibly unmount a (4.x) filesystem that has null_mounts<br />
still mounted in it, the system '''will crash''' within 10-15 mins.<br />
<br />
== Misc jail Items ==<br />
<br />
We are overselling hard drive space on jail2, jail8, jail9, a couple jails on jail17, jail4, jail12 and jail18.<br />
Even though the vn file shows 4G size, it doesn’t actually occupy that amount of space on the disk. So be careful not to fill up drives where we’re overselling – use oversellcheck to confirm you’re not oversold by more than 10G.<br />
There are other truncated jails, they are generally noted in a the file on the root system: /root/truncated<br />
<br />
The act of moving a truncated vn to another system un-does the truncating- the truncated vn is filled with 0’s and it occupies physical disk space for which it’s configured. So, you should use dumpremote to preserve the truncation.<br />
<br />
= FreeBSD VPS Management Tools =<br />
<br />
These files are located in /usr/local/jail/rc.d and /usr/local/jail/bin<br />
<br />
== jailmake ==<br />
<br />
Applies to 7.x+ <br />
On older systems syntax differs, run jailmake once to see.<br />
<br />
Note: this procedure differs on mx2 which is 7.x but still uses gvinum<br />
<br />
# run js to figure out which md’s are in use, which disk has enough space, IP to put it on<br />
# use col00xxx for both hostnames if they don’t give you a hostname<br />
# copy over dir, ip and password to pending customer screen<br />
<br />
<tt>Usage: jailmake IP[,IP] CID disk[1|2|3] md# hostname shorthost ipfw# email [size in GB]</tt><br />
<br />
Ex: <br />
<br />
Jail2# jailmake 69.55.234.66 col01334 3 97 vps.bsd.it vps 1334 fb@bsd.it<br />
<br />
== jailps ==<br />
jailps [hostname]<br />
DEPRECATED FOR jps: displays processes belonging to/running inside a jail. The command<br />
takes one (optional) argument – the hostname of the jail you wish to query. If you don’t <br />
supply an argument, all processes on the machine are listed and grouped by jail. <br />
<br />
== jps ==<br />
jps [hostname]<br />
displays processes belonging to/running inside a jail. The command<br />
takes one (optional) argument – the hostname or ID of the jail you wish to query. <br />
<br />
== jailkill ==<br />
jailkill <hostname><br />
stops all process running in a jail.<br />
<br />
You can also run:<br />
jailkill <JID><br />
<br />
=== problems ===<br />
Occasionally you will hit an issue where jail will not kill off:<br />
<br />
<pre>jail9# jailkill www.domain.com<br />
www.domain.com .. killed: none<br />
jail9#</pre><br />
Because no processes are running under that hostname. You cannot use jailps.pl either:<br />
<br />
<pre>jail9# jailps www.domain.com<br />
www.domain.com doesn’t exist on this server<br />
jail9#</pre><br />
<br />
The reasons for this are usually:<br />
* the jail is no longer running<br />
<br />
* the jail's hostname has changed<br />
In this case, <br />
<br />
>=6.x: run a <tt>jls|grep <jail's IP></tt> to find the correct hostname, then update the quad file, then kill the jail.<br />
<br />
<6.x: the first step is to cat their /etc/rc.conf file to see if you can tell what they set the new hostname to. This very often works. For example:<br />
<br />
cat /mnt/data2/198.78.65.136-col00261-DIR/etc/rc.conf<br />
<br />
But maybe they set the hostname with the hostname command, and the original hostname is still in /etc/rc.conf.<br />
<br />
The welcome email clearly states that they should tell us if they change their hostname, so there is no problem in just emailing them and asking them what they set the new hostname to.<br />
<br />
Once you know the new hostname OR if a customer simply emails to inform you that they have set the hostname to something different, you need to edit the quad and safe files that their system is in to input the new hostname.<br />
<br />
However, if push comes to shove and you cannot find out the hostname from them or from their system, then you need to start doing some detective work.<br />
<br />
The easiest thing to do is run jailps looking for a hostname similar to their original hostname. Or you could get into the /bin/sh shell by running:<br />
<br />
/bin/sh<br />
<br />
and then looking at every hostname of every process:<br />
<br />
for f in `ls /proc` ; do cat /proc/$f/status ; done<br />
<br />
and scanning for a hostname that is either similar to their original hostname, or that you don't see in any of the quad safe files.<br />
<br />
This is very brute force though, and it is possible that catting every file in /proc is dangerous - I don't recommend it. A better thing would be to identify any processes that you know belong to this system – perhaps the reason you are trying to find this system is because they are running something bad - and just catting the status from only that PID.<br />
<br />
Somewhere there’s a jail where there may be 2 systems named www. Look at /etc/rc.conf and make sure they’re both really www. If they are, jailkill www, jailps www to make sure not running. Then immediately restart the other one, as the fqdn (as found from a rev nslookup)<br />
<br />
* on >=6.x the hostname may not yet be hashed:<br />
<pre>jail9 /# jls<br />
JID Hostname Path IP Address(es)<br />
1 bitnet.dgate.org /mnt/data1/69.55.232.50-col02094-DIR 69.55.232.50<br />
2 ns3.hctc.net /mnt/data1/69.55.234.52-col01925-DIR 69.55.234.52<br />
3 bsd1 /mnt/data1/69.55.232.44-col00155-DIR 69.55.232.44<br />
4 let2.bbag.org /mnt/data1/69.55.230.92-col00202-DIR 69.55.230.92<br />
5 post.org /mnt/data2/69.55.232.51-col02095-DIR 69.55.232.51 ...<br />
6 ns2 /mnt/data1/69.55.232.47-col01506-DIR 69.55.232.47 ...<br />
7 arlen.server.net /mnt/data1/69.55.232.52-col01171-DIR 69.55.232.52<br />
8 deskfood.com /mnt/data1/69.55.232.71-col00419-DIR 69.55.232.71<br />
9 mirage.confluentforms.com /mnt/data1/69.55.232.54-col02105-DIR 69.55.232.54 ...<br />
10 beachmember.com /mnt/data1/69.55.232.59-col02107-DIR 69.55.232.59<br />
11 www.agottem.com /mnt/data1/69.55.232.60-col02109-DIR 69.55.232.60<br />
12 sdhobbit.myglance.org /mnt/data1/69.55.236.82-col01708-DIR 69.55.236.82<br />
13 ns1.jnielsen.net /mnt/data1/69.55.234.48-col00204-DIR 69.55.234.48 ...<br />
14 ymt.rollingegg.net /mnt/data2/69.55.236.71-col01678-DIR 69.55.236.71<br />
15 verse.unixlore.net /mnt/data1/69.55.232.58-col02131-DIR 69.55.232.58<br />
16 smcc-mail.org /mnt/data2/69.55.232.68-col02144-DIR 69.55.232.68<br />
17 kasoutsuki.w4jdh.net /mnt/data2/69.55.232.46-col02147-DIR 69.55.232.46<br />
18 dili.thium.net /mnt/data2/69.55.232.80-col01901-DIR 69.55.232.80<br />
20 www.tekmarsis.com /mnt/data2/69.55.232.66-col02155-DIR 69.55.232.66<br />
21 vps.yoxel.net /mnt/data2/69.55.236.67-col01673-DIR 69.55.236.67<br />
22 smitty.twitalertz.com /mnt/data2/69.55.232.84-col02153-DIR 69.55.232.84<br />
23 deliver4.klatha.com /mnt/data2/69.55.232.67-col02160-DIR 69.55.232.67<br />
24 nideffer.com /mnt/data2/69.55.232.65-col00412-DIR 69.55.232.65<br />
25 usa.hanyuan.com /mnt/data2/69.55.232.57-col02163-DIR 69.55.232.57<br />
26 daifuku.ppbh.com /mnt/data2/69.55.236.91-col01720-DIR 69.55.236.91<br />
27 collins.greencape.net /mnt/data2/69.55.232.83-col01294-DIR 69.55.232.83<br />
28 ragebox.com /mnt/data2/69.55.230.104-col01278-DIR 69.55.230.104<br />
29 outside.mt.net /mnt/data2/69.55.232.72-col02166-DIR 69.55.232.72<br />
30 vps.payneful.ca /mnt/data2/69.55.234.98-col01999-DIR 69.55.234.98<br />
31 higgins /mnt/data2/69.55.232.87-col02165-DIR 69.55.232.87 ...<br />
32 ozymandius /mnt/data2/69.55.228.96-col01233-DIR 69.55.228.96<br />
33 trusted.realtors.org /mnt/data2/69.55.238.72-col02170-DIR 69.55.238.72<br />
34 jc1.flanderous.com /mnt/data2/69.55.239.22-col01504-DIR 69.55.239.22<br />
36 guppylog.com /mnt/data2/69.55.238.73-col00036-DIR 69.55.238.73<br />
40 haliohost.com /mnt/data2/69.55.234.41-col01916-DIR 69.55.234.41 ...<br />
41 satyr.jorge.cc /mnt/data1/69.55.232.70-col01963-DIR 69.55.232.70<br />
jail9 /# jailkill satyr.jorge.cc<br />
ERROR: jail_: jail "satyr,jorge,cc" not found<br />
</pre><br />
<br />
Note how it's saying <tt>satyr,jorge,cc</tt> is not found, and not <tt>satyr.jorge.cc</tt>. <br />
<br />
The jail subsystem tracks things using comma-delimited hostnames. That is created every few hours:<br />
<br />
jail9 /# crontab -l<br />
0 0,6,12,18 * * * /usr/local/jail/bin/sync_jail_names<br />
<br />
So if we run this manually:<br />
jail9 /# /usr/local/jail/bin/sync_jail_names<br />
<br />
Then kill the jail:<br />
jail9 /# jailkill satyr.jorge.cc<br />
successfully killed: satyr,jorge,cc</pre><br />
<br />
It worked.<br />
<br />
<br />
If you ever see this when trying to kill a jail:<br />
<br />
<pre># jailkill e-scribe.com<br />
killing JID: 6 hostname: e-scribe.com<br />
3 procs running<br />
3 procs running<br />
3 procs running<br />
3 procs running<br />
...</pre><br />
<br />
<tt>[[#jailkill|jailkill]]</tt> probably got lost trying to kill off the jail. Just ctrl-c the jailkill process, then run a jailps on the hostname, and <tt>kill -9</tt> any process which is still running. Keep running jailps and <tt>kill -9</tt> till all processes are gone.<br />
<br />
== jailpsall ==<br />
jailpsall<br />
will run a jailps on all jails configured in the quad files (this is different from<br />
jailps with no arguments as it won’t help you find a “hidden” system)<br />
<br />
== jailpsw ==<br />
jailpsw<br />
will run a jailps with an extra -w to provide wider output<br />
<br />
== jt (>=7.x) ==<br />
jt<br />
displays the top 20 processes on the server (the top 20 processes from top) and <br />
which jail owns them. This is very helpful for determining who is doing what when<br />
the server is very busy.<br />
<br />
== jtop (>=7.x) ==<br />
jtop<br />
a wrapper for top displaying processes on the server and which jail owns them. Constantly updates, like top. <br />
<br />
== jtop (<7.x) ==<br />
jtop<br />
displays the top 20 processes on the server (the top 20 processes from top) and <br />
which jail owns them. This is very helpful for determining who is doing what when<br />
the server is very busy.<br />
<br />
== stopjail ==<br />
stopjail <hostname> [1]<br />
this will jailkill, umount and vnconfig –u a jail. If passed an optional 2nd<br />
argument, it will not exit before umounting and un-vnconfig’ing in the event<br />
jailkill returns no processes killed. This is useful if you just want to umount<br />
and vnconfig –u a jail you’ve already killed. It is intelligent in that it won’t <br />
try to umount or vnconfig –u if it’s not necessary.<br />
<br />
== startjail ==<br />
startjail <hostname><br />
this will start vnconfig, mount (including linprocfs and null-mounts), and start a jail.<br />
Essentially, it reads the jail’s relevant block from the right quad file and executes it.<br />
It is intelligent in that it won’t try to mount or vnconfig if it’s not necessary.<br />
<br />
== jpid ==<br />
jpid <pid><br />
displays information about a process – including which jail owns it.<br />
It’s the equivalent of running cat /proc/<pid>/status<br />
<br />
== canceljail ==<br />
canceljail <hostname> [1]<br />
this will stop a jail (the equivalent of stopjail), check for backups (offer to remove them <br />
from the backup server and the backup.config), rename the vnfile, remove the dir, and <br />
edit quad/safe. If passed an optional 2nd argument, it will not exit upon failing to kill<br />
and processes owned by the jail. This is useful if you just want to cancel a jail which <br />
is already stopped.<br />
<br />
== jls ==<br />
jls [-v]<br />
Lists all jails running:<br />
<pre>JID #REF IP Address Hostname Path<br />
101 135 69.55.224.148 mail.pc9.org /mnt/data2/69.55.224.148-col01034-DIR</pre><br />
<br />
#REF is the number of references or procs(?) running<br />
<br />
Running with -v will give you all IPs assigned to each jail (7.2 up)<br />
<pre>JID #REF Hostname Path IP Address(es)<br />
101 139 mail.pc9.org /mnt/data2/69.55.224.148-col01034-DIR 69.55.224.14869.55.234.85</pre><br />
<br />
== startalljails ==<br />
startalljails<br />
7.2+ only. This will parse through quad1 and start all jails. It utilizes lockfiles so it won’t try to start a jail more than once- therefore multiple instances can be running in parallel without fear of starting a jail twice. If a jail startup gets stuck, you can ^C without fear of killing the script. IMPORTANT- before running startalljails you should make sure you ran preboot once as it will clear out all the lockfiles and enable startalljails to work properly.<br />
<br />
== aaccheck.sh ==<br />
aaccheck.sh<br />
displayes the output of container list and task list from aaccli<br />
<br />
== backup ==<br />
backup<br />
backup script called nightly to update jail scripts and do customer backups<br />
<br />
== buildsafe ==<br />
buildsafe<br />
creates safe files based on quads (automatically removing the fsck’s). This will destructively overwrite safe files<br />
<br />
== checkload.pl ==<br />
checkload.pl<br />
this was intended to be setup as a cronjob to watch processes on a jail when the load <br />
rises above a certain level. Not currently in use.<br />
<br />
== checkprio.pl ==<br />
checkprio.pl<br />
will look for any process (other than the current shell’s csh, sh, sshd procs) with a non-normal priority and normalize it<br />
<br />
== diskusagemon == <br />
diskusagemon <mount point> <1k blocks><br />
watches a mount point’s disk use, when it reaches the level specified in the 2nd argument,<br />
it exits. This is useful when doing a restore and you want to be paged as it’s nearing completion.<br />
Best used as: <tt>diskusagemon /asd/asd 1234; pagexxx</tt><br />
<br />
== dumprestore ==<br />
dumprestore <dumpfile><br />
this is a perl expect script which automatically enters ‘1’ and ‘y’. It seems to cause restore to fail<br />
to set owner permissions on large restores.<br />
<br />
== g ==<br />
g <search><br />
greps the quad/safe files for the given search parameter<br />
<br />
== gather.pl ==<br />
gather.pl<br />
gathers up data about jails configured and writes to a file. Used for audits against the db<br />
<br />
== gb ==<br />
gb <search><br />
greps backup.config for the given search parameter<br />
<br />
== gbg ==<br />
gbg <search><br />
greps backup.config for the given search parameter and presents just the directories (for clean pasting)<br />
<br />
== ipfwbackup ==<br />
ipfwbackup<br />
writes ipfw traffic count data to a logfile<br />
<br />
== ipfwreset ==<br />
ipfwreset<br />
writes ipfw traffic count data to a logfile and resets counters to 0<br />
<br />
== js ==<br />
js<br />
output varies by OS version, but generally provides information about the base jail:<br />
- which vn’s are in use<br />
- disk usage<br />
- info about the contents of quads<br />
- the # of inodes represented by the jails contained in the group (133.2 in the example below), and how many jails per data mount, as well as subtotals<br />
- ips bound to the base machine but not in use by a jail<br />
- free gvinum volumes, or unused vn’s or used md’s<br />
<br />
<pre>/usr/local/jail/rc.d/quad1:<br />
/mnt/data1 133.2 (1)<br />
/mnt/data2 1040.5 (7)<br />
total 1173.7 (8)<br />
/usr/local/jail/rc.d/quad2:<br />
/mnt/data1 983.4 (6)<br />
total 983.4 (6)<br />
/usr/local/jail/rc.d/quad3:<br />
/mnt/data1 693.4 (4)<br />
/mnt/data2 371.6 (3)<br />
total 1065 (7)<br />
/usr/local/jail/rc.d/quad4:<br />
/mnt/data1 466.6 (3)<br />
/mnt/data2 882.2 (5)<br />
total 1348.8 (8)<br />
/mnt/data1: 2276.6 (14)<br />
/mnt/data2: 2294.3 (15)<br />
<br />
Available IPs:<br />
69.55.230.11 69.55.230.13 69.55.228.200<br />
<br />
Available volumes:<br />
v78 /mnt/data2 2G<br />
v79 /mnt/data2 2G<br />
v80 /mnt/data2 2G</pre> <br />
<br />
== load.pl ==<br />
load.pl<br />
feeds info to load mrtg - executed by inetd.<br />
<br />
== makevirginjail ==<br />
makevirginjail<br />
Only on some systems, makes an empty jail (doesn't do restore step)<br />
<br />
== mb == <br />
mb <mount|umount><br />
(nfs) mounts and umounts dirs to backup2. Shortcuts are mbm and mbu to mount and unmount. <br />
<br />
== notify.sh ==<br />
notify.sh<br />
emails reboot@johncompanies.com – intended to be called at boot time to alert us to a machine which panics and reboots and isn’t caught by bb or castle.<br />
<br />
== orphanedbackupwatch ==<br />
orphanedbackupwatch<br />
looks for directories on backup2 which aren’t configured in backup.config and offers to delete them<br />
<br />
== postboot ==<br />
postboot<br />
to be run after a machine reboot and quad/safe’s are done executing. It will:<br />
* do chmod 666 on each jail’s /dev/null<br />
* add ipfw counts<br />
* run jailpsall (so you can see if a configured jail isn’t running)<br />
<br />
== preboot ==<br />
preboot<br />
to be run before running quad/safe – checks for misconfigurations: <br />
* a jail configured in a quad but not a safe<br />
* a jail is listed more than once in a quad<br />
* the ip assigned to a jail isn’t configured on the machine<br />
* alias numbering skips in the rc.conf (resulting in the above)<br />
* orphaned vnfile's that aren't mentioned in a quad/safe<br />
* ip mismatches between dir/vnfile name and the jail’s ip<br />
* dir/vnfiles's in quad/safe that don’t exist <br />
<br />
== quadanalyze.pl ==<br />
quadanalyze.pl<br />
called by js, produces the info (seen above with js explanation) about the contents of quad (inode count, # of jails, etc.)<br />
<br />
== rsync.backup ==<br />
rsync.backup<br />
does customer backups (relies on backup.config)<br />
<br />
== taskdone ==<br />
taskdone<br />
when called will email support@johncompanies.com with the hostname of the machine from which it was executed as the subject<br />
<br />
== topten ==<br />
topten<br />
summarizes the top 10 traffic users (called by ipfwreset)<br />
<br />
== trafficgather.pl ==<br />
trafficgather.pl [yy-mm]<br />
sends a traffic usage summary by jail to support@johncomapnies.com and payments@johncompanies.com. Optional arguments are year and month (must be in the past). If not passed, assumes last month. Relies on traffic logs created by ipfwreset and ipfwbackup<br />
<br />
== trafficwatch.pl ==<br />
trafficwatch.pl<br />
checks traffic usage and emails support@johncomapnies.com when a jail reaches the warning level (35G) and the limit (40G). We really aren’t using this anymore now that we have netflow.<br />
<br />
== trafstats ==<br />
trafstats<br />
writes ipfw traffic usage info by jail to a file called jc_traffic_dump in each jail’s / dir<br />
<br />
== truncate_jailmake ==<br />
truncate_jailmake<br />
a version of jailmake which creates truncated vnfiles.<br />
<br />
== vb ==<br />
vb<br />
the equivalent of: <tt>vi /usr/local/jail/bin/backup.config</tt><br />
<br />
== vs (freebsd) ==<br />
vs<n><br />
the equivalent of: <tt>vi /usr/local/jail/rc.d/safe<n></tt><br />
<br />
vq<n><br />
the equivalent of: vi /usr/local/jail/rc.d/quad<n><br />
<br />
== dumpremote ==<br />
dumpremote <user@machine> </remote/location/file-dump> <vnX><br />
ex: dumpremote user@10.1.4.117 /mnt/data3/remote.echoditto.com-dump 7<br />
this will dump a vn filesystem to a remote machine and location<br />
<br />
== oversellcheck ==<br />
oversellcheck<br />
displays how much a disk is oversold or undersold taking into account truncated vn files. Only for use on 4.x systems<br />
<br />
== mvbackups (freebsd) ==<br />
mvbackups <dir> (1.1.1.1-col00001-DIR) <target_machine> (jail1) <target_dir> (data1)<br />
moves backups from one location to another on the backup server, and provides you with option to remove entries from current backup.config, and simple paste command to add the config to backup.config on the target server<br />
<br />
== jailnice ==<br />
jailnice <hostname><br />
applies <tt>renice 19 [PID]</tt> and <tt>rtprio 31 –[PID]</tt> to each process in the given jail<br />
<br />
== dumpremoterestore ==<br />
dumpremoterestore <device> <ip of target machine> <dir on target machine><br />
ex: dumpremoterestore /dev/vn51 10.1.4.118 /mnt/data2/69.55.239.45-col00688-DIR<br />
dumps a device and restores it to a directory on a remote machine. Requires that you enable root ssh on the <br />
remote machine.<br />
<br />
== psj ==<br />
psj<br />
shows just the procs running on the base system – a ps auxw but without jail’d procs present<br />
<br />
== perc5iraidchk ==<br />
perc5iraidchk<br />
checks for degraded arrays on Dell 2950 systems with Perc5/6 controllers<br />
<br />
== perc4eraidchk ==<br />
perc4eraidchk<br />
checks for degraded arrays on Dell 2850 systems with Perc4e/Di controllers<br />
<br />
= Virtuozzo VPS =<br />
<br />
Note: We use VEID (Virtual Environment ID) and CTID (Container ID) interchangably. Similarly, VE and CT. They mean the same thing.<br />
VZPP = VirtuoZzo Power Panel (the control panel for each CT)<br />
<br />
All linux systems exist in /vz, /vz1 or /vz2 - since each linux machine holds roughly 60-90 customers, there will be roughly 30-45 in each partition.<br />
<br />
The actual filesystem of the system in question is in:<br />
<br />
/vz(1-2)/private/(VEID)<br />
<br />
Where VEID is the identifier for that system - an all-numeric string larger than 100.<br />
<br />
The actual mounted and running systems are in the corresponding:<br />
<br />
/vz(1-2)/root/(VEID)<br />
<br />
But we rarely interact with any system from this mount point.<br />
<br />
You should never need to touch the root portion of their system – however you can traverse their filesystem by going to <tt>/vz(1-2)/private/(VEID)/root</tt> (<tt>/vz(1-2)/private/(VEID)/fs/root</tt> on 4.x systems) the root of their filesystem is in that directory, and their entire system is underneath that.<br />
<br />
Every VE has a startup script in <tt>/etc/sysconfig/vz-scripts</tt> (which is symlinked as <tt>/vzconf</tt> on all systems) - the VE startup script is simply named <tt>(VEID).conf</tt> - it contains all the system parameters for that VE:<br />
<br />
<pre># Configuration file generated by vzsplit for 60 VE<br />
# on HN with total amount of physical mem 2011 Mb<br />
<br />
VERSION="2"<br />
CLASSID="2"<br />
<br />
ONBOOT="yes"<br />
<br />
KMEMSIZE="8100000:8200000"<br />
LOCKEDPAGES="322:322"<br />
PRIVVMPAGES="610000:615000"<br />
SHMPAGES="33000:34500"<br />
NUMPROC="410:415"<br />
PHYSPAGES="0:2147483647"<br />
VMGUARPAGES="13019:2147483647"<br />
OOMGUARPAGES="13019:2147483647"<br />
NUMTCPSOCK="1210:1215"<br />
NUMFLOCK="107:117"<br />
NUMPTY="19:19"<br />
NUMSIGINFO="274:274"<br />
TCPSNDBUF="1800000:1900000"<br />
TCPRCVBUF="1800000:1900000"<br />
OTHERSOCKBUF="900000:950000"<br />
DGRAMRCVBUF="200000:200000"<br />
NUMOTHERSOCK="650:660"<br />
DCACHE="786432:818029"<br />
NUMFILE="7500:7600"<br />
AVNUMPROC="51:51"<br />
IPTENTRIES="155:155"<br />
DISKSPACE="4194304:4613734"<br />
DISKINODES="400000:420000"<br />
CPUUNITS="1412"<br />
QUOTAUGIDLIMIT="2000"<br />
VE_ROOT="/vz1/root/636"<br />
VE_PRIVATE="/vz1/private/636"<br />
NAMESERVER="69.55.225.225 69.55.230.3"<br />
OSTEMPLATE="vzredhat-7.3/20030305"<br />
VE_TYPE="regular"<br />
IP_ADDRESS="69.55.225.229"<br />
HOSTNAME="textengine.net"</pre><br />
<br />
<br />
As you can see, the hostname is set here, the disk space is set here, the number of inodes, the number of files that can be open, the number of tcp sockets, etc. - all are set here.<br />
<br />
In fact, everything that can be set on this customer system is set in this conf file.<br />
<br />
<br />
All interaction with the customer system is done with the VEID. You start the system by running:<br />
<br />
vzctl start 999<br />
<br />
You stop it by running:<br />
<br />
vzctl stop 999<br />
<br />
You execute commands in it by running:<br />
<br />
vzctl exec 999 df -k<br />
<br />
You enter into it, via a root-shell backdoor with:<br />
<br />
vzctl enter 999<br />
<br />
and you set parameters for the system, while it is still running, with:<br />
<br />
vzctl set 999 --diskspace 6100000:6200000 --save<br />
<br />
<tt>vzctl</tt> is the most commonly used command - we have aliased <tt>v</tt> to <tt>vzctl</tt> since we use it so often. We’ll continue to use <tt>vzctl</tt> in our examples, but feel free to use just <tt>v</tt>.<br />
<br />
Let's say the user wants more diskspace. You can cat their conf file and see:<br />
<br />
DISKSPACE="4194304:4613734"<br />
<br />
So right now they have 4gigs of space. You can then change it to 6 with:<br />
<br />
vzctl set 999 --diskspace 6100000:6200000 --save<br />
<br />
IMPORTANT: all issuances of the vzctl set command need to end with <tt>–save</tt> - if they don't, the setting will be set, but it will not be saved to the conf file, and they will not have those settings next time they boot.<br />
<br />
All of the tunables in the conf file can be set with the vzctl set command. Note that in the conf file, and on the vzctl set command line, we always issue two numbers seperated by a colon - that is because we are setting the hard and soft limits. Always set the hard limit slightly above the soft limit, as you see it is in the conf file for all those settings.<br />
<br />
There are also things you can set with `<tt>vzctl set</tt>` that are not in the conf file as settings, per se. For instance, you can add IPs:<br />
<br />
vzctl set 999 --ipadd 10.10.10.10 --save<br />
<br />
or multiple IPs:<br />
<br />
vzctl set 999 --ipadd 10.10.10.10 --ipadd 10.10.20.30 --save<br />
<br />
or change the hostname:<br />
<br />
vzctl set 999 --hostname www.example.com --save<br />
<br />
You can even set the nameservers:<br />
<br />
vzctl set 999 --nameserver 198.78.66.4 --nameserver 198.78.70.180 --save<br />
<br />
Although you probably will never do that.<br />
<br />
You can disable a VPS from being started (by VZPP or reboot) (4.x):<br />
<br />
vzctl set 999 --disabled yes --save <br />
<br />
You can disable a VPS from being started (by VZPP or reboot) (<=3.x):<br />
<br />
vzctl set 999 --onboot=no --save <br />
<br />
You can disable a VPS from using his control panel:<br />
<br />
vzctl set 999 --offline_management=no --save <br />
<br />
You can suspend a VPS, so it can be resumed in the same state it was in when it was stopped (4.x):<br />
<br />
vzctl suspend 999<br />
<br />
and to resume it:<br />
<br />
vzctl resume 999<br />
<br />
to see who owns process:<br />
vzpid <PID><br />
<br />
to mount up an unmounted ve:<br />
vzctl mount 827<br />
<br />
<br />
To see network stats for CT's:<br />
<pre># vznetstat<br />
VEID Net.Class Output(bytes) Input(bytes)<br />
-----------------------------------------------<br />
24218 1 484M 39M<br />
24245 1 463M 143M<br />
2451 1 2224M 265M<br />
2454 1 2616M 385M<br />
4149 1 125M 68M<br />
418 1 1560M 34M<br />
472 1 1219M 315M<br />
726 1 628M 317M<br />
763 1 223M 82M<br />
771 1 4234M 437M<br />
-----------------------------------------------<br />
Total: 13780M 2090M</pre><br />
<br />
<br />
One thing that sometimes comes up on older systems that we created with smaller defaults is that the system would run out of inodes. The user will email and say they cannot create any more files or grow any files larger, but they will also say that they are not out of diskspace ... they are running:<br />
<br />
df -k<br />
<br />
and seeing how much space is free - and they are not out of space. They are most likely out of inodes - which they would see by running:<br />
<br />
df -i<br />
<br />
So, the first thing you should do is enter their system with:<br />
<br />
vzctl enter 999<br />
<br />
and run: <tt>df -i</tt><br />
<br />
to confirm your theory. Then exit their system. Then simply cat their conf file and see what their inodes are set to (probably 200000:200000, since that was the old default on the older systems) and run:<br />
<br />
vzctl set 999 --diskinodes 400000:400000 --save<br />
<br />
If they are not out of inodes, then a good possibility is that they have maxed out their numfile configuration variable, which controls how many files they can have in their system. The current default is 7500 (which nobody has ever hit), but the old default was as low as 2000, so you would run something like:<br />
<br />
vzctl set 999 --numfile 7500:7500 --save<br />
<br />
----<br />
<br />
You cannot start or stop a VE if your pwd is its private (/vz/private/999) or root (/vz/root/999) directories, or anywhere below them.<br />
<br />
<br />
== Recovering from a crash (linux) ==<br />
<br />
=== Diagnose whether you have a crash ===<br />
The most important thing is to get the machine and all ve’s back up as soon as possible. Note the time, you’ll need to create a crash log entry (Mgmt. -> Reference -> CrashLog). The first thing to do is head over to the [[Screen#Screen_Organization|serial console screen]] and see if there’s any kernel error messages output. Try to copy any messages (or just a sample of repeating messages) you see into the notes section of the crash log – these will also likely need to be sent to virtuozzo for interpretation. If the messages are spewing too fast, hit ^O + H to start a screen log dump which you can ob1182.pts-38.bb serve after the machine is rebooted. Additionally, if the machine is responsive, you can get a trace to send to virtuozzo by hooking up a kvm and entering these 3 sequences:<br />
<pre>alt+print screen+m<br />
alt+print screen+p<br />
alt+print screen+t</pre><br />
<br />
If there are no messages, the machine may just be really busy- wait a bit (5-10min) to see if it comes back. If it's still pinging, odds are its very busy. If it doesn't come back, or the messages indicate a fatal error, you will need to proceed with a power cycle (ctrl+alt+del will not work).<br />
<br />
=== Power cycle the server ===<br />
If this machine is not a Dell 2950 with a [[DRAC/RMM#DRAC|DRAC card]] (i.e. if you can’t ssh into the DRAC card and issue racadm serveraction hardreset, then you will need someone at the data center to power the macine off, wait 30 sec, then turn it back on. Make sure to re-attach via console (<tt>tip virtxx</tt>) immediately after power down. <br />
<br />
=== (Re)attach to the console ===<br />
Stay on the console the entire time during boot. As the BIOS posts- look out for the RAID card output- does everything look healthy? The output may be scrambled, look for "DEGRADED" or "FAILED". Once the OS starts booting you will be disconnected (dropped back to the shell on the console server) a couple times during the boot up. The reason you want to quickly re-attach is two-fold: 1. If you don’t reattach quickly then you won’t get any console output, 2. you want to be attached before the server ''potentially'' starts (an extensive) fsck. If you attach after the fsck begins, you’ll have seen no indication it started an fsck and the server will appear frozen during startup- no output, no response. <br />
<br />
=== Start containers/VE's/VPSs ===<br />
When the machine begins to start VE’s, it’s safe to leave the console and login via ssh. All virts should be set to auto start all the VEs after a crash. Further, most (newer) virts are set to “fastboot” it’s VE’s (to find out, do:<br />
grep -i fast /etc/sysconfig/vz <br />
and look for <tt>VZFASTBOOT=yes</tt>). If this was set prior to the machine’s crash (setting it after the machine boots will not have any effect until the vz service is restarted) it will start each ve as fast as possible, in serial, then go thru each VE (serially), shutting it down running a vzquota (disk usage) check, then bringing it back up. The benefit is that all VE’s are brought up quickly (within 15min or so depending on the #), the downside is a customer watching closely will notice 2 outages – 1st the machine crash, 2nd their quota check (which will be a much shorter downtime- on the order of a few minutes). <br />
<br />
Where “fastboot” is not set to yes (i.e on quar1), vz will start them consecutively, checking the quotas one at a time, and the 60th VE may not start until an hour or two later - this is not acceptable.<br />
<br />
The good news is, if you run vzctl start for a VE that is already started, you will simply get an error: <tt>VE is already started</tt>. Further, if you attempt to vzctl start a VE that is in the process of being started, you will simply get an error: unable to lock VE. So, there is no danger in simply running scripts to start smaller sets of VEs. If the system is not autostarting, then there is no issue, and even if it does, when it conflicts, one process (yours or the autostart) will lose, and just move on to the next one.<br />
<br />
A script has been written to assist with ve starts: [[#startvirt.pl|startvirt.pl]] which will start 6 ve’s at once until there are no more left. If startvirt.pl is used on a system where “fastboot” was on, it will circumvent the fastboot for ve’s started by startvirt.pl – they will go through the complete quota check before starting- therefore this is not advisable when a system has crashed. When a system is booted cleanly, and there's no need for vzquota checks, then startvirt.pl is safe and advisable to run.<br />
<br />
=== Make sure all containers are running ===<br />
You can quickly get a feel for how many ve’s are started by running:<br />
<br />
<pre>[root@virt4 log]# vs<br />
VEID 16066 exist mounted running<br />
VEID 16067 exist mounted running<br />
VEID 4102 exist mounted running<br />
VEID 4112 exist mounted running<br />
VEID 4116 exist mounted running<br />
VEID 4122 exist mounted running<br />
VEID 4123 exist mounted running<br />
VEID 4124 exist mounted running<br />
VEID 4132 exist mounted running<br />
VEID 4148 exist mounted running<br />
VEID 4151 exist mounted running<br />
VEID 4155 exist mounted running<br />
VEID 42 exist mounted running<br />
VEID 432 exist mounted running<br />
VEID 434 exist mounted running<br />
VEID 442 exist mounted running<br />
VEID 450 exist mounted running<br />
VEID 452 exist mounted running<br />
VEID 453 exist mounted running<br />
VEID 454 exist mounted running<br />
VEID 462 exist mounted running<br />
VEID 463 exist mounted running<br />
VEID 464 exist mounted running<br />
VEID 465 exist mounted running<br />
VEID 477 exist mounted running<br />
VEID 484 exist mounted running<br />
VEID 486 exist mounted running<br />
VEID 490 exist mounted running</pre><br />
<br />
So to see how many ve’s have started:<br />
<pre>[root@virt11 root]# vs | grep running | wc -l<br />
39</pre><br />
<br />
And to see how many haven’t:<br />
<pre>[root@virt11 root]# vs | grep down | wc -l<br />
0</pre><br />
<br />
And how many we should have running:<br />
<pre>[root@virt11 root]# vs | wc -l<br />
39</pre><br />
<br />
Another tool you can use to see which ve’s have started, among other things is [[#vzstat|vzstat]]. It will give you CPU, memory, and other stats on each ve and the overall system. It’s a good thing to watch as ve’s are starting (note the VENum parameter, it will tell you how many have started):<br />
<br />
<pre>4:37pm, up 3 days, 5:31, 1 user, load average: 1.57, 1.68, 1.79<br />
VENum 40, procs 1705: running 2, sleeping 1694, unint 0, zombie 9, stopped 0<br />
CPU [ OK ]: VEs 57%, VE0 0%, user 8%, sys 7%, idle 85%, lat(ms) 412/2<br />
Mem [ OK ]: total 6057MB, free 9MB/54MB (low/high), lat(ms) 0/0<br />
Swap [ OK ]: tot 6142MB, free 4953MB, in 0.000MB/s, out 0.000MB/s<br />
Net [ OK ]: tot: in 0.043MB/s 402pkt/s, out 0.382MB/s 4116pkt/s<br />
Disks [ OK ]: in 0.002MB/s, out 0.000MB/s<br />
<br />
VEID ST %VM %KM PROC CPU SOCK FCNT MLAT IP<br />
1 OK 1.0/17 0.0/0.4 0/32/256 0.0/0.5 39/1256 0 9 69.55.227.152<br />
21 OK 1.3/39 0.1/0.2 0/46/410 0.2/2.8 23/1860 0 6 69.55.239.60<br />
133 OK 3.1/39 0.1/0.3 1/34/410 6.3/2.8 98/1860 0 0 69.55.227.147<br />
263 OK 2.3/39 0.1/0.2 0/56/410 0.3/2.8 34/1860 0 1 69.55.237.74<br />
456 OK 17/39 0.1/0.2 0/100/410 0.1/2.8 48/1860 0 11 69.55.236.65<br />
476 OK 0.6/39 0.0/0.2 0/33/410 0.1/2.8 96/1860 0 10 69.55.227.151<br />
524 OK 1.8/39 0.1/0.2 0/33/410 0.0/2.8 28/1860 0 0 69.55.227.153<br />
594 OK 3.1/39 0.1/0.2 0/45/410 0.0/2.8 87/1860 0 1 69.55.239.40<br />
670 OK 7.7/39 0.2/0.3 0/98/410 0.0/2.8 64/1860 0 216 69.55.225.136<br />
691 OK 2.0/39 0.1/0.2 0/31/410 0.0/0.7 25/1860 0 1 69.55.234.96<br />
744 OK 0.1/17 0.0/0.5 0/10/410 0.0/0.7 7/1860 0 6 69.55.224.253<br />
755 OK 1.1/39 0.0/0.2 0/27/410 0.0/2.8 33/1860 0 0 192.168.1.4<br />
835 OK 1.1/39 0.0/0.2 0/19/410 0.0/2.8 5/1860 0 0 69.55.227.134<br />
856 OK 0.3/39 0.0/0.2 0/13/410 0.0/2.8 16/1860 0 0 69.55.227.137<br />
936 OK 3.2/52 0.2/0.4 0/75/410 0.2/0.7 69/1910 0 8 69.55.224.181<br />
1020 OK 3.9/39 0.1/0.2 0/60/410 0.1/0.7 55/1860 0 8 69.55.227.52<br />
1027 OK 0.3/39 0.0/0.2 0/14/410 0.0/2.8 17/1860 0 0 69.55.227.83<br />
1029 OK 1.9/39 0.1/0.2 0/48/410 0.2/2.8 25/1860 0 5 69.55.227.85<br />
1032 OK 12/39 0.1/0.4 0/80/410 0.0/2.8 41/1860 0 8 69.55.227.90</pre><br />
<br />
When you are all done, you will want to make sure that all the VEs really did get started, run vs one more time.<br />
<br />
Note the time all ve’s are back up and enter that into and save the crash log entry.<br />
<br />
Occasionally, a ve will not start automatically. The most common reason for a ve not to come up normally is the ve was at it’s disk limit before the crash, and will not start since they’re over the limit. To overcome this, set the disk space to current usage level (the system will give this to you when it fails to start), start the ve, then re-set the disk space back to the prior level. Lastly, contact the customer to let them know they’re out of disk (or allocate more disk if they're entitled to more).<br />
<br />
== Hitting performance barriers and fixing them ==<br />
<br />
There are multiple modes virtuozzo offers to allocate resources to a ve. We utilize 2: SLM and UBC parameters<br />
On our 4.x systems, we use all SLM – it’s simpler to manage and understand. There are a few systems on virt19/18 that may also use SLM. Everything else uses UBC. <br />
You can tell a SLM ve by:<br />
<br />
SLMMODE="all"<br />
<br />
in their conf file. <br />
<br />
TODO: detail SLM modes and parameters.<br />
<br />
If someone is in SLM mode and they hit memory resource limits, they simply need to upgrade to more memory.<br />
<br />
The following applies to everyone else (UBC).<br />
<br />
Customers will often email and say that they are getting out of memory errors - a common one is "cannot fork" ... basically, anytime you see something odd like this, it means they are hitting one of their limits that is in place in their conf file.<br />
<br />
The conf file, however, simply shows their limits - how do we know what they are currently at ?<br />
<br />
The answer is a file called v - this file contains the current status (and peaks) of their performance settings, and also counts how many times they have hit the barrier. The output of the file looks like this:<br />
<br />
<pre>764: kmemsize 384113 898185 8100000 8200000 0<br />
lockedpages 0 0 322 322 0<br />
privvmpages 1292 7108 610000 615000 0<br />
shmpages 270 528 33000 34500 0<br />
dummy 0 0 0 0 0<br />
numproc 8 23 410 415 0<br />
physpages 48 5624 0 2147483647 0<br />
vmguarpages 0 0 13019 2147483647 0<br />
oomguarpages 641 6389 13019 2147483647 0<br />
numtcpsock 3 21 1210 1215 0<br />
numflock 1 3 107 117 0<br />
numpty 0 2 19 19 0<br />
numsiginfo 0 4 274 274 0<br />
tcpsndbuf 0 80928 1800000 1900000 0 <br />
tcprcvbuf 0 108976 1800000 1900000 0<br />
othersockbuf 2224 37568 900000 950000 0<br />
dgramrcvbuf 0 4272 200000 200000 0<br />
numothersock 3 9 650 660 0<br />
dcachesize 53922 100320 786432 818029 0<br />
numfile 161 382 7500 7600 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
numiptent 4 4 155 155 0</pre><br />
<br />
The first column is the name of the counter in question - the same names we saw in the systems conf file. The second column is the _current_ value of that counter, the third column is the max that that counter has ever risen to, the fourth column is the soft limit, and the fifth column is the hard limit (which is the same as the numbers in that systems conf file).<br />
<br />
The sixth number is the failcount - how many times the current usage has risen to hit the barrier. It will increase as soon as the current usage hits the soft limit.<br />
<br />
The problem with /proc/user_beancounters is that it actually contains that set of data for every running VE - so you can't just cat /proc/user_beancounters - it is too long and you get info for every other running system.<br />
<br />
You can vzctl enter the system and run:<br />
<br />
vzctl enter 9999<br />
cat /proc/user_beancounters<br />
<br />
inside their system, and you will just see the stats for their particular system, but entering their system every time you want to see it is combersome.<br />
<br />
So, I wrote a simple script called "vzs" which simply greps for the VEID, and spits out the next 20 or so lines (however many lines there are in the output, I forget) after it. For instance:<br />
<br />
<pre># vzs 765:<br />
765: kmemsize 2007936 2562780 8100000 8200000 0<br />
lockedpages 0 8 322 322 0<br />
privvmpages 26925 71126 610000 615000 0<br />
shmpages 16654 16750 33000 34500 0<br />
dummy 0 0 0 0 0<br />
numproc 41 57 410 415 0<br />
physpages 1794 49160 0 2147483647 0<br />
vmguarpages 0 0 13019 2147483647 0<br />
oomguarpages 4780 51270 13019 2147483647 0<br />
numtcpsock 23 37 1210 1215 0<br />
numflock 17 39 107 117 0<br />
numpty 1 3 19 19 0<br />
numsiginfo 0 6 274 274 0<br />
tcpsndbuf 22240 333600 1800000 1900000 0<br />
tcprcvbuf 0 222656 1800000 1900000 0<br />
othersockbuf 104528 414944 900000 950000 0<br />
dgramrcvbuf 0 4448 200000 200000 0<br />
numothersock 73 105 650 660 0<br />
dcachesize 247038 309111 786432 818029 0<br />
numfile 904 1231 7500 7600 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
numiptent 4 4 155 155 0</pre><br />
<br />
That showed us just the portion of /proc/user_beancounters for system 765.<br />
<br />
When you run the vzs command, always add a : after the VEID.<br />
<br />
So, if a customer complains about some out of memory errors, or no more files, or no more ptys, or just has an unspecific complain about processes dying, etc., the very first thing you need to do is check their beancounters with vzs. Usually you will spot an item that has a high failcount and needs to be upped.<br />
<br />
At that point you could simply up the counter with `vzctl set`. Generally pick a number 10-20% higher than the old one, and make the hard limit slightly larger than the the soft limit. However our systems now come in several levels and those levels have more/different memory allocations. If someone is complaining about something other than a memory limit (pty, numiptent, numflock), it’s generally safe to increase it, at least to the same level as what’s in the /vzconf/4unlimited file on the newest virt. If someone is hitting a memory limit, first make sure they are given what they deserve:<br />
<br />
(refer to mgmt -> payments -> packages)<br />
<br />
To set those levels, you use the [[#setmem|setmem]] command. <br />
<br />
The alternate (DEPRECATED) method would be to use one of 3 commands:<br />
256 <veid><br />
300 <veid><br />
384 <veid><br />
512 <veid><br />
<br />
If the levels were not right (you’d run vzs <veid> before and after to see the effect) tell the customer they’ve been adjusted and be done with it. If the levels were right, tell the customer they must upgrade to a higher package, tell them how to see level (control panel) and that they can reboot their system to escape this lockup contidion.<br />
<br />
Customers can also complain that their site is totally unreachable, or complain that it is down ... if the underlying machine is up, and all seems well, you may notice in the beancounters that network-specific counters are failing - such as numtcpsock, tcpsndbuf or tcprcvbuf. This will keep them from talking on the network and make it seem like their system is down. Again, just up the limits and things should be fine.<br />
<br />
On virts 1-4, you should first look at the default settings for that item on a later virt, such as virt 8 - we have increased the defaults a lot since the early machines. So, if you are going to up a counter on virt2, instead of upping it by 10-20%, instead up it to the new default that you see on virt8.<br />
<br />
== Moving a VE to another virt (migrate/migrateonline) ==<br />
<br />
This will take a while to complete - and it is best to do this at night when the load is light on both machines.<br />
<br />
There are different methods for this, depending on which version of virtuozzo is installed on the src. and dst. virt. <br />
To check which version is running: <br />
[root@virt12 private]# cat /etc/virtuozzo-release<br />
Virtuozzo release 2.6.0<br />
<br />
Ok, let's say that the VE is 1212, and vital stats are:<br />
<pre>[root@virt12 sbin]# vc 1212<br />
VE_ROOT="/vz1/root/1212"<br />
VE_PRIVATE="/vz1/private/1212"<br />
OSTEMPLATE="fedora-core-2/20040903"<br />
IP_ADDRESS="69.55.229.84"<br />
TEMPLATES="devel-fc2/20040903 php-fc2/20040813 mysql-fc2/20040812 postgresql-fc2/20040813 mod_perl-fc2/20040812 mod_ssl-fc2/20040811 jre-fc2/20040823 jdk-fc2/20040823 mailman-fc2/20040823 analog-fc2/20040824 proftpd-fc2/20040818 tomcat-fc2/20040823 usermin-fc2/20040909 webmin-fc2/20040909 uw-imap-fc2/20040830 phpBB-fc2/20040831 spamassassin-fc2/20040910 PostNuke-fc2/20040824 sl-webalizer-fc2/20040<br />
818"<br />
<br />
[root@virt12 sbin]# vzctl exec 1212 df -h<br />
Filesystem Size Used Avail Use% Mounted on<br />
/dev/vzfs 4.0G 405M 3.7G 10% /</pre><br />
<br />
From this you can see that he’s using (and will minimally need free on the dst server) ~400MB, and he’s running on a Fedora 2 template, version 20040903. He’s also got a bunch of other templates installed. It’s is '''vital''' that '''all''' these templates exist on the dst system. To confirm that, on the dst system run:<br />
<br />
For < 3.0:<br />
<br />
<pre>[root@virt14 private]# vzpkgls | grep fc2<br />
devel-fc2 20040903<br />
PostNuke-fc2 20040824<br />
analog-fc2 20040824<br />
awstats-fc2 20040824<br />
bbClone-fc2 20040824<br />
jdk-fc2 20040823<br />
jre-fc2 20040823<br />
mailman-fc2 20040823<br />
mod_frontpage-fc2 20040816<br />
mod_perl-fc2 20040812<br />
mod_ssl-fc2 20040811<br />
mysql-fc2 20040812<br />
openwebmail-fc2 20040817<br />
php-fc2 20040813<br />
phpBB-fc2 20040831<br />
postgresql-fc2 20040813<br />
proftpd-fc2 20040818<br />
sl-webalizer-fc2 20040818<br />
spamassassin-fc2 20040910<br />
tomcat-fc2 20040823<br />
usermin-fc2 20040909<br />
uw-imap-fc2 20040830<br />
webmin-fc2 20040909<br />
[root@virt14 private]# vzpkgls | grep fedora<br />
fedora-core-1 20040121 20040818<br />
fedora-core-devel-1 20040121 20040818<br />
fedora-core-2 20040903<br />
[root@virt14 private]#</pre><br />
<br />
For these older systems, you can simply match up the date on the template. <br />
<br />
For >= 3.0:<br />
<br />
<pre>[root@virt19 /vz2/private]# vzpkg list<br />
centos-5-x86 2008-01-07 22:05:57<br />
centos-5-x86 devel<br />
centos-5-x86 jre<br />
centos-5-x86 jsdk<br />
centos-5-x86 mod_perl<br />
centos-5-x86 mod_ssl<br />
centos-5-x86 mysql<br />
centos-5-x86 php<br />
centos-5-x86 plesk9<br />
centos-5-x86 plesk9-antivirus<br />
centos-5-x86 plesk9-api<br />
centos-5-x86 plesk9-atmail<br />
centos-5-x86 plesk9-backup<br />
centos-5-x86 plesk9-horde<br />
centos-5-x86 plesk9-mailman<br />
centos-5-x86 plesk9-mod-bw<br />
centos-5-x86 plesk9-postfix<br />
centos-5-x86 plesk9-ppwse<br />
centos-5-x86 plesk9-psa-firewall<br />
centos-5-x86 plesk9-psa-vpn<br />
centos-5-x86 plesk9-psa-fileserver<br />
centos-5-x86 plesk9-qmail<br />
centos-5-x86 plesk9-sb-publish<br />
centos-5-x86 plesk9-vault<br />
centos-5-x86 plesk9-vault-most-popular<br />
centos-5-x86 plesk9-watchdog</pre><br />
<br />
On these newer systems, it's difficult to tell whether the template on the dst matches exactly the src. Just cause a centos-5-x86 is listed on both servers doesn't mean all the same packages are there on the dst. To truly know, you must perform a sample rsync:<br />
<br />
rsync -avn /vz/template/centos/5/x86/ root@10.1.4.61:/vz/template/centos/5/x86/<br />
<br />
if you see a ton of output from the dry run command, then clearly there are some differences. You may opt to let the rsync complete (without running in dry run mode) the only downside is you've now used up more space on the dst and also the centos template will be a mess with old and new data- it will be difficult if not impossible to undo (if someday we wanted to reclaim the space).<br />
<br />
If you choose to merge templates, you should closely inspect the dry run output. You should also take care to exclude anything in the /config directory. For example:<br />
<br />
rsync -av -e ssh --stats --exclude=x86/config /vz/template/ubuntu/10.04/ root@10.1.4.62:/vz/template/ubuntu/10.04/<br />
<br />
Which will avoid this directory and contents:<br />
<pre>[root@virt11 /vz2/private]# ls /vz/template/ubuntu/10.04/x86/config*<br />
app os</pre><br />
<br />
This is important to avoid since the config may differ on the destination and we are really only interested in making sure the pacakges are there, not overwriting a newer config with an older one.<br />
<br />
If the dst system was missing a template, you have 2 choices: <br />
# put the missing template on the dst system. 2 choices here: <br />
## Install the template from rpm (found under backup2: /mnt/data4/vzrpms/distro/) or <br />
## rsync over the template (found under /vz/template) - see above<br />
# put the ve on a system which has all the proper templates<br />
<br />
=== pre-seeding a migration ===<br />
<br />
When migrating a customer (or when doing many) depending on how much data you have to transfer, it can take some time. Further, it can be difficult to gauge when a migration will complete or how long it will take. To help speed up the process and get a better idea about how long it will take you can pre-transfer a customer's data to the destination server. If done correctly, vzmigrate will see the pre-transferred data and pick up where you left off, having much less to transfer (just changed/new files). <br />
<br />
We believe vzmigrate uses rsync to do it's transfer. Therefore not only can you use rsync to do a pre-seed, you can also run rsync to see what is causing a repeatedly-failing vzmigrate to fail. <br />
<br />
There's no magic to a pre-seed, you just need to make sure it's named correctly.<br />
<br />
Given:<br />
<br />
source: /vz1/private/1234<br />
<br />
and you want to migrate to /vz2 on the target system, your rsync would look like:<br />
<br />
rsync -av /vz1/private/1234/ root@x.x.x.x:/vz2/private/1234.migrated/<br />
<br />
After running that successful rsync, the ensuing migrateonline (or migrate) will take much less time to complete- depending on the # of files to be analyzed and the # of changed files. In any case, it'll be much much faster than had you just started the migration from scratch.<br />
<br />
Further, as we discuss elsewhere in this topic, a failed migration can be moved from <tt>/vz/private/1234</tt> to <tt>/vz/private/1234.migrated</tt> on the destination if you want to restart a failed migration. This should '''only''' be done if the migration failed and the CT is not running on the destination HN.<br />
<br />
=== migrateonline intructions: src >=3.x -> dst>=3.x ===<br />
<br />
A script called [[#migrateonline|migrateonline]] was written to handle this kind of move. It is basically a wrapper for <tt>vzmigrate</tt> – vzmigrate is a util to seamlessly- as no no reboot of the ve necessary- move a ve from one host to another. This wrapper was initially written cause vz virtuozzo version 2.6.0 has a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables, causing problems when the ve was started up on the dst system. [[#migrate|migrate]] mitigates that. Since it makes multiple ssh connections to the dst virt, it’s a good idea to put the pub key for the src virt in the authorized_keys file on the dst virt. In addition, migrateonline emails ve owners when their migration starts and stops. For this to happen they need to put email addresses (on a single line, space delimited) in a file on their system: /migrate_notify. If the optional target dir is not specified, the ve will be moved to the same private/root location as it was on the src virt. Note: <tt>migrate</tt> is equivalent to <tt>migrateonline</tt>, but will <tt>migrate</tt> a ve AND restart it in the process.<br />
<br />
<pre>[root@virt12 sbin]# migrateonline<br />
usage: /usr/local/sbin/migrateonline <ip of node migrating to> <veid> [target dir: vz | vz1 | vz2]<br />
[root@virt12 sbin]# migrateonline 10.1.4.64 1212 vz<br />
starting to migrate 1212 at Sat Mar 26 22:40:38 PST 2005<br />
Turning off offline_management<br />
Saved parameters for VE 1212<br />
migrating with no start on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
clearing the arp cache<br />
now going to 10.1.4.64 and clear cache and starting<br />
starting it<br />
Delete port redirection<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
finished migrating 1212 at Sat Mar 26 22 22:52:01 PST 2005<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine.<br />
<br />
If they had backups, use the mvbackups command to move their backups to the new server:<br />
<br />
mvbackups 1212 virt14 vz<br />
<br />
Rename the ve<br />
<br />
[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/migrated-1212<br />
<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/old-1212-migrated-20120404-noarchive<br />
<br />
Update the customer’s systems in mgmt to reflect the new path and server.<br />
<br />
<br />
IF migrateonline does not work, you can try again using simply migrate- this will result in a brief reboot for the ve.<br />
Before you try again, make sure of a few things:<br />
<br />
Depending on where in the migration died, there may be partial data on the dst system in 1 of 2 places:<br />
(given the example above)<br />
<br />
/vz/private/1212<br />
<br />
or <br />
<br />
/vz/private/1212.migrated<br />
<br />
before you run migrate again, you'll want to rename so that all data is in <br />
1212.migrated:<br />
<br />
mv /vz/private/1212 /vz/private/1212.migrated<br />
<br />
this way, it will pick up where it left off and transfer only new files.<br />
<br />
Likewise, if you want to speed up a migration, you can pre-seed the dst as follows:<br />
<br />
[root@virt12 sbin]# rsync -avSH /vz/private/1212/ root@10.1.4.64:/vz/private/1212.migrated/<br />
<br />
then when you run migrate or migrateonline, it will only need to move the changed files- the migration will complete quickly<br />
<br />
=== migrate manually (if migrate/online fails) ===<br />
<br />
Lets say for whatever reason the migration fails. If it fails with [[#migrateonline|migrateonline]], you should try [[#migrate|migrate]] (which will reboot the customer, so notify them ahead of time).<br />
<br />
You may want to run a [[#pre-seeding_a_migration|pre-seed]] rsync to see if you can find the problem. On older virts, we've seen this problem due to a large logfile (which you can find and encourage the customer to remove/compress):<br />
for f in `find / -size +1048576k`; do ls -lh $f; done<br />
<br />
If the rsync or [[#migrate|migrate]] fails, <br />
<br />
You can always move someone manually:<br />
<br />
1. stop ve: <br><br />
v stop 1234<br />
<br />
2. copy over data<br><br />
rsync -avSH /vz/private/1234/ root@1.1.1.1:/vzX/private/1234/<br />
<br />
NOTE: if you've previously seeded the data (run rsync while the VE was up/running), and this is a subsequent rsync, make sure the last rsync you do (while the VE is not running, has the --delete option in the rsync)<br />
<br />
3. copy over conf<br><br />
scp /vzconf/1234.conf root@1.1.1.1:/vzconf<br />
<br />
4. on dst, edit the conf to reflect the right vzX dir<br><br />
vi /vzconf/1234.conf<br />
<br />
5. on src remove the IPs<br><br />
ipdel 1234 2.2.2.2 3.3.3.3<br />
<br />
6. on dst add IPs <br><br />
ipadd 1234 2.2.2.2 3.3.3.3<br />
<br />
7. on dst, start ve: <br><br />
v start 1324<br />
<br />
8. cancel, then archive ve on src per above instrs.<br />
<br />
=== migrate src=2.6.0 -> dst>=2.6.0, or mass-migration with customer notify ===<br />
<br />
A script called <tt>migrate</tt> was written to handle this kind of move. It is basically a wrapper for vzmigrate – vzmigrate is a util to seamlessly move a ve from one host to another. This wrapper was initially written cause vz virtuozzo version 2.6.0 has a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables, causing problems when the ve was started up on the dst system. migrate mitigates that. Since it makes multiple ssh connections to the dst virt, it’s a good idea to put the pub key for the src virt in the authorized_keys file on the dst virt. In addition, migrate emails ve owners when their migration starts and stops. For this to happen they need to put email addresses (on a single line, space delimited) in a file on their system: /migrate_notify. If the optional target dir is not specified, the ve will be moved to the same private/root location as it was on the src virt. Note: migrateonline is equivalent to migrate, but will migrate a ve from one 2.6 '''kernel''' machine to another 2.6 kernel machine without restarting the ve.<br />
<br />
<pre>[root@virt12 sbin]# migrate<br />
usage: /usr/local/sbin/migrate <ip of node migrating to> <veid> [target dir: vz | vz1 | vz2]<br />
[root@virt12 sbin]# migrate 10.1.4.64 1212 vz<br />
starting to migrate 1212 at Sat Mar 26 22:40:38 PST 2005<br />
Turning off offline_management<br />
Saved parameters for VE 1212<br />
migrating with no start on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
clearing the arp cache<br />
now going to 10.1.4.64 and clear cache and starting<br />
starting it<br />
Delete port redirection<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
finished migrating 1212 at Sat Mar 26 22 22:52:01 PST 2005<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine (backup2 or mail).<br />
Cancel the ve (first we have to rename things which migrate changed so cancelve will find them):<br />
<br />
[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/1212.conf<br />
<br />
On 2.6.1 you’ll also have to move the private area:<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/1212<br />
<br />
<pre>[root@virt12 sbin]# cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, [[#cancelve|cancelve]] would offer to remove them. You want to say '''no''' to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
To move everyone off a system, you’d do:<br />
for f in `vl`; do migrate <ip> $f; done<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
=== vzmigrate: src=2.6.1 -> dst>=2.6.0 ===<br />
<br />
This version of vzmigrate works properly with regard to handling ips. It will not notify ve owners of moves as in the above example. Other than that it’s essentially the same.<br />
<br />
<pre>[root@virt12 sbin]# vzmigrate 10.1.4.64 -r no 1212:1212:/vz/private/1212:/vz/root/1212<br />
migrating on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine (backup2 or mail).<br />
Cancel the ve (first we have to rename things which vzmigrate changed so cancelve will find them):<br />
<br />
<pre>[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/1212.conf<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/1212<br />
<br />
[root@virt12 sbin]# cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, <tt>cancelve</tt> would offer to remove them. You want to say no to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
=== src=2.5.x ===<br />
<br />
First, go to the private dir:<br />
<br />
cd /vz1/private/<br />
<br />
Stop the VE - make sure it stops totally cleanly.<br />
<br />
vzctl stop 1212<br />
<br />
Then you’d use vemove - a script written to copy over the config, create tarballs of the ve’s data on the destination virt, and cancel the ve on the source system (in this example we’re going to put a ve that was in /vz1/private on the src virt, in /vz/private on the dst virt):<br />
<br />
<pre>[root@virt12 sbin]# vemove<br />
ERROR: Usage: vemove veid target_ip target_path_dir<br />
[root@virt12 sbin]# vemove 1212 10.1.4.64 /vz/private/1212<br />
tar cfpP - 1212 --ignore-failed-read | (ssh -2 -c arcfour 10.1.4.64 "split - -b 1024m /vz/private/1212.tar" )<br />
scp /vzconf/1212.conf 10.1.4.64:/vzconf<br />
cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, cancelve would offer to remove them. You want to say no to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
When you are done, go to /vz/private on the dst virt you will have files like this:<br />
<br />
<pre>1212.taraa<br />
1212.tarab<br />
1212.tarac</pre><br />
<br />
Each one 1024m (or less, for the last one) in size.<br />
<br />
on the dst server and run:<br />
<br />
cat 1212.tar?? | tar xpPBf -<br />
<br />
and after 20 mins or so it will be totally untarred. Now since the conf<br />
file is already there, you can go ahead and start the system.<br />
<br />
vzctl start 1212<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
NOTE: you MUST tar the system up using the virtuozzo version of tar that<br />
is on all the virt systems, and further you MUST untar the tarball with<br />
the virtuozzo tar, using these options: `<tt>tar xpPBf -</tt>`<br />
<br />
If you tar up an entire VE and move it to a non-virtuozzo machine, that is<br />
ok, and you can untar it there with normal tar commands, but do not untar<br />
it and then repack it with a normal tar and expect it to work - you need<br />
to use virtuozzo tar commands on virtuozzo tarballs to make it work.<br />
<br />
The backups are sort of an exception, since we are just (usually)<br />
restoring user data that was created after we gave them the system, and<br />
therefore has nothing to do with magic symlinks or vz-rpms, etc.<br />
<br />
== Moving a VE on the same virt ==<br />
<br />
Easy way:<br><br />
Scenario 1: ve 123 is to be renamed 1231 and moved from vz1 to vz<br />
<br />
vzmlocal 123:1231:/vz/private/1231:/vz/root/1231<br />
<br />
Scenario 2: ve 123 is to be moved vz1 to vz<br />
<br />
vzmlocal 123:123:/vz/private/123:/vz/root/123<br />
<br />
vzmlocal will reboot the ve at the end of the move<br />
<br />
Manual/old way:<br />
<br />
1) <tt>vzctl stop 123</tt><br><br />
2) <tt>mv /vz1/private/123 /vz/private/.</tt><br><br />
(or cp -a if you want to copy)<br />
3) in <tt>/etc/sysconfig/vz-scripts/123.conf</tt> change value<br><br />
of '<tt>VE_PRIVATE</tt>' variable to point to a new private area location<br />
4) <tt>vzctl start 123</tt><br><br />
5) update backups if needed: <tt>mvbackups 123 virtX virt1 vz</tt><br><br />
6) update management scerens<br><br />
<br />
Notes: a) absolute path to private area is stored in quota file <tt>/var/vzquota/quota.123</tt> - so during first startup quota will be recalculated.<br><br />
b) if you're going to write some script to do a job, you MUST be sure that $VEID won't be expanded to '' in ve config file - ie. you need to escape '$'. Otherwise you might have:<br />
<br />
VE_PRIVATE="/vz/private/"<br />
<br />
in config, and 'vzctl destroy' for this VE ID '''will remove everything under /vz/private/ directory'''.<br />
<br />
== Adding a veth device to a VE ==<br />
<br />
Not totally sure what this is, but a customer asked for it and here's what we did (as instructed by vz support):<br />
<br />
<pre>v set 99 --netif_add eth99 --save<br />
ipdel 99 69.55.230.58<br />
v set 99 --ifname eth99 --ipadd 69.55.230.58 --save<br />
v set 99 --ifname eth99 --gateway 69.55.230.1 --save<br />
<br />
vznetcfg net new veth_net<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
net99 active eth0 veth77.77,veth99.99<br />
veth_net active<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
br0 bridge veth_net<br />
br99 bridge net99<br />
veth99.99 veth net99<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic net99 69.55.227.70/24<br />
<br />
vznetcfg br attach br0 eth0<br />
<br />
(will remove 99 from orig net and move to veth_net)<br />
vznetcfg net addif veth_net veth99.99<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
net99 active<br />
veth_net active eth0 veth77.77,veth99.99<br />
<br />
(delete the old crap)<br />
vznetcfg net del net99<br />
<br />
<br />
Then, to add another device in<br />
<br />
v set 77 --netif_add eth77 --save<br />
ipdel 77 69.55.230.78<br />
v set 77 --ifname eth77 --ipadd 69.55.230.78 --save<br />
v set 77 --ifname eth77 --gateway 69.55.230.1 --save<br />
v set 77 --save --ifname eth77 --network veth_net<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
veth77.77 veth<br />
br0 bridge veth_net<br />
veth99.99 veth veth_net<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic veth_net 69.55.227.70/24<br />
<br />
vznetcfg net addif veth_net veth77.77<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
veth77.77 veth veth_net<br />
br0 bridge veth_net<br />
veth99.99 veth veth_net<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic veth_net 69.55.227.70/24<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
veth_net active eth0 veth77.77,veth99.99<br />
<br />
another example<br />
<br />
v set 1182 --netif_add eth1182 --save<br />
ipdel 1182 69.55.236.217<br />
v set 1182 --ifname eth1182 --ipadd 69.55.236.217 --save<br />
v set 1182 --ifname eth1182 --gateway 69.55.236.1 --save<br />
vznetcfg net addif veth_net veth1182.1182<br />
v set 1182 --save --ifname eth1182 --network veth_net<br />
<br />
<br />
unused/not working commands:<br />
ifconfig veth99.0 0<br />
vznetcfg net list<br />
vznetcfg br new br99 net99<br />
vznetcfg br attach br99 eth0<br />
vznetcfg br show<br />
vznetcfg if list<br />
vznetcfg net addif net99 veth99.99<br />
vznetcfg net addif eth0 net99<br />
<br />
---------<br />
<br />
vznetcfg br new br1182 net1182<br />
<br />
vznetcfg br attach br99 eth0<br />
vznetcfg if list<br />
vznetcfg net addif net99 veth99.99<br />
vznetcfg net addif eth0 net99<br />
<br />
<br />
<br />
vznetcfg net addif eth0 net1182<br />
<br />
<br />
# vznetcfg net addif veth99.0 net99<br />
--- 8< ---<br />
<br />
vznetcfg net new net<br />
# vznetcfg net addif eth0 net99<br />
<br />
<br />
# vzctl set 99 --save --netif_add eth0 (at this stage veth99.0 interface have to appear<br />
on node)<br />
# vzctl set 99 --save --ifname eth0 --ipadd 69.55.230.58 (and probably few more arguments<br />
here - see 'man vzctl')<br />
# vznetcfg net addif veth99.0 net99</pre><br />
<br />
<br />
== Assigning/remove ip from a VE ==<br />
<br />
1. Add or remove ips:<br />
ipdel 1234 1.1.1.1 2.2.2.2<br />
ipadd 1234 1.1.1.1 2.2.2.2<br />
<br />
2. update Mgmt screens<br />
<br />
3. offer to update any DNS we do for them<br />
<br />
4. check to see if we had rules for old IP in firwall<br />
<br />
== Enabling tun device for a ve ==<br />
Note, there’s a command for this: [[#addtun|addtun]]<br />
<br />
For posterity:<br />
Make sure the tun.o module is already loaded before Virtuozzo is started: <br />
lsmod <br />
Allow the VPS to use the TUN/TAP device: <br />
vzctl set 101 --devices c:10:200:rw --save <br />
Create the corresponding device inside the VPS and set the proper permissions: <br />
vzctl exec 101 mkdir -p /dev/net <br />
vzctl exec 101 mknod /dev/net/tun c 10 200 <br />
vzctl exec 101 chmod 600 /dev/net/tun<br />
<br />
== Remaking a system (on same virt) ==<br />
<br />
1. [[#cancelve|cancelve]] (or v destroy x - ONLY if you're POSITIVE no data needs to be saved)<br />
<br />
2. [[#vemake|vemake]] using same veid<br />
<br />
3. [[#mvbackups|mvbackups]] or [[#vb|vb]] (if new mount point)<br />
<br />
4. update mgmt with new dir/ip <br />
<br />
5. update firewall if changed ip<br />
<br />
== Re-initialize quota for a VE ==<br />
<br />
There’s a commamd for this now: [[#clearquota|clearquota]]<br />
<br />
For posterity:<br />
<br />
vzctl stop 1<br />
vzquota drop 1<br />
vzctl start 1<br />
<br />
<br />
== Traffic accounting on linux ==<br />
<br />
DEPRECATED - all tracking is done via bwdb now. This is how we used to track traffic.<br />
<br />
TODO: update for diff versions of vz<br />
<br />
Unlike FreeBSD, where we have to add firewall count rules to the system to count the traffic, on virtuozzo counts the traffic for us. You an see the current traffic stats by running `vznetstat`:<br />
<br />
<pre># vznetstat<br />
VEID Net.Class Output(bytes) Input(bytes)<br />
-----------------------------------------------<br />
24218 1 484M 39M<br />
24245 1 463M 143M<br />
2451 1 2224M 265M<br />
2454 1 2616M 385M<br />
4149 1 125M 68M<br />
418 1 1560M 34M<br />
472 1 1219M 315M<br />
726 1 628M 317M<br />
763 1 223M 82M<br />
771 1 4234M 437M<br />
-----------------------------------------------<br />
Total: 13780M 2090M</pre><br />
<br />
As you can see the VEID is on a line with the in and out bytes. So, we simply run a cron job:<br />
<br />
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /root/vztrafdump.sh<br />
<br />
Just like we do on FreeBSD - this one goes through all the VEs in /vz/private and greps the line from vznetstat that matches them and dumps it in /jc_traffic_dump on their system. Then it does it again for all the VEs in /vz1/private. It is important to note that vznetstat runs only once, and the grepping is done from a temporary file that contains that output - we do this because running vznetstat once for each VE that we read out of /vz/private and /vz1/private would take way too long and be too intensive.<br />
<br />
You do not need to do anything to facilitate this other than make sure that that cron job is running - the vznetstat counters are always running, and any new VEs that are added to the system will be accounted for automatically.<br />
<br />
Traffic resetting no longer works with vz 2.6, so we disable the vztrafdump.sh on those virts.<br />
<br />
== Watchdog script ==<br />
<br />
On some of the older virts, we have a watchdog running that kills procs that are deemed bad per the following:<br />
<br />
/root/watchdog from quar1<br />
<br />
<pre>if echo $line | awk '{print $(NF-3)}' | grep [5-9]...<br />
then<br />
# 50-90%<br />
if echo $line | awk '{print $(NF-1)}' | grep "...:.."<br />
then<br />
# running for > 99min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
if echo $line | awk '{print $(NF-1)}' | grep "....m"<br />
then<br />
# running for > 1000min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
fi<br />
<br />
if echo $line | awk '{print $(NF-3)}' | grep [1-9]...<br />
then<br />
# running for 10-90 percent<br />
if echo $line | awk '{print $NF}' | egrep 'cfusion|counter|vchkpw'<br />
then<br />
<br />
if echo $line | awk '{print $(NF-1)}' | grep "[2-9]:.."<br />
then<br />
# between 2-9min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
elif echo $line | awk '{print $(NF-1)}' | grep "[0-9][0-9]:.."<br />
then<br />
# up to 99min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
fi<br />
fi</pre><br />
<br />
<br />
== Misc Linux Items ==<br />
<br />
We are overselling hard drive space ... when you configure a linux system with a certain amount of disk space (the default is 4gigs) you do not actually use up 4gigs of space on the system. The diskspace setting for a user is simply a cap, and they only use up as much space on the actual disk drive as they are actually using.<br />
<br />
When you create a new linux system, even though there are some 300 RPMs or so installed, if you run `df -k` you will see that the entire 4gig partition is empty - no space is being used. This is because the files in their system are "magic symlinks" to the template for their OS that is in /vz/template - however, any changes to any of those files will "disconnect" them and they will immediately begin using space in their system. Further, any new files uploaded (even if those new files overwrite existing files) will take up space on the partition.<br />
<br />
<br />
if you see this:<br />
<br />
<pre>[root@virt8 root]# vzctl stop 160 ; vzctl start 160<br />
VE is not running<br />
Starting VE ...<br />
VE is unmounted<br />
VE is mounted<br />
Adding IP address(es): 69.55.226.83 69.55.226.84<br />
bash ERROR: Can't change file /etc/sysconfig/network<br />
Deleting IP address(es): 69.55.226.83 69.55.226.84<br />
VE is unmounted<br />
[root@virt8 root]#</pre><br />
<br />
it probably means they no longer have /bin/bash - copy one in for them<br />
<br />
ALSO: another possibility is that they have removed the `ed` RPM from their system - it needs to be reinstalled into their system. But since their system is down, this is tricky ...<br />
<br />
VE startup scripts used by 'vzctl' want package 'ed' to be available inside VE. So if package 'ed' will be enabled in OS template config and OS template itself VE #827 is based on - this error should be fixed.<br />
<br />
yes, it is possible to add RPM to VE while it not running.<br />
Try to do following:<br />
<br />
<pre># cd /vz/template/<OS_template_with_ed_package>/<br />
# vzctl mount 827<br />
# rpm -Uvh --root /vz/root/827 --veid 827 ed-0.2-25.i386.vz.rpm</pre><br />
<br />
Usually theres an error, but its ok<br />
<br />
Note: replace 'ed-0.2-25.i386.vz.rpm' in last command with actual<br />
version of 'ed' package you have.<br />
<br />
----<br />
<br />
So how do I know what template the user has ? cat their conf file and it is listed in there. For example, if the conf file has:<br />
<br />
<pre>[root@virt12 sbin]# vc 1103<br />
…snip…<br />
OSTEMPLATE="debian-3.0/20030822"<br />
TEMPLATES="mod_perl-deb30/20030707 mod_ssl-deb30/20030703 mysql-deb30/20030707 proftpd-deb30/20030703 webmin-deb30/20030823 "<br />
…</pre><br />
<br />
then they are on debian 3.0, all of their system RPMs are in /vz/template/debian-3.0, and they are using version 20030822 of that debian 3.0 template. Also, they’ve also got additional packages installed (mod_perl, mod_ssl, etc). Those are also found under /vz/template<br />
<br />
----<br />
<br />
Edits needed to run java:<br />
<br />
When we first created the VEs, the default setting for privvmpages was 93000:94000 ... which was high enough that most people never had problems ... however, you can;t run java or jdk or tomcat or anything java related with that setting. We have found that by setting privvmpages to 610000:615000 that java runs just fine. That is now the default setting. It is exceedingly rare that anyone needs it higher than that, although we have seen it once or twice.<br />
<br />
Any problems with java at all - the first thing you need to do is see if the failcnt has raised for privvmpages.<br />
<br />
<pre># vzctl start 160<br />
Starting VE ...<br />
vzquota : (error) Quota on syscall for 160: Device or resource busy<br />
Running vzquota on failed for VE 160 [3]</pre><br />
<br />
This is because my pwd is _in_ their private directory - you can't start it until you move out<br />
<br />
People seem to have trouble with php if they are clueless newbies. Here are two common problems/solutions:<br />
<br />
no... but i figured it out myself. problem was the php.ini file that came<br />
vanilla with the account was not configured to work with apache (the<br />
ENGINE directive was set to off).<br />
<br />
everything else seems fine now.<br />
<br />
----<br />
<br />
the problem was in the php.ini file. I noticed that is wasnt showing<br />
the code when it was in an html file so I looked at the php.ini file<br />
and had to change it so it recognized <? tags aswell as <?php tags.<br />
<br />
Also, make sure added to httpd.conf<br />
AddType application/x-httpd-php .php<br />
<br />
----<br />
<br />
You can set the time zone:<br />
<br />
You can change the timezone by doing this:<br />
<br />
ln -sf /usr/share/zoneinfo/<zone> /etc/localtime<br />
<br />
where <zone> is the zone you want in the /usr/share/zoneinfo/ directory.<br />
<br />
----<br />
<br />
Failing shm_open calls:<br />
<br />
first, please check if /dev/shm is mounted inside VE.<br />
'cat /proc/mounts' command should show something like this:<br />
tmpfs /dev/shm tmpfs rw 0 0<br />
<br />
If /dev/shm is not mounted you have 2 ways to solve issue:<br />
1. execute following command inside VE (doesn't require VE reboot):<br />
mount -t tmpfs none /dev/shm<br />
2. add following string to /etc/fstab inside VE and reboot it:<br />
tmpfs /dev/shm tmpfs defaults 0 0<br />
<br />
----<br />
<br />
You can have a mounted but not running ve<br />
Just:<br />
vzctl mount <veid><br />
<br />
----<br />
<br />
When a debian sys can’t get on the network, and you try:<br />
<br />
<pre>vzctl set 1046 --ipadd 69.55.227.117<br />
Adding IP address(es): 69.55.227.117<br />
Failed to bring up lo.<br />
Failed to bring up venet0.</pre><br />
<br />
They probably removed iproute package, which must be the one from swsoft. To restore:<br />
<pre># dpkg -i --veid=1046 --admindir=/vz1/private/1046/root/var/lib/dpkg --instdir=/vz1/private/1046/root/ /vz/template/debian-3.0/iproute_20010824-8_i386.vz.deb<br />
(Reading database ... 16007 files and directories currently installed.)<br />
Preparing to replace iproute 20010824-8 (using .../iproute_20010824-8_i386.vz.deb) ...<br />
Unpacking replacement iproute ...<br />
Setting up iproute (20010824-8) ...</pre><br />
<br />
Then restart their ve<br />
<br />
----<br />
<br />
in a ve i do:<br />
<br />
<pre>cd /<br />
du -h .</pre><br />
<br />
and get: 483M .<br />
<br />
i do:<br />
<br />
<pre>bash-2.05a# df -h<br />
Filesystem Size Used Avail Use% Mounted on<br />
/dev/vzfs 4.0G 2.3G 1.7G 56% /</pre><br />
<br />
how can this be?<br />
<br />
Is it possible that quota file was corrupted somehow? Please try to: <br />
<pre>vzctl stop <VEID><br />
vzquota drop <VEID><br />
vzquota init <VEID><br />
vzctl start <VEID></pre><br />
<br />
----<br />
<br />
How to stop vz from starting after reboot:<br />
<br />
VIRTUOZZO=no <br />
in <br />
/etc/sysconfig/vz<br />
<br />
To start: <br />
service vz start<br />
(after setting VIRTUOZZO=yes in /etc/sysconfig/vz)<br />
<br />
service vz restart will do some kind of 'soft reboot' -- restart all<br />
VPSes and reload modules without rebooting the node<br />
<br />
if you need to shut down all VPSes really really fast, run killall -9 init<br />
<br />
----<br />
<br />
Postfix tip:<br />
<br />
You may want to tweak settings: default_process_limit=10<br />
<br />
= Virtuozzo VPS Management Tools =<br />
<br />
== vm ==<br />
<br />
TODO<br />
<br />
== cancelve ==<br />
cancelve <veid><br />
this will:<br />
* stop a ve<br />
* check for backups (offer to remove them from the backup server <br />
and the backup.config)<br />
* rename the private dir<br />
* check for PTR, provide the commands to reset to default<br />
* and rename the ve’s config<br />
* remind you to remove firewall rules<br />
* remind you to remove DNS entries<br />
<br />
== ipadd ==<br />
ipadd <veid> <ip> [ip] [ip]<br />
add’s ip(s) to a ve<br />
<br />
== ipdel ==<br />
ipdel <veid> <ip> [ip] [ip]<br />
removes ip(s) from a ve<br />
<br />
== vc ==<br />
vc <veid><br />
the equivalent of: <tt>cat /vzconf/<veid>.conf</tt><br />
<br />
== vl ==<br />
vl<br />
will displays a list of ve #’s, 1 per line. (ostensibly to use in a for loop)<br />
<br />
== vp ==<br />
vp <veid><br />
the equivalent of: <tt>vzps auxww –E <veid></tt><br />
<br />
== vpe ==<br />
DEPRECATED<br />
vpe <veid> <br />
this will allow you to do a vp when a ve is running out of control, the equivalent of (deprecated since vp operates outside the VPS): <br />
<pre>vzctl set <veid> --kmemsize 2100000:2200000<br />
vzctl exec <veid> ps auxw<br />
vzctl set <veid> --kmemsize (ve’s orig lvalue):(ve’s orig hvalue)</pre><br />
<br />
== vt ==<br />
vt <veid><br />
the equivalent of: <tt>vztop –E <veid></tt><br />
<br />
== vr ==<br />
vr <veid><br />
the equivalent of: <tt>vzctl stop <veid>; vzctl start <veid></tt><br />
You can run this even if the ve is down - the stop command will just fail<br />
<br />
== vs ==<br />
vs [veid]<br />
displays status (output of <tt>vzctl status <veid></tt>) on each ve configured on the system (as determined by <tt>/vzconf/*.conf</tt>)<br />
If passed an argument, gives the status for just that ve. <br />
A running system looks like:<br />
<tt>VEID 16066 exist mounted running</tt><br />
<br />
A ve which is not running (but does exist) looks like:<br />
<tt>VEID 9990 exist unmounted down</tt><br />
<br />
A ve which is not running and doesn’t exist looks like:<br />
<tt>VEID 421 deleted unmounted down</tt><br />
<br />
== vs2 ==<br />
vs2 [veid]<br />
this is similar to vs in that it displays status (output of <tt>vzctl status <veid></tt>) on each ve,<br />
but the difference is it’s list comes from doing an ls on the data dirs. This was meant to catch <br />
the rare case where a ve configured but exists. <br />
<br />
== vw ==<br />
vw [veid]<br />
displays the output of ‘<tt>w</tt>’ (the equivalent of <tt>vzctl exec <veid> w</tt>) for each configured ve (as determined by <tt>/vzconf/*.conf</tt>). Useful for determing which ve is contributing to a heavily-loaded system.<br />
If passed an argument, gives ‘<tt>w</tt>‘ output for just that ve. <br />
Ex:<br />
<pre>[root@virt2 etc]# vw<br />
134<br />
10:52pm up 79 days, 6:14, 0 users, load average: 0.02, 0.02, 0.00<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT<br />
16027<br />
2:52pm up 7 days, 19:54, 0 users, load average: 0.00, 0.00, 0.00<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT<br />
16055<br />
2:52pm up 79 days, 6:38, 0 users, load average: 0.00, 0.04, 0.07<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT</pre><br />
<br />
== vwe ==<br />
vwe [constraint]<br />
just like <tt>vw</tt>, but takes a constraint as an argument, only show’s ve’s with loads >= the constraint provided. If no constraint is provided, 1 is used by default<br />
<br />
== vzs ==<br />
vzs [veid]<br />
displays the beancounter status for all ve’s, or a particular ve if an argument is passed<br />
<br />
== ve ==<br />
ve <veid><br />
the equivalent of: <tt>vzctl enter <veid></tt><br />
<br />
== vx ==<br />
vx <veid> <command><br />
the equivalent of: <tt>/usr/sbin/vzctl exec <veid> <command></tt><br />
<br />
== dprocs ==<br />
dprocs [count]<br />
a script which outputs a continuous report (or a certain number of reports if an option is passed) of processes stuck in the D state and which VPS’s those procs belong to.<br />
<br />
== setmem ==<br />
setmem <VEID> <256|512|768|1024|2048><br />
adjusts the memory resources for the VE<br />
<br />
== afacheck.sh ==<br />
afacheck.sh<br />
displays the health/status of containers and mirrors on an adaptec card (currently quar1, tempvirt1-2, virt9, virt10)- all other are LSI<br />
<br />
== backup ==<br />
backup<br />
backup script called nightly to update virt scripts and do customer backups<br />
<br />
== checkload.pl ==<br />
checkload.pl<br />
this was intended to be setup as a cronjob to watch processes on a virt when the load <br />
rises above a certain level. Not currently in use.<br />
<br />
== findbackuppigs.pl ==<br />
findbackuppigs.pl<br />
looks for files larger than 50MB which customers have asked us to backup. Emails matches<br />
to linux@johncompanies.com<br />
<br />
== gatherlinux.pl ==<br />
gatherlinux.pl<br />
gathers up data about ve’s configured and writes to a file. Used for audits against the db<br />
<br />
== gb ==<br />
gb <search><br />
greps backup.config for the given search parameter<br />
<br />
== gbg ==<br />
gbg <search><br />
greps backup.config for the given search parameter and presents just the directories (for clean pasting)<br />
<br />
== linuxtrafficgather.pl ==<br />
linuxtrafficgather.pl [yy-mm]<br />
sends a traffic usage summary by ve to support@johncomapnies.com and payments@johncompanies.com.<br />
Optional arguments are year and month (must be in the past). If not passed, assumes last month. Relies on <br />
traffic logs created by netstatreset and netstatbackup<br />
<br />
== linuxtrafficwatch.pl ==<br />
linuxtrafficwatch.pl<br />
checks traffic usage and emails support@johncomapnies.com when a ve reaches the warning level (35G) and<br />
the limit (40G). Works on virtuozzo versions <= 2.6.x. We really aren’t using this anymore now that we have netflow.<br />
<br />
== linuxtrafficwatch2.pl ==<br />
linuxtrafficwatch2.pl<br />
checks traffic usage and emails support@johncomapnies.com when a ve reaches the warning level (35G) and<br />
the limit (40G). Works on virtuozzo version 2.6.x. We really aren’t using this anymore now that we have netflow.<br />
<br />
== load.pl ==<br />
load.pl<br />
feeds info to load mrtg - executed by inetd.<br />
<br />
== mb (linux) ==<br />
mb <mount|umount><br />
(nfs) mounts and umounts dirs to backup2. Shortcuts are mbm and mbu to mount and unmount. <br />
<br />
== migrate ==<br />
migrate <ip of node migrating to> <veid> <target_veid> [target dir: vz | vz1 | vz2]<br />
this is basically a wrapper for <tt>vzmigrate</tt> – vzmigrate is a util to seamlessly move a ve from one host to another. This wrapper was written cause vz virtuozzo version 2.6 had a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables. This script mitigates that. Since it makes multiple ssh connections to the target host, it’s a good idea to put the pub key for the src system in the authorized_keys file on the target host. In addition, it emails ve owners when their migration starts and stops (if they place email addresses in a file on their system: /migrate_notify). To move everyone off a system, you’d do:<br />
for f in `vl`; do migrate <ip> $f; done<br />
<br />
== migrateonline ==<br />
migrateonline <ip of node migrating to> <veid> <target_veid> [target dir: vz | vz1 | vz2]<br />
this is the same as migrate but will migrate a ve in <tt>–online</tt> mode which means it won’t be shut down at the end of the migration. This only works when migrating ve’s between 2 machines running a 2.6 kernel (currently tempvirt1-2. virt16-19, virt12). If you get an error that the machine you’re trying to migrate to has a different CPU or features, etc, then you have to edit the file and add the –f switch to the vzmigrate line- you can basically ignore this kind of warning (but never ignore a warning about missing templates on the destination node). NOTE: This edit (if made to migrateonline) will be overwritten by the base script during each night’s backup.<br />
<br />
== netstatbackup ==<br />
DEPRECATED<br />
netstatbackup <br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.5.x. <br />
<br />
== netstatbackup2 ==<br />
DEPRECATED<br />
netstatbackup2<br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.6.x. <br />
<br />
== netstatreset ==<br />
DEPRECATED<br />
netstatreset<br />
writes traffic count data to a logfile and resets counters to 0. Works on virtuozzo versions 2.5.x <br />
<br />
== netstatreset2 ==<br />
DEPRECATED<br />
netstatreset2<br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.6.x. <br />
<br />
== orphanedbackupwatchlinux ==<br />
orphanedbackupwatchlinux <br />
looks for directories on backup2 which aren’t configured in backup.config and offers to <br />
delete them<br />
<br />
== rsync.backup (linux) ==<br />
rsync.backup<br />
does customer backups (relies on backup.config)<br />
<br />
== startvirt.pl ==<br />
startvirt.pl<br />
forks off start ve commands – keeps 6 running at a time. This is not to be used on systems where fastboot is enabled as it circumvents the benefit of the fastboot. The script will occasionally not exit gracefully and will continue to use up CPU, so it should be watched. Also, don’t exit from the script till you’re sure all ve’s are started – if you do you need to start them manually and may have to free up locks. On some systems, the startvirt script doesn’t exit cleanly and you have to ^C out of it. Be careful though- doing so can leave some VE’s in an odd bootup state and you may need to ‘vr’ them manually. You should check to see which ve’s aren’t running and/or confirm all have started when ^C’ing out of startvirt.<br />
<br />
== taskdone (linux) ==<br />
taskdone<br />
when called will email support@johncompanies.com with the hostname of the machine from which it was <br />
executed as the subject<br />
<br />
== vb (linux) ==<br />
vb<br />
the equivalent of: <tt>vi /usr/local/sbin/backup.config</tt><br />
<br />
== vemakeXX ==<br />
DEPRECATED<br />
vemakerh9 <br />
ve create script for RH9 (see vemake)<br />
<br />
vemakedebian30 <br />
ve create script for debian 3.0 (Woody) (see vemake)<br />
<br />
vemakedebian31 <br />
ve create script for debian 3.1 (Sarge) (see vemake)<br />
<br />
vemakedebian40 <br />
ve create script for debian 4.0 (Etch) (see vemake)<br />
<br />
vemakefedora, vemakefedora2, vemakefedora4, vemakefedora5, vemakefedora6, vemakefedora7<br />
ve create script for fedora core 1, 2, 4, 5, 6, 7 respectively (see vemake)<br />
<br />
vemakecentos3, vemakecentos4<br />
ve create script for centos 3, 4 respectively (see vemake)<br />
<br />
vemakesuse, vemakesuse93, vemakesuse100<br />
ve create script for suse 9.2, 9.3, 10.0 respectively (see vemake)<br />
<br />
vemakeubuntu5, vemakeubuntu606, vemakeubuntu606 vemakeubuntu704<br />
ve create script for ubuntu 5.10, 6.06, 6.10, 7.04 respectively (see vemake)<br />
<br />
== vemove ==<br />
DEPRECATED<br />
vemove <veid> <target_ip> </vz/private/123><br />
this script simplifies the old way of moving ve’s from one system to another - in short moving a ve to or from a virt running virtuozzo < 2.6.x<br />
It’s the equivalent of:<br />
<tt>tar cfpP - <veid> --ignore-failed-read | (ssh -2 -c arcfour <target_ip> "split - -b 1024m </vz/private/123>.tar" )</tt>This should only be used if migrate/vzmigrate can’t be used. <br />
<br />
== vim.watchdog ==<br />
vim.watchdog <br />
looks for and kills procs matching “vi|vim|nano|pine|elm” that are running for a long time & consuming lots of cpu. Works on virtuozzo versions 2.5.x<br />
<br />
== vim.watchdog2 ==<br />
vim.watchdog2<br />
looks for and kills procs matching “vi|vim|nano|pine|elm” that are running for a long time & consuming lots of cpu.<br />
Works on virtuozzo versions 2.6.x.<br />
<br />
== vzmigrate ==<br />
vzmigrate <target_ip> -r no <veid>:[dst veid]:[dst /vzX/private/veid]:[dst /vzX/root/veid]<br />
(this is the raw command “wrapped” by migrate/migrateonline) this will seamlessly move a ve from one host to another. The ve will run for the duration of the migration till the very end when it’s shut down, ip moved and started up on the target system. The filesystem on the src will remain. This should be watched – occasionally the move will timeout and leave the system shut down. If target private and root aren’t specified it just puts it in /vz. Only works when both systems are running virtuozzo 2.6.x<br />
<br />
== vztrafdump.sh ==<br />
DEPRECATED<br />
vztrafdump.sh<br />
writes traffic usage info by ve to a file called jc_traffic_dump in each ve’s / dir. Works on virtuozzo versions <= 2.5.x. <br />
<br />
== vztrafdump2.sh ==<br />
DEPRECATED<br />
vztrafdump2.sh<br />
writes traffic usage info by ve to a file called jc_traffic_dump in each ve’s / dir. Works on virtuozzo versions 2.6.x. <br />
<br />
== addtun ==<br />
addtun <veid><br />
Add’s tun device to ve.<br />
<br />
== bwcap ==<br />
bwcap <veid> <kbps><br />
Ex: <tt>bwcap 1234 512</tt><br />
Caps a VE’s bandwidth to the amount given<br />
<br />
== setdisk ==<br />
setdisk <veid> <diskspace in GB><br />
Ex: <tt>setdisk 1234 5</tt><br />
Gives a VE’s a given amount of disk space<br />
<br />
== vdf ==<br />
vdf <veid> <br />
the equivalent of: <tt>vzctl exec <veid> df –h</tt><br />
<br />
== vdff ==<br />
vdff<br />
runs a (condensed) vdf for all ve’s in your pwd (must be run from /vz/privateN)<br />
<br />
== mvbackups ==<br />
mvbackups <veid> <target_machine> (virt1) <target_dir> (vz1)<br />
moves backups from one location to another on the backup server, and provides you with option to remove entries from current backup.config, and simple paste command to add the config to backup.config on the target server<br />
<br />
== checkquota ==<br />
checkquota<br />
for all the ve’s in the cwd (run from /vz/private, /vz1/private, etc) reports what vz quota says they’re using and what the actual usage is (as reported by du)<br />
<br />
== clearquota ==<br />
clearquota <veid><br />
Recalculates a ve’s quota, prints out the usage before and after. The equivalent of:<br />
<tt>vdf <veid>; v stop <veid>; vzquota drop <veid>; v start <veid>; vdf <veid></tt><br />
<br />
== dprocs ==<br />
dprocs<br />
Sometimes the server’s have a large number of processes get stuck in the D state- this script shows (every 3 secs) which VE’s have D procs, which procs<br />
are stuck and a running average of the top “offenders”<br />
<br />
== vzstat ==<br />
vstat<br />
sort of like top for VZ. sort VEs by CPU usage by pressing 'o' and then 'c' keys<br />
<br />
== stopvirt ==<br />
stopvirt<br />
will stop VEs as fast as it can, 6 at a time. May not exit when complete so you should watch [[#vzstat|vzstat]] in another window.</div>Supporthttps://wiki.jcihosting.com/index.php?title=VPS_Management&diff=1022VPS Management2013-03-11T23:13:43Z<p>Support: /* migrate manually (if migrate/online fails) */</p>
<hr />
<div>= Common Problems =<br />
== Login to any machine without a password ==<br />
<br />
This is possible via the use of ssh keys. The process is thus:<br />
<br />
1. place the public key for your user (root@mail) in the /root/.ssh/authorized_keys file on the server you wish to login to<br />
cat /root/.ssh/id_dsa.pub<br />
(paste that into authorized_keys on the target server). If the file doesn't exist, create it.<br />
<br />
2. enable root login (usually only applies to FreeBSD). Edit the /etc/ssh/sshd_config on the target server and change:<br />
<tt>#PermitRootLogin no</tt><br />
to<br />
<tt>PermitRootLogin yes</tt><br />
<br />
3. Restart the sshd on the target machine. First, find the sshd process: <br />
jailps <hostname> | grep sshd <br />
or <br />
vp <VEID> | grep sshd<br />
<br />
Look for the process resembling:<br />
root 17296 0.0 0.0 5280 1036 ? Ss 2011 4:27 /usr/sbin/sshd <br />
(this is the sshd)<br />
<br />
Not:<br />
root 6270 0.5 0.0 6808 2536 ? Ss 14:33 0:00 sshd: root [priv]<br />
(this is an sshd child- someone already ssh'd in as root)<br />
<br />
Restart the sshd: <br />
kill -1 <PID><br />
<br />
Ex:<br />
kill -1 17296<br />
<br />
You may now ssh in.<br />
<br />
Once you're done, IF you enabled root login, you should repeat steps 2 and 3 to disable root logins.<br />
<br />
== Letting someone in who has locked themselves out (killed sshd, lost pwd) ==<br />
<br />
There are two ways people frequently lock themselves out - either they forget a password, or they kill off sshd somehow.<br />
<br />
These are actually both fairly easy to solve. First, let's say someone kills off their sshd, or somehow mangles /etc/ssh/sshd_config such that it no longer lets them in.<br />
<br />
Their email may be very short, or it may have all sorts of details about how you should fix sshd_config to let them in ... just ignore all of this. They can fix their own mangled sshd. Fixing this is very simple. First, edit the /etc/inetd.conf on their system and uncomment the telnet line:<br />
<br />
telnet stream tcp nowait root /usr/libexec/telnetd telnetd<br />
#telnet stream tcp6 nowait root /usr/libexec/telnetd telnetd<br />
<br />
(just leave the tcp6 version of telnet commented)<br />
<br />
Then, use jailps to list the processes on their system, and find their inetd process. Then simply:<br />
<br />
kill -HUP (pid)<br />
<br />
where (pid) is the PID of their inetd process. Now they have telnet running on their system and they can log in and do whatever they need to do.<br />
<br />
The only complications that could occur are:<br />
<br />
a) their firewall config on our firewall has port 23 blocked, in which case you will need to open that - will be covered in a different lesson.<br />
<br />
b) they are not running inetd, so you can't HUP it. If this happens, edit their /etc/rc.conf, add the inetd_enable="YES" line, and then kill<br />
their jail with /tmp/jailkill.pl - then restart their jail with the jail line from their quad/safe file. Easy.<br />
<br />
If they have forgotten a password,<br />
<br />
On 6.x+ you can reset their password with:<br />
jexec <jailID from jls> passwd root<br />
<br />
Note: the default password for 6.x jails is 8ico2987, for 4.x it is p455agfa<br />
<br />
On 4.x, you need to cd to their etc directory<br />
... for instance:<br />
<br />
cd /mnt/data2/198.78.65.136-col00261-DIR/etc<br />
<br />
and run:<br />
<br />
vipw -d .<br />
<br />
Then paste in these two lines (theres a paste with these):<br />
<br />
root:$1$krszPxhk$xkCepSnz3mIikT3vCtJCt0:0:0::0:0:Charlie &:/root:/bin/csh<br />
user:$1$Mx9p5Npk$QdMU6c8YQqp2FW2M3irEh/:1001:1001::0:0:User &:/home/user:/bin/sh<br />
<br />
overwriting the lines they already have for "user" and "root" - then just tell them that both user and root have been reset to the default password of p455agfa.<br />
<br />
For linux, just passwd inside shell or <br />
vzctl set <veid> --userpasswd root:p455agfa –save<br />
<br />
Starting in 2009 we began giving out randomized passwords for FreeBSD and Linux as the default password. That is stored with each system in Mgmt. You should look for and reset the password to that password in the event of a reset and refer the customer to use their original password from their welcome email- this way we don’t have to send the password again via email (in clear text).<br />
<br />
<br />
== sendmail can’t be contacted from ext ip (only locally) ==<br />
<br />
By default redhat puts this line in sendmail.mc:<br />
<br />
DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')<br />
<br />
which makes it only answer on localhost. Comment it out like:<br />
<br />
dnl DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')<br />
<br />
and then rebuild sendmail.cf with:<br />
<br />
m4 /etc/mail/sendmail.mc > /etc/sendmail.cf<br />
<br />
== virt doesn’t properly let go of ve’s ip(s) when moved to another system ==<br />
<br />
On virtuozzo 2.6 systems, it's been observed that when moving ips from one virt to another that sometimes the routing table will not get updated to reflect the removal of the ip addresses.<br />
<br />
A recent example was a customer that was moving to a new ve on a new virt and the ip addresses were traded between the two ve's. After the trade the two systems were not able to talk to each other. When looking at the routing table for the old system all the ip addresses were still in the routing table as being local, like so:<br />
<br />
<pre>netstat -rn | grep 69.55.225.149<br />
69.55.225.149 0.0.0.0 255.255.255.255 UH 40 0 0 venet0</pre><br />
<br />
This was preventing traffic to the other system from being routed properly.<br />
The solution is to manually delete the route:<br />
<br />
route delete 69.55.225.149 gw 0.0.0.0<br />
<br />
Supposedly, this was fixed in 2.6.1<br />
<br />
== sshd on FreeBSD 6.2 segfaults ==<br />
<br />
First try to reinstall ssh<br />
<br />
<pre>cd /usr/src/secure<br />
cd lib/libssh<br />
make depend && make all install<br />
cd ../../usr.sbin/sshd<br />
make depend && make all install<br />
cd ../../usr.bin/ssh<br />
make depend && make all install</pre><br />
<br />
Failing that, find the library that’s messed up:<br />
<br />
<pre>ldd /usr/sbin/sshd<br />
libssh.so.3 => /usr/lib/libssh.so.3 (0x280a3000) <br />
libutil.so.5 => /lib/libutil.so.5 (0x280d8000) <br />
libz.so.3 => /lib/libz.so.3 (0x280e4000) <br />
libwrap.so.4 => /usr/lib/libwrap.so.4 (0x280f5000) <br />
libpam.so.3 => /usr/lib/libpam.so.3 (0x280fc000) <br />
libbsm.so.1 => /usr/lib/libbsm.so.1 (0x28103000) <br />
libgssapi.so.8 => /usr/lib/libgssapi.so.8 (0x28112000) <br />
libkrb5.so.8 => /usr/lib/libkrb5.so.8 (0x28120000) <br />
libasn1.so.8 => /usr/lib/libasn1.so.8 (0x28154000) <br />
libcom_err.so.3 => /usr/lib/libcom_err.so.3 (0x28175000) <br />
libroken.so.8 => /usr/lib/libroken.so.8 (0x28177000) <br />
libcrypto.so.4 => /lib/libcrypto.so.4 (0x28183000) <br />
libcrypt.so.3 => /lib/libcrypt.so.3 (0x28276000) <br />
libc.so.6 => /lib/libc.so.6 (0x2828e000) <br />
libmd.so.3 => /lib/libmd.so.3 (0x28373000)</pre><br />
<br />
md5 them and compare to other jail hosts or jails running on host<br />
<br />
for libcrypto reinstall:<br />
<pre>/usr/src/crypto<br />
make depend && make all install</pre><br />
<br />
= FreeBSD VPS =<br />
<br />
== Starting jails: Quad/Safe Files ==<br />
<br />
FreeBSD customer systems do not start up automatically at boot time. When one of our freebsd machines boots up, it boots up, and does nothing else. To start jails, we put the commands to start each jail into a shell script(s) and run the script(s). Jail startup is something that needs to be actively monitored, which is why we don’t just run the script automatically. More on monitoring later.<br />
<br />
NOTE: >=7.x we have moved to 1 quad file: <tt>quad1</tt>. Startups are not done by running each quad, but rather [[#startalljails|startalljails]] which relies on the contents of <tt>quad1</tt>. The specifics of this are lower in this article. What follows here applies for pre 7.x systems.<br />
<br />
There are eight files in <tt>/usr/local/jail/rc.d</tt>:<br />
<br />
<pre>jail3# ls /usr/local/jail/rc.d/<br />
quad1 quad2 quad3 quad4 safe1 safe2 safe3 safe4<br />
jail3#</pre><br />
<br />
four quad files and four safe files.<br />
<br />
Each file contains an even number of system startup blocks (total number of jails divided by 4)<br />
<br />
The reason for this is, if we make one large script to startup all the systems at boot time, it will take too long - the first system in the script will start up right after system boot, which is great, but the last system may not start for another 20 minutes.<br />
<br />
Since there is no way to parralelize this during the startup procedure, we simply open four terminals (in screen window 9) and run each script, one in each terminal. This way they all run simultaneously, and the very last system in each startup script gets started in 1/4th the time it would if there was one large file<br />
<br />
The files are generally organized so that quad/safe 1&2 have only jails from disk 1, and quad/safe 3&4 have jails from disk 2. This helps ensure that only 2 fscks on any disk are going on at once. Further, they are balanced so that all quad/safe’s finish executing around the same time. We do this by making sure each quad/safe has a similar number of jails and represents a similar number of inodes (see js).<br />
<br />
The other, very important reason we do it this way, and this is the reason there are quad files and safe files, is that in the event of a system crash, every single vn-backed filesystem that was mounted at the time of system crash needs to be fsck'd. However, fsck'ing takes time, so if we shut the system down gracefully, we don't want to fsck.<br />
<br />
Therefore, we have two sets of scripts - the four quad scripts are identical to the four safe scripts except for the fact that the quad scripts contain fsck commands for each filesystem.<br />
<br />
So, if you shut a system down gracefully, start four terminals and run safe1 in window one, and safe2 in window 2, and so on.<br />
<br />
If you crash, start four terminals (or go to screen window 9) and run quad1 in window one, and quad2 in window 2, and so on.<br />
<br />
Here is a snip of (a 4.x version) quad2 from jail17:<br />
<br />
<pre>vnconfig /dev/vn16 /mnt/data2/69.55.228.7-col00820<br />
fsck -y /dev/vn16<br />
mount /dev/vn16c /mnt/data2/69.55.228.7-col00820-DIR<br />
chmod 0666 /mnt/data2/69.55.228.7-col00820-DIR/dev/null<br />
jail /mnt/data2/69.55.228.7-col00820-DIR mail1.phimail.com 69.55.228.7 /bin/sh /etc/rc<br />
<br />
# moved to data2 col00368<br />
#vnconfig /dev/vn28 /mnt/data2/69.55.236.132-col00368<br />
#fsck -y /dev/vn28<br />
#mount /dev/vn28c /mnt/data2/69.55.236.132-col00368-DIR<br />
#chmod 0666 /mnt/data2/69.55.236.132-col00368-DIR/dev/null<br />
#jail /mnt/data2/69.55.236.132-col00368-DIR limehouse.org 69.55.236.132 /bin/sh /etc/rc<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
vnconfig /dev/vn22 /mnt/data2/69.55.228.13-col01063<br />
fsck -y /dev/vn22<br />
mount /dev/vn22c /mnt/data2/69.55.228.13-col01063-DIR<br />
chmod 0666 /mnt/data2/69.55.228.13-col01063-DIR/dev/null<br />
jail /mnt/data2/69.55.228.13-col01063-DIR www.widestream.com.au 69.55.228.13 /bin/sh /etc/rc<br />
<br />
# cancelled col00106<br />
#vnconfig /dev/vn15 /mnt/data2/69.55.238.5-col00106<br />
#fsck -y /dev/vn15<br />
#mount /dev/vn15c /mnt/data2/69.55.238.5-col00106-DIR<br />
#chmod 0666 /mnt/data2/69.55.238.5-col00106-DIR/dev/null<br />
#jail /mnt/data2/69.55.238.5-col00106-DIR mail.azebu.net 69.55.238.5 /bin/sh /etc/rc</pre><br />
<br />
As you can see, two of the systems specified are commented out - presumably those customers cancelled, or were moved to new servers.<br />
<br />
As you can see, the vnconfig line is the simpler command line, not the longer one that was used when it was first configured. As you can see, all that is done is, vnconfig the filesystem, then fsck it, then mount it. The fourth command is the `jail` command used to start the system – but that will be covered later.<br />
<br />
Here is the safe2 file from jail17:<br />
<br />
<pre>vnconfig /dev/vn16 /mnt/data2/69.55.228.7-col00820<br />
mount /dev/vn16c /mnt/data2/69.55.228.7-col00820-DIR<br />
chmod 0666 /mnt/data2/69.55.228.7-col00820-DIR/dev/null<br />
jail /mnt/data2/69.55.228.7-col00820-DIR mail1.phimail.com 69.55.228.7 /bin/sh /etc/rc<br />
<br />
# moved to data2 col00368<br />
#vnconfig /dev/vn28 /mnt/data2/69.55.236.132-col00368<br />
#mount /dev/vn28c /mnt/data2/69.55.236.132-col00368-DIR<br />
#chmod 0666 /mnt/data2/69.55.236.132-col00368-DIR/dev/null<br />
#jail /mnt/data2/69.55.236.132-col00368-DIR limehouse.org 69.55.236.132 /bin/sh /etc/rc<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
vnconfig /dev/vn22 /mnt/data2/69.55.228.13-col01063<br />
mount /dev/vn22c /mnt/data2/69.55.228.13-col01063-DIR<br />
chmod 0666 /mnt/data2/69.55.228.13-col01063-DIR/dev/null<br />
jail /mnt/data2/69.55.228.13-col01063-DIR www.widestream.com.au 69.55.228.13 /bin/sh /etc/rc<br />
<br />
# cancelled col00106<br />
#vnconfig /dev/vn15 /mnt/data2/69.55.238.5-col00106<br />
#mount /dev/vn15c /mnt/data2/69.55.238.5-col00106-DIR<br />
#chmod 0666 /mnt/data2/69.55.238.5-col00106-DIR/dev/null<br />
#jail /mnt/data2/69.55.238.5-col00106-DIR mail.azebu.net 69.55.238.5 /bin/sh /etc/rc</pre><br />
<br />
As you can see, it is exactly the same, but it does not have the fsck lines.<br />
<br />
Take a look at the last entry - note that the file is named:<br />
<br />
/mnt/data2/69.55.238.5-col00106<br />
<br />
and the mount point is named:<br />
<br />
/mnt/data2/69.55.238.5-col00106-DIR<br />
<br />
This is the general format on all the FreeBSD systems. The file is always named:<br />
<br />
IP-custnumber<br />
<br />
and the directory is named:<br />
<br />
IP-custnumber-DIR<br />
<br />
If you run safe when you need a fsck, the mount will fail and jail will fail:<br />
<br />
# mount /dev/vn1c /mnt/data2/jails/65.248.2.131-ns1.kozubik.com-DIR<br />
mount: /dev/vn1c: Operation not permitted<br />
<br />
No reboot needed, just run the quad script<br />
<br />
Starting with 6.x jails, we added block delimiters to the quad/safe files, the block looks like:<br />
<br />
<pre>echo '## begin ##: nuie.solaris.mu'<br />
fsck -y /dev/concat/v30v31a<br />
mount /dev/concat/v30v31a /mnt/data1/69.55.228.218-col01441-DIR<br />
mount_devfs devfs /mnt/data1/69.55.228.218-col01441-DIR/dev<br />
devfs -m /mnt/data1/69.55.228.218-col01441-DIR/dev rule -s 3 applyset<br />
jail /mnt/data1/69.55.228.218-col01441-DIR nuie.solaris.mu 69.55.228.218 /bin/sh /etc/rc<br />
echo '## end ##: nuie.solaris.mu'</pre><br />
<br />
These are more than just informative when running quad/safe’s, the echo lines MUST be present for certain tools to work properly. So it’s important that any updates to the hostname also be updated on the 2 echo lines. For example, if you try to startjail a jail with a hostname which is on the jail line but not the echo lines, the command will return with host not found.<br />
<br />
=== FreeBSD 7.x+ notes ===<br />
<br />
Starting with the release of FreeBSD 7.x, we are doing jail startups in a slightly different way. First, thereis only 1 file: <tt>/usr/local/jail/rc.d/quad1</tt><br />
There are no other quads or corresponding safe files. The reason for this is twofold, 1. We can pass –C to fsck which will tell is to skip the fsck if the fs is clean (no more need for safe files), 2. We have a new startup script which can be launched multiple times, running in parallel to start jails, where quad1 is the master jail file. <br />
Quad1 could still be run as a shell script, but it would take a very long time for it to run completely so it’s not advisable; or you should break it down into smaller chunks (like quad1, quad2, quad3, etc)<br />
<br />
Here is a snip of (a 7.x version) quad1 from jail2:<br />
<br />
<pre>echo '## begin ##: projects.tw.com'<br />
mdconfig -a -t vnode -f /mnt/data1/69.55.230.46-col01213 -u 50<br />
fsck -Cy /dev/md50c<br />
mount /dev/md50c /mnt/data1/69.55.230.46-col01213-DIR<br />
mount -t devfs devfs /mnt/data1/69.55.230.46-col01213-DIR/dev<br />
devfs -m /mnt/data1/69.55.230.46-col01213-DIR/dev rule -s 3 applyset<br />
jail /mnt/data1/69.55.230.46-col01213-DIR projects.tw.com 69.55.230.46 /bin/sh /etc/rc<br />
echo '## end ##: projects.tw.com'</pre><br />
<br />
Cancelled jails are no longer commented out and stored in quad1, rather they’re moved to <tt>/usr/local/jail/rc.d/deprecated</tt> <br />
<br />
To start these jails, start the 4 ssh sessions as you would for a normal crash and then instead of running quad1-4, instead run startalljails in each window. IMPORTANT- before running startalljails you should make sure you ran preboot once as it will clear out all the lockfiles and enable startalljails to work properly.<br />
<br />
== Problems with the quad/safe files ==<br />
<br />
When you run the quad/safe files, there are two problems that can occur - either a particular system will hang during initialization, OR a system will spit out output to the screen, impeding your ability to do anything. Or both.<br />
<br />
First off, when you start a jail, you see output like this:<br />
<br />
<pre>Skipping disk checks ...<br />
adjkerntz[25285]: sysctl(put_wallclock): Operation not permitted<br />
Doing initial network setup:.<br />
ifconfig: ioctl (SIOCDIFADDR): permission denied<br />
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384<br />
Additional routing options: TCP keepalive=YESsysctl:<br />
net.inet.tcp.always_keepalive: Operation not permitted.<br />
Routing daemons:.<br />
Additional daemons: syslogd.<br />
Doing additional network setup:.<br />
Starting final network daemons:.<br />
ELF ldconfig path: /usr/lib /usr/lib/compat /usr/X11R6/lib /usr/local/lib<br />
a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout /usr/X11R6/lib/aout<br />
Starting standard daemons: inetd cron sshd sendmail sendmail-clientmqueue.<br />
Initial rc.i386 initialization:.<br />
Configuring syscons: blanktime.<br />
Additional ABI support:.<br />
Local package initialization:.<br />
Additional TCP options:.</pre><br />
<br />
Now, let's look at this line, near the end:<br />
<br />
Local package initialization:.<br />
<br />
This is where a list of daemons that are set to start at boot time willshow up. You might see something like:<br />
<br />
Local package initialization: mysqld apache sendmail sendmail-clientmqueue<br />
<br />
Or something like this:<br />
<br />
Local package initialization: postgres postfix apache<br />
<br />
The problem is that many systems (about 4-5 per machine) will hang on that line. Basically it will get to some of the way through the total daemons to be started:<br />
<br />
Local package initialization: mysqld apache<br />
<br />
and will just sit there. Forever.<br />
<br />
Fortunately, pressing ctrl-c will break out of it. Not only will it break out of it, but it will also continue on that same line and start the other daemons:<br />
<br />
Local package initialization: mysqld apache ^c sendmail-clientmqueue<br />
<br />
and then continue on to finish the startup, and then move to the next system to be started.<br />
<br />
So what does this mean? It means that if a machine crashes, and you start four screen-windows to run four quads or four safes, you need to periodically cycle between them and see if any systems are stuck at that point, causing their quad/safe file to hang. A good rule of thumb is, if you see a system at that point in the startup, give it another 100 seconds - if it is still at the exact same spot, hit ctrl-c. Its also a good idea to go back into the quad file (just before the first command in the jail startup block) and note that this jail tends to need a control-c or more time as follows:<br />
<pre>echo '### NOTE ### slow sendmail'<br />
echo '### NOTE ###: ^C @ Starting sendmail.'</pre><br />
<br />
'''NEVER''' hit ctrl-c repeatedly if you don't get an immediate response - that will cause the following jail’s startup commands to be aborted.<br />
<br />
A second problem that can occur is that a jail - maybe the first one in that particular quad/safe, maybe the last one, or maybe one in the middle, will start spitting out status or error messages from one of its init scripts. This is not a problem - basically, hit enter a few times and see if you get a prompt - if you do get a prompt, that means that the quad/safe script has already completed. Therefore it is safe to log out (and log out of the user that you su'd from) and then log back in (if necessary).<br />
<br />
The tricky thing is, if a system in the middle starts flooding with messages, and you hit enter a few times and don't get a prompt. Are you not getting a prompt because some subsequent system is hanging at the initialization, as we discussede above ? Or are you not getting a prompt because that quad file is currently running an fsck ? Usually you can tell by scrolling back in screen’s history to see what it was doing before you started getting the messages.<br />
<br />
If you don’t get clues from history, you have to use your judgement - instead of giving it 100 seconds to respond, perhaps give it 2-3 mins ... if you still get no response (no prompt) when you hit enter, hit ctrl-c. However, be aware that you might still be hitting ctrl-c in the middle of an fsck. This means you will get an error like "filesystem still marked dirty" and then the vnconfig for it will fail and so will the jail command, and the next system in the quad file will then start starting up.<br />
<br />
If this happens, just wait until the end of all the quad files have finished, and start that system manually.<br />
<br />
If things really get weird, like a screen flooded with errors, and you can't get a prompt, and ctrl-c does nothing, then you need to just eventually (give it ten mins or so) just kill that window with ctrl-p, then k, and then log in again and manually check which systems are now running and which aren't, and manually start up any that are not.<br />
<br />
Don't EVER risk running a particular quad/safe file a second time.<br />
If the quad/safe script gets executed twice, reboot the machine immediately.<br />
<br />
So, for all the above reasons, anytime a machine crashes and you run all the quads or all the safes, '''always''' check every jail afterwards to make sure it is running - even if you have no hangs or complications at all.<br />
Run this command:<br />
<br />
<tt>[[#jailpsall|jailpsall]]</tt> <br />
<br />
Note: [[#postboot|postboot]] also populates ipfw counts, so it '''should not be run multiple times''', use <tt>jailpsall</tt> for subsequent extensive ps’ing<br />
<br />
And make sure they all show as running. If one does not show as running, check its /etc/rc.conf file first to see if maybe it is using a different hostname first before starting it manually.<br />
<br />
One thing we have implemented to alleviate these startup hangs and noisy jails, is to put jail start blocks that are slow or hangy at the bottom of the safe/quad file. Further, for each bad jail we note in each quad/safe just before the start block something like:<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
<br />
That way we’ll be prepared to ^C when we see that message appear during the quad/safe startup process. If you observe a new, undocumented hang, '''after''' the quad/safe has finished, place a line similar to the above in the quad file, move the jail start block to the end of the file, then run [[#buildsafe|buildsafe]]<br />
<br />
<br />
== Recovering from a crash (FreeBSD) ==<br />
<br />
=== Diagnose whether you have a crash ===<br />
The most important thing is to get the machine and all jails back up as soon as possible. Note the time, you’ll need to create a crash log entry (Mgmt. -> Reference -> CrashLog). The first thing to do is head over to the [[Screen#Screen_Organization|serial console screen]] and see if there’s any kernel error messages output. Try to copy any messages (or just a sample of repeating messages) you see into the notes section of the crash log. If there are no messages, the machine may just be really busy- wait a bit (5-10min) to see if it comes back. If it's still pinging, odds are its very busy. Note, if you see messages about swap space exhausted, the server is obviously out of memory, however it may recover briefly enough for you to get a jtop in to see who's lauched a ton of procs (most likely) and then issue a quick jailkill to get it back under control.<br />
<br />
If it doesn't come back, or the messages indicate a fatal error, you will need to proceed with a power cycle (ctrl+alt+del will not work).<br />
<br />
=== Power cycle the server ===<br />
If this machine is not a Dell 2950 with a [[DRAC/RMM#DRAC|DRAC card]] (i.e. if you can’t ssh into the DRAC card (as root, using the standard root pass) and issue <br />
racadm serveraction hardreset<br />
then you will need someone at the data center power the macine off, wait 30 sec, then turn it back on. Make sure to re-attach via console:<br />
tip jailX<br />
immediately after power down. <br />
<br />
=== (Re)attach to the console ===<br />
Stay on the console the entire time during boot. As the BIOS posts- look out for the RAID card output- does everything look healthy? The output may be scrambled, look for "DEGRADED" or "FAILED". Once the OS starts booting you will be disconnected (dropped back to the shell on the console server) a couple times during the boot up. The reason you want to quickly re-attach is two-fold: 1. If you don’t reattach quickly then you won’t get any console output, 2. you want to be attached before the server ''potentially'' starts (an extensive) fsck. If you attach after the fsck begins, you’ll have seen no indication it started an fsck and the server will appear frozen during startup- no output, no response. <br />
<br />
IMPORTANT NOTE: on some older FreeBSD systems, there will be no output to the video (KVM) console as it boots up. The console output is redirected to the serial port ... so if a jail crashes, and you attach a kvm, the output during the bootup procedure will not be shown on the screen. However, when the bootup is done, you will get a login prompt on the screen and will be able to log in as normal. <tt>/boot/loader.conf</tt> is where serial console redirect output lives, so comment that if you want to catch output on kvm.<br />
On newer systems it sends most output to both locations. <br />
<br />
=== Assess the heath of the server ===<br />
Once the server boots up fully, you should be able to ssh in. Look around- make sure all the mounts are there and reporting the correct size/usage (i.e. /mnt/data1 /mnt/data2 /mnt/data3 - look in /etc/fstab to determine which mount points should be there), check to see if RAID mirrors are healthy. See [[RAID_Cards#Common_CLI_commands_.28megacli.29|megacli]], [[#aaccheck|aaccheck]]<br />
<br />
Before you start the jails, you need to run [[#preboot|preboot]]. This will do some assurance checks to make sure things are prepped to start the jails. Any issues that come out of preboot need to be addressed before starting jails.<br />
<br />
=== Start jails ===<br />
[[#Starting_jails:_Quad.2FSafe_Files|More on starting jails]]<br />
Customer jails (the VPSs) do not start up automatically at boot time. When a FreeBSD machines boots up, it boots up, and does nothing else. To start jails, we put the commands to start each jail into a shell script(s) and run the script(s). Jail startup is something that needs to be actively monitored, which is why we don’t just run the script automatically. <br />
<br />
In order to start jails, we run the quad files: quad1 quad2 quad3 and quad4 (on new systems there is only quad1). If the machine was cleanly rebooted- which wouldn't be the case if this was a crash, you may run the safe files (safe1 safe2 safe3 safe4) in lieu of quads. <br />
<br />
Open up 4 logins to the server (use the windows in [[Screen#Screen_Organization|a9]])<br />
In each of the 4 windows you will:<br />
<br />
If there is a [[#startalljails|startalljails]] script (and only quad1), run that command in each of the 4 windows. It will parse through the quad1 file and start each jail. Follow the instructions [[#Problems_with_the_quad.2Fsafe_files|here]] for monitoring startup. Note that you can be a little more lenient with jails that take awhile to start- startalljails will work around the slow jails and start the rest. As long as there aren't 4 jails which are "hung" during startup, the rest will get started eventually.<br />
-or-<br />
If there is no startalljails script, there will be multiple quad files. In each of the 4 windows, start each of the quads. i.e. start quad1 in window1, quad2 in window2 and so on. DO NOT start any quad twice. It will crash the server. If you accidentally do this, just jailkill all the jails which are in the quad and run the quad again. Follow the instructions here for monitoring quad startup.<br />
<br />
Note the time the last jail boots- this is what you will enter in the crash log.<br />
<br />
Save the crash log.<br />
<br />
=== Check to make sure all jails have started ===<br />
There's a simple script which will make sure all jails have started, and enter the ipfw counter rules: [[#postboot|postboot]] <br />
Run postboot, which will do a jailps on each jail it finds (excluding commented out jails) in the quad file(s). We're looking for 2 things:<br />
# systems spawning out of control or too many procs<br />
# jails which haven't started<br />
On 7.x and newer systems it will print out the problems (which jails haven't started) at the conclusion of postboot. <br />
On older systems you will need to watch closely to see if/when there's a problem, namely:<br />
<br />
[hostname] doesnt exist on this server<br />
<br />
When you get this message, it means one of 2 things:<br />
1. the jail really didn't start:<br />
When a jail doesn't start it usually boils down to a problem in the quad file. Perhaps the path name is wrong (data1 vs data2) or the name of the vn/mdfile is wrong. Once this is corrected, you will need to run the commands from the quad file manually, or you may use <tt>startjail <hostname></tt><br />
<br />
2. the customer has changed their hostname (and not told us) so their jail ''is'' running, just under a different hostname:<br />
On systems with jls, this is easy to rectify. First, get the customer info: <tt>g <hostname></tt><br />
Then look for the customer in jls: <tt>jls | grep <col0XXXX></tt><br />
From there you will see their new hostname- you should update that hostname in the quad file: don't forget to edit it on the <tt>## begin ##</tt> and <tt>## end ##</tt> lines, and in mgmt. <br />
On older systems without jls, this will be harder, you will need to look further to see their hostname- perhaps its in their /etc/rc.conf<br />
<br />
<br />
Once all jails are started, do some spot checks- try to ssh or browse to some customers, just to make sure things are really ok.<br />
<br />
== Adding disk to a 7.x/8.x jail ==<br />
or<br />
== Moving customer to a different drive (md) ==<br />
<br />
NOTE: this doesn’t apply to mx2 which uses gvinum. Use same procedure as 6.x<br />
NOTE: if you unmount before mdconfig, re-mdconfig (attach) then unmount then mdconfig -u again <br />
<br />
<br />
(parts to change/customize are <tt><span style="color:red">red</span></tt>)<br />
<br />
<br />
If someone wants more disk space, there’s a paste for it, send it to them (explains about downtime, etc).<br />
<br />
1. Figure out the space avail from <tt>js</tt>. Ideally, you want to put the customers new space on a different partition (and create the new md on the new partition). <br />
<br />
<br />
2. make a mental note of how much space they're currently using<br />
<br />
<br />
3. <tt>jailkill <hostname></tt> <br />
<br />
<br />
4. Umount it (including their devfs) but leave the md config’d (so if you use stopjail, you will have to re-mdconfig it)<br />
<br />
<br />
5. <tt>g <customerID></tt> to get the info (IP/cust#) needed to feed the new mdfile and mount name, and to see the current md device. <br />
<br />
<br />
6a. When there's enough room to place new system on an alternate, or the same drive:<br />
USE CAUTION not to overwrite (touch, mdconfig) existing md!!<br><br />
<tt>touch /mnt/data<span style="color:red">3/69.55.234.66-col01334</span><br><br />
mdconfig -a -t vnode -s 10g -f /mnt/data3/69.55.234.66-col01334 -u 97<br><br />
newfs /dev/md97</tt><br />
<br />
<br />
Optional- if new space is on a different drive, move the mount point directory AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
<tt>mount /dev/md<span style="color:red">97</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<br />
confirm you are mounted to /dev/md<span style="color:red">97</span> and space is correct:<br><br />
<tt>df .</tt> <br />
<br />
<br />
do the dump and pipe directly to restore:<br><br />
<tt>dump -0a -f - /dev/md<span style="color:red">1</span> | restore -r -f - <br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use <tt>df</tt> to confirm the restored data size matches the original usage figure<br />
<br />
<br />
md-unconfig old system:<br><br />
<tt>mdconfig -d -u <span style="color:red">1</span></tt><br />
<br />
<br />
archive old mdfile. <br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.237.26-col00241</span> /mnt/data<span style="color:red">1/old-col00241-mdfile-noarchive-20091211</span></tt><br />
<br />
<br />
edit quad (vq1) to point to new (/mnt/data<span style="color:red">3</span>) location AND new md number (md<span style="color:red">97</span>)<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
6b. When there's not enough room on an alternate partition, or on the same drive...but there is enough room if you were to remove the existing customer's space:<br />
<br />
<br />
mount backup nfs mounts:<br><br />
<tt>mbm</tt><br> <br />
(run <tt>df</tt> to confirm backup mounts are mounted)<br />
<br />
<br />
dump the customer to backup2 or backup1<br><br />
<tt>dump -0a -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> /dev/md<span style="color:red">1</span></tt><br><br />
(when complete WITHOUT errors, <tt>du</tt> the dump file to confirm it matches size, roughly, with usage)<br />
<br />
<br />
unconfigure and remove old mdfile<br><br />
<tt>mdconfig -d -u <span style="color:red">1</span><br><br />
rm /mnt/data<span style="color:red">1/69.55.237.26-col00241</span></tt><br><br />
(there should now be enough space to recreate your bigger system. If not, run sync a couple times)<br />
<br />
<br />
create the new system (ok to reuse old mdfile and md#):<br><br />
<tt>touch /mnt/data<span style="color:red">1/69.55.234.66-col01334</span><br><br />
mdconfig -a -t vnode -s <span style="color:red">10</span>g -f /mnt/data<span style="color:red">1/69.55.234.66-col01334</span> -u <span style="color:red">1</span><br><br />
newfs /dev/md<span style="color:red">1</span><br><br />
mount /dev/md<span style="color:red">1</span> /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> <br><br />
cd /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<br />
confirm you are mounted to /dev/md<span style="color:red">1</span> and space is correct:<br><br />
<tt>df .</tt> <br />
<br />
<br />
do the restore from the dumpfile on the backup server:<br><br />
<tt>restore -r -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> .<br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
umount nfs:<br><br />
<tt>mbu</tt><br />
<br />
<br />
If md# changed (or mount point), edit quad (<tt>vq1</tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new md number (md<span style="color:red">1</span>)<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
7. update disk (and dir if applicable) in mgmt screen<br />
<br />
<br />
8. update backup list AND move backups, if applicatble<br />
<br />
Ex: <tt>mvbackups <span style="color:red">69.55.237.26-col00241</span> jail<span style="color:red">9</span> data<span style="color:red">3</span></tt><br />
<br />
<br />
9. Optional: archive old mdfile<br />
<br />
<tt>mbm<br><br />
gzip -c old-col01588-mdfile-noarchive-20120329 > /deprecated/old-col01588-mdfile-noarchive-20120329.gz<br><br />
mbu<br><br />
rm old-col01588-mdfile-noarchive-20120329</tt><br />
<br />
== Adding disk to a 6.x jail (gvinum/gconcat) ==<br />
or<br />
== Moving customer to a different drive (gvinum/gconcat) ==<br />
<br />
(parts to change are <span style="color:red">highlighted</span>)<br />
<br />
<br />
If someone wants more disk space, there’s a paste for it, send it to them (explains about downtime, etc).<br />
<br />
<br />
1. Figure out the space avail from [[#js|js]]. Ideally, you want to put the customers new space on a different partition (and create the new md on the new partition). <br />
<br />
<br />
2. make a mental note of how much space they're currently using<br />
<br />
<br />
3. <tt>[[#stopjail|stopjail]] <hostname></tt> <br />
<br />
<br />
4. <tt>[[#g|g]] <customerID></tt> to get the info (IP/cust#) needed to feed the new mount name and existing volume/device. <br />
<br />
<br />
5a. When there's enough room to place new system on an alternate, or the same drive (using only UNUSED - including if it's in use by the system in question - gvinum volumes):<br />
<br />
<br />
Configure the new device:<br><br />
A. for a 2G system (single gvinum volume):<br><br />
<tt>bsdlabel -r -w /dev/gvinum/v<span style="color:red">123</span><br><br />
newfs /dev/gvinum/v<span style="color:red">123</span>a</tt><br><br />
<br />
-or- <br />
<br />
<br />
B. for a >2G system (create a gconcat volume):<br><br />
try to grab a contiguous block of gvinum volumes. gconcat volumes MAY NOT span drives (i.e. you cannot use a gvinum volume from data3 and a volume from data2 in the same gconcat volume). <br />
<br />
<tt>gconcat label <span style="color:red">v82-v84 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84</span><br><br />
bsdlabel -r -w /dev/concat/<span style="color:red">v82-v84</span><br><br />
newfs /dev/concat/<span style="color:red">v82-v84</span>a</tt><br />
<br />
<br />
Other valid gconcat examples:<br><br />
<tt>gconcat label v82-v84v109v112 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84 /dev/gvinum/v109 /dev/gvinum/v112<br><br />
gconcat label v82v83 /dev/gvinum/v82 /dev/gvinum/v83</tt> <br><br />
Note, long names will truncate: v144v145v148-v115 will truncate to v144v145v148-v1 (so you will refer to it as v144v145v148-v1 thereafter)<br />
<br />
<br />
Optional- if new volume is on a different drive, move the mount point directory (get the drive from js output) AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
<br />
confirm you are mounted to the device (<tt>/dev/gvinum/v<span style="color:red">123</span>a</tt> OR <tt>/dev/concat/<span style="color:red">v82-v84</span></tt>) and space is correct:<br><br />
A. <tt>mount /dev/gvinum/v<span style="color:red">123</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br><br />
-or-<br><br />
B. <tt>mount /dev/concat/<span style="color:red">v82-v84</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<tt>cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
df .</tt> <br />
<br />
<br />
do the dump and pipe directly to restore:<br><br />
<tt>dump -0a -f - /dev/gvinum/v<span style="color:red">1</span> | restore -r -f - <br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
edit quad (<tt>vq<span style="color:red">1</span></tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new volume (<tt>/dev/gvinum/v<span style="color:red">123</span>a</tt> or <tt>/dev/concat/<span style="color:red">v82-v84</span>a</tt>) , run <tt>buildsafe</tt><br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
5b. When there's not enough room on an alternate partition, or on the same drive...but there is enough room if you were to remove the existing customer's space (i.e. if you want/need to reuse the existing gvinum volumes and add on more):<br />
<br />
<br />
mount backup nfs mounts:<br><br />
<tt>mbm</tt> <br><br />
(run df to confirm backup mounts are mounted)<br />
<br />
<br />
dump the customer to backup2 or backup1<br><br />
<tt>dump -0a -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> /dev/<span style="color:red">gconcat/v106-v107</span></tt><br><br />
(when complete WITHOUT errors, du the dump file to confirm it matches size, roughly, with usage)<br />
<br />
<br />
unconfigure the old gconcat volume<br><br />
list member gvinum volumes:<br />
<br />
<tt>gconcat list <span style="color:red">v106v107</span></tt><br />
<br />
<br />
Output will resemble:<br><br />
<pre>Geom name: v106v107<br />
State: UP<br />
Status: Total=2, Online=2<br />
Type: AUTOMATIC<br />
ID: 3530663882<br />
Providers:<br />
1. Name: concat/v106v107<br />
Mediasize: 4294966272 (4.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e2<br />
Consumers:<br />
1. Name: gvinum/sd/v106.p0.s0<br />
Mediasize: 2147483648 (2.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e3<br />
Start: 0<br />
End: 2147483136<br />
2. Name: gvinum/sd/v107.p0.s0<br />
Mediasize: 2147483648 (2.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e3<br />
Start: 2147483136<br />
End: 4294966272</pre><br />
<br />
stop volume and clear members<br><br />
<tt>gconcat stop <span style="color:red">v106v107</span><br><br />
gconcat clear <span style="color:red">gvinum/sd/v106.p0.s0 gvinum/sd/v107.p0.s0</span></tt><br />
<br />
<br />
create new device- and its ok to reuse old/former members<br><br />
try to grab a contiguous block of gvinum volumes. gconcat volumes MAY NOT span drives (i.e. you cannot use a gvinum volume from data3 and a volume from data2 in the same gconcat volume). <br><br />
<tt>gconcat label <span style="color:red">v82-v84v106v107 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84 /dev/gvinum/v106 /dev/gvinum/v107</span> <br><br />
bsdlabel -r -w /dev/concat/<span style="color:red">v82-v84v106v107</span><br><br />
newfs /dev/concat/<span style="color:red">v82-v84v106v107</span>a</tt><br />
<br />
<br />
Optional- if new volume is on a different drive, move the mount point directory (get the drive from js output) AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
confirm you are mounted to the device (/dev/concat/<span style="color:red">v82-v84v106v107</span>) and space is correct:<br><br />
<tt>mount /dev/concat/<span style="color:red">v82-v84v106v107</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<tt>cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
df .</tt> <br />
<br />
do the restore from the dumpfile on the backup server:<br><br />
<tt>restore -r -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> .<br><br />
rm restoresymtable</tt><br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
edit quad (<tt>vq<span style="color:red">1</span></tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new volume (<tt>/dev/concat/<span style="color:red">v82-v84v106v107</span>a</tt>), run buildsafe<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
TODO: clean up/clear old gvin/gconcat vol<br />
<br />
<br />
6. update disk (and dir if applicable) in mgmt screen<br />
<br />
<br />
7. update backup list AND move backups, if applicatble<br />
<br />
<br />
Ex: <tt>mvbackups <span style="color:red">69.55.237.26-col00241</span> jail<span style="color:red">9</span> data<span style="color:red">3</span></tt><br />
<br />
<br />
<br />
DEPRECATED - steps to tack on a new gvin to existing gconcat- leads to corrupted fs<br />
bsdlabel -e /dev/concat/v82-v84<br />
<br />
To figure out new size of the c partition, multiply 4194304 by the # of 2G gvinum volumes and subtract the # of 2G volumes:<br />
10G: 4194304 * 5 – 5 = 20971515<br />
8G: 4194304 * 4 – 4 = 16777212<br />
6G: 4194304 * 3 – 3 = 12582909<br />
4G: 4194304 * 2 – 2 = 8388606<br />
<br />
To figure out the new size of the a partition, subtract 16 from the c partition:<br />
10G: 20971515 – 16 = 20971499<br />
8G: 16777212 – 16 = 16777196<br />
6G: 12582909 – 16 = 12582893<br />
4G: 8388606 – 16 = 8388590<br />
<br />
Orig:<br />
8 partitions:<br />
# size offset fstype [fsize bsize bps/cpg]<br />
a: 8388590 16 4.2BSD 2048 16384 28552<br />
c: 8388606 0 unused 0 0 # "raw" part, don't edit<br />
<br />
New:<br />
8 partitions:<br />
# size offset fstype [fsize bsize bps/cpg]<br />
a: 12582893 16 4.2BSD 2048 16384 28552<br />
c: 12582909 0 unused 0 0 # "raw" part, don't edit<br />
<br />
sync; sync<br />
<br />
growfs /dev/concat/v82-v84a<br />
<br />
fsck –fy /dev/concat/v82-v84a<br />
<br />
sync<br />
<br />
fsck –fy /dev/concat/v82-v84a<br />
<br />
(keep running fsck’s till NO errors)<br />
<br />
== Problems un-mounting - and with mount_null’s ==<br />
<br />
If you cannot unmount a filesystem, beacuse it says the filesystem is busy, it is because of three things:<br />
<br />
a) the jail is still running<br />
<br />
b) you are actually in that directory, even though the jail is stopped<br />
<br />
c) there are still dev, null_mount or linprocfs mount points mounted inside that directory.<br />
<br />
d) when trying to umount null_mounts that are really long and you get an error like “No such file or directory”, it’s an OS bug where the dir is truncated. No known fix<br />
<br />
e) there are still files open somewhere inside the dir. Use <tt>fstat | grep <cid></tt> to find the process that has files open<br />
<br />
f) Starting with 6.x, the jail mechanism does a poor job of keeping track of processes running in a jail and if it thinks there are still procs running, it will refuse to umount the disk. If this is happening you should see a low number in the #REF column when you run jls. In this case you ''can'' safely <tt>umount –f</tt> the mount. <br />
<br />
Please note -if you forcibly unmount a (4.x) filesystem that has null_mounts<br />
still mounted in it, the system '''will crash''' within 10-15 mins.<br />
<br />
== Misc jail Items ==<br />
<br />
We are overselling hard drive space on jail2, jail8, jail9, a couple jails on jail17, jail4, jail12 and jail18.<br />
Even though the vn file shows 4G size, it doesn’t actually occupy that amount of space on the disk. So be careful not to fill up drives where we’re overselling – use oversellcheck to confirm you’re not oversold by more than 10G.<br />
There are other truncated jails, they are generally noted in a the file on the root system: /root/truncated<br />
<br />
The act of moving a truncated vn to another system un-does the truncating- the truncated vn is filled with 0’s and it occupies physical disk space for which it’s configured. So, you should use dumpremote to preserve the truncation.<br />
<br />
= FreeBSD VPS Management Tools =<br />
<br />
These files are located in /usr/local/jail/rc.d and /usr/local/jail/bin<br />
<br />
== jailmake ==<br />
<br />
Applies to 7.x+ <br />
On older systems syntax differs, run jailmake once to see.<br />
<br />
Note: this procedure differs on mx2 which is 7.x but still uses gvinum<br />
<br />
# run js to figure out which md’s are in use, which disk has enough space, IP to put it on<br />
# use col00xxx for both hostnames if they don’t give you a hostname<br />
# copy over dir, ip and password to pending customer screen<br />
<br />
<tt>Usage: jailmake IP[,IP] CID disk[1|2|3] md# hostname shorthost ipfw# email [size in GB]</tt><br />
<br />
Ex: <br />
<br />
Jail2# jailmake 69.55.234.66 col01334 3 97 vps.bsd.it vps 1334 fb@bsd.it<br />
<br />
== jailps ==<br />
jailps [hostname]<br />
DEPRECATED FOR jps: displays processes belonging to/running inside a jail. The command<br />
takes one (optional) argument – the hostname of the jail you wish to query. If you don’t <br />
supply an argument, all processes on the machine are listed and grouped by jail. <br />
<br />
== jps ==<br />
jps [hostname]<br />
displays processes belonging to/running inside a jail. The command<br />
takes one (optional) argument – the hostname or ID of the jail you wish to query. <br />
<br />
== jailkill ==<br />
jailkill <hostname><br />
stops all process running in a jail.<br />
<br />
You can also run:<br />
jailkill <JID><br />
<br />
=== problems ===<br />
Occasionally you will hit an issue where jail will not kill off:<br />
<br />
<pre>jail9# jailkill www.domain.com<br />
www.domain.com .. killed: none<br />
jail9#</pre><br />
Because no processes are running under that hostname. You cannot use jailps.pl either:<br />
<br />
<pre>jail9# jailps www.domain.com<br />
www.domain.com doesn’t exist on this server<br />
jail9#</pre><br />
<br />
The reasons for this are usually:<br />
* the jail is no longer running<br />
<br />
* the jail's hostname has changed<br />
In this case, <br />
<br />
>=6.x: run a <tt>jls|grep <jail's IP></tt> to find the correct hostname, then update the quad file, then kill the jail.<br />
<br />
<6.x: the first step is to cat their /etc/rc.conf file to see if you can tell what they set the new hostname to. This very often works. For example:<br />
<br />
cat /mnt/data2/198.78.65.136-col00261-DIR/etc/rc.conf<br />
<br />
But maybe they set the hostname with the hostname command, and the original hostname is still in /etc/rc.conf.<br />
<br />
The welcome email clearly states that they should tell us if they change their hostname, so there is no problem in just emailing them and asking them what they set the new hostname to.<br />
<br />
Once you know the new hostname OR if a customer simply emails to inform you that they have set the hostname to something different, you need to edit the quad and safe files that their system is in to input the new hostname.<br />
<br />
However, if push comes to shove and you cannot find out the hostname from them or from their system, then you need to start doing some detective work.<br />
<br />
The easiest thing to do is run jailps looking for a hostname similar to their original hostname. Or you could get into the /bin/sh shell by running:<br />
<br />
/bin/sh<br />
<br />
and then looking at every hostname of every process:<br />
<br />
for f in `ls /proc` ; do cat /proc/$f/status ; done<br />
<br />
and scanning for a hostname that is either similar to their original hostname, or that you don't see in any of the quad safe files.<br />
<br />
This is very brute force though, and it is possible that catting every file in /proc is dangerous - I don't recommend it. A better thing would be to identify any processes that you know belong to this system – perhaps the reason you are trying to find this system is because they are running something bad - and just catting the status from only that PID.<br />
<br />
Somewhere there’s a jail where there may be 2 systems named www. Look at /etc/rc.conf and make sure they’re both really www. If they are, jailkill www, jailps www to make sure not running. Then immediately restart the other one, as the fqdn (as found from a rev nslookup)<br />
<br />
* on >=6.x the hostname may not yet be hashed:<br />
<pre>jail9 /# jls<br />
JID Hostname Path IP Address(es)<br />
1 bitnet.dgate.org /mnt/data1/69.55.232.50-col02094-DIR 69.55.232.50<br />
2 ns3.hctc.net /mnt/data1/69.55.234.52-col01925-DIR 69.55.234.52<br />
3 bsd1 /mnt/data1/69.55.232.44-col00155-DIR 69.55.232.44<br />
4 let2.bbag.org /mnt/data1/69.55.230.92-col00202-DIR 69.55.230.92<br />
5 post.org /mnt/data2/69.55.232.51-col02095-DIR 69.55.232.51 ...<br />
6 ns2 /mnt/data1/69.55.232.47-col01506-DIR 69.55.232.47 ...<br />
7 arlen.server.net /mnt/data1/69.55.232.52-col01171-DIR 69.55.232.52<br />
8 deskfood.com /mnt/data1/69.55.232.71-col00419-DIR 69.55.232.71<br />
9 mirage.confluentforms.com /mnt/data1/69.55.232.54-col02105-DIR 69.55.232.54 ...<br />
10 beachmember.com /mnt/data1/69.55.232.59-col02107-DIR 69.55.232.59<br />
11 www.agottem.com /mnt/data1/69.55.232.60-col02109-DIR 69.55.232.60<br />
12 sdhobbit.myglance.org /mnt/data1/69.55.236.82-col01708-DIR 69.55.236.82<br />
13 ns1.jnielsen.net /mnt/data1/69.55.234.48-col00204-DIR 69.55.234.48 ...<br />
14 ymt.rollingegg.net /mnt/data2/69.55.236.71-col01678-DIR 69.55.236.71<br />
15 verse.unixlore.net /mnt/data1/69.55.232.58-col02131-DIR 69.55.232.58<br />
16 smcc-mail.org /mnt/data2/69.55.232.68-col02144-DIR 69.55.232.68<br />
17 kasoutsuki.w4jdh.net /mnt/data2/69.55.232.46-col02147-DIR 69.55.232.46<br />
18 dili.thium.net /mnt/data2/69.55.232.80-col01901-DIR 69.55.232.80<br />
20 www.tekmarsis.com /mnt/data2/69.55.232.66-col02155-DIR 69.55.232.66<br />
21 vps.yoxel.net /mnt/data2/69.55.236.67-col01673-DIR 69.55.236.67<br />
22 smitty.twitalertz.com /mnt/data2/69.55.232.84-col02153-DIR 69.55.232.84<br />
23 deliver4.klatha.com /mnt/data2/69.55.232.67-col02160-DIR 69.55.232.67<br />
24 nideffer.com /mnt/data2/69.55.232.65-col00412-DIR 69.55.232.65<br />
25 usa.hanyuan.com /mnt/data2/69.55.232.57-col02163-DIR 69.55.232.57<br />
26 daifuku.ppbh.com /mnt/data2/69.55.236.91-col01720-DIR 69.55.236.91<br />
27 collins.greencape.net /mnt/data2/69.55.232.83-col01294-DIR 69.55.232.83<br />
28 ragebox.com /mnt/data2/69.55.230.104-col01278-DIR 69.55.230.104<br />
29 outside.mt.net /mnt/data2/69.55.232.72-col02166-DIR 69.55.232.72<br />
30 vps.payneful.ca /mnt/data2/69.55.234.98-col01999-DIR 69.55.234.98<br />
31 higgins /mnt/data2/69.55.232.87-col02165-DIR 69.55.232.87 ...<br />
32 ozymandius /mnt/data2/69.55.228.96-col01233-DIR 69.55.228.96<br />
33 trusted.realtors.org /mnt/data2/69.55.238.72-col02170-DIR 69.55.238.72<br />
34 jc1.flanderous.com /mnt/data2/69.55.239.22-col01504-DIR 69.55.239.22<br />
36 guppylog.com /mnt/data2/69.55.238.73-col00036-DIR 69.55.238.73<br />
40 haliohost.com /mnt/data2/69.55.234.41-col01916-DIR 69.55.234.41 ...<br />
41 satyr.jorge.cc /mnt/data1/69.55.232.70-col01963-DIR 69.55.232.70<br />
jail9 /# jailkill satyr.jorge.cc<br />
ERROR: jail_: jail "satyr,jorge,cc" not found<br />
</pre><br />
<br />
Note how it's saying <tt>satyr,jorge,cc</tt> is not found, and not <tt>satyr.jorge.cc</tt>. <br />
<br />
The jail subsystem tracks things using comma-delimited hostnames. That is created every few hours:<br />
<br />
jail9 /# crontab -l<br />
0 0,6,12,18 * * * /usr/local/jail/bin/sync_jail_names<br />
<br />
So if we run this manually:<br />
jail9 /# /usr/local/jail/bin/sync_jail_names<br />
<br />
Then kill the jail:<br />
jail9 /# jailkill satyr.jorge.cc<br />
successfully killed: satyr,jorge,cc</pre><br />
<br />
It worked.<br />
<br />
<br />
If you ever see this when trying to kill a jail:<br />
<br />
<pre># jailkill e-scribe.com<br />
killing JID: 6 hostname: e-scribe.com<br />
3 procs running<br />
3 procs running<br />
3 procs running<br />
3 procs running<br />
...</pre><br />
<br />
<tt>[[#jailkill|jailkill]]</tt> probably got lost trying to kill off the jail. Just ctrl-c the jailkill process, then run a jailps on the hostname, and <tt>kill -9</tt> any process which is still running. Keep running jailps and <tt>kill -9</tt> till all processes are gone.<br />
<br />
== jailpsall ==<br />
jailpsall<br />
will run a jailps on all jails configured in the quad files (this is different from<br />
jailps with no arguments as it won’t help you find a “hidden” system)<br />
<br />
== jailpsw ==<br />
jailpsw<br />
will run a jailps with an extra -w to provide wider output<br />
<br />
== jt (>=7.x) ==<br />
jt<br />
displays the top 20 processes on the server (the top 20 processes from top) and <br />
which jail owns them. This is very helpful for determining who is doing what when<br />
the server is very busy.<br />
<br />
== jtop (>=7.x) ==<br />
jtop<br />
a wrapper for top displaying processes on the server and which jail owns them. Constantly updates, like top. <br />
<br />
== jtop (<7.x) ==<br />
jtop<br />
displays the top 20 processes on the server (the top 20 processes from top) and <br />
which jail owns them. This is very helpful for determining who is doing what when<br />
the server is very busy.<br />
<br />
== stopjail ==<br />
stopjail <hostname> [1]<br />
this will jailkill, umount and vnconfig –u a jail. If passed an optional 2nd<br />
argument, it will not exit before umounting and un-vnconfig’ing in the event<br />
jailkill returns no processes killed. This is useful if you just want to umount<br />
and vnconfig –u a jail you’ve already killed. It is intelligent in that it won’t <br />
try to umount or vnconfig –u if it’s not necessary.<br />
<br />
== startjail ==<br />
startjail <hostname><br />
this will start vnconfig, mount (including linprocfs and null-mounts), and start a jail.<br />
Essentially, it reads the jail’s relevant block from the right quad file and executes it.<br />
It is intelligent in that it won’t try to mount or vnconfig if it’s not necessary.<br />
<br />
== jpid ==<br />
jpid <pid><br />
displays information about a process – including which jail owns it.<br />
It’s the equivalent of running cat /proc/<pid>/status<br />
<br />
== canceljail ==<br />
canceljail <hostname> [1]<br />
this will stop a jail (the equivalent of stopjail), check for backups (offer to remove them <br />
from the backup server and the backup.config), rename the vnfile, remove the dir, and <br />
edit quad/safe. If passed an optional 2nd argument, it will not exit upon failing to kill<br />
and processes owned by the jail. This is useful if you just want to cancel a jail which <br />
is already stopped.<br />
<br />
== jls ==<br />
jls [-v]<br />
Lists all jails running:<br />
<pre>JID #REF IP Address Hostname Path<br />
101 135 69.55.224.148 mail.pc9.org /mnt/data2/69.55.224.148-col01034-DIR</pre><br />
<br />
#REF is the number of references or procs(?) running<br />
<br />
Running with -v will give you all IPs assigned to each jail (7.2 up)<br />
<pre>JID #REF Hostname Path IP Address(es)<br />
101 139 mail.pc9.org /mnt/data2/69.55.224.148-col01034-DIR 69.55.224.14869.55.234.85</pre><br />
<br />
== startalljails ==<br />
startalljails<br />
7.2+ only. This will parse through quad1 and start all jails. It utilizes lockfiles so it won’t try to start a jail more than once- therefore multiple instances can be running in parallel without fear of starting a jail twice. If a jail startup gets stuck, you can ^C without fear of killing the script. IMPORTANT- before running startalljails you should make sure you ran preboot once as it will clear out all the lockfiles and enable startalljails to work properly.<br />
<br />
== aaccheck.sh ==<br />
aaccheck.sh<br />
displayes the output of container list and task list from aaccli<br />
<br />
== backup ==<br />
backup<br />
backup script called nightly to update jail scripts and do customer backups<br />
<br />
== buildsafe ==<br />
buildsafe<br />
creates safe files based on quads (automatically removing the fsck’s). This will destructively overwrite safe files<br />
<br />
== checkload.pl ==<br />
checkload.pl<br />
this was intended to be setup as a cronjob to watch processes on a jail when the load <br />
rises above a certain level. Not currently in use.<br />
<br />
== checkprio.pl ==<br />
checkprio.pl<br />
will look for any process (other than the current shell’s csh, sh, sshd procs) with a non-normal priority and normalize it<br />
<br />
== diskusagemon == <br />
diskusagemon <mount point> <1k blocks><br />
watches a mount point’s disk use, when it reaches the level specified in the 2nd argument,<br />
it exits. This is useful when doing a restore and you want to be paged as it’s nearing completion.<br />
Best used as: <tt>diskusagemon /asd/asd 1234; pagexxx</tt><br />
<br />
== dumprestore ==<br />
dumprestore <dumpfile><br />
this is a perl expect script which automatically enters ‘1’ and ‘y’. It seems to cause restore to fail<br />
to set owner permissions on large restores.<br />
<br />
== g ==<br />
g <search><br />
greps the quad/safe files for the given search parameter<br />
<br />
== gather.pl ==<br />
gather.pl<br />
gathers up data about jails configured and writes to a file. Used for audits against the db<br />
<br />
== gb ==<br />
gb <search><br />
greps backup.config for the given search parameter<br />
<br />
== gbg ==<br />
gbg <search><br />
greps backup.config for the given search parameter and presents just the directories (for clean pasting)<br />
<br />
== ipfwbackup ==<br />
ipfwbackup<br />
writes ipfw traffic count data to a logfile<br />
<br />
== ipfwreset ==<br />
ipfwreset<br />
writes ipfw traffic count data to a logfile and resets counters to 0<br />
<br />
== js ==<br />
js<br />
output varies by OS version, but generally provides information about the base jail:<br />
- which vn’s are in use<br />
- disk usage<br />
- info about the contents of quads<br />
- the # of inodes represented by the jails contained in the group (133.2 in the example below), and how many jails per data mount, as well as subtotals<br />
- ips bound to the base machine but not in use by a jail<br />
- free gvinum volumes, or unused vn’s or used md’s<br />
<br />
<pre>/usr/local/jail/rc.d/quad1:<br />
/mnt/data1 133.2 (1)<br />
/mnt/data2 1040.5 (7)<br />
total 1173.7 (8)<br />
/usr/local/jail/rc.d/quad2:<br />
/mnt/data1 983.4 (6)<br />
total 983.4 (6)<br />
/usr/local/jail/rc.d/quad3:<br />
/mnt/data1 693.4 (4)<br />
/mnt/data2 371.6 (3)<br />
total 1065 (7)<br />
/usr/local/jail/rc.d/quad4:<br />
/mnt/data1 466.6 (3)<br />
/mnt/data2 882.2 (5)<br />
total 1348.8 (8)<br />
/mnt/data1: 2276.6 (14)<br />
/mnt/data2: 2294.3 (15)<br />
<br />
Available IPs:<br />
69.55.230.11 69.55.230.13 69.55.228.200<br />
<br />
Available volumes:<br />
v78 /mnt/data2 2G<br />
v79 /mnt/data2 2G<br />
v80 /mnt/data2 2G</pre> <br />
<br />
== load.pl ==<br />
load.pl<br />
feeds info to load mrtg - executed by inetd.<br />
<br />
== makevirginjail ==<br />
makevirginjail<br />
Only on some systems, makes an empty jail (doesn't do restore step)<br />
<br />
== mb == <br />
mb <mount|umount><br />
(nfs) mounts and umounts dirs to backup2. Shortcuts are mbm and mbu to mount and unmount. <br />
<br />
== notify.sh ==<br />
notify.sh<br />
emails reboot@johncompanies.com – intended to be called at boot time to alert us to a machine which panics and reboots and isn’t caught by bb or castle.<br />
<br />
== orphanedbackupwatch ==<br />
orphanedbackupwatch<br />
looks for directories on backup2 which aren’t configured in backup.config and offers to delete them<br />
<br />
== postboot ==<br />
postboot<br />
to be run after a machine reboot and quad/safe’s are done executing. It will:<br />
* do chmod 666 on each jail’s /dev/null<br />
* add ipfw counts<br />
* run jailpsall (so you can see if a configured jail isn’t running)<br />
<br />
== preboot ==<br />
preboot<br />
to be run before running quad/safe – checks for misconfigurations: <br />
* a jail configured in a quad but not a safe<br />
* a jail is listed more than once in a quad<br />
* the ip assigned to a jail isn’t configured on the machine<br />
* alias numbering skips in the rc.conf (resulting in the above)<br />
* orphaned vnfile's that aren't mentioned in a quad/safe<br />
* ip mismatches between dir/vnfile name and the jail’s ip<br />
* dir/vnfiles's in quad/safe that don’t exist <br />
<br />
== quadanalyze.pl ==<br />
quadanalyze.pl<br />
called by js, produces the info (seen above with js explanation) about the contents of quad (inode count, # of jails, etc.)<br />
<br />
== rsync.backup ==<br />
rsync.backup<br />
does customer backups (relies on backup.config)<br />
<br />
== taskdone ==<br />
taskdone<br />
when called will email support@johncompanies.com with the hostname of the machine from which it was executed as the subject<br />
<br />
== topten ==<br />
topten<br />
summarizes the top 10 traffic users (called by ipfwreset)<br />
<br />
== trafficgather.pl ==<br />
trafficgather.pl [yy-mm]<br />
sends a traffic usage summary by jail to support@johncomapnies.com and payments@johncompanies.com. Optional arguments are year and month (must be in the past). If not passed, assumes last month. Relies on traffic logs created by ipfwreset and ipfwbackup<br />
<br />
== trafficwatch.pl ==<br />
trafficwatch.pl<br />
checks traffic usage and emails support@johncomapnies.com when a jail reaches the warning level (35G) and the limit (40G). We really aren’t using this anymore now that we have netflow.<br />
<br />
== trafstats ==<br />
trafstats<br />
writes ipfw traffic usage info by jail to a file called jc_traffic_dump in each jail’s / dir<br />
<br />
== truncate_jailmake ==<br />
truncate_jailmake<br />
a version of jailmake which creates truncated vnfiles.<br />
<br />
== vb ==<br />
vb<br />
the equivalent of: <tt>vi /usr/local/jail/bin/backup.config</tt><br />
<br />
== vs (freebsd) ==<br />
vs<n><br />
the equivalent of: <tt>vi /usr/local/jail/rc.d/safe<n></tt><br />
<br />
vq<n><br />
the equivalent of: vi /usr/local/jail/rc.d/quad<n><br />
<br />
== dumpremote ==<br />
dumpremote <user@machine> </remote/location/file-dump> <vnX><br />
ex: dumpremote user@10.1.4.117 /mnt/data3/remote.echoditto.com-dump 7<br />
this will dump a vn filesystem to a remote machine and location<br />
<br />
== oversellcheck ==<br />
oversellcheck<br />
displays how much a disk is oversold or undersold taking into account truncated vn files. Only for use on 4.x systems<br />
<br />
== mvbackups (freebsd) ==<br />
mvbackups <dir> (1.1.1.1-col00001-DIR) <target_machine> (jail1) <target_dir> (data1)<br />
moves backups from one location to another on the backup server, and provides you with option to remove entries from current backup.config, and simple paste command to add the config to backup.config on the target server<br />
<br />
== jailnice ==<br />
jailnice <hostname><br />
applies <tt>renice 19 [PID]</tt> and <tt>rtprio 31 –[PID]</tt> to each process in the given jail<br />
<br />
== dumpremoterestore ==<br />
dumpremoterestore <device> <ip of target machine> <dir on target machine><br />
ex: dumpremoterestore /dev/vn51 10.1.4.118 /mnt/data2/69.55.239.45-col00688-DIR<br />
dumps a device and restores it to a directory on a remote machine. Requires that you enable root ssh on the <br />
remote machine.<br />
<br />
== psj ==<br />
psj<br />
shows just the procs running on the base system – a ps auxw but without jail’d procs present<br />
<br />
== perc5iraidchk ==<br />
perc5iraidchk<br />
checks for degraded arrays on Dell 2950 systems with Perc5/6 controllers<br />
<br />
== perc4eraidchk ==<br />
perc4eraidchk<br />
checks for degraded arrays on Dell 2850 systems with Perc4e/Di controllers<br />
<br />
= Virtuozzo VPS =<br />
<br />
Note: We use VEID (Virtual Environment ID) and CTID (Container ID) interchangably. Similarly, VE and CT. They mean the same thing.<br />
VZPP = VirtuoZzo Power Panel (the control panel for each CT)<br />
<br />
All linux systems exist in /vz, /vz1 or /vz2 - since each linux machine holds roughly 60-90 customers, there will be roughly 30-45 in each partition.<br />
<br />
The actual filesystem of the system in question is in:<br />
<br />
/vz(1-2)/private/(VEID)<br />
<br />
Where VEID is the identifier for that system - an all-numeric string larger than 100.<br />
<br />
The actual mounted and running systems are in the corresponding:<br />
<br />
/vz(1-2)/root/(VEID)<br />
<br />
But we rarely interact with any system from this mount point.<br />
<br />
You should never need to touch the root portion of their system – however you can traverse their filesystem by going to <tt>/vz(1-2)/private/(VEID)/root</tt> (<tt>/vz(1-2)/private/(VEID)/fs/root</tt> on 4.x systems) the root of their filesystem is in that directory, and their entire system is underneath that.<br />
<br />
Every VE has a startup script in <tt>/etc/sysconfig/vz-scripts</tt> (which is symlinked as <tt>/vzconf</tt> on all systems) - the VE startup script is simply named <tt>(VEID).conf</tt> - it contains all the system parameters for that VE:<br />
<br />
<pre># Configuration file generated by vzsplit for 60 VE<br />
# on HN with total amount of physical mem 2011 Mb<br />
<br />
VERSION="2"<br />
CLASSID="2"<br />
<br />
ONBOOT="yes"<br />
<br />
KMEMSIZE="8100000:8200000"<br />
LOCKEDPAGES="322:322"<br />
PRIVVMPAGES="610000:615000"<br />
SHMPAGES="33000:34500"<br />
NUMPROC="410:415"<br />
PHYSPAGES="0:2147483647"<br />
VMGUARPAGES="13019:2147483647"<br />
OOMGUARPAGES="13019:2147483647"<br />
NUMTCPSOCK="1210:1215"<br />
NUMFLOCK="107:117"<br />
NUMPTY="19:19"<br />
NUMSIGINFO="274:274"<br />
TCPSNDBUF="1800000:1900000"<br />
TCPRCVBUF="1800000:1900000"<br />
OTHERSOCKBUF="900000:950000"<br />
DGRAMRCVBUF="200000:200000"<br />
NUMOTHERSOCK="650:660"<br />
DCACHE="786432:818029"<br />
NUMFILE="7500:7600"<br />
AVNUMPROC="51:51"<br />
IPTENTRIES="155:155"<br />
DISKSPACE="4194304:4613734"<br />
DISKINODES="400000:420000"<br />
CPUUNITS="1412"<br />
QUOTAUGIDLIMIT="2000"<br />
VE_ROOT="/vz1/root/636"<br />
VE_PRIVATE="/vz1/private/636"<br />
NAMESERVER="69.55.225.225 69.55.230.3"<br />
OSTEMPLATE="vzredhat-7.3/20030305"<br />
VE_TYPE="regular"<br />
IP_ADDRESS="69.55.225.229"<br />
HOSTNAME="textengine.net"</pre><br />
<br />
<br />
As you can see, the hostname is set here, the disk space is set here, the number of inodes, the number of files that can be open, the number of tcp sockets, etc. - all are set here.<br />
<br />
In fact, everything that can be set on this customer system is set in this conf file.<br />
<br />
<br />
All interaction with the customer system is done with the VEID. You start the system by running:<br />
<br />
vzctl start 999<br />
<br />
You stop it by running:<br />
<br />
vzctl stop 999<br />
<br />
You execute commands in it by running:<br />
<br />
vzctl exec 999 df -k<br />
<br />
You enter into it, via a root-shell backdoor with:<br />
<br />
vzctl enter 999<br />
<br />
and you set parameters for the system, while it is still running, with:<br />
<br />
vzctl set 999 --diskspace 6100000:6200000 --save<br />
<br />
<tt>vzctl</tt> is the most commonly used command - we have aliased <tt>v</tt> to <tt>vzctl</tt> since we use it so often. We’ll continue to use <tt>vzctl</tt> in our examples, but feel free to use just <tt>v</tt>.<br />
<br />
Let's say the user wants more diskspace. You can cat their conf file and see:<br />
<br />
DISKSPACE="4194304:4613734"<br />
<br />
So right now they have 4gigs of space. You can then change it to 6 with:<br />
<br />
vzctl set 999 --diskspace 6100000:6200000 --save<br />
<br />
IMPORTANT: all issuances of the vzctl set command need to end with <tt>–save</tt> - if they don't, the setting will be set, but it will not be saved to the conf file, and they will not have those settings next time they boot.<br />
<br />
All of the tunables in the conf file can be set with the vzctl set command. Note that in the conf file, and on the vzctl set command line, we always issue two numbers seperated by a colon - that is because we are setting the hard and soft limits. Always set the hard limit slightly above the soft limit, as you see it is in the conf file for all those settings.<br />
<br />
There are also things you can set with `<tt>vzctl set</tt>` that are not in the conf file as settings, per se. For instance, you can add IPs:<br />
<br />
vzctl set 999 --ipadd 10.10.10.10 --save<br />
<br />
or multiple IPs:<br />
<br />
vzctl set 999 --ipadd 10.10.10.10 --ipadd 10.10.20.30 --save<br />
<br />
or change the hostname:<br />
<br />
vzctl set 999 --hostname www.example.com --save<br />
<br />
You can even set the nameservers:<br />
<br />
vzctl set 999 --nameserver 198.78.66.4 --nameserver 198.78.70.180 --save<br />
<br />
Although you probably will never do that.<br />
<br />
You can disable a VPS from being started (by VZPP or reboot) (4.x):<br />
<br />
vzctl set 999 --disabled yes --save <br />
<br />
You can disable a VPS from being started (by VZPP or reboot) (<=3.x):<br />
<br />
vzctl set 999 --onboot=no --save <br />
<br />
You can disable a VPS from using his control panel:<br />
<br />
vzctl set 999 --offline_management=no --save <br />
<br />
You can suspend a VPS, so it can be resumed in the same state it was in when it was stopped (4.x):<br />
<br />
vzctl suspend 999<br />
<br />
and to resume it:<br />
<br />
vzctl resume 999<br />
<br />
to see who owns process:<br />
vzpid <PID><br />
<br />
to mount up an unmounted ve:<br />
vzctl mount 827<br />
<br />
<br />
To see network stats for CT's:<br />
<pre># vznetstat<br />
VEID Net.Class Output(bytes) Input(bytes)<br />
-----------------------------------------------<br />
24218 1 484M 39M<br />
24245 1 463M 143M<br />
2451 1 2224M 265M<br />
2454 1 2616M 385M<br />
4149 1 125M 68M<br />
418 1 1560M 34M<br />
472 1 1219M 315M<br />
726 1 628M 317M<br />
763 1 223M 82M<br />
771 1 4234M 437M<br />
-----------------------------------------------<br />
Total: 13780M 2090M</pre><br />
<br />
<br />
One thing that sometimes comes up on older systems that we created with smaller defaults is that the system would run out of inodes. The user will email and say they cannot create any more files or grow any files larger, but they will also say that they are not out of diskspace ... they are running:<br />
<br />
df -k<br />
<br />
and seeing how much space is free - and they are not out of space. They are most likely out of inodes - which they would see by running:<br />
<br />
df -i<br />
<br />
So, the first thing you should do is enter their system with:<br />
<br />
vzctl enter 999<br />
<br />
and run: <tt>df -i</tt><br />
<br />
to confirm your theory. Then exit their system. Then simply cat their conf file and see what their inodes are set to (probably 200000:200000, since that was the old default on the older systems) and run:<br />
<br />
vzctl set 999 --diskinodes 400000:400000 --save<br />
<br />
If they are not out of inodes, then a good possibility is that they have maxed out their numfile configuration variable, which controls how many files they can have in their system. The current default is 7500 (which nobody has ever hit), but the old default was as low as 2000, so you would run something like:<br />
<br />
vzctl set 999 --numfile 7500:7500 --save<br />
<br />
----<br />
<br />
You cannot start or stop a VE if your pwd is its private (/vz/private/999) or root (/vz/root/999) directories, or anywhere below them.<br />
<br />
<br />
== Recovering from a crash (linux) ==<br />
<br />
=== Diagnose whether you have a crash ===<br />
The most important thing is to get the machine and all ve’s back up as soon as possible. Note the time, you’ll need to create a crash log entry (Mgmt. -> Reference -> CrashLog). The first thing to do is head over to the [[Screen#Screen_Organization|serial console screen]] and see if there’s any kernel error messages output. Try to copy any messages (or just a sample of repeating messages) you see into the notes section of the crash log – these will also likely need to be sent to virtuozzo for interpretation. If the messages are spewing too fast, hit ^O + H to start a screen log dump which you can ob1182.pts-38.bb serve after the machine is rebooted. Additionally, if the machine is responsive, you can get a trace to send to virtuozzo by hooking up a kvm and entering these 3 sequences:<br />
<pre>alt+print screen+m<br />
alt+print screen+p<br />
alt+print screen+t</pre><br />
<br />
If there are no messages, the machine may just be really busy- wait a bit (5-10min) to see if it comes back. If it's still pinging, odds are its very busy. If it doesn't come back, or the messages indicate a fatal error, you will need to proceed with a power cycle (ctrl+alt+del will not work).<br />
<br />
=== Power cycle the server ===<br />
If this machine is not a Dell 2950 with a [[DRAC/RMM#DRAC|DRAC card]] (i.e. if you can’t ssh into the DRAC card and issue racadm serveraction hardreset, then you will need someone at the data center to power the macine off, wait 30 sec, then turn it back on. Make sure to re-attach via console (<tt>tip virtxx</tt>) immediately after power down. <br />
<br />
=== (Re)attach to the console ===<br />
Stay on the console the entire time during boot. As the BIOS posts- look out for the RAID card output- does everything look healthy? The output may be scrambled, look for "DEGRADED" or "FAILED". Once the OS starts booting you will be disconnected (dropped back to the shell on the console server) a couple times during the boot up. The reason you want to quickly re-attach is two-fold: 1. If you don’t reattach quickly then you won’t get any console output, 2. you want to be attached before the server ''potentially'' starts (an extensive) fsck. If you attach after the fsck begins, you’ll have seen no indication it started an fsck and the server will appear frozen during startup- no output, no response. <br />
<br />
=== Start containers/VE's/VPSs ===<br />
When the machine begins to start VE’s, it’s safe to leave the console and login via ssh. All virts should be set to auto start all the VEs after a crash. Further, most (newer) virts are set to “fastboot” it’s VE’s (to find out, do:<br />
grep -i fast /etc/sysconfig/vz <br />
and look for <tt>VZFASTBOOT=yes</tt>). If this was set prior to the machine’s crash (setting it after the machine boots will not have any effect until the vz service is restarted) it will start each ve as fast as possible, in serial, then go thru each VE (serially), shutting it down running a vzquota (disk usage) check, then bringing it back up. The benefit is that all VE’s are brought up quickly (within 15min or so depending on the #), the downside is a customer watching closely will notice 2 outages – 1st the machine crash, 2nd their quota check (which will be a much shorter downtime- on the order of a few minutes). <br />
<br />
Where “fastboot” is not set to yes (i.e on quar1), vz will start them consecutively, checking the quotas one at a time, and the 60th VE may not start until an hour or two later - this is not acceptable.<br />
<br />
The good news is, if you run vzctl start for a VE that is already started, you will simply get an error: <tt>VE is already started</tt>. Further, if you attempt to vzctl start a VE that is in the process of being started, you will simply get an error: unable to lock VE. So, there is no danger in simply running scripts to start smaller sets of VEs. If the system is not autostarting, then there is no issue, and even if it does, when it conflicts, one process (yours or the autostart) will lose, and just move on to the next one.<br />
<br />
A script has been written to assist with ve starts: [[#startvirt.pl|startvirt.pl]] which will start 6 ve’s at once until there are no more left. If startvirt.pl is used on a system where “fastboot” was on, it will circumvent the fastboot for ve’s started by startvirt.pl – they will go through the complete quota check before starting- therefore this is not advisable when a system has crashed. When a system is booted cleanly, and there's no need for vzquota checks, then startvirt.pl is safe and advisable to run.<br />
<br />
=== Make sure all containers are running ===<br />
You can quickly get a feel for how many ve’s are started by running:<br />
<br />
<pre>[root@virt4 log]# vs<br />
VEID 16066 exist mounted running<br />
VEID 16067 exist mounted running<br />
VEID 4102 exist mounted running<br />
VEID 4112 exist mounted running<br />
VEID 4116 exist mounted running<br />
VEID 4122 exist mounted running<br />
VEID 4123 exist mounted running<br />
VEID 4124 exist mounted running<br />
VEID 4132 exist mounted running<br />
VEID 4148 exist mounted running<br />
VEID 4151 exist mounted running<br />
VEID 4155 exist mounted running<br />
VEID 42 exist mounted running<br />
VEID 432 exist mounted running<br />
VEID 434 exist mounted running<br />
VEID 442 exist mounted running<br />
VEID 450 exist mounted running<br />
VEID 452 exist mounted running<br />
VEID 453 exist mounted running<br />
VEID 454 exist mounted running<br />
VEID 462 exist mounted running<br />
VEID 463 exist mounted running<br />
VEID 464 exist mounted running<br />
VEID 465 exist mounted running<br />
VEID 477 exist mounted running<br />
VEID 484 exist mounted running<br />
VEID 486 exist mounted running<br />
VEID 490 exist mounted running</pre><br />
<br />
So to see how many ve’s have started:<br />
<pre>[root@virt11 root]# vs | grep running | wc -l<br />
39</pre><br />
<br />
And to see how many haven’t:<br />
<pre>[root@virt11 root]# vs | grep down | wc -l<br />
0</pre><br />
<br />
And how many we should have running:<br />
<pre>[root@virt11 root]# vs | wc -l<br />
39</pre><br />
<br />
Another tool you can use to see which ve’s have started, among other things is [[#vzstat|vzstat]]. It will give you CPU, memory, and other stats on each ve and the overall system. It’s a good thing to watch as ve’s are starting (note the VENum parameter, it will tell you how many have started):<br />
<br />
<pre>4:37pm, up 3 days, 5:31, 1 user, load average: 1.57, 1.68, 1.79<br />
VENum 40, procs 1705: running 2, sleeping 1694, unint 0, zombie 9, stopped 0<br />
CPU [ OK ]: VEs 57%, VE0 0%, user 8%, sys 7%, idle 85%, lat(ms) 412/2<br />
Mem [ OK ]: total 6057MB, free 9MB/54MB (low/high), lat(ms) 0/0<br />
Swap [ OK ]: tot 6142MB, free 4953MB, in 0.000MB/s, out 0.000MB/s<br />
Net [ OK ]: tot: in 0.043MB/s 402pkt/s, out 0.382MB/s 4116pkt/s<br />
Disks [ OK ]: in 0.002MB/s, out 0.000MB/s<br />
<br />
VEID ST %VM %KM PROC CPU SOCK FCNT MLAT IP<br />
1 OK 1.0/17 0.0/0.4 0/32/256 0.0/0.5 39/1256 0 9 69.55.227.152<br />
21 OK 1.3/39 0.1/0.2 0/46/410 0.2/2.8 23/1860 0 6 69.55.239.60<br />
133 OK 3.1/39 0.1/0.3 1/34/410 6.3/2.8 98/1860 0 0 69.55.227.147<br />
263 OK 2.3/39 0.1/0.2 0/56/410 0.3/2.8 34/1860 0 1 69.55.237.74<br />
456 OK 17/39 0.1/0.2 0/100/410 0.1/2.8 48/1860 0 11 69.55.236.65<br />
476 OK 0.6/39 0.0/0.2 0/33/410 0.1/2.8 96/1860 0 10 69.55.227.151<br />
524 OK 1.8/39 0.1/0.2 0/33/410 0.0/2.8 28/1860 0 0 69.55.227.153<br />
594 OK 3.1/39 0.1/0.2 0/45/410 0.0/2.8 87/1860 0 1 69.55.239.40<br />
670 OK 7.7/39 0.2/0.3 0/98/410 0.0/2.8 64/1860 0 216 69.55.225.136<br />
691 OK 2.0/39 0.1/0.2 0/31/410 0.0/0.7 25/1860 0 1 69.55.234.96<br />
744 OK 0.1/17 0.0/0.5 0/10/410 0.0/0.7 7/1860 0 6 69.55.224.253<br />
755 OK 1.1/39 0.0/0.2 0/27/410 0.0/2.8 33/1860 0 0 192.168.1.4<br />
835 OK 1.1/39 0.0/0.2 0/19/410 0.0/2.8 5/1860 0 0 69.55.227.134<br />
856 OK 0.3/39 0.0/0.2 0/13/410 0.0/2.8 16/1860 0 0 69.55.227.137<br />
936 OK 3.2/52 0.2/0.4 0/75/410 0.2/0.7 69/1910 0 8 69.55.224.181<br />
1020 OK 3.9/39 0.1/0.2 0/60/410 0.1/0.7 55/1860 0 8 69.55.227.52<br />
1027 OK 0.3/39 0.0/0.2 0/14/410 0.0/2.8 17/1860 0 0 69.55.227.83<br />
1029 OK 1.9/39 0.1/0.2 0/48/410 0.2/2.8 25/1860 0 5 69.55.227.85<br />
1032 OK 12/39 0.1/0.4 0/80/410 0.0/2.8 41/1860 0 8 69.55.227.90</pre><br />
<br />
When you are all done, you will want to make sure that all the VEs really did get started, run vs one more time.<br />
<br />
Note the time all ve’s are back up and enter that into and save the crash log entry.<br />
<br />
Occasionally, a ve will not start automatically. The most common reason for a ve not to come up normally is the ve was at it’s disk limit before the crash, and will not start since they’re over the limit. To overcome this, set the disk space to current usage level (the system will give this to you when it fails to start), start the ve, then re-set the disk space back to the prior level. Lastly, contact the customer to let them know they’re out of disk (or allocate more disk if they're entitled to more).<br />
<br />
== Hitting performance barriers and fixing them ==<br />
<br />
There are multiple modes virtuozzo offers to allocate resources to a ve. We utilize 2: SLM and UBC parameters<br />
On our 4.x systems, we use all SLM – it’s simpler to manage and understand. There are a few systems on virt19/18 that may also use SLM. Everything else uses UBC. <br />
You can tell a SLM ve by:<br />
<br />
SLMMODE="all"<br />
<br />
in their conf file. <br />
<br />
TODO: detail SLM modes and parameters.<br />
<br />
If someone is in SLM mode and they hit memory resource limits, they simply need to upgrade to more memory.<br />
<br />
The following applies to everyone else (UBC).<br />
<br />
Customers will often email and say that they are getting out of memory errors - a common one is "cannot fork" ... basically, anytime you see something odd like this, it means they are hitting one of their limits that is in place in their conf file.<br />
<br />
The conf file, however, simply shows their limits - how do we know what they are currently at ?<br />
<br />
The answer is a file called v - this file contains the current status (and peaks) of their performance settings, and also counts how many times they have hit the barrier. The output of the file looks like this:<br />
<br />
<pre>764: kmemsize 384113 898185 8100000 8200000 0<br />
lockedpages 0 0 322 322 0<br />
privvmpages 1292 7108 610000 615000 0<br />
shmpages 270 528 33000 34500 0<br />
dummy 0 0 0 0 0<br />
numproc 8 23 410 415 0<br />
physpages 48 5624 0 2147483647 0<br />
vmguarpages 0 0 13019 2147483647 0<br />
oomguarpages 641 6389 13019 2147483647 0<br />
numtcpsock 3 21 1210 1215 0<br />
numflock 1 3 107 117 0<br />
numpty 0 2 19 19 0<br />
numsiginfo 0 4 274 274 0<br />
tcpsndbuf 0 80928 1800000 1900000 0 <br />
tcprcvbuf 0 108976 1800000 1900000 0<br />
othersockbuf 2224 37568 900000 950000 0<br />
dgramrcvbuf 0 4272 200000 200000 0<br />
numothersock 3 9 650 660 0<br />
dcachesize 53922 100320 786432 818029 0<br />
numfile 161 382 7500 7600 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
numiptent 4 4 155 155 0</pre><br />
<br />
The first column is the name of the counter in question - the same names we saw in the systems conf file. The second column is the _current_ value of that counter, the third column is the max that that counter has ever risen to, the fourth column is the soft limit, and the fifth column is the hard limit (which is the same as the numbers in that systems conf file).<br />
<br />
The sixth number is the failcount - how many times the current usage has risen to hit the barrier. It will increase as soon as the current usage hits the soft limit.<br />
<br />
The problem with /proc/user_beancounters is that it actually contains that set of data for every running VE - so you can't just cat /proc/user_beancounters - it is too long and you get info for every other running system.<br />
<br />
You can vzctl enter the system and run:<br />
<br />
vzctl enter 9999<br />
cat /proc/user_beancounters<br />
<br />
inside their system, and you will just see the stats for their particular system, but entering their system every time you want to see it is combersome.<br />
<br />
So, I wrote a simple script called "vzs" which simply greps for the VEID, and spits out the next 20 or so lines (however many lines there are in the output, I forget) after it. For instance:<br />
<br />
<pre># vzs 765:<br />
765: kmemsize 2007936 2562780 8100000 8200000 0<br />
lockedpages 0 8 322 322 0<br />
privvmpages 26925 71126 610000 615000 0<br />
shmpages 16654 16750 33000 34500 0<br />
dummy 0 0 0 0 0<br />
numproc 41 57 410 415 0<br />
physpages 1794 49160 0 2147483647 0<br />
vmguarpages 0 0 13019 2147483647 0<br />
oomguarpages 4780 51270 13019 2147483647 0<br />
numtcpsock 23 37 1210 1215 0<br />
numflock 17 39 107 117 0<br />
numpty 1 3 19 19 0<br />
numsiginfo 0 6 274 274 0<br />
tcpsndbuf 22240 333600 1800000 1900000 0<br />
tcprcvbuf 0 222656 1800000 1900000 0<br />
othersockbuf 104528 414944 900000 950000 0<br />
dgramrcvbuf 0 4448 200000 200000 0<br />
numothersock 73 105 650 660 0<br />
dcachesize 247038 309111 786432 818029 0<br />
numfile 904 1231 7500 7600 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
numiptent 4 4 155 155 0</pre><br />
<br />
That showed us just the portion of /proc/user_beancounters for system 765.<br />
<br />
When you run the vzs command, always add a : after the VEID.<br />
<br />
So, if a customer complains about some out of memory errors, or no more files, or no more ptys, or just has an unspecific complain about processes dying, etc., the very first thing you need to do is check their beancounters with vzs. Usually you will spot an item that has a high failcount and needs to be upped.<br />
<br />
At that point you could simply up the counter with `vzctl set`. Generally pick a number 10-20% higher than the old one, and make the hard limit slightly larger than the the soft limit. However our systems now come in several levels and those levels have more/different memory allocations. If someone is complaining about something other than a memory limit (pty, numiptent, numflock), it’s generally safe to increase it, at least to the same level as what’s in the /vzconf/4unlimited file on the newest virt. If someone is hitting a memory limit, first make sure they are given what they deserve:<br />
<br />
(refer to mgmt -> payments -> packages)<br />
<br />
To set those levels, you use the [[#setmem|setmem]] command. <br />
<br />
The alternate (DEPRECATED) method would be to use one of 3 commands:<br />
256 <veid><br />
300 <veid><br />
384 <veid><br />
512 <veid><br />
<br />
If the levels were not right (you’d run vzs <veid> before and after to see the effect) tell the customer they’ve been adjusted and be done with it. If the levels were right, tell the customer they must upgrade to a higher package, tell them how to see level (control panel) and that they can reboot their system to escape this lockup contidion.<br />
<br />
Customers can also complain that their site is totally unreachable, or complain that it is down ... if the underlying machine is up, and all seems well, you may notice in the beancounters that network-specific counters are failing - such as numtcpsock, tcpsndbuf or tcprcvbuf. This will keep them from talking on the network and make it seem like their system is down. Again, just up the limits and things should be fine.<br />
<br />
On virts 1-4, you should first look at the default settings for that item on a later virt, such as virt 8 - we have increased the defaults a lot since the early machines. So, if you are going to up a counter on virt2, instead of upping it by 10-20%, instead up it to the new default that you see on virt8.<br />
<br />
== Moving a VE to another virt (migrate/migrateonline) ==<br />
<br />
This will take a while to complete - and it is best to do this at night when the load is light on both machines.<br />
<br />
There are different methods for this, depending on which version of virtuozzo is installed on the src. and dst. virt. <br />
To check which version is running: <br />
[root@virt12 private]# cat /etc/virtuozzo-release<br />
Virtuozzo release 2.6.0<br />
<br />
Ok, let's say that the VE is 1212, and vital stats are:<br />
<pre>[root@virt12 sbin]# vc 1212<br />
VE_ROOT="/vz1/root/1212"<br />
VE_PRIVATE="/vz1/private/1212"<br />
OSTEMPLATE="fedora-core-2/20040903"<br />
IP_ADDRESS="69.55.229.84"<br />
TEMPLATES="devel-fc2/20040903 php-fc2/20040813 mysql-fc2/20040812 postgresql-fc2/20040813 mod_perl-fc2/20040812 mod_ssl-fc2/20040811 jre-fc2/20040823 jdk-fc2/20040823 mailman-fc2/20040823 analog-fc2/20040824 proftpd-fc2/20040818 tomcat-fc2/20040823 usermin-fc2/20040909 webmin-fc2/20040909 uw-imap-fc2/20040830 phpBB-fc2/20040831 spamassassin-fc2/20040910 PostNuke-fc2/20040824 sl-webalizer-fc2/20040<br />
818"<br />
<br />
[root@virt12 sbin]# vzctl exec 1212 df -h<br />
Filesystem Size Used Avail Use% Mounted on<br />
/dev/vzfs 4.0G 405M 3.7G 10% /</pre><br />
<br />
From this you can see that he’s using (and will minimally need free on the dst server) ~400MB, and he’s running on a Fedora 2 template, version 20040903. He’s also got a bunch of other templates installed. It’s is '''vital''' that '''all''' these templates exist on the dst system. To confirm that, on the dst system run:<br />
<br />
For < 3.0:<br />
<br />
<pre>[root@virt14 private]# vzpkgls | grep fc2<br />
devel-fc2 20040903<br />
PostNuke-fc2 20040824<br />
analog-fc2 20040824<br />
awstats-fc2 20040824<br />
bbClone-fc2 20040824<br />
jdk-fc2 20040823<br />
jre-fc2 20040823<br />
mailman-fc2 20040823<br />
mod_frontpage-fc2 20040816<br />
mod_perl-fc2 20040812<br />
mod_ssl-fc2 20040811<br />
mysql-fc2 20040812<br />
openwebmail-fc2 20040817<br />
php-fc2 20040813<br />
phpBB-fc2 20040831<br />
postgresql-fc2 20040813<br />
proftpd-fc2 20040818<br />
sl-webalizer-fc2 20040818<br />
spamassassin-fc2 20040910<br />
tomcat-fc2 20040823<br />
usermin-fc2 20040909<br />
uw-imap-fc2 20040830<br />
webmin-fc2 20040909<br />
[root@virt14 private]# vzpkgls | grep fedora<br />
fedora-core-1 20040121 20040818<br />
fedora-core-devel-1 20040121 20040818<br />
fedora-core-2 20040903<br />
[root@virt14 private]#</pre><br />
<br />
For these older systems, you can simply match up the date on the template. <br />
<br />
For >= 3.0:<br />
<br />
<pre>[root@virt19 /vz2/private]# vzpkg list<br />
centos-5-x86 2008-01-07 22:05:57<br />
centos-5-x86 devel<br />
centos-5-x86 jre<br />
centos-5-x86 jsdk<br />
centos-5-x86 mod_perl<br />
centos-5-x86 mod_ssl<br />
centos-5-x86 mysql<br />
centos-5-x86 php<br />
centos-5-x86 plesk9<br />
centos-5-x86 plesk9-antivirus<br />
centos-5-x86 plesk9-api<br />
centos-5-x86 plesk9-atmail<br />
centos-5-x86 plesk9-backup<br />
centos-5-x86 plesk9-horde<br />
centos-5-x86 plesk9-mailman<br />
centos-5-x86 plesk9-mod-bw<br />
centos-5-x86 plesk9-postfix<br />
centos-5-x86 plesk9-ppwse<br />
centos-5-x86 plesk9-psa-firewall<br />
centos-5-x86 plesk9-psa-vpn<br />
centos-5-x86 plesk9-psa-fileserver<br />
centos-5-x86 plesk9-qmail<br />
centos-5-x86 plesk9-sb-publish<br />
centos-5-x86 plesk9-vault<br />
centos-5-x86 plesk9-vault-most-popular<br />
centos-5-x86 plesk9-watchdog</pre><br />
<br />
On these newer systems, it's difficult to tell whether the template on the dst matches exactly the src. Just cause a centos-5-x86 is listed on both servers doesn't mean all the same packages are there on the dst. To truly know, you must perform a sample rsync:<br />
<br />
rsync -avn /vz/template/centos/5/x86/ root@10.1.4.61:/vz/template/centos/5/x86/<br />
<br />
if you see a ton of output from the dry run command, then clearly there are some differences. You may opt to let the rsync complete (without running in dry run mode) the only downside is you've now used up more space on the dst and also the centos template will be a mess with old and new data- it will be difficult if not impossible to undo (if someday we wanted to reclaim the space).<br />
<br />
If you choose to merge templates, you should closely inspect the dry run output. You should also take care to exclude anything in the /config directory. For example:<br />
<br />
rsync -av -e ssh --stats --exclude=x86/config /vz/template/ubuntu/10.04/ root@10.1.4.62:/vz/template/ubuntu/10.04/<br />
<br />
Which will avoid this directory and contents:<br />
<pre>[root@virt11 /vz2/private]# ls /vz/template/ubuntu/10.04/x86/config*<br />
app os</pre><br />
<br />
This is important to avoid since the config may differ on the destination and we are really only interested in making sure the pacakges are there, not overwriting a newer config with an older one.<br />
<br />
If the dst system was missing a template, you have 2 choices: <br />
# put the missing template on the dst system. 2 choices here: <br />
## Install the template from rpm (found under backup2: /mnt/data4/vzrpms/distro/) or <br />
## rsync over the template (found under /vz/template) - see above<br />
# put the ve on a system which has all the proper templates<br />
<br />
=== pre-seeding a migration ===<br />
<br />
When migrating a customer (or when doing many) depending on how much data you have to transfer, it can take some time. Further, it can be difficult to gauge when a migration will complete or how long it will take. To help speed up the process and get a better idea about how long it will take you can pre-transfer a customer's data to the destination server. If done correctly, vzmigrate will see the pre-transferred data and pick up where you left off, having much less to transfer (just changed/new files). <br />
<br />
We believe vzmigrate uses rsync to do it's transfer. Therefore not only can you use rsync to do a pre-seed, you can also run rsync to see what is causing a repeatedly-failing vzmigrate to fail. <br />
<br />
There's no magic to a pre-seed, you just need to make sure it's named correctly.<br />
<br />
Given:<br />
<br />
source: /vz1/private/1234<br />
<br />
and you want to migrate to /vz2 on the target system, your rsync would look like:<br />
<br />
rsync -av /vz1/private/1234/ root@x.x.x.x:/vz2/private/1234.migrated/<br />
<br />
After running that successful rsync, the ensuing migrateonline (or migrate) will take much less time to complete- depending on the # of files to be analyzed and the # of changed files. In any case, it'll be much much faster than had you just started the migration from scratch.<br />
<br />
Further, as we discuss elsewhere in this topic, a failed migration can be moved from <tt>/vz/private/1234</tt> to <tt>/vz/private/1234.migrated</tt> on the destination if you want to restart a failed migration. This should '''only''' be done if the migration failed and the CT is not running on the destination HN.<br />
<br />
=== migrateonline intructions: src >=3.x -> dst>=3.x ===<br />
<br />
A script called [[#migrateonline|migrateonline]] was written to handle this kind of move. It is basically a wrapper for <tt>vzmigrate</tt> – vzmigrate is a util to seamlessly- as no no reboot of the ve necessary- move a ve from one host to another. This wrapper was initially written cause vz virtuozzo version 2.6.0 has a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables, causing problems when the ve was started up on the dst system. [[#migrate|migrate]] mitigates that. Since it makes multiple ssh connections to the dst virt, it’s a good idea to put the pub key for the src virt in the authorized_keys file on the dst virt. In addition, migrateonline emails ve owners when their migration starts and stops. For this to happen they need to put email addresses (on a single line, space delimited) in a file on their system: /migrate_notify. If the optional target dir is not specified, the ve will be moved to the same private/root location as it was on the src virt. Note: <tt>migrate</tt> is equivalent to <tt>migrateonline</tt>, but will <tt>migrate</tt> a ve AND restart it in the process.<br />
<br />
<pre>[root@virt12 sbin]# migrateonline<br />
usage: /usr/local/sbin/migrateonline <ip of node migrating to> <veid> [target dir: vz | vz1 | vz2]<br />
[root@virt12 sbin]# migrateonline 10.1.4.64 1212 vz<br />
starting to migrate 1212 at Sat Mar 26 22:40:38 PST 2005<br />
Turning off offline_management<br />
Saved parameters for VE 1212<br />
migrating with no start on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
clearing the arp cache<br />
now going to 10.1.4.64 and clear cache and starting<br />
starting it<br />
Delete port redirection<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
finished migrating 1212 at Sat Mar 26 22 22:52:01 PST 2005<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine.<br />
<br />
If they had backups, use the mvbackups command to move their backups to the new server:<br />
<br />
mvbackups 1212 virt14 vz<br />
<br />
Rename the ve<br />
<br />
[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/migrated-1212<br />
<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/old-1212-migrated-20120404-noarchive<br />
<br />
Update the customer’s systems in mgmt to reflect the new path and server.<br />
<br />
<br />
IF migrateonline does not work, you can try again using simply migrate- this will result in a brief reboot for the ve.<br />
Before you try again, make sure of a few things:<br />
<br />
Depending on where in the migration died, there may be partial data on the dst system in 1 of 2 places:<br />
(given the example above)<br />
<br />
/vz/private/1212<br />
<br />
or <br />
<br />
/vz/private/1212.migrated<br />
<br />
before you run migrate again, you'll want to rename so that all data is in <br />
1212.migrated:<br />
<br />
mv /vz/private/1212 /vz/private/1212.migrated<br />
<br />
this way, it will pick up where it left off and transfer only new files.<br />
<br />
Likewise, if you want to speed up a migration, you can pre-seed the dst as follows:<br />
<br />
[root@virt12 sbin]# rsync -avSH /vz/private/1212/ root@10.1.4.64:/vz/private/1212.migrated/<br />
<br />
then when you run migrate or migrateonline, it will only need to move the changed files- the migration will complete quickly<br />
<br />
=== migrate manually (if migrate/online fails) ===<br />
<br />
Lets say for whatever reason the migration fails. If it fails with [[#migrateonline|migrateonline]], you should try [[#migrate|migrate]] (which will reboot the customer, so notify them ahead of time).<br />
<br />
You may want to run a pre-seed rsync to see if you can find the problem. On older virts, we've seen this problem due to a large logfile (which you can find and encourage the customer to remove/compress):<br />
for f in `find / -size +1048576k`; do ls -lh $f; done<br />
<br />
If the rsync or [[#migrate|migrate]] fails, <br />
<br />
You can always move someone manually:<br />
<br />
1. stop ve: <br><br />
v stop 1234<br />
<br />
2. copy over data<br><br />
rsync -avSH /vz/private/1234/ root@1.1.1.1:/vzX/private/1234/<br />
<br />
NOTE: if you've previously seeded the data (run rsync while the VE was up/running), and this is a subsequent rsync, make sure the last rsync you do (while the VE is not running, has the --delete option in the rsync)<br />
<br />
3. copy over conf<br><br />
scp /vzconf/1234.conf root@1.1.1.1:/vzconf<br />
<br />
4. on dst, edit the conf to reflect the right vzX dir<br><br />
vi /vzconf/1234.conf<br />
<br />
5. on src remove the IPs<br><br />
ipdel 1234 2.2.2.2 3.3.3.3<br />
<br />
6. on dst add IPs <br><br />
ipadd 1234 2.2.2.2 3.3.3.3<br />
<br />
7. on dst, start ve: <br><br />
v start 1324<br />
<br />
8. cancel, then archive ve on src per above instrs.<br />
<br />
=== migrate src=2.6.0 -> dst>=2.6.0, or mass-migration with customer notify ===<br />
<br />
A script called <tt>migrate</tt> was written to handle this kind of move. It is basically a wrapper for vzmigrate – vzmigrate is a util to seamlessly move a ve from one host to another. This wrapper was initially written cause vz virtuozzo version 2.6.0 has a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables, causing problems when the ve was started up on the dst system. migrate mitigates that. Since it makes multiple ssh connections to the dst virt, it’s a good idea to put the pub key for the src virt in the authorized_keys file on the dst virt. In addition, migrate emails ve owners when their migration starts and stops. For this to happen they need to put email addresses (on a single line, space delimited) in a file on their system: /migrate_notify. If the optional target dir is not specified, the ve will be moved to the same private/root location as it was on the src virt. Note: migrateonline is equivalent to migrate, but will migrate a ve from one 2.6 '''kernel''' machine to another 2.6 kernel machine without restarting the ve.<br />
<br />
<pre>[root@virt12 sbin]# migrate<br />
usage: /usr/local/sbin/migrate <ip of node migrating to> <veid> [target dir: vz | vz1 | vz2]<br />
[root@virt12 sbin]# migrate 10.1.4.64 1212 vz<br />
starting to migrate 1212 at Sat Mar 26 22:40:38 PST 2005<br />
Turning off offline_management<br />
Saved parameters for VE 1212<br />
migrating with no start on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
clearing the arp cache<br />
now going to 10.1.4.64 and clear cache and starting<br />
starting it<br />
Delete port redirection<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
finished migrating 1212 at Sat Mar 26 22 22:52:01 PST 2005<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine (backup2 or mail).<br />
Cancel the ve (first we have to rename things which migrate changed so cancelve will find them):<br />
<br />
[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/1212.conf<br />
<br />
On 2.6.1 you’ll also have to move the private area:<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/1212<br />
<br />
<pre>[root@virt12 sbin]# cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, [[#cancelve|cancelve]] would offer to remove them. You want to say '''no''' to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
To move everyone off a system, you’d do:<br />
for f in `vl`; do migrate <ip> $f; done<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
=== vzmigrate: src=2.6.1 -> dst>=2.6.0 ===<br />
<br />
This version of vzmigrate works properly with regard to handling ips. It will not notify ve owners of moves as in the above example. Other than that it’s essentially the same.<br />
<br />
<pre>[root@virt12 sbin]# vzmigrate 10.1.4.64 -r no 1212:1212:/vz/private/1212:/vz/root/1212<br />
migrating on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine (backup2 or mail).<br />
Cancel the ve (first we have to rename things which vzmigrate changed so cancelve will find them):<br />
<br />
<pre>[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/1212.conf<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/1212<br />
<br />
[root@virt12 sbin]# cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, <tt>cancelve</tt> would offer to remove them. You want to say no to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
=== src=2.5.x ===<br />
<br />
First, go to the private dir:<br />
<br />
cd /vz1/private/<br />
<br />
Stop the VE - make sure it stops totally cleanly.<br />
<br />
vzctl stop 1212<br />
<br />
Then you’d use vemove - a script written to copy over the config, create tarballs of the ve’s data on the destination virt, and cancel the ve on the source system (in this example we’re going to put a ve that was in /vz1/private on the src virt, in /vz/private on the dst virt):<br />
<br />
<pre>[root@virt12 sbin]# vemove<br />
ERROR: Usage: vemove veid target_ip target_path_dir<br />
[root@virt12 sbin]# vemove 1212 10.1.4.64 /vz/private/1212<br />
tar cfpP - 1212 --ignore-failed-read | (ssh -2 -c arcfour 10.1.4.64 "split - -b 1024m /vz/private/1212.tar" )<br />
scp /vzconf/1212.conf 10.1.4.64:/vzconf<br />
cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, cancelve would offer to remove them. You want to say no to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
When you are done, go to /vz/private on the dst virt you will have files like this:<br />
<br />
<pre>1212.taraa<br />
1212.tarab<br />
1212.tarac</pre><br />
<br />
Each one 1024m (or less, for the last one) in size.<br />
<br />
on the dst server and run:<br />
<br />
cat 1212.tar?? | tar xpPBf -<br />
<br />
and after 20 mins or so it will be totally untarred. Now since the conf<br />
file is already there, you can go ahead and start the system.<br />
<br />
vzctl start 1212<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
NOTE: you MUST tar the system up using the virtuozzo version of tar that<br />
is on all the virt systems, and further you MUST untar the tarball with<br />
the virtuozzo tar, using these options: `<tt>tar xpPBf -</tt>`<br />
<br />
If you tar up an entire VE and move it to a non-virtuozzo machine, that is<br />
ok, and you can untar it there with normal tar commands, but do not untar<br />
it and then repack it with a normal tar and expect it to work - you need<br />
to use virtuozzo tar commands on virtuozzo tarballs to make it work.<br />
<br />
The backups are sort of an exception, since we are just (usually)<br />
restoring user data that was created after we gave them the system, and<br />
therefore has nothing to do with magic symlinks or vz-rpms, etc.<br />
<br />
== Moving a VE on the same virt ==<br />
<br />
Easy way:<br><br />
Scenario 1: ve 123 is to be renamed 1231 and moved from vz1 to vz<br />
<br />
vzmlocal 123:1231:/vz/private/1231:/vz/root/1231<br />
<br />
Scenario 2: ve 123 is to be moved vz1 to vz<br />
<br />
vzmlocal 123:123:/vz/private/123:/vz/root/123<br />
<br />
vzmlocal will reboot the ve at the end of the move<br />
<br />
Manual/old way:<br />
<br />
1) <tt>vzctl stop 123</tt><br><br />
2) <tt>mv /vz1/private/123 /vz/private/.</tt><br><br />
(or cp -a if you want to copy)<br />
3) in <tt>/etc/sysconfig/vz-scripts/123.conf</tt> change value<br><br />
of '<tt>VE_PRIVATE</tt>' variable to point to a new private area location<br />
4) <tt>vzctl start 123</tt><br><br />
5) update backups if needed: <tt>mvbackups 123 virtX virt1 vz</tt><br><br />
6) update management scerens<br><br />
<br />
Notes: a) absolute path to private area is stored in quota file <tt>/var/vzquota/quota.123</tt> - so during first startup quota will be recalculated.<br><br />
b) if you're going to write some script to do a job, you MUST be sure that $VEID won't be expanded to '' in ve config file - ie. you need to escape '$'. Otherwise you might have:<br />
<br />
VE_PRIVATE="/vz/private/"<br />
<br />
in config, and 'vzctl destroy' for this VE ID '''will remove everything under /vz/private/ directory'''.<br />
<br />
== Adding a veth device to a VE ==<br />
<br />
Not totally sure what this is, but a customer asked for it and here's what we did (as instructed by vz support):<br />
<br />
<pre>v set 99 --netif_add eth99 --save<br />
ipdel 99 69.55.230.58<br />
v set 99 --ifname eth99 --ipadd 69.55.230.58 --save<br />
v set 99 --ifname eth99 --gateway 69.55.230.1 --save<br />
<br />
vznetcfg net new veth_net<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
net99 active eth0 veth77.77,veth99.99<br />
veth_net active<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
br0 bridge veth_net<br />
br99 bridge net99<br />
veth99.99 veth net99<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic net99 69.55.227.70/24<br />
<br />
vznetcfg br attach br0 eth0<br />
<br />
(will remove 99 from orig net and move to veth_net)<br />
vznetcfg net addif veth_net veth99.99<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
net99 active<br />
veth_net active eth0 veth77.77,veth99.99<br />
<br />
(delete the old crap)<br />
vznetcfg net del net99<br />
<br />
<br />
Then, to add another device in<br />
<br />
v set 77 --netif_add eth77 --save<br />
ipdel 77 69.55.230.78<br />
v set 77 --ifname eth77 --ipadd 69.55.230.78 --save<br />
v set 77 --ifname eth77 --gateway 69.55.230.1 --save<br />
v set 77 --save --ifname eth77 --network veth_net<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
veth77.77 veth<br />
br0 bridge veth_net<br />
veth99.99 veth veth_net<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic veth_net 69.55.227.70/24<br />
<br />
vznetcfg net addif veth_net veth77.77<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
veth77.77 veth veth_net<br />
br0 bridge veth_net<br />
veth99.99 veth veth_net<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic veth_net 69.55.227.70/24<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
veth_net active eth0 veth77.77,veth99.99<br />
<br />
another example<br />
<br />
v set 1182 --netif_add eth1182 --save<br />
ipdel 1182 69.55.236.217<br />
v set 1182 --ifname eth1182 --ipadd 69.55.236.217 --save<br />
v set 1182 --ifname eth1182 --gateway 69.55.236.1 --save<br />
vznetcfg net addif veth_net veth1182.1182<br />
v set 1182 --save --ifname eth1182 --network veth_net<br />
<br />
<br />
unused/not working commands:<br />
ifconfig veth99.0 0<br />
vznetcfg net list<br />
vznetcfg br new br99 net99<br />
vznetcfg br attach br99 eth0<br />
vznetcfg br show<br />
vznetcfg if list<br />
vznetcfg net addif net99 veth99.99<br />
vznetcfg net addif eth0 net99<br />
<br />
---------<br />
<br />
vznetcfg br new br1182 net1182<br />
<br />
vznetcfg br attach br99 eth0<br />
vznetcfg if list<br />
vznetcfg net addif net99 veth99.99<br />
vznetcfg net addif eth0 net99<br />
<br />
<br />
<br />
vznetcfg net addif eth0 net1182<br />
<br />
<br />
# vznetcfg net addif veth99.0 net99<br />
--- 8< ---<br />
<br />
vznetcfg net new net<br />
# vznetcfg net addif eth0 net99<br />
<br />
<br />
# vzctl set 99 --save --netif_add eth0 (at this stage veth99.0 interface have to appear<br />
on node)<br />
# vzctl set 99 --save --ifname eth0 --ipadd 69.55.230.58 (and probably few more arguments<br />
here - see 'man vzctl')<br />
# vznetcfg net addif veth99.0 net99</pre><br />
<br />
<br />
== Assigning/remove ip from a VE ==<br />
<br />
1. Add or remove ips:<br />
ipdel 1234 1.1.1.1 2.2.2.2<br />
ipadd 1234 1.1.1.1 2.2.2.2<br />
<br />
2. update Mgmt screens<br />
<br />
3. offer to update any DNS we do for them<br />
<br />
4. check to see if we had rules for old IP in firwall<br />
<br />
== Enabling tun device for a ve ==<br />
Note, there’s a command for this: [[#addtun|addtun]]<br />
<br />
For posterity:<br />
Make sure the tun.o module is already loaded before Virtuozzo is started: <br />
lsmod <br />
Allow the VPS to use the TUN/TAP device: <br />
vzctl set 101 --devices c:10:200:rw --save <br />
Create the corresponding device inside the VPS and set the proper permissions: <br />
vzctl exec 101 mkdir -p /dev/net <br />
vzctl exec 101 mknod /dev/net/tun c 10 200 <br />
vzctl exec 101 chmod 600 /dev/net/tun<br />
<br />
== Remaking a system (on same virt) ==<br />
<br />
1. [[#cancelve|cancelve]] (or v destroy x - ONLY if you're POSITIVE no data needs to be saved)<br />
<br />
2. [[#vemake|vemake]] using same veid<br />
<br />
3. [[#mvbackups|mvbackups]] or [[#vb|vb]] (if new mount point)<br />
<br />
4. update mgmt with new dir/ip <br />
<br />
5. update firewall if changed ip<br />
<br />
== Re-initialize quota for a VE ==<br />
<br />
There’s a commamd for this now: [[#clearquota|clearquota]]<br />
<br />
For posterity:<br />
<br />
vzctl stop 1<br />
vzquota drop 1<br />
vzctl start 1<br />
<br />
<br />
== Traffic accounting on linux ==<br />
<br />
DEPRECATED - all tracking is done via bwdb now. This is how we used to track traffic.<br />
<br />
TODO: update for diff versions of vz<br />
<br />
Unlike FreeBSD, where we have to add firewall count rules to the system to count the traffic, on virtuozzo counts the traffic for us. You an see the current traffic stats by running `vznetstat`:<br />
<br />
<pre># vznetstat<br />
VEID Net.Class Output(bytes) Input(bytes)<br />
-----------------------------------------------<br />
24218 1 484M 39M<br />
24245 1 463M 143M<br />
2451 1 2224M 265M<br />
2454 1 2616M 385M<br />
4149 1 125M 68M<br />
418 1 1560M 34M<br />
472 1 1219M 315M<br />
726 1 628M 317M<br />
763 1 223M 82M<br />
771 1 4234M 437M<br />
-----------------------------------------------<br />
Total: 13780M 2090M</pre><br />
<br />
As you can see the VEID is on a line with the in and out bytes. So, we simply run a cron job:<br />
<br />
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /root/vztrafdump.sh<br />
<br />
Just like we do on FreeBSD - this one goes through all the VEs in /vz/private and greps the line from vznetstat that matches them and dumps it in /jc_traffic_dump on their system. Then it does it again for all the VEs in /vz1/private. It is important to note that vznetstat runs only once, and the grepping is done from a temporary file that contains that output - we do this because running vznetstat once for each VE that we read out of /vz/private and /vz1/private would take way too long and be too intensive.<br />
<br />
You do not need to do anything to facilitate this other than make sure that that cron job is running - the vznetstat counters are always running, and any new VEs that are added to the system will be accounted for automatically.<br />
<br />
Traffic resetting no longer works with vz 2.6, so we disable the vztrafdump.sh on those virts.<br />
<br />
== Watchdog script ==<br />
<br />
On some of the older virts, we have a watchdog running that kills procs that are deemed bad per the following:<br />
<br />
/root/watchdog from quar1<br />
<br />
<pre>if echo $line | awk '{print $(NF-3)}' | grep [5-9]...<br />
then<br />
# 50-90%<br />
if echo $line | awk '{print $(NF-1)}' | grep "...:.."<br />
then<br />
# running for > 99min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
if echo $line | awk '{print $(NF-1)}' | grep "....m"<br />
then<br />
# running for > 1000min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
fi<br />
<br />
if echo $line | awk '{print $(NF-3)}' | grep [1-9]...<br />
then<br />
# running for 10-90 percent<br />
if echo $line | awk '{print $NF}' | egrep 'cfusion|counter|vchkpw'<br />
then<br />
<br />
if echo $line | awk '{print $(NF-1)}' | grep "[2-9]:.."<br />
then<br />
# between 2-9min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
elif echo $line | awk '{print $(NF-1)}' | grep "[0-9][0-9]:.."<br />
then<br />
# up to 99min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
fi<br />
fi</pre><br />
<br />
<br />
== Misc Linux Items ==<br />
<br />
We are overselling hard drive space ... when you configure a linux system with a certain amount of disk space (the default is 4gigs) you do not actually use up 4gigs of space on the system. The diskspace setting for a user is simply a cap, and they only use up as much space on the actual disk drive as they are actually using.<br />
<br />
When you create a new linux system, even though there are some 300 RPMs or so installed, if you run `df -k` you will see that the entire 4gig partition is empty - no space is being used. This is because the files in their system are "magic symlinks" to the template for their OS that is in /vz/template - however, any changes to any of those files will "disconnect" them and they will immediately begin using space in their system. Further, any new files uploaded (even if those new files overwrite existing files) will take up space on the partition.<br />
<br />
<br />
if you see this:<br />
<br />
<pre>[root@virt8 root]# vzctl stop 160 ; vzctl start 160<br />
VE is not running<br />
Starting VE ...<br />
VE is unmounted<br />
VE is mounted<br />
Adding IP address(es): 69.55.226.83 69.55.226.84<br />
bash ERROR: Can't change file /etc/sysconfig/network<br />
Deleting IP address(es): 69.55.226.83 69.55.226.84<br />
VE is unmounted<br />
[root@virt8 root]#</pre><br />
<br />
it probably means they no longer have /bin/bash - copy one in for them<br />
<br />
ALSO: another possibility is that they have removed the `ed` RPM from their system - it needs to be reinstalled into their system. But since their system is down, this is tricky ...<br />
<br />
VE startup scripts used by 'vzctl' want package 'ed' to be available inside VE. So if package 'ed' will be enabled in OS template config and OS template itself VE #827 is based on - this error should be fixed.<br />
<br />
yes, it is possible to add RPM to VE while it not running.<br />
Try to do following:<br />
<br />
<pre># cd /vz/template/<OS_template_with_ed_package>/<br />
# vzctl mount 827<br />
# rpm -Uvh --root /vz/root/827 --veid 827 ed-0.2-25.i386.vz.rpm</pre><br />
<br />
Usually theres an error, but its ok<br />
<br />
Note: replace 'ed-0.2-25.i386.vz.rpm' in last command with actual<br />
version of 'ed' package you have.<br />
<br />
----<br />
<br />
So how do I know what template the user has ? cat their conf file and it is listed in there. For example, if the conf file has:<br />
<br />
<pre>[root@virt12 sbin]# vc 1103<br />
…snip…<br />
OSTEMPLATE="debian-3.0/20030822"<br />
TEMPLATES="mod_perl-deb30/20030707 mod_ssl-deb30/20030703 mysql-deb30/20030707 proftpd-deb30/20030703 webmin-deb30/20030823 "<br />
…</pre><br />
<br />
then they are on debian 3.0, all of their system RPMs are in /vz/template/debian-3.0, and they are using version 20030822 of that debian 3.0 template. Also, they’ve also got additional packages installed (mod_perl, mod_ssl, etc). Those are also found under /vz/template<br />
<br />
----<br />
<br />
Edits needed to run java:<br />
<br />
When we first created the VEs, the default setting for privvmpages was 93000:94000 ... which was high enough that most people never had problems ... however, you can;t run java or jdk or tomcat or anything java related with that setting. We have found that by setting privvmpages to 610000:615000 that java runs just fine. That is now the default setting. It is exceedingly rare that anyone needs it higher than that, although we have seen it once or twice.<br />
<br />
Any problems with java at all - the first thing you need to do is see if the failcnt has raised for privvmpages.<br />
<br />
<pre># vzctl start 160<br />
Starting VE ...<br />
vzquota : (error) Quota on syscall for 160: Device or resource busy<br />
Running vzquota on failed for VE 160 [3]</pre><br />
<br />
This is because my pwd is _in_ their private directory - you can't start it until you move out<br />
<br />
People seem to have trouble with php if they are clueless newbies. Here are two common problems/solutions:<br />
<br />
no... but i figured it out myself. problem was the php.ini file that came<br />
vanilla with the account was not configured to work with apache (the<br />
ENGINE directive was set to off).<br />
<br />
everything else seems fine now.<br />
<br />
----<br />
<br />
the problem was in the php.ini file. I noticed that is wasnt showing<br />
the code when it was in an html file so I looked at the php.ini file<br />
and had to change it so it recognized <? tags aswell as <?php tags.<br />
<br />
Also, make sure added to httpd.conf<br />
AddType application/x-httpd-php .php<br />
<br />
----<br />
<br />
You can set the time zone:<br />
<br />
You can change the timezone by doing this:<br />
<br />
ln -sf /usr/share/zoneinfo/<zone> /etc/localtime<br />
<br />
where <zone> is the zone you want in the /usr/share/zoneinfo/ directory.<br />
<br />
----<br />
<br />
Failing shm_open calls:<br />
<br />
first, please check if /dev/shm is mounted inside VE.<br />
'cat /proc/mounts' command should show something like this:<br />
tmpfs /dev/shm tmpfs rw 0 0<br />
<br />
If /dev/shm is not mounted you have 2 ways to solve issue:<br />
1. execute following command inside VE (doesn't require VE reboot):<br />
mount -t tmpfs none /dev/shm<br />
2. add following string to /etc/fstab inside VE and reboot it:<br />
tmpfs /dev/shm tmpfs defaults 0 0<br />
<br />
----<br />
<br />
You can have a mounted but not running ve<br />
Just:<br />
vzctl mount <veid><br />
<br />
----<br />
<br />
When a debian sys can’t get on the network, and you try:<br />
<br />
<pre>vzctl set 1046 --ipadd 69.55.227.117<br />
Adding IP address(es): 69.55.227.117<br />
Failed to bring up lo.<br />
Failed to bring up venet0.</pre><br />
<br />
They probably removed iproute package, which must be the one from swsoft. To restore:<br />
<pre># dpkg -i --veid=1046 --admindir=/vz1/private/1046/root/var/lib/dpkg --instdir=/vz1/private/1046/root/ /vz/template/debian-3.0/iproute_20010824-8_i386.vz.deb<br />
(Reading database ... 16007 files and directories currently installed.)<br />
Preparing to replace iproute 20010824-8 (using .../iproute_20010824-8_i386.vz.deb) ...<br />
Unpacking replacement iproute ...<br />
Setting up iproute (20010824-8) ...</pre><br />
<br />
Then restart their ve<br />
<br />
----<br />
<br />
in a ve i do:<br />
<br />
<pre>cd /<br />
du -h .</pre><br />
<br />
and get: 483M .<br />
<br />
i do:<br />
<br />
<pre>bash-2.05a# df -h<br />
Filesystem Size Used Avail Use% Mounted on<br />
/dev/vzfs 4.0G 2.3G 1.7G 56% /</pre><br />
<br />
how can this be?<br />
<br />
Is it possible that quota file was corrupted somehow? Please try to: <br />
<pre>vzctl stop <VEID><br />
vzquota drop <VEID><br />
vzquota init <VEID><br />
vzctl start <VEID></pre><br />
<br />
----<br />
<br />
How to stop vz from starting after reboot:<br />
<br />
VIRTUOZZO=no <br />
in <br />
/etc/sysconfig/vz<br />
<br />
To start: <br />
service vz start<br />
(after setting VIRTUOZZO=yes in /etc/sysconfig/vz)<br />
<br />
service vz restart will do some kind of 'soft reboot' -- restart all<br />
VPSes and reload modules without rebooting the node<br />
<br />
if you need to shut down all VPSes really really fast, run killall -9 init<br />
<br />
----<br />
<br />
Postfix tip:<br />
<br />
You may want to tweak settings: default_process_limit=10<br />
<br />
= Virtuozzo VPS Management Tools =<br />
<br />
== vm ==<br />
<br />
TODO<br />
<br />
== cancelve ==<br />
cancelve <veid><br />
this will:<br />
* stop a ve<br />
* check for backups (offer to remove them from the backup server <br />
and the backup.config)<br />
* rename the private dir<br />
* check for PTR, provide the commands to reset to default<br />
* and rename the ve’s config<br />
* remind you to remove firewall rules<br />
* remind you to remove DNS entries<br />
<br />
== ipadd ==<br />
ipadd <veid> <ip> [ip] [ip]<br />
add’s ip(s) to a ve<br />
<br />
== ipdel ==<br />
ipdel <veid> <ip> [ip] [ip]<br />
removes ip(s) from a ve<br />
<br />
== vc ==<br />
vc <veid><br />
the equivalent of: <tt>cat /vzconf/<veid>.conf</tt><br />
<br />
== vl ==<br />
vl<br />
will displays a list of ve #’s, 1 per line. (ostensibly to use in a for loop)<br />
<br />
== vp ==<br />
vp <veid><br />
the equivalent of: <tt>vzps auxww –E <veid></tt><br />
<br />
== vpe ==<br />
DEPRECATED<br />
vpe <veid> <br />
this will allow you to do a vp when a ve is running out of control, the equivalent of (deprecated since vp operates outside the VPS): <br />
<pre>vzctl set <veid> --kmemsize 2100000:2200000<br />
vzctl exec <veid> ps auxw<br />
vzctl set <veid> --kmemsize (ve’s orig lvalue):(ve’s orig hvalue)</pre><br />
<br />
== vt ==<br />
vt <veid><br />
the equivalent of: <tt>vztop –E <veid></tt><br />
<br />
== vr ==<br />
vr <veid><br />
the equivalent of: <tt>vzctl stop <veid>; vzctl start <veid></tt><br />
You can run this even if the ve is down - the stop command will just fail<br />
<br />
== vs ==<br />
vs [veid]<br />
displays status (output of <tt>vzctl status <veid></tt>) on each ve configured on the system (as determined by <tt>/vzconf/*.conf</tt>)<br />
If passed an argument, gives the status for just that ve. <br />
A running system looks like:<br />
<tt>VEID 16066 exist mounted running</tt><br />
<br />
A ve which is not running (but does exist) looks like:<br />
<tt>VEID 9990 exist unmounted down</tt><br />
<br />
A ve which is not running and doesn’t exist looks like:<br />
<tt>VEID 421 deleted unmounted down</tt><br />
<br />
== vs2 ==<br />
vs2 [veid]<br />
this is similar to vs in that it displays status (output of <tt>vzctl status <veid></tt>) on each ve,<br />
but the difference is it’s list comes from doing an ls on the data dirs. This was meant to catch <br />
the rare case where a ve configured but exists. <br />
<br />
== vw ==<br />
vw [veid]<br />
displays the output of ‘<tt>w</tt>’ (the equivalent of <tt>vzctl exec <veid> w</tt>) for each configured ve (as determined by <tt>/vzconf/*.conf</tt>). Useful for determing which ve is contributing to a heavily-loaded system.<br />
If passed an argument, gives ‘<tt>w</tt>‘ output for just that ve. <br />
Ex:<br />
<pre>[root@virt2 etc]# vw<br />
134<br />
10:52pm up 79 days, 6:14, 0 users, load average: 0.02, 0.02, 0.00<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT<br />
16027<br />
2:52pm up 7 days, 19:54, 0 users, load average: 0.00, 0.00, 0.00<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT<br />
16055<br />
2:52pm up 79 days, 6:38, 0 users, load average: 0.00, 0.04, 0.07<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT</pre><br />
<br />
== vwe ==<br />
vwe [constraint]<br />
just like <tt>vw</tt>, but takes a constraint as an argument, only show’s ve’s with loads >= the constraint provided. If no constraint is provided, 1 is used by default<br />
<br />
== vzs ==<br />
vzs [veid]<br />
displays the beancounter status for all ve’s, or a particular ve if an argument is passed<br />
<br />
== ve ==<br />
ve <veid><br />
the equivalent of: <tt>vzctl enter <veid></tt><br />
<br />
== vx ==<br />
vx <veid> <command><br />
the equivalent of: <tt>/usr/sbin/vzctl exec <veid> <command></tt><br />
<br />
== dprocs ==<br />
dprocs [count]<br />
a script which outputs a continuous report (or a certain number of reports if an option is passed) of processes stuck in the D state and which VPS’s those procs belong to.<br />
<br />
== setmem ==<br />
setmem <VEID> <256|512|768|1024|2048><br />
adjusts the memory resources for the VE<br />
<br />
== afacheck.sh ==<br />
afacheck.sh<br />
displays the health/status of containers and mirrors on an adaptec card (currently quar1, tempvirt1-2, virt9, virt10)- all other are LSI<br />
<br />
== backup ==<br />
backup<br />
backup script called nightly to update virt scripts and do customer backups<br />
<br />
== checkload.pl ==<br />
checkload.pl<br />
this was intended to be setup as a cronjob to watch processes on a virt when the load <br />
rises above a certain level. Not currently in use.<br />
<br />
== findbackuppigs.pl ==<br />
findbackuppigs.pl<br />
looks for files larger than 50MB which customers have asked us to backup. Emails matches<br />
to linux@johncompanies.com<br />
<br />
== gatherlinux.pl ==<br />
gatherlinux.pl<br />
gathers up data about ve’s configured and writes to a file. Used for audits against the db<br />
<br />
== gb ==<br />
gb <search><br />
greps backup.config for the given search parameter<br />
<br />
== gbg ==<br />
gbg <search><br />
greps backup.config for the given search parameter and presents just the directories (for clean pasting)<br />
<br />
== linuxtrafficgather.pl ==<br />
linuxtrafficgather.pl [yy-mm]<br />
sends a traffic usage summary by ve to support@johncomapnies.com and payments@johncompanies.com.<br />
Optional arguments are year and month (must be in the past). If not passed, assumes last month. Relies on <br />
traffic logs created by netstatreset and netstatbackup<br />
<br />
== linuxtrafficwatch.pl ==<br />
linuxtrafficwatch.pl<br />
checks traffic usage and emails support@johncomapnies.com when a ve reaches the warning level (35G) and<br />
the limit (40G). Works on virtuozzo versions <= 2.6.x. We really aren’t using this anymore now that we have netflow.<br />
<br />
== linuxtrafficwatch2.pl ==<br />
linuxtrafficwatch2.pl<br />
checks traffic usage and emails support@johncomapnies.com when a ve reaches the warning level (35G) and<br />
the limit (40G). Works on virtuozzo version 2.6.x. We really aren’t using this anymore now that we have netflow.<br />
<br />
== load.pl ==<br />
load.pl<br />
feeds info to load mrtg - executed by inetd.<br />
<br />
== mb (linux) ==<br />
mb <mount|umount><br />
(nfs) mounts and umounts dirs to backup2. Shortcuts are mbm and mbu to mount and unmount. <br />
<br />
== migrate ==<br />
migrate <ip of node migrating to> <veid> <target_veid> [target dir: vz | vz1 | vz2]<br />
this is basically a wrapper for <tt>vzmigrate</tt> – vzmigrate is a util to seamlessly move a ve from one host to another. This wrapper was written cause vz virtuozzo version 2.6 had a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables. This script mitigates that. Since it makes multiple ssh connections to the target host, it’s a good idea to put the pub key for the src system in the authorized_keys file on the target host. In addition, it emails ve owners when their migration starts and stops (if they place email addresses in a file on their system: /migrate_notify). To move everyone off a system, you’d do:<br />
for f in `vl`; do migrate <ip> $f; done<br />
<br />
== migrateonline ==<br />
migrateonline <ip of node migrating to> <veid> <target_veid> [target dir: vz | vz1 | vz2]<br />
this is the same as migrate but will migrate a ve in <tt>–online</tt> mode which means it won’t be shut down at the end of the migration. This only works when migrating ve’s between 2 machines running a 2.6 kernel (currently tempvirt1-2. virt16-19, virt12). If you get an error that the machine you’re trying to migrate to has a different CPU or features, etc, then you have to edit the file and add the –f switch to the vzmigrate line- you can basically ignore this kind of warning (but never ignore a warning about missing templates on the destination node). NOTE: This edit (if made to migrateonline) will be overwritten by the base script during each night’s backup.<br />
<br />
== netstatbackup ==<br />
DEPRECATED<br />
netstatbackup <br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.5.x. <br />
<br />
== netstatbackup2 ==<br />
DEPRECATED<br />
netstatbackup2<br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.6.x. <br />
<br />
== netstatreset ==<br />
DEPRECATED<br />
netstatreset<br />
writes traffic count data to a logfile and resets counters to 0. Works on virtuozzo versions 2.5.x <br />
<br />
== netstatreset2 ==<br />
DEPRECATED<br />
netstatreset2<br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.6.x. <br />
<br />
== orphanedbackupwatchlinux ==<br />
orphanedbackupwatchlinux <br />
looks for directories on backup2 which aren’t configured in backup.config and offers to <br />
delete them<br />
<br />
== rsync.backup (linux) ==<br />
rsync.backup<br />
does customer backups (relies on backup.config)<br />
<br />
== startvirt.pl ==<br />
startvirt.pl<br />
forks off start ve commands – keeps 6 running at a time. This is not to be used on systems where fastboot is enabled as it circumvents the benefit of the fastboot. The script will occasionally not exit gracefully and will continue to use up CPU, so it should be watched. Also, don’t exit from the script till you’re sure all ve’s are started – if you do you need to start them manually and may have to free up locks. On some systems, the startvirt script doesn’t exit cleanly and you have to ^C out of it. Be careful though- doing so can leave some VE’s in an odd bootup state and you may need to ‘vr’ them manually. You should check to see which ve’s aren’t running and/or confirm all have started when ^C’ing out of startvirt.<br />
<br />
== taskdone (linux) ==<br />
taskdone<br />
when called will email support@johncompanies.com with the hostname of the machine from which it was <br />
executed as the subject<br />
<br />
== vb (linux) ==<br />
vb<br />
the equivalent of: <tt>vi /usr/local/sbin/backup.config</tt><br />
<br />
== vemakeXX ==<br />
DEPRECATED<br />
vemakerh9 <br />
ve create script for RH9 (see vemake)<br />
<br />
vemakedebian30 <br />
ve create script for debian 3.0 (Woody) (see vemake)<br />
<br />
vemakedebian31 <br />
ve create script for debian 3.1 (Sarge) (see vemake)<br />
<br />
vemakedebian40 <br />
ve create script for debian 4.0 (Etch) (see vemake)<br />
<br />
vemakefedora, vemakefedora2, vemakefedora4, vemakefedora5, vemakefedora6, vemakefedora7<br />
ve create script for fedora core 1, 2, 4, 5, 6, 7 respectively (see vemake)<br />
<br />
vemakecentos3, vemakecentos4<br />
ve create script for centos 3, 4 respectively (see vemake)<br />
<br />
vemakesuse, vemakesuse93, vemakesuse100<br />
ve create script for suse 9.2, 9.3, 10.0 respectively (see vemake)<br />
<br />
vemakeubuntu5, vemakeubuntu606, vemakeubuntu606 vemakeubuntu704<br />
ve create script for ubuntu 5.10, 6.06, 6.10, 7.04 respectively (see vemake)<br />
<br />
== vemove ==<br />
DEPRECATED<br />
vemove <veid> <target_ip> </vz/private/123><br />
this script simplifies the old way of moving ve’s from one system to another - in short moving a ve to or from a virt running virtuozzo < 2.6.x<br />
It’s the equivalent of:<br />
<tt>tar cfpP - <veid> --ignore-failed-read | (ssh -2 -c arcfour <target_ip> "split - -b 1024m </vz/private/123>.tar" )</tt>This should only be used if migrate/vzmigrate can’t be used. <br />
<br />
== vim.watchdog ==<br />
vim.watchdog <br />
looks for and kills procs matching “vi|vim|nano|pine|elm” that are running for a long time & consuming lots of cpu. Works on virtuozzo versions 2.5.x<br />
<br />
== vim.watchdog2 ==<br />
vim.watchdog2<br />
looks for and kills procs matching “vi|vim|nano|pine|elm” that are running for a long time & consuming lots of cpu.<br />
Works on virtuozzo versions 2.6.x.<br />
<br />
== vzmigrate ==<br />
vzmigrate <target_ip> -r no <veid>:[dst veid]:[dst /vzX/private/veid]:[dst /vzX/root/veid]<br />
(this is the raw command “wrapped” by migrate/migrateonline) this will seamlessly move a ve from one host to another. The ve will run for the duration of the migration till the very end when it’s shut down, ip moved and started up on the target system. The filesystem on the src will remain. This should be watched – occasionally the move will timeout and leave the system shut down. If target private and root aren’t specified it just puts it in /vz. Only works when both systems are running virtuozzo 2.6.x<br />
<br />
== vztrafdump.sh ==<br />
DEPRECATED<br />
vztrafdump.sh<br />
writes traffic usage info by ve to a file called jc_traffic_dump in each ve’s / dir. Works on virtuozzo versions <= 2.5.x. <br />
<br />
== vztrafdump2.sh ==<br />
DEPRECATED<br />
vztrafdump2.sh<br />
writes traffic usage info by ve to a file called jc_traffic_dump in each ve’s / dir. Works on virtuozzo versions 2.6.x. <br />
<br />
== addtun ==<br />
addtun <veid><br />
Add’s tun device to ve.<br />
<br />
== bwcap ==<br />
bwcap <veid> <kbps><br />
Ex: <tt>bwcap 1234 512</tt><br />
Caps a VE’s bandwidth to the amount given<br />
<br />
== setdisk ==<br />
setdisk <veid> <diskspace in GB><br />
Ex: <tt>setdisk 1234 5</tt><br />
Gives a VE’s a given amount of disk space<br />
<br />
== vdf ==<br />
vdf <veid> <br />
the equivalent of: <tt>vzctl exec <veid> df –h</tt><br />
<br />
== vdff ==<br />
vdff<br />
runs a (condensed) vdf for all ve’s in your pwd (must be run from /vz/privateN)<br />
<br />
== mvbackups ==<br />
mvbackups <veid> <target_machine> (virt1) <target_dir> (vz1)<br />
moves backups from one location to another on the backup server, and provides you with option to remove entries from current backup.config, and simple paste command to add the config to backup.config on the target server<br />
<br />
== checkquota ==<br />
checkquota<br />
for all the ve’s in the cwd (run from /vz/private, /vz1/private, etc) reports what vz quota says they’re using and what the actual usage is (as reported by du)<br />
<br />
== clearquota ==<br />
clearquota <veid><br />
Recalculates a ve’s quota, prints out the usage before and after. The equivalent of:<br />
<tt>vdf <veid>; v stop <veid>; vzquota drop <veid>; v start <veid>; vdf <veid></tt><br />
<br />
== dprocs ==<br />
dprocs<br />
Sometimes the server’s have a large number of processes get stuck in the D state- this script shows (every 3 secs) which VE’s have D procs, which procs<br />
are stuck and a running average of the top “offenders”<br />
<br />
== vzstat ==<br />
vstat<br />
sort of like top for VZ. sort VEs by CPU usage by pressing 'o' and then 'c' keys<br />
<br />
== stopvirt ==<br />
stopvirt<br />
will stop VEs as fast as it can, 6 at a time. May not exit when complete so you should watch [[#vzstat|vzstat]] in another window.</div>Supporthttps://wiki.jcihosting.com/index.php?title=VPS_Management&diff=1021VPS Management2013-03-11T23:11:07Z<p>Support: /* Traffic accounting on linux */</p>
<hr />
<div>= Common Problems =<br />
== Login to any machine without a password ==<br />
<br />
This is possible via the use of ssh keys. The process is thus:<br />
<br />
1. place the public key for your user (root@mail) in the /root/.ssh/authorized_keys file on the server you wish to login to<br />
cat /root/.ssh/id_dsa.pub<br />
(paste that into authorized_keys on the target server). If the file doesn't exist, create it.<br />
<br />
2. enable root login (usually only applies to FreeBSD). Edit the /etc/ssh/sshd_config on the target server and change:<br />
<tt>#PermitRootLogin no</tt><br />
to<br />
<tt>PermitRootLogin yes</tt><br />
<br />
3. Restart the sshd on the target machine. First, find the sshd process: <br />
jailps <hostname> | grep sshd <br />
or <br />
vp <VEID> | grep sshd<br />
<br />
Look for the process resembling:<br />
root 17296 0.0 0.0 5280 1036 ? Ss 2011 4:27 /usr/sbin/sshd <br />
(this is the sshd)<br />
<br />
Not:<br />
root 6270 0.5 0.0 6808 2536 ? Ss 14:33 0:00 sshd: root [priv]<br />
(this is an sshd child- someone already ssh'd in as root)<br />
<br />
Restart the sshd: <br />
kill -1 <PID><br />
<br />
Ex:<br />
kill -1 17296<br />
<br />
You may now ssh in.<br />
<br />
Once you're done, IF you enabled root login, you should repeat steps 2 and 3 to disable root logins.<br />
<br />
== Letting someone in who has locked themselves out (killed sshd, lost pwd) ==<br />
<br />
There are two ways people frequently lock themselves out - either they forget a password, or they kill off sshd somehow.<br />
<br />
These are actually both fairly easy to solve. First, let's say someone kills off their sshd, or somehow mangles /etc/ssh/sshd_config such that it no longer lets them in.<br />
<br />
Their email may be very short, or it may have all sorts of details about how you should fix sshd_config to let them in ... just ignore all of this. They can fix their own mangled sshd. Fixing this is very simple. First, edit the /etc/inetd.conf on their system and uncomment the telnet line:<br />
<br />
telnet stream tcp nowait root /usr/libexec/telnetd telnetd<br />
#telnet stream tcp6 nowait root /usr/libexec/telnetd telnetd<br />
<br />
(just leave the tcp6 version of telnet commented)<br />
<br />
Then, use jailps to list the processes on their system, and find their inetd process. Then simply:<br />
<br />
kill -HUP (pid)<br />
<br />
where (pid) is the PID of their inetd process. Now they have telnet running on their system and they can log in and do whatever they need to do.<br />
<br />
The only complications that could occur are:<br />
<br />
a) their firewall config on our firewall has port 23 blocked, in which case you will need to open that - will be covered in a different lesson.<br />
<br />
b) they are not running inetd, so you can't HUP it. If this happens, edit their /etc/rc.conf, add the inetd_enable="YES" line, and then kill<br />
their jail with /tmp/jailkill.pl - then restart their jail with the jail line from their quad/safe file. Easy.<br />
<br />
If they have forgotten a password,<br />
<br />
On 6.x+ you can reset their password with:<br />
jexec <jailID from jls> passwd root<br />
<br />
Note: the default password for 6.x jails is 8ico2987, for 4.x it is p455agfa<br />
<br />
On 4.x, you need to cd to their etc directory<br />
... for instance:<br />
<br />
cd /mnt/data2/198.78.65.136-col00261-DIR/etc<br />
<br />
and run:<br />
<br />
vipw -d .<br />
<br />
Then paste in these two lines (theres a paste with these):<br />
<br />
root:$1$krszPxhk$xkCepSnz3mIikT3vCtJCt0:0:0::0:0:Charlie &:/root:/bin/csh<br />
user:$1$Mx9p5Npk$QdMU6c8YQqp2FW2M3irEh/:1001:1001::0:0:User &:/home/user:/bin/sh<br />
<br />
overwriting the lines they already have for "user" and "root" - then just tell them that both user and root have been reset to the default password of p455agfa.<br />
<br />
For linux, just passwd inside shell or <br />
vzctl set <veid> --userpasswd root:p455agfa –save<br />
<br />
Starting in 2009 we began giving out randomized passwords for FreeBSD and Linux as the default password. That is stored with each system in Mgmt. You should look for and reset the password to that password in the event of a reset and refer the customer to use their original password from their welcome email- this way we don’t have to send the password again via email (in clear text).<br />
<br />
<br />
== sendmail can’t be contacted from ext ip (only locally) ==<br />
<br />
By default redhat puts this line in sendmail.mc:<br />
<br />
DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')<br />
<br />
which makes it only answer on localhost. Comment it out like:<br />
<br />
dnl DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')<br />
<br />
and then rebuild sendmail.cf with:<br />
<br />
m4 /etc/mail/sendmail.mc > /etc/sendmail.cf<br />
<br />
== virt doesn’t properly let go of ve’s ip(s) when moved to another system ==<br />
<br />
On virtuozzo 2.6 systems, it's been observed that when moving ips from one virt to another that sometimes the routing table will not get updated to reflect the removal of the ip addresses.<br />
<br />
A recent example was a customer that was moving to a new ve on a new virt and the ip addresses were traded between the two ve's. After the trade the two systems were not able to talk to each other. When looking at the routing table for the old system all the ip addresses were still in the routing table as being local, like so:<br />
<br />
<pre>netstat -rn | grep 69.55.225.149<br />
69.55.225.149 0.0.0.0 255.255.255.255 UH 40 0 0 venet0</pre><br />
<br />
This was preventing traffic to the other system from being routed properly.<br />
The solution is to manually delete the route:<br />
<br />
route delete 69.55.225.149 gw 0.0.0.0<br />
<br />
Supposedly, this was fixed in 2.6.1<br />
<br />
== sshd on FreeBSD 6.2 segfaults ==<br />
<br />
First try to reinstall ssh<br />
<br />
<pre>cd /usr/src/secure<br />
cd lib/libssh<br />
make depend && make all install<br />
cd ../../usr.sbin/sshd<br />
make depend && make all install<br />
cd ../../usr.bin/ssh<br />
make depend && make all install</pre><br />
<br />
Failing that, find the library that’s messed up:<br />
<br />
<pre>ldd /usr/sbin/sshd<br />
libssh.so.3 => /usr/lib/libssh.so.3 (0x280a3000) <br />
libutil.so.5 => /lib/libutil.so.5 (0x280d8000) <br />
libz.so.3 => /lib/libz.so.3 (0x280e4000) <br />
libwrap.so.4 => /usr/lib/libwrap.so.4 (0x280f5000) <br />
libpam.so.3 => /usr/lib/libpam.so.3 (0x280fc000) <br />
libbsm.so.1 => /usr/lib/libbsm.so.1 (0x28103000) <br />
libgssapi.so.8 => /usr/lib/libgssapi.so.8 (0x28112000) <br />
libkrb5.so.8 => /usr/lib/libkrb5.so.8 (0x28120000) <br />
libasn1.so.8 => /usr/lib/libasn1.so.8 (0x28154000) <br />
libcom_err.so.3 => /usr/lib/libcom_err.so.3 (0x28175000) <br />
libroken.so.8 => /usr/lib/libroken.so.8 (0x28177000) <br />
libcrypto.so.4 => /lib/libcrypto.so.4 (0x28183000) <br />
libcrypt.so.3 => /lib/libcrypt.so.3 (0x28276000) <br />
libc.so.6 => /lib/libc.so.6 (0x2828e000) <br />
libmd.so.3 => /lib/libmd.so.3 (0x28373000)</pre><br />
<br />
md5 them and compare to other jail hosts or jails running on host<br />
<br />
for libcrypto reinstall:<br />
<pre>/usr/src/crypto<br />
make depend && make all install</pre><br />
<br />
= FreeBSD VPS =<br />
<br />
== Starting jails: Quad/Safe Files ==<br />
<br />
FreeBSD customer systems do not start up automatically at boot time. When one of our freebsd machines boots up, it boots up, and does nothing else. To start jails, we put the commands to start each jail into a shell script(s) and run the script(s). Jail startup is something that needs to be actively monitored, which is why we don’t just run the script automatically. More on monitoring later.<br />
<br />
NOTE: >=7.x we have moved to 1 quad file: <tt>quad1</tt>. Startups are not done by running each quad, but rather [[#startalljails|startalljails]] which relies on the contents of <tt>quad1</tt>. The specifics of this are lower in this article. What follows here applies for pre 7.x systems.<br />
<br />
There are eight files in <tt>/usr/local/jail/rc.d</tt>:<br />
<br />
<pre>jail3# ls /usr/local/jail/rc.d/<br />
quad1 quad2 quad3 quad4 safe1 safe2 safe3 safe4<br />
jail3#</pre><br />
<br />
four quad files and four safe files.<br />
<br />
Each file contains an even number of system startup blocks (total number of jails divided by 4)<br />
<br />
The reason for this is, if we make one large script to startup all the systems at boot time, it will take too long - the first system in the script will start up right after system boot, which is great, but the last system may not start for another 20 minutes.<br />
<br />
Since there is no way to parralelize this during the startup procedure, we simply open four terminals (in screen window 9) and run each script, one in each terminal. This way they all run simultaneously, and the very last system in each startup script gets started in 1/4th the time it would if there was one large file<br />
<br />
The files are generally organized so that quad/safe 1&2 have only jails from disk 1, and quad/safe 3&4 have jails from disk 2. This helps ensure that only 2 fscks on any disk are going on at once. Further, they are balanced so that all quad/safe’s finish executing around the same time. We do this by making sure each quad/safe has a similar number of jails and represents a similar number of inodes (see js).<br />
<br />
The other, very important reason we do it this way, and this is the reason there are quad files and safe files, is that in the event of a system crash, every single vn-backed filesystem that was mounted at the time of system crash needs to be fsck'd. However, fsck'ing takes time, so if we shut the system down gracefully, we don't want to fsck.<br />
<br />
Therefore, we have two sets of scripts - the four quad scripts are identical to the four safe scripts except for the fact that the quad scripts contain fsck commands for each filesystem.<br />
<br />
So, if you shut a system down gracefully, start four terminals and run safe1 in window one, and safe2 in window 2, and so on.<br />
<br />
If you crash, start four terminals (or go to screen window 9) and run quad1 in window one, and quad2 in window 2, and so on.<br />
<br />
Here is a snip of (a 4.x version) quad2 from jail17:<br />
<br />
<pre>vnconfig /dev/vn16 /mnt/data2/69.55.228.7-col00820<br />
fsck -y /dev/vn16<br />
mount /dev/vn16c /mnt/data2/69.55.228.7-col00820-DIR<br />
chmod 0666 /mnt/data2/69.55.228.7-col00820-DIR/dev/null<br />
jail /mnt/data2/69.55.228.7-col00820-DIR mail1.phimail.com 69.55.228.7 /bin/sh /etc/rc<br />
<br />
# moved to data2 col00368<br />
#vnconfig /dev/vn28 /mnt/data2/69.55.236.132-col00368<br />
#fsck -y /dev/vn28<br />
#mount /dev/vn28c /mnt/data2/69.55.236.132-col00368-DIR<br />
#chmod 0666 /mnt/data2/69.55.236.132-col00368-DIR/dev/null<br />
#jail /mnt/data2/69.55.236.132-col00368-DIR limehouse.org 69.55.236.132 /bin/sh /etc/rc<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
vnconfig /dev/vn22 /mnt/data2/69.55.228.13-col01063<br />
fsck -y /dev/vn22<br />
mount /dev/vn22c /mnt/data2/69.55.228.13-col01063-DIR<br />
chmod 0666 /mnt/data2/69.55.228.13-col01063-DIR/dev/null<br />
jail /mnt/data2/69.55.228.13-col01063-DIR www.widestream.com.au 69.55.228.13 /bin/sh /etc/rc<br />
<br />
# cancelled col00106<br />
#vnconfig /dev/vn15 /mnt/data2/69.55.238.5-col00106<br />
#fsck -y /dev/vn15<br />
#mount /dev/vn15c /mnt/data2/69.55.238.5-col00106-DIR<br />
#chmod 0666 /mnt/data2/69.55.238.5-col00106-DIR/dev/null<br />
#jail /mnt/data2/69.55.238.5-col00106-DIR mail.azebu.net 69.55.238.5 /bin/sh /etc/rc</pre><br />
<br />
As you can see, two of the systems specified are commented out - presumably those customers cancelled, or were moved to new servers.<br />
<br />
As you can see, the vnconfig line is the simpler command line, not the longer one that was used when it was first configured. As you can see, all that is done is, vnconfig the filesystem, then fsck it, then mount it. The fourth command is the `jail` command used to start the system – but that will be covered later.<br />
<br />
Here is the safe2 file from jail17:<br />
<br />
<pre>vnconfig /dev/vn16 /mnt/data2/69.55.228.7-col00820<br />
mount /dev/vn16c /mnt/data2/69.55.228.7-col00820-DIR<br />
chmod 0666 /mnt/data2/69.55.228.7-col00820-DIR/dev/null<br />
jail /mnt/data2/69.55.228.7-col00820-DIR mail1.phimail.com 69.55.228.7 /bin/sh /etc/rc<br />
<br />
# moved to data2 col00368<br />
#vnconfig /dev/vn28 /mnt/data2/69.55.236.132-col00368<br />
#mount /dev/vn28c /mnt/data2/69.55.236.132-col00368-DIR<br />
#chmod 0666 /mnt/data2/69.55.236.132-col00368-DIR/dev/null<br />
#jail /mnt/data2/69.55.236.132-col00368-DIR limehouse.org 69.55.236.132 /bin/sh /etc/rc<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
vnconfig /dev/vn22 /mnt/data2/69.55.228.13-col01063<br />
mount /dev/vn22c /mnt/data2/69.55.228.13-col01063-DIR<br />
chmod 0666 /mnt/data2/69.55.228.13-col01063-DIR/dev/null<br />
jail /mnt/data2/69.55.228.13-col01063-DIR www.widestream.com.au 69.55.228.13 /bin/sh /etc/rc<br />
<br />
# cancelled col00106<br />
#vnconfig /dev/vn15 /mnt/data2/69.55.238.5-col00106<br />
#mount /dev/vn15c /mnt/data2/69.55.238.5-col00106-DIR<br />
#chmod 0666 /mnt/data2/69.55.238.5-col00106-DIR/dev/null<br />
#jail /mnt/data2/69.55.238.5-col00106-DIR mail.azebu.net 69.55.238.5 /bin/sh /etc/rc</pre><br />
<br />
As you can see, it is exactly the same, but it does not have the fsck lines.<br />
<br />
Take a look at the last entry - note that the file is named:<br />
<br />
/mnt/data2/69.55.238.5-col00106<br />
<br />
and the mount point is named:<br />
<br />
/mnt/data2/69.55.238.5-col00106-DIR<br />
<br />
This is the general format on all the FreeBSD systems. The file is always named:<br />
<br />
IP-custnumber<br />
<br />
and the directory is named:<br />
<br />
IP-custnumber-DIR<br />
<br />
If you run safe when you need a fsck, the mount will fail and jail will fail:<br />
<br />
# mount /dev/vn1c /mnt/data2/jails/65.248.2.131-ns1.kozubik.com-DIR<br />
mount: /dev/vn1c: Operation not permitted<br />
<br />
No reboot needed, just run the quad script<br />
<br />
Starting with 6.x jails, we added block delimiters to the quad/safe files, the block looks like:<br />
<br />
<pre>echo '## begin ##: nuie.solaris.mu'<br />
fsck -y /dev/concat/v30v31a<br />
mount /dev/concat/v30v31a /mnt/data1/69.55.228.218-col01441-DIR<br />
mount_devfs devfs /mnt/data1/69.55.228.218-col01441-DIR/dev<br />
devfs -m /mnt/data1/69.55.228.218-col01441-DIR/dev rule -s 3 applyset<br />
jail /mnt/data1/69.55.228.218-col01441-DIR nuie.solaris.mu 69.55.228.218 /bin/sh /etc/rc<br />
echo '## end ##: nuie.solaris.mu'</pre><br />
<br />
These are more than just informative when running quad/safe’s, the echo lines MUST be present for certain tools to work properly. So it’s important that any updates to the hostname also be updated on the 2 echo lines. For example, if you try to startjail a jail with a hostname which is on the jail line but not the echo lines, the command will return with host not found.<br />
<br />
=== FreeBSD 7.x+ notes ===<br />
<br />
Starting with the release of FreeBSD 7.x, we are doing jail startups in a slightly different way. First, thereis only 1 file: <tt>/usr/local/jail/rc.d/quad1</tt><br />
There are no other quads or corresponding safe files. The reason for this is twofold, 1. We can pass –C to fsck which will tell is to skip the fsck if the fs is clean (no more need for safe files), 2. We have a new startup script which can be launched multiple times, running in parallel to start jails, where quad1 is the master jail file. <br />
Quad1 could still be run as a shell script, but it would take a very long time for it to run completely so it’s not advisable; or you should break it down into smaller chunks (like quad1, quad2, quad3, etc)<br />
<br />
Here is a snip of (a 7.x version) quad1 from jail2:<br />
<br />
<pre>echo '## begin ##: projects.tw.com'<br />
mdconfig -a -t vnode -f /mnt/data1/69.55.230.46-col01213 -u 50<br />
fsck -Cy /dev/md50c<br />
mount /dev/md50c /mnt/data1/69.55.230.46-col01213-DIR<br />
mount -t devfs devfs /mnt/data1/69.55.230.46-col01213-DIR/dev<br />
devfs -m /mnt/data1/69.55.230.46-col01213-DIR/dev rule -s 3 applyset<br />
jail /mnt/data1/69.55.230.46-col01213-DIR projects.tw.com 69.55.230.46 /bin/sh /etc/rc<br />
echo '## end ##: projects.tw.com'</pre><br />
<br />
Cancelled jails are no longer commented out and stored in quad1, rather they’re moved to <tt>/usr/local/jail/rc.d/deprecated</tt> <br />
<br />
To start these jails, start the 4 ssh sessions as you would for a normal crash and then instead of running quad1-4, instead run startalljails in each window. IMPORTANT- before running startalljails you should make sure you ran preboot once as it will clear out all the lockfiles and enable startalljails to work properly.<br />
<br />
== Problems with the quad/safe files ==<br />
<br />
When you run the quad/safe files, there are two problems that can occur - either a particular system will hang during initialization, OR a system will spit out output to the screen, impeding your ability to do anything. Or both.<br />
<br />
First off, when you start a jail, you see output like this:<br />
<br />
<pre>Skipping disk checks ...<br />
adjkerntz[25285]: sysctl(put_wallclock): Operation not permitted<br />
Doing initial network setup:.<br />
ifconfig: ioctl (SIOCDIFADDR): permission denied<br />
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384<br />
Additional routing options: TCP keepalive=YESsysctl:<br />
net.inet.tcp.always_keepalive: Operation not permitted.<br />
Routing daemons:.<br />
Additional daemons: syslogd.<br />
Doing additional network setup:.<br />
Starting final network daemons:.<br />
ELF ldconfig path: /usr/lib /usr/lib/compat /usr/X11R6/lib /usr/local/lib<br />
a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout /usr/X11R6/lib/aout<br />
Starting standard daemons: inetd cron sshd sendmail sendmail-clientmqueue.<br />
Initial rc.i386 initialization:.<br />
Configuring syscons: blanktime.<br />
Additional ABI support:.<br />
Local package initialization:.<br />
Additional TCP options:.</pre><br />
<br />
Now, let's look at this line, near the end:<br />
<br />
Local package initialization:.<br />
<br />
This is where a list of daemons that are set to start at boot time willshow up. You might see something like:<br />
<br />
Local package initialization: mysqld apache sendmail sendmail-clientmqueue<br />
<br />
Or something like this:<br />
<br />
Local package initialization: postgres postfix apache<br />
<br />
The problem is that many systems (about 4-5 per machine) will hang on that line. Basically it will get to some of the way through the total daemons to be started:<br />
<br />
Local package initialization: mysqld apache<br />
<br />
and will just sit there. Forever.<br />
<br />
Fortunately, pressing ctrl-c will break out of it. Not only will it break out of it, but it will also continue on that same line and start the other daemons:<br />
<br />
Local package initialization: mysqld apache ^c sendmail-clientmqueue<br />
<br />
and then continue on to finish the startup, and then move to the next system to be started.<br />
<br />
So what does this mean? It means that if a machine crashes, and you start four screen-windows to run four quads or four safes, you need to periodically cycle between them and see if any systems are stuck at that point, causing their quad/safe file to hang. A good rule of thumb is, if you see a system at that point in the startup, give it another 100 seconds - if it is still at the exact same spot, hit ctrl-c. Its also a good idea to go back into the quad file (just before the first command in the jail startup block) and note that this jail tends to need a control-c or more time as follows:<br />
<pre>echo '### NOTE ### slow sendmail'<br />
echo '### NOTE ###: ^C @ Starting sendmail.'</pre><br />
<br />
'''NEVER''' hit ctrl-c repeatedly if you don't get an immediate response - that will cause the following jail’s startup commands to be aborted.<br />
<br />
A second problem that can occur is that a jail - maybe the first one in that particular quad/safe, maybe the last one, or maybe one in the middle, will start spitting out status or error messages from one of its init scripts. This is not a problem - basically, hit enter a few times and see if you get a prompt - if you do get a prompt, that means that the quad/safe script has already completed. Therefore it is safe to log out (and log out of the user that you su'd from) and then log back in (if necessary).<br />
<br />
The tricky thing is, if a system in the middle starts flooding with messages, and you hit enter a few times and don't get a prompt. Are you not getting a prompt because some subsequent system is hanging at the initialization, as we discussede above ? Or are you not getting a prompt because that quad file is currently running an fsck ? Usually you can tell by scrolling back in screen’s history to see what it was doing before you started getting the messages.<br />
<br />
If you don’t get clues from history, you have to use your judgement - instead of giving it 100 seconds to respond, perhaps give it 2-3 mins ... if you still get no response (no prompt) when you hit enter, hit ctrl-c. However, be aware that you might still be hitting ctrl-c in the middle of an fsck. This means you will get an error like "filesystem still marked dirty" and then the vnconfig for it will fail and so will the jail command, and the next system in the quad file will then start starting up.<br />
<br />
If this happens, just wait until the end of all the quad files have finished, and start that system manually.<br />
<br />
If things really get weird, like a screen flooded with errors, and you can't get a prompt, and ctrl-c does nothing, then you need to just eventually (give it ten mins or so) just kill that window with ctrl-p, then k, and then log in again and manually check which systems are now running and which aren't, and manually start up any that are not.<br />
<br />
Don't EVER risk running a particular quad/safe file a second time.<br />
If the quad/safe script gets executed twice, reboot the machine immediately.<br />
<br />
So, for all the above reasons, anytime a machine crashes and you run all the quads or all the safes, '''always''' check every jail afterwards to make sure it is running - even if you have no hangs or complications at all.<br />
Run this command:<br />
<br />
<tt>[[#jailpsall|jailpsall]]</tt> <br />
<br />
Note: [[#postboot|postboot]] also populates ipfw counts, so it '''should not be run multiple times''', use <tt>jailpsall</tt> for subsequent extensive ps’ing<br />
<br />
And make sure they all show as running. If one does not show as running, check its /etc/rc.conf file first to see if maybe it is using a different hostname first before starting it manually.<br />
<br />
One thing we have implemented to alleviate these startup hangs and noisy jails, is to put jail start blocks that are slow or hangy at the bottom of the safe/quad file. Further, for each bad jail we note in each quad/safe just before the start block something like:<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
<br />
That way we’ll be prepared to ^C when we see that message appear during the quad/safe startup process. If you observe a new, undocumented hang, '''after''' the quad/safe has finished, place a line similar to the above in the quad file, move the jail start block to the end of the file, then run [[#buildsafe|buildsafe]]<br />
<br />
<br />
== Recovering from a crash (FreeBSD) ==<br />
<br />
=== Diagnose whether you have a crash ===<br />
The most important thing is to get the machine and all jails back up as soon as possible. Note the time, you’ll need to create a crash log entry (Mgmt. -> Reference -> CrashLog). The first thing to do is head over to the [[Screen#Screen_Organization|serial console screen]] and see if there’s any kernel error messages output. Try to copy any messages (or just a sample of repeating messages) you see into the notes section of the crash log. If there are no messages, the machine may just be really busy- wait a bit (5-10min) to see if it comes back. If it's still pinging, odds are its very busy. Note, if you see messages about swap space exhausted, the server is obviously out of memory, however it may recover briefly enough for you to get a jtop in to see who's lauched a ton of procs (most likely) and then issue a quick jailkill to get it back under control.<br />
<br />
If it doesn't come back, or the messages indicate a fatal error, you will need to proceed with a power cycle (ctrl+alt+del will not work).<br />
<br />
=== Power cycle the server ===<br />
If this machine is not a Dell 2950 with a [[DRAC/RMM#DRAC|DRAC card]] (i.e. if you can’t ssh into the DRAC card (as root, using the standard root pass) and issue <br />
racadm serveraction hardreset<br />
then you will need someone at the data center power the macine off, wait 30 sec, then turn it back on. Make sure to re-attach via console:<br />
tip jailX<br />
immediately after power down. <br />
<br />
=== (Re)attach to the console ===<br />
Stay on the console the entire time during boot. As the BIOS posts- look out for the RAID card output- does everything look healthy? The output may be scrambled, look for "DEGRADED" or "FAILED". Once the OS starts booting you will be disconnected (dropped back to the shell on the console server) a couple times during the boot up. The reason you want to quickly re-attach is two-fold: 1. If you don’t reattach quickly then you won’t get any console output, 2. you want to be attached before the server ''potentially'' starts (an extensive) fsck. If you attach after the fsck begins, you’ll have seen no indication it started an fsck and the server will appear frozen during startup- no output, no response. <br />
<br />
IMPORTANT NOTE: on some older FreeBSD systems, there will be no output to the video (KVM) console as it boots up. The console output is redirected to the serial port ... so if a jail crashes, and you attach a kvm, the output during the bootup procedure will not be shown on the screen. However, when the bootup is done, you will get a login prompt on the screen and will be able to log in as normal. <tt>/boot/loader.conf</tt> is where serial console redirect output lives, so comment that if you want to catch output on kvm.<br />
On newer systems it sends most output to both locations. <br />
<br />
=== Assess the heath of the server ===<br />
Once the server boots up fully, you should be able to ssh in. Look around- make sure all the mounts are there and reporting the correct size/usage (i.e. /mnt/data1 /mnt/data2 /mnt/data3 - look in /etc/fstab to determine which mount points should be there), check to see if RAID mirrors are healthy. See [[RAID_Cards#Common_CLI_commands_.28megacli.29|megacli]], [[#aaccheck|aaccheck]]<br />
<br />
Before you start the jails, you need to run [[#preboot|preboot]]. This will do some assurance checks to make sure things are prepped to start the jails. Any issues that come out of preboot need to be addressed before starting jails.<br />
<br />
=== Start jails ===<br />
[[#Starting_jails:_Quad.2FSafe_Files|More on starting jails]]<br />
Customer jails (the VPSs) do not start up automatically at boot time. When a FreeBSD machines boots up, it boots up, and does nothing else. To start jails, we put the commands to start each jail into a shell script(s) and run the script(s). Jail startup is something that needs to be actively monitored, which is why we don’t just run the script automatically. <br />
<br />
In order to start jails, we run the quad files: quad1 quad2 quad3 and quad4 (on new systems there is only quad1). If the machine was cleanly rebooted- which wouldn't be the case if this was a crash, you may run the safe files (safe1 safe2 safe3 safe4) in lieu of quads. <br />
<br />
Open up 4 logins to the server (use the windows in [[Screen#Screen_Organization|a9]])<br />
In each of the 4 windows you will:<br />
<br />
If there is a [[#startalljails|startalljails]] script (and only quad1), run that command in each of the 4 windows. It will parse through the quad1 file and start each jail. Follow the instructions [[#Problems_with_the_quad.2Fsafe_files|here]] for monitoring startup. Note that you can be a little more lenient with jails that take awhile to start- startalljails will work around the slow jails and start the rest. As long as there aren't 4 jails which are "hung" during startup, the rest will get started eventually.<br />
-or-<br />
If there is no startalljails script, there will be multiple quad files. In each of the 4 windows, start each of the quads. i.e. start quad1 in window1, quad2 in window2 and so on. DO NOT start any quad twice. It will crash the server. If you accidentally do this, just jailkill all the jails which are in the quad and run the quad again. Follow the instructions here for monitoring quad startup.<br />
<br />
Note the time the last jail boots- this is what you will enter in the crash log.<br />
<br />
Save the crash log.<br />
<br />
=== Check to make sure all jails have started ===<br />
There's a simple script which will make sure all jails have started, and enter the ipfw counter rules: [[#postboot|postboot]] <br />
Run postboot, which will do a jailps on each jail it finds (excluding commented out jails) in the quad file(s). We're looking for 2 things:<br />
# systems spawning out of control or too many procs<br />
# jails which haven't started<br />
On 7.x and newer systems it will print out the problems (which jails haven't started) at the conclusion of postboot. <br />
On older systems you will need to watch closely to see if/when there's a problem, namely:<br />
<br />
[hostname] doesnt exist on this server<br />
<br />
When you get this message, it means one of 2 things:<br />
1. the jail really didn't start:<br />
When a jail doesn't start it usually boils down to a problem in the quad file. Perhaps the path name is wrong (data1 vs data2) or the name of the vn/mdfile is wrong. Once this is corrected, you will need to run the commands from the quad file manually, or you may use <tt>startjail <hostname></tt><br />
<br />
2. the customer has changed their hostname (and not told us) so their jail ''is'' running, just under a different hostname:<br />
On systems with jls, this is easy to rectify. First, get the customer info: <tt>g <hostname></tt><br />
Then look for the customer in jls: <tt>jls | grep <col0XXXX></tt><br />
From there you will see their new hostname- you should update that hostname in the quad file: don't forget to edit it on the <tt>## begin ##</tt> and <tt>## end ##</tt> lines, and in mgmt. <br />
On older systems without jls, this will be harder, you will need to look further to see their hostname- perhaps its in their /etc/rc.conf<br />
<br />
<br />
Once all jails are started, do some spot checks- try to ssh or browse to some customers, just to make sure things are really ok.<br />
<br />
== Adding disk to a 7.x/8.x jail ==<br />
or<br />
== Moving customer to a different drive (md) ==<br />
<br />
NOTE: this doesn’t apply to mx2 which uses gvinum. Use same procedure as 6.x<br />
NOTE: if you unmount before mdconfig, re-mdconfig (attach) then unmount then mdconfig -u again <br />
<br />
<br />
(parts to change/customize are <tt><span style="color:red">red</span></tt>)<br />
<br />
<br />
If someone wants more disk space, there’s a paste for it, send it to them (explains about downtime, etc).<br />
<br />
1. Figure out the space avail from <tt>js</tt>. Ideally, you want to put the customers new space on a different partition (and create the new md on the new partition). <br />
<br />
<br />
2. make a mental note of how much space they're currently using<br />
<br />
<br />
3. <tt>jailkill <hostname></tt> <br />
<br />
<br />
4. Umount it (including their devfs) but leave the md config’d (so if you use stopjail, you will have to re-mdconfig it)<br />
<br />
<br />
5. <tt>g <customerID></tt> to get the info (IP/cust#) needed to feed the new mdfile and mount name, and to see the current md device. <br />
<br />
<br />
6a. When there's enough room to place new system on an alternate, or the same drive:<br />
USE CAUTION not to overwrite (touch, mdconfig) existing md!!<br><br />
<tt>touch /mnt/data<span style="color:red">3/69.55.234.66-col01334</span><br><br />
mdconfig -a -t vnode -s 10g -f /mnt/data3/69.55.234.66-col01334 -u 97<br><br />
newfs /dev/md97</tt><br />
<br />
<br />
Optional- if new space is on a different drive, move the mount point directory AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
<tt>mount /dev/md<span style="color:red">97</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<br />
confirm you are mounted to /dev/md<span style="color:red">97</span> and space is correct:<br><br />
<tt>df .</tt> <br />
<br />
<br />
do the dump and pipe directly to restore:<br><br />
<tt>dump -0a -f - /dev/md<span style="color:red">1</span> | restore -r -f - <br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use <tt>df</tt> to confirm the restored data size matches the original usage figure<br />
<br />
<br />
md-unconfig old system:<br><br />
<tt>mdconfig -d -u <span style="color:red">1</span></tt><br />
<br />
<br />
archive old mdfile. <br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.237.26-col00241</span> /mnt/data<span style="color:red">1/old-col00241-mdfile-noarchive-20091211</span></tt><br />
<br />
<br />
edit quad (vq1) to point to new (/mnt/data<span style="color:red">3</span>) location AND new md number (md<span style="color:red">97</span>)<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
6b. When there's not enough room on an alternate partition, or on the same drive...but there is enough room if you were to remove the existing customer's space:<br />
<br />
<br />
mount backup nfs mounts:<br><br />
<tt>mbm</tt><br> <br />
(run <tt>df</tt> to confirm backup mounts are mounted)<br />
<br />
<br />
dump the customer to backup2 or backup1<br><br />
<tt>dump -0a -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> /dev/md<span style="color:red">1</span></tt><br><br />
(when complete WITHOUT errors, <tt>du</tt> the dump file to confirm it matches size, roughly, with usage)<br />
<br />
<br />
unconfigure and remove old mdfile<br><br />
<tt>mdconfig -d -u <span style="color:red">1</span><br><br />
rm /mnt/data<span style="color:red">1/69.55.237.26-col00241</span></tt><br><br />
(there should now be enough space to recreate your bigger system. If not, run sync a couple times)<br />
<br />
<br />
create the new system (ok to reuse old mdfile and md#):<br><br />
<tt>touch /mnt/data<span style="color:red">1/69.55.234.66-col01334</span><br><br />
mdconfig -a -t vnode -s <span style="color:red">10</span>g -f /mnt/data<span style="color:red">1/69.55.234.66-col01334</span> -u <span style="color:red">1</span><br><br />
newfs /dev/md<span style="color:red">1</span><br><br />
mount /dev/md<span style="color:red">1</span> /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> <br><br />
cd /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<br />
confirm you are mounted to /dev/md<span style="color:red">1</span> and space is correct:<br><br />
<tt>df .</tt> <br />
<br />
<br />
do the restore from the dumpfile on the backup server:<br><br />
<tt>restore -r -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> .<br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
umount nfs:<br><br />
<tt>mbu</tt><br />
<br />
<br />
If md# changed (or mount point), edit quad (<tt>vq1</tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new md number (md<span style="color:red">1</span>)<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
7. update disk (and dir if applicable) in mgmt screen<br />
<br />
<br />
8. update backup list AND move backups, if applicatble<br />
<br />
Ex: <tt>mvbackups <span style="color:red">69.55.237.26-col00241</span> jail<span style="color:red">9</span> data<span style="color:red">3</span></tt><br />
<br />
<br />
9. Optional: archive old mdfile<br />
<br />
<tt>mbm<br><br />
gzip -c old-col01588-mdfile-noarchive-20120329 > /deprecated/old-col01588-mdfile-noarchive-20120329.gz<br><br />
mbu<br><br />
rm old-col01588-mdfile-noarchive-20120329</tt><br />
<br />
== Adding disk to a 6.x jail (gvinum/gconcat) ==<br />
or<br />
== Moving customer to a different drive (gvinum/gconcat) ==<br />
<br />
(parts to change are <span style="color:red">highlighted</span>)<br />
<br />
<br />
If someone wants more disk space, there’s a paste for it, send it to them (explains about downtime, etc).<br />
<br />
<br />
1. Figure out the space avail from [[#js|js]]. Ideally, you want to put the customers new space on a different partition (and create the new md on the new partition). <br />
<br />
<br />
2. make a mental note of how much space they're currently using<br />
<br />
<br />
3. <tt>[[#stopjail|stopjail]] <hostname></tt> <br />
<br />
<br />
4. <tt>[[#g|g]] <customerID></tt> to get the info (IP/cust#) needed to feed the new mount name and existing volume/device. <br />
<br />
<br />
5a. When there's enough room to place new system on an alternate, or the same drive (using only UNUSED - including if it's in use by the system in question - gvinum volumes):<br />
<br />
<br />
Configure the new device:<br><br />
A. for a 2G system (single gvinum volume):<br><br />
<tt>bsdlabel -r -w /dev/gvinum/v<span style="color:red">123</span><br><br />
newfs /dev/gvinum/v<span style="color:red">123</span>a</tt><br><br />
<br />
-or- <br />
<br />
<br />
B. for a >2G system (create a gconcat volume):<br><br />
try to grab a contiguous block of gvinum volumes. gconcat volumes MAY NOT span drives (i.e. you cannot use a gvinum volume from data3 and a volume from data2 in the same gconcat volume). <br />
<br />
<tt>gconcat label <span style="color:red">v82-v84 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84</span><br><br />
bsdlabel -r -w /dev/concat/<span style="color:red">v82-v84</span><br><br />
newfs /dev/concat/<span style="color:red">v82-v84</span>a</tt><br />
<br />
<br />
Other valid gconcat examples:<br><br />
<tt>gconcat label v82-v84v109v112 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84 /dev/gvinum/v109 /dev/gvinum/v112<br><br />
gconcat label v82v83 /dev/gvinum/v82 /dev/gvinum/v83</tt> <br><br />
Note, long names will truncate: v144v145v148-v115 will truncate to v144v145v148-v1 (so you will refer to it as v144v145v148-v1 thereafter)<br />
<br />
<br />
Optional- if new volume is on a different drive, move the mount point directory (get the drive from js output) AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
<br />
confirm you are mounted to the device (<tt>/dev/gvinum/v<span style="color:red">123</span>a</tt> OR <tt>/dev/concat/<span style="color:red">v82-v84</span></tt>) and space is correct:<br><br />
A. <tt>mount /dev/gvinum/v<span style="color:red">123</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br><br />
-or-<br><br />
B. <tt>mount /dev/concat/<span style="color:red">v82-v84</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<tt>cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
df .</tt> <br />
<br />
<br />
do the dump and pipe directly to restore:<br><br />
<tt>dump -0a -f - /dev/gvinum/v<span style="color:red">1</span> | restore -r -f - <br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
edit quad (<tt>vq<span style="color:red">1</span></tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new volume (<tt>/dev/gvinum/v<span style="color:red">123</span>a</tt> or <tt>/dev/concat/<span style="color:red">v82-v84</span>a</tt>) , run <tt>buildsafe</tt><br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
5b. When there's not enough room on an alternate partition, or on the same drive...but there is enough room if you were to remove the existing customer's space (i.e. if you want/need to reuse the existing gvinum volumes and add on more):<br />
<br />
<br />
mount backup nfs mounts:<br><br />
<tt>mbm</tt> <br><br />
(run df to confirm backup mounts are mounted)<br />
<br />
<br />
dump the customer to backup2 or backup1<br><br />
<tt>dump -0a -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> /dev/<span style="color:red">gconcat/v106-v107</span></tt><br><br />
(when complete WITHOUT errors, du the dump file to confirm it matches size, roughly, with usage)<br />
<br />
<br />
unconfigure the old gconcat volume<br><br />
list member gvinum volumes:<br />
<br />
<tt>gconcat list <span style="color:red">v106v107</span></tt><br />
<br />
<br />
Output will resemble:<br><br />
<pre>Geom name: v106v107<br />
State: UP<br />
Status: Total=2, Online=2<br />
Type: AUTOMATIC<br />
ID: 3530663882<br />
Providers:<br />
1. Name: concat/v106v107<br />
Mediasize: 4294966272 (4.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e2<br />
Consumers:<br />
1. Name: gvinum/sd/v106.p0.s0<br />
Mediasize: 2147483648 (2.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e3<br />
Start: 0<br />
End: 2147483136<br />
2. Name: gvinum/sd/v107.p0.s0<br />
Mediasize: 2147483648 (2.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e3<br />
Start: 2147483136<br />
End: 4294966272</pre><br />
<br />
stop volume and clear members<br><br />
<tt>gconcat stop <span style="color:red">v106v107</span><br><br />
gconcat clear <span style="color:red">gvinum/sd/v106.p0.s0 gvinum/sd/v107.p0.s0</span></tt><br />
<br />
<br />
create new device- and its ok to reuse old/former members<br><br />
try to grab a contiguous block of gvinum volumes. gconcat volumes MAY NOT span drives (i.e. you cannot use a gvinum volume from data3 and a volume from data2 in the same gconcat volume). <br><br />
<tt>gconcat label <span style="color:red">v82-v84v106v107 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84 /dev/gvinum/v106 /dev/gvinum/v107</span> <br><br />
bsdlabel -r -w /dev/concat/<span style="color:red">v82-v84v106v107</span><br><br />
newfs /dev/concat/<span style="color:red">v82-v84v106v107</span>a</tt><br />
<br />
<br />
Optional- if new volume is on a different drive, move the mount point directory (get the drive from js output) AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
confirm you are mounted to the device (/dev/concat/<span style="color:red">v82-v84v106v107</span>) and space is correct:<br><br />
<tt>mount /dev/concat/<span style="color:red">v82-v84v106v107</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<tt>cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
df .</tt> <br />
<br />
do the restore from the dumpfile on the backup server:<br><br />
<tt>restore -r -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> .<br><br />
rm restoresymtable</tt><br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
edit quad (<tt>vq<span style="color:red">1</span></tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new volume (<tt>/dev/concat/<span style="color:red">v82-v84v106v107</span>a</tt>), run buildsafe<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
TODO: clean up/clear old gvin/gconcat vol<br />
<br />
<br />
6. update disk (and dir if applicable) in mgmt screen<br />
<br />
<br />
7. update backup list AND move backups, if applicatble<br />
<br />
<br />
Ex: <tt>mvbackups <span style="color:red">69.55.237.26-col00241</span> jail<span style="color:red">9</span> data<span style="color:red">3</span></tt><br />
<br />
<br />
<br />
DEPRECATED - steps to tack on a new gvin to existing gconcat- leads to corrupted fs<br />
bsdlabel -e /dev/concat/v82-v84<br />
<br />
To figure out new size of the c partition, multiply 4194304 by the # of 2G gvinum volumes and subtract the # of 2G volumes:<br />
10G: 4194304 * 5 – 5 = 20971515<br />
8G: 4194304 * 4 – 4 = 16777212<br />
6G: 4194304 * 3 – 3 = 12582909<br />
4G: 4194304 * 2 – 2 = 8388606<br />
<br />
To figure out the new size of the a partition, subtract 16 from the c partition:<br />
10G: 20971515 – 16 = 20971499<br />
8G: 16777212 – 16 = 16777196<br />
6G: 12582909 – 16 = 12582893<br />
4G: 8388606 – 16 = 8388590<br />
<br />
Orig:<br />
8 partitions:<br />
# size offset fstype [fsize bsize bps/cpg]<br />
a: 8388590 16 4.2BSD 2048 16384 28552<br />
c: 8388606 0 unused 0 0 # "raw" part, don't edit<br />
<br />
New:<br />
8 partitions:<br />
# size offset fstype [fsize bsize bps/cpg]<br />
a: 12582893 16 4.2BSD 2048 16384 28552<br />
c: 12582909 0 unused 0 0 # "raw" part, don't edit<br />
<br />
sync; sync<br />
<br />
growfs /dev/concat/v82-v84a<br />
<br />
fsck –fy /dev/concat/v82-v84a<br />
<br />
sync<br />
<br />
fsck –fy /dev/concat/v82-v84a<br />
<br />
(keep running fsck’s till NO errors)<br />
<br />
== Problems un-mounting - and with mount_null’s ==<br />
<br />
If you cannot unmount a filesystem, beacuse it says the filesystem is busy, it is because of three things:<br />
<br />
a) the jail is still running<br />
<br />
b) you are actually in that directory, even though the jail is stopped<br />
<br />
c) there are still dev, null_mount or linprocfs mount points mounted inside that directory.<br />
<br />
d) when trying to umount null_mounts that are really long and you get an error like “No such file or directory”, it’s an OS bug where the dir is truncated. No known fix<br />
<br />
e) there are still files open somewhere inside the dir. Use <tt>fstat | grep <cid></tt> to find the process that has files open<br />
<br />
f) Starting with 6.x, the jail mechanism does a poor job of keeping track of processes running in a jail and if it thinks there are still procs running, it will refuse to umount the disk. If this is happening you should see a low number in the #REF column when you run jls. In this case you ''can'' safely <tt>umount –f</tt> the mount. <br />
<br />
Please note -if you forcibly unmount a (4.x) filesystem that has null_mounts<br />
still mounted in it, the system '''will crash''' within 10-15 mins.<br />
<br />
== Misc jail Items ==<br />
<br />
We are overselling hard drive space on jail2, jail8, jail9, a couple jails on jail17, jail4, jail12 and jail18.<br />
Even though the vn file shows 4G size, it doesn’t actually occupy that amount of space on the disk. So be careful not to fill up drives where we’re overselling – use oversellcheck to confirm you’re not oversold by more than 10G.<br />
There are other truncated jails, they are generally noted in a the file on the root system: /root/truncated<br />
<br />
The act of moving a truncated vn to another system un-does the truncating- the truncated vn is filled with 0’s and it occupies physical disk space for which it’s configured. So, you should use dumpremote to preserve the truncation.<br />
<br />
= FreeBSD VPS Management Tools =<br />
<br />
These files are located in /usr/local/jail/rc.d and /usr/local/jail/bin<br />
<br />
== jailmake ==<br />
<br />
Applies to 7.x+ <br />
On older systems syntax differs, run jailmake once to see.<br />
<br />
Note: this procedure differs on mx2 which is 7.x but still uses gvinum<br />
<br />
# run js to figure out which md’s are in use, which disk has enough space, IP to put it on<br />
# use col00xxx for both hostnames if they don’t give you a hostname<br />
# copy over dir, ip and password to pending customer screen<br />
<br />
<tt>Usage: jailmake IP[,IP] CID disk[1|2|3] md# hostname shorthost ipfw# email [size in GB]</tt><br />
<br />
Ex: <br />
<br />
Jail2# jailmake 69.55.234.66 col01334 3 97 vps.bsd.it vps 1334 fb@bsd.it<br />
<br />
== jailps ==<br />
jailps [hostname]<br />
DEPRECATED FOR jps: displays processes belonging to/running inside a jail. The command<br />
takes one (optional) argument – the hostname of the jail you wish to query. If you don’t <br />
supply an argument, all processes on the machine are listed and grouped by jail. <br />
<br />
== jps ==<br />
jps [hostname]<br />
displays processes belonging to/running inside a jail. The command<br />
takes one (optional) argument – the hostname or ID of the jail you wish to query. <br />
<br />
== jailkill ==<br />
jailkill <hostname><br />
stops all process running in a jail.<br />
<br />
You can also run:<br />
jailkill <JID><br />
<br />
=== problems ===<br />
Occasionally you will hit an issue where jail will not kill off:<br />
<br />
<pre>jail9# jailkill www.domain.com<br />
www.domain.com .. killed: none<br />
jail9#</pre><br />
Because no processes are running under that hostname. You cannot use jailps.pl either:<br />
<br />
<pre>jail9# jailps www.domain.com<br />
www.domain.com doesn’t exist on this server<br />
jail9#</pre><br />
<br />
The reasons for this are usually:<br />
* the jail is no longer running<br />
<br />
* the jail's hostname has changed<br />
In this case, <br />
<br />
>=6.x: run a <tt>jls|grep <jail's IP></tt> to find the correct hostname, then update the quad file, then kill the jail.<br />
<br />
<6.x: the first step is to cat their /etc/rc.conf file to see if you can tell what they set the new hostname to. This very often works. For example:<br />
<br />
cat /mnt/data2/198.78.65.136-col00261-DIR/etc/rc.conf<br />
<br />
But maybe they set the hostname with the hostname command, and the original hostname is still in /etc/rc.conf.<br />
<br />
The welcome email clearly states that they should tell us if they change their hostname, so there is no problem in just emailing them and asking them what they set the new hostname to.<br />
<br />
Once you know the new hostname OR if a customer simply emails to inform you that they have set the hostname to something different, you need to edit the quad and safe files that their system is in to input the new hostname.<br />
<br />
However, if push comes to shove and you cannot find out the hostname from them or from their system, then you need to start doing some detective work.<br />
<br />
The easiest thing to do is run jailps looking for a hostname similar to their original hostname. Or you could get into the /bin/sh shell by running:<br />
<br />
/bin/sh<br />
<br />
and then looking at every hostname of every process:<br />
<br />
for f in `ls /proc` ; do cat /proc/$f/status ; done<br />
<br />
and scanning for a hostname that is either similar to their original hostname, or that you don't see in any of the quad safe files.<br />
<br />
This is very brute force though, and it is possible that catting every file in /proc is dangerous - I don't recommend it. A better thing would be to identify any processes that you know belong to this system – perhaps the reason you are trying to find this system is because they are running something bad - and just catting the status from only that PID.<br />
<br />
Somewhere there’s a jail where there may be 2 systems named www. Look at /etc/rc.conf and make sure they’re both really www. If they are, jailkill www, jailps www to make sure not running. Then immediately restart the other one, as the fqdn (as found from a rev nslookup)<br />
<br />
* on >=6.x the hostname may not yet be hashed:<br />
<pre>jail9 /# jls<br />
JID Hostname Path IP Address(es)<br />
1 bitnet.dgate.org /mnt/data1/69.55.232.50-col02094-DIR 69.55.232.50<br />
2 ns3.hctc.net /mnt/data1/69.55.234.52-col01925-DIR 69.55.234.52<br />
3 bsd1 /mnt/data1/69.55.232.44-col00155-DIR 69.55.232.44<br />
4 let2.bbag.org /mnt/data1/69.55.230.92-col00202-DIR 69.55.230.92<br />
5 post.org /mnt/data2/69.55.232.51-col02095-DIR 69.55.232.51 ...<br />
6 ns2 /mnt/data1/69.55.232.47-col01506-DIR 69.55.232.47 ...<br />
7 arlen.server.net /mnt/data1/69.55.232.52-col01171-DIR 69.55.232.52<br />
8 deskfood.com /mnt/data1/69.55.232.71-col00419-DIR 69.55.232.71<br />
9 mirage.confluentforms.com /mnt/data1/69.55.232.54-col02105-DIR 69.55.232.54 ...<br />
10 beachmember.com /mnt/data1/69.55.232.59-col02107-DIR 69.55.232.59<br />
11 www.agottem.com /mnt/data1/69.55.232.60-col02109-DIR 69.55.232.60<br />
12 sdhobbit.myglance.org /mnt/data1/69.55.236.82-col01708-DIR 69.55.236.82<br />
13 ns1.jnielsen.net /mnt/data1/69.55.234.48-col00204-DIR 69.55.234.48 ...<br />
14 ymt.rollingegg.net /mnt/data2/69.55.236.71-col01678-DIR 69.55.236.71<br />
15 verse.unixlore.net /mnt/data1/69.55.232.58-col02131-DIR 69.55.232.58<br />
16 smcc-mail.org /mnt/data2/69.55.232.68-col02144-DIR 69.55.232.68<br />
17 kasoutsuki.w4jdh.net /mnt/data2/69.55.232.46-col02147-DIR 69.55.232.46<br />
18 dili.thium.net /mnt/data2/69.55.232.80-col01901-DIR 69.55.232.80<br />
20 www.tekmarsis.com /mnt/data2/69.55.232.66-col02155-DIR 69.55.232.66<br />
21 vps.yoxel.net /mnt/data2/69.55.236.67-col01673-DIR 69.55.236.67<br />
22 smitty.twitalertz.com /mnt/data2/69.55.232.84-col02153-DIR 69.55.232.84<br />
23 deliver4.klatha.com /mnt/data2/69.55.232.67-col02160-DIR 69.55.232.67<br />
24 nideffer.com /mnt/data2/69.55.232.65-col00412-DIR 69.55.232.65<br />
25 usa.hanyuan.com /mnt/data2/69.55.232.57-col02163-DIR 69.55.232.57<br />
26 daifuku.ppbh.com /mnt/data2/69.55.236.91-col01720-DIR 69.55.236.91<br />
27 collins.greencape.net /mnt/data2/69.55.232.83-col01294-DIR 69.55.232.83<br />
28 ragebox.com /mnt/data2/69.55.230.104-col01278-DIR 69.55.230.104<br />
29 outside.mt.net /mnt/data2/69.55.232.72-col02166-DIR 69.55.232.72<br />
30 vps.payneful.ca /mnt/data2/69.55.234.98-col01999-DIR 69.55.234.98<br />
31 higgins /mnt/data2/69.55.232.87-col02165-DIR 69.55.232.87 ...<br />
32 ozymandius /mnt/data2/69.55.228.96-col01233-DIR 69.55.228.96<br />
33 trusted.realtors.org /mnt/data2/69.55.238.72-col02170-DIR 69.55.238.72<br />
34 jc1.flanderous.com /mnt/data2/69.55.239.22-col01504-DIR 69.55.239.22<br />
36 guppylog.com /mnt/data2/69.55.238.73-col00036-DIR 69.55.238.73<br />
40 haliohost.com /mnt/data2/69.55.234.41-col01916-DIR 69.55.234.41 ...<br />
41 satyr.jorge.cc /mnt/data1/69.55.232.70-col01963-DIR 69.55.232.70<br />
jail9 /# jailkill satyr.jorge.cc<br />
ERROR: jail_: jail "satyr,jorge,cc" not found<br />
</pre><br />
<br />
Note how it's saying <tt>satyr,jorge,cc</tt> is not found, and not <tt>satyr.jorge.cc</tt>. <br />
<br />
The jail subsystem tracks things using comma-delimited hostnames. That is created every few hours:<br />
<br />
jail9 /# crontab -l<br />
0 0,6,12,18 * * * /usr/local/jail/bin/sync_jail_names<br />
<br />
So if we run this manually:<br />
jail9 /# /usr/local/jail/bin/sync_jail_names<br />
<br />
Then kill the jail:<br />
jail9 /# jailkill satyr.jorge.cc<br />
successfully killed: satyr,jorge,cc</pre><br />
<br />
It worked.<br />
<br />
<br />
If you ever see this when trying to kill a jail:<br />
<br />
<pre># jailkill e-scribe.com<br />
killing JID: 6 hostname: e-scribe.com<br />
3 procs running<br />
3 procs running<br />
3 procs running<br />
3 procs running<br />
...</pre><br />
<br />
<tt>[[#jailkill|jailkill]]</tt> probably got lost trying to kill off the jail. Just ctrl-c the jailkill process, then run a jailps on the hostname, and <tt>kill -9</tt> any process which is still running. Keep running jailps and <tt>kill -9</tt> till all processes are gone.<br />
<br />
== jailpsall ==<br />
jailpsall<br />
will run a jailps on all jails configured in the quad files (this is different from<br />
jailps with no arguments as it won’t help you find a “hidden” system)<br />
<br />
== jailpsw ==<br />
jailpsw<br />
will run a jailps with an extra -w to provide wider output<br />
<br />
== jt (>=7.x) ==<br />
jt<br />
displays the top 20 processes on the server (the top 20 processes from top) and <br />
which jail owns them. This is very helpful for determining who is doing what when<br />
the server is very busy.<br />
<br />
== jtop (>=7.x) ==<br />
jtop<br />
a wrapper for top displaying processes on the server and which jail owns them. Constantly updates, like top. <br />
<br />
== jtop (<7.x) ==<br />
jtop<br />
displays the top 20 processes on the server (the top 20 processes from top) and <br />
which jail owns them. This is very helpful for determining who is doing what when<br />
the server is very busy.<br />
<br />
== stopjail ==<br />
stopjail <hostname> [1]<br />
this will jailkill, umount and vnconfig –u a jail. If passed an optional 2nd<br />
argument, it will not exit before umounting and un-vnconfig’ing in the event<br />
jailkill returns no processes killed. This is useful if you just want to umount<br />
and vnconfig –u a jail you’ve already killed. It is intelligent in that it won’t <br />
try to umount or vnconfig –u if it’s not necessary.<br />
<br />
== startjail ==<br />
startjail <hostname><br />
this will start vnconfig, mount (including linprocfs and null-mounts), and start a jail.<br />
Essentially, it reads the jail’s relevant block from the right quad file and executes it.<br />
It is intelligent in that it won’t try to mount or vnconfig if it’s not necessary.<br />
<br />
== jpid ==<br />
jpid <pid><br />
displays information about a process – including which jail owns it.<br />
It’s the equivalent of running cat /proc/<pid>/status<br />
<br />
== canceljail ==<br />
canceljail <hostname> [1]<br />
this will stop a jail (the equivalent of stopjail), check for backups (offer to remove them <br />
from the backup server and the backup.config), rename the vnfile, remove the dir, and <br />
edit quad/safe. If passed an optional 2nd argument, it will not exit upon failing to kill<br />
and processes owned by the jail. This is useful if you just want to cancel a jail which <br />
is already stopped.<br />
<br />
== jls ==<br />
jls [-v]<br />
Lists all jails running:<br />
<pre>JID #REF IP Address Hostname Path<br />
101 135 69.55.224.148 mail.pc9.org /mnt/data2/69.55.224.148-col01034-DIR</pre><br />
<br />
#REF is the number of references or procs(?) running<br />
<br />
Running with -v will give you all IPs assigned to each jail (7.2 up)<br />
<pre>JID #REF Hostname Path IP Address(es)<br />
101 139 mail.pc9.org /mnt/data2/69.55.224.148-col01034-DIR 69.55.224.14869.55.234.85</pre><br />
<br />
== startalljails ==<br />
startalljails<br />
7.2+ only. This will parse through quad1 and start all jails. It utilizes lockfiles so it won’t try to start a jail more than once- therefore multiple instances can be running in parallel without fear of starting a jail twice. If a jail startup gets stuck, you can ^C without fear of killing the script. IMPORTANT- before running startalljails you should make sure you ran preboot once as it will clear out all the lockfiles and enable startalljails to work properly.<br />
<br />
== aaccheck.sh ==<br />
aaccheck.sh<br />
displayes the output of container list and task list from aaccli<br />
<br />
== backup ==<br />
backup<br />
backup script called nightly to update jail scripts and do customer backups<br />
<br />
== buildsafe ==<br />
buildsafe<br />
creates safe files based on quads (automatically removing the fsck’s). This will destructively overwrite safe files<br />
<br />
== checkload.pl ==<br />
checkload.pl<br />
this was intended to be setup as a cronjob to watch processes on a jail when the load <br />
rises above a certain level. Not currently in use.<br />
<br />
== checkprio.pl ==<br />
checkprio.pl<br />
will look for any process (other than the current shell’s csh, sh, sshd procs) with a non-normal priority and normalize it<br />
<br />
== diskusagemon == <br />
diskusagemon <mount point> <1k blocks><br />
watches a mount point’s disk use, when it reaches the level specified in the 2nd argument,<br />
it exits. This is useful when doing a restore and you want to be paged as it’s nearing completion.<br />
Best used as: <tt>diskusagemon /asd/asd 1234; pagexxx</tt><br />
<br />
== dumprestore ==<br />
dumprestore <dumpfile><br />
this is a perl expect script which automatically enters ‘1’ and ‘y’. It seems to cause restore to fail<br />
to set owner permissions on large restores.<br />
<br />
== g ==<br />
g <search><br />
greps the quad/safe files for the given search parameter<br />
<br />
== gather.pl ==<br />
gather.pl<br />
gathers up data about jails configured and writes to a file. Used for audits against the db<br />
<br />
== gb ==<br />
gb <search><br />
greps backup.config for the given search parameter<br />
<br />
== gbg ==<br />
gbg <search><br />
greps backup.config for the given search parameter and presents just the directories (for clean pasting)<br />
<br />
== ipfwbackup ==<br />
ipfwbackup<br />
writes ipfw traffic count data to a logfile<br />
<br />
== ipfwreset ==<br />
ipfwreset<br />
writes ipfw traffic count data to a logfile and resets counters to 0<br />
<br />
== js ==<br />
js<br />
output varies by OS version, but generally provides information about the base jail:<br />
- which vn’s are in use<br />
- disk usage<br />
- info about the contents of quads<br />
- the # of inodes represented by the jails contained in the group (133.2 in the example below), and how many jails per data mount, as well as subtotals<br />
- ips bound to the base machine but not in use by a jail<br />
- free gvinum volumes, or unused vn’s or used md’s<br />
<br />
<pre>/usr/local/jail/rc.d/quad1:<br />
/mnt/data1 133.2 (1)<br />
/mnt/data2 1040.5 (7)<br />
total 1173.7 (8)<br />
/usr/local/jail/rc.d/quad2:<br />
/mnt/data1 983.4 (6)<br />
total 983.4 (6)<br />
/usr/local/jail/rc.d/quad3:<br />
/mnt/data1 693.4 (4)<br />
/mnt/data2 371.6 (3)<br />
total 1065 (7)<br />
/usr/local/jail/rc.d/quad4:<br />
/mnt/data1 466.6 (3)<br />
/mnt/data2 882.2 (5)<br />
total 1348.8 (8)<br />
/mnt/data1: 2276.6 (14)<br />
/mnt/data2: 2294.3 (15)<br />
<br />
Available IPs:<br />
69.55.230.11 69.55.230.13 69.55.228.200<br />
<br />
Available volumes:<br />
v78 /mnt/data2 2G<br />
v79 /mnt/data2 2G<br />
v80 /mnt/data2 2G</pre> <br />
<br />
== load.pl ==<br />
load.pl<br />
feeds info to load mrtg - executed by inetd.<br />
<br />
== makevirginjail ==<br />
makevirginjail<br />
Only on some systems, makes an empty jail (doesn't do restore step)<br />
<br />
== mb == <br />
mb <mount|umount><br />
(nfs) mounts and umounts dirs to backup2. Shortcuts are mbm and mbu to mount and unmount. <br />
<br />
== notify.sh ==<br />
notify.sh<br />
emails reboot@johncompanies.com – intended to be called at boot time to alert us to a machine which panics and reboots and isn’t caught by bb or castle.<br />
<br />
== orphanedbackupwatch ==<br />
orphanedbackupwatch<br />
looks for directories on backup2 which aren’t configured in backup.config and offers to delete them<br />
<br />
== postboot ==<br />
postboot<br />
to be run after a machine reboot and quad/safe’s are done executing. It will:<br />
* do chmod 666 on each jail’s /dev/null<br />
* add ipfw counts<br />
* run jailpsall (so you can see if a configured jail isn’t running)<br />
<br />
== preboot ==<br />
preboot<br />
to be run before running quad/safe – checks for misconfigurations: <br />
* a jail configured in a quad but not a safe<br />
* a jail is listed more than once in a quad<br />
* the ip assigned to a jail isn’t configured on the machine<br />
* alias numbering skips in the rc.conf (resulting in the above)<br />
* orphaned vnfile's that aren't mentioned in a quad/safe<br />
* ip mismatches between dir/vnfile name and the jail’s ip<br />
* dir/vnfiles's in quad/safe that don’t exist <br />
<br />
== quadanalyze.pl ==<br />
quadanalyze.pl<br />
called by js, produces the info (seen above with js explanation) about the contents of quad (inode count, # of jails, etc.)<br />
<br />
== rsync.backup ==<br />
rsync.backup<br />
does customer backups (relies on backup.config)<br />
<br />
== taskdone ==<br />
taskdone<br />
when called will email support@johncompanies.com with the hostname of the machine from which it was executed as the subject<br />
<br />
== topten ==<br />
topten<br />
summarizes the top 10 traffic users (called by ipfwreset)<br />
<br />
== trafficgather.pl ==<br />
trafficgather.pl [yy-mm]<br />
sends a traffic usage summary by jail to support@johncomapnies.com and payments@johncompanies.com. Optional arguments are year and month (must be in the past). If not passed, assumes last month. Relies on traffic logs created by ipfwreset and ipfwbackup<br />
<br />
== trafficwatch.pl ==<br />
trafficwatch.pl<br />
checks traffic usage and emails support@johncomapnies.com when a jail reaches the warning level (35G) and the limit (40G). We really aren’t using this anymore now that we have netflow.<br />
<br />
== trafstats ==<br />
trafstats<br />
writes ipfw traffic usage info by jail to a file called jc_traffic_dump in each jail’s / dir<br />
<br />
== truncate_jailmake ==<br />
truncate_jailmake<br />
a version of jailmake which creates truncated vnfiles.<br />
<br />
== vb ==<br />
vb<br />
the equivalent of: <tt>vi /usr/local/jail/bin/backup.config</tt><br />
<br />
== vs (freebsd) ==<br />
vs<n><br />
the equivalent of: <tt>vi /usr/local/jail/rc.d/safe<n></tt><br />
<br />
vq<n><br />
the equivalent of: vi /usr/local/jail/rc.d/quad<n><br />
<br />
== dumpremote ==<br />
dumpremote <user@machine> </remote/location/file-dump> <vnX><br />
ex: dumpremote user@10.1.4.117 /mnt/data3/remote.echoditto.com-dump 7<br />
this will dump a vn filesystem to a remote machine and location<br />
<br />
== oversellcheck ==<br />
oversellcheck<br />
displays how much a disk is oversold or undersold taking into account truncated vn files. Only for use on 4.x systems<br />
<br />
== mvbackups (freebsd) ==<br />
mvbackups <dir> (1.1.1.1-col00001-DIR) <target_machine> (jail1) <target_dir> (data1)<br />
moves backups from one location to another on the backup server, and provides you with option to remove entries from current backup.config, and simple paste command to add the config to backup.config on the target server<br />
<br />
== jailnice ==<br />
jailnice <hostname><br />
applies <tt>renice 19 [PID]</tt> and <tt>rtprio 31 –[PID]</tt> to each process in the given jail<br />
<br />
== dumpremoterestore ==<br />
dumpremoterestore <device> <ip of target machine> <dir on target machine><br />
ex: dumpremoterestore /dev/vn51 10.1.4.118 /mnt/data2/69.55.239.45-col00688-DIR<br />
dumps a device and restores it to a directory on a remote machine. Requires that you enable root ssh on the <br />
remote machine.<br />
<br />
== psj ==<br />
psj<br />
shows just the procs running on the base system – a ps auxw but without jail’d procs present<br />
<br />
== perc5iraidchk ==<br />
perc5iraidchk<br />
checks for degraded arrays on Dell 2950 systems with Perc5/6 controllers<br />
<br />
== perc4eraidchk ==<br />
perc4eraidchk<br />
checks for degraded arrays on Dell 2850 systems with Perc4e/Di controllers<br />
<br />
= Virtuozzo VPS =<br />
<br />
Note: We use VEID (Virtual Environment ID) and CTID (Container ID) interchangably. Similarly, VE and CT. They mean the same thing.<br />
VZPP = VirtuoZzo Power Panel (the control panel for each CT)<br />
<br />
All linux systems exist in /vz, /vz1 or /vz2 - since each linux machine holds roughly 60-90 customers, there will be roughly 30-45 in each partition.<br />
<br />
The actual filesystem of the system in question is in:<br />
<br />
/vz(1-2)/private/(VEID)<br />
<br />
Where VEID is the identifier for that system - an all-numeric string larger than 100.<br />
<br />
The actual mounted and running systems are in the corresponding:<br />
<br />
/vz(1-2)/root/(VEID)<br />
<br />
But we rarely interact with any system from this mount point.<br />
<br />
You should never need to touch the root portion of their system – however you can traverse their filesystem by going to <tt>/vz(1-2)/private/(VEID)/root</tt> (<tt>/vz(1-2)/private/(VEID)/fs/root</tt> on 4.x systems) the root of their filesystem is in that directory, and their entire system is underneath that.<br />
<br />
Every VE has a startup script in <tt>/etc/sysconfig/vz-scripts</tt> (which is symlinked as <tt>/vzconf</tt> on all systems) - the VE startup script is simply named <tt>(VEID).conf</tt> - it contains all the system parameters for that VE:<br />
<br />
<pre># Configuration file generated by vzsplit for 60 VE<br />
# on HN with total amount of physical mem 2011 Mb<br />
<br />
VERSION="2"<br />
CLASSID="2"<br />
<br />
ONBOOT="yes"<br />
<br />
KMEMSIZE="8100000:8200000"<br />
LOCKEDPAGES="322:322"<br />
PRIVVMPAGES="610000:615000"<br />
SHMPAGES="33000:34500"<br />
NUMPROC="410:415"<br />
PHYSPAGES="0:2147483647"<br />
VMGUARPAGES="13019:2147483647"<br />
OOMGUARPAGES="13019:2147483647"<br />
NUMTCPSOCK="1210:1215"<br />
NUMFLOCK="107:117"<br />
NUMPTY="19:19"<br />
NUMSIGINFO="274:274"<br />
TCPSNDBUF="1800000:1900000"<br />
TCPRCVBUF="1800000:1900000"<br />
OTHERSOCKBUF="900000:950000"<br />
DGRAMRCVBUF="200000:200000"<br />
NUMOTHERSOCK="650:660"<br />
DCACHE="786432:818029"<br />
NUMFILE="7500:7600"<br />
AVNUMPROC="51:51"<br />
IPTENTRIES="155:155"<br />
DISKSPACE="4194304:4613734"<br />
DISKINODES="400000:420000"<br />
CPUUNITS="1412"<br />
QUOTAUGIDLIMIT="2000"<br />
VE_ROOT="/vz1/root/636"<br />
VE_PRIVATE="/vz1/private/636"<br />
NAMESERVER="69.55.225.225 69.55.230.3"<br />
OSTEMPLATE="vzredhat-7.3/20030305"<br />
VE_TYPE="regular"<br />
IP_ADDRESS="69.55.225.229"<br />
HOSTNAME="textengine.net"</pre><br />
<br />
<br />
As you can see, the hostname is set here, the disk space is set here, the number of inodes, the number of files that can be open, the number of tcp sockets, etc. - all are set here.<br />
<br />
In fact, everything that can be set on this customer system is set in this conf file.<br />
<br />
<br />
All interaction with the customer system is done with the VEID. You start the system by running:<br />
<br />
vzctl start 999<br />
<br />
You stop it by running:<br />
<br />
vzctl stop 999<br />
<br />
You execute commands in it by running:<br />
<br />
vzctl exec 999 df -k<br />
<br />
You enter into it, via a root-shell backdoor with:<br />
<br />
vzctl enter 999<br />
<br />
and you set parameters for the system, while it is still running, with:<br />
<br />
vzctl set 999 --diskspace 6100000:6200000 --save<br />
<br />
<tt>vzctl</tt> is the most commonly used command - we have aliased <tt>v</tt> to <tt>vzctl</tt> since we use it so often. We’ll continue to use <tt>vzctl</tt> in our examples, but feel free to use just <tt>v</tt>.<br />
<br />
Let's say the user wants more diskspace. You can cat their conf file and see:<br />
<br />
DISKSPACE="4194304:4613734"<br />
<br />
So right now they have 4gigs of space. You can then change it to 6 with:<br />
<br />
vzctl set 999 --diskspace 6100000:6200000 --save<br />
<br />
IMPORTANT: all issuances of the vzctl set command need to end with <tt>–save</tt> - if they don't, the setting will be set, but it will not be saved to the conf file, and they will not have those settings next time they boot.<br />
<br />
All of the tunables in the conf file can be set with the vzctl set command. Note that in the conf file, and on the vzctl set command line, we always issue two numbers seperated by a colon - that is because we are setting the hard and soft limits. Always set the hard limit slightly above the soft limit, as you see it is in the conf file for all those settings.<br />
<br />
There are also things you can set with `<tt>vzctl set</tt>` that are not in the conf file as settings, per se. For instance, you can add IPs:<br />
<br />
vzctl set 999 --ipadd 10.10.10.10 --save<br />
<br />
or multiple IPs:<br />
<br />
vzctl set 999 --ipadd 10.10.10.10 --ipadd 10.10.20.30 --save<br />
<br />
or change the hostname:<br />
<br />
vzctl set 999 --hostname www.example.com --save<br />
<br />
You can even set the nameservers:<br />
<br />
vzctl set 999 --nameserver 198.78.66.4 --nameserver 198.78.70.180 --save<br />
<br />
Although you probably will never do that.<br />
<br />
You can disable a VPS from being started (by VZPP or reboot) (4.x):<br />
<br />
vzctl set 999 --disabled yes --save <br />
<br />
You can disable a VPS from being started (by VZPP or reboot) (<=3.x):<br />
<br />
vzctl set 999 --onboot=no --save <br />
<br />
You can disable a VPS from using his control panel:<br />
<br />
vzctl set 999 --offline_management=no --save <br />
<br />
You can suspend a VPS, so it can be resumed in the same state it was in when it was stopped (4.x):<br />
<br />
vzctl suspend 999<br />
<br />
and to resume it:<br />
<br />
vzctl resume 999<br />
<br />
to see who owns process:<br />
vzpid <PID><br />
<br />
to mount up an unmounted ve:<br />
vzctl mount 827<br />
<br />
<br />
To see network stats for CT's:<br />
<pre># vznetstat<br />
VEID Net.Class Output(bytes) Input(bytes)<br />
-----------------------------------------------<br />
24218 1 484M 39M<br />
24245 1 463M 143M<br />
2451 1 2224M 265M<br />
2454 1 2616M 385M<br />
4149 1 125M 68M<br />
418 1 1560M 34M<br />
472 1 1219M 315M<br />
726 1 628M 317M<br />
763 1 223M 82M<br />
771 1 4234M 437M<br />
-----------------------------------------------<br />
Total: 13780M 2090M</pre><br />
<br />
<br />
One thing that sometimes comes up on older systems that we created with smaller defaults is that the system would run out of inodes. The user will email and say they cannot create any more files or grow any files larger, but they will also say that they are not out of diskspace ... they are running:<br />
<br />
df -k<br />
<br />
and seeing how much space is free - and they are not out of space. They are most likely out of inodes - which they would see by running:<br />
<br />
df -i<br />
<br />
So, the first thing you should do is enter their system with:<br />
<br />
vzctl enter 999<br />
<br />
and run: <tt>df -i</tt><br />
<br />
to confirm your theory. Then exit their system. Then simply cat their conf file and see what their inodes are set to (probably 200000:200000, since that was the old default on the older systems) and run:<br />
<br />
vzctl set 999 --diskinodes 400000:400000 --save<br />
<br />
If they are not out of inodes, then a good possibility is that they have maxed out their numfile configuration variable, which controls how many files they can have in their system. The current default is 7500 (which nobody has ever hit), but the old default was as low as 2000, so you would run something like:<br />
<br />
vzctl set 999 --numfile 7500:7500 --save<br />
<br />
----<br />
<br />
You cannot start or stop a VE if your pwd is its private (/vz/private/999) or root (/vz/root/999) directories, or anywhere below them.<br />
<br />
<br />
== Recovering from a crash (linux) ==<br />
<br />
=== Diagnose whether you have a crash ===<br />
The most important thing is to get the machine and all ve’s back up as soon as possible. Note the time, you’ll need to create a crash log entry (Mgmt. -> Reference -> CrashLog). The first thing to do is head over to the [[Screen#Screen_Organization|serial console screen]] and see if there’s any kernel error messages output. Try to copy any messages (or just a sample of repeating messages) you see into the notes section of the crash log – these will also likely need to be sent to virtuozzo for interpretation. If the messages are spewing too fast, hit ^O + H to start a screen log dump which you can ob1182.pts-38.bb serve after the machine is rebooted. Additionally, if the machine is responsive, you can get a trace to send to virtuozzo by hooking up a kvm and entering these 3 sequences:<br />
<pre>alt+print screen+m<br />
alt+print screen+p<br />
alt+print screen+t</pre><br />
<br />
If there are no messages, the machine may just be really busy- wait a bit (5-10min) to see if it comes back. If it's still pinging, odds are its very busy. If it doesn't come back, or the messages indicate a fatal error, you will need to proceed with a power cycle (ctrl+alt+del will not work).<br />
<br />
=== Power cycle the server ===<br />
If this machine is not a Dell 2950 with a [[DRAC/RMM#DRAC|DRAC card]] (i.e. if you can’t ssh into the DRAC card and issue racadm serveraction hardreset, then you will need someone at the data center to power the macine off, wait 30 sec, then turn it back on. Make sure to re-attach via console (<tt>tip virtxx</tt>) immediately after power down. <br />
<br />
=== (Re)attach to the console ===<br />
Stay on the console the entire time during boot. As the BIOS posts- look out for the RAID card output- does everything look healthy? The output may be scrambled, look for "DEGRADED" or "FAILED". Once the OS starts booting you will be disconnected (dropped back to the shell on the console server) a couple times during the boot up. The reason you want to quickly re-attach is two-fold: 1. If you don’t reattach quickly then you won’t get any console output, 2. you want to be attached before the server ''potentially'' starts (an extensive) fsck. If you attach after the fsck begins, you’ll have seen no indication it started an fsck and the server will appear frozen during startup- no output, no response. <br />
<br />
=== Start containers/VE's/VPSs ===<br />
When the machine begins to start VE’s, it’s safe to leave the console and login via ssh. All virts should be set to auto start all the VEs after a crash. Further, most (newer) virts are set to “fastboot” it’s VE’s (to find out, do:<br />
grep -i fast /etc/sysconfig/vz <br />
and look for <tt>VZFASTBOOT=yes</tt>). If this was set prior to the machine’s crash (setting it after the machine boots will not have any effect until the vz service is restarted) it will start each ve as fast as possible, in serial, then go thru each VE (serially), shutting it down running a vzquota (disk usage) check, then bringing it back up. The benefit is that all VE’s are brought up quickly (within 15min or so depending on the #), the downside is a customer watching closely will notice 2 outages – 1st the machine crash, 2nd their quota check (which will be a much shorter downtime- on the order of a few minutes). <br />
<br />
Where “fastboot” is not set to yes (i.e on quar1), vz will start them consecutively, checking the quotas one at a time, and the 60th VE may not start until an hour or two later - this is not acceptable.<br />
<br />
The good news is, if you run vzctl start for a VE that is already started, you will simply get an error: <tt>VE is already started</tt>. Further, if you attempt to vzctl start a VE that is in the process of being started, you will simply get an error: unable to lock VE. So, there is no danger in simply running scripts to start smaller sets of VEs. If the system is not autostarting, then there is no issue, and even if it does, when it conflicts, one process (yours or the autostart) will lose, and just move on to the next one.<br />
<br />
A script has been written to assist with ve starts: [[#startvirt.pl|startvirt.pl]] which will start 6 ve’s at once until there are no more left. If startvirt.pl is used on a system where “fastboot” was on, it will circumvent the fastboot for ve’s started by startvirt.pl – they will go through the complete quota check before starting- therefore this is not advisable when a system has crashed. When a system is booted cleanly, and there's no need for vzquota checks, then startvirt.pl is safe and advisable to run.<br />
<br />
=== Make sure all containers are running ===<br />
You can quickly get a feel for how many ve’s are started by running:<br />
<br />
<pre>[root@virt4 log]# vs<br />
VEID 16066 exist mounted running<br />
VEID 16067 exist mounted running<br />
VEID 4102 exist mounted running<br />
VEID 4112 exist mounted running<br />
VEID 4116 exist mounted running<br />
VEID 4122 exist mounted running<br />
VEID 4123 exist mounted running<br />
VEID 4124 exist mounted running<br />
VEID 4132 exist mounted running<br />
VEID 4148 exist mounted running<br />
VEID 4151 exist mounted running<br />
VEID 4155 exist mounted running<br />
VEID 42 exist mounted running<br />
VEID 432 exist mounted running<br />
VEID 434 exist mounted running<br />
VEID 442 exist mounted running<br />
VEID 450 exist mounted running<br />
VEID 452 exist mounted running<br />
VEID 453 exist mounted running<br />
VEID 454 exist mounted running<br />
VEID 462 exist mounted running<br />
VEID 463 exist mounted running<br />
VEID 464 exist mounted running<br />
VEID 465 exist mounted running<br />
VEID 477 exist mounted running<br />
VEID 484 exist mounted running<br />
VEID 486 exist mounted running<br />
VEID 490 exist mounted running</pre><br />
<br />
So to see how many ve’s have started:<br />
<pre>[root@virt11 root]# vs | grep running | wc -l<br />
39</pre><br />
<br />
And to see how many haven’t:<br />
<pre>[root@virt11 root]# vs | grep down | wc -l<br />
0</pre><br />
<br />
And how many we should have running:<br />
<pre>[root@virt11 root]# vs | wc -l<br />
39</pre><br />
<br />
Another tool you can use to see which ve’s have started, among other things is [[#vzstat|vzstat]]. It will give you CPU, memory, and other stats on each ve and the overall system. It’s a good thing to watch as ve’s are starting (note the VENum parameter, it will tell you how many have started):<br />
<br />
<pre>4:37pm, up 3 days, 5:31, 1 user, load average: 1.57, 1.68, 1.79<br />
VENum 40, procs 1705: running 2, sleeping 1694, unint 0, zombie 9, stopped 0<br />
CPU [ OK ]: VEs 57%, VE0 0%, user 8%, sys 7%, idle 85%, lat(ms) 412/2<br />
Mem [ OK ]: total 6057MB, free 9MB/54MB (low/high), lat(ms) 0/0<br />
Swap [ OK ]: tot 6142MB, free 4953MB, in 0.000MB/s, out 0.000MB/s<br />
Net [ OK ]: tot: in 0.043MB/s 402pkt/s, out 0.382MB/s 4116pkt/s<br />
Disks [ OK ]: in 0.002MB/s, out 0.000MB/s<br />
<br />
VEID ST %VM %KM PROC CPU SOCK FCNT MLAT IP<br />
1 OK 1.0/17 0.0/0.4 0/32/256 0.0/0.5 39/1256 0 9 69.55.227.152<br />
21 OK 1.3/39 0.1/0.2 0/46/410 0.2/2.8 23/1860 0 6 69.55.239.60<br />
133 OK 3.1/39 0.1/0.3 1/34/410 6.3/2.8 98/1860 0 0 69.55.227.147<br />
263 OK 2.3/39 0.1/0.2 0/56/410 0.3/2.8 34/1860 0 1 69.55.237.74<br />
456 OK 17/39 0.1/0.2 0/100/410 0.1/2.8 48/1860 0 11 69.55.236.65<br />
476 OK 0.6/39 0.0/0.2 0/33/410 0.1/2.8 96/1860 0 10 69.55.227.151<br />
524 OK 1.8/39 0.1/0.2 0/33/410 0.0/2.8 28/1860 0 0 69.55.227.153<br />
594 OK 3.1/39 0.1/0.2 0/45/410 0.0/2.8 87/1860 0 1 69.55.239.40<br />
670 OK 7.7/39 0.2/0.3 0/98/410 0.0/2.8 64/1860 0 216 69.55.225.136<br />
691 OK 2.0/39 0.1/0.2 0/31/410 0.0/0.7 25/1860 0 1 69.55.234.96<br />
744 OK 0.1/17 0.0/0.5 0/10/410 0.0/0.7 7/1860 0 6 69.55.224.253<br />
755 OK 1.1/39 0.0/0.2 0/27/410 0.0/2.8 33/1860 0 0 192.168.1.4<br />
835 OK 1.1/39 0.0/0.2 0/19/410 0.0/2.8 5/1860 0 0 69.55.227.134<br />
856 OK 0.3/39 0.0/0.2 0/13/410 0.0/2.8 16/1860 0 0 69.55.227.137<br />
936 OK 3.2/52 0.2/0.4 0/75/410 0.2/0.7 69/1910 0 8 69.55.224.181<br />
1020 OK 3.9/39 0.1/0.2 0/60/410 0.1/0.7 55/1860 0 8 69.55.227.52<br />
1027 OK 0.3/39 0.0/0.2 0/14/410 0.0/2.8 17/1860 0 0 69.55.227.83<br />
1029 OK 1.9/39 0.1/0.2 0/48/410 0.2/2.8 25/1860 0 5 69.55.227.85<br />
1032 OK 12/39 0.1/0.4 0/80/410 0.0/2.8 41/1860 0 8 69.55.227.90</pre><br />
<br />
When you are all done, you will want to make sure that all the VEs really did get started, run vs one more time.<br />
<br />
Note the time all ve’s are back up and enter that into and save the crash log entry.<br />
<br />
Occasionally, a ve will not start automatically. The most common reason for a ve not to come up normally is the ve was at it’s disk limit before the crash, and will not start since they’re over the limit. To overcome this, set the disk space to current usage level (the system will give this to you when it fails to start), start the ve, then re-set the disk space back to the prior level. Lastly, contact the customer to let them know they’re out of disk (or allocate more disk if they're entitled to more).<br />
<br />
== Hitting performance barriers and fixing them ==<br />
<br />
There are multiple modes virtuozzo offers to allocate resources to a ve. We utilize 2: SLM and UBC parameters<br />
On our 4.x systems, we use all SLM – it’s simpler to manage and understand. There are a few systems on virt19/18 that may also use SLM. Everything else uses UBC. <br />
You can tell a SLM ve by:<br />
<br />
SLMMODE="all"<br />
<br />
in their conf file. <br />
<br />
TODO: detail SLM modes and parameters.<br />
<br />
If someone is in SLM mode and they hit memory resource limits, they simply need to upgrade to more memory.<br />
<br />
The following applies to everyone else (UBC).<br />
<br />
Customers will often email and say that they are getting out of memory errors - a common one is "cannot fork" ... basically, anytime you see something odd like this, it means they are hitting one of their limits that is in place in their conf file.<br />
<br />
The conf file, however, simply shows their limits - how do we know what they are currently at ?<br />
<br />
The answer is a file called v - this file contains the current status (and peaks) of their performance settings, and also counts how many times they have hit the barrier. The output of the file looks like this:<br />
<br />
<pre>764: kmemsize 384113 898185 8100000 8200000 0<br />
lockedpages 0 0 322 322 0<br />
privvmpages 1292 7108 610000 615000 0<br />
shmpages 270 528 33000 34500 0<br />
dummy 0 0 0 0 0<br />
numproc 8 23 410 415 0<br />
physpages 48 5624 0 2147483647 0<br />
vmguarpages 0 0 13019 2147483647 0<br />
oomguarpages 641 6389 13019 2147483647 0<br />
numtcpsock 3 21 1210 1215 0<br />
numflock 1 3 107 117 0<br />
numpty 0 2 19 19 0<br />
numsiginfo 0 4 274 274 0<br />
tcpsndbuf 0 80928 1800000 1900000 0 <br />
tcprcvbuf 0 108976 1800000 1900000 0<br />
othersockbuf 2224 37568 900000 950000 0<br />
dgramrcvbuf 0 4272 200000 200000 0<br />
numothersock 3 9 650 660 0<br />
dcachesize 53922 100320 786432 818029 0<br />
numfile 161 382 7500 7600 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
numiptent 4 4 155 155 0</pre><br />
<br />
The first column is the name of the counter in question - the same names we saw in the systems conf file. The second column is the _current_ value of that counter, the third column is the max that that counter has ever risen to, the fourth column is the soft limit, and the fifth column is the hard limit (which is the same as the numbers in that systems conf file).<br />
<br />
The sixth number is the failcount - how many times the current usage has risen to hit the barrier. It will increase as soon as the current usage hits the soft limit.<br />
<br />
The problem with /proc/user_beancounters is that it actually contains that set of data for every running VE - so you can't just cat /proc/user_beancounters - it is too long and you get info for every other running system.<br />
<br />
You can vzctl enter the system and run:<br />
<br />
vzctl enter 9999<br />
cat /proc/user_beancounters<br />
<br />
inside their system, and you will just see the stats for their particular system, but entering their system every time you want to see it is combersome.<br />
<br />
So, I wrote a simple script called "vzs" which simply greps for the VEID, and spits out the next 20 or so lines (however many lines there are in the output, I forget) after it. For instance:<br />
<br />
<pre># vzs 765:<br />
765: kmemsize 2007936 2562780 8100000 8200000 0<br />
lockedpages 0 8 322 322 0<br />
privvmpages 26925 71126 610000 615000 0<br />
shmpages 16654 16750 33000 34500 0<br />
dummy 0 0 0 0 0<br />
numproc 41 57 410 415 0<br />
physpages 1794 49160 0 2147483647 0<br />
vmguarpages 0 0 13019 2147483647 0<br />
oomguarpages 4780 51270 13019 2147483647 0<br />
numtcpsock 23 37 1210 1215 0<br />
numflock 17 39 107 117 0<br />
numpty 1 3 19 19 0<br />
numsiginfo 0 6 274 274 0<br />
tcpsndbuf 22240 333600 1800000 1900000 0<br />
tcprcvbuf 0 222656 1800000 1900000 0<br />
othersockbuf 104528 414944 900000 950000 0<br />
dgramrcvbuf 0 4448 200000 200000 0<br />
numothersock 73 105 650 660 0<br />
dcachesize 247038 309111 786432 818029 0<br />
numfile 904 1231 7500 7600 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
numiptent 4 4 155 155 0</pre><br />
<br />
That showed us just the portion of /proc/user_beancounters for system 765.<br />
<br />
When you run the vzs command, always add a : after the VEID.<br />
<br />
So, if a customer complains about some out of memory errors, or no more files, or no more ptys, or just has an unspecific complain about processes dying, etc., the very first thing you need to do is check their beancounters with vzs. Usually you will spot an item that has a high failcount and needs to be upped.<br />
<br />
At that point you could simply up the counter with `vzctl set`. Generally pick a number 10-20% higher than the old one, and make the hard limit slightly larger than the the soft limit. However our systems now come in several levels and those levels have more/different memory allocations. If someone is complaining about something other than a memory limit (pty, numiptent, numflock), it’s generally safe to increase it, at least to the same level as what’s in the /vzconf/4unlimited file on the newest virt. If someone is hitting a memory limit, first make sure they are given what they deserve:<br />
<br />
(refer to mgmt -> payments -> packages)<br />
<br />
To set those levels, you use the [[#setmem|setmem]] command. <br />
<br />
The alternate (DEPRECATED) method would be to use one of 3 commands:<br />
256 <veid><br />
300 <veid><br />
384 <veid><br />
512 <veid><br />
<br />
If the levels were not right (you’d run vzs <veid> before and after to see the effect) tell the customer they’ve been adjusted and be done with it. If the levels were right, tell the customer they must upgrade to a higher package, tell them how to see level (control panel) and that they can reboot their system to escape this lockup contidion.<br />
<br />
Customers can also complain that their site is totally unreachable, or complain that it is down ... if the underlying machine is up, and all seems well, you may notice in the beancounters that network-specific counters are failing - such as numtcpsock, tcpsndbuf or tcprcvbuf. This will keep them from talking on the network and make it seem like their system is down. Again, just up the limits and things should be fine.<br />
<br />
On virts 1-4, you should first look at the default settings for that item on a later virt, such as virt 8 - we have increased the defaults a lot since the early machines. So, if you are going to up a counter on virt2, instead of upping it by 10-20%, instead up it to the new default that you see on virt8.<br />
<br />
== Moving a VE to another virt (migrate/migrateonline) ==<br />
<br />
This will take a while to complete - and it is best to do this at night when the load is light on both machines.<br />
<br />
There are different methods for this, depending on which version of virtuozzo is installed on the src. and dst. virt. <br />
To check which version is running: <br />
[root@virt12 private]# cat /etc/virtuozzo-release<br />
Virtuozzo release 2.6.0<br />
<br />
Ok, let's say that the VE is 1212, and vital stats are:<br />
<pre>[root@virt12 sbin]# vc 1212<br />
VE_ROOT="/vz1/root/1212"<br />
VE_PRIVATE="/vz1/private/1212"<br />
OSTEMPLATE="fedora-core-2/20040903"<br />
IP_ADDRESS="69.55.229.84"<br />
TEMPLATES="devel-fc2/20040903 php-fc2/20040813 mysql-fc2/20040812 postgresql-fc2/20040813 mod_perl-fc2/20040812 mod_ssl-fc2/20040811 jre-fc2/20040823 jdk-fc2/20040823 mailman-fc2/20040823 analog-fc2/20040824 proftpd-fc2/20040818 tomcat-fc2/20040823 usermin-fc2/20040909 webmin-fc2/20040909 uw-imap-fc2/20040830 phpBB-fc2/20040831 spamassassin-fc2/20040910 PostNuke-fc2/20040824 sl-webalizer-fc2/20040<br />
818"<br />
<br />
[root@virt12 sbin]# vzctl exec 1212 df -h<br />
Filesystem Size Used Avail Use% Mounted on<br />
/dev/vzfs 4.0G 405M 3.7G 10% /</pre><br />
<br />
From this you can see that he’s using (and will minimally need free on the dst server) ~400MB, and he’s running on a Fedora 2 template, version 20040903. He’s also got a bunch of other templates installed. It’s is '''vital''' that '''all''' these templates exist on the dst system. To confirm that, on the dst system run:<br />
<br />
For < 3.0:<br />
<br />
<pre>[root@virt14 private]# vzpkgls | grep fc2<br />
devel-fc2 20040903<br />
PostNuke-fc2 20040824<br />
analog-fc2 20040824<br />
awstats-fc2 20040824<br />
bbClone-fc2 20040824<br />
jdk-fc2 20040823<br />
jre-fc2 20040823<br />
mailman-fc2 20040823<br />
mod_frontpage-fc2 20040816<br />
mod_perl-fc2 20040812<br />
mod_ssl-fc2 20040811<br />
mysql-fc2 20040812<br />
openwebmail-fc2 20040817<br />
php-fc2 20040813<br />
phpBB-fc2 20040831<br />
postgresql-fc2 20040813<br />
proftpd-fc2 20040818<br />
sl-webalizer-fc2 20040818<br />
spamassassin-fc2 20040910<br />
tomcat-fc2 20040823<br />
usermin-fc2 20040909<br />
uw-imap-fc2 20040830<br />
webmin-fc2 20040909<br />
[root@virt14 private]# vzpkgls | grep fedora<br />
fedora-core-1 20040121 20040818<br />
fedora-core-devel-1 20040121 20040818<br />
fedora-core-2 20040903<br />
[root@virt14 private]#</pre><br />
<br />
For these older systems, you can simply match up the date on the template. <br />
<br />
For >= 3.0:<br />
<br />
<pre>[root@virt19 /vz2/private]# vzpkg list<br />
centos-5-x86 2008-01-07 22:05:57<br />
centos-5-x86 devel<br />
centos-5-x86 jre<br />
centos-5-x86 jsdk<br />
centos-5-x86 mod_perl<br />
centos-5-x86 mod_ssl<br />
centos-5-x86 mysql<br />
centos-5-x86 php<br />
centos-5-x86 plesk9<br />
centos-5-x86 plesk9-antivirus<br />
centos-5-x86 plesk9-api<br />
centos-5-x86 plesk9-atmail<br />
centos-5-x86 plesk9-backup<br />
centos-5-x86 plesk9-horde<br />
centos-5-x86 plesk9-mailman<br />
centos-5-x86 plesk9-mod-bw<br />
centos-5-x86 plesk9-postfix<br />
centos-5-x86 plesk9-ppwse<br />
centos-5-x86 plesk9-psa-firewall<br />
centos-5-x86 plesk9-psa-vpn<br />
centos-5-x86 plesk9-psa-fileserver<br />
centos-5-x86 plesk9-qmail<br />
centos-5-x86 plesk9-sb-publish<br />
centos-5-x86 plesk9-vault<br />
centos-5-x86 plesk9-vault-most-popular<br />
centos-5-x86 plesk9-watchdog</pre><br />
<br />
On these newer systems, it's difficult to tell whether the template on the dst matches exactly the src. Just cause a centos-5-x86 is listed on both servers doesn't mean all the same packages are there on the dst. To truly know, you must perform a sample rsync:<br />
<br />
rsync -avn /vz/template/centos/5/x86/ root@10.1.4.61:/vz/template/centos/5/x86/<br />
<br />
if you see a ton of output from the dry run command, then clearly there are some differences. You may opt to let the rsync complete (without running in dry run mode) the only downside is you've now used up more space on the dst and also the centos template will be a mess with old and new data- it will be difficult if not impossible to undo (if someday we wanted to reclaim the space).<br />
<br />
If you choose to merge templates, you should closely inspect the dry run output. You should also take care to exclude anything in the /config directory. For example:<br />
<br />
rsync -av -e ssh --stats --exclude=x86/config /vz/template/ubuntu/10.04/ root@10.1.4.62:/vz/template/ubuntu/10.04/<br />
<br />
Which will avoid this directory and contents:<br />
<pre>[root@virt11 /vz2/private]# ls /vz/template/ubuntu/10.04/x86/config*<br />
app os</pre><br />
<br />
This is important to avoid since the config may differ on the destination and we are really only interested in making sure the pacakges are there, not overwriting a newer config with an older one.<br />
<br />
If the dst system was missing a template, you have 2 choices: <br />
# put the missing template on the dst system. 2 choices here: <br />
## Install the template from rpm (found under backup2: /mnt/data4/vzrpms/distro/) or <br />
## rsync over the template (found under /vz/template) - see above<br />
# put the ve on a system which has all the proper templates<br />
<br />
=== pre-seeding a migration ===<br />
<br />
When migrating a customer (or when doing many) depending on how much data you have to transfer, it can take some time. Further, it can be difficult to gauge when a migration will complete or how long it will take. To help speed up the process and get a better idea about how long it will take you can pre-transfer a customer's data to the destination server. If done correctly, vzmigrate will see the pre-transferred data and pick up where you left off, having much less to transfer (just changed/new files). <br />
<br />
We believe vzmigrate uses rsync to do it's transfer. Therefore not only can you use rsync to do a pre-seed, you can also run rsync to see what is causing a repeatedly-failing vzmigrate to fail. <br />
<br />
There's no magic to a pre-seed, you just need to make sure it's named correctly.<br />
<br />
Given:<br />
<br />
source: /vz1/private/1234<br />
<br />
and you want to migrate to /vz2 on the target system, your rsync would look like:<br />
<br />
rsync -av /vz1/private/1234/ root@x.x.x.x:/vz2/private/1234.migrated/<br />
<br />
After running that successful rsync, the ensuing migrateonline (or migrate) will take much less time to complete- depending on the # of files to be analyzed and the # of changed files. In any case, it'll be much much faster than had you just started the migration from scratch.<br />
<br />
Further, as we discuss elsewhere in this topic, a failed migration can be moved from <tt>/vz/private/1234</tt> to <tt>/vz/private/1234.migrated</tt> on the destination if you want to restart a failed migration. This should '''only''' be done if the migration failed and the CT is not running on the destination HN.<br />
<br />
=== migrateonline intructions: src >=3.x -> dst>=3.x ===<br />
<br />
A script called [[#migrateonline|migrateonline]] was written to handle this kind of move. It is basically a wrapper for <tt>vzmigrate</tt> – vzmigrate is a util to seamlessly- as no no reboot of the ve necessary- move a ve from one host to another. This wrapper was initially written cause vz virtuozzo version 2.6.0 has a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables, causing problems when the ve was started up on the dst system. [[#migrate|migrate]] mitigates that. Since it makes multiple ssh connections to the dst virt, it’s a good idea to put the pub key for the src virt in the authorized_keys file on the dst virt. In addition, migrateonline emails ve owners when their migration starts and stops. For this to happen they need to put email addresses (on a single line, space delimited) in a file on their system: /migrate_notify. If the optional target dir is not specified, the ve will be moved to the same private/root location as it was on the src virt. Note: <tt>migrate</tt> is equivalent to <tt>migrateonline</tt>, but will <tt>migrate</tt> a ve AND restart it in the process.<br />
<br />
<pre>[root@virt12 sbin]# migrateonline<br />
usage: /usr/local/sbin/migrateonline <ip of node migrating to> <veid> [target dir: vz | vz1 | vz2]<br />
[root@virt12 sbin]# migrateonline 10.1.4.64 1212 vz<br />
starting to migrate 1212 at Sat Mar 26 22:40:38 PST 2005<br />
Turning off offline_management<br />
Saved parameters for VE 1212<br />
migrating with no start on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
clearing the arp cache<br />
now going to 10.1.4.64 and clear cache and starting<br />
starting it<br />
Delete port redirection<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
finished migrating 1212 at Sat Mar 26 22 22:52:01 PST 2005<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine.<br />
<br />
If they had backups, use the mvbackups command to move their backups to the new server:<br />
<br />
mvbackups 1212 virt14 vz<br />
<br />
Rename the ve<br />
<br />
[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/migrated-1212<br />
<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/old-1212-migrated-20120404-noarchive<br />
<br />
Update the customer’s systems in mgmt to reflect the new path and server.<br />
<br />
<br />
IF migrateonline does not work, you can try again using simply migrate- this will result in a brief reboot for the ve.<br />
Before you try again, make sure of a few things:<br />
<br />
Depending on where in the migration died, there may be partial data on the dst system in 1 of 2 places:<br />
(given the example above)<br />
<br />
/vz/private/1212<br />
<br />
or <br />
<br />
/vz/private/1212.migrated<br />
<br />
before you run migrate again, you'll want to rename so that all data is in <br />
1212.migrated:<br />
<br />
mv /vz/private/1212 /vz/private/1212.migrated<br />
<br />
this way, it will pick up where it left off and transfer only new files.<br />
<br />
Likewise, if you want to speed up a migration, you can pre-seed the dst as follows:<br />
<br />
[root@virt12 sbin]# rsync -avSH /vz/private/1212/ root@10.1.4.64:/vz/private/1212.migrated/<br />
<br />
then when you run migrate or migrateonline, it will only need to move the changed files- the migration will complete quickly<br />
<br />
=== migrate manually (if migrate/online fails) ===<br />
<br />
Lets say for whatever reason the migration fails. You can always move someone manually:<br />
<br />
1. stop ve: <br><br />
v stop 1234<br />
<br />
2. copy over data<br><br />
rsync -avSH /vz/private/1234/ root@1.1.1.1:/vzX/private/1234/<br />
<br />
NOTE: if you've previously seeded the data (run rsync while the VE was up/running), and this is a subsequent rsync, make sure the last rsync you do (while the VE is not running, has the --delete option in the rsync)<br />
<br />
3. copy over conf<br><br />
scp /vzconf/1234.conf root@1.1.1.1:/vzconf<br />
<br />
4. on dst, edit the conf to reflect the right vzX dir<br><br />
vi /vzconf/1234.conf<br />
<br />
5. on src remove the IPs<br><br />
ipdel 1234 2.2.2.2 3.3.3.3<br />
<br />
6. on dst add IPs <br><br />
ipadd 1234 2.2.2.2 3.3.3.3<br />
<br />
7. on dst, start ve: <br><br />
v start 1324<br />
<br />
8. cancel, then archive ve on src per above instrs.<br />
<br />
=== migrate src=2.6.0 -> dst>=2.6.0, or mass-migration with customer notify ===<br />
<br />
A script called <tt>migrate</tt> was written to handle this kind of move. It is basically a wrapper for vzmigrate – vzmigrate is a util to seamlessly move a ve from one host to another. This wrapper was initially written cause vz virtuozzo version 2.6.0 has a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables, causing problems when the ve was started up on the dst system. migrate mitigates that. Since it makes multiple ssh connections to the dst virt, it’s a good idea to put the pub key for the src virt in the authorized_keys file on the dst virt. In addition, migrate emails ve owners when their migration starts and stops. For this to happen they need to put email addresses (on a single line, space delimited) in a file on their system: /migrate_notify. If the optional target dir is not specified, the ve will be moved to the same private/root location as it was on the src virt. Note: migrateonline is equivalent to migrate, but will migrate a ve from one 2.6 '''kernel''' machine to another 2.6 kernel machine without restarting the ve.<br />
<br />
<pre>[root@virt12 sbin]# migrate<br />
usage: /usr/local/sbin/migrate <ip of node migrating to> <veid> [target dir: vz | vz1 | vz2]<br />
[root@virt12 sbin]# migrate 10.1.4.64 1212 vz<br />
starting to migrate 1212 at Sat Mar 26 22:40:38 PST 2005<br />
Turning off offline_management<br />
Saved parameters for VE 1212<br />
migrating with no start on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
clearing the arp cache<br />
now going to 10.1.4.64 and clear cache and starting<br />
starting it<br />
Delete port redirection<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
finished migrating 1212 at Sat Mar 26 22 22:52:01 PST 2005<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine (backup2 or mail).<br />
Cancel the ve (first we have to rename things which migrate changed so cancelve will find them):<br />
<br />
[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/1212.conf<br />
<br />
On 2.6.1 you’ll also have to move the private area:<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/1212<br />
<br />
<pre>[root@virt12 sbin]# cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, [[#cancelve|cancelve]] would offer to remove them. You want to say '''no''' to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
To move everyone off a system, you’d do:<br />
for f in `vl`; do migrate <ip> $f; done<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
=== vzmigrate: src=2.6.1 -> dst>=2.6.0 ===<br />
<br />
This version of vzmigrate works properly with regard to handling ips. It will not notify ve owners of moves as in the above example. Other than that it’s essentially the same.<br />
<br />
<pre>[root@virt12 sbin]# vzmigrate 10.1.4.64 -r no 1212:1212:/vz/private/1212:/vz/root/1212<br />
migrating on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine (backup2 or mail).<br />
Cancel the ve (first we have to rename things which vzmigrate changed so cancelve will find them):<br />
<br />
<pre>[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/1212.conf<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/1212<br />
<br />
[root@virt12 sbin]# cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, <tt>cancelve</tt> would offer to remove them. You want to say no to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
=== src=2.5.x ===<br />
<br />
First, go to the private dir:<br />
<br />
cd /vz1/private/<br />
<br />
Stop the VE - make sure it stops totally cleanly.<br />
<br />
vzctl stop 1212<br />
<br />
Then you’d use vemove - a script written to copy over the config, create tarballs of the ve’s data on the destination virt, and cancel the ve on the source system (in this example we’re going to put a ve that was in /vz1/private on the src virt, in /vz/private on the dst virt):<br />
<br />
<pre>[root@virt12 sbin]# vemove<br />
ERROR: Usage: vemove veid target_ip target_path_dir<br />
[root@virt12 sbin]# vemove 1212 10.1.4.64 /vz/private/1212<br />
tar cfpP - 1212 --ignore-failed-read | (ssh -2 -c arcfour 10.1.4.64 "split - -b 1024m /vz/private/1212.tar" )<br />
scp /vzconf/1212.conf 10.1.4.64:/vzconf<br />
cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, cancelve would offer to remove them. You want to say no to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
When you are done, go to /vz/private on the dst virt you will have files like this:<br />
<br />
<pre>1212.taraa<br />
1212.tarab<br />
1212.tarac</pre><br />
<br />
Each one 1024m (or less, for the last one) in size.<br />
<br />
on the dst server and run:<br />
<br />
cat 1212.tar?? | tar xpPBf -<br />
<br />
and after 20 mins or so it will be totally untarred. Now since the conf<br />
file is already there, you can go ahead and start the system.<br />
<br />
vzctl start 1212<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
NOTE: you MUST tar the system up using the virtuozzo version of tar that<br />
is on all the virt systems, and further you MUST untar the tarball with<br />
the virtuozzo tar, using these options: `<tt>tar xpPBf -</tt>`<br />
<br />
If you tar up an entire VE and move it to a non-virtuozzo machine, that is<br />
ok, and you can untar it there with normal tar commands, but do not untar<br />
it and then repack it with a normal tar and expect it to work - you need<br />
to use virtuozzo tar commands on virtuozzo tarballs to make it work.<br />
<br />
The backups are sort of an exception, since we are just (usually)<br />
restoring user data that was created after we gave them the system, and<br />
therefore has nothing to do with magic symlinks or vz-rpms, etc.<br />
<br />
== Moving a VE on the same virt ==<br />
<br />
Easy way:<br><br />
Scenario 1: ve 123 is to be renamed 1231 and moved from vz1 to vz<br />
<br />
vzmlocal 123:1231:/vz/private/1231:/vz/root/1231<br />
<br />
Scenario 2: ve 123 is to be moved vz1 to vz<br />
<br />
vzmlocal 123:123:/vz/private/123:/vz/root/123<br />
<br />
vzmlocal will reboot the ve at the end of the move<br />
<br />
Manual/old way:<br />
<br />
1) <tt>vzctl stop 123</tt><br><br />
2) <tt>mv /vz1/private/123 /vz/private/.</tt><br><br />
(or cp -a if you want to copy)<br />
3) in <tt>/etc/sysconfig/vz-scripts/123.conf</tt> change value<br><br />
of '<tt>VE_PRIVATE</tt>' variable to point to a new private area location<br />
4) <tt>vzctl start 123</tt><br><br />
5) update backups if needed: <tt>mvbackups 123 virtX virt1 vz</tt><br><br />
6) update management scerens<br><br />
<br />
Notes: a) absolute path to private area is stored in quota file <tt>/var/vzquota/quota.123</tt> - so during first startup quota will be recalculated.<br><br />
b) if you're going to write some script to do a job, you MUST be sure that $VEID won't be expanded to '' in ve config file - ie. you need to escape '$'. Otherwise you might have:<br />
<br />
VE_PRIVATE="/vz/private/"<br />
<br />
in config, and 'vzctl destroy' for this VE ID '''will remove everything under /vz/private/ directory'''.<br />
<br />
== Adding a veth device to a VE ==<br />
<br />
Not totally sure what this is, but a customer asked for it and here's what we did (as instructed by vz support):<br />
<br />
<pre>v set 99 --netif_add eth99 --save<br />
ipdel 99 69.55.230.58<br />
v set 99 --ifname eth99 --ipadd 69.55.230.58 --save<br />
v set 99 --ifname eth99 --gateway 69.55.230.1 --save<br />
<br />
vznetcfg net new veth_net<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
net99 active eth0 veth77.77,veth99.99<br />
veth_net active<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
br0 bridge veth_net<br />
br99 bridge net99<br />
veth99.99 veth net99<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic net99 69.55.227.70/24<br />
<br />
vznetcfg br attach br0 eth0<br />
<br />
(will remove 99 from orig net and move to veth_net)<br />
vznetcfg net addif veth_net veth99.99<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
net99 active<br />
veth_net active eth0 veth77.77,veth99.99<br />
<br />
(delete the old crap)<br />
vznetcfg net del net99<br />
<br />
<br />
Then, to add another device in<br />
<br />
v set 77 --netif_add eth77 --save<br />
ipdel 77 69.55.230.78<br />
v set 77 --ifname eth77 --ipadd 69.55.230.78 --save<br />
v set 77 --ifname eth77 --gateway 69.55.230.1 --save<br />
v set 77 --save --ifname eth77 --network veth_net<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
veth77.77 veth<br />
br0 bridge veth_net<br />
veth99.99 veth veth_net<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic veth_net 69.55.227.70/24<br />
<br />
vznetcfg net addif veth_net veth77.77<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
veth77.77 veth veth_net<br />
br0 bridge veth_net<br />
veth99.99 veth veth_net<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic veth_net 69.55.227.70/24<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
veth_net active eth0 veth77.77,veth99.99<br />
<br />
another example<br />
<br />
v set 1182 --netif_add eth1182 --save<br />
ipdel 1182 69.55.236.217<br />
v set 1182 --ifname eth1182 --ipadd 69.55.236.217 --save<br />
v set 1182 --ifname eth1182 --gateway 69.55.236.1 --save<br />
vznetcfg net addif veth_net veth1182.1182<br />
v set 1182 --save --ifname eth1182 --network veth_net<br />
<br />
<br />
unused/not working commands:<br />
ifconfig veth99.0 0<br />
vznetcfg net list<br />
vznetcfg br new br99 net99<br />
vznetcfg br attach br99 eth0<br />
vznetcfg br show<br />
vznetcfg if list<br />
vznetcfg net addif net99 veth99.99<br />
vznetcfg net addif eth0 net99<br />
<br />
---------<br />
<br />
vznetcfg br new br1182 net1182<br />
<br />
vznetcfg br attach br99 eth0<br />
vznetcfg if list<br />
vznetcfg net addif net99 veth99.99<br />
vznetcfg net addif eth0 net99<br />
<br />
<br />
<br />
vznetcfg net addif eth0 net1182<br />
<br />
<br />
# vznetcfg net addif veth99.0 net99<br />
--- 8< ---<br />
<br />
vznetcfg net new net<br />
# vznetcfg net addif eth0 net99<br />
<br />
<br />
# vzctl set 99 --save --netif_add eth0 (at this stage veth99.0 interface have to appear<br />
on node)<br />
# vzctl set 99 --save --ifname eth0 --ipadd 69.55.230.58 (and probably few more arguments<br />
here - see 'man vzctl')<br />
# vznetcfg net addif veth99.0 net99</pre><br />
<br />
<br />
== Assigning/remove ip from a VE ==<br />
<br />
1. Add or remove ips:<br />
ipdel 1234 1.1.1.1 2.2.2.2<br />
ipadd 1234 1.1.1.1 2.2.2.2<br />
<br />
2. update Mgmt screens<br />
<br />
3. offer to update any DNS we do for them<br />
<br />
4. check to see if we had rules for old IP in firwall<br />
<br />
== Enabling tun device for a ve ==<br />
Note, there’s a command for this: [[#addtun|addtun]]<br />
<br />
For posterity:<br />
Make sure the tun.o module is already loaded before Virtuozzo is started: <br />
lsmod <br />
Allow the VPS to use the TUN/TAP device: <br />
vzctl set 101 --devices c:10:200:rw --save <br />
Create the corresponding device inside the VPS and set the proper permissions: <br />
vzctl exec 101 mkdir -p /dev/net <br />
vzctl exec 101 mknod /dev/net/tun c 10 200 <br />
vzctl exec 101 chmod 600 /dev/net/tun<br />
<br />
== Remaking a system (on same virt) ==<br />
<br />
1. [[#cancelve|cancelve]] (or v destroy x - ONLY if you're POSITIVE no data needs to be saved)<br />
<br />
2. [[#vemake|vemake]] using same veid<br />
<br />
3. [[#mvbackups|mvbackups]] or [[#vb|vb]] (if new mount point)<br />
<br />
4. update mgmt with new dir/ip <br />
<br />
5. update firewall if changed ip<br />
<br />
== Re-initialize quota for a VE ==<br />
<br />
There’s a commamd for this now: [[#clearquota|clearquota]]<br />
<br />
For posterity:<br />
<br />
vzctl stop 1<br />
vzquota drop 1<br />
vzctl start 1<br />
<br />
<br />
== Traffic accounting on linux ==<br />
<br />
DEPRECATED - all tracking is done via bwdb now. This is how we used to track traffic.<br />
<br />
TODO: update for diff versions of vz<br />
<br />
Unlike FreeBSD, where we have to add firewall count rules to the system to count the traffic, on virtuozzo counts the traffic for us. You an see the current traffic stats by running `vznetstat`:<br />
<br />
<pre># vznetstat<br />
VEID Net.Class Output(bytes) Input(bytes)<br />
-----------------------------------------------<br />
24218 1 484M 39M<br />
24245 1 463M 143M<br />
2451 1 2224M 265M<br />
2454 1 2616M 385M<br />
4149 1 125M 68M<br />
418 1 1560M 34M<br />
472 1 1219M 315M<br />
726 1 628M 317M<br />
763 1 223M 82M<br />
771 1 4234M 437M<br />
-----------------------------------------------<br />
Total: 13780M 2090M</pre><br />
<br />
As you can see the VEID is on a line with the in and out bytes. So, we simply run a cron job:<br />
<br />
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /root/vztrafdump.sh<br />
<br />
Just like we do on FreeBSD - this one goes through all the VEs in /vz/private and greps the line from vznetstat that matches them and dumps it in /jc_traffic_dump on their system. Then it does it again for all the VEs in /vz1/private. It is important to note that vznetstat runs only once, and the grepping is done from a temporary file that contains that output - we do this because running vznetstat once for each VE that we read out of /vz/private and /vz1/private would take way too long and be too intensive.<br />
<br />
You do not need to do anything to facilitate this other than make sure that that cron job is running - the vznetstat counters are always running, and any new VEs that are added to the system will be accounted for automatically.<br />
<br />
Traffic resetting no longer works with vz 2.6, so we disable the vztrafdump.sh on those virts.<br />
<br />
== Watchdog script ==<br />
<br />
On some of the older virts, we have a watchdog running that kills procs that are deemed bad per the following:<br />
<br />
/root/watchdog from quar1<br />
<br />
<pre>if echo $line | awk '{print $(NF-3)}' | grep [5-9]...<br />
then<br />
# 50-90%<br />
if echo $line | awk '{print $(NF-1)}' | grep "...:.."<br />
then<br />
# running for > 99min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
if echo $line | awk '{print $(NF-1)}' | grep "....m"<br />
then<br />
# running for > 1000min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
fi<br />
<br />
if echo $line | awk '{print $(NF-3)}' | grep [1-9]...<br />
then<br />
# running for 10-90 percent<br />
if echo $line | awk '{print $NF}' | egrep 'cfusion|counter|vchkpw'<br />
then<br />
<br />
if echo $line | awk '{print $(NF-1)}' | grep "[2-9]:.."<br />
then<br />
# between 2-9min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
elif echo $line | awk '{print $(NF-1)}' | grep "[0-9][0-9]:.."<br />
then<br />
# up to 99min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
fi<br />
fi</pre><br />
<br />
<br />
== Misc Linux Items ==<br />
<br />
We are overselling hard drive space ... when you configure a linux system with a certain amount of disk space (the default is 4gigs) you do not actually use up 4gigs of space on the system. The diskspace setting for a user is simply a cap, and they only use up as much space on the actual disk drive as they are actually using.<br />
<br />
When you create a new linux system, even though there are some 300 RPMs or so installed, if you run `df -k` you will see that the entire 4gig partition is empty - no space is being used. This is because the files in their system are "magic symlinks" to the template for their OS that is in /vz/template - however, any changes to any of those files will "disconnect" them and they will immediately begin using space in their system. Further, any new files uploaded (even if those new files overwrite existing files) will take up space on the partition.<br />
<br />
<br />
if you see this:<br />
<br />
<pre>[root@virt8 root]# vzctl stop 160 ; vzctl start 160<br />
VE is not running<br />
Starting VE ...<br />
VE is unmounted<br />
VE is mounted<br />
Adding IP address(es): 69.55.226.83 69.55.226.84<br />
bash ERROR: Can't change file /etc/sysconfig/network<br />
Deleting IP address(es): 69.55.226.83 69.55.226.84<br />
VE is unmounted<br />
[root@virt8 root]#</pre><br />
<br />
it probably means they no longer have /bin/bash - copy one in for them<br />
<br />
ALSO: another possibility is that they have removed the `ed` RPM from their system - it needs to be reinstalled into their system. But since their system is down, this is tricky ...<br />
<br />
VE startup scripts used by 'vzctl' want package 'ed' to be available inside VE. So if package 'ed' will be enabled in OS template config and OS template itself VE #827 is based on - this error should be fixed.<br />
<br />
yes, it is possible to add RPM to VE while it not running.<br />
Try to do following:<br />
<br />
<pre># cd /vz/template/<OS_template_with_ed_package>/<br />
# vzctl mount 827<br />
# rpm -Uvh --root /vz/root/827 --veid 827 ed-0.2-25.i386.vz.rpm</pre><br />
<br />
Usually theres an error, but its ok<br />
<br />
Note: replace 'ed-0.2-25.i386.vz.rpm' in last command with actual<br />
version of 'ed' package you have.<br />
<br />
----<br />
<br />
So how do I know what template the user has ? cat their conf file and it is listed in there. For example, if the conf file has:<br />
<br />
<pre>[root@virt12 sbin]# vc 1103<br />
…snip…<br />
OSTEMPLATE="debian-3.0/20030822"<br />
TEMPLATES="mod_perl-deb30/20030707 mod_ssl-deb30/20030703 mysql-deb30/20030707 proftpd-deb30/20030703 webmin-deb30/20030823 "<br />
…</pre><br />
<br />
then they are on debian 3.0, all of their system RPMs are in /vz/template/debian-3.0, and they are using version 20030822 of that debian 3.0 template. Also, they’ve also got additional packages installed (mod_perl, mod_ssl, etc). Those are also found under /vz/template<br />
<br />
----<br />
<br />
Edits needed to run java:<br />
<br />
When we first created the VEs, the default setting for privvmpages was 93000:94000 ... which was high enough that most people never had problems ... however, you can;t run java or jdk or tomcat or anything java related with that setting. We have found that by setting privvmpages to 610000:615000 that java runs just fine. That is now the default setting. It is exceedingly rare that anyone needs it higher than that, although we have seen it once or twice.<br />
<br />
Any problems with java at all - the first thing you need to do is see if the failcnt has raised for privvmpages.<br />
<br />
<pre># vzctl start 160<br />
Starting VE ...<br />
vzquota : (error) Quota on syscall for 160: Device or resource busy<br />
Running vzquota on failed for VE 160 [3]</pre><br />
<br />
This is because my pwd is _in_ their private directory - you can't start it until you move out<br />
<br />
People seem to have trouble with php if they are clueless newbies. Here are two common problems/solutions:<br />
<br />
no... but i figured it out myself. problem was the php.ini file that came<br />
vanilla with the account was not configured to work with apache (the<br />
ENGINE directive was set to off).<br />
<br />
everything else seems fine now.<br />
<br />
----<br />
<br />
the problem was in the php.ini file. I noticed that is wasnt showing<br />
the code when it was in an html file so I looked at the php.ini file<br />
and had to change it so it recognized <? tags aswell as <?php tags.<br />
<br />
Also, make sure added to httpd.conf<br />
AddType application/x-httpd-php .php<br />
<br />
----<br />
<br />
You can set the time zone:<br />
<br />
You can change the timezone by doing this:<br />
<br />
ln -sf /usr/share/zoneinfo/<zone> /etc/localtime<br />
<br />
where <zone> is the zone you want in the /usr/share/zoneinfo/ directory.<br />
<br />
----<br />
<br />
Failing shm_open calls:<br />
<br />
first, please check if /dev/shm is mounted inside VE.<br />
'cat /proc/mounts' command should show something like this:<br />
tmpfs /dev/shm tmpfs rw 0 0<br />
<br />
If /dev/shm is not mounted you have 2 ways to solve issue:<br />
1. execute following command inside VE (doesn't require VE reboot):<br />
mount -t tmpfs none /dev/shm<br />
2. add following string to /etc/fstab inside VE and reboot it:<br />
tmpfs /dev/shm tmpfs defaults 0 0<br />
<br />
----<br />
<br />
You can have a mounted but not running ve<br />
Just:<br />
vzctl mount <veid><br />
<br />
----<br />
<br />
When a debian sys can’t get on the network, and you try:<br />
<br />
<pre>vzctl set 1046 --ipadd 69.55.227.117<br />
Adding IP address(es): 69.55.227.117<br />
Failed to bring up lo.<br />
Failed to bring up venet0.</pre><br />
<br />
They probably removed iproute package, which must be the one from swsoft. To restore:<br />
<pre># dpkg -i --veid=1046 --admindir=/vz1/private/1046/root/var/lib/dpkg --instdir=/vz1/private/1046/root/ /vz/template/debian-3.0/iproute_20010824-8_i386.vz.deb<br />
(Reading database ... 16007 files and directories currently installed.)<br />
Preparing to replace iproute 20010824-8 (using .../iproute_20010824-8_i386.vz.deb) ...<br />
Unpacking replacement iproute ...<br />
Setting up iproute (20010824-8) ...</pre><br />
<br />
Then restart their ve<br />
<br />
----<br />
<br />
in a ve i do:<br />
<br />
<pre>cd /<br />
du -h .</pre><br />
<br />
and get: 483M .<br />
<br />
i do:<br />
<br />
<pre>bash-2.05a# df -h<br />
Filesystem Size Used Avail Use% Mounted on<br />
/dev/vzfs 4.0G 2.3G 1.7G 56% /</pre><br />
<br />
how can this be?<br />
<br />
Is it possible that quota file was corrupted somehow? Please try to: <br />
<pre>vzctl stop <VEID><br />
vzquota drop <VEID><br />
vzquota init <VEID><br />
vzctl start <VEID></pre><br />
<br />
----<br />
<br />
How to stop vz from starting after reboot:<br />
<br />
VIRTUOZZO=no <br />
in <br />
/etc/sysconfig/vz<br />
<br />
To start: <br />
service vz start<br />
(after setting VIRTUOZZO=yes in /etc/sysconfig/vz)<br />
<br />
service vz restart will do some kind of 'soft reboot' -- restart all<br />
VPSes and reload modules without rebooting the node<br />
<br />
if you need to shut down all VPSes really really fast, run killall -9 init<br />
<br />
----<br />
<br />
Postfix tip:<br />
<br />
You may want to tweak settings: default_process_limit=10<br />
<br />
= Virtuozzo VPS Management Tools =<br />
<br />
== vm ==<br />
<br />
TODO<br />
<br />
== cancelve ==<br />
cancelve <veid><br />
this will:<br />
* stop a ve<br />
* check for backups (offer to remove them from the backup server <br />
and the backup.config)<br />
* rename the private dir<br />
* check for PTR, provide the commands to reset to default<br />
* and rename the ve’s config<br />
* remind you to remove firewall rules<br />
* remind you to remove DNS entries<br />
<br />
== ipadd ==<br />
ipadd <veid> <ip> [ip] [ip]<br />
add’s ip(s) to a ve<br />
<br />
== ipdel ==<br />
ipdel <veid> <ip> [ip] [ip]<br />
removes ip(s) from a ve<br />
<br />
== vc ==<br />
vc <veid><br />
the equivalent of: <tt>cat /vzconf/<veid>.conf</tt><br />
<br />
== vl ==<br />
vl<br />
will displays a list of ve #’s, 1 per line. (ostensibly to use in a for loop)<br />
<br />
== vp ==<br />
vp <veid><br />
the equivalent of: <tt>vzps auxww –E <veid></tt><br />
<br />
== vpe ==<br />
DEPRECATED<br />
vpe <veid> <br />
this will allow you to do a vp when a ve is running out of control, the equivalent of (deprecated since vp operates outside the VPS): <br />
<pre>vzctl set <veid> --kmemsize 2100000:2200000<br />
vzctl exec <veid> ps auxw<br />
vzctl set <veid> --kmemsize (ve’s orig lvalue):(ve’s orig hvalue)</pre><br />
<br />
== vt ==<br />
vt <veid><br />
the equivalent of: <tt>vztop –E <veid></tt><br />
<br />
== vr ==<br />
vr <veid><br />
the equivalent of: <tt>vzctl stop <veid>; vzctl start <veid></tt><br />
You can run this even if the ve is down - the stop command will just fail<br />
<br />
== vs ==<br />
vs [veid]<br />
displays status (output of <tt>vzctl status <veid></tt>) on each ve configured on the system (as determined by <tt>/vzconf/*.conf</tt>)<br />
If passed an argument, gives the status for just that ve. <br />
A running system looks like:<br />
<tt>VEID 16066 exist mounted running</tt><br />
<br />
A ve which is not running (but does exist) looks like:<br />
<tt>VEID 9990 exist unmounted down</tt><br />
<br />
A ve which is not running and doesn’t exist looks like:<br />
<tt>VEID 421 deleted unmounted down</tt><br />
<br />
== vs2 ==<br />
vs2 [veid]<br />
this is similar to vs in that it displays status (output of <tt>vzctl status <veid></tt>) on each ve,<br />
but the difference is it’s list comes from doing an ls on the data dirs. This was meant to catch <br />
the rare case where a ve configured but exists. <br />
<br />
== vw ==<br />
vw [veid]<br />
displays the output of ‘<tt>w</tt>’ (the equivalent of <tt>vzctl exec <veid> w</tt>) for each configured ve (as determined by <tt>/vzconf/*.conf</tt>). Useful for determing which ve is contributing to a heavily-loaded system.<br />
If passed an argument, gives ‘<tt>w</tt>‘ output for just that ve. <br />
Ex:<br />
<pre>[root@virt2 etc]# vw<br />
134<br />
10:52pm up 79 days, 6:14, 0 users, load average: 0.02, 0.02, 0.00<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT<br />
16027<br />
2:52pm up 7 days, 19:54, 0 users, load average: 0.00, 0.00, 0.00<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT<br />
16055<br />
2:52pm up 79 days, 6:38, 0 users, load average: 0.00, 0.04, 0.07<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT</pre><br />
<br />
== vwe ==<br />
vwe [constraint]<br />
just like <tt>vw</tt>, but takes a constraint as an argument, only show’s ve’s with loads >= the constraint provided. If no constraint is provided, 1 is used by default<br />
<br />
== vzs ==<br />
vzs [veid]<br />
displays the beancounter status for all ve’s, or a particular ve if an argument is passed<br />
<br />
== ve ==<br />
ve <veid><br />
the equivalent of: <tt>vzctl enter <veid></tt><br />
<br />
== vx ==<br />
vx <veid> <command><br />
the equivalent of: <tt>/usr/sbin/vzctl exec <veid> <command></tt><br />
<br />
== dprocs ==<br />
dprocs [count]<br />
a script which outputs a continuous report (or a certain number of reports if an option is passed) of processes stuck in the D state and which VPS’s those procs belong to.<br />
<br />
== setmem ==<br />
setmem <VEID> <256|512|768|1024|2048><br />
adjusts the memory resources for the VE<br />
<br />
== afacheck.sh ==<br />
afacheck.sh<br />
displays the health/status of containers and mirrors on an adaptec card (currently quar1, tempvirt1-2, virt9, virt10)- all other are LSI<br />
<br />
== backup ==<br />
backup<br />
backup script called nightly to update virt scripts and do customer backups<br />
<br />
== checkload.pl ==<br />
checkload.pl<br />
this was intended to be setup as a cronjob to watch processes on a virt when the load <br />
rises above a certain level. Not currently in use.<br />
<br />
== findbackuppigs.pl ==<br />
findbackuppigs.pl<br />
looks for files larger than 50MB which customers have asked us to backup. Emails matches<br />
to linux@johncompanies.com<br />
<br />
== gatherlinux.pl ==<br />
gatherlinux.pl<br />
gathers up data about ve’s configured and writes to a file. Used for audits against the db<br />
<br />
== gb ==<br />
gb <search><br />
greps backup.config for the given search parameter<br />
<br />
== gbg ==<br />
gbg <search><br />
greps backup.config for the given search parameter and presents just the directories (for clean pasting)<br />
<br />
== linuxtrafficgather.pl ==<br />
linuxtrafficgather.pl [yy-mm]<br />
sends a traffic usage summary by ve to support@johncomapnies.com and payments@johncompanies.com.<br />
Optional arguments are year and month (must be in the past). If not passed, assumes last month. Relies on <br />
traffic logs created by netstatreset and netstatbackup<br />
<br />
== linuxtrafficwatch.pl ==<br />
linuxtrafficwatch.pl<br />
checks traffic usage and emails support@johncomapnies.com when a ve reaches the warning level (35G) and<br />
the limit (40G). Works on virtuozzo versions <= 2.6.x. We really aren’t using this anymore now that we have netflow.<br />
<br />
== linuxtrafficwatch2.pl ==<br />
linuxtrafficwatch2.pl<br />
checks traffic usage and emails support@johncomapnies.com when a ve reaches the warning level (35G) and<br />
the limit (40G). Works on virtuozzo version 2.6.x. We really aren’t using this anymore now that we have netflow.<br />
<br />
== load.pl ==<br />
load.pl<br />
feeds info to load mrtg - executed by inetd.<br />
<br />
== mb (linux) ==<br />
mb <mount|umount><br />
(nfs) mounts and umounts dirs to backup2. Shortcuts are mbm and mbu to mount and unmount. <br />
<br />
== migrate ==<br />
migrate <ip of node migrating to> <veid> <target_veid> [target dir: vz | vz1 | vz2]<br />
this is basically a wrapper for <tt>vzmigrate</tt> – vzmigrate is a util to seamlessly move a ve from one host to another. This wrapper was written cause vz virtuozzo version 2.6 had a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables. This script mitigates that. Since it makes multiple ssh connections to the target host, it’s a good idea to put the pub key for the src system in the authorized_keys file on the target host. In addition, it emails ve owners when their migration starts and stops (if they place email addresses in a file on their system: /migrate_notify). To move everyone off a system, you’d do:<br />
for f in `vl`; do migrate <ip> $f; done<br />
<br />
== migrateonline ==<br />
migrateonline <ip of node migrating to> <veid> <target_veid> [target dir: vz | vz1 | vz2]<br />
this is the same as migrate but will migrate a ve in <tt>–online</tt> mode which means it won’t be shut down at the end of the migration. This only works when migrating ve’s between 2 machines running a 2.6 kernel (currently tempvirt1-2. virt16-19, virt12). If you get an error that the machine you’re trying to migrate to has a different CPU or features, etc, then you have to edit the file and add the –f switch to the vzmigrate line- you can basically ignore this kind of warning (but never ignore a warning about missing templates on the destination node). NOTE: This edit (if made to migrateonline) will be overwritten by the base script during each night’s backup.<br />
<br />
== netstatbackup ==<br />
DEPRECATED<br />
netstatbackup <br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.5.x. <br />
<br />
== netstatbackup2 ==<br />
DEPRECATED<br />
netstatbackup2<br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.6.x. <br />
<br />
== netstatreset ==<br />
DEPRECATED<br />
netstatreset<br />
writes traffic count data to a logfile and resets counters to 0. Works on virtuozzo versions 2.5.x <br />
<br />
== netstatreset2 ==<br />
DEPRECATED<br />
netstatreset2<br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.6.x. <br />
<br />
== orphanedbackupwatchlinux ==<br />
orphanedbackupwatchlinux <br />
looks for directories on backup2 which aren’t configured in backup.config and offers to <br />
delete them<br />
<br />
== rsync.backup (linux) ==<br />
rsync.backup<br />
does customer backups (relies on backup.config)<br />
<br />
== startvirt.pl ==<br />
startvirt.pl<br />
forks off start ve commands – keeps 6 running at a time. This is not to be used on systems where fastboot is enabled as it circumvents the benefit of the fastboot. The script will occasionally not exit gracefully and will continue to use up CPU, so it should be watched. Also, don’t exit from the script till you’re sure all ve’s are started – if you do you need to start them manually and may have to free up locks. On some systems, the startvirt script doesn’t exit cleanly and you have to ^C out of it. Be careful though- doing so can leave some VE’s in an odd bootup state and you may need to ‘vr’ them manually. You should check to see which ve’s aren’t running and/or confirm all have started when ^C’ing out of startvirt.<br />
<br />
== taskdone (linux) ==<br />
taskdone<br />
when called will email support@johncompanies.com with the hostname of the machine from which it was <br />
executed as the subject<br />
<br />
== vb (linux) ==<br />
vb<br />
the equivalent of: <tt>vi /usr/local/sbin/backup.config</tt><br />
<br />
== vemakeXX ==<br />
DEPRECATED<br />
vemakerh9 <br />
ve create script for RH9 (see vemake)<br />
<br />
vemakedebian30 <br />
ve create script for debian 3.0 (Woody) (see vemake)<br />
<br />
vemakedebian31 <br />
ve create script for debian 3.1 (Sarge) (see vemake)<br />
<br />
vemakedebian40 <br />
ve create script for debian 4.0 (Etch) (see vemake)<br />
<br />
vemakefedora, vemakefedora2, vemakefedora4, vemakefedora5, vemakefedora6, vemakefedora7<br />
ve create script for fedora core 1, 2, 4, 5, 6, 7 respectively (see vemake)<br />
<br />
vemakecentos3, vemakecentos4<br />
ve create script for centos 3, 4 respectively (see vemake)<br />
<br />
vemakesuse, vemakesuse93, vemakesuse100<br />
ve create script for suse 9.2, 9.3, 10.0 respectively (see vemake)<br />
<br />
vemakeubuntu5, vemakeubuntu606, vemakeubuntu606 vemakeubuntu704<br />
ve create script for ubuntu 5.10, 6.06, 6.10, 7.04 respectively (see vemake)<br />
<br />
== vemove ==<br />
DEPRECATED<br />
vemove <veid> <target_ip> </vz/private/123><br />
this script simplifies the old way of moving ve’s from one system to another - in short moving a ve to or from a virt running virtuozzo < 2.6.x<br />
It’s the equivalent of:<br />
<tt>tar cfpP - <veid> --ignore-failed-read | (ssh -2 -c arcfour <target_ip> "split - -b 1024m </vz/private/123>.tar" )</tt>This should only be used if migrate/vzmigrate can’t be used. <br />
<br />
== vim.watchdog ==<br />
vim.watchdog <br />
looks for and kills procs matching “vi|vim|nano|pine|elm” that are running for a long time & consuming lots of cpu. Works on virtuozzo versions 2.5.x<br />
<br />
== vim.watchdog2 ==<br />
vim.watchdog2<br />
looks for and kills procs matching “vi|vim|nano|pine|elm” that are running for a long time & consuming lots of cpu.<br />
Works on virtuozzo versions 2.6.x.<br />
<br />
== vzmigrate ==<br />
vzmigrate <target_ip> -r no <veid>:[dst veid]:[dst /vzX/private/veid]:[dst /vzX/root/veid]<br />
(this is the raw command “wrapped” by migrate/migrateonline) this will seamlessly move a ve from one host to another. The ve will run for the duration of the migration till the very end when it’s shut down, ip moved and started up on the target system. The filesystem on the src will remain. This should be watched – occasionally the move will timeout and leave the system shut down. If target private and root aren’t specified it just puts it in /vz. Only works when both systems are running virtuozzo 2.6.x<br />
<br />
== vztrafdump.sh ==<br />
DEPRECATED<br />
vztrafdump.sh<br />
writes traffic usage info by ve to a file called jc_traffic_dump in each ve’s / dir. Works on virtuozzo versions <= 2.5.x. <br />
<br />
== vztrafdump2.sh ==<br />
DEPRECATED<br />
vztrafdump2.sh<br />
writes traffic usage info by ve to a file called jc_traffic_dump in each ve’s / dir. Works on virtuozzo versions 2.6.x. <br />
<br />
== addtun ==<br />
addtun <veid><br />
Add’s tun device to ve.<br />
<br />
== bwcap ==<br />
bwcap <veid> <kbps><br />
Ex: <tt>bwcap 1234 512</tt><br />
Caps a VE’s bandwidth to the amount given<br />
<br />
== setdisk ==<br />
setdisk <veid> <diskspace in GB><br />
Ex: <tt>setdisk 1234 5</tt><br />
Gives a VE’s a given amount of disk space<br />
<br />
== vdf ==<br />
vdf <veid> <br />
the equivalent of: <tt>vzctl exec <veid> df –h</tt><br />
<br />
== vdff ==<br />
vdff<br />
runs a (condensed) vdf for all ve’s in your pwd (must be run from /vz/privateN)<br />
<br />
== mvbackups ==<br />
mvbackups <veid> <target_machine> (virt1) <target_dir> (vz1)<br />
moves backups from one location to another on the backup server, and provides you with option to remove entries from current backup.config, and simple paste command to add the config to backup.config on the target server<br />
<br />
== checkquota ==<br />
checkquota<br />
for all the ve’s in the cwd (run from /vz/private, /vz1/private, etc) reports what vz quota says they’re using and what the actual usage is (as reported by du)<br />
<br />
== clearquota ==<br />
clearquota <veid><br />
Recalculates a ve’s quota, prints out the usage before and after. The equivalent of:<br />
<tt>vdf <veid>; v stop <veid>; vzquota drop <veid>; v start <veid>; vdf <veid></tt><br />
<br />
== dprocs ==<br />
dprocs<br />
Sometimes the server’s have a large number of processes get stuck in the D state- this script shows (every 3 secs) which VE’s have D procs, which procs<br />
are stuck and a running average of the top “offenders”<br />
<br />
== vzstat ==<br />
vstat<br />
sort of like top for VZ. sort VEs by CPU usage by pressing 'o' and then 'c' keys<br />
<br />
== stopvirt ==<br />
stopvirt<br />
will stop VEs as fast as it can, 6 at a time. May not exit when complete so you should watch [[#vzstat|vzstat]] in another window.</div>Supporthttps://wiki.jcihosting.com/index.php?title=Virtuozzo_/_Linux_Reference&diff=1020Virtuozzo / Linux Reference2013-03-11T23:10:05Z<p>Support: /* Updating kernel on virt */</p>
<hr />
<div>= Updating VZ =<br />
<br />
Update Virtuozzo to version 4.0.0<br />
Apply minor updates for Virtuozzo 3.0.0<br />
<br />
...<br />
When you get to the last screen, you will be given an option to <br />
<br />
( ) Automatically reboot after update<br />
(*) Reboot later manually<br />
<br />
make sure you choose to reboot later.<br />
<br />
= Migrating Templates =<br />
One nice feature VZ offers is the ability to run a virtual environment (VE or CT as it's now called) based on a particular ditribution on a host which may or may not share the same distribution. For instance, you could run a CT running Ubuntu while the host machine (Hardware Node or HN) is running CentOS. This is accomplished via the use of OS templates. As long as the HN contains the OS template on which a particular CT relies, it will run there. As such, if you want to move a CT from one HN to another, you will need to make sure the OS template exists there.<br />
<br />
Modern versions of VZ (>= 4.x) will check for and automatically move the OS template over to the host as you're (vz)migrating the CT over to a new HN. However we like to make sure the template is in place prior to moving the CT. <br />
<br />
The first step is to see if the template exists on the host. To see OS templates installed:<br />
<br />
2.x (shows OS and application templates):<br />
<pre># vzpkgls<br />
mod_ssl-rh9 20040624<br />
mysql-rh9 20030814 20050412<br />
openwebmail-rh9 20050427<br />
php-rh9 20050506<br />
phpBB-rh9 20041229<br />
postgresql-rh9 20050719<br />
sl-webalizer-rh9 20040119<br />
spamassassin-rh9 20040127<br />
tomcat-rh9 20040524<br />
usermin-rh9 20040116</pre><br />
<br />
>= 3.x (shows just OS templates, run without -O to see application as well):<br />
<pre># vzpkg list -O<br />
ubuntu-10.04-x86 2010-06-01 11:39:05<br />
ubuntu-11.04-x86 2011-06-22 14:23:59<br />
ubuntu-9.10-x86 2010-03-11 17:39:51<br />
centos-5-x86 2010-03-11 17:12:27<br />
debian-5.0-x86 2010-03-11 17:33:34</pre><br />
<br />
<br />
All template files are located:<br />
<br />
<pre># ls /vz/template/<br />
ColdFusion-fc1 jre-fc1 openwebmail-fc1 sl-webalizer-fc1<br />
ColdFusion-fc2 jre-fc2 openwebmail-fc2 sl-webalizer-fc2<br />
ColdFusion-rh9 jre-rh9 openwebmail-rh9 sl-webalizer-rh9<br />
PostNuke-fc1 jre-suse92 openwebmail-suse92 spamassassin-fc1<br />
PostNuke-fc2 jrun php-deb31 spamassassin-fc2<br />
PostNuke-rh9 mailman-deb31 php-fc1 spamassassin-rh9<br />
analog-fc1 mailman-fc1 php-fc2 spamassassin-suse92<br />
analog-fc2 mailman-fc2 php-rh9 suse-9.2<br />
analog-rh9 mailman-rh9 php-suse92 tomcat-fc1<br />
awstats-fc1 mailman-suse92 phpBB-fc1 tomcat-fc2<br />
awstats-rh9 mod_frontpage-fc1 phpBB-fc2 tomcat-rh9<br />
bbClone-fc1 mod_frontpage-fc2 phpBB-rh9 tomcat-suse92<br />
bbClone-fc2 mod_frontpage-rh9 phpmyadmin-deb31 urchin-fc1<br />
bbClone-rh9 mod_frontpage-suse92 postgresql-deb31 usermin-fc1<br />
conf mod_perl-deb31 postgresql-fc1 usermin-fc2<br />
debian-3.1 mod_perl-fc1 postgresql-fc2 usermin-rh9<br />
devel-deb31 mod_perl-fc2 postgresql-rh9 uw-imap-fc1<br />
devel-fc2 mod_perl-rh9 postgresql-suse92 uw-imap-fc2<br />
devel-suse92 mod_perl-suse92 proftpd-deb31 uw-imap-rh9<br />
fedora-core-1 mod_ssl-fc1 proftpd-fc1 vzredhat-7.3<br />
fedora-core-2 mod_ssl-fc2 proftpd-fc2 vzredhat-8.0<br />
jdk-deb31 mod_ssl-rh9 proftpd-rh9 webalizer-suse92<br />
jdk-fc1 mysql-deb31 proftpd-suse92 webmin-fc1<br />
jdk-fc2 mysql-fc1 redhat-7.2 webmin-fc2<br />
jdk-rh9 mysql-fc2 redhat-9 webmin-rh9<br />
jdk-suse92 mysql-rh9 redhat-as3-minimal<br />
jre-deb31 mysql-suse92 redhat-devel-9<br />
</pre><br />
<br />
What we see here are the base OS templates: <tt>redhat-9, fedora-core-1</tt> and supporting application templates: <tt>postgresql-rh9, mailman-rh9, jdk-fc1, mod_ssl-fc1</tt>. So if you wanted to copy over all of redhat 9 you'd need to copy the contents of:<br />
<pre># ls -d /vz/template/*rh9*<br />
/vz/template/ColdFusion-rh9 /vz/template/mod_frontpage-rh9 /vz/template/proftpd-rh9<br />
/vz/template/PostNuke-rh9 /vz/template/mod_perl-rh9 /vz/template/sl-webalizer-rh9<br />
/vz/template/analog-rh9 /vz/template/mod_ssl-rh9 /vz/template/spamassassin-rh9<br />
/vz/template/awstats-rh9 /vz/template/mysql-rh9 /vz/template/tomcat-rh9<br />
/vz/template/bbClone-rh9 /vz/template/openwebmail-rh9 /vz/template/usermin-rh9<br />
/vz/template/jdk-rh9 /vz/template/php-rh9 /vz/template/uw-imap-rh9<br />
/vz/template/jre-rh9 /vz/template/phpBB-rh9 /vz/template/webmin-rh9<br />
/vz/template/mailman-rh9 /vz/template/postgresql-rh9<br />
<br />
# ls -d /vz/template/*redhat-9*<br />
/vz/template/redhat-9<br />
</pre><br />
<br />
To get a template from one HN to another, it simply needs to be rsync'd over to the target HN. It's rsync'd over in this fashion:<br />
<br />
rsync -av -e ssh /vz/template/redhat-9 root@10.1.4.65:/vz/template<br />
rsync -av -e ssh /vz/template/*rh9 root@10.1.4.65:/vz/template<br />
<br />
Newer (>= 3.x) VZ employs "EZ" templates which are organized a little differently:<br />
<br />
<pre># ls /vz/template/<br />
cache CmdTool.log debian redhat-as3-minimal-20080630.tar.gz vc<br />
centos conf redhat-as3-minimal ubuntu<br />
<br />
# ls /vz/template/ubuntu/<br />
10.04 11.04 9.10<br />
# ls /vz/template/ubuntu/10.04/<br />
x86<br />
# ls /vz/template/ubuntu/10.04/x86/<br />
aap_1.091-1_all<br />
aap-doc_1.091-1_all<br />
adduser_3.112ubuntu1_all<br />
anacron_2.3-13.1ubuntu11_i386<br />
apache2_2.2.14-5ubuntu8.2_i386<br />
apache2_2.2.14-5ubuntu8.4_i386<br />
apache2_2.2.14-5ubuntu8.6_i386<br />
apache2_2.2.14-5ubuntu8.7_i386<br />
apache2_2.2.14-5ubuntu8.8_i386<br />
apache2_2.2.14-5ubuntu8.9_i386<br />
<...snip><br />
<br />
# ls /vz/template/cache/<br />
centos-5-x86.tar.gz suse-11.3-x86.tar.gz ubuntu-9.10-x86.tar.gz<br />
debian-5.0-x86.tar.gz ubuntu-10.04-x86.tar.gz<br />
fedora-core-14-x86.tar.gz ubuntu-11.04-x86.tar.gz<br />
</pre><br />
<br />
So what we see is the entire OS and all applications are in 1 directory, and organized by distribution and release version. So to move Ubuntu 10.04:<br />
<br />
rsync -av -e ssh /vz/template/ubuntu/10.04 root@10.1.4.69:/vz/template/ubuntu<br />
<br />
and you also need to move the cache:<br />
<br />
rsync -av -e ssh /vz/template/cache/ubuntu-10.04-x86.tar.gz root@10.1.4.69:/vz/template/cache/<br />
<br />
Once the transfer is complete, you will be able to run <tt>vzpkgls</tt> or <tt>vzpkg list -O</tt> and see the template(s) listed on the target HN.<br />
<br />
So far, this all supposes template doesn't exist on the target- very simple, just copy it over. If the template exists already on the target HN, this requires closer inspection. With older templates, simply moving over the templates would be the end of it- you see the version listed when you do a <tt>vzplgls</tt>:<br />
<br />
mysql-rh9 20030814 20050412<br />
<br />
this means there are 2 versions of the mysql package for RH9: 20030814 and 20050412<br />
As an aside, you can't copy over just 1 of them, but if you rsync'd over the <tt>mysql-rh9</tt> directory, you'd see 20030814 and 20050412 on the target HN. As long as the versions match up, you're good to go.<br />
<br />
With EZ templates, the versions are not as easy to decipher. When you look at the <tt>vzpkg list</tt> output, that simply tells you when a cache was created. A cache is simply a snapshot of all the data needed for the latest versions of the OS and it's applications at the time the cache was created. It is a date, and thus tells you nothing about the versions within. So for instance, if you had a cache for Ubuntu 10.04 from 2007 and you ran <tt>vzpkg create cache ubuntu-10.04-x86</tt> it would re-download the latest version of apache, mysql and so on. And any ''subsequent'' ubuntu 10.04 CT's created thereafter would use those newer versions, whereas older CT's based on 10.04 would use older templates. You can see how this works by looking at the files under:<br />
<br />
<pre># ls /vz/template/ubuntu/10.04/x86/<br />
aap_1.091-1_all<br />
aap-doc_1.091-1_all<br />
adduser_3.112ubuntu1_all<br />
anacron_2.3-13.1ubuntu11_i386<br />
apache2_2.2.14-5ubuntu8.2_i386<br />
apache2_2.2.14-5ubuntu8.4_i386<br />
apache2_2.2.14-5ubuntu8.6_i386<br />
apache2_2.2.14-5ubuntu8.7_i386<br />
apache2_2.2.14-5ubuntu8.8_i386<br />
apache2_2.2.14-5ubuntu8.9_i386<br />
<...snip><br />
</pre><br />
<br />
You can see that this template has in it 6 different versions of apache2. This means that this template was merged with other 10.04 templates and/or it has been cached a few times, each time a new apache was available. The unfortunate thing is it's almost impossible to know which CT's are using which version of apache so it's hard to try to prune this down and remove unwanted/unneeded applications.<br />
<br />
So, if you see another HN has Ubuntu 10.04 on it, you need to see/confirm that it has all the exact files your CT needs. You can run a test rsync to see what ''would'' be transferred (what's missing):<br />
<br />
rsync -avn --stats -e ssh /vz/template/ubuntu/10.04 root@10.1.4.69:/vz/template/ubuntu<br />
<br />
Based on the amount of data you get back from the rsync, you will be able to decide whether or not to remove the -n and allow the full transfer to go through. The downside to doing the transfer is the situation described above- you've permanently joined these templates and now you may have 10 versions of apache on the target HN. There's one other potential pitfall to this kind of merging- it will also potentially copy over shared files, for which there is no particular version, most of which are in <tt>/vz/template/ubuntu/10.04/x86/config</tt>: <br />
<br />
cat /vz/template/ubuntu/10.04/x86/config/os/default/repositories<br />
/vz/template/ubuntu/10.04/x86/config/os/default/repositories<br />
<br />
Here are some specific things to look out for. Case: copying centos-5 on virt19 to virt12. We dumped the rsync output to a file so we could inspect:<br />
<br />
[root@virt19 /vz/template]# rsync -an -e ssh /vz/template/centos/ root@10.1.4.62:/vz/template/centos/ > v12_c5<br />
<br />
(note, that can be dangerous, better to add on full path <tt>/vz/template/centos/5</tt>)<br />
<br />
So in the output file we see things like:<br />
<br />
<pre><br />
x86/base0/cachecookie<br />
x86/base0/primary.xml.gz <br />
x86/base0/primary.xml.gz.sqlite <br />
x86/base0/repomd.xml <br />
x86/base1/cachecookie <br />
x86/base1/filelists.xml.gz <br />
x86/base1/filelists.xml.gz.sqlite <br />
x86/base1/primary.xml.gz <br />
x86/base1/primary.xml.gz.sqlite <br />
x86/base1/repomd.xml<br />
x86/base2/cachecookie<br />
</pre><br />
<br />
Which make us nervous cause those are not versioned files, i.e.:<br />
5/x86/MAKEDEV-3.23-1.2.i386/usr/sbin/mksock<br />
<br />
So those files will replace existing files on v12 rather than the case of the latter where it's likely being added. So caution should be given and deference to whichever version is newer (which rsync should take care of, but that will depend on the date each file was created and perhaps not the actual data within).<br />
<br />
If we look further in the file we see:<br />
<br />
x86/psa-appvault-moodle-1.8-8202920080409011254.noarch/usr/local/psa/var/cgitory/moodle-1.<br />
9/htdocs/lib/editor/htmlarea/plugins/TableOperations/img/cell-split.gif<br />
<br />
and other files looking like:<br />
5/x86/plesk-base-9.0.1-cos5.build90090127.18.i586/etc/sw/keys/info<br />
<br />
Which means plesk is here. We don't need plesk on v12 (the CT moving over doesn't use it) so we rerun the rsync and exclude it:<br />
<br />
[root@virt19 /vz/template]# rsync -an -e ssh --exclude="plesk*" --exclude="psa*" /vz/template/centos/ root@10.1.4.62:/vz/template/centos/ > v12_c5<br />
<br />
and now it's much smaller:<br />
<br />
[root@virt19 /vz/template]# cat v12_c5|wc -l<br />
97104<br />
<br />
Before running with excludes it was:<br />
[root@virt19 /vz/template]# cat v12_c5|wc -l <br />
238238<br />
<br />
So again, look at what you're doing. Generally, combining templates is fine however.<br />
<br />
If you're happy with the merge, run the rsync again without the "-n" to let it actually do the transfer.<br />
<br />
Once done, don't forget to add the new template to the HN in Management -> Reference -> Templates<br />
<br />
= getting help =<br />
<br />
= vzup2date =<br />
TODO <br />
= adding new templates =<br />
TODO<br />
<br />
= Updating kernel on virt =<br />
<br />
We now use [[#vzup2date|vzup2date]] to update systems and kernels. What follows is what we used to do when we had to download kernels and install manually on old systems:<br />
<br />
<br />
1. install kernel and modules<br />
<pre># rpm -ivh vzkernel-enterprise-2.4.20-021stab028.12.777.i686.rpm \<br />
vzmodules-enterprise-2.4.20-021stab028.12.swsoft.i686.rpm<br />
vzkernel-smp ##################################################<br />
vzmodules-smp ##################################################</pre><br />
<br />
DO NOT USE the "rpm -Uhv" command to install the kernel, otherwise all the previously installed kernels may be removed from your system. <br />
<br />
2. update lilo/grub<br />
<br />
Lilo:<br />
<br />
vi /etc/lilo.conf<br />
<br />
rename old Virtuozzo kernel label from "linux+virtuozzo" to "virtuozzo_old"<br />
and add a new entry pointing to the newly installed virtuozzo kernel image.<br />
For example:<br />
<br />
<pre>prompt<br />
timeout=50<br />
default=linux+virtuozzo<br />
append="debug panic=5 console=tty console=ttyS0,38400"<br />
boot=/dev/sda<br />
map=/boot/map<br />
install=/boot/boot.b<br />
message=/boot/message<br />
linear<br />
<br />
image=/boot/vmlinuz-2.4.18-3smp<br />
label=linux<br />
initrd=/boot/initrd-2.4.18-3smp.img<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.18-3<br />
label=linux-up<br />
initrd=/boot/initrd-2.4.18-3.img<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-020stab009.5.777-enterprise<br />
label=2.4.20-9.5.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-020stab009.8.777-enterprise<br />
label=2.4.20-9.8.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-020stab009.21.777-enterprise<br />
label=2.4.20-9.21.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-020stab009.24.777-enterprise<br />
label=2.4.20-9.24.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-021stab028.3.777-enterprise<br />
label=2.4.20-28.3.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-021stab028.5.777-enterprise<br />
label=old_linux+vz<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-021stab028.12.777-enterprise<br />
label=linux+virtuozzo<br />
read-only<br />
root=/dev/sda1</pre><br />
<br />
With this configuration, the new kernel is the default kernel to use at boot. The original kernel entry has been retained with the label "old_linux+vz ".<br />
<br />
Grub:<br />
<br />
vi /boot/grub /menu.lst<br />
<br />
Add new entry at the top.<br />
For example:<br />
<br />
<pre>serial --unit=0 --speed=38400<br />
terminal --timeout=10 serial console<br />
default=0<br />
timeout=10<br />
splashimage=(hd0,0)/boot/grub/splash.xpm.gz<br />
title Virtuozzo 2.6.1 (2.4.20-021stab028.12.777-enterprise)<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-22.4.20-021stab028.12.777-enterprise root=/dev/sda1 console=tty0 \ console=ttyS0,38400<br />
title Virtuozzo 2.6.1 (2.4.20-021stab028.3.777-enterprise)<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.20-021stab028.3.777-enterprise root=/dev/sda1 console=tty0 \ console=ttyS0,38400<br />
title Red Hat Linux (2.4.18-3smp)<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.18-3smp ro root=/dev/sda1<br />
initrd /boot/initrd-2.4.18-3smp.img<br />
title Red Hat Linux-up (2.4.18-3)<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.18-3 ro root=/dev/sda1<br />
initrd /boot/initrd-2.4.18-3.img<br />
title Linux with Virtuozzo 2.5<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.20-020stab009.3.777-smp ro root=/dev/sda1<br />
title vz-enterpriseo<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.20-020stab009.21.777-enterprise ro root=/dev/sda1<br />
title vz-enterprise<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.20-020stab009.24.777-enterprise ro root=/dev/sda1 \ console=tty0 console=ttyS0,38400</pre><br />
<br />
3. run the lilo (if using lilo)<br />
<br />
<pre># lilo<br />
Added linux<br />
Added linux-up<br />
Added virtuozzo_old<br />
Added linux+virtuozzo *</pre><br />
<br />
4. reboot the server<br />
shutdown -r 23:05<br />
<br />
when it comes time for the shutdown, run [[VPS_Management#stopvirt|stopvirt]] to get things shut down faster<br />
<br />
<br />
= Remaking service VE (<=3.x only) =<br />
<br />
Occasionally the control panel for VEs (aka VZPP) will stop working and you just need to reinstall the service VE (which runs the control panels for all VEs on the HN). Here's how that's done :<br />
<br />
<pre>vzctl stop 1<br />
ipdel 1 69.55.234.152<br />
vzmlocal 1:2<br />
vzctl set 2 --save --onboot no<br />
vzsveinstall -t redhat-as3-minimal -d /root/Rel261/HW/RPMS -s 69.55.234.152 --skip-client</pre><br />
<br />
on 3.0:<br />
<br />
<pre>vzsveinstall -t redhat-as3-minimal -d /vz/Rel300/HW/RPMS -s 69.55.229.151 --skip-client<br />
<br />
vzctl stop 1<br />
vzctl mount 1<br />
vzctl mount 2<br />
/bin/cp -aRf /vz/root/2/home/vzagent0 /vz/root/1/home<br />
/bin/cp -aRf /vz/root/2/var/lib/mysql/VZADB /vz/root/1/var/lib/mysql<br />
/bin/cp -af /vz/root/2/etc/shadow /vz/root/1/etc/<br />
/bin/cp -aRf /vz/root/2/var/vzcp/static/vz/skins/ /vz/root/1/var/vzcp/static/vz<br />
/bin/cp -af /vz/root/2/etc/vzcp/pp/menu.xml /vz/root/1/etc/vzcp/pp/<br />
/bin/cp -af /vz/root/2/etc/vzcp/pp/dashboard.xml /vz/root/1/etc/vzcp/pp/<br />
<br />
vzctl umount 2<br />
vzctl umount 1<br />
vzctl start 1</pre></div>Supporthttps://wiki.jcihosting.com/index.php?title=VPS_Management&diff=1019VPS Management2013-03-11T23:08:10Z<p>Support: /* vzstat */</p>
<hr />
<div>= Common Problems =<br />
== Login to any machine without a password ==<br />
<br />
This is possible via the use of ssh keys. The process is thus:<br />
<br />
1. place the public key for your user (root@mail) in the /root/.ssh/authorized_keys file on the server you wish to login to<br />
cat /root/.ssh/id_dsa.pub<br />
(paste that into authorized_keys on the target server). If the file doesn't exist, create it.<br />
<br />
2. enable root login (usually only applies to FreeBSD). Edit the /etc/ssh/sshd_config on the target server and change:<br />
<tt>#PermitRootLogin no</tt><br />
to<br />
<tt>PermitRootLogin yes</tt><br />
<br />
3. Restart the sshd on the target machine. First, find the sshd process: <br />
jailps <hostname> | grep sshd <br />
or <br />
vp <VEID> | grep sshd<br />
<br />
Look for the process resembling:<br />
root 17296 0.0 0.0 5280 1036 ? Ss 2011 4:27 /usr/sbin/sshd <br />
(this is the sshd)<br />
<br />
Not:<br />
root 6270 0.5 0.0 6808 2536 ? Ss 14:33 0:00 sshd: root [priv]<br />
(this is an sshd child- someone already ssh'd in as root)<br />
<br />
Restart the sshd: <br />
kill -1 <PID><br />
<br />
Ex:<br />
kill -1 17296<br />
<br />
You may now ssh in.<br />
<br />
Once you're done, IF you enabled root login, you should repeat steps 2 and 3 to disable root logins.<br />
<br />
== Letting someone in who has locked themselves out (killed sshd, lost pwd) ==<br />
<br />
There are two ways people frequently lock themselves out - either they forget a password, or they kill off sshd somehow.<br />
<br />
These are actually both fairly easy to solve. First, let's say someone kills off their sshd, or somehow mangles /etc/ssh/sshd_config such that it no longer lets them in.<br />
<br />
Their email may be very short, or it may have all sorts of details about how you should fix sshd_config to let them in ... just ignore all of this. They can fix their own mangled sshd. Fixing this is very simple. First, edit the /etc/inetd.conf on their system and uncomment the telnet line:<br />
<br />
telnet stream tcp nowait root /usr/libexec/telnetd telnetd<br />
#telnet stream tcp6 nowait root /usr/libexec/telnetd telnetd<br />
<br />
(just leave the tcp6 version of telnet commented)<br />
<br />
Then, use jailps to list the processes on their system, and find their inetd process. Then simply:<br />
<br />
kill -HUP (pid)<br />
<br />
where (pid) is the PID of their inetd process. Now they have telnet running on their system and they can log in and do whatever they need to do.<br />
<br />
The only complications that could occur are:<br />
<br />
a) their firewall config on our firewall has port 23 blocked, in which case you will need to open that - will be covered in a different lesson.<br />
<br />
b) they are not running inetd, so you can't HUP it. If this happens, edit their /etc/rc.conf, add the inetd_enable="YES" line, and then kill<br />
their jail with /tmp/jailkill.pl - then restart their jail with the jail line from their quad/safe file. Easy.<br />
<br />
If they have forgotten a password,<br />
<br />
On 6.x+ you can reset their password with:<br />
jexec <jailID from jls> passwd root<br />
<br />
Note: the default password for 6.x jails is 8ico2987, for 4.x it is p455agfa<br />
<br />
On 4.x, you need to cd to their etc directory<br />
... for instance:<br />
<br />
cd /mnt/data2/198.78.65.136-col00261-DIR/etc<br />
<br />
and run:<br />
<br />
vipw -d .<br />
<br />
Then paste in these two lines (theres a paste with these):<br />
<br />
root:$1$krszPxhk$xkCepSnz3mIikT3vCtJCt0:0:0::0:0:Charlie &:/root:/bin/csh<br />
user:$1$Mx9p5Npk$QdMU6c8YQqp2FW2M3irEh/:1001:1001::0:0:User &:/home/user:/bin/sh<br />
<br />
overwriting the lines they already have for "user" and "root" - then just tell them that both user and root have been reset to the default password of p455agfa.<br />
<br />
For linux, just passwd inside shell or <br />
vzctl set <veid> --userpasswd root:p455agfa –save<br />
<br />
Starting in 2009 we began giving out randomized passwords for FreeBSD and Linux as the default password. That is stored with each system in Mgmt. You should look for and reset the password to that password in the event of a reset and refer the customer to use their original password from their welcome email- this way we don’t have to send the password again via email (in clear text).<br />
<br />
<br />
== sendmail can’t be contacted from ext ip (only locally) ==<br />
<br />
By default redhat puts this line in sendmail.mc:<br />
<br />
DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')<br />
<br />
which makes it only answer on localhost. Comment it out like:<br />
<br />
dnl DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')<br />
<br />
and then rebuild sendmail.cf with:<br />
<br />
m4 /etc/mail/sendmail.mc > /etc/sendmail.cf<br />
<br />
== virt doesn’t properly let go of ve’s ip(s) when moved to another system ==<br />
<br />
On virtuozzo 2.6 systems, it's been observed that when moving ips from one virt to another that sometimes the routing table will not get updated to reflect the removal of the ip addresses.<br />
<br />
A recent example was a customer that was moving to a new ve on a new virt and the ip addresses were traded between the two ve's. After the trade the two systems were not able to talk to each other. When looking at the routing table for the old system all the ip addresses were still in the routing table as being local, like so:<br />
<br />
<pre>netstat -rn | grep 69.55.225.149<br />
69.55.225.149 0.0.0.0 255.255.255.255 UH 40 0 0 venet0</pre><br />
<br />
This was preventing traffic to the other system from being routed properly.<br />
The solution is to manually delete the route:<br />
<br />
route delete 69.55.225.149 gw 0.0.0.0<br />
<br />
Supposedly, this was fixed in 2.6.1<br />
<br />
== sshd on FreeBSD 6.2 segfaults ==<br />
<br />
First try to reinstall ssh<br />
<br />
<pre>cd /usr/src/secure<br />
cd lib/libssh<br />
make depend && make all install<br />
cd ../../usr.sbin/sshd<br />
make depend && make all install<br />
cd ../../usr.bin/ssh<br />
make depend && make all install</pre><br />
<br />
Failing that, find the library that’s messed up:<br />
<br />
<pre>ldd /usr/sbin/sshd<br />
libssh.so.3 => /usr/lib/libssh.so.3 (0x280a3000) <br />
libutil.so.5 => /lib/libutil.so.5 (0x280d8000) <br />
libz.so.3 => /lib/libz.so.3 (0x280e4000) <br />
libwrap.so.4 => /usr/lib/libwrap.so.4 (0x280f5000) <br />
libpam.so.3 => /usr/lib/libpam.so.3 (0x280fc000) <br />
libbsm.so.1 => /usr/lib/libbsm.so.1 (0x28103000) <br />
libgssapi.so.8 => /usr/lib/libgssapi.so.8 (0x28112000) <br />
libkrb5.so.8 => /usr/lib/libkrb5.so.8 (0x28120000) <br />
libasn1.so.8 => /usr/lib/libasn1.so.8 (0x28154000) <br />
libcom_err.so.3 => /usr/lib/libcom_err.so.3 (0x28175000) <br />
libroken.so.8 => /usr/lib/libroken.so.8 (0x28177000) <br />
libcrypto.so.4 => /lib/libcrypto.so.4 (0x28183000) <br />
libcrypt.so.3 => /lib/libcrypt.so.3 (0x28276000) <br />
libc.so.6 => /lib/libc.so.6 (0x2828e000) <br />
libmd.so.3 => /lib/libmd.so.3 (0x28373000)</pre><br />
<br />
md5 them and compare to other jail hosts or jails running on host<br />
<br />
for libcrypto reinstall:<br />
<pre>/usr/src/crypto<br />
make depend && make all install</pre><br />
<br />
= FreeBSD VPS =<br />
<br />
== Starting jails: Quad/Safe Files ==<br />
<br />
FreeBSD customer systems do not start up automatically at boot time. When one of our freebsd machines boots up, it boots up, and does nothing else. To start jails, we put the commands to start each jail into a shell script(s) and run the script(s). Jail startup is something that needs to be actively monitored, which is why we don’t just run the script automatically. More on monitoring later.<br />
<br />
NOTE: >=7.x we have moved to 1 quad file: <tt>quad1</tt>. Startups are not done by running each quad, but rather [[#startalljails|startalljails]] which relies on the contents of <tt>quad1</tt>. The specifics of this are lower in this article. What follows here applies for pre 7.x systems.<br />
<br />
There are eight files in <tt>/usr/local/jail/rc.d</tt>:<br />
<br />
<pre>jail3# ls /usr/local/jail/rc.d/<br />
quad1 quad2 quad3 quad4 safe1 safe2 safe3 safe4<br />
jail3#</pre><br />
<br />
four quad files and four safe files.<br />
<br />
Each file contains an even number of system startup blocks (total number of jails divided by 4)<br />
<br />
The reason for this is, if we make one large script to startup all the systems at boot time, it will take too long - the first system in the script will start up right after system boot, which is great, but the last system may not start for another 20 minutes.<br />
<br />
Since there is no way to parralelize this during the startup procedure, we simply open four terminals (in screen window 9) and run each script, one in each terminal. This way they all run simultaneously, and the very last system in each startup script gets started in 1/4th the time it would if there was one large file<br />
<br />
The files are generally organized so that quad/safe 1&2 have only jails from disk 1, and quad/safe 3&4 have jails from disk 2. This helps ensure that only 2 fscks on any disk are going on at once. Further, they are balanced so that all quad/safe’s finish executing around the same time. We do this by making sure each quad/safe has a similar number of jails and represents a similar number of inodes (see js).<br />
<br />
The other, very important reason we do it this way, and this is the reason there are quad files and safe files, is that in the event of a system crash, every single vn-backed filesystem that was mounted at the time of system crash needs to be fsck'd. However, fsck'ing takes time, so if we shut the system down gracefully, we don't want to fsck.<br />
<br />
Therefore, we have two sets of scripts - the four quad scripts are identical to the four safe scripts except for the fact that the quad scripts contain fsck commands for each filesystem.<br />
<br />
So, if you shut a system down gracefully, start four terminals and run safe1 in window one, and safe2 in window 2, and so on.<br />
<br />
If you crash, start four terminals (or go to screen window 9) and run quad1 in window one, and quad2 in window 2, and so on.<br />
<br />
Here is a snip of (a 4.x version) quad2 from jail17:<br />
<br />
<pre>vnconfig /dev/vn16 /mnt/data2/69.55.228.7-col00820<br />
fsck -y /dev/vn16<br />
mount /dev/vn16c /mnt/data2/69.55.228.7-col00820-DIR<br />
chmod 0666 /mnt/data2/69.55.228.7-col00820-DIR/dev/null<br />
jail /mnt/data2/69.55.228.7-col00820-DIR mail1.phimail.com 69.55.228.7 /bin/sh /etc/rc<br />
<br />
# moved to data2 col00368<br />
#vnconfig /dev/vn28 /mnt/data2/69.55.236.132-col00368<br />
#fsck -y /dev/vn28<br />
#mount /dev/vn28c /mnt/data2/69.55.236.132-col00368-DIR<br />
#chmod 0666 /mnt/data2/69.55.236.132-col00368-DIR/dev/null<br />
#jail /mnt/data2/69.55.236.132-col00368-DIR limehouse.org 69.55.236.132 /bin/sh /etc/rc<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
vnconfig /dev/vn22 /mnt/data2/69.55.228.13-col01063<br />
fsck -y /dev/vn22<br />
mount /dev/vn22c /mnt/data2/69.55.228.13-col01063-DIR<br />
chmod 0666 /mnt/data2/69.55.228.13-col01063-DIR/dev/null<br />
jail /mnt/data2/69.55.228.13-col01063-DIR www.widestream.com.au 69.55.228.13 /bin/sh /etc/rc<br />
<br />
# cancelled col00106<br />
#vnconfig /dev/vn15 /mnt/data2/69.55.238.5-col00106<br />
#fsck -y /dev/vn15<br />
#mount /dev/vn15c /mnt/data2/69.55.238.5-col00106-DIR<br />
#chmod 0666 /mnt/data2/69.55.238.5-col00106-DIR/dev/null<br />
#jail /mnt/data2/69.55.238.5-col00106-DIR mail.azebu.net 69.55.238.5 /bin/sh /etc/rc</pre><br />
<br />
As you can see, two of the systems specified are commented out - presumably those customers cancelled, or were moved to new servers.<br />
<br />
As you can see, the vnconfig line is the simpler command line, not the longer one that was used when it was first configured. As you can see, all that is done is, vnconfig the filesystem, then fsck it, then mount it. The fourth command is the `jail` command used to start the system – but that will be covered later.<br />
<br />
Here is the safe2 file from jail17:<br />
<br />
<pre>vnconfig /dev/vn16 /mnt/data2/69.55.228.7-col00820<br />
mount /dev/vn16c /mnt/data2/69.55.228.7-col00820-DIR<br />
chmod 0666 /mnt/data2/69.55.228.7-col00820-DIR/dev/null<br />
jail /mnt/data2/69.55.228.7-col00820-DIR mail1.phimail.com 69.55.228.7 /bin/sh /etc/rc<br />
<br />
# moved to data2 col00368<br />
#vnconfig /dev/vn28 /mnt/data2/69.55.236.132-col00368<br />
#mount /dev/vn28c /mnt/data2/69.55.236.132-col00368-DIR<br />
#chmod 0666 /mnt/data2/69.55.236.132-col00368-DIR/dev/null<br />
#jail /mnt/data2/69.55.236.132-col00368-DIR limehouse.org 69.55.236.132 /bin/sh /etc/rc<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
vnconfig /dev/vn22 /mnt/data2/69.55.228.13-col01063<br />
mount /dev/vn22c /mnt/data2/69.55.228.13-col01063-DIR<br />
chmod 0666 /mnt/data2/69.55.228.13-col01063-DIR/dev/null<br />
jail /mnt/data2/69.55.228.13-col01063-DIR www.widestream.com.au 69.55.228.13 /bin/sh /etc/rc<br />
<br />
# cancelled col00106<br />
#vnconfig /dev/vn15 /mnt/data2/69.55.238.5-col00106<br />
#mount /dev/vn15c /mnt/data2/69.55.238.5-col00106-DIR<br />
#chmod 0666 /mnt/data2/69.55.238.5-col00106-DIR/dev/null<br />
#jail /mnt/data2/69.55.238.5-col00106-DIR mail.azebu.net 69.55.238.5 /bin/sh /etc/rc</pre><br />
<br />
As you can see, it is exactly the same, but it does not have the fsck lines.<br />
<br />
Take a look at the last entry - note that the file is named:<br />
<br />
/mnt/data2/69.55.238.5-col00106<br />
<br />
and the mount point is named:<br />
<br />
/mnt/data2/69.55.238.5-col00106-DIR<br />
<br />
This is the general format on all the FreeBSD systems. The file is always named:<br />
<br />
IP-custnumber<br />
<br />
and the directory is named:<br />
<br />
IP-custnumber-DIR<br />
<br />
If you run safe when you need a fsck, the mount will fail and jail will fail:<br />
<br />
# mount /dev/vn1c /mnt/data2/jails/65.248.2.131-ns1.kozubik.com-DIR<br />
mount: /dev/vn1c: Operation not permitted<br />
<br />
No reboot needed, just run the quad script<br />
<br />
Starting with 6.x jails, we added block delimiters to the quad/safe files, the block looks like:<br />
<br />
<pre>echo '## begin ##: nuie.solaris.mu'<br />
fsck -y /dev/concat/v30v31a<br />
mount /dev/concat/v30v31a /mnt/data1/69.55.228.218-col01441-DIR<br />
mount_devfs devfs /mnt/data1/69.55.228.218-col01441-DIR/dev<br />
devfs -m /mnt/data1/69.55.228.218-col01441-DIR/dev rule -s 3 applyset<br />
jail /mnt/data1/69.55.228.218-col01441-DIR nuie.solaris.mu 69.55.228.218 /bin/sh /etc/rc<br />
echo '## end ##: nuie.solaris.mu'</pre><br />
<br />
These are more than just informative when running quad/safe’s, the echo lines MUST be present for certain tools to work properly. So it’s important that any updates to the hostname also be updated on the 2 echo lines. For example, if you try to startjail a jail with a hostname which is on the jail line but not the echo lines, the command will return with host not found.<br />
<br />
=== FreeBSD 7.x+ notes ===<br />
<br />
Starting with the release of FreeBSD 7.x, we are doing jail startups in a slightly different way. First, thereis only 1 file: <tt>/usr/local/jail/rc.d/quad1</tt><br />
There are no other quads or corresponding safe files. The reason for this is twofold, 1. We can pass –C to fsck which will tell is to skip the fsck if the fs is clean (no more need for safe files), 2. We have a new startup script which can be launched multiple times, running in parallel to start jails, where quad1 is the master jail file. <br />
Quad1 could still be run as a shell script, but it would take a very long time for it to run completely so it’s not advisable; or you should break it down into smaller chunks (like quad1, quad2, quad3, etc)<br />
<br />
Here is a snip of (a 7.x version) quad1 from jail2:<br />
<br />
<pre>echo '## begin ##: projects.tw.com'<br />
mdconfig -a -t vnode -f /mnt/data1/69.55.230.46-col01213 -u 50<br />
fsck -Cy /dev/md50c<br />
mount /dev/md50c /mnt/data1/69.55.230.46-col01213-DIR<br />
mount -t devfs devfs /mnt/data1/69.55.230.46-col01213-DIR/dev<br />
devfs -m /mnt/data1/69.55.230.46-col01213-DIR/dev rule -s 3 applyset<br />
jail /mnt/data1/69.55.230.46-col01213-DIR projects.tw.com 69.55.230.46 /bin/sh /etc/rc<br />
echo '## end ##: projects.tw.com'</pre><br />
<br />
Cancelled jails are no longer commented out and stored in quad1, rather they’re moved to <tt>/usr/local/jail/rc.d/deprecated</tt> <br />
<br />
To start these jails, start the 4 ssh sessions as you would for a normal crash and then instead of running quad1-4, instead run startalljails in each window. IMPORTANT- before running startalljails you should make sure you ran preboot once as it will clear out all the lockfiles and enable startalljails to work properly.<br />
<br />
== Problems with the quad/safe files ==<br />
<br />
When you run the quad/safe files, there are two problems that can occur - either a particular system will hang during initialization, OR a system will spit out output to the screen, impeding your ability to do anything. Or both.<br />
<br />
First off, when you start a jail, you see output like this:<br />
<br />
<pre>Skipping disk checks ...<br />
adjkerntz[25285]: sysctl(put_wallclock): Operation not permitted<br />
Doing initial network setup:.<br />
ifconfig: ioctl (SIOCDIFADDR): permission denied<br />
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384<br />
Additional routing options: TCP keepalive=YESsysctl:<br />
net.inet.tcp.always_keepalive: Operation not permitted.<br />
Routing daemons:.<br />
Additional daemons: syslogd.<br />
Doing additional network setup:.<br />
Starting final network daemons:.<br />
ELF ldconfig path: /usr/lib /usr/lib/compat /usr/X11R6/lib /usr/local/lib<br />
a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout /usr/X11R6/lib/aout<br />
Starting standard daemons: inetd cron sshd sendmail sendmail-clientmqueue.<br />
Initial rc.i386 initialization:.<br />
Configuring syscons: blanktime.<br />
Additional ABI support:.<br />
Local package initialization:.<br />
Additional TCP options:.</pre><br />
<br />
Now, let's look at this line, near the end:<br />
<br />
Local package initialization:.<br />
<br />
This is where a list of daemons that are set to start at boot time willshow up. You might see something like:<br />
<br />
Local package initialization: mysqld apache sendmail sendmail-clientmqueue<br />
<br />
Or something like this:<br />
<br />
Local package initialization: postgres postfix apache<br />
<br />
The problem is that many systems (about 4-5 per machine) will hang on that line. Basically it will get to some of the way through the total daemons to be started:<br />
<br />
Local package initialization: mysqld apache<br />
<br />
and will just sit there. Forever.<br />
<br />
Fortunately, pressing ctrl-c will break out of it. Not only will it break out of it, but it will also continue on that same line and start the other daemons:<br />
<br />
Local package initialization: mysqld apache ^c sendmail-clientmqueue<br />
<br />
and then continue on to finish the startup, and then move to the next system to be started.<br />
<br />
So what does this mean? It means that if a machine crashes, and you start four screen-windows to run four quads or four safes, you need to periodically cycle between them and see if any systems are stuck at that point, causing their quad/safe file to hang. A good rule of thumb is, if you see a system at that point in the startup, give it another 100 seconds - if it is still at the exact same spot, hit ctrl-c. Its also a good idea to go back into the quad file (just before the first command in the jail startup block) and note that this jail tends to need a control-c or more time as follows:<br />
<pre>echo '### NOTE ### slow sendmail'<br />
echo '### NOTE ###: ^C @ Starting sendmail.'</pre><br />
<br />
'''NEVER''' hit ctrl-c repeatedly if you don't get an immediate response - that will cause the following jail’s startup commands to be aborted.<br />
<br />
A second problem that can occur is that a jail - maybe the first one in that particular quad/safe, maybe the last one, or maybe one in the middle, will start spitting out status or error messages from one of its init scripts. This is not a problem - basically, hit enter a few times and see if you get a prompt - if you do get a prompt, that means that the quad/safe script has already completed. Therefore it is safe to log out (and log out of the user that you su'd from) and then log back in (if necessary).<br />
<br />
The tricky thing is, if a system in the middle starts flooding with messages, and you hit enter a few times and don't get a prompt. Are you not getting a prompt because some subsequent system is hanging at the initialization, as we discussede above ? Or are you not getting a prompt because that quad file is currently running an fsck ? Usually you can tell by scrolling back in screen’s history to see what it was doing before you started getting the messages.<br />
<br />
If you don’t get clues from history, you have to use your judgement - instead of giving it 100 seconds to respond, perhaps give it 2-3 mins ... if you still get no response (no prompt) when you hit enter, hit ctrl-c. However, be aware that you might still be hitting ctrl-c in the middle of an fsck. This means you will get an error like "filesystem still marked dirty" and then the vnconfig for it will fail and so will the jail command, and the next system in the quad file will then start starting up.<br />
<br />
If this happens, just wait until the end of all the quad files have finished, and start that system manually.<br />
<br />
If things really get weird, like a screen flooded with errors, and you can't get a prompt, and ctrl-c does nothing, then you need to just eventually (give it ten mins or so) just kill that window with ctrl-p, then k, and then log in again and manually check which systems are now running and which aren't, and manually start up any that are not.<br />
<br />
Don't EVER risk running a particular quad/safe file a second time.<br />
If the quad/safe script gets executed twice, reboot the machine immediately.<br />
<br />
So, for all the above reasons, anytime a machine crashes and you run all the quads or all the safes, '''always''' check every jail afterwards to make sure it is running - even if you have no hangs or complications at all.<br />
Run this command:<br />
<br />
<tt>[[#jailpsall|jailpsall]]</tt> <br />
<br />
Note: [[#postboot|postboot]] also populates ipfw counts, so it '''should not be run multiple times''', use <tt>jailpsall</tt> for subsequent extensive ps’ing<br />
<br />
And make sure they all show as running. If one does not show as running, check its /etc/rc.conf file first to see if maybe it is using a different hostname first before starting it manually.<br />
<br />
One thing we have implemented to alleviate these startup hangs and noisy jails, is to put jail start blocks that are slow or hangy at the bottom of the safe/quad file. Further, for each bad jail we note in each quad/safe just before the start block something like:<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
<br />
That way we’ll be prepared to ^C when we see that message appear during the quad/safe startup process. If you observe a new, undocumented hang, '''after''' the quad/safe has finished, place a line similar to the above in the quad file, move the jail start block to the end of the file, then run [[#buildsafe|buildsafe]]<br />
<br />
<br />
== Recovering from a crash (FreeBSD) ==<br />
<br />
=== Diagnose whether you have a crash ===<br />
The most important thing is to get the machine and all jails back up as soon as possible. Note the time, you’ll need to create a crash log entry (Mgmt. -> Reference -> CrashLog). The first thing to do is head over to the [[Screen#Screen_Organization|serial console screen]] and see if there’s any kernel error messages output. Try to copy any messages (or just a sample of repeating messages) you see into the notes section of the crash log. If there are no messages, the machine may just be really busy- wait a bit (5-10min) to see if it comes back. If it's still pinging, odds are its very busy. Note, if you see messages about swap space exhausted, the server is obviously out of memory, however it may recover briefly enough for you to get a jtop in to see who's lauched a ton of procs (most likely) and then issue a quick jailkill to get it back under control.<br />
<br />
If it doesn't come back, or the messages indicate a fatal error, you will need to proceed with a power cycle (ctrl+alt+del will not work).<br />
<br />
=== Power cycle the server ===<br />
If this machine is not a Dell 2950 with a [[DRAC/RMM#DRAC|DRAC card]] (i.e. if you can’t ssh into the DRAC card (as root, using the standard root pass) and issue <br />
racadm serveraction hardreset<br />
then you will need someone at the data center power the macine off, wait 30 sec, then turn it back on. Make sure to re-attach via console:<br />
tip jailX<br />
immediately after power down. <br />
<br />
=== (Re)attach to the console ===<br />
Stay on the console the entire time during boot. As the BIOS posts- look out for the RAID card output- does everything look healthy? The output may be scrambled, look for "DEGRADED" or "FAILED". Once the OS starts booting you will be disconnected (dropped back to the shell on the console server) a couple times during the boot up. The reason you want to quickly re-attach is two-fold: 1. If you don’t reattach quickly then you won’t get any console output, 2. you want to be attached before the server ''potentially'' starts (an extensive) fsck. If you attach after the fsck begins, you’ll have seen no indication it started an fsck and the server will appear frozen during startup- no output, no response. <br />
<br />
IMPORTANT NOTE: on some older FreeBSD systems, there will be no output to the video (KVM) console as it boots up. The console output is redirected to the serial port ... so if a jail crashes, and you attach a kvm, the output during the bootup procedure will not be shown on the screen. However, when the bootup is done, you will get a login prompt on the screen and will be able to log in as normal. <tt>/boot/loader.conf</tt> is where serial console redirect output lives, so comment that if you want to catch output on kvm.<br />
On newer systems it sends most output to both locations. <br />
<br />
=== Assess the heath of the server ===<br />
Once the server boots up fully, you should be able to ssh in. Look around- make sure all the mounts are there and reporting the correct size/usage (i.e. /mnt/data1 /mnt/data2 /mnt/data3 - look in /etc/fstab to determine which mount points should be there), check to see if RAID mirrors are healthy. See [[RAID_Cards#Common_CLI_commands_.28megacli.29|megacli]], [[#aaccheck|aaccheck]]<br />
<br />
Before you start the jails, you need to run [[#preboot|preboot]]. This will do some assurance checks to make sure things are prepped to start the jails. Any issues that come out of preboot need to be addressed before starting jails.<br />
<br />
=== Start jails ===<br />
[[#Starting_jails:_Quad.2FSafe_Files|More on starting jails]]<br />
Customer jails (the VPSs) do not start up automatically at boot time. When a FreeBSD machines boots up, it boots up, and does nothing else. To start jails, we put the commands to start each jail into a shell script(s) and run the script(s). Jail startup is something that needs to be actively monitored, which is why we don’t just run the script automatically. <br />
<br />
In order to start jails, we run the quad files: quad1 quad2 quad3 and quad4 (on new systems there is only quad1). If the machine was cleanly rebooted- which wouldn't be the case if this was a crash, you may run the safe files (safe1 safe2 safe3 safe4) in lieu of quads. <br />
<br />
Open up 4 logins to the server (use the windows in [[Screen#Screen_Organization|a9]])<br />
In each of the 4 windows you will:<br />
<br />
If there is a [[#startalljails|startalljails]] script (and only quad1), run that command in each of the 4 windows. It will parse through the quad1 file and start each jail. Follow the instructions [[#Problems_with_the_quad.2Fsafe_files|here]] for monitoring startup. Note that you can be a little more lenient with jails that take awhile to start- startalljails will work around the slow jails and start the rest. As long as there aren't 4 jails which are "hung" during startup, the rest will get started eventually.<br />
-or-<br />
If there is no startalljails script, there will be multiple quad files. In each of the 4 windows, start each of the quads. i.e. start quad1 in window1, quad2 in window2 and so on. DO NOT start any quad twice. It will crash the server. If you accidentally do this, just jailkill all the jails which are in the quad and run the quad again. Follow the instructions here for monitoring quad startup.<br />
<br />
Note the time the last jail boots- this is what you will enter in the crash log.<br />
<br />
Save the crash log.<br />
<br />
=== Check to make sure all jails have started ===<br />
There's a simple script which will make sure all jails have started, and enter the ipfw counter rules: [[#postboot|postboot]] <br />
Run postboot, which will do a jailps on each jail it finds (excluding commented out jails) in the quad file(s). We're looking for 2 things:<br />
# systems spawning out of control or too many procs<br />
# jails which haven't started<br />
On 7.x and newer systems it will print out the problems (which jails haven't started) at the conclusion of postboot. <br />
On older systems you will need to watch closely to see if/when there's a problem, namely:<br />
<br />
[hostname] doesnt exist on this server<br />
<br />
When you get this message, it means one of 2 things:<br />
1. the jail really didn't start:<br />
When a jail doesn't start it usually boils down to a problem in the quad file. Perhaps the path name is wrong (data1 vs data2) or the name of the vn/mdfile is wrong. Once this is corrected, you will need to run the commands from the quad file manually, or you may use <tt>startjail <hostname></tt><br />
<br />
2. the customer has changed their hostname (and not told us) so their jail ''is'' running, just under a different hostname:<br />
On systems with jls, this is easy to rectify. First, get the customer info: <tt>g <hostname></tt><br />
Then look for the customer in jls: <tt>jls | grep <col0XXXX></tt><br />
From there you will see their new hostname- you should update that hostname in the quad file: don't forget to edit it on the <tt>## begin ##</tt> and <tt>## end ##</tt> lines, and in mgmt. <br />
On older systems without jls, this will be harder, you will need to look further to see their hostname- perhaps its in their /etc/rc.conf<br />
<br />
<br />
Once all jails are started, do some spot checks- try to ssh or browse to some customers, just to make sure things are really ok.<br />
<br />
== Adding disk to a 7.x/8.x jail ==<br />
or<br />
== Moving customer to a different drive (md) ==<br />
<br />
NOTE: this doesn’t apply to mx2 which uses gvinum. Use same procedure as 6.x<br />
NOTE: if you unmount before mdconfig, re-mdconfig (attach) then unmount then mdconfig -u again <br />
<br />
<br />
(parts to change/customize are <tt><span style="color:red">red</span></tt>)<br />
<br />
<br />
If someone wants more disk space, there’s a paste for it, send it to them (explains about downtime, etc).<br />
<br />
1. Figure out the space avail from <tt>js</tt>. Ideally, you want to put the customers new space on a different partition (and create the new md on the new partition). <br />
<br />
<br />
2. make a mental note of how much space they're currently using<br />
<br />
<br />
3. <tt>jailkill <hostname></tt> <br />
<br />
<br />
4. Umount it (including their devfs) but leave the md config’d (so if you use stopjail, you will have to re-mdconfig it)<br />
<br />
<br />
5. <tt>g <customerID></tt> to get the info (IP/cust#) needed to feed the new mdfile and mount name, and to see the current md device. <br />
<br />
<br />
6a. When there's enough room to place new system on an alternate, or the same drive:<br />
USE CAUTION not to overwrite (touch, mdconfig) existing md!!<br><br />
<tt>touch /mnt/data<span style="color:red">3/69.55.234.66-col01334</span><br><br />
mdconfig -a -t vnode -s 10g -f /mnt/data3/69.55.234.66-col01334 -u 97<br><br />
newfs /dev/md97</tt><br />
<br />
<br />
Optional- if new space is on a different drive, move the mount point directory AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
<tt>mount /dev/md<span style="color:red">97</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<br />
confirm you are mounted to /dev/md<span style="color:red">97</span> and space is correct:<br><br />
<tt>df .</tt> <br />
<br />
<br />
do the dump and pipe directly to restore:<br><br />
<tt>dump -0a -f - /dev/md<span style="color:red">1</span> | restore -r -f - <br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use <tt>df</tt> to confirm the restored data size matches the original usage figure<br />
<br />
<br />
md-unconfig old system:<br><br />
<tt>mdconfig -d -u <span style="color:red">1</span></tt><br />
<br />
<br />
archive old mdfile. <br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.237.26-col00241</span> /mnt/data<span style="color:red">1/old-col00241-mdfile-noarchive-20091211</span></tt><br />
<br />
<br />
edit quad (vq1) to point to new (/mnt/data<span style="color:red">3</span>) location AND new md number (md<span style="color:red">97</span>)<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
6b. When there's not enough room on an alternate partition, or on the same drive...but there is enough room if you were to remove the existing customer's space:<br />
<br />
<br />
mount backup nfs mounts:<br><br />
<tt>mbm</tt><br> <br />
(run <tt>df</tt> to confirm backup mounts are mounted)<br />
<br />
<br />
dump the customer to backup2 or backup1<br><br />
<tt>dump -0a -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> /dev/md<span style="color:red">1</span></tt><br><br />
(when complete WITHOUT errors, <tt>du</tt> the dump file to confirm it matches size, roughly, with usage)<br />
<br />
<br />
unconfigure and remove old mdfile<br><br />
<tt>mdconfig -d -u <span style="color:red">1</span><br><br />
rm /mnt/data<span style="color:red">1/69.55.237.26-col00241</span></tt><br><br />
(there should now be enough space to recreate your bigger system. If not, run sync a couple times)<br />
<br />
<br />
create the new system (ok to reuse old mdfile and md#):<br><br />
<tt>touch /mnt/data<span style="color:red">1/69.55.234.66-col01334</span><br><br />
mdconfig -a -t vnode -s <span style="color:red">10</span>g -f /mnt/data<span style="color:red">1/69.55.234.66-col01334</span> -u <span style="color:red">1</span><br><br />
newfs /dev/md<span style="color:red">1</span><br><br />
mount /dev/md<span style="color:red">1</span> /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> <br><br />
cd /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<br />
confirm you are mounted to /dev/md<span style="color:red">1</span> and space is correct:<br><br />
<tt>df .</tt> <br />
<br />
<br />
do the restore from the dumpfile on the backup server:<br><br />
<tt>restore -r -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> .<br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
umount nfs:<br><br />
<tt>mbu</tt><br />
<br />
<br />
If md# changed (or mount point), edit quad (<tt>vq1</tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new md number (md<span style="color:red">1</span>)<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
7. update disk (and dir if applicable) in mgmt screen<br />
<br />
<br />
8. update backup list AND move backups, if applicatble<br />
<br />
Ex: <tt>mvbackups <span style="color:red">69.55.237.26-col00241</span> jail<span style="color:red">9</span> data<span style="color:red">3</span></tt><br />
<br />
<br />
9. Optional: archive old mdfile<br />
<br />
<tt>mbm<br><br />
gzip -c old-col01588-mdfile-noarchive-20120329 > /deprecated/old-col01588-mdfile-noarchive-20120329.gz<br><br />
mbu<br><br />
rm old-col01588-mdfile-noarchive-20120329</tt><br />
<br />
== Adding disk to a 6.x jail (gvinum/gconcat) ==<br />
or<br />
== Moving customer to a different drive (gvinum/gconcat) ==<br />
<br />
(parts to change are <span style="color:red">highlighted</span>)<br />
<br />
<br />
If someone wants more disk space, there’s a paste for it, send it to them (explains about downtime, etc).<br />
<br />
<br />
1. Figure out the space avail from [[#js|js]]. Ideally, you want to put the customers new space on a different partition (and create the new md on the new partition). <br />
<br />
<br />
2. make a mental note of how much space they're currently using<br />
<br />
<br />
3. <tt>[[#stopjail|stopjail]] <hostname></tt> <br />
<br />
<br />
4. <tt>[[#g|g]] <customerID></tt> to get the info (IP/cust#) needed to feed the new mount name and existing volume/device. <br />
<br />
<br />
5a. When there's enough room to place new system on an alternate, or the same drive (using only UNUSED - including if it's in use by the system in question - gvinum volumes):<br />
<br />
<br />
Configure the new device:<br><br />
A. for a 2G system (single gvinum volume):<br><br />
<tt>bsdlabel -r -w /dev/gvinum/v<span style="color:red">123</span><br><br />
newfs /dev/gvinum/v<span style="color:red">123</span>a</tt><br><br />
<br />
-or- <br />
<br />
<br />
B. for a >2G system (create a gconcat volume):<br><br />
try to grab a contiguous block of gvinum volumes. gconcat volumes MAY NOT span drives (i.e. you cannot use a gvinum volume from data3 and a volume from data2 in the same gconcat volume). <br />
<br />
<tt>gconcat label <span style="color:red">v82-v84 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84</span><br><br />
bsdlabel -r -w /dev/concat/<span style="color:red">v82-v84</span><br><br />
newfs /dev/concat/<span style="color:red">v82-v84</span>a</tt><br />
<br />
<br />
Other valid gconcat examples:<br><br />
<tt>gconcat label v82-v84v109v112 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84 /dev/gvinum/v109 /dev/gvinum/v112<br><br />
gconcat label v82v83 /dev/gvinum/v82 /dev/gvinum/v83</tt> <br><br />
Note, long names will truncate: v144v145v148-v115 will truncate to v144v145v148-v1 (so you will refer to it as v144v145v148-v1 thereafter)<br />
<br />
<br />
Optional- if new volume is on a different drive, move the mount point directory (get the drive from js output) AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
<br />
confirm you are mounted to the device (<tt>/dev/gvinum/v<span style="color:red">123</span>a</tt> OR <tt>/dev/concat/<span style="color:red">v82-v84</span></tt>) and space is correct:<br><br />
A. <tt>mount /dev/gvinum/v<span style="color:red">123</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br><br />
-or-<br><br />
B. <tt>mount /dev/concat/<span style="color:red">v82-v84</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<tt>cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
df .</tt> <br />
<br />
<br />
do the dump and pipe directly to restore:<br><br />
<tt>dump -0a -f - /dev/gvinum/v<span style="color:red">1</span> | restore -r -f - <br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
edit quad (<tt>vq<span style="color:red">1</span></tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new volume (<tt>/dev/gvinum/v<span style="color:red">123</span>a</tt> or <tt>/dev/concat/<span style="color:red">v82-v84</span>a</tt>) , run <tt>buildsafe</tt><br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
5b. When there's not enough room on an alternate partition, or on the same drive...but there is enough room if you were to remove the existing customer's space (i.e. if you want/need to reuse the existing gvinum volumes and add on more):<br />
<br />
<br />
mount backup nfs mounts:<br><br />
<tt>mbm</tt> <br><br />
(run df to confirm backup mounts are mounted)<br />
<br />
<br />
dump the customer to backup2 or backup1<br><br />
<tt>dump -0a -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> /dev/<span style="color:red">gconcat/v106-v107</span></tt><br><br />
(when complete WITHOUT errors, du the dump file to confirm it matches size, roughly, with usage)<br />
<br />
<br />
unconfigure the old gconcat volume<br><br />
list member gvinum volumes:<br />
<br />
<tt>gconcat list <span style="color:red">v106v107</span></tt><br />
<br />
<br />
Output will resemble:<br><br />
<pre>Geom name: v106v107<br />
State: UP<br />
Status: Total=2, Online=2<br />
Type: AUTOMATIC<br />
ID: 3530663882<br />
Providers:<br />
1. Name: concat/v106v107<br />
Mediasize: 4294966272 (4.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e2<br />
Consumers:<br />
1. Name: gvinum/sd/v106.p0.s0<br />
Mediasize: 2147483648 (2.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e3<br />
Start: 0<br />
End: 2147483136<br />
2. Name: gvinum/sd/v107.p0.s0<br />
Mediasize: 2147483648 (2.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e3<br />
Start: 2147483136<br />
End: 4294966272</pre><br />
<br />
stop volume and clear members<br><br />
<tt>gconcat stop <span style="color:red">v106v107</span><br><br />
gconcat clear <span style="color:red">gvinum/sd/v106.p0.s0 gvinum/sd/v107.p0.s0</span></tt><br />
<br />
<br />
create new device- and its ok to reuse old/former members<br><br />
try to grab a contiguous block of gvinum volumes. gconcat volumes MAY NOT span drives (i.e. you cannot use a gvinum volume from data3 and a volume from data2 in the same gconcat volume). <br><br />
<tt>gconcat label <span style="color:red">v82-v84v106v107 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84 /dev/gvinum/v106 /dev/gvinum/v107</span> <br><br />
bsdlabel -r -w /dev/concat/<span style="color:red">v82-v84v106v107</span><br><br />
newfs /dev/concat/<span style="color:red">v82-v84v106v107</span>a</tt><br />
<br />
<br />
Optional- if new volume is on a different drive, move the mount point directory (get the drive from js output) AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
confirm you are mounted to the device (/dev/concat/<span style="color:red">v82-v84v106v107</span>) and space is correct:<br><br />
<tt>mount /dev/concat/<span style="color:red">v82-v84v106v107</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<tt>cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
df .</tt> <br />
<br />
do the restore from the dumpfile on the backup server:<br><br />
<tt>restore -r -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> .<br><br />
rm restoresymtable</tt><br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
edit quad (<tt>vq<span style="color:red">1</span></tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new volume (<tt>/dev/concat/<span style="color:red">v82-v84v106v107</span>a</tt>), run buildsafe<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
TODO: clean up/clear old gvin/gconcat vol<br />
<br />
<br />
6. update disk (and dir if applicable) in mgmt screen<br />
<br />
<br />
7. update backup list AND move backups, if applicatble<br />
<br />
<br />
Ex: <tt>mvbackups <span style="color:red">69.55.237.26-col00241</span> jail<span style="color:red">9</span> data<span style="color:red">3</span></tt><br />
<br />
<br />
<br />
DEPRECATED - steps to tack on a new gvin to existing gconcat- leads to corrupted fs<br />
bsdlabel -e /dev/concat/v82-v84<br />
<br />
To figure out new size of the c partition, multiply 4194304 by the # of 2G gvinum volumes and subtract the # of 2G volumes:<br />
10G: 4194304 * 5 – 5 = 20971515<br />
8G: 4194304 * 4 – 4 = 16777212<br />
6G: 4194304 * 3 – 3 = 12582909<br />
4G: 4194304 * 2 – 2 = 8388606<br />
<br />
To figure out the new size of the a partition, subtract 16 from the c partition:<br />
10G: 20971515 – 16 = 20971499<br />
8G: 16777212 – 16 = 16777196<br />
6G: 12582909 – 16 = 12582893<br />
4G: 8388606 – 16 = 8388590<br />
<br />
Orig:<br />
8 partitions:<br />
# size offset fstype [fsize bsize bps/cpg]<br />
a: 8388590 16 4.2BSD 2048 16384 28552<br />
c: 8388606 0 unused 0 0 # "raw" part, don't edit<br />
<br />
New:<br />
8 partitions:<br />
# size offset fstype [fsize bsize bps/cpg]<br />
a: 12582893 16 4.2BSD 2048 16384 28552<br />
c: 12582909 0 unused 0 0 # "raw" part, don't edit<br />
<br />
sync; sync<br />
<br />
growfs /dev/concat/v82-v84a<br />
<br />
fsck –fy /dev/concat/v82-v84a<br />
<br />
sync<br />
<br />
fsck –fy /dev/concat/v82-v84a<br />
<br />
(keep running fsck’s till NO errors)<br />
<br />
== Problems un-mounting - and with mount_null’s ==<br />
<br />
If you cannot unmount a filesystem, beacuse it says the filesystem is busy, it is because of three things:<br />
<br />
a) the jail is still running<br />
<br />
b) you are actually in that directory, even though the jail is stopped<br />
<br />
c) there are still dev, null_mount or linprocfs mount points mounted inside that directory.<br />
<br />
d) when trying to umount null_mounts that are really long and you get an error like “No such file or directory”, it’s an OS bug where the dir is truncated. No known fix<br />
<br />
e) there are still files open somewhere inside the dir. Use <tt>fstat | grep <cid></tt> to find the process that has files open<br />
<br />
f) Starting with 6.x, the jail mechanism does a poor job of keeping track of processes running in a jail and if it thinks there are still procs running, it will refuse to umount the disk. If this is happening you should see a low number in the #REF column when you run jls. In this case you ''can'' safely <tt>umount –f</tt> the mount. <br />
<br />
Please note -if you forcibly unmount a (4.x) filesystem that has null_mounts<br />
still mounted in it, the system '''will crash''' within 10-15 mins.<br />
<br />
== Misc jail Items ==<br />
<br />
We are overselling hard drive space on jail2, jail8, jail9, a couple jails on jail17, jail4, jail12 and jail18.<br />
Even though the vn file shows 4G size, it doesn’t actually occupy that amount of space on the disk. So be careful not to fill up drives where we’re overselling – use oversellcheck to confirm you’re not oversold by more than 10G.<br />
There are other truncated jails, they are generally noted in a the file on the root system: /root/truncated<br />
<br />
The act of moving a truncated vn to another system un-does the truncating- the truncated vn is filled with 0’s and it occupies physical disk space for which it’s configured. So, you should use dumpremote to preserve the truncation.<br />
<br />
= FreeBSD VPS Management Tools =<br />
<br />
These files are located in /usr/local/jail/rc.d and /usr/local/jail/bin<br />
<br />
== jailmake ==<br />
<br />
Applies to 7.x+ <br />
On older systems syntax differs, run jailmake once to see.<br />
<br />
Note: this procedure differs on mx2 which is 7.x but still uses gvinum<br />
<br />
# run js to figure out which md’s are in use, which disk has enough space, IP to put it on<br />
# use col00xxx for both hostnames if they don’t give you a hostname<br />
# copy over dir, ip and password to pending customer screen<br />
<br />
<tt>Usage: jailmake IP[,IP] CID disk[1|2|3] md# hostname shorthost ipfw# email [size in GB]</tt><br />
<br />
Ex: <br />
<br />
Jail2# jailmake 69.55.234.66 col01334 3 97 vps.bsd.it vps 1334 fb@bsd.it<br />
<br />
== jailps ==<br />
jailps [hostname]<br />
DEPRECATED FOR jps: displays processes belonging to/running inside a jail. The command<br />
takes one (optional) argument – the hostname of the jail you wish to query. If you don’t <br />
supply an argument, all processes on the machine are listed and grouped by jail. <br />
<br />
== jps ==<br />
jps [hostname]<br />
displays processes belonging to/running inside a jail. The command<br />
takes one (optional) argument – the hostname or ID of the jail you wish to query. <br />
<br />
== jailkill ==<br />
jailkill <hostname><br />
stops all process running in a jail.<br />
<br />
You can also run:<br />
jailkill <JID><br />
<br />
=== problems ===<br />
Occasionally you will hit an issue where jail will not kill off:<br />
<br />
<pre>jail9# jailkill www.domain.com<br />
www.domain.com .. killed: none<br />
jail9#</pre><br />
Because no processes are running under that hostname. You cannot use jailps.pl either:<br />
<br />
<pre>jail9# jailps www.domain.com<br />
www.domain.com doesn’t exist on this server<br />
jail9#</pre><br />
<br />
The reasons for this are usually:<br />
* the jail is no longer running<br />
<br />
* the jail's hostname has changed<br />
In this case, <br />
<br />
>=6.x: run a <tt>jls|grep <jail's IP></tt> to find the correct hostname, then update the quad file, then kill the jail.<br />
<br />
<6.x: the first step is to cat their /etc/rc.conf file to see if you can tell what they set the new hostname to. This very often works. For example:<br />
<br />
cat /mnt/data2/198.78.65.136-col00261-DIR/etc/rc.conf<br />
<br />
But maybe they set the hostname with the hostname command, and the original hostname is still in /etc/rc.conf.<br />
<br />
The welcome email clearly states that they should tell us if they change their hostname, so there is no problem in just emailing them and asking them what they set the new hostname to.<br />
<br />
Once you know the new hostname OR if a customer simply emails to inform you that they have set the hostname to something different, you need to edit the quad and safe files that their system is in to input the new hostname.<br />
<br />
However, if push comes to shove and you cannot find out the hostname from them or from their system, then you need to start doing some detective work.<br />
<br />
The easiest thing to do is run jailps looking for a hostname similar to their original hostname. Or you could get into the /bin/sh shell by running:<br />
<br />
/bin/sh<br />
<br />
and then looking at every hostname of every process:<br />
<br />
for f in `ls /proc` ; do cat /proc/$f/status ; done<br />
<br />
and scanning for a hostname that is either similar to their original hostname, or that you don't see in any of the quad safe files.<br />
<br />
This is very brute force though, and it is possible that catting every file in /proc is dangerous - I don't recommend it. A better thing would be to identify any processes that you know belong to this system – perhaps the reason you are trying to find this system is because they are running something bad - and just catting the status from only that PID.<br />
<br />
Somewhere there’s a jail where there may be 2 systems named www. Look at /etc/rc.conf and make sure they’re both really www. If they are, jailkill www, jailps www to make sure not running. Then immediately restart the other one, as the fqdn (as found from a rev nslookup)<br />
<br />
* on >=6.x the hostname may not yet be hashed:<br />
<pre>jail9 /# jls<br />
JID Hostname Path IP Address(es)<br />
1 bitnet.dgate.org /mnt/data1/69.55.232.50-col02094-DIR 69.55.232.50<br />
2 ns3.hctc.net /mnt/data1/69.55.234.52-col01925-DIR 69.55.234.52<br />
3 bsd1 /mnt/data1/69.55.232.44-col00155-DIR 69.55.232.44<br />
4 let2.bbag.org /mnt/data1/69.55.230.92-col00202-DIR 69.55.230.92<br />
5 post.org /mnt/data2/69.55.232.51-col02095-DIR 69.55.232.51 ...<br />
6 ns2 /mnt/data1/69.55.232.47-col01506-DIR 69.55.232.47 ...<br />
7 arlen.server.net /mnt/data1/69.55.232.52-col01171-DIR 69.55.232.52<br />
8 deskfood.com /mnt/data1/69.55.232.71-col00419-DIR 69.55.232.71<br />
9 mirage.confluentforms.com /mnt/data1/69.55.232.54-col02105-DIR 69.55.232.54 ...<br />
10 beachmember.com /mnt/data1/69.55.232.59-col02107-DIR 69.55.232.59<br />
11 www.agottem.com /mnt/data1/69.55.232.60-col02109-DIR 69.55.232.60<br />
12 sdhobbit.myglance.org /mnt/data1/69.55.236.82-col01708-DIR 69.55.236.82<br />
13 ns1.jnielsen.net /mnt/data1/69.55.234.48-col00204-DIR 69.55.234.48 ...<br />
14 ymt.rollingegg.net /mnt/data2/69.55.236.71-col01678-DIR 69.55.236.71<br />
15 verse.unixlore.net /mnt/data1/69.55.232.58-col02131-DIR 69.55.232.58<br />
16 smcc-mail.org /mnt/data2/69.55.232.68-col02144-DIR 69.55.232.68<br />
17 kasoutsuki.w4jdh.net /mnt/data2/69.55.232.46-col02147-DIR 69.55.232.46<br />
18 dili.thium.net /mnt/data2/69.55.232.80-col01901-DIR 69.55.232.80<br />
20 www.tekmarsis.com /mnt/data2/69.55.232.66-col02155-DIR 69.55.232.66<br />
21 vps.yoxel.net /mnt/data2/69.55.236.67-col01673-DIR 69.55.236.67<br />
22 smitty.twitalertz.com /mnt/data2/69.55.232.84-col02153-DIR 69.55.232.84<br />
23 deliver4.klatha.com /mnt/data2/69.55.232.67-col02160-DIR 69.55.232.67<br />
24 nideffer.com /mnt/data2/69.55.232.65-col00412-DIR 69.55.232.65<br />
25 usa.hanyuan.com /mnt/data2/69.55.232.57-col02163-DIR 69.55.232.57<br />
26 daifuku.ppbh.com /mnt/data2/69.55.236.91-col01720-DIR 69.55.236.91<br />
27 collins.greencape.net /mnt/data2/69.55.232.83-col01294-DIR 69.55.232.83<br />
28 ragebox.com /mnt/data2/69.55.230.104-col01278-DIR 69.55.230.104<br />
29 outside.mt.net /mnt/data2/69.55.232.72-col02166-DIR 69.55.232.72<br />
30 vps.payneful.ca /mnt/data2/69.55.234.98-col01999-DIR 69.55.234.98<br />
31 higgins /mnt/data2/69.55.232.87-col02165-DIR 69.55.232.87 ...<br />
32 ozymandius /mnt/data2/69.55.228.96-col01233-DIR 69.55.228.96<br />
33 trusted.realtors.org /mnt/data2/69.55.238.72-col02170-DIR 69.55.238.72<br />
34 jc1.flanderous.com /mnt/data2/69.55.239.22-col01504-DIR 69.55.239.22<br />
36 guppylog.com /mnt/data2/69.55.238.73-col00036-DIR 69.55.238.73<br />
40 haliohost.com /mnt/data2/69.55.234.41-col01916-DIR 69.55.234.41 ...<br />
41 satyr.jorge.cc /mnt/data1/69.55.232.70-col01963-DIR 69.55.232.70<br />
jail9 /# jailkill satyr.jorge.cc<br />
ERROR: jail_: jail "satyr,jorge,cc" not found<br />
</pre><br />
<br />
Note how it's saying <tt>satyr,jorge,cc</tt> is not found, and not <tt>satyr.jorge.cc</tt>. <br />
<br />
The jail subsystem tracks things using comma-delimited hostnames. That is created every few hours:<br />
<br />
jail9 /# crontab -l<br />
0 0,6,12,18 * * * /usr/local/jail/bin/sync_jail_names<br />
<br />
So if we run this manually:<br />
jail9 /# /usr/local/jail/bin/sync_jail_names<br />
<br />
Then kill the jail:<br />
jail9 /# jailkill satyr.jorge.cc<br />
successfully killed: satyr,jorge,cc</pre><br />
<br />
It worked.<br />
<br />
<br />
If you ever see this when trying to kill a jail:<br />
<br />
<pre># jailkill e-scribe.com<br />
killing JID: 6 hostname: e-scribe.com<br />
3 procs running<br />
3 procs running<br />
3 procs running<br />
3 procs running<br />
...</pre><br />
<br />
<tt>[[#jailkill|jailkill]]</tt> probably got lost trying to kill off the jail. Just ctrl-c the jailkill process, then run a jailps on the hostname, and <tt>kill -9</tt> any process which is still running. Keep running jailps and <tt>kill -9</tt> till all processes are gone.<br />
<br />
== jailpsall ==<br />
jailpsall<br />
will run a jailps on all jails configured in the quad files (this is different from<br />
jailps with no arguments as it won’t help you find a “hidden” system)<br />
<br />
== jailpsw ==<br />
jailpsw<br />
will run a jailps with an extra -w to provide wider output<br />
<br />
== jt (>=7.x) ==<br />
jt<br />
displays the top 20 processes on the server (the top 20 processes from top) and <br />
which jail owns them. This is very helpful for determining who is doing what when<br />
the server is very busy.<br />
<br />
== jtop (>=7.x) ==<br />
jtop<br />
a wrapper for top displaying processes on the server and which jail owns them. Constantly updates, like top. <br />
<br />
== jtop (<7.x) ==<br />
jtop<br />
displays the top 20 processes on the server (the top 20 processes from top) and <br />
which jail owns them. This is very helpful for determining who is doing what when<br />
the server is very busy.<br />
<br />
== stopjail ==<br />
stopjail <hostname> [1]<br />
this will jailkill, umount and vnconfig –u a jail. If passed an optional 2nd<br />
argument, it will not exit before umounting and un-vnconfig’ing in the event<br />
jailkill returns no processes killed. This is useful if you just want to umount<br />
and vnconfig –u a jail you’ve already killed. It is intelligent in that it won’t <br />
try to umount or vnconfig –u if it’s not necessary.<br />
<br />
== startjail ==<br />
startjail <hostname><br />
this will start vnconfig, mount (including linprocfs and null-mounts), and start a jail.<br />
Essentially, it reads the jail’s relevant block from the right quad file and executes it.<br />
It is intelligent in that it won’t try to mount or vnconfig if it’s not necessary.<br />
<br />
== jpid ==<br />
jpid <pid><br />
displays information about a process – including which jail owns it.<br />
It’s the equivalent of running cat /proc/<pid>/status<br />
<br />
== canceljail ==<br />
canceljail <hostname> [1]<br />
this will stop a jail (the equivalent of stopjail), check for backups (offer to remove them <br />
from the backup server and the backup.config), rename the vnfile, remove the dir, and <br />
edit quad/safe. If passed an optional 2nd argument, it will not exit upon failing to kill<br />
and processes owned by the jail. This is useful if you just want to cancel a jail which <br />
is already stopped.<br />
<br />
== jls ==<br />
jls [-v]<br />
Lists all jails running:<br />
<pre>JID #REF IP Address Hostname Path<br />
101 135 69.55.224.148 mail.pc9.org /mnt/data2/69.55.224.148-col01034-DIR</pre><br />
<br />
#REF is the number of references or procs(?) running<br />
<br />
Running with -v will give you all IPs assigned to each jail (7.2 up)<br />
<pre>JID #REF Hostname Path IP Address(es)<br />
101 139 mail.pc9.org /mnt/data2/69.55.224.148-col01034-DIR 69.55.224.14869.55.234.85</pre><br />
<br />
== startalljails ==<br />
startalljails<br />
7.2+ only. This will parse through quad1 and start all jails. It utilizes lockfiles so it won’t try to start a jail more than once- therefore multiple instances can be running in parallel without fear of starting a jail twice. If a jail startup gets stuck, you can ^C without fear of killing the script. IMPORTANT- before running startalljails you should make sure you ran preboot once as it will clear out all the lockfiles and enable startalljails to work properly.<br />
<br />
== aaccheck.sh ==<br />
aaccheck.sh<br />
displayes the output of container list and task list from aaccli<br />
<br />
== backup ==<br />
backup<br />
backup script called nightly to update jail scripts and do customer backups<br />
<br />
== buildsafe ==<br />
buildsafe<br />
creates safe files based on quads (automatically removing the fsck’s). This will destructively overwrite safe files<br />
<br />
== checkload.pl ==<br />
checkload.pl<br />
this was intended to be setup as a cronjob to watch processes on a jail when the load <br />
rises above a certain level. Not currently in use.<br />
<br />
== checkprio.pl ==<br />
checkprio.pl<br />
will look for any process (other than the current shell’s csh, sh, sshd procs) with a non-normal priority and normalize it<br />
<br />
== diskusagemon == <br />
diskusagemon <mount point> <1k blocks><br />
watches a mount point’s disk use, when it reaches the level specified in the 2nd argument,<br />
it exits. This is useful when doing a restore and you want to be paged as it’s nearing completion.<br />
Best used as: <tt>diskusagemon /asd/asd 1234; pagexxx</tt><br />
<br />
== dumprestore ==<br />
dumprestore <dumpfile><br />
this is a perl expect script which automatically enters ‘1’ and ‘y’. It seems to cause restore to fail<br />
to set owner permissions on large restores.<br />
<br />
== g ==<br />
g <search><br />
greps the quad/safe files for the given search parameter<br />
<br />
== gather.pl ==<br />
gather.pl<br />
gathers up data about jails configured and writes to a file. Used for audits against the db<br />
<br />
== gb ==<br />
gb <search><br />
greps backup.config for the given search parameter<br />
<br />
== gbg ==<br />
gbg <search><br />
greps backup.config for the given search parameter and presents just the directories (for clean pasting)<br />
<br />
== ipfwbackup ==<br />
ipfwbackup<br />
writes ipfw traffic count data to a logfile<br />
<br />
== ipfwreset ==<br />
ipfwreset<br />
writes ipfw traffic count data to a logfile and resets counters to 0<br />
<br />
== js ==<br />
js<br />
output varies by OS version, but generally provides information about the base jail:<br />
- which vn’s are in use<br />
- disk usage<br />
- info about the contents of quads<br />
- the # of inodes represented by the jails contained in the group (133.2 in the example below), and how many jails per data mount, as well as subtotals<br />
- ips bound to the base machine but not in use by a jail<br />
- free gvinum volumes, or unused vn’s or used md’s<br />
<br />
<pre>/usr/local/jail/rc.d/quad1:<br />
/mnt/data1 133.2 (1)<br />
/mnt/data2 1040.5 (7)<br />
total 1173.7 (8)<br />
/usr/local/jail/rc.d/quad2:<br />
/mnt/data1 983.4 (6)<br />
total 983.4 (6)<br />
/usr/local/jail/rc.d/quad3:<br />
/mnt/data1 693.4 (4)<br />
/mnt/data2 371.6 (3)<br />
total 1065 (7)<br />
/usr/local/jail/rc.d/quad4:<br />
/mnt/data1 466.6 (3)<br />
/mnt/data2 882.2 (5)<br />
total 1348.8 (8)<br />
/mnt/data1: 2276.6 (14)<br />
/mnt/data2: 2294.3 (15)<br />
<br />
Available IPs:<br />
69.55.230.11 69.55.230.13 69.55.228.200<br />
<br />
Available volumes:<br />
v78 /mnt/data2 2G<br />
v79 /mnt/data2 2G<br />
v80 /mnt/data2 2G</pre> <br />
<br />
== load.pl ==<br />
load.pl<br />
feeds info to load mrtg - executed by inetd.<br />
<br />
== makevirginjail ==<br />
makevirginjail<br />
Only on some systems, makes an empty jail (doesn't do restore step)<br />
<br />
== mb == <br />
mb <mount|umount><br />
(nfs) mounts and umounts dirs to backup2. Shortcuts are mbm and mbu to mount and unmount. <br />
<br />
== notify.sh ==<br />
notify.sh<br />
emails reboot@johncompanies.com – intended to be called at boot time to alert us to a machine which panics and reboots and isn’t caught by bb or castle.<br />
<br />
== orphanedbackupwatch ==<br />
orphanedbackupwatch<br />
looks for directories on backup2 which aren’t configured in backup.config and offers to delete them<br />
<br />
== postboot ==<br />
postboot<br />
to be run after a machine reboot and quad/safe’s are done executing. It will:<br />
* do chmod 666 on each jail’s /dev/null<br />
* add ipfw counts<br />
* run jailpsall (so you can see if a configured jail isn’t running)<br />
<br />
== preboot ==<br />
preboot<br />
to be run before running quad/safe – checks for misconfigurations: <br />
* a jail configured in a quad but not a safe<br />
* a jail is listed more than once in a quad<br />
* the ip assigned to a jail isn’t configured on the machine<br />
* alias numbering skips in the rc.conf (resulting in the above)<br />
* orphaned vnfile's that aren't mentioned in a quad/safe<br />
* ip mismatches between dir/vnfile name and the jail’s ip<br />
* dir/vnfiles's in quad/safe that don’t exist <br />
<br />
== quadanalyze.pl ==<br />
quadanalyze.pl<br />
called by js, produces the info (seen above with js explanation) about the contents of quad (inode count, # of jails, etc.)<br />
<br />
== rsync.backup ==<br />
rsync.backup<br />
does customer backups (relies on backup.config)<br />
<br />
== taskdone ==<br />
taskdone<br />
when called will email support@johncompanies.com with the hostname of the machine from which it was executed as the subject<br />
<br />
== topten ==<br />
topten<br />
summarizes the top 10 traffic users (called by ipfwreset)<br />
<br />
== trafficgather.pl ==<br />
trafficgather.pl [yy-mm]<br />
sends a traffic usage summary by jail to support@johncomapnies.com and payments@johncompanies.com. Optional arguments are year and month (must be in the past). If not passed, assumes last month. Relies on traffic logs created by ipfwreset and ipfwbackup<br />
<br />
== trafficwatch.pl ==<br />
trafficwatch.pl<br />
checks traffic usage and emails support@johncomapnies.com when a jail reaches the warning level (35G) and the limit (40G). We really aren’t using this anymore now that we have netflow.<br />
<br />
== trafstats ==<br />
trafstats<br />
writes ipfw traffic usage info by jail to a file called jc_traffic_dump in each jail’s / dir<br />
<br />
== truncate_jailmake ==<br />
truncate_jailmake<br />
a version of jailmake which creates truncated vnfiles.<br />
<br />
== vb ==<br />
vb<br />
the equivalent of: <tt>vi /usr/local/jail/bin/backup.config</tt><br />
<br />
== vs (freebsd) ==<br />
vs<n><br />
the equivalent of: <tt>vi /usr/local/jail/rc.d/safe<n></tt><br />
<br />
vq<n><br />
the equivalent of: vi /usr/local/jail/rc.d/quad<n><br />
<br />
== dumpremote ==<br />
dumpremote <user@machine> </remote/location/file-dump> <vnX><br />
ex: dumpremote user@10.1.4.117 /mnt/data3/remote.echoditto.com-dump 7<br />
this will dump a vn filesystem to a remote machine and location<br />
<br />
== oversellcheck ==<br />
oversellcheck<br />
displays how much a disk is oversold or undersold taking into account truncated vn files. Only for use on 4.x systems<br />
<br />
== mvbackups (freebsd) ==<br />
mvbackups <dir> (1.1.1.1-col00001-DIR) <target_machine> (jail1) <target_dir> (data1)<br />
moves backups from one location to another on the backup server, and provides you with option to remove entries from current backup.config, and simple paste command to add the config to backup.config on the target server<br />
<br />
== jailnice ==<br />
jailnice <hostname><br />
applies <tt>renice 19 [PID]</tt> and <tt>rtprio 31 –[PID]</tt> to each process in the given jail<br />
<br />
== dumpremoterestore ==<br />
dumpremoterestore <device> <ip of target machine> <dir on target machine><br />
ex: dumpremoterestore /dev/vn51 10.1.4.118 /mnt/data2/69.55.239.45-col00688-DIR<br />
dumps a device and restores it to a directory on a remote machine. Requires that you enable root ssh on the <br />
remote machine.<br />
<br />
== psj ==<br />
psj<br />
shows just the procs running on the base system – a ps auxw but without jail’d procs present<br />
<br />
== perc5iraidchk ==<br />
perc5iraidchk<br />
checks for degraded arrays on Dell 2950 systems with Perc5/6 controllers<br />
<br />
== perc4eraidchk ==<br />
perc4eraidchk<br />
checks for degraded arrays on Dell 2850 systems with Perc4e/Di controllers<br />
<br />
= Virtuozzo VPS =<br />
<br />
Note: We use VEID (Virtual Environment ID) and CTID (Container ID) interchangably. Similarly, VE and CT. They mean the same thing.<br />
VZPP = VirtuoZzo Power Panel (the control panel for each CT)<br />
<br />
All linux systems exist in /vz, /vz1 or /vz2 - since each linux machine holds roughly 60-90 customers, there will be roughly 30-45 in each partition.<br />
<br />
The actual filesystem of the system in question is in:<br />
<br />
/vz(1-2)/private/(VEID)<br />
<br />
Where VEID is the identifier for that system - an all-numeric string larger than 100.<br />
<br />
The actual mounted and running systems are in the corresponding:<br />
<br />
/vz(1-2)/root/(VEID)<br />
<br />
But we rarely interact with any system from this mount point.<br />
<br />
You should never need to touch the root portion of their system – however you can traverse their filesystem by going to <tt>/vz(1-2)/private/(VEID)/root</tt> (<tt>/vz(1-2)/private/(VEID)/fs/root</tt> on 4.x systems) the root of their filesystem is in that directory, and their entire system is underneath that.<br />
<br />
Every VE has a startup script in <tt>/etc/sysconfig/vz-scripts</tt> (which is symlinked as <tt>/vzconf</tt> on all systems) - the VE startup script is simply named <tt>(VEID).conf</tt> - it contains all the system parameters for that VE:<br />
<br />
<pre># Configuration file generated by vzsplit for 60 VE<br />
# on HN with total amount of physical mem 2011 Mb<br />
<br />
VERSION="2"<br />
CLASSID="2"<br />
<br />
ONBOOT="yes"<br />
<br />
KMEMSIZE="8100000:8200000"<br />
LOCKEDPAGES="322:322"<br />
PRIVVMPAGES="610000:615000"<br />
SHMPAGES="33000:34500"<br />
NUMPROC="410:415"<br />
PHYSPAGES="0:2147483647"<br />
VMGUARPAGES="13019:2147483647"<br />
OOMGUARPAGES="13019:2147483647"<br />
NUMTCPSOCK="1210:1215"<br />
NUMFLOCK="107:117"<br />
NUMPTY="19:19"<br />
NUMSIGINFO="274:274"<br />
TCPSNDBUF="1800000:1900000"<br />
TCPRCVBUF="1800000:1900000"<br />
OTHERSOCKBUF="900000:950000"<br />
DGRAMRCVBUF="200000:200000"<br />
NUMOTHERSOCK="650:660"<br />
DCACHE="786432:818029"<br />
NUMFILE="7500:7600"<br />
AVNUMPROC="51:51"<br />
IPTENTRIES="155:155"<br />
DISKSPACE="4194304:4613734"<br />
DISKINODES="400000:420000"<br />
CPUUNITS="1412"<br />
QUOTAUGIDLIMIT="2000"<br />
VE_ROOT="/vz1/root/636"<br />
VE_PRIVATE="/vz1/private/636"<br />
NAMESERVER="69.55.225.225 69.55.230.3"<br />
OSTEMPLATE="vzredhat-7.3/20030305"<br />
VE_TYPE="regular"<br />
IP_ADDRESS="69.55.225.229"<br />
HOSTNAME="textengine.net"</pre><br />
<br />
<br />
As you can see, the hostname is set here, the disk space is set here, the number of inodes, the number of files that can be open, the number of tcp sockets, etc. - all are set here.<br />
<br />
In fact, everything that can be set on this customer system is set in this conf file.<br />
<br />
<br />
All interaction with the customer system is done with the VEID. You start the system by running:<br />
<br />
vzctl start 999<br />
<br />
You stop it by running:<br />
<br />
vzctl stop 999<br />
<br />
You execute commands in it by running:<br />
<br />
vzctl exec 999 df -k<br />
<br />
You enter into it, via a root-shell backdoor with:<br />
<br />
vzctl enter 999<br />
<br />
and you set parameters for the system, while it is still running, with:<br />
<br />
vzctl set 999 --diskspace 6100000:6200000 --save<br />
<br />
<tt>vzctl</tt> is the most commonly used command - we have aliased <tt>v</tt> to <tt>vzctl</tt> since we use it so often. We’ll continue to use <tt>vzctl</tt> in our examples, but feel free to use just <tt>v</tt>.<br />
<br />
Let's say the user wants more diskspace. You can cat their conf file and see:<br />
<br />
DISKSPACE="4194304:4613734"<br />
<br />
So right now they have 4gigs of space. You can then change it to 6 with:<br />
<br />
vzctl set 999 --diskspace 6100000:6200000 --save<br />
<br />
IMPORTANT: all issuances of the vzctl set command need to end with <tt>–save</tt> - if they don't, the setting will be set, but it will not be saved to the conf file, and they will not have those settings next time they boot.<br />
<br />
All of the tunables in the conf file can be set with the vzctl set command. Note that in the conf file, and on the vzctl set command line, we always issue two numbers seperated by a colon - that is because we are setting the hard and soft limits. Always set the hard limit slightly above the soft limit, as you see it is in the conf file for all those settings.<br />
<br />
There are also things you can set with `<tt>vzctl set</tt>` that are not in the conf file as settings, per se. For instance, you can add IPs:<br />
<br />
vzctl set 999 --ipadd 10.10.10.10 --save<br />
<br />
or multiple IPs:<br />
<br />
vzctl set 999 --ipadd 10.10.10.10 --ipadd 10.10.20.30 --save<br />
<br />
or change the hostname:<br />
<br />
vzctl set 999 --hostname www.example.com --save<br />
<br />
You can even set the nameservers:<br />
<br />
vzctl set 999 --nameserver 198.78.66.4 --nameserver 198.78.70.180 --save<br />
<br />
Although you probably will never do that.<br />
<br />
You can disable a VPS from being started (by VZPP or reboot) (4.x):<br />
<br />
vzctl set 999 --disabled yes --save <br />
<br />
You can disable a VPS from being started (by VZPP or reboot) (<=3.x):<br />
<br />
vzctl set 999 --onboot=no --save <br />
<br />
You can disable a VPS from using his control panel:<br />
<br />
vzctl set 999 --offline_management=no --save <br />
<br />
You can suspend a VPS, so it can be resumed in the same state it was in when it was stopped (4.x):<br />
<br />
vzctl suspend 999<br />
<br />
and to resume it:<br />
<br />
vzctl resume 999<br />
<br />
to see who owns process:<br />
vzpid <PID><br />
<br />
to mount up an unmounted ve:<br />
vzctl mount 827<br />
<br />
<br />
To see network stats for CT's:<br />
<pre># vznetstat<br />
VEID Net.Class Output(bytes) Input(bytes)<br />
-----------------------------------------------<br />
24218 1 484M 39M<br />
24245 1 463M 143M<br />
2451 1 2224M 265M<br />
2454 1 2616M 385M<br />
4149 1 125M 68M<br />
418 1 1560M 34M<br />
472 1 1219M 315M<br />
726 1 628M 317M<br />
763 1 223M 82M<br />
771 1 4234M 437M<br />
-----------------------------------------------<br />
Total: 13780M 2090M</pre><br />
<br />
<br />
One thing that sometimes comes up on older systems that we created with smaller defaults is that the system would run out of inodes. The user will email and say they cannot create any more files or grow any files larger, but they will also say that they are not out of diskspace ... they are running:<br />
<br />
df -k<br />
<br />
and seeing how much space is free - and they are not out of space. They are most likely out of inodes - which they would see by running:<br />
<br />
df -i<br />
<br />
So, the first thing you should do is enter their system with:<br />
<br />
vzctl enter 999<br />
<br />
and run: <tt>df -i</tt><br />
<br />
to confirm your theory. Then exit their system. Then simply cat their conf file and see what their inodes are set to (probably 200000:200000, since that was the old default on the older systems) and run:<br />
<br />
vzctl set 999 --diskinodes 400000:400000 --save<br />
<br />
If they are not out of inodes, then a good possibility is that they have maxed out their numfile configuration variable, which controls how many files they can have in their system. The current default is 7500 (which nobody has ever hit), but the old default was as low as 2000, so you would run something like:<br />
<br />
vzctl set 999 --numfile 7500:7500 --save<br />
<br />
----<br />
<br />
You cannot start or stop a VE if your pwd is its private (/vz/private/999) or root (/vz/root/999) directories, or anywhere below them.<br />
<br />
<br />
== Recovering from a crash (linux) ==<br />
<br />
=== Diagnose whether you have a crash ===<br />
The most important thing is to get the machine and all ve’s back up as soon as possible. Note the time, you’ll need to create a crash log entry (Mgmt. -> Reference -> CrashLog). The first thing to do is head over to the [[Screen#Screen_Organization|serial console screen]] and see if there’s any kernel error messages output. Try to copy any messages (or just a sample of repeating messages) you see into the notes section of the crash log – these will also likely need to be sent to virtuozzo for interpretation. If the messages are spewing too fast, hit ^O + H to start a screen log dump which you can ob1182.pts-38.bb serve after the machine is rebooted. Additionally, if the machine is responsive, you can get a trace to send to virtuozzo by hooking up a kvm and entering these 3 sequences:<br />
<pre>alt+print screen+m<br />
alt+print screen+p<br />
alt+print screen+t</pre><br />
<br />
If there are no messages, the machine may just be really busy- wait a bit (5-10min) to see if it comes back. If it's still pinging, odds are its very busy. If it doesn't come back, or the messages indicate a fatal error, you will need to proceed with a power cycle (ctrl+alt+del will not work).<br />
<br />
=== Power cycle the server ===<br />
If this machine is not a Dell 2950 with a [[DRAC/RMM#DRAC|DRAC card]] (i.e. if you can’t ssh into the DRAC card and issue racadm serveraction hardreset, then you will need someone at the data center to power the macine off, wait 30 sec, then turn it back on. Make sure to re-attach via console (<tt>tip virtxx</tt>) immediately after power down. <br />
<br />
=== (Re)attach to the console ===<br />
Stay on the console the entire time during boot. As the BIOS posts- look out for the RAID card output- does everything look healthy? The output may be scrambled, look for "DEGRADED" or "FAILED". Once the OS starts booting you will be disconnected (dropped back to the shell on the console server) a couple times during the boot up. The reason you want to quickly re-attach is two-fold: 1. If you don’t reattach quickly then you won’t get any console output, 2. you want to be attached before the server ''potentially'' starts (an extensive) fsck. If you attach after the fsck begins, you’ll have seen no indication it started an fsck and the server will appear frozen during startup- no output, no response. <br />
<br />
=== Start containers/VE's/VPSs ===<br />
When the machine begins to start VE’s, it’s safe to leave the console and login via ssh. All virts should be set to auto start all the VEs after a crash. Further, most (newer) virts are set to “fastboot” it’s VE’s (to find out, do:<br />
grep -i fast /etc/sysconfig/vz <br />
and look for <tt>VZFASTBOOT=yes</tt>). If this was set prior to the machine’s crash (setting it after the machine boots will not have any effect until the vz service is restarted) it will start each ve as fast as possible, in serial, then go thru each VE (serially), shutting it down running a vzquota (disk usage) check, then bringing it back up. The benefit is that all VE’s are brought up quickly (within 15min or so depending on the #), the downside is a customer watching closely will notice 2 outages – 1st the machine crash, 2nd their quota check (which will be a much shorter downtime- on the order of a few minutes). <br />
<br />
Where “fastboot” is not set to yes (i.e on quar1), vz will start them consecutively, checking the quotas one at a time, and the 60th VE may not start until an hour or two later - this is not acceptable.<br />
<br />
The good news is, if you run vzctl start for a VE that is already started, you will simply get an error: <tt>VE is already started</tt>. Further, if you attempt to vzctl start a VE that is in the process of being started, you will simply get an error: unable to lock VE. So, there is no danger in simply running scripts to start smaller sets of VEs. If the system is not autostarting, then there is no issue, and even if it does, when it conflicts, one process (yours or the autostart) will lose, and just move on to the next one.<br />
<br />
A script has been written to assist with ve starts: [[#startvirt.pl|startvirt.pl]] which will start 6 ve’s at once until there are no more left. If startvirt.pl is used on a system where “fastboot” was on, it will circumvent the fastboot for ve’s started by startvirt.pl – they will go through the complete quota check before starting- therefore this is not advisable when a system has crashed. When a system is booted cleanly, and there's no need for vzquota checks, then startvirt.pl is safe and advisable to run.<br />
<br />
=== Make sure all containers are running ===<br />
You can quickly get a feel for how many ve’s are started by running:<br />
<br />
<pre>[root@virt4 log]# vs<br />
VEID 16066 exist mounted running<br />
VEID 16067 exist mounted running<br />
VEID 4102 exist mounted running<br />
VEID 4112 exist mounted running<br />
VEID 4116 exist mounted running<br />
VEID 4122 exist mounted running<br />
VEID 4123 exist mounted running<br />
VEID 4124 exist mounted running<br />
VEID 4132 exist mounted running<br />
VEID 4148 exist mounted running<br />
VEID 4151 exist mounted running<br />
VEID 4155 exist mounted running<br />
VEID 42 exist mounted running<br />
VEID 432 exist mounted running<br />
VEID 434 exist mounted running<br />
VEID 442 exist mounted running<br />
VEID 450 exist mounted running<br />
VEID 452 exist mounted running<br />
VEID 453 exist mounted running<br />
VEID 454 exist mounted running<br />
VEID 462 exist mounted running<br />
VEID 463 exist mounted running<br />
VEID 464 exist mounted running<br />
VEID 465 exist mounted running<br />
VEID 477 exist mounted running<br />
VEID 484 exist mounted running<br />
VEID 486 exist mounted running<br />
VEID 490 exist mounted running</pre><br />
<br />
So to see how many ve’s have started:<br />
<pre>[root@virt11 root]# vs | grep running | wc -l<br />
39</pre><br />
<br />
And to see how many haven’t:<br />
<pre>[root@virt11 root]# vs | grep down | wc -l<br />
0</pre><br />
<br />
And how many we should have running:<br />
<pre>[root@virt11 root]# vs | wc -l<br />
39</pre><br />
<br />
Another tool you can use to see which ve’s have started, among other things is [[#vzstat|vzstat]]. It will give you CPU, memory, and other stats on each ve and the overall system. It’s a good thing to watch as ve’s are starting (note the VENum parameter, it will tell you how many have started):<br />
<br />
<pre>4:37pm, up 3 days, 5:31, 1 user, load average: 1.57, 1.68, 1.79<br />
VENum 40, procs 1705: running 2, sleeping 1694, unint 0, zombie 9, stopped 0<br />
CPU [ OK ]: VEs 57%, VE0 0%, user 8%, sys 7%, idle 85%, lat(ms) 412/2<br />
Mem [ OK ]: total 6057MB, free 9MB/54MB (low/high), lat(ms) 0/0<br />
Swap [ OK ]: tot 6142MB, free 4953MB, in 0.000MB/s, out 0.000MB/s<br />
Net [ OK ]: tot: in 0.043MB/s 402pkt/s, out 0.382MB/s 4116pkt/s<br />
Disks [ OK ]: in 0.002MB/s, out 0.000MB/s<br />
<br />
VEID ST %VM %KM PROC CPU SOCK FCNT MLAT IP<br />
1 OK 1.0/17 0.0/0.4 0/32/256 0.0/0.5 39/1256 0 9 69.55.227.152<br />
21 OK 1.3/39 0.1/0.2 0/46/410 0.2/2.8 23/1860 0 6 69.55.239.60<br />
133 OK 3.1/39 0.1/0.3 1/34/410 6.3/2.8 98/1860 0 0 69.55.227.147<br />
263 OK 2.3/39 0.1/0.2 0/56/410 0.3/2.8 34/1860 0 1 69.55.237.74<br />
456 OK 17/39 0.1/0.2 0/100/410 0.1/2.8 48/1860 0 11 69.55.236.65<br />
476 OK 0.6/39 0.0/0.2 0/33/410 0.1/2.8 96/1860 0 10 69.55.227.151<br />
524 OK 1.8/39 0.1/0.2 0/33/410 0.0/2.8 28/1860 0 0 69.55.227.153<br />
594 OK 3.1/39 0.1/0.2 0/45/410 0.0/2.8 87/1860 0 1 69.55.239.40<br />
670 OK 7.7/39 0.2/0.3 0/98/410 0.0/2.8 64/1860 0 216 69.55.225.136<br />
691 OK 2.0/39 0.1/0.2 0/31/410 0.0/0.7 25/1860 0 1 69.55.234.96<br />
744 OK 0.1/17 0.0/0.5 0/10/410 0.0/0.7 7/1860 0 6 69.55.224.253<br />
755 OK 1.1/39 0.0/0.2 0/27/410 0.0/2.8 33/1860 0 0 192.168.1.4<br />
835 OK 1.1/39 0.0/0.2 0/19/410 0.0/2.8 5/1860 0 0 69.55.227.134<br />
856 OK 0.3/39 0.0/0.2 0/13/410 0.0/2.8 16/1860 0 0 69.55.227.137<br />
936 OK 3.2/52 0.2/0.4 0/75/410 0.2/0.7 69/1910 0 8 69.55.224.181<br />
1020 OK 3.9/39 0.1/0.2 0/60/410 0.1/0.7 55/1860 0 8 69.55.227.52<br />
1027 OK 0.3/39 0.0/0.2 0/14/410 0.0/2.8 17/1860 0 0 69.55.227.83<br />
1029 OK 1.9/39 0.1/0.2 0/48/410 0.2/2.8 25/1860 0 5 69.55.227.85<br />
1032 OK 12/39 0.1/0.4 0/80/410 0.0/2.8 41/1860 0 8 69.55.227.90</pre><br />
<br />
When you are all done, you will want to make sure that all the VEs really did get started, run vs one more time.<br />
<br />
Note the time all ve’s are back up and enter that into and save the crash log entry.<br />
<br />
Occasionally, a ve will not start automatically. The most common reason for a ve not to come up normally is the ve was at it’s disk limit before the crash, and will not start since they’re over the limit. To overcome this, set the disk space to current usage level (the system will give this to you when it fails to start), start the ve, then re-set the disk space back to the prior level. Lastly, contact the customer to let them know they’re out of disk (or allocate more disk if they're entitled to more).<br />
<br />
== Hitting performance barriers and fixing them ==<br />
<br />
There are multiple modes virtuozzo offers to allocate resources to a ve. We utilize 2: SLM and UBC parameters<br />
On our 4.x systems, we use all SLM – it’s simpler to manage and understand. There are a few systems on virt19/18 that may also use SLM. Everything else uses UBC. <br />
You can tell a SLM ve by:<br />
<br />
SLMMODE="all"<br />
<br />
in their conf file. <br />
<br />
TODO: detail SLM modes and parameters.<br />
<br />
If someone is in SLM mode and they hit memory resource limits, they simply need to upgrade to more memory.<br />
<br />
The following applies to everyone else (UBC).<br />
<br />
Customers will often email and say that they are getting out of memory errors - a common one is "cannot fork" ... basically, anytime you see something odd like this, it means they are hitting one of their limits that is in place in their conf file.<br />
<br />
The conf file, however, simply shows their limits - how do we know what they are currently at ?<br />
<br />
The answer is a file called v - this file contains the current status (and peaks) of their performance settings, and also counts how many times they have hit the barrier. The output of the file looks like this:<br />
<br />
<pre>764: kmemsize 384113 898185 8100000 8200000 0<br />
lockedpages 0 0 322 322 0<br />
privvmpages 1292 7108 610000 615000 0<br />
shmpages 270 528 33000 34500 0<br />
dummy 0 0 0 0 0<br />
numproc 8 23 410 415 0<br />
physpages 48 5624 0 2147483647 0<br />
vmguarpages 0 0 13019 2147483647 0<br />
oomguarpages 641 6389 13019 2147483647 0<br />
numtcpsock 3 21 1210 1215 0<br />
numflock 1 3 107 117 0<br />
numpty 0 2 19 19 0<br />
numsiginfo 0 4 274 274 0<br />
tcpsndbuf 0 80928 1800000 1900000 0 <br />
tcprcvbuf 0 108976 1800000 1900000 0<br />
othersockbuf 2224 37568 900000 950000 0<br />
dgramrcvbuf 0 4272 200000 200000 0<br />
numothersock 3 9 650 660 0<br />
dcachesize 53922 100320 786432 818029 0<br />
numfile 161 382 7500 7600 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
numiptent 4 4 155 155 0</pre><br />
<br />
The first column is the name of the counter in question - the same names we saw in the systems conf file. The second column is the _current_ value of that counter, the third column is the max that that counter has ever risen to, the fourth column is the soft limit, and the fifth column is the hard limit (which is the same as the numbers in that systems conf file).<br />
<br />
The sixth number is the failcount - how many times the current usage has risen to hit the barrier. It will increase as soon as the current usage hits the soft limit.<br />
<br />
The problem with /proc/user_beancounters is that it actually contains that set of data for every running VE - so you can't just cat /proc/user_beancounters - it is too long and you get info for every other running system.<br />
<br />
You can vzctl enter the system and run:<br />
<br />
vzctl enter 9999<br />
cat /proc/user_beancounters<br />
<br />
inside their system, and you will just see the stats for their particular system, but entering their system every time you want to see it is combersome.<br />
<br />
So, I wrote a simple script called "vzs" which simply greps for the VEID, and spits out the next 20 or so lines (however many lines there are in the output, I forget) after it. For instance:<br />
<br />
<pre># vzs 765:<br />
765: kmemsize 2007936 2562780 8100000 8200000 0<br />
lockedpages 0 8 322 322 0<br />
privvmpages 26925 71126 610000 615000 0<br />
shmpages 16654 16750 33000 34500 0<br />
dummy 0 0 0 0 0<br />
numproc 41 57 410 415 0<br />
physpages 1794 49160 0 2147483647 0<br />
vmguarpages 0 0 13019 2147483647 0<br />
oomguarpages 4780 51270 13019 2147483647 0<br />
numtcpsock 23 37 1210 1215 0<br />
numflock 17 39 107 117 0<br />
numpty 1 3 19 19 0<br />
numsiginfo 0 6 274 274 0<br />
tcpsndbuf 22240 333600 1800000 1900000 0<br />
tcprcvbuf 0 222656 1800000 1900000 0<br />
othersockbuf 104528 414944 900000 950000 0<br />
dgramrcvbuf 0 4448 200000 200000 0<br />
numothersock 73 105 650 660 0<br />
dcachesize 247038 309111 786432 818029 0<br />
numfile 904 1231 7500 7600 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
numiptent 4 4 155 155 0</pre><br />
<br />
That showed us just the portion of /proc/user_beancounters for system 765.<br />
<br />
When you run the vzs command, always add a : after the VEID.<br />
<br />
So, if a customer complains about some out of memory errors, or no more files, or no more ptys, or just has an unspecific complain about processes dying, etc., the very first thing you need to do is check their beancounters with vzs. Usually you will spot an item that has a high failcount and needs to be upped.<br />
<br />
At that point you could simply up the counter with `vzctl set`. Generally pick a number 10-20% higher than the old one, and make the hard limit slightly larger than the the soft limit. However our systems now come in several levels and those levels have more/different memory allocations. If someone is complaining about something other than a memory limit (pty, numiptent, numflock), it’s generally safe to increase it, at least to the same level as what’s in the /vzconf/4unlimited file on the newest virt. If someone is hitting a memory limit, first make sure they are given what they deserve:<br />
<br />
(refer to mgmt -> payments -> packages)<br />
<br />
To set those levels, you use the [[#setmem|setmem]] command. <br />
<br />
The alternate (DEPRECATED) method would be to use one of 3 commands:<br />
256 <veid><br />
300 <veid><br />
384 <veid><br />
512 <veid><br />
<br />
If the levels were not right (you’d run vzs <veid> before and after to see the effect) tell the customer they’ve been adjusted and be done with it. If the levels were right, tell the customer they must upgrade to a higher package, tell them how to see level (control panel) and that they can reboot their system to escape this lockup contidion.<br />
<br />
Customers can also complain that their site is totally unreachable, or complain that it is down ... if the underlying machine is up, and all seems well, you may notice in the beancounters that network-specific counters are failing - such as numtcpsock, tcpsndbuf or tcprcvbuf. This will keep them from talking on the network and make it seem like their system is down. Again, just up the limits and things should be fine.<br />
<br />
On virts 1-4, you should first look at the default settings for that item on a later virt, such as virt 8 - we have increased the defaults a lot since the early machines. So, if you are going to up a counter on virt2, instead of upping it by 10-20%, instead up it to the new default that you see on virt8.<br />
<br />
== Moving a VE to another virt (migrate/migrateonline) ==<br />
<br />
This will take a while to complete - and it is best to do this at night when the load is light on both machines.<br />
<br />
There are different methods for this, depending on which version of virtuozzo is installed on the src. and dst. virt. <br />
To check which version is running: <br />
[root@virt12 private]# cat /etc/virtuozzo-release<br />
Virtuozzo release 2.6.0<br />
<br />
Ok, let's say that the VE is 1212, and vital stats are:<br />
<pre>[root@virt12 sbin]# vc 1212<br />
VE_ROOT="/vz1/root/1212"<br />
VE_PRIVATE="/vz1/private/1212"<br />
OSTEMPLATE="fedora-core-2/20040903"<br />
IP_ADDRESS="69.55.229.84"<br />
TEMPLATES="devel-fc2/20040903 php-fc2/20040813 mysql-fc2/20040812 postgresql-fc2/20040813 mod_perl-fc2/20040812 mod_ssl-fc2/20040811 jre-fc2/20040823 jdk-fc2/20040823 mailman-fc2/20040823 analog-fc2/20040824 proftpd-fc2/20040818 tomcat-fc2/20040823 usermin-fc2/20040909 webmin-fc2/20040909 uw-imap-fc2/20040830 phpBB-fc2/20040831 spamassassin-fc2/20040910 PostNuke-fc2/20040824 sl-webalizer-fc2/20040<br />
818"<br />
<br />
[root@virt12 sbin]# vzctl exec 1212 df -h<br />
Filesystem Size Used Avail Use% Mounted on<br />
/dev/vzfs 4.0G 405M 3.7G 10% /</pre><br />
<br />
From this you can see that he’s using (and will minimally need free on the dst server) ~400MB, and he’s running on a Fedora 2 template, version 20040903. He’s also got a bunch of other templates installed. It’s is '''vital''' that '''all''' these templates exist on the dst system. To confirm that, on the dst system run:<br />
<br />
For < 3.0:<br />
<br />
<pre>[root@virt14 private]# vzpkgls | grep fc2<br />
devel-fc2 20040903<br />
PostNuke-fc2 20040824<br />
analog-fc2 20040824<br />
awstats-fc2 20040824<br />
bbClone-fc2 20040824<br />
jdk-fc2 20040823<br />
jre-fc2 20040823<br />
mailman-fc2 20040823<br />
mod_frontpage-fc2 20040816<br />
mod_perl-fc2 20040812<br />
mod_ssl-fc2 20040811<br />
mysql-fc2 20040812<br />
openwebmail-fc2 20040817<br />
php-fc2 20040813<br />
phpBB-fc2 20040831<br />
postgresql-fc2 20040813<br />
proftpd-fc2 20040818<br />
sl-webalizer-fc2 20040818<br />
spamassassin-fc2 20040910<br />
tomcat-fc2 20040823<br />
usermin-fc2 20040909<br />
uw-imap-fc2 20040830<br />
webmin-fc2 20040909<br />
[root@virt14 private]# vzpkgls | grep fedora<br />
fedora-core-1 20040121 20040818<br />
fedora-core-devel-1 20040121 20040818<br />
fedora-core-2 20040903<br />
[root@virt14 private]#</pre><br />
<br />
For these older systems, you can simply match up the date on the template. <br />
<br />
For >= 3.0:<br />
<br />
<pre>[root@virt19 /vz2/private]# vzpkg list<br />
centos-5-x86 2008-01-07 22:05:57<br />
centos-5-x86 devel<br />
centos-5-x86 jre<br />
centos-5-x86 jsdk<br />
centos-5-x86 mod_perl<br />
centos-5-x86 mod_ssl<br />
centos-5-x86 mysql<br />
centos-5-x86 php<br />
centos-5-x86 plesk9<br />
centos-5-x86 plesk9-antivirus<br />
centos-5-x86 plesk9-api<br />
centos-5-x86 plesk9-atmail<br />
centos-5-x86 plesk9-backup<br />
centos-5-x86 plesk9-horde<br />
centos-5-x86 plesk9-mailman<br />
centos-5-x86 plesk9-mod-bw<br />
centos-5-x86 plesk9-postfix<br />
centos-5-x86 plesk9-ppwse<br />
centos-5-x86 plesk9-psa-firewall<br />
centos-5-x86 plesk9-psa-vpn<br />
centos-5-x86 plesk9-psa-fileserver<br />
centos-5-x86 plesk9-qmail<br />
centos-5-x86 plesk9-sb-publish<br />
centos-5-x86 plesk9-vault<br />
centos-5-x86 plesk9-vault-most-popular<br />
centos-5-x86 plesk9-watchdog</pre><br />
<br />
On these newer systems, it's difficult to tell whether the template on the dst matches exactly the src. Just cause a centos-5-x86 is listed on both servers doesn't mean all the same packages are there on the dst. To truly know, you must perform a sample rsync:<br />
<br />
rsync -avn /vz/template/centos/5/x86/ root@10.1.4.61:/vz/template/centos/5/x86/<br />
<br />
if you see a ton of output from the dry run command, then clearly there are some differences. You may opt to let the rsync complete (without running in dry run mode) the only downside is you've now used up more space on the dst and also the centos template will be a mess with old and new data- it will be difficult if not impossible to undo (if someday we wanted to reclaim the space).<br />
<br />
If you choose to merge templates, you should closely inspect the dry run output. You should also take care to exclude anything in the /config directory. For example:<br />
<br />
rsync -av -e ssh --stats --exclude=x86/config /vz/template/ubuntu/10.04/ root@10.1.4.62:/vz/template/ubuntu/10.04/<br />
<br />
Which will avoid this directory and contents:<br />
<pre>[root@virt11 /vz2/private]# ls /vz/template/ubuntu/10.04/x86/config*<br />
app os</pre><br />
<br />
This is important to avoid since the config may differ on the destination and we are really only interested in making sure the pacakges are there, not overwriting a newer config with an older one.<br />
<br />
If the dst system was missing a template, you have 2 choices: <br />
# put the missing template on the dst system. 2 choices here: <br />
## Install the template from rpm (found under backup2: /mnt/data4/vzrpms/distro/) or <br />
## rsync over the template (found under /vz/template) - see above<br />
# put the ve on a system which has all the proper templates<br />
<br />
=== pre-seeding a migration ===<br />
<br />
When migrating a customer (or when doing many) depending on how much data you have to transfer, it can take some time. Further, it can be difficult to gauge when a migration will complete or how long it will take. To help speed up the process and get a better idea about how long it will take you can pre-transfer a customer's data to the destination server. If done correctly, vzmigrate will see the pre-transferred data and pick up where you left off, having much less to transfer (just changed/new files). <br />
<br />
We believe vzmigrate uses rsync to do it's transfer. Therefore not only can you use rsync to do a pre-seed, you can also run rsync to see what is causing a repeatedly-failing vzmigrate to fail. <br />
<br />
There's no magic to a pre-seed, you just need to make sure it's named correctly.<br />
<br />
Given:<br />
<br />
source: /vz1/private/1234<br />
<br />
and you want to migrate to /vz2 on the target system, your rsync would look like:<br />
<br />
rsync -av /vz1/private/1234/ root@x.x.x.x:/vz2/private/1234.migrated/<br />
<br />
After running that successful rsync, the ensuing migrateonline (or migrate) will take much less time to complete- depending on the # of files to be analyzed and the # of changed files. In any case, it'll be much much faster than had you just started the migration from scratch.<br />
<br />
Further, as we discuss elsewhere in this topic, a failed migration can be moved from <tt>/vz/private/1234</tt> to <tt>/vz/private/1234.migrated</tt> on the destination if you want to restart a failed migration. This should '''only''' be done if the migration failed and the CT is not running on the destination HN.<br />
<br />
=== migrateonline intructions: src >=3.x -> dst>=3.x ===<br />
<br />
A script called [[#migrateonline|migrateonline]] was written to handle this kind of move. It is basically a wrapper for <tt>vzmigrate</tt> – vzmigrate is a util to seamlessly- as no no reboot of the ve necessary- move a ve from one host to another. This wrapper was initially written cause vz virtuozzo version 2.6.0 has a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables, causing problems when the ve was started up on the dst system. [[#migrate|migrate]] mitigates that. Since it makes multiple ssh connections to the dst virt, it’s a good idea to put the pub key for the src virt in the authorized_keys file on the dst virt. In addition, migrateonline emails ve owners when their migration starts and stops. For this to happen they need to put email addresses (on a single line, space delimited) in a file on their system: /migrate_notify. If the optional target dir is not specified, the ve will be moved to the same private/root location as it was on the src virt. Note: <tt>migrate</tt> is equivalent to <tt>migrateonline</tt>, but will <tt>migrate</tt> a ve AND restart it in the process.<br />
<br />
<pre>[root@virt12 sbin]# migrateonline<br />
usage: /usr/local/sbin/migrateonline <ip of node migrating to> <veid> [target dir: vz | vz1 | vz2]<br />
[root@virt12 sbin]# migrateonline 10.1.4.64 1212 vz<br />
starting to migrate 1212 at Sat Mar 26 22:40:38 PST 2005<br />
Turning off offline_management<br />
Saved parameters for VE 1212<br />
migrating with no start on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
clearing the arp cache<br />
now going to 10.1.4.64 and clear cache and starting<br />
starting it<br />
Delete port redirection<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
finished migrating 1212 at Sat Mar 26 22 22:52:01 PST 2005<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine.<br />
<br />
If they had backups, use the mvbackups command to move their backups to the new server:<br />
<br />
mvbackups 1212 virt14 vz<br />
<br />
Rename the ve<br />
<br />
[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/migrated-1212<br />
<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/old-1212-migrated-20120404-noarchive<br />
<br />
Update the customer’s systems in mgmt to reflect the new path and server.<br />
<br />
<br />
IF migrateonline does not work, you can try again using simply migrate- this will result in a brief reboot for the ve.<br />
Before you try again, make sure of a few things:<br />
<br />
Depending on where in the migration died, there may be partial data on the dst system in 1 of 2 places:<br />
(given the example above)<br />
<br />
/vz/private/1212<br />
<br />
or <br />
<br />
/vz/private/1212.migrated<br />
<br />
before you run migrate again, you'll want to rename so that all data is in <br />
1212.migrated:<br />
<br />
mv /vz/private/1212 /vz/private/1212.migrated<br />
<br />
this way, it will pick up where it left off and transfer only new files.<br />
<br />
Likewise, if you want to speed up a migration, you can pre-seed the dst as follows:<br />
<br />
[root@virt12 sbin]# rsync -avSH /vz/private/1212/ root@10.1.4.64:/vz/private/1212.migrated/<br />
<br />
then when you run migrate or migrateonline, it will only need to move the changed files- the migration will complete quickly<br />
<br />
=== migrate manually (if migrate/online fails) ===<br />
<br />
Lets say for whatever reason the migration fails. You can always move someone manually:<br />
<br />
1. stop ve: <br><br />
v stop 1234<br />
<br />
2. copy over data<br><br />
rsync -avSH /vz/private/1234/ root@1.1.1.1:/vzX/private/1234/<br />
<br />
NOTE: if you've previously seeded the data (run rsync while the VE was up/running), and this is a subsequent rsync, make sure the last rsync you do (while the VE is not running, has the --delete option in the rsync)<br />
<br />
3. copy over conf<br><br />
scp /vzconf/1234.conf root@1.1.1.1:/vzconf<br />
<br />
4. on dst, edit the conf to reflect the right vzX dir<br><br />
vi /vzconf/1234.conf<br />
<br />
5. on src remove the IPs<br><br />
ipdel 1234 2.2.2.2 3.3.3.3<br />
<br />
6. on dst add IPs <br><br />
ipadd 1234 2.2.2.2 3.3.3.3<br />
<br />
7. on dst, start ve: <br><br />
v start 1324<br />
<br />
8. cancel, then archive ve on src per above instrs.<br />
<br />
=== migrate src=2.6.0 -> dst>=2.6.0, or mass-migration with customer notify ===<br />
<br />
A script called <tt>migrate</tt> was written to handle this kind of move. It is basically a wrapper for vzmigrate – vzmigrate is a util to seamlessly move a ve from one host to another. This wrapper was initially written cause vz virtuozzo version 2.6.0 has a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables, causing problems when the ve was started up on the dst system. migrate mitigates that. Since it makes multiple ssh connections to the dst virt, it’s a good idea to put the pub key for the src virt in the authorized_keys file on the dst virt. In addition, migrate emails ve owners when their migration starts and stops. For this to happen they need to put email addresses (on a single line, space delimited) in a file on their system: /migrate_notify. If the optional target dir is not specified, the ve will be moved to the same private/root location as it was on the src virt. Note: migrateonline is equivalent to migrate, but will migrate a ve from one 2.6 '''kernel''' machine to another 2.6 kernel machine without restarting the ve.<br />
<br />
<pre>[root@virt12 sbin]# migrate<br />
usage: /usr/local/sbin/migrate <ip of node migrating to> <veid> [target dir: vz | vz1 | vz2]<br />
[root@virt12 sbin]# migrate 10.1.4.64 1212 vz<br />
starting to migrate 1212 at Sat Mar 26 22:40:38 PST 2005<br />
Turning off offline_management<br />
Saved parameters for VE 1212<br />
migrating with no start on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
clearing the arp cache<br />
now going to 10.1.4.64 and clear cache and starting<br />
starting it<br />
Delete port redirection<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
finished migrating 1212 at Sat Mar 26 22 22:52:01 PST 2005<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine (backup2 or mail).<br />
Cancel the ve (first we have to rename things which migrate changed so cancelve will find them):<br />
<br />
[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/1212.conf<br />
<br />
On 2.6.1 you’ll also have to move the private area:<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/1212<br />
<br />
<pre>[root@virt12 sbin]# cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, [[#cancelve|cancelve]] would offer to remove them. You want to say '''no''' to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
To move everyone off a system, you’d do:<br />
for f in `vl`; do migrate <ip> $f; done<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
=== vzmigrate: src=2.6.1 -> dst>=2.6.0 ===<br />
<br />
This version of vzmigrate works properly with regard to handling ips. It will not notify ve owners of moves as in the above example. Other than that it’s essentially the same.<br />
<br />
<pre>[root@virt12 sbin]# vzmigrate 10.1.4.64 -r no 1212:1212:/vz/private/1212:/vz/root/1212<br />
migrating on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine (backup2 or mail).<br />
Cancel the ve (first we have to rename things which vzmigrate changed so cancelve will find them):<br />
<br />
<pre>[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/1212.conf<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/1212<br />
<br />
[root@virt12 sbin]# cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, <tt>cancelve</tt> would offer to remove them. You want to say no to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
=== src=2.5.x ===<br />
<br />
First, go to the private dir:<br />
<br />
cd /vz1/private/<br />
<br />
Stop the VE - make sure it stops totally cleanly.<br />
<br />
vzctl stop 1212<br />
<br />
Then you’d use vemove - a script written to copy over the config, create tarballs of the ve’s data on the destination virt, and cancel the ve on the source system (in this example we’re going to put a ve that was in /vz1/private on the src virt, in /vz/private on the dst virt):<br />
<br />
<pre>[root@virt12 sbin]# vemove<br />
ERROR: Usage: vemove veid target_ip target_path_dir<br />
[root@virt12 sbin]# vemove 1212 10.1.4.64 /vz/private/1212<br />
tar cfpP - 1212 --ignore-failed-read | (ssh -2 -c arcfour 10.1.4.64 "split - -b 1024m /vz/private/1212.tar" )<br />
scp /vzconf/1212.conf 10.1.4.64:/vzconf<br />
cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, cancelve would offer to remove them. You want to say no to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
When you are done, go to /vz/private on the dst virt you will have files like this:<br />
<br />
<pre>1212.taraa<br />
1212.tarab<br />
1212.tarac</pre><br />
<br />
Each one 1024m (or less, for the last one) in size.<br />
<br />
on the dst server and run:<br />
<br />
cat 1212.tar?? | tar xpPBf -<br />
<br />
and after 20 mins or so it will be totally untarred. Now since the conf<br />
file is already there, you can go ahead and start the system.<br />
<br />
vzctl start 1212<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
NOTE: you MUST tar the system up using the virtuozzo version of tar that<br />
is on all the virt systems, and further you MUST untar the tarball with<br />
the virtuozzo tar, using these options: `<tt>tar xpPBf -</tt>`<br />
<br />
If you tar up an entire VE and move it to a non-virtuozzo machine, that is<br />
ok, and you can untar it there with normal tar commands, but do not untar<br />
it and then repack it with a normal tar and expect it to work - you need<br />
to use virtuozzo tar commands on virtuozzo tarballs to make it work.<br />
<br />
The backups are sort of an exception, since we are just (usually)<br />
restoring user data that was created after we gave them the system, and<br />
therefore has nothing to do with magic symlinks or vz-rpms, etc.<br />
<br />
== Moving a VE on the same virt ==<br />
<br />
Easy way:<br><br />
Scenario 1: ve 123 is to be renamed 1231 and moved from vz1 to vz<br />
<br />
vzmlocal 123:1231:/vz/private/1231:/vz/root/1231<br />
<br />
Scenario 2: ve 123 is to be moved vz1 to vz<br />
<br />
vzmlocal 123:123:/vz/private/123:/vz/root/123<br />
<br />
vzmlocal will reboot the ve at the end of the move<br />
<br />
Manual/old way:<br />
<br />
1) <tt>vzctl stop 123</tt><br><br />
2) <tt>mv /vz1/private/123 /vz/private/.</tt><br><br />
(or cp -a if you want to copy)<br />
3) in <tt>/etc/sysconfig/vz-scripts/123.conf</tt> change value<br><br />
of '<tt>VE_PRIVATE</tt>' variable to point to a new private area location<br />
4) <tt>vzctl start 123</tt><br><br />
5) update backups if needed: <tt>mvbackups 123 virtX virt1 vz</tt><br><br />
6) update management scerens<br><br />
<br />
Notes: a) absolute path to private area is stored in quota file <tt>/var/vzquota/quota.123</tt> - so during first startup quota will be recalculated.<br><br />
b) if you're going to write some script to do a job, you MUST be sure that $VEID won't be expanded to '' in ve config file - ie. you need to escape '$'. Otherwise you might have:<br />
<br />
VE_PRIVATE="/vz/private/"<br />
<br />
in config, and 'vzctl destroy' for this VE ID '''will remove everything under /vz/private/ directory'''.<br />
<br />
== Adding a veth device to a VE ==<br />
<br />
Not totally sure what this is, but a customer asked for it and here's what we did (as instructed by vz support):<br />
<br />
<pre>v set 99 --netif_add eth99 --save<br />
ipdel 99 69.55.230.58<br />
v set 99 --ifname eth99 --ipadd 69.55.230.58 --save<br />
v set 99 --ifname eth99 --gateway 69.55.230.1 --save<br />
<br />
vznetcfg net new veth_net<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
net99 active eth0 veth77.77,veth99.99<br />
veth_net active<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
br0 bridge veth_net<br />
br99 bridge net99<br />
veth99.99 veth net99<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic net99 69.55.227.70/24<br />
<br />
vznetcfg br attach br0 eth0<br />
<br />
(will remove 99 from orig net and move to veth_net)<br />
vznetcfg net addif veth_net veth99.99<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
net99 active<br />
veth_net active eth0 veth77.77,veth99.99<br />
<br />
(delete the old crap)<br />
vznetcfg net del net99<br />
<br />
<br />
Then, to add another device in<br />
<br />
v set 77 --netif_add eth77 --save<br />
ipdel 77 69.55.230.78<br />
v set 77 --ifname eth77 --ipadd 69.55.230.78 --save<br />
v set 77 --ifname eth77 --gateway 69.55.230.1 --save<br />
v set 77 --save --ifname eth77 --network veth_net<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
veth77.77 veth<br />
br0 bridge veth_net<br />
veth99.99 veth veth_net<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic veth_net 69.55.227.70/24<br />
<br />
vznetcfg net addif veth_net veth77.77<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
veth77.77 veth veth_net<br />
br0 bridge veth_net<br />
veth99.99 veth veth_net<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic veth_net 69.55.227.70/24<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
veth_net active eth0 veth77.77,veth99.99<br />
<br />
another example<br />
<br />
v set 1182 --netif_add eth1182 --save<br />
ipdel 1182 69.55.236.217<br />
v set 1182 --ifname eth1182 --ipadd 69.55.236.217 --save<br />
v set 1182 --ifname eth1182 --gateway 69.55.236.1 --save<br />
vznetcfg net addif veth_net veth1182.1182<br />
v set 1182 --save --ifname eth1182 --network veth_net<br />
<br />
<br />
unused/not working commands:<br />
ifconfig veth99.0 0<br />
vznetcfg net list<br />
vznetcfg br new br99 net99<br />
vznetcfg br attach br99 eth0<br />
vznetcfg br show<br />
vznetcfg if list<br />
vznetcfg net addif net99 veth99.99<br />
vznetcfg net addif eth0 net99<br />
<br />
---------<br />
<br />
vznetcfg br new br1182 net1182<br />
<br />
vznetcfg br attach br99 eth0<br />
vznetcfg if list<br />
vznetcfg net addif net99 veth99.99<br />
vznetcfg net addif eth0 net99<br />
<br />
<br />
<br />
vznetcfg net addif eth0 net1182<br />
<br />
<br />
# vznetcfg net addif veth99.0 net99<br />
--- 8< ---<br />
<br />
vznetcfg net new net<br />
# vznetcfg net addif eth0 net99<br />
<br />
<br />
# vzctl set 99 --save --netif_add eth0 (at this stage veth99.0 interface have to appear<br />
on node)<br />
# vzctl set 99 --save --ifname eth0 --ipadd 69.55.230.58 (and probably few more arguments<br />
here - see 'man vzctl')<br />
# vznetcfg net addif veth99.0 net99</pre><br />
<br />
<br />
== Assigning/remove ip from a VE ==<br />
<br />
1. Add or remove ips:<br />
ipdel 1234 1.1.1.1 2.2.2.2<br />
ipadd 1234 1.1.1.1 2.2.2.2<br />
<br />
2. update Mgmt screens<br />
<br />
3. offer to update any DNS we do for them<br />
<br />
4. check to see if we had rules for old IP in firwall<br />
<br />
== Enabling tun device for a ve ==<br />
Note, there’s a command for this: [[#addtun|addtun]]<br />
<br />
For posterity:<br />
Make sure the tun.o module is already loaded before Virtuozzo is started: <br />
lsmod <br />
Allow the VPS to use the TUN/TAP device: <br />
vzctl set 101 --devices c:10:200:rw --save <br />
Create the corresponding device inside the VPS and set the proper permissions: <br />
vzctl exec 101 mkdir -p /dev/net <br />
vzctl exec 101 mknod /dev/net/tun c 10 200 <br />
vzctl exec 101 chmod 600 /dev/net/tun<br />
<br />
== Remaking a system (on same virt) ==<br />
<br />
1. [[#cancelve|cancelve]] (or v destroy x - ONLY if you're POSITIVE no data needs to be saved)<br />
<br />
2. [[#vemake|vemake]] using same veid<br />
<br />
3. [[#mvbackups|mvbackups]] or [[#vb|vb]] (if new mount point)<br />
<br />
4. update mgmt with new dir/ip <br />
<br />
5. update firewall if changed ip<br />
<br />
== Traffic accounting on linux ==<br />
<br />
DEPRECATED - all tracking is done via bwdb now. This is how we used to track traffic.<br />
<br />
TODO: update for diff versions of vz<br />
<br />
Unlike FreeBSD, where we have to add firewall count rules to the system to count the traffic, on virtuozzo counts the traffic for us. You an see the current traffic stats by running `vznetstat`:<br />
<br />
<pre># vznetstat<br />
VEID Net.Class Output(bytes) Input(bytes)<br />
-----------------------------------------------<br />
24218 1 484M 39M<br />
24245 1 463M 143M<br />
2451 1 2224M 265M<br />
2454 1 2616M 385M<br />
4149 1 125M 68M<br />
418 1 1560M 34M<br />
472 1 1219M 315M<br />
726 1 628M 317M<br />
763 1 223M 82M<br />
771 1 4234M 437M<br />
-----------------------------------------------<br />
Total: 13780M 2090M</pre><br />
<br />
As you can see the VEID is on a line with the in and out bytes. So, we simply run a cron job:<br />
<br />
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /root/vztrafdump.sh<br />
<br />
Just like we do on FreeBSD - this one goes through all the VEs in /vz/private and greps the line from vznetstat that matches them and dumps it in /jc_traffic_dump on their system. Then it does it again for all the VEs in /vz1/private. It is important to note that vznetstat runs only once, and the grepping is done from a temporary file that contains that output - we do this because running vznetstat once for each VE that we read out of /vz/private and /vz1/private would take way too long and be too intensive.<br />
<br />
You do not need to do anything to facilitate this other than make sure that that cron job is running - the vznetstat counters are always running, and any new VEs that are added to the system will be accounted for automatically.<br />
<br />
Traffic resetting no longer works with vz 2.6, so we disable the vztrafdump.sh on those virts.<br />
<br />
== Watchdog script ==<br />
<br />
On some of the older virts, we have a watchdog running that kills procs that are deemed bad per the following:<br />
<br />
/root/watchdog from quar1<br />
<br />
<pre>if echo $line | awk '{print $(NF-3)}' | grep [5-9]...<br />
then<br />
# 50-90%<br />
if echo $line | awk '{print $(NF-1)}' | grep "...:.."<br />
then<br />
# running for > 99min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
if echo $line | awk '{print $(NF-1)}' | grep "....m"<br />
then<br />
# running for > 1000min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
fi<br />
<br />
if echo $line | awk '{print $(NF-3)}' | grep [1-9]...<br />
then<br />
# running for 10-90 percent<br />
if echo $line | awk '{print $NF}' | egrep 'cfusion|counter|vchkpw'<br />
then<br />
<br />
if echo $line | awk '{print $(NF-1)}' | grep "[2-9]:.."<br />
then<br />
# between 2-9min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
elif echo $line | awk '{print $(NF-1)}' | grep "[0-9][0-9]:.."<br />
then<br />
# up to 99min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
fi<br />
fi</pre><br />
<br />
<br />
== Misc Linux Items ==<br />
<br />
We are overselling hard drive space ... when you configure a linux system with a certain amount of disk space (the default is 4gigs) you do not actually use up 4gigs of space on the system. The diskspace setting for a user is simply a cap, and they only use up as much space on the actual disk drive as they are actually using.<br />
<br />
When you create a new linux system, even though there are some 300 RPMs or so installed, if you run `df -k` you will see that the entire 4gig partition is empty - no space is being used. This is because the files in their system are "magic symlinks" to the template for their OS that is in /vz/template - however, any changes to any of those files will "disconnect" them and they will immediately begin using space in their system. Further, any new files uploaded (even if those new files overwrite existing files) will take up space on the partition.<br />
<br />
<br />
if you see this:<br />
<br />
<pre>[root@virt8 root]# vzctl stop 160 ; vzctl start 160<br />
VE is not running<br />
Starting VE ...<br />
VE is unmounted<br />
VE is mounted<br />
Adding IP address(es): 69.55.226.83 69.55.226.84<br />
bash ERROR: Can't change file /etc/sysconfig/network<br />
Deleting IP address(es): 69.55.226.83 69.55.226.84<br />
VE is unmounted<br />
[root@virt8 root]#</pre><br />
<br />
it probably means they no longer have /bin/bash - copy one in for them<br />
<br />
ALSO: another possibility is that they have removed the `ed` RPM from their system - it needs to be reinstalled into their system. But since their system is down, this is tricky ...<br />
<br />
VE startup scripts used by 'vzctl' want package 'ed' to be available inside VE. So if package 'ed' will be enabled in OS template config and OS template itself VE #827 is based on - this error should be fixed.<br />
<br />
yes, it is possible to add RPM to VE while it not running.<br />
Try to do following:<br />
<br />
<pre># cd /vz/template/<OS_template_with_ed_package>/<br />
# vzctl mount 827<br />
# rpm -Uvh --root /vz/root/827 --veid 827 ed-0.2-25.i386.vz.rpm</pre><br />
<br />
Usually theres an error, but its ok<br />
<br />
Note: replace 'ed-0.2-25.i386.vz.rpm' in last command with actual<br />
version of 'ed' package you have.<br />
<br />
----<br />
<br />
So how do I know what template the user has ? cat their conf file and it is listed in there. For example, if the conf file has:<br />
<br />
<pre>[root@virt12 sbin]# vc 1103<br />
…snip…<br />
OSTEMPLATE="debian-3.0/20030822"<br />
TEMPLATES="mod_perl-deb30/20030707 mod_ssl-deb30/20030703 mysql-deb30/20030707 proftpd-deb30/20030703 webmin-deb30/20030823 "<br />
…</pre><br />
<br />
then they are on debian 3.0, all of their system RPMs are in /vz/template/debian-3.0, and they are using version 20030822 of that debian 3.0 template. Also, they’ve also got additional packages installed (mod_perl, mod_ssl, etc). Those are also found under /vz/template<br />
<br />
----<br />
<br />
Edits needed to run java:<br />
<br />
When we first created the VEs, the default setting for privvmpages was 93000:94000 ... which was high enough that most people never had problems ... however, you can;t run java or jdk or tomcat or anything java related with that setting. We have found that by setting privvmpages to 610000:615000 that java runs just fine. That is now the default setting. It is exceedingly rare that anyone needs it higher than that, although we have seen it once or twice.<br />
<br />
Any problems with java at all - the first thing you need to do is see if the failcnt has raised for privvmpages.<br />
<br />
<pre># vzctl start 160<br />
Starting VE ...<br />
vzquota : (error) Quota on syscall for 160: Device or resource busy<br />
Running vzquota on failed for VE 160 [3]</pre><br />
<br />
This is because my pwd is _in_ their private directory - you can't start it until you move out<br />
<br />
People seem to have trouble with php if they are clueless newbies. Here are two common problems/solutions:<br />
<br />
no... but i figured it out myself. problem was the php.ini file that came<br />
vanilla with the account was not configured to work with apache (the<br />
ENGINE directive was set to off).<br />
<br />
everything else seems fine now.<br />
<br />
----<br />
<br />
the problem was in the php.ini file. I noticed that is wasnt showing<br />
the code when it was in an html file so I looked at the php.ini file<br />
and had to change it so it recognized <? tags aswell as <?php tags.<br />
<br />
Also, make sure added to httpd.conf<br />
AddType application/x-httpd-php .php<br />
<br />
----<br />
<br />
You can set the time zone:<br />
<br />
You can change the timezone by doing this:<br />
<br />
ln -sf /usr/share/zoneinfo/<zone> /etc/localtime<br />
<br />
where <zone> is the zone you want in the /usr/share/zoneinfo/ directory.<br />
<br />
----<br />
<br />
Failing shm_open calls:<br />
<br />
first, please check if /dev/shm is mounted inside VE.<br />
'cat /proc/mounts' command should show something like this:<br />
tmpfs /dev/shm tmpfs rw 0 0<br />
<br />
If /dev/shm is not mounted you have 2 ways to solve issue:<br />
1. execute following command inside VE (doesn't require VE reboot):<br />
mount -t tmpfs none /dev/shm<br />
2. add following string to /etc/fstab inside VE and reboot it:<br />
tmpfs /dev/shm tmpfs defaults 0 0<br />
<br />
----<br />
<br />
You can have a mounted but not running ve<br />
Just:<br />
vzctl mount <veid><br />
<br />
----<br />
<br />
When a debian sys can’t get on the network, and you try:<br />
<br />
<pre>vzctl set 1046 --ipadd 69.55.227.117<br />
Adding IP address(es): 69.55.227.117<br />
Failed to bring up lo.<br />
Failed to bring up venet0.</pre><br />
<br />
They probably removed iproute package, which must be the one from swsoft. To restore:<br />
<pre># dpkg -i --veid=1046 --admindir=/vz1/private/1046/root/var/lib/dpkg --instdir=/vz1/private/1046/root/ /vz/template/debian-3.0/iproute_20010824-8_i386.vz.deb<br />
(Reading database ... 16007 files and directories currently installed.)<br />
Preparing to replace iproute 20010824-8 (using .../iproute_20010824-8_i386.vz.deb) ...<br />
Unpacking replacement iproute ...<br />
Setting up iproute (20010824-8) ...</pre><br />
<br />
Then restart their ve<br />
<br />
----<br />
<br />
in a ve i do:<br />
<br />
<pre>cd /<br />
du -h .</pre><br />
<br />
and get: 483M .<br />
<br />
i do:<br />
<br />
<pre>bash-2.05a# df -h<br />
Filesystem Size Used Avail Use% Mounted on<br />
/dev/vzfs 4.0G 2.3G 1.7G 56% /</pre><br />
<br />
how can this be?<br />
<br />
Is it possible that quota file was corrupted somehow? Please try to: <br />
<pre>vzctl stop <VEID><br />
vzquota drop <VEID><br />
vzquota init <VEID><br />
vzctl start <VEID></pre><br />
<br />
----<br />
<br />
How to stop vz from starting after reboot:<br />
<br />
VIRTUOZZO=no <br />
in <br />
/etc/sysconfig/vz<br />
<br />
To start: <br />
service vz start<br />
(after setting VIRTUOZZO=yes in /etc/sysconfig/vz)<br />
<br />
service vz restart will do some kind of 'soft reboot' -- restart all<br />
VPSes and reload modules without rebooting the node<br />
<br />
if you need to shut down all VPSes really really fast, run killall -9 init<br />
<br />
----<br />
<br />
Postfix tip:<br />
<br />
You may want to tweak settings: default_process_limit=10<br />
<br />
= Virtuozzo VPS Management Tools =<br />
<br />
== vm ==<br />
<br />
TODO<br />
<br />
== cancelve ==<br />
cancelve <veid><br />
this will:<br />
* stop a ve<br />
* check for backups (offer to remove them from the backup server <br />
and the backup.config)<br />
* rename the private dir<br />
* check for PTR, provide the commands to reset to default<br />
* and rename the ve’s config<br />
* remind you to remove firewall rules<br />
* remind you to remove DNS entries<br />
<br />
== ipadd ==<br />
ipadd <veid> <ip> [ip] [ip]<br />
add’s ip(s) to a ve<br />
<br />
== ipdel ==<br />
ipdel <veid> <ip> [ip] [ip]<br />
removes ip(s) from a ve<br />
<br />
== vc ==<br />
vc <veid><br />
the equivalent of: <tt>cat /vzconf/<veid>.conf</tt><br />
<br />
== vl ==<br />
vl<br />
will displays a list of ve #’s, 1 per line. (ostensibly to use in a for loop)<br />
<br />
== vp ==<br />
vp <veid><br />
the equivalent of: <tt>vzps auxww –E <veid></tt><br />
<br />
== vpe ==<br />
DEPRECATED<br />
vpe <veid> <br />
this will allow you to do a vp when a ve is running out of control, the equivalent of (deprecated since vp operates outside the VPS): <br />
<pre>vzctl set <veid> --kmemsize 2100000:2200000<br />
vzctl exec <veid> ps auxw<br />
vzctl set <veid> --kmemsize (ve’s orig lvalue):(ve’s orig hvalue)</pre><br />
<br />
== vt ==<br />
vt <veid><br />
the equivalent of: <tt>vztop –E <veid></tt><br />
<br />
== vr ==<br />
vr <veid><br />
the equivalent of: <tt>vzctl stop <veid>; vzctl start <veid></tt><br />
You can run this even if the ve is down - the stop command will just fail<br />
<br />
== vs ==<br />
vs [veid]<br />
displays status (output of <tt>vzctl status <veid></tt>) on each ve configured on the system (as determined by <tt>/vzconf/*.conf</tt>)<br />
If passed an argument, gives the status for just that ve. <br />
A running system looks like:<br />
<tt>VEID 16066 exist mounted running</tt><br />
<br />
A ve which is not running (but does exist) looks like:<br />
<tt>VEID 9990 exist unmounted down</tt><br />
<br />
A ve which is not running and doesn’t exist looks like:<br />
<tt>VEID 421 deleted unmounted down</tt><br />
<br />
== vs2 ==<br />
vs2 [veid]<br />
this is similar to vs in that it displays status (output of <tt>vzctl status <veid></tt>) on each ve,<br />
but the difference is it’s list comes from doing an ls on the data dirs. This was meant to catch <br />
the rare case where a ve configured but exists. <br />
<br />
== vw ==<br />
vw [veid]<br />
displays the output of ‘<tt>w</tt>’ (the equivalent of <tt>vzctl exec <veid> w</tt>) for each configured ve (as determined by <tt>/vzconf/*.conf</tt>). Useful for determing which ve is contributing to a heavily-loaded system.<br />
If passed an argument, gives ‘<tt>w</tt>‘ output for just that ve. <br />
Ex:<br />
<pre>[root@virt2 etc]# vw<br />
134<br />
10:52pm up 79 days, 6:14, 0 users, load average: 0.02, 0.02, 0.00<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT<br />
16027<br />
2:52pm up 7 days, 19:54, 0 users, load average: 0.00, 0.00, 0.00<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT<br />
16055<br />
2:52pm up 79 days, 6:38, 0 users, load average: 0.00, 0.04, 0.07<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT</pre><br />
<br />
== vwe ==<br />
vwe [constraint]<br />
just like <tt>vw</tt>, but takes a constraint as an argument, only show’s ve’s with loads >= the constraint provided. If no constraint is provided, 1 is used by default<br />
<br />
== vzs ==<br />
vzs [veid]<br />
displays the beancounter status for all ve’s, or a particular ve if an argument is passed<br />
<br />
== ve ==<br />
ve <veid><br />
the equivalent of: <tt>vzctl enter <veid></tt><br />
<br />
== vx ==<br />
vx <veid> <command><br />
the equivalent of: <tt>/usr/sbin/vzctl exec <veid> <command></tt><br />
<br />
== dprocs ==<br />
dprocs [count]<br />
a script which outputs a continuous report (or a certain number of reports if an option is passed) of processes stuck in the D state and which VPS’s those procs belong to.<br />
<br />
== setmem ==<br />
setmem <VEID> <256|512|768|1024|2048><br />
adjusts the memory resources for the VE<br />
<br />
== afacheck.sh ==<br />
afacheck.sh<br />
displays the health/status of containers and mirrors on an adaptec card (currently quar1, tempvirt1-2, virt9, virt10)- all other are LSI<br />
<br />
== backup ==<br />
backup<br />
backup script called nightly to update virt scripts and do customer backups<br />
<br />
== checkload.pl ==<br />
checkload.pl<br />
this was intended to be setup as a cronjob to watch processes on a virt when the load <br />
rises above a certain level. Not currently in use.<br />
<br />
== findbackuppigs.pl ==<br />
findbackuppigs.pl<br />
looks for files larger than 50MB which customers have asked us to backup. Emails matches<br />
to linux@johncompanies.com<br />
<br />
== gatherlinux.pl ==<br />
gatherlinux.pl<br />
gathers up data about ve’s configured and writes to a file. Used for audits against the db<br />
<br />
== gb ==<br />
gb <search><br />
greps backup.config for the given search parameter<br />
<br />
== gbg ==<br />
gbg <search><br />
greps backup.config for the given search parameter and presents just the directories (for clean pasting)<br />
<br />
== linuxtrafficgather.pl ==<br />
linuxtrafficgather.pl [yy-mm]<br />
sends a traffic usage summary by ve to support@johncomapnies.com and payments@johncompanies.com.<br />
Optional arguments are year and month (must be in the past). If not passed, assumes last month. Relies on <br />
traffic logs created by netstatreset and netstatbackup<br />
<br />
== linuxtrafficwatch.pl ==<br />
linuxtrafficwatch.pl<br />
checks traffic usage and emails support@johncomapnies.com when a ve reaches the warning level (35G) and<br />
the limit (40G). Works on virtuozzo versions <= 2.6.x. We really aren’t using this anymore now that we have netflow.<br />
<br />
== linuxtrafficwatch2.pl ==<br />
linuxtrafficwatch2.pl<br />
checks traffic usage and emails support@johncomapnies.com when a ve reaches the warning level (35G) and<br />
the limit (40G). Works on virtuozzo version 2.6.x. We really aren’t using this anymore now that we have netflow.<br />
<br />
== load.pl ==<br />
load.pl<br />
feeds info to load mrtg - executed by inetd.<br />
<br />
== mb (linux) ==<br />
mb <mount|umount><br />
(nfs) mounts and umounts dirs to backup2. Shortcuts are mbm and mbu to mount and unmount. <br />
<br />
== migrate ==<br />
migrate <ip of node migrating to> <veid> <target_veid> [target dir: vz | vz1 | vz2]<br />
this is basically a wrapper for <tt>vzmigrate</tt> – vzmigrate is a util to seamlessly move a ve from one host to another. This wrapper was written cause vz virtuozzo version 2.6 had a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables. This script mitigates that. Since it makes multiple ssh connections to the target host, it’s a good idea to put the pub key for the src system in the authorized_keys file on the target host. In addition, it emails ve owners when their migration starts and stops (if they place email addresses in a file on their system: /migrate_notify). To move everyone off a system, you’d do:<br />
for f in `vl`; do migrate <ip> $f; done<br />
<br />
== migrateonline ==<br />
migrateonline <ip of node migrating to> <veid> <target_veid> [target dir: vz | vz1 | vz2]<br />
this is the same as migrate but will migrate a ve in <tt>–online</tt> mode which means it won’t be shut down at the end of the migration. This only works when migrating ve’s between 2 machines running a 2.6 kernel (currently tempvirt1-2. virt16-19, virt12). If you get an error that the machine you’re trying to migrate to has a different CPU or features, etc, then you have to edit the file and add the –f switch to the vzmigrate line- you can basically ignore this kind of warning (but never ignore a warning about missing templates on the destination node). NOTE: This edit (if made to migrateonline) will be overwritten by the base script during each night’s backup.<br />
<br />
== netstatbackup ==<br />
DEPRECATED<br />
netstatbackup <br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.5.x. <br />
<br />
== netstatbackup2 ==<br />
DEPRECATED<br />
netstatbackup2<br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.6.x. <br />
<br />
== netstatreset ==<br />
DEPRECATED<br />
netstatreset<br />
writes traffic count data to a logfile and resets counters to 0. Works on virtuozzo versions 2.5.x <br />
<br />
== netstatreset2 ==<br />
DEPRECATED<br />
netstatreset2<br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.6.x. <br />
<br />
== orphanedbackupwatchlinux ==<br />
orphanedbackupwatchlinux <br />
looks for directories on backup2 which aren’t configured in backup.config and offers to <br />
delete them<br />
<br />
== rsync.backup (linux) ==<br />
rsync.backup<br />
does customer backups (relies on backup.config)<br />
<br />
== startvirt.pl ==<br />
startvirt.pl<br />
forks off start ve commands – keeps 6 running at a time. This is not to be used on systems where fastboot is enabled as it circumvents the benefit of the fastboot. The script will occasionally not exit gracefully and will continue to use up CPU, so it should be watched. Also, don’t exit from the script till you’re sure all ve’s are started – if you do you need to start them manually and may have to free up locks. On some systems, the startvirt script doesn’t exit cleanly and you have to ^C out of it. Be careful though- doing so can leave some VE’s in an odd bootup state and you may need to ‘vr’ them manually. You should check to see which ve’s aren’t running and/or confirm all have started when ^C’ing out of startvirt.<br />
<br />
== taskdone (linux) ==<br />
taskdone<br />
when called will email support@johncompanies.com with the hostname of the machine from which it was <br />
executed as the subject<br />
<br />
== vb (linux) ==<br />
vb<br />
the equivalent of: <tt>vi /usr/local/sbin/backup.config</tt><br />
<br />
== vemakeXX ==<br />
DEPRECATED<br />
vemakerh9 <br />
ve create script for RH9 (see vemake)<br />
<br />
vemakedebian30 <br />
ve create script for debian 3.0 (Woody) (see vemake)<br />
<br />
vemakedebian31 <br />
ve create script for debian 3.1 (Sarge) (see vemake)<br />
<br />
vemakedebian40 <br />
ve create script for debian 4.0 (Etch) (see vemake)<br />
<br />
vemakefedora, vemakefedora2, vemakefedora4, vemakefedora5, vemakefedora6, vemakefedora7<br />
ve create script for fedora core 1, 2, 4, 5, 6, 7 respectively (see vemake)<br />
<br />
vemakecentos3, vemakecentos4<br />
ve create script for centos 3, 4 respectively (see vemake)<br />
<br />
vemakesuse, vemakesuse93, vemakesuse100<br />
ve create script for suse 9.2, 9.3, 10.0 respectively (see vemake)<br />
<br />
vemakeubuntu5, vemakeubuntu606, vemakeubuntu606 vemakeubuntu704<br />
ve create script for ubuntu 5.10, 6.06, 6.10, 7.04 respectively (see vemake)<br />
<br />
== vemove ==<br />
DEPRECATED<br />
vemove <veid> <target_ip> </vz/private/123><br />
this script simplifies the old way of moving ve’s from one system to another - in short moving a ve to or from a virt running virtuozzo < 2.6.x<br />
It’s the equivalent of:<br />
<tt>tar cfpP - <veid> --ignore-failed-read | (ssh -2 -c arcfour <target_ip> "split - -b 1024m </vz/private/123>.tar" )</tt>This should only be used if migrate/vzmigrate can’t be used. <br />
<br />
== vim.watchdog ==<br />
vim.watchdog <br />
looks for and kills procs matching “vi|vim|nano|pine|elm” that are running for a long time & consuming lots of cpu. Works on virtuozzo versions 2.5.x<br />
<br />
== vim.watchdog2 ==<br />
vim.watchdog2<br />
looks for and kills procs matching “vi|vim|nano|pine|elm” that are running for a long time & consuming lots of cpu.<br />
Works on virtuozzo versions 2.6.x.<br />
<br />
== vzmigrate ==<br />
vzmigrate <target_ip> -r no <veid>:[dst veid]:[dst /vzX/private/veid]:[dst /vzX/root/veid]<br />
(this is the raw command “wrapped” by migrate/migrateonline) this will seamlessly move a ve from one host to another. The ve will run for the duration of the migration till the very end when it’s shut down, ip moved and started up on the target system. The filesystem on the src will remain. This should be watched – occasionally the move will timeout and leave the system shut down. If target private and root aren’t specified it just puts it in /vz. Only works when both systems are running virtuozzo 2.6.x<br />
<br />
== vztrafdump.sh ==<br />
DEPRECATED<br />
vztrafdump.sh<br />
writes traffic usage info by ve to a file called jc_traffic_dump in each ve’s / dir. Works on virtuozzo versions <= 2.5.x. <br />
<br />
== vztrafdump2.sh ==<br />
DEPRECATED<br />
vztrafdump2.sh<br />
writes traffic usage info by ve to a file called jc_traffic_dump in each ve’s / dir. Works on virtuozzo versions 2.6.x. <br />
<br />
== addtun ==<br />
addtun <veid><br />
Add’s tun device to ve.<br />
<br />
== bwcap ==<br />
bwcap <veid> <kbps><br />
Ex: <tt>bwcap 1234 512</tt><br />
Caps a VE’s bandwidth to the amount given<br />
<br />
== setdisk ==<br />
setdisk <veid> <diskspace in GB><br />
Ex: <tt>setdisk 1234 5</tt><br />
Gives a VE’s a given amount of disk space<br />
<br />
== vdf ==<br />
vdf <veid> <br />
the equivalent of: <tt>vzctl exec <veid> df –h</tt><br />
<br />
== vdff ==<br />
vdff<br />
runs a (condensed) vdf for all ve’s in your pwd (must be run from /vz/privateN)<br />
<br />
== mvbackups ==<br />
mvbackups <veid> <target_machine> (virt1) <target_dir> (vz1)<br />
moves backups from one location to another on the backup server, and provides you with option to remove entries from current backup.config, and simple paste command to add the config to backup.config on the target server<br />
<br />
== checkquota ==<br />
checkquota<br />
for all the ve’s in the cwd (run from /vz/private, /vz1/private, etc) reports what vz quota says they’re using and what the actual usage is (as reported by du)<br />
<br />
== clearquota ==<br />
clearquota <veid><br />
Recalculates a ve’s quota, prints out the usage before and after. The equivalent of:<br />
<tt>vdf <veid>; v stop <veid>; vzquota drop <veid>; v start <veid>; vdf <veid></tt><br />
<br />
== dprocs ==<br />
dprocs<br />
Sometimes the server’s have a large number of processes get stuck in the D state- this script shows (every 3 secs) which VE’s have D procs, which procs<br />
are stuck and a running average of the top “offenders”<br />
<br />
== vzstat ==<br />
vstat<br />
sort of like top for VZ. sort VEs by CPU usage by pressing 'o' and then 'c' keys<br />
<br />
== stopvirt ==<br />
stopvirt<br />
will stop VEs as fast as it can, 6 at a time. May not exit when complete so you should watch [[#vzstat|vzstat]] in another window.</div>Supporthttps://wiki.jcihosting.com/index.php?title=Virtuozzo_/_Linux_Reference&diff=1018Virtuozzo / Linux Reference2013-03-11T23:07:11Z<p>Support: /* adding new templates */</p>
<hr />
<div>= Updating VZ =<br />
<br />
Update Virtuozzo to version 4.0.0<br />
Apply minor updates for Virtuozzo 3.0.0<br />
<br />
...<br />
When you get to the last screen, you will be given an option to <br />
<br />
( ) Automatically reboot after update<br />
(*) Reboot later manually<br />
<br />
make sure you choose to reboot later.<br />
<br />
= Migrating Templates =<br />
One nice feature VZ offers is the ability to run a virtual environment (VE or CT as it's now called) based on a particular ditribution on a host which may or may not share the same distribution. For instance, you could run a CT running Ubuntu while the host machine (Hardware Node or HN) is running CentOS. This is accomplished via the use of OS templates. As long as the HN contains the OS template on which a particular CT relies, it will run there. As such, if you want to move a CT from one HN to another, you will need to make sure the OS template exists there.<br />
<br />
Modern versions of VZ (>= 4.x) will check for and automatically move the OS template over to the host as you're (vz)migrating the CT over to a new HN. However we like to make sure the template is in place prior to moving the CT. <br />
<br />
The first step is to see if the template exists on the host. To see OS templates installed:<br />
<br />
2.x (shows OS and application templates):<br />
<pre># vzpkgls<br />
mod_ssl-rh9 20040624<br />
mysql-rh9 20030814 20050412<br />
openwebmail-rh9 20050427<br />
php-rh9 20050506<br />
phpBB-rh9 20041229<br />
postgresql-rh9 20050719<br />
sl-webalizer-rh9 20040119<br />
spamassassin-rh9 20040127<br />
tomcat-rh9 20040524<br />
usermin-rh9 20040116</pre><br />
<br />
>= 3.x (shows just OS templates, run without -O to see application as well):<br />
<pre># vzpkg list -O<br />
ubuntu-10.04-x86 2010-06-01 11:39:05<br />
ubuntu-11.04-x86 2011-06-22 14:23:59<br />
ubuntu-9.10-x86 2010-03-11 17:39:51<br />
centos-5-x86 2010-03-11 17:12:27<br />
debian-5.0-x86 2010-03-11 17:33:34</pre><br />
<br />
<br />
All template files are located:<br />
<br />
<pre># ls /vz/template/<br />
ColdFusion-fc1 jre-fc1 openwebmail-fc1 sl-webalizer-fc1<br />
ColdFusion-fc2 jre-fc2 openwebmail-fc2 sl-webalizer-fc2<br />
ColdFusion-rh9 jre-rh9 openwebmail-rh9 sl-webalizer-rh9<br />
PostNuke-fc1 jre-suse92 openwebmail-suse92 spamassassin-fc1<br />
PostNuke-fc2 jrun php-deb31 spamassassin-fc2<br />
PostNuke-rh9 mailman-deb31 php-fc1 spamassassin-rh9<br />
analog-fc1 mailman-fc1 php-fc2 spamassassin-suse92<br />
analog-fc2 mailman-fc2 php-rh9 suse-9.2<br />
analog-rh9 mailman-rh9 php-suse92 tomcat-fc1<br />
awstats-fc1 mailman-suse92 phpBB-fc1 tomcat-fc2<br />
awstats-rh9 mod_frontpage-fc1 phpBB-fc2 tomcat-rh9<br />
bbClone-fc1 mod_frontpage-fc2 phpBB-rh9 tomcat-suse92<br />
bbClone-fc2 mod_frontpage-rh9 phpmyadmin-deb31 urchin-fc1<br />
bbClone-rh9 mod_frontpage-suse92 postgresql-deb31 usermin-fc1<br />
conf mod_perl-deb31 postgresql-fc1 usermin-fc2<br />
debian-3.1 mod_perl-fc1 postgresql-fc2 usermin-rh9<br />
devel-deb31 mod_perl-fc2 postgresql-rh9 uw-imap-fc1<br />
devel-fc2 mod_perl-rh9 postgresql-suse92 uw-imap-fc2<br />
devel-suse92 mod_perl-suse92 proftpd-deb31 uw-imap-rh9<br />
fedora-core-1 mod_ssl-fc1 proftpd-fc1 vzredhat-7.3<br />
fedora-core-2 mod_ssl-fc2 proftpd-fc2 vzredhat-8.0<br />
jdk-deb31 mod_ssl-rh9 proftpd-rh9 webalizer-suse92<br />
jdk-fc1 mysql-deb31 proftpd-suse92 webmin-fc1<br />
jdk-fc2 mysql-fc1 redhat-7.2 webmin-fc2<br />
jdk-rh9 mysql-fc2 redhat-9 webmin-rh9<br />
jdk-suse92 mysql-rh9 redhat-as3-minimal<br />
jre-deb31 mysql-suse92 redhat-devel-9<br />
</pre><br />
<br />
What we see here are the base OS templates: <tt>redhat-9, fedora-core-1</tt> and supporting application templates: <tt>postgresql-rh9, mailman-rh9, jdk-fc1, mod_ssl-fc1</tt>. So if you wanted to copy over all of redhat 9 you'd need to copy the contents of:<br />
<pre># ls -d /vz/template/*rh9*<br />
/vz/template/ColdFusion-rh9 /vz/template/mod_frontpage-rh9 /vz/template/proftpd-rh9<br />
/vz/template/PostNuke-rh9 /vz/template/mod_perl-rh9 /vz/template/sl-webalizer-rh9<br />
/vz/template/analog-rh9 /vz/template/mod_ssl-rh9 /vz/template/spamassassin-rh9<br />
/vz/template/awstats-rh9 /vz/template/mysql-rh9 /vz/template/tomcat-rh9<br />
/vz/template/bbClone-rh9 /vz/template/openwebmail-rh9 /vz/template/usermin-rh9<br />
/vz/template/jdk-rh9 /vz/template/php-rh9 /vz/template/uw-imap-rh9<br />
/vz/template/jre-rh9 /vz/template/phpBB-rh9 /vz/template/webmin-rh9<br />
/vz/template/mailman-rh9 /vz/template/postgresql-rh9<br />
<br />
# ls -d /vz/template/*redhat-9*<br />
/vz/template/redhat-9<br />
</pre><br />
<br />
To get a template from one HN to another, it simply needs to be rsync'd over to the target HN. It's rsync'd over in this fashion:<br />
<br />
rsync -av -e ssh /vz/template/redhat-9 root@10.1.4.65:/vz/template<br />
rsync -av -e ssh /vz/template/*rh9 root@10.1.4.65:/vz/template<br />
<br />
Newer (>= 3.x) VZ employs "EZ" templates which are organized a little differently:<br />
<br />
<pre># ls /vz/template/<br />
cache CmdTool.log debian redhat-as3-minimal-20080630.tar.gz vc<br />
centos conf redhat-as3-minimal ubuntu<br />
<br />
# ls /vz/template/ubuntu/<br />
10.04 11.04 9.10<br />
# ls /vz/template/ubuntu/10.04/<br />
x86<br />
# ls /vz/template/ubuntu/10.04/x86/<br />
aap_1.091-1_all<br />
aap-doc_1.091-1_all<br />
adduser_3.112ubuntu1_all<br />
anacron_2.3-13.1ubuntu11_i386<br />
apache2_2.2.14-5ubuntu8.2_i386<br />
apache2_2.2.14-5ubuntu8.4_i386<br />
apache2_2.2.14-5ubuntu8.6_i386<br />
apache2_2.2.14-5ubuntu8.7_i386<br />
apache2_2.2.14-5ubuntu8.8_i386<br />
apache2_2.2.14-5ubuntu8.9_i386<br />
<...snip><br />
<br />
# ls /vz/template/cache/<br />
centos-5-x86.tar.gz suse-11.3-x86.tar.gz ubuntu-9.10-x86.tar.gz<br />
debian-5.0-x86.tar.gz ubuntu-10.04-x86.tar.gz<br />
fedora-core-14-x86.tar.gz ubuntu-11.04-x86.tar.gz<br />
</pre><br />
<br />
So what we see is the entire OS and all applications are in 1 directory, and organized by distribution and release version. So to move Ubuntu 10.04:<br />
<br />
rsync -av -e ssh /vz/template/ubuntu/10.04 root@10.1.4.69:/vz/template/ubuntu<br />
<br />
and you also need to move the cache:<br />
<br />
rsync -av -e ssh /vz/template/cache/ubuntu-10.04-x86.tar.gz root@10.1.4.69:/vz/template/cache/<br />
<br />
Once the transfer is complete, you will be able to run <tt>vzpkgls</tt> or <tt>vzpkg list -O</tt> and see the template(s) listed on the target HN.<br />
<br />
So far, this all supposes template doesn't exist on the target- very simple, just copy it over. If the template exists already on the target HN, this requires closer inspection. With older templates, simply moving over the templates would be the end of it- you see the version listed when you do a <tt>vzplgls</tt>:<br />
<br />
mysql-rh9 20030814 20050412<br />
<br />
this means there are 2 versions of the mysql package for RH9: 20030814 and 20050412<br />
As an aside, you can't copy over just 1 of them, but if you rsync'd over the <tt>mysql-rh9</tt> directory, you'd see 20030814 and 20050412 on the target HN. As long as the versions match up, you're good to go.<br />
<br />
With EZ templates, the versions are not as easy to decipher. When you look at the <tt>vzpkg list</tt> output, that simply tells you when a cache was created. A cache is simply a snapshot of all the data needed for the latest versions of the OS and it's applications at the time the cache was created. It is a date, and thus tells you nothing about the versions within. So for instance, if you had a cache for Ubuntu 10.04 from 2007 and you ran <tt>vzpkg create cache ubuntu-10.04-x86</tt> it would re-download the latest version of apache, mysql and so on. And any ''subsequent'' ubuntu 10.04 CT's created thereafter would use those newer versions, whereas older CT's based on 10.04 would use older templates. You can see how this works by looking at the files under:<br />
<br />
<pre># ls /vz/template/ubuntu/10.04/x86/<br />
aap_1.091-1_all<br />
aap-doc_1.091-1_all<br />
adduser_3.112ubuntu1_all<br />
anacron_2.3-13.1ubuntu11_i386<br />
apache2_2.2.14-5ubuntu8.2_i386<br />
apache2_2.2.14-5ubuntu8.4_i386<br />
apache2_2.2.14-5ubuntu8.6_i386<br />
apache2_2.2.14-5ubuntu8.7_i386<br />
apache2_2.2.14-5ubuntu8.8_i386<br />
apache2_2.2.14-5ubuntu8.9_i386<br />
<...snip><br />
</pre><br />
<br />
You can see that this template has in it 6 different versions of apache2. This means that this template was merged with other 10.04 templates and/or it has been cached a few times, each time a new apache was available. The unfortunate thing is it's almost impossible to know which CT's are using which version of apache so it's hard to try to prune this down and remove unwanted/unneeded applications.<br />
<br />
So, if you see another HN has Ubuntu 10.04 on it, you need to see/confirm that it has all the exact files your CT needs. You can run a test rsync to see what ''would'' be transferred (what's missing):<br />
<br />
rsync -avn --stats -e ssh /vz/template/ubuntu/10.04 root@10.1.4.69:/vz/template/ubuntu<br />
<br />
Based on the amount of data you get back from the rsync, you will be able to decide whether or not to remove the -n and allow the full transfer to go through. The downside to doing the transfer is the situation described above- you've permanently joined these templates and now you may have 10 versions of apache on the target HN. There's one other potential pitfall to this kind of merging- it will also potentially copy over shared files, for which there is no particular version, most of which are in <tt>/vz/template/ubuntu/10.04/x86/config</tt>: <br />
<br />
cat /vz/template/ubuntu/10.04/x86/config/os/default/repositories<br />
/vz/template/ubuntu/10.04/x86/config/os/default/repositories<br />
<br />
Here are some specific things to look out for. Case: copying centos-5 on virt19 to virt12. We dumped the rsync output to a file so we could inspect:<br />
<br />
[root@virt19 /vz/template]# rsync -an -e ssh /vz/template/centos/ root@10.1.4.62:/vz/template/centos/ > v12_c5<br />
<br />
(note, that can be dangerous, better to add on full path <tt>/vz/template/centos/5</tt>)<br />
<br />
So in the output file we see things like:<br />
<br />
<pre><br />
x86/base0/cachecookie<br />
x86/base0/primary.xml.gz <br />
x86/base0/primary.xml.gz.sqlite <br />
x86/base0/repomd.xml <br />
x86/base1/cachecookie <br />
x86/base1/filelists.xml.gz <br />
x86/base1/filelists.xml.gz.sqlite <br />
x86/base1/primary.xml.gz <br />
x86/base1/primary.xml.gz.sqlite <br />
x86/base1/repomd.xml<br />
x86/base2/cachecookie<br />
</pre><br />
<br />
Which make us nervous cause those are not versioned files, i.e.:<br />
5/x86/MAKEDEV-3.23-1.2.i386/usr/sbin/mksock<br />
<br />
So those files will replace existing files on v12 rather than the case of the latter where it's likely being added. So caution should be given and deference to whichever version is newer (which rsync should take care of, but that will depend on the date each file was created and perhaps not the actual data within).<br />
<br />
If we look further in the file we see:<br />
<br />
x86/psa-appvault-moodle-1.8-8202920080409011254.noarch/usr/local/psa/var/cgitory/moodle-1.<br />
9/htdocs/lib/editor/htmlarea/plugins/TableOperations/img/cell-split.gif<br />
<br />
and other files looking like:<br />
5/x86/plesk-base-9.0.1-cos5.build90090127.18.i586/etc/sw/keys/info<br />
<br />
Which means plesk is here. We don't need plesk on v12 (the CT moving over doesn't use it) so we rerun the rsync and exclude it:<br />
<br />
[root@virt19 /vz/template]# rsync -an -e ssh --exclude="plesk*" --exclude="psa*" /vz/template/centos/ root@10.1.4.62:/vz/template/centos/ > v12_c5<br />
<br />
and now it's much smaller:<br />
<br />
[root@virt19 /vz/template]# cat v12_c5|wc -l<br />
97104<br />
<br />
Before running with excludes it was:<br />
[root@virt19 /vz/template]# cat v12_c5|wc -l <br />
238238<br />
<br />
So again, look at what you're doing. Generally, combining templates is fine however.<br />
<br />
If you're happy with the merge, run the rsync again without the "-n" to let it actually do the transfer.<br />
<br />
Once done, don't forget to add the new template to the HN in Management -> Reference -> Templates<br />
<br />
= getting help =<br />
<br />
= vzup2date =<br />
TODO <br />
= adding new templates =<br />
TODO<br />
<br />
= Updating kernel on virt =<br />
<br />
We now use [[#vzup2date|vzup2date]] to update systems and kernels. What follows is what we used to do when we had to download kernels and install manually on old systems:<br />
<br />
<br />
1. install kernel and modules<br />
<pre># rpm -ivh vzkernel-enterprise-2.4.20-021stab028.12.777.i686.rpm \<br />
vzmodules-enterprise-2.4.20-021stab028.12.swsoft.i686.rpm<br />
vzkernel-smp ##################################################<br />
vzmodules-smp ##################################################</pre><br />
<br />
DO NOT USE the "rpm -Uhv" command to install the kernel, otherwise all the previously installed kernels may be removed from your system. <br />
<br />
2. update lilo/grub<br />
<br />
Lilo:<br />
<br />
vi /etc/lilo.conf<br />
<br />
rename old Virtuozzo kernel label from "linux+virtuozzo" to "virtuozzo_old"<br />
and add a new entry pointing to the newly installed virtuozzo kernel image.<br />
For example:<br />
<br />
<pre>prompt<br />
timeout=50<br />
default=linux+virtuozzo<br />
append="debug panic=5 console=tty console=ttyS0,38400"<br />
boot=/dev/sda<br />
map=/boot/map<br />
install=/boot/boot.b<br />
message=/boot/message<br />
linear<br />
<br />
image=/boot/vmlinuz-2.4.18-3smp<br />
label=linux<br />
initrd=/boot/initrd-2.4.18-3smp.img<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.18-3<br />
label=linux-up<br />
initrd=/boot/initrd-2.4.18-3.img<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-020stab009.5.777-enterprise<br />
label=2.4.20-9.5.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-020stab009.8.777-enterprise<br />
label=2.4.20-9.8.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-020stab009.21.777-enterprise<br />
label=2.4.20-9.21.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-020stab009.24.777-enterprise<br />
label=2.4.20-9.24.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-021stab028.3.777-enterprise<br />
label=2.4.20-28.3.777<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-021stab028.5.777-enterprise<br />
label=old_linux+vz<br />
read-only<br />
root=/dev/sda1<br />
<br />
image=/boot/vmlinuz-2.4.20-021stab028.12.777-enterprise<br />
label=linux+virtuozzo<br />
read-only<br />
root=/dev/sda1</pre><br />
<br />
With this configuration, the new kernel is the default kernel to use at boot. The original kernel entry has been retained with the label "old_linux+vz ".<br />
<br />
Grub:<br />
<br />
vi /boot/grub /menu.lst<br />
<br />
Add new entry at the top.<br />
For example:<br />
<br />
<pre>serial --unit=0 --speed=38400<br />
terminal --timeout=10 serial console<br />
default=0<br />
timeout=10<br />
splashimage=(hd0,0)/boot/grub/splash.xpm.gz<br />
title Virtuozzo 2.6.1 (2.4.20-021stab028.12.777-enterprise)<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-22.4.20-021stab028.12.777-enterprise root=/dev/sda1 console=tty0 \ console=ttyS0,38400<br />
title Virtuozzo 2.6.1 (2.4.20-021stab028.3.777-enterprise)<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.20-021stab028.3.777-enterprise root=/dev/sda1 console=tty0 \ console=ttyS0,38400<br />
title Red Hat Linux (2.4.18-3smp)<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.18-3smp ro root=/dev/sda1<br />
initrd /boot/initrd-2.4.18-3smp.img<br />
title Red Hat Linux-up (2.4.18-3)<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.18-3 ro root=/dev/sda1<br />
initrd /boot/initrd-2.4.18-3.img<br />
title Linux with Virtuozzo 2.5<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.20-020stab009.3.777-smp ro root=/dev/sda1<br />
title vz-enterpriseo<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.20-020stab009.21.777-enterprise ro root=/dev/sda1<br />
title vz-enterprise<br />
root (hd0,0)<br />
kernel /boot/vmlinuz-2.4.20-020stab009.24.777-enterprise ro root=/dev/sda1 \ console=tty0 console=ttyS0,38400</pre><br />
<br />
3. run the lilo (if using lilo)<br />
<br />
<pre># lilo<br />
Added linux<br />
Added linux-up<br />
Added virtuozzo_old<br />
Added linux+virtuozzo *</pre><br />
<br />
4. reboot the server<br />
shutdown -r 23:05<br />
<br />
when it comes time for the shutdown, run [[VPS_Management#stopvirt|stopvirt]] to get things shut down faster</div>Supporthttps://wiki.jcihosting.com/index.php?title=VPS_Management&diff=1017VPS Management2013-03-11T23:04:12Z<p>Support: /* Remaking a system (on same virt) */</p>
<hr />
<div>= Common Problems =<br />
== Login to any machine without a password ==<br />
<br />
This is possible via the use of ssh keys. The process is thus:<br />
<br />
1. place the public key for your user (root@mail) in the /root/.ssh/authorized_keys file on the server you wish to login to<br />
cat /root/.ssh/id_dsa.pub<br />
(paste that into authorized_keys on the target server). If the file doesn't exist, create it.<br />
<br />
2. enable root login (usually only applies to FreeBSD). Edit the /etc/ssh/sshd_config on the target server and change:<br />
<tt>#PermitRootLogin no</tt><br />
to<br />
<tt>PermitRootLogin yes</tt><br />
<br />
3. Restart the sshd on the target machine. First, find the sshd process: <br />
jailps <hostname> | grep sshd <br />
or <br />
vp <VEID> | grep sshd<br />
<br />
Look for the process resembling:<br />
root 17296 0.0 0.0 5280 1036 ? Ss 2011 4:27 /usr/sbin/sshd <br />
(this is the sshd)<br />
<br />
Not:<br />
root 6270 0.5 0.0 6808 2536 ? Ss 14:33 0:00 sshd: root [priv]<br />
(this is an sshd child- someone already ssh'd in as root)<br />
<br />
Restart the sshd: <br />
kill -1 <PID><br />
<br />
Ex:<br />
kill -1 17296<br />
<br />
You may now ssh in.<br />
<br />
Once you're done, IF you enabled root login, you should repeat steps 2 and 3 to disable root logins.<br />
<br />
== Letting someone in who has locked themselves out (killed sshd, lost pwd) ==<br />
<br />
There are two ways people frequently lock themselves out - either they forget a password, or they kill off sshd somehow.<br />
<br />
These are actually both fairly easy to solve. First, let's say someone kills off their sshd, or somehow mangles /etc/ssh/sshd_config such that it no longer lets them in.<br />
<br />
Their email may be very short, or it may have all sorts of details about how you should fix sshd_config to let them in ... just ignore all of this. They can fix their own mangled sshd. Fixing this is very simple. First, edit the /etc/inetd.conf on their system and uncomment the telnet line:<br />
<br />
telnet stream tcp nowait root /usr/libexec/telnetd telnetd<br />
#telnet stream tcp6 nowait root /usr/libexec/telnetd telnetd<br />
<br />
(just leave the tcp6 version of telnet commented)<br />
<br />
Then, use jailps to list the processes on their system, and find their inetd process. Then simply:<br />
<br />
kill -HUP (pid)<br />
<br />
where (pid) is the PID of their inetd process. Now they have telnet running on their system and they can log in and do whatever they need to do.<br />
<br />
The only complications that could occur are:<br />
<br />
a) their firewall config on our firewall has port 23 blocked, in which case you will need to open that - will be covered in a different lesson.<br />
<br />
b) they are not running inetd, so you can't HUP it. If this happens, edit their /etc/rc.conf, add the inetd_enable="YES" line, and then kill<br />
their jail with /tmp/jailkill.pl - then restart their jail with the jail line from their quad/safe file. Easy.<br />
<br />
If they have forgotten a password,<br />
<br />
On 6.x+ you can reset their password with:<br />
jexec <jailID from jls> passwd root<br />
<br />
Note: the default password for 6.x jails is 8ico2987, for 4.x it is p455agfa<br />
<br />
On 4.x, you need to cd to their etc directory<br />
... for instance:<br />
<br />
cd /mnt/data2/198.78.65.136-col00261-DIR/etc<br />
<br />
and run:<br />
<br />
vipw -d .<br />
<br />
Then paste in these two lines (theres a paste with these):<br />
<br />
root:$1$krszPxhk$xkCepSnz3mIikT3vCtJCt0:0:0::0:0:Charlie &:/root:/bin/csh<br />
user:$1$Mx9p5Npk$QdMU6c8YQqp2FW2M3irEh/:1001:1001::0:0:User &:/home/user:/bin/sh<br />
<br />
overwriting the lines they already have for "user" and "root" - then just tell them that both user and root have been reset to the default password of p455agfa.<br />
<br />
For linux, just passwd inside shell or <br />
vzctl set <veid> --userpasswd root:p455agfa –save<br />
<br />
Starting in 2009 we began giving out randomized passwords for FreeBSD and Linux as the default password. That is stored with each system in Mgmt. You should look for and reset the password to that password in the event of a reset and refer the customer to use their original password from their welcome email- this way we don’t have to send the password again via email (in clear text).<br />
<br />
<br />
== sendmail can’t be contacted from ext ip (only locally) ==<br />
<br />
By default redhat puts this line in sendmail.mc:<br />
<br />
DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')<br />
<br />
which makes it only answer on localhost. Comment it out like:<br />
<br />
dnl DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')<br />
<br />
and then rebuild sendmail.cf with:<br />
<br />
m4 /etc/mail/sendmail.mc > /etc/sendmail.cf<br />
<br />
== virt doesn’t properly let go of ve’s ip(s) when moved to another system ==<br />
<br />
On virtuozzo 2.6 systems, it's been observed that when moving ips from one virt to another that sometimes the routing table will not get updated to reflect the removal of the ip addresses.<br />
<br />
A recent example was a customer that was moving to a new ve on a new virt and the ip addresses were traded between the two ve's. After the trade the two systems were not able to talk to each other. When looking at the routing table for the old system all the ip addresses were still in the routing table as being local, like so:<br />
<br />
<pre>netstat -rn | grep 69.55.225.149<br />
69.55.225.149 0.0.0.0 255.255.255.255 UH 40 0 0 venet0</pre><br />
<br />
This was preventing traffic to the other system from being routed properly.<br />
The solution is to manually delete the route:<br />
<br />
route delete 69.55.225.149 gw 0.0.0.0<br />
<br />
Supposedly, this was fixed in 2.6.1<br />
<br />
== sshd on FreeBSD 6.2 segfaults ==<br />
<br />
First try to reinstall ssh<br />
<br />
<pre>cd /usr/src/secure<br />
cd lib/libssh<br />
make depend && make all install<br />
cd ../../usr.sbin/sshd<br />
make depend && make all install<br />
cd ../../usr.bin/ssh<br />
make depend && make all install</pre><br />
<br />
Failing that, find the library that’s messed up:<br />
<br />
<pre>ldd /usr/sbin/sshd<br />
libssh.so.3 => /usr/lib/libssh.so.3 (0x280a3000) <br />
libutil.so.5 => /lib/libutil.so.5 (0x280d8000) <br />
libz.so.3 => /lib/libz.so.3 (0x280e4000) <br />
libwrap.so.4 => /usr/lib/libwrap.so.4 (0x280f5000) <br />
libpam.so.3 => /usr/lib/libpam.so.3 (0x280fc000) <br />
libbsm.so.1 => /usr/lib/libbsm.so.1 (0x28103000) <br />
libgssapi.so.8 => /usr/lib/libgssapi.so.8 (0x28112000) <br />
libkrb5.so.8 => /usr/lib/libkrb5.so.8 (0x28120000) <br />
libasn1.so.8 => /usr/lib/libasn1.so.8 (0x28154000) <br />
libcom_err.so.3 => /usr/lib/libcom_err.so.3 (0x28175000) <br />
libroken.so.8 => /usr/lib/libroken.so.8 (0x28177000) <br />
libcrypto.so.4 => /lib/libcrypto.so.4 (0x28183000) <br />
libcrypt.so.3 => /lib/libcrypt.so.3 (0x28276000) <br />
libc.so.6 => /lib/libc.so.6 (0x2828e000) <br />
libmd.so.3 => /lib/libmd.so.3 (0x28373000)</pre><br />
<br />
md5 them and compare to other jail hosts or jails running on host<br />
<br />
for libcrypto reinstall:<br />
<pre>/usr/src/crypto<br />
make depend && make all install</pre><br />
<br />
= FreeBSD VPS =<br />
<br />
== Starting jails: Quad/Safe Files ==<br />
<br />
FreeBSD customer systems do not start up automatically at boot time. When one of our freebsd machines boots up, it boots up, and does nothing else. To start jails, we put the commands to start each jail into a shell script(s) and run the script(s). Jail startup is something that needs to be actively monitored, which is why we don’t just run the script automatically. More on monitoring later.<br />
<br />
NOTE: >=7.x we have moved to 1 quad file: <tt>quad1</tt>. Startups are not done by running each quad, but rather [[#startalljails|startalljails]] which relies on the contents of <tt>quad1</tt>. The specifics of this are lower in this article. What follows here applies for pre 7.x systems.<br />
<br />
There are eight files in <tt>/usr/local/jail/rc.d</tt>:<br />
<br />
<pre>jail3# ls /usr/local/jail/rc.d/<br />
quad1 quad2 quad3 quad4 safe1 safe2 safe3 safe4<br />
jail3#</pre><br />
<br />
four quad files and four safe files.<br />
<br />
Each file contains an even number of system startup blocks (total number of jails divided by 4)<br />
<br />
The reason for this is, if we make one large script to startup all the systems at boot time, it will take too long - the first system in the script will start up right after system boot, which is great, but the last system may not start for another 20 minutes.<br />
<br />
Since there is no way to parralelize this during the startup procedure, we simply open four terminals (in screen window 9) and run each script, one in each terminal. This way they all run simultaneously, and the very last system in each startup script gets started in 1/4th the time it would if there was one large file<br />
<br />
The files are generally organized so that quad/safe 1&2 have only jails from disk 1, and quad/safe 3&4 have jails from disk 2. This helps ensure that only 2 fscks on any disk are going on at once. Further, they are balanced so that all quad/safe’s finish executing around the same time. We do this by making sure each quad/safe has a similar number of jails and represents a similar number of inodes (see js).<br />
<br />
The other, very important reason we do it this way, and this is the reason there are quad files and safe files, is that in the event of a system crash, every single vn-backed filesystem that was mounted at the time of system crash needs to be fsck'd. However, fsck'ing takes time, so if we shut the system down gracefully, we don't want to fsck.<br />
<br />
Therefore, we have two sets of scripts - the four quad scripts are identical to the four safe scripts except for the fact that the quad scripts contain fsck commands for each filesystem.<br />
<br />
So, if you shut a system down gracefully, start four terminals and run safe1 in window one, and safe2 in window 2, and so on.<br />
<br />
If you crash, start four terminals (or go to screen window 9) and run quad1 in window one, and quad2 in window 2, and so on.<br />
<br />
Here is a snip of (a 4.x version) quad2 from jail17:<br />
<br />
<pre>vnconfig /dev/vn16 /mnt/data2/69.55.228.7-col00820<br />
fsck -y /dev/vn16<br />
mount /dev/vn16c /mnt/data2/69.55.228.7-col00820-DIR<br />
chmod 0666 /mnt/data2/69.55.228.7-col00820-DIR/dev/null<br />
jail /mnt/data2/69.55.228.7-col00820-DIR mail1.phimail.com 69.55.228.7 /bin/sh /etc/rc<br />
<br />
# moved to data2 col00368<br />
#vnconfig /dev/vn28 /mnt/data2/69.55.236.132-col00368<br />
#fsck -y /dev/vn28<br />
#mount /dev/vn28c /mnt/data2/69.55.236.132-col00368-DIR<br />
#chmod 0666 /mnt/data2/69.55.236.132-col00368-DIR/dev/null<br />
#jail /mnt/data2/69.55.236.132-col00368-DIR limehouse.org 69.55.236.132 /bin/sh /etc/rc<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
vnconfig /dev/vn22 /mnt/data2/69.55.228.13-col01063<br />
fsck -y /dev/vn22<br />
mount /dev/vn22c /mnt/data2/69.55.228.13-col01063-DIR<br />
chmod 0666 /mnt/data2/69.55.228.13-col01063-DIR/dev/null<br />
jail /mnt/data2/69.55.228.13-col01063-DIR www.widestream.com.au 69.55.228.13 /bin/sh /etc/rc<br />
<br />
# cancelled col00106<br />
#vnconfig /dev/vn15 /mnt/data2/69.55.238.5-col00106<br />
#fsck -y /dev/vn15<br />
#mount /dev/vn15c /mnt/data2/69.55.238.5-col00106-DIR<br />
#chmod 0666 /mnt/data2/69.55.238.5-col00106-DIR/dev/null<br />
#jail /mnt/data2/69.55.238.5-col00106-DIR mail.azebu.net 69.55.238.5 /bin/sh /etc/rc</pre><br />
<br />
As you can see, two of the systems specified are commented out - presumably those customers cancelled, or were moved to new servers.<br />
<br />
As you can see, the vnconfig line is the simpler command line, not the longer one that was used when it was first configured. As you can see, all that is done is, vnconfig the filesystem, then fsck it, then mount it. The fourth command is the `jail` command used to start the system – but that will be covered later.<br />
<br />
Here is the safe2 file from jail17:<br />
<br />
<pre>vnconfig /dev/vn16 /mnt/data2/69.55.228.7-col00820<br />
mount /dev/vn16c /mnt/data2/69.55.228.7-col00820-DIR<br />
chmod 0666 /mnt/data2/69.55.228.7-col00820-DIR/dev/null<br />
jail /mnt/data2/69.55.228.7-col00820-DIR mail1.phimail.com 69.55.228.7 /bin/sh /etc/rc<br />
<br />
# moved to data2 col00368<br />
#vnconfig /dev/vn28 /mnt/data2/69.55.236.132-col00368<br />
#mount /dev/vn28c /mnt/data2/69.55.236.132-col00368-DIR<br />
#chmod 0666 /mnt/data2/69.55.236.132-col00368-DIR/dev/null<br />
#jail /mnt/data2/69.55.236.132-col00368-DIR limehouse.org 69.55.236.132 /bin/sh /etc/rc<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
vnconfig /dev/vn22 /mnt/data2/69.55.228.13-col01063<br />
mount /dev/vn22c /mnt/data2/69.55.228.13-col01063-DIR<br />
chmod 0666 /mnt/data2/69.55.228.13-col01063-DIR/dev/null<br />
jail /mnt/data2/69.55.228.13-col01063-DIR www.widestream.com.au 69.55.228.13 /bin/sh /etc/rc<br />
<br />
# cancelled col00106<br />
#vnconfig /dev/vn15 /mnt/data2/69.55.238.5-col00106<br />
#mount /dev/vn15c /mnt/data2/69.55.238.5-col00106-DIR<br />
#chmod 0666 /mnt/data2/69.55.238.5-col00106-DIR/dev/null<br />
#jail /mnt/data2/69.55.238.5-col00106-DIR mail.azebu.net 69.55.238.5 /bin/sh /etc/rc</pre><br />
<br />
As you can see, it is exactly the same, but it does not have the fsck lines.<br />
<br />
Take a look at the last entry - note that the file is named:<br />
<br />
/mnt/data2/69.55.238.5-col00106<br />
<br />
and the mount point is named:<br />
<br />
/mnt/data2/69.55.238.5-col00106-DIR<br />
<br />
This is the general format on all the FreeBSD systems. The file is always named:<br />
<br />
IP-custnumber<br />
<br />
and the directory is named:<br />
<br />
IP-custnumber-DIR<br />
<br />
If you run safe when you need a fsck, the mount will fail and jail will fail:<br />
<br />
# mount /dev/vn1c /mnt/data2/jails/65.248.2.131-ns1.kozubik.com-DIR<br />
mount: /dev/vn1c: Operation not permitted<br />
<br />
No reboot needed, just run the quad script<br />
<br />
Starting with 6.x jails, we added block delimiters to the quad/safe files, the block looks like:<br />
<br />
<pre>echo '## begin ##: nuie.solaris.mu'<br />
fsck -y /dev/concat/v30v31a<br />
mount /dev/concat/v30v31a /mnt/data1/69.55.228.218-col01441-DIR<br />
mount_devfs devfs /mnt/data1/69.55.228.218-col01441-DIR/dev<br />
devfs -m /mnt/data1/69.55.228.218-col01441-DIR/dev rule -s 3 applyset<br />
jail /mnt/data1/69.55.228.218-col01441-DIR nuie.solaris.mu 69.55.228.218 /bin/sh /etc/rc<br />
echo '## end ##: nuie.solaris.mu'</pre><br />
<br />
These are more than just informative when running quad/safe’s, the echo lines MUST be present for certain tools to work properly. So it’s important that any updates to the hostname also be updated on the 2 echo lines. For example, if you try to startjail a jail with a hostname which is on the jail line but not the echo lines, the command will return with host not found.<br />
<br />
=== FreeBSD 7.x+ notes ===<br />
<br />
Starting with the release of FreeBSD 7.x, we are doing jail startups in a slightly different way. First, thereis only 1 file: <tt>/usr/local/jail/rc.d/quad1</tt><br />
There are no other quads or corresponding safe files. The reason for this is twofold, 1. We can pass –C to fsck which will tell is to skip the fsck if the fs is clean (no more need for safe files), 2. We have a new startup script which can be launched multiple times, running in parallel to start jails, where quad1 is the master jail file. <br />
Quad1 could still be run as a shell script, but it would take a very long time for it to run completely so it’s not advisable; or you should break it down into smaller chunks (like quad1, quad2, quad3, etc)<br />
<br />
Here is a snip of (a 7.x version) quad1 from jail2:<br />
<br />
<pre>echo '## begin ##: projects.tw.com'<br />
mdconfig -a -t vnode -f /mnt/data1/69.55.230.46-col01213 -u 50<br />
fsck -Cy /dev/md50c<br />
mount /dev/md50c /mnt/data1/69.55.230.46-col01213-DIR<br />
mount -t devfs devfs /mnt/data1/69.55.230.46-col01213-DIR/dev<br />
devfs -m /mnt/data1/69.55.230.46-col01213-DIR/dev rule -s 3 applyset<br />
jail /mnt/data1/69.55.230.46-col01213-DIR projects.tw.com 69.55.230.46 /bin/sh /etc/rc<br />
echo '## end ##: projects.tw.com'</pre><br />
<br />
Cancelled jails are no longer commented out and stored in quad1, rather they’re moved to <tt>/usr/local/jail/rc.d/deprecated</tt> <br />
<br />
To start these jails, start the 4 ssh sessions as you would for a normal crash and then instead of running quad1-4, instead run startalljails in each window. IMPORTANT- before running startalljails you should make sure you ran preboot once as it will clear out all the lockfiles and enable startalljails to work properly.<br />
<br />
== Problems with the quad/safe files ==<br />
<br />
When you run the quad/safe files, there are two problems that can occur - either a particular system will hang during initialization, OR a system will spit out output to the screen, impeding your ability to do anything. Or both.<br />
<br />
First off, when you start a jail, you see output like this:<br />
<br />
<pre>Skipping disk checks ...<br />
adjkerntz[25285]: sysctl(put_wallclock): Operation not permitted<br />
Doing initial network setup:.<br />
ifconfig: ioctl (SIOCDIFADDR): permission denied<br />
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384<br />
Additional routing options: TCP keepalive=YESsysctl:<br />
net.inet.tcp.always_keepalive: Operation not permitted.<br />
Routing daemons:.<br />
Additional daemons: syslogd.<br />
Doing additional network setup:.<br />
Starting final network daemons:.<br />
ELF ldconfig path: /usr/lib /usr/lib/compat /usr/X11R6/lib /usr/local/lib<br />
a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout /usr/X11R6/lib/aout<br />
Starting standard daemons: inetd cron sshd sendmail sendmail-clientmqueue.<br />
Initial rc.i386 initialization:.<br />
Configuring syscons: blanktime.<br />
Additional ABI support:.<br />
Local package initialization:.<br />
Additional TCP options:.</pre><br />
<br />
Now, let's look at this line, near the end:<br />
<br />
Local package initialization:.<br />
<br />
This is where a list of daemons that are set to start at boot time willshow up. You might see something like:<br />
<br />
Local package initialization: mysqld apache sendmail sendmail-clientmqueue<br />
<br />
Or something like this:<br />
<br />
Local package initialization: postgres postfix apache<br />
<br />
The problem is that many systems (about 4-5 per machine) will hang on that line. Basically it will get to some of the way through the total daemons to be started:<br />
<br />
Local package initialization: mysqld apache<br />
<br />
and will just sit there. Forever.<br />
<br />
Fortunately, pressing ctrl-c will break out of it. Not only will it break out of it, but it will also continue on that same line and start the other daemons:<br />
<br />
Local package initialization: mysqld apache ^c sendmail-clientmqueue<br />
<br />
and then continue on to finish the startup, and then move to the next system to be started.<br />
<br />
So what does this mean? It means that if a machine crashes, and you start four screen-windows to run four quads or four safes, you need to periodically cycle between them and see if any systems are stuck at that point, causing their quad/safe file to hang. A good rule of thumb is, if you see a system at that point in the startup, give it another 100 seconds - if it is still at the exact same spot, hit ctrl-c. Its also a good idea to go back into the quad file (just before the first command in the jail startup block) and note that this jail tends to need a control-c or more time as follows:<br />
<pre>echo '### NOTE ### slow sendmail'<br />
echo '### NOTE ###: ^C @ Starting sendmail.'</pre><br />
<br />
'''NEVER''' hit ctrl-c repeatedly if you don't get an immediate response - that will cause the following jail’s startup commands to be aborted.<br />
<br />
A second problem that can occur is that a jail - maybe the first one in that particular quad/safe, maybe the last one, or maybe one in the middle, will start spitting out status or error messages from one of its init scripts. This is not a problem - basically, hit enter a few times and see if you get a prompt - if you do get a prompt, that means that the quad/safe script has already completed. Therefore it is safe to log out (and log out of the user that you su'd from) and then log back in (if necessary).<br />
<br />
The tricky thing is, if a system in the middle starts flooding with messages, and you hit enter a few times and don't get a prompt. Are you not getting a prompt because some subsequent system is hanging at the initialization, as we discussede above ? Or are you not getting a prompt because that quad file is currently running an fsck ? Usually you can tell by scrolling back in screen’s history to see what it was doing before you started getting the messages.<br />
<br />
If you don’t get clues from history, you have to use your judgement - instead of giving it 100 seconds to respond, perhaps give it 2-3 mins ... if you still get no response (no prompt) when you hit enter, hit ctrl-c. However, be aware that you might still be hitting ctrl-c in the middle of an fsck. This means you will get an error like "filesystem still marked dirty" and then the vnconfig for it will fail and so will the jail command, and the next system in the quad file will then start starting up.<br />
<br />
If this happens, just wait until the end of all the quad files have finished, and start that system manually.<br />
<br />
If things really get weird, like a screen flooded with errors, and you can't get a prompt, and ctrl-c does nothing, then you need to just eventually (give it ten mins or so) just kill that window with ctrl-p, then k, and then log in again and manually check which systems are now running and which aren't, and manually start up any that are not.<br />
<br />
Don't EVER risk running a particular quad/safe file a second time.<br />
If the quad/safe script gets executed twice, reboot the machine immediately.<br />
<br />
So, for all the above reasons, anytime a machine crashes and you run all the quads or all the safes, '''always''' check every jail afterwards to make sure it is running - even if you have no hangs or complications at all.<br />
Run this command:<br />
<br />
<tt>[[#jailpsall|jailpsall]]</tt> <br />
<br />
Note: [[#postboot|postboot]] also populates ipfw counts, so it '''should not be run multiple times''', use <tt>jailpsall</tt> for subsequent extensive ps’ing<br />
<br />
And make sure they all show as running. If one does not show as running, check its /etc/rc.conf file first to see if maybe it is using a different hostname first before starting it manually.<br />
<br />
One thing we have implemented to alleviate these startup hangs and noisy jails, is to put jail start blocks that are slow or hangy at the bottom of the safe/quad file. Further, for each bad jail we note in each quad/safe just before the start block something like:<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
<br />
That way we’ll be prepared to ^C when we see that message appear during the quad/safe startup process. If you observe a new, undocumented hang, '''after''' the quad/safe has finished, place a line similar to the above in the quad file, move the jail start block to the end of the file, then run [[#buildsafe|buildsafe]]<br />
<br />
<br />
== Recovering from a crash (FreeBSD) ==<br />
<br />
=== Diagnose whether you have a crash ===<br />
The most important thing is to get the machine and all jails back up as soon as possible. Note the time, you’ll need to create a crash log entry (Mgmt. -> Reference -> CrashLog). The first thing to do is head over to the [[Screen#Screen_Organization|serial console screen]] and see if there’s any kernel error messages output. Try to copy any messages (or just a sample of repeating messages) you see into the notes section of the crash log. If there are no messages, the machine may just be really busy- wait a bit (5-10min) to see if it comes back. If it's still pinging, odds are its very busy. Note, if you see messages about swap space exhausted, the server is obviously out of memory, however it may recover briefly enough for you to get a jtop in to see who's lauched a ton of procs (most likely) and then issue a quick jailkill to get it back under control.<br />
<br />
If it doesn't come back, or the messages indicate a fatal error, you will need to proceed with a power cycle (ctrl+alt+del will not work).<br />
<br />
=== Power cycle the server ===<br />
If this machine is not a Dell 2950 with a [[DRAC/RMM#DRAC|DRAC card]] (i.e. if you can’t ssh into the DRAC card (as root, using the standard root pass) and issue <br />
racadm serveraction hardreset<br />
then you will need someone at the data center power the macine off, wait 30 sec, then turn it back on. Make sure to re-attach via console:<br />
tip jailX<br />
immediately after power down. <br />
<br />
=== (Re)attach to the console ===<br />
Stay on the console the entire time during boot. As the BIOS posts- look out for the RAID card output- does everything look healthy? The output may be scrambled, look for "DEGRADED" or "FAILED". Once the OS starts booting you will be disconnected (dropped back to the shell on the console server) a couple times during the boot up. The reason you want to quickly re-attach is two-fold: 1. If you don’t reattach quickly then you won’t get any console output, 2. you want to be attached before the server ''potentially'' starts (an extensive) fsck. If you attach after the fsck begins, you’ll have seen no indication it started an fsck and the server will appear frozen during startup- no output, no response. <br />
<br />
IMPORTANT NOTE: on some older FreeBSD systems, there will be no output to the video (KVM) console as it boots up. The console output is redirected to the serial port ... so if a jail crashes, and you attach a kvm, the output during the bootup procedure will not be shown on the screen. However, when the bootup is done, you will get a login prompt on the screen and will be able to log in as normal. <tt>/boot/loader.conf</tt> is where serial console redirect output lives, so comment that if you want to catch output on kvm.<br />
On newer systems it sends most output to both locations. <br />
<br />
=== Assess the heath of the server ===<br />
Once the server boots up fully, you should be able to ssh in. Look around- make sure all the mounts are there and reporting the correct size/usage (i.e. /mnt/data1 /mnt/data2 /mnt/data3 - look in /etc/fstab to determine which mount points should be there), check to see if RAID mirrors are healthy. See [[RAID_Cards#Common_CLI_commands_.28megacli.29|megacli]], [[#aaccheck|aaccheck]]<br />
<br />
Before you start the jails, you need to run [[#preboot|preboot]]. This will do some assurance checks to make sure things are prepped to start the jails. Any issues that come out of preboot need to be addressed before starting jails.<br />
<br />
=== Start jails ===<br />
[[#Starting_jails:_Quad.2FSafe_Files|More on starting jails]]<br />
Customer jails (the VPSs) do not start up automatically at boot time. When a FreeBSD machines boots up, it boots up, and does nothing else. To start jails, we put the commands to start each jail into a shell script(s) and run the script(s). Jail startup is something that needs to be actively monitored, which is why we don’t just run the script automatically. <br />
<br />
In order to start jails, we run the quad files: quad1 quad2 quad3 and quad4 (on new systems there is only quad1). If the machine was cleanly rebooted- which wouldn't be the case if this was a crash, you may run the safe files (safe1 safe2 safe3 safe4) in lieu of quads. <br />
<br />
Open up 4 logins to the server (use the windows in [[Screen#Screen_Organization|a9]])<br />
In each of the 4 windows you will:<br />
<br />
If there is a [[#startalljails|startalljails]] script (and only quad1), run that command in each of the 4 windows. It will parse through the quad1 file and start each jail. Follow the instructions [[#Problems_with_the_quad.2Fsafe_files|here]] for monitoring startup. Note that you can be a little more lenient with jails that take awhile to start- startalljails will work around the slow jails and start the rest. As long as there aren't 4 jails which are "hung" during startup, the rest will get started eventually.<br />
-or-<br />
If there is no startalljails script, there will be multiple quad files. In each of the 4 windows, start each of the quads. i.e. start quad1 in window1, quad2 in window2 and so on. DO NOT start any quad twice. It will crash the server. If you accidentally do this, just jailkill all the jails which are in the quad and run the quad again. Follow the instructions here for monitoring quad startup.<br />
<br />
Note the time the last jail boots- this is what you will enter in the crash log.<br />
<br />
Save the crash log.<br />
<br />
=== Check to make sure all jails have started ===<br />
There's a simple script which will make sure all jails have started, and enter the ipfw counter rules: [[#postboot|postboot]] <br />
Run postboot, which will do a jailps on each jail it finds (excluding commented out jails) in the quad file(s). We're looking for 2 things:<br />
# systems spawning out of control or too many procs<br />
# jails which haven't started<br />
On 7.x and newer systems it will print out the problems (which jails haven't started) at the conclusion of postboot. <br />
On older systems you will need to watch closely to see if/when there's a problem, namely:<br />
<br />
[hostname] doesnt exist on this server<br />
<br />
When you get this message, it means one of 2 things:<br />
1. the jail really didn't start:<br />
When a jail doesn't start it usually boils down to a problem in the quad file. Perhaps the path name is wrong (data1 vs data2) or the name of the vn/mdfile is wrong. Once this is corrected, you will need to run the commands from the quad file manually, or you may use <tt>startjail <hostname></tt><br />
<br />
2. the customer has changed their hostname (and not told us) so their jail ''is'' running, just under a different hostname:<br />
On systems with jls, this is easy to rectify. First, get the customer info: <tt>g <hostname></tt><br />
Then look for the customer in jls: <tt>jls | grep <col0XXXX></tt><br />
From there you will see their new hostname- you should update that hostname in the quad file: don't forget to edit it on the <tt>## begin ##</tt> and <tt>## end ##</tt> lines, and in mgmt. <br />
On older systems without jls, this will be harder, you will need to look further to see their hostname- perhaps its in their /etc/rc.conf<br />
<br />
<br />
Once all jails are started, do some spot checks- try to ssh or browse to some customers, just to make sure things are really ok.<br />
<br />
== Adding disk to a 7.x/8.x jail ==<br />
or<br />
== Moving customer to a different drive (md) ==<br />
<br />
NOTE: this doesn’t apply to mx2 which uses gvinum. Use same procedure as 6.x<br />
NOTE: if you unmount before mdconfig, re-mdconfig (attach) then unmount then mdconfig -u again <br />
<br />
<br />
(parts to change/customize are <tt><span style="color:red">red</span></tt>)<br />
<br />
<br />
If someone wants more disk space, there’s a paste for it, send it to them (explains about downtime, etc).<br />
<br />
1. Figure out the space avail from <tt>js</tt>. Ideally, you want to put the customers new space on a different partition (and create the new md on the new partition). <br />
<br />
<br />
2. make a mental note of how much space they're currently using<br />
<br />
<br />
3. <tt>jailkill <hostname></tt> <br />
<br />
<br />
4. Umount it (including their devfs) but leave the md config’d (so if you use stopjail, you will have to re-mdconfig it)<br />
<br />
<br />
5. <tt>g <customerID></tt> to get the info (IP/cust#) needed to feed the new mdfile and mount name, and to see the current md device. <br />
<br />
<br />
6a. When there's enough room to place new system on an alternate, or the same drive:<br />
USE CAUTION not to overwrite (touch, mdconfig) existing md!!<br><br />
<tt>touch /mnt/data<span style="color:red">3/69.55.234.66-col01334</span><br><br />
mdconfig -a -t vnode -s 10g -f /mnt/data3/69.55.234.66-col01334 -u 97<br><br />
newfs /dev/md97</tt><br />
<br />
<br />
Optional- if new space is on a different drive, move the mount point directory AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
<tt>mount /dev/md<span style="color:red">97</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<br />
confirm you are mounted to /dev/md<span style="color:red">97</span> and space is correct:<br><br />
<tt>df .</tt> <br />
<br />
<br />
do the dump and pipe directly to restore:<br><br />
<tt>dump -0a -f - /dev/md<span style="color:red">1</span> | restore -r -f - <br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use <tt>df</tt> to confirm the restored data size matches the original usage figure<br />
<br />
<br />
md-unconfig old system:<br><br />
<tt>mdconfig -d -u <span style="color:red">1</span></tt><br />
<br />
<br />
archive old mdfile. <br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.237.26-col00241</span> /mnt/data<span style="color:red">1/old-col00241-mdfile-noarchive-20091211</span></tt><br />
<br />
<br />
edit quad (vq1) to point to new (/mnt/data<span style="color:red">3</span>) location AND new md number (md<span style="color:red">97</span>)<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
6b. When there's not enough room on an alternate partition, or on the same drive...but there is enough room if you were to remove the existing customer's space:<br />
<br />
<br />
mount backup nfs mounts:<br><br />
<tt>mbm</tt><br> <br />
(run <tt>df</tt> to confirm backup mounts are mounted)<br />
<br />
<br />
dump the customer to backup2 or backup1<br><br />
<tt>dump -0a -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> /dev/md<span style="color:red">1</span></tt><br><br />
(when complete WITHOUT errors, <tt>du</tt> the dump file to confirm it matches size, roughly, with usage)<br />
<br />
<br />
unconfigure and remove old mdfile<br><br />
<tt>mdconfig -d -u <span style="color:red">1</span><br><br />
rm /mnt/data<span style="color:red">1/69.55.237.26-col00241</span></tt><br><br />
(there should now be enough space to recreate your bigger system. If not, run sync a couple times)<br />
<br />
<br />
create the new system (ok to reuse old mdfile and md#):<br><br />
<tt>touch /mnt/data<span style="color:red">1/69.55.234.66-col01334</span><br><br />
mdconfig -a -t vnode -s <span style="color:red">10</span>g -f /mnt/data<span style="color:red">1/69.55.234.66-col01334</span> -u <span style="color:red">1</span><br><br />
newfs /dev/md<span style="color:red">1</span><br><br />
mount /dev/md<span style="color:red">1</span> /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> <br><br />
cd /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<br />
confirm you are mounted to /dev/md<span style="color:red">1</span> and space is correct:<br><br />
<tt>df .</tt> <br />
<br />
<br />
do the restore from the dumpfile on the backup server:<br><br />
<tt>restore -r -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> .<br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
umount nfs:<br><br />
<tt>mbu</tt><br />
<br />
<br />
If md# changed (or mount point), edit quad (<tt>vq1</tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new md number (md<span style="color:red">1</span>)<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
7. update disk (and dir if applicable) in mgmt screen<br />
<br />
<br />
8. update backup list AND move backups, if applicatble<br />
<br />
Ex: <tt>mvbackups <span style="color:red">69.55.237.26-col00241</span> jail<span style="color:red">9</span> data<span style="color:red">3</span></tt><br />
<br />
<br />
9. Optional: archive old mdfile<br />
<br />
<tt>mbm<br><br />
gzip -c old-col01588-mdfile-noarchive-20120329 > /deprecated/old-col01588-mdfile-noarchive-20120329.gz<br><br />
mbu<br><br />
rm old-col01588-mdfile-noarchive-20120329</tt><br />
<br />
== Adding disk to a 6.x jail (gvinum/gconcat) ==<br />
or<br />
== Moving customer to a different drive (gvinum/gconcat) ==<br />
<br />
(parts to change are <span style="color:red">highlighted</span>)<br />
<br />
<br />
If someone wants more disk space, there’s a paste for it, send it to them (explains about downtime, etc).<br />
<br />
<br />
1. Figure out the space avail from [[#js|js]]. Ideally, you want to put the customers new space on a different partition (and create the new md on the new partition). <br />
<br />
<br />
2. make a mental note of how much space they're currently using<br />
<br />
<br />
3. <tt>[[#stopjail|stopjail]] <hostname></tt> <br />
<br />
<br />
4. <tt>[[#g|g]] <customerID></tt> to get the info (IP/cust#) needed to feed the new mount name and existing volume/device. <br />
<br />
<br />
5a. When there's enough room to place new system on an alternate, or the same drive (using only UNUSED - including if it's in use by the system in question - gvinum volumes):<br />
<br />
<br />
Configure the new device:<br><br />
A. for a 2G system (single gvinum volume):<br><br />
<tt>bsdlabel -r -w /dev/gvinum/v<span style="color:red">123</span><br><br />
newfs /dev/gvinum/v<span style="color:red">123</span>a</tt><br><br />
<br />
-or- <br />
<br />
<br />
B. for a >2G system (create a gconcat volume):<br><br />
try to grab a contiguous block of gvinum volumes. gconcat volumes MAY NOT span drives (i.e. you cannot use a gvinum volume from data3 and a volume from data2 in the same gconcat volume). <br />
<br />
<tt>gconcat label <span style="color:red">v82-v84 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84</span><br><br />
bsdlabel -r -w /dev/concat/<span style="color:red">v82-v84</span><br><br />
newfs /dev/concat/<span style="color:red">v82-v84</span>a</tt><br />
<br />
<br />
Other valid gconcat examples:<br><br />
<tt>gconcat label v82-v84v109v112 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84 /dev/gvinum/v109 /dev/gvinum/v112<br><br />
gconcat label v82v83 /dev/gvinum/v82 /dev/gvinum/v83</tt> <br><br />
Note, long names will truncate: v144v145v148-v115 will truncate to v144v145v148-v1 (so you will refer to it as v144v145v148-v1 thereafter)<br />
<br />
<br />
Optional- if new volume is on a different drive, move the mount point directory (get the drive from js output) AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
<br />
confirm you are mounted to the device (<tt>/dev/gvinum/v<span style="color:red">123</span>a</tt> OR <tt>/dev/concat/<span style="color:red">v82-v84</span></tt>) and space is correct:<br><br />
A. <tt>mount /dev/gvinum/v<span style="color:red">123</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br><br />
-or-<br><br />
B. <tt>mount /dev/concat/<span style="color:red">v82-v84</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<tt>cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
df .</tt> <br />
<br />
<br />
do the dump and pipe directly to restore:<br><br />
<tt>dump -0a -f - /dev/gvinum/v<span style="color:red">1</span> | restore -r -f - <br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
edit quad (<tt>vq<span style="color:red">1</span></tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new volume (<tt>/dev/gvinum/v<span style="color:red">123</span>a</tt> or <tt>/dev/concat/<span style="color:red">v82-v84</span>a</tt>) , run <tt>buildsafe</tt><br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
5b. When there's not enough room on an alternate partition, or on the same drive...but there is enough room if you were to remove the existing customer's space (i.e. if you want/need to reuse the existing gvinum volumes and add on more):<br />
<br />
<br />
mount backup nfs mounts:<br><br />
<tt>mbm</tt> <br><br />
(run df to confirm backup mounts are mounted)<br />
<br />
<br />
dump the customer to backup2 or backup1<br><br />
<tt>dump -0a -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> /dev/<span style="color:red">gconcat/v106-v107</span></tt><br><br />
(when complete WITHOUT errors, du the dump file to confirm it matches size, roughly, with usage)<br />
<br />
<br />
unconfigure the old gconcat volume<br><br />
list member gvinum volumes:<br />
<br />
<tt>gconcat list <span style="color:red">v106v107</span></tt><br />
<br />
<br />
Output will resemble:<br><br />
<pre>Geom name: v106v107<br />
State: UP<br />
Status: Total=2, Online=2<br />
Type: AUTOMATIC<br />
ID: 3530663882<br />
Providers:<br />
1. Name: concat/v106v107<br />
Mediasize: 4294966272 (4.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e2<br />
Consumers:<br />
1. Name: gvinum/sd/v106.p0.s0<br />
Mediasize: 2147483648 (2.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e3<br />
Start: 0<br />
End: 2147483136<br />
2. Name: gvinum/sd/v107.p0.s0<br />
Mediasize: 2147483648 (2.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e3<br />
Start: 2147483136<br />
End: 4294966272</pre><br />
<br />
stop volume and clear members<br><br />
<tt>gconcat stop <span style="color:red">v106v107</span><br><br />
gconcat clear <span style="color:red">gvinum/sd/v106.p0.s0 gvinum/sd/v107.p0.s0</span></tt><br />
<br />
<br />
create new device- and its ok to reuse old/former members<br><br />
try to grab a contiguous block of gvinum volumes. gconcat volumes MAY NOT span drives (i.e. you cannot use a gvinum volume from data3 and a volume from data2 in the same gconcat volume). <br><br />
<tt>gconcat label <span style="color:red">v82-v84v106v107 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84 /dev/gvinum/v106 /dev/gvinum/v107</span> <br><br />
bsdlabel -r -w /dev/concat/<span style="color:red">v82-v84v106v107</span><br><br />
newfs /dev/concat/<span style="color:red">v82-v84v106v107</span>a</tt><br />
<br />
<br />
Optional- if new volume is on a different drive, move the mount point directory (get the drive from js output) AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
confirm you are mounted to the device (/dev/concat/<span style="color:red">v82-v84v106v107</span>) and space is correct:<br><br />
<tt>mount /dev/concat/<span style="color:red">v82-v84v106v107</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<tt>cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
df .</tt> <br />
<br />
do the restore from the dumpfile on the backup server:<br><br />
<tt>restore -r -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> .<br><br />
rm restoresymtable</tt><br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
edit quad (<tt>vq<span style="color:red">1</span></tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new volume (<tt>/dev/concat/<span style="color:red">v82-v84v106v107</span>a</tt>), run buildsafe<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
TODO: clean up/clear old gvin/gconcat vol<br />
<br />
<br />
6. update disk (and dir if applicable) in mgmt screen<br />
<br />
<br />
7. update backup list AND move backups, if applicatble<br />
<br />
<br />
Ex: <tt>mvbackups <span style="color:red">69.55.237.26-col00241</span> jail<span style="color:red">9</span> data<span style="color:red">3</span></tt><br />
<br />
<br />
<br />
DEPRECATED - steps to tack on a new gvin to existing gconcat- leads to corrupted fs<br />
bsdlabel -e /dev/concat/v82-v84<br />
<br />
To figure out new size of the c partition, multiply 4194304 by the # of 2G gvinum volumes and subtract the # of 2G volumes:<br />
10G: 4194304 * 5 – 5 = 20971515<br />
8G: 4194304 * 4 – 4 = 16777212<br />
6G: 4194304 * 3 – 3 = 12582909<br />
4G: 4194304 * 2 – 2 = 8388606<br />
<br />
To figure out the new size of the a partition, subtract 16 from the c partition:<br />
10G: 20971515 – 16 = 20971499<br />
8G: 16777212 – 16 = 16777196<br />
6G: 12582909 – 16 = 12582893<br />
4G: 8388606 – 16 = 8388590<br />
<br />
Orig:<br />
8 partitions:<br />
# size offset fstype [fsize bsize bps/cpg]<br />
a: 8388590 16 4.2BSD 2048 16384 28552<br />
c: 8388606 0 unused 0 0 # "raw" part, don't edit<br />
<br />
New:<br />
8 partitions:<br />
# size offset fstype [fsize bsize bps/cpg]<br />
a: 12582893 16 4.2BSD 2048 16384 28552<br />
c: 12582909 0 unused 0 0 # "raw" part, don't edit<br />
<br />
sync; sync<br />
<br />
growfs /dev/concat/v82-v84a<br />
<br />
fsck –fy /dev/concat/v82-v84a<br />
<br />
sync<br />
<br />
fsck –fy /dev/concat/v82-v84a<br />
<br />
(keep running fsck’s till NO errors)<br />
<br />
== Problems un-mounting - and with mount_null’s ==<br />
<br />
If you cannot unmount a filesystem, beacuse it says the filesystem is busy, it is because of three things:<br />
<br />
a) the jail is still running<br />
<br />
b) you are actually in that directory, even though the jail is stopped<br />
<br />
c) there are still dev, null_mount or linprocfs mount points mounted inside that directory.<br />
<br />
d) when trying to umount null_mounts that are really long and you get an error like “No such file or directory”, it’s an OS bug where the dir is truncated. No known fix<br />
<br />
e) there are still files open somewhere inside the dir. Use <tt>fstat | grep <cid></tt> to find the process that has files open<br />
<br />
f) Starting with 6.x, the jail mechanism does a poor job of keeping track of processes running in a jail and if it thinks there are still procs running, it will refuse to umount the disk. If this is happening you should see a low number in the #REF column when you run jls. In this case you ''can'' safely <tt>umount –f</tt> the mount. <br />
<br />
Please note -if you forcibly unmount a (4.x) filesystem that has null_mounts<br />
still mounted in it, the system '''will crash''' within 10-15 mins.<br />
<br />
== Misc jail Items ==<br />
<br />
We are overselling hard drive space on jail2, jail8, jail9, a couple jails on jail17, jail4, jail12 and jail18.<br />
Even though the vn file shows 4G size, it doesn’t actually occupy that amount of space on the disk. So be careful not to fill up drives where we’re overselling – use oversellcheck to confirm you’re not oversold by more than 10G.<br />
There are other truncated jails, they are generally noted in a the file on the root system: /root/truncated<br />
<br />
The act of moving a truncated vn to another system un-does the truncating- the truncated vn is filled with 0’s and it occupies physical disk space for which it’s configured. So, you should use dumpremote to preserve the truncation.<br />
<br />
= FreeBSD VPS Management Tools =<br />
<br />
These files are located in /usr/local/jail/rc.d and /usr/local/jail/bin<br />
<br />
== jailmake ==<br />
<br />
Applies to 7.x+ <br />
On older systems syntax differs, run jailmake once to see.<br />
<br />
Note: this procedure differs on mx2 which is 7.x but still uses gvinum<br />
<br />
# run js to figure out which md’s are in use, which disk has enough space, IP to put it on<br />
# use col00xxx for both hostnames if they don’t give you a hostname<br />
# copy over dir, ip and password to pending customer screen<br />
<br />
<tt>Usage: jailmake IP[,IP] CID disk[1|2|3] md# hostname shorthost ipfw# email [size in GB]</tt><br />
<br />
Ex: <br />
<br />
Jail2# jailmake 69.55.234.66 col01334 3 97 vps.bsd.it vps 1334 fb@bsd.it<br />
<br />
== jailps ==<br />
jailps [hostname]<br />
DEPRECATED FOR jps: displays processes belonging to/running inside a jail. The command<br />
takes one (optional) argument – the hostname of the jail you wish to query. If you don’t <br />
supply an argument, all processes on the machine are listed and grouped by jail. <br />
<br />
== jps ==<br />
jps [hostname]<br />
displays processes belonging to/running inside a jail. The command<br />
takes one (optional) argument – the hostname or ID of the jail you wish to query. <br />
<br />
== jailkill ==<br />
jailkill <hostname><br />
stops all process running in a jail.<br />
<br />
You can also run:<br />
jailkill <JID><br />
<br />
=== problems ===<br />
Occasionally you will hit an issue where jail will not kill off:<br />
<br />
<pre>jail9# jailkill www.domain.com<br />
www.domain.com .. killed: none<br />
jail9#</pre><br />
Because no processes are running under that hostname. You cannot use jailps.pl either:<br />
<br />
<pre>jail9# jailps www.domain.com<br />
www.domain.com doesn’t exist on this server<br />
jail9#</pre><br />
<br />
The reasons for this are usually:<br />
* the jail is no longer running<br />
<br />
* the jail's hostname has changed<br />
In this case, <br />
<br />
>=6.x: run a <tt>jls|grep <jail's IP></tt> to find the correct hostname, then update the quad file, then kill the jail.<br />
<br />
<6.x: the first step is to cat their /etc/rc.conf file to see if you can tell what they set the new hostname to. This very often works. For example:<br />
<br />
cat /mnt/data2/198.78.65.136-col00261-DIR/etc/rc.conf<br />
<br />
But maybe they set the hostname with the hostname command, and the original hostname is still in /etc/rc.conf.<br />
<br />
The welcome email clearly states that they should tell us if they change their hostname, so there is no problem in just emailing them and asking them what they set the new hostname to.<br />
<br />
Once you know the new hostname OR if a customer simply emails to inform you that they have set the hostname to something different, you need to edit the quad and safe files that their system is in to input the new hostname.<br />
<br />
However, if push comes to shove and you cannot find out the hostname from them or from their system, then you need to start doing some detective work.<br />
<br />
The easiest thing to do is run jailps looking for a hostname similar to their original hostname. Or you could get into the /bin/sh shell by running:<br />
<br />
/bin/sh<br />
<br />
and then looking at every hostname of every process:<br />
<br />
for f in `ls /proc` ; do cat /proc/$f/status ; done<br />
<br />
and scanning for a hostname that is either similar to their original hostname, or that you don't see in any of the quad safe files.<br />
<br />
This is very brute force though, and it is possible that catting every file in /proc is dangerous - I don't recommend it. A better thing would be to identify any processes that you know belong to this system – perhaps the reason you are trying to find this system is because they are running something bad - and just catting the status from only that PID.<br />
<br />
Somewhere there’s a jail where there may be 2 systems named www. Look at /etc/rc.conf and make sure they’re both really www. If they are, jailkill www, jailps www to make sure not running. Then immediately restart the other one, as the fqdn (as found from a rev nslookup)<br />
<br />
* on >=6.x the hostname may not yet be hashed:<br />
<pre>jail9 /# jls<br />
JID Hostname Path IP Address(es)<br />
1 bitnet.dgate.org /mnt/data1/69.55.232.50-col02094-DIR 69.55.232.50<br />
2 ns3.hctc.net /mnt/data1/69.55.234.52-col01925-DIR 69.55.234.52<br />
3 bsd1 /mnt/data1/69.55.232.44-col00155-DIR 69.55.232.44<br />
4 let2.bbag.org /mnt/data1/69.55.230.92-col00202-DIR 69.55.230.92<br />
5 post.org /mnt/data2/69.55.232.51-col02095-DIR 69.55.232.51 ...<br />
6 ns2 /mnt/data1/69.55.232.47-col01506-DIR 69.55.232.47 ...<br />
7 arlen.server.net /mnt/data1/69.55.232.52-col01171-DIR 69.55.232.52<br />
8 deskfood.com /mnt/data1/69.55.232.71-col00419-DIR 69.55.232.71<br />
9 mirage.confluentforms.com /mnt/data1/69.55.232.54-col02105-DIR 69.55.232.54 ...<br />
10 beachmember.com /mnt/data1/69.55.232.59-col02107-DIR 69.55.232.59<br />
11 www.agottem.com /mnt/data1/69.55.232.60-col02109-DIR 69.55.232.60<br />
12 sdhobbit.myglance.org /mnt/data1/69.55.236.82-col01708-DIR 69.55.236.82<br />
13 ns1.jnielsen.net /mnt/data1/69.55.234.48-col00204-DIR 69.55.234.48 ...<br />
14 ymt.rollingegg.net /mnt/data2/69.55.236.71-col01678-DIR 69.55.236.71<br />
15 verse.unixlore.net /mnt/data1/69.55.232.58-col02131-DIR 69.55.232.58<br />
16 smcc-mail.org /mnt/data2/69.55.232.68-col02144-DIR 69.55.232.68<br />
17 kasoutsuki.w4jdh.net /mnt/data2/69.55.232.46-col02147-DIR 69.55.232.46<br />
18 dili.thium.net /mnt/data2/69.55.232.80-col01901-DIR 69.55.232.80<br />
20 www.tekmarsis.com /mnt/data2/69.55.232.66-col02155-DIR 69.55.232.66<br />
21 vps.yoxel.net /mnt/data2/69.55.236.67-col01673-DIR 69.55.236.67<br />
22 smitty.twitalertz.com /mnt/data2/69.55.232.84-col02153-DIR 69.55.232.84<br />
23 deliver4.klatha.com /mnt/data2/69.55.232.67-col02160-DIR 69.55.232.67<br />
24 nideffer.com /mnt/data2/69.55.232.65-col00412-DIR 69.55.232.65<br />
25 usa.hanyuan.com /mnt/data2/69.55.232.57-col02163-DIR 69.55.232.57<br />
26 daifuku.ppbh.com /mnt/data2/69.55.236.91-col01720-DIR 69.55.236.91<br />
27 collins.greencape.net /mnt/data2/69.55.232.83-col01294-DIR 69.55.232.83<br />
28 ragebox.com /mnt/data2/69.55.230.104-col01278-DIR 69.55.230.104<br />
29 outside.mt.net /mnt/data2/69.55.232.72-col02166-DIR 69.55.232.72<br />
30 vps.payneful.ca /mnt/data2/69.55.234.98-col01999-DIR 69.55.234.98<br />
31 higgins /mnt/data2/69.55.232.87-col02165-DIR 69.55.232.87 ...<br />
32 ozymandius /mnt/data2/69.55.228.96-col01233-DIR 69.55.228.96<br />
33 trusted.realtors.org /mnt/data2/69.55.238.72-col02170-DIR 69.55.238.72<br />
34 jc1.flanderous.com /mnt/data2/69.55.239.22-col01504-DIR 69.55.239.22<br />
36 guppylog.com /mnt/data2/69.55.238.73-col00036-DIR 69.55.238.73<br />
40 haliohost.com /mnt/data2/69.55.234.41-col01916-DIR 69.55.234.41 ...<br />
41 satyr.jorge.cc /mnt/data1/69.55.232.70-col01963-DIR 69.55.232.70<br />
jail9 /# jailkill satyr.jorge.cc<br />
ERROR: jail_: jail "satyr,jorge,cc" not found<br />
</pre><br />
<br />
Note how it's saying <tt>satyr,jorge,cc</tt> is not found, and not <tt>satyr.jorge.cc</tt>. <br />
<br />
The jail subsystem tracks things using comma-delimited hostnames. That is created every few hours:<br />
<br />
jail9 /# crontab -l<br />
0 0,6,12,18 * * * /usr/local/jail/bin/sync_jail_names<br />
<br />
So if we run this manually:<br />
jail9 /# /usr/local/jail/bin/sync_jail_names<br />
<br />
Then kill the jail:<br />
jail9 /# jailkill satyr.jorge.cc<br />
successfully killed: satyr,jorge,cc</pre><br />
<br />
It worked.<br />
<br />
<br />
If you ever see this when trying to kill a jail:<br />
<br />
<pre># jailkill e-scribe.com<br />
killing JID: 6 hostname: e-scribe.com<br />
3 procs running<br />
3 procs running<br />
3 procs running<br />
3 procs running<br />
...</pre><br />
<br />
<tt>[[#jailkill|jailkill]]</tt> probably got lost trying to kill off the jail. Just ctrl-c the jailkill process, then run a jailps on the hostname, and <tt>kill -9</tt> any process which is still running. Keep running jailps and <tt>kill -9</tt> till all processes are gone.<br />
<br />
== jailpsall ==<br />
jailpsall<br />
will run a jailps on all jails configured in the quad files (this is different from<br />
jailps with no arguments as it won’t help you find a “hidden” system)<br />
<br />
== jailpsw ==<br />
jailpsw<br />
will run a jailps with an extra -w to provide wider output<br />
<br />
== jt (>=7.x) ==<br />
jt<br />
displays the top 20 processes on the server (the top 20 processes from top) and <br />
which jail owns them. This is very helpful for determining who is doing what when<br />
the server is very busy.<br />
<br />
== jtop (>=7.x) ==<br />
jtop<br />
a wrapper for top displaying processes on the server and which jail owns them. Constantly updates, like top. <br />
<br />
== jtop (<7.x) ==<br />
jtop<br />
displays the top 20 processes on the server (the top 20 processes from top) and <br />
which jail owns them. This is very helpful for determining who is doing what when<br />
the server is very busy.<br />
<br />
== stopjail ==<br />
stopjail <hostname> [1]<br />
this will jailkill, umount and vnconfig –u a jail. If passed an optional 2nd<br />
argument, it will not exit before umounting and un-vnconfig’ing in the event<br />
jailkill returns no processes killed. This is useful if you just want to umount<br />
and vnconfig –u a jail you’ve already killed. It is intelligent in that it won’t <br />
try to umount or vnconfig –u if it’s not necessary.<br />
<br />
== startjail ==<br />
startjail <hostname><br />
this will start vnconfig, mount (including linprocfs and null-mounts), and start a jail.<br />
Essentially, it reads the jail’s relevant block from the right quad file and executes it.<br />
It is intelligent in that it won’t try to mount or vnconfig if it’s not necessary.<br />
<br />
== jpid ==<br />
jpid <pid><br />
displays information about a process – including which jail owns it.<br />
It’s the equivalent of running cat /proc/<pid>/status<br />
<br />
== canceljail ==<br />
canceljail <hostname> [1]<br />
this will stop a jail (the equivalent of stopjail), check for backups (offer to remove them <br />
from the backup server and the backup.config), rename the vnfile, remove the dir, and <br />
edit quad/safe. If passed an optional 2nd argument, it will not exit upon failing to kill<br />
and processes owned by the jail. This is useful if you just want to cancel a jail which <br />
is already stopped.<br />
<br />
== jls ==<br />
jls [-v]<br />
Lists all jails running:<br />
<pre>JID #REF IP Address Hostname Path<br />
101 135 69.55.224.148 mail.pc9.org /mnt/data2/69.55.224.148-col01034-DIR</pre><br />
<br />
#REF is the number of references or procs(?) running<br />
<br />
Running with -v will give you all IPs assigned to each jail (7.2 up)<br />
<pre>JID #REF Hostname Path IP Address(es)<br />
101 139 mail.pc9.org /mnt/data2/69.55.224.148-col01034-DIR 69.55.224.14869.55.234.85</pre><br />
<br />
== startalljails ==<br />
startalljails<br />
7.2+ only. This will parse through quad1 and start all jails. It utilizes lockfiles so it won’t try to start a jail more than once- therefore multiple instances can be running in parallel without fear of starting a jail twice. If a jail startup gets stuck, you can ^C without fear of killing the script. IMPORTANT- before running startalljails you should make sure you ran preboot once as it will clear out all the lockfiles and enable startalljails to work properly.<br />
<br />
== aaccheck.sh ==<br />
aaccheck.sh<br />
displayes the output of container list and task list from aaccli<br />
<br />
== backup ==<br />
backup<br />
backup script called nightly to update jail scripts and do customer backups<br />
<br />
== buildsafe ==<br />
buildsafe<br />
creates safe files based on quads (automatically removing the fsck’s). This will destructively overwrite safe files<br />
<br />
== checkload.pl ==<br />
checkload.pl<br />
this was intended to be setup as a cronjob to watch processes on a jail when the load <br />
rises above a certain level. Not currently in use.<br />
<br />
== checkprio.pl ==<br />
checkprio.pl<br />
will look for any process (other than the current shell’s csh, sh, sshd procs) with a non-normal priority and normalize it<br />
<br />
== diskusagemon == <br />
diskusagemon <mount point> <1k blocks><br />
watches a mount point’s disk use, when it reaches the level specified in the 2nd argument,<br />
it exits. This is useful when doing a restore and you want to be paged as it’s nearing completion.<br />
Best used as: <tt>diskusagemon /asd/asd 1234; pagexxx</tt><br />
<br />
== dumprestore ==<br />
dumprestore <dumpfile><br />
this is a perl expect script which automatically enters ‘1’ and ‘y’. It seems to cause restore to fail<br />
to set owner permissions on large restores.<br />
<br />
== g ==<br />
g <search><br />
greps the quad/safe files for the given search parameter<br />
<br />
== gather.pl ==<br />
gather.pl<br />
gathers up data about jails configured and writes to a file. Used for audits against the db<br />
<br />
== gb ==<br />
gb <search><br />
greps backup.config for the given search parameter<br />
<br />
== gbg ==<br />
gbg <search><br />
greps backup.config for the given search parameter and presents just the directories (for clean pasting)<br />
<br />
== ipfwbackup ==<br />
ipfwbackup<br />
writes ipfw traffic count data to a logfile<br />
<br />
== ipfwreset ==<br />
ipfwreset<br />
writes ipfw traffic count data to a logfile and resets counters to 0<br />
<br />
== js ==<br />
js<br />
output varies by OS version, but generally provides information about the base jail:<br />
- which vn’s are in use<br />
- disk usage<br />
- info about the contents of quads<br />
- the # of inodes represented by the jails contained in the group (133.2 in the example below), and how many jails per data mount, as well as subtotals<br />
- ips bound to the base machine but not in use by a jail<br />
- free gvinum volumes, or unused vn’s or used md’s<br />
<br />
<pre>/usr/local/jail/rc.d/quad1:<br />
/mnt/data1 133.2 (1)<br />
/mnt/data2 1040.5 (7)<br />
total 1173.7 (8)<br />
/usr/local/jail/rc.d/quad2:<br />
/mnt/data1 983.4 (6)<br />
total 983.4 (6)<br />
/usr/local/jail/rc.d/quad3:<br />
/mnt/data1 693.4 (4)<br />
/mnt/data2 371.6 (3)<br />
total 1065 (7)<br />
/usr/local/jail/rc.d/quad4:<br />
/mnt/data1 466.6 (3)<br />
/mnt/data2 882.2 (5)<br />
total 1348.8 (8)<br />
/mnt/data1: 2276.6 (14)<br />
/mnt/data2: 2294.3 (15)<br />
<br />
Available IPs:<br />
69.55.230.11 69.55.230.13 69.55.228.200<br />
<br />
Available volumes:<br />
v78 /mnt/data2 2G<br />
v79 /mnt/data2 2G<br />
v80 /mnt/data2 2G</pre> <br />
<br />
== load.pl ==<br />
load.pl<br />
feeds info to load mrtg - executed by inetd.<br />
<br />
== makevirginjail ==<br />
makevirginjail<br />
Only on some systems, makes an empty jail (doesn't do restore step)<br />
<br />
== mb == <br />
mb <mount|umount><br />
(nfs) mounts and umounts dirs to backup2. Shortcuts are mbm and mbu to mount and unmount. <br />
<br />
== notify.sh ==<br />
notify.sh<br />
emails reboot@johncompanies.com – intended to be called at boot time to alert us to a machine which panics and reboots and isn’t caught by bb or castle.<br />
<br />
== orphanedbackupwatch ==<br />
orphanedbackupwatch<br />
looks for directories on backup2 which aren’t configured in backup.config and offers to delete them<br />
<br />
== postboot ==<br />
postboot<br />
to be run after a machine reboot and quad/safe’s are done executing. It will:<br />
* do chmod 666 on each jail’s /dev/null<br />
* add ipfw counts<br />
* run jailpsall (so you can see if a configured jail isn’t running)<br />
<br />
== preboot ==<br />
preboot<br />
to be run before running quad/safe – checks for misconfigurations: <br />
* a jail configured in a quad but not a safe<br />
* a jail is listed more than once in a quad<br />
* the ip assigned to a jail isn’t configured on the machine<br />
* alias numbering skips in the rc.conf (resulting in the above)<br />
* orphaned vnfile's that aren't mentioned in a quad/safe<br />
* ip mismatches between dir/vnfile name and the jail’s ip<br />
* dir/vnfiles's in quad/safe that don’t exist <br />
<br />
== quadanalyze.pl ==<br />
quadanalyze.pl<br />
called by js, produces the info (seen above with js explanation) about the contents of quad (inode count, # of jails, etc.)<br />
<br />
== rsync.backup ==<br />
rsync.backup<br />
does customer backups (relies on backup.config)<br />
<br />
== taskdone ==<br />
taskdone<br />
when called will email support@johncompanies.com with the hostname of the machine from which it was executed as the subject<br />
<br />
== topten ==<br />
topten<br />
summarizes the top 10 traffic users (called by ipfwreset)<br />
<br />
== trafficgather.pl ==<br />
trafficgather.pl [yy-mm]<br />
sends a traffic usage summary by jail to support@johncomapnies.com and payments@johncompanies.com. Optional arguments are year and month (must be in the past). If not passed, assumes last month. Relies on traffic logs created by ipfwreset and ipfwbackup<br />
<br />
== trafficwatch.pl ==<br />
trafficwatch.pl<br />
checks traffic usage and emails support@johncomapnies.com when a jail reaches the warning level (35G) and the limit (40G). We really aren’t using this anymore now that we have netflow.<br />
<br />
== trafstats ==<br />
trafstats<br />
writes ipfw traffic usage info by jail to a file called jc_traffic_dump in each jail’s / dir<br />
<br />
== truncate_jailmake ==<br />
truncate_jailmake<br />
a version of jailmake which creates truncated vnfiles.<br />
<br />
== vb ==<br />
vb<br />
the equivalent of: <tt>vi /usr/local/jail/bin/backup.config</tt><br />
<br />
== vs (freebsd) ==<br />
vs<n><br />
the equivalent of: <tt>vi /usr/local/jail/rc.d/safe<n></tt><br />
<br />
vq<n><br />
the equivalent of: vi /usr/local/jail/rc.d/quad<n><br />
<br />
== dumpremote ==<br />
dumpremote <user@machine> </remote/location/file-dump> <vnX><br />
ex: dumpremote user@10.1.4.117 /mnt/data3/remote.echoditto.com-dump 7<br />
this will dump a vn filesystem to a remote machine and location<br />
<br />
== oversellcheck ==<br />
oversellcheck<br />
displays how much a disk is oversold or undersold taking into account truncated vn files. Only for use on 4.x systems<br />
<br />
== mvbackups (freebsd) ==<br />
mvbackups <dir> (1.1.1.1-col00001-DIR) <target_machine> (jail1) <target_dir> (data1)<br />
moves backups from one location to another on the backup server, and provides you with option to remove entries from current backup.config, and simple paste command to add the config to backup.config on the target server<br />
<br />
== jailnice ==<br />
jailnice <hostname><br />
applies <tt>renice 19 [PID]</tt> and <tt>rtprio 31 –[PID]</tt> to each process in the given jail<br />
<br />
== dumpremoterestore ==<br />
dumpremoterestore <device> <ip of target machine> <dir on target machine><br />
ex: dumpremoterestore /dev/vn51 10.1.4.118 /mnt/data2/69.55.239.45-col00688-DIR<br />
dumps a device and restores it to a directory on a remote machine. Requires that you enable root ssh on the <br />
remote machine.<br />
<br />
== psj ==<br />
psj<br />
shows just the procs running on the base system – a ps auxw but without jail’d procs present<br />
<br />
== perc5iraidchk ==<br />
perc5iraidchk<br />
checks for degraded arrays on Dell 2950 systems with Perc5/6 controllers<br />
<br />
== perc4eraidchk ==<br />
perc4eraidchk<br />
checks for degraded arrays on Dell 2850 systems with Perc4e/Di controllers<br />
<br />
= Virtuozzo VPS =<br />
<br />
Note: We use VEID (Virtual Environment ID) and CTID (Container ID) interchangably. Similarly, VE and CT. They mean the same thing.<br />
VZPP = VirtuoZzo Power Panel (the control panel for each CT)<br />
<br />
All linux systems exist in /vz, /vz1 or /vz2 - since each linux machine holds roughly 60-90 customers, there will be roughly 30-45 in each partition.<br />
<br />
The actual filesystem of the system in question is in:<br />
<br />
/vz(1-2)/private/(VEID)<br />
<br />
Where VEID is the identifier for that system - an all-numeric string larger than 100.<br />
<br />
The actual mounted and running systems are in the corresponding:<br />
<br />
/vz(1-2)/root/(VEID)<br />
<br />
But we rarely interact with any system from this mount point.<br />
<br />
You should never need to touch the root portion of their system – however you can traverse their filesystem by going to <tt>/vz(1-2)/private/(VEID)/root</tt> (<tt>/vz(1-2)/private/(VEID)/fs/root</tt> on 4.x systems) the root of their filesystem is in that directory, and their entire system is underneath that.<br />
<br />
Every VE has a startup script in <tt>/etc/sysconfig/vz-scripts</tt> (which is symlinked as <tt>/vzconf</tt> on all systems) - the VE startup script is simply named <tt>(VEID).conf</tt> - it contains all the system parameters for that VE:<br />
<br />
<pre># Configuration file generated by vzsplit for 60 VE<br />
# on HN with total amount of physical mem 2011 Mb<br />
<br />
VERSION="2"<br />
CLASSID="2"<br />
<br />
ONBOOT="yes"<br />
<br />
KMEMSIZE="8100000:8200000"<br />
LOCKEDPAGES="322:322"<br />
PRIVVMPAGES="610000:615000"<br />
SHMPAGES="33000:34500"<br />
NUMPROC="410:415"<br />
PHYSPAGES="0:2147483647"<br />
VMGUARPAGES="13019:2147483647"<br />
OOMGUARPAGES="13019:2147483647"<br />
NUMTCPSOCK="1210:1215"<br />
NUMFLOCK="107:117"<br />
NUMPTY="19:19"<br />
NUMSIGINFO="274:274"<br />
TCPSNDBUF="1800000:1900000"<br />
TCPRCVBUF="1800000:1900000"<br />
OTHERSOCKBUF="900000:950000"<br />
DGRAMRCVBUF="200000:200000"<br />
NUMOTHERSOCK="650:660"<br />
DCACHE="786432:818029"<br />
NUMFILE="7500:7600"<br />
AVNUMPROC="51:51"<br />
IPTENTRIES="155:155"<br />
DISKSPACE="4194304:4613734"<br />
DISKINODES="400000:420000"<br />
CPUUNITS="1412"<br />
QUOTAUGIDLIMIT="2000"<br />
VE_ROOT="/vz1/root/636"<br />
VE_PRIVATE="/vz1/private/636"<br />
NAMESERVER="69.55.225.225 69.55.230.3"<br />
OSTEMPLATE="vzredhat-7.3/20030305"<br />
VE_TYPE="regular"<br />
IP_ADDRESS="69.55.225.229"<br />
HOSTNAME="textengine.net"</pre><br />
<br />
<br />
As you can see, the hostname is set here, the disk space is set here, the number of inodes, the number of files that can be open, the number of tcp sockets, etc. - all are set here.<br />
<br />
In fact, everything that can be set on this customer system is set in this conf file.<br />
<br />
<br />
All interaction with the customer system is done with the VEID. You start the system by running:<br />
<br />
vzctl start 999<br />
<br />
You stop it by running:<br />
<br />
vzctl stop 999<br />
<br />
You execute commands in it by running:<br />
<br />
vzctl exec 999 df -k<br />
<br />
You enter into it, via a root-shell backdoor with:<br />
<br />
vzctl enter 999<br />
<br />
and you set parameters for the system, while it is still running, with:<br />
<br />
vzctl set 999 --diskspace 6100000:6200000 --save<br />
<br />
<tt>vzctl</tt> is the most commonly used command - we have aliased <tt>v</tt> to <tt>vzctl</tt> since we use it so often. We’ll continue to use <tt>vzctl</tt> in our examples, but feel free to use just <tt>v</tt>.<br />
<br />
Let's say the user wants more diskspace. You can cat their conf file and see:<br />
<br />
DISKSPACE="4194304:4613734"<br />
<br />
So right now they have 4gigs of space. You can then change it to 6 with:<br />
<br />
vzctl set 999 --diskspace 6100000:6200000 --save<br />
<br />
IMPORTANT: all issuances of the vzctl set command need to end with <tt>–save</tt> - if they don't, the setting will be set, but it will not be saved to the conf file, and they will not have those settings next time they boot.<br />
<br />
All of the tunables in the conf file can be set with the vzctl set command. Note that in the conf file, and on the vzctl set command line, we always issue two numbers seperated by a colon - that is because we are setting the hard and soft limits. Always set the hard limit slightly above the soft limit, as you see it is in the conf file for all those settings.<br />
<br />
There are also things you can set with `<tt>vzctl set</tt>` that are not in the conf file as settings, per se. For instance, you can add IPs:<br />
<br />
vzctl set 999 --ipadd 10.10.10.10 --save<br />
<br />
or multiple IPs:<br />
<br />
vzctl set 999 --ipadd 10.10.10.10 --ipadd 10.10.20.30 --save<br />
<br />
or change the hostname:<br />
<br />
vzctl set 999 --hostname www.example.com --save<br />
<br />
You can even set the nameservers:<br />
<br />
vzctl set 999 --nameserver 198.78.66.4 --nameserver 198.78.70.180 --save<br />
<br />
Although you probably will never do that.<br />
<br />
You can disable a VPS from being started (by VZPP or reboot) (4.x):<br />
<br />
vzctl set 999 --disabled yes --save <br />
<br />
You can disable a VPS from being started (by VZPP or reboot) (<=3.x):<br />
<br />
vzctl set 999 --onboot=no --save <br />
<br />
You can disable a VPS from using his control panel:<br />
<br />
vzctl set 999 --offline_management=no --save <br />
<br />
You can suspend a VPS, so it can be resumed in the same state it was in when it was stopped (4.x):<br />
<br />
vzctl suspend 999<br />
<br />
and to resume it:<br />
<br />
vzctl resume 999<br />
<br />
to see who owns process:<br />
vzpid <PID><br />
<br />
to mount up an unmounted ve:<br />
vzctl mount 827<br />
<br />
<br />
To see network stats for CT's:<br />
<pre># vznetstat<br />
VEID Net.Class Output(bytes) Input(bytes)<br />
-----------------------------------------------<br />
24218 1 484M 39M<br />
24245 1 463M 143M<br />
2451 1 2224M 265M<br />
2454 1 2616M 385M<br />
4149 1 125M 68M<br />
418 1 1560M 34M<br />
472 1 1219M 315M<br />
726 1 628M 317M<br />
763 1 223M 82M<br />
771 1 4234M 437M<br />
-----------------------------------------------<br />
Total: 13780M 2090M</pre><br />
<br />
<br />
One thing that sometimes comes up on older systems that we created with smaller defaults is that the system would run out of inodes. The user will email and say they cannot create any more files or grow any files larger, but they will also say that they are not out of diskspace ... they are running:<br />
<br />
df -k<br />
<br />
and seeing how much space is free - and they are not out of space. They are most likely out of inodes - which they would see by running:<br />
<br />
df -i<br />
<br />
So, the first thing you should do is enter their system with:<br />
<br />
vzctl enter 999<br />
<br />
and run: <tt>df -i</tt><br />
<br />
to confirm your theory. Then exit their system. Then simply cat their conf file and see what their inodes are set to (probably 200000:200000, since that was the old default on the older systems) and run:<br />
<br />
vzctl set 999 --diskinodes 400000:400000 --save<br />
<br />
If they are not out of inodes, then a good possibility is that they have maxed out their numfile configuration variable, which controls how many files they can have in their system. The current default is 7500 (which nobody has ever hit), but the old default was as low as 2000, so you would run something like:<br />
<br />
vzctl set 999 --numfile 7500:7500 --save<br />
<br />
----<br />
<br />
You cannot start or stop a VE if your pwd is its private (/vz/private/999) or root (/vz/root/999) directories, or anywhere below them.<br />
<br />
<br />
== Recovering from a crash (linux) ==<br />
<br />
=== Diagnose whether you have a crash ===<br />
The most important thing is to get the machine and all ve’s back up as soon as possible. Note the time, you’ll need to create a crash log entry (Mgmt. -> Reference -> CrashLog). The first thing to do is head over to the [[Screen#Screen_Organization|serial console screen]] and see if there’s any kernel error messages output. Try to copy any messages (or just a sample of repeating messages) you see into the notes section of the crash log – these will also likely need to be sent to virtuozzo for interpretation. If the messages are spewing too fast, hit ^O + H to start a screen log dump which you can ob1182.pts-38.bb serve after the machine is rebooted. Additionally, if the machine is responsive, you can get a trace to send to virtuozzo by hooking up a kvm and entering these 3 sequences:<br />
<pre>alt+print screen+m<br />
alt+print screen+p<br />
alt+print screen+t</pre><br />
<br />
If there are no messages, the machine may just be really busy- wait a bit (5-10min) to see if it comes back. If it's still pinging, odds are its very busy. If it doesn't come back, or the messages indicate a fatal error, you will need to proceed with a power cycle (ctrl+alt+del will not work).<br />
<br />
=== Power cycle the server ===<br />
If this machine is not a Dell 2950 with a [[DRAC/RMM#DRAC|DRAC card]] (i.e. if you can’t ssh into the DRAC card and issue racadm serveraction hardreset, then you will need someone at the data center to power the macine off, wait 30 sec, then turn it back on. Make sure to re-attach via console (<tt>tip virtxx</tt>) immediately after power down. <br />
<br />
=== (Re)attach to the console ===<br />
Stay on the console the entire time during boot. As the BIOS posts- look out for the RAID card output- does everything look healthy? The output may be scrambled, look for "DEGRADED" or "FAILED". Once the OS starts booting you will be disconnected (dropped back to the shell on the console server) a couple times during the boot up. The reason you want to quickly re-attach is two-fold: 1. If you don’t reattach quickly then you won’t get any console output, 2. you want to be attached before the server ''potentially'' starts (an extensive) fsck. If you attach after the fsck begins, you’ll have seen no indication it started an fsck and the server will appear frozen during startup- no output, no response. <br />
<br />
=== Start containers/VE's/VPSs ===<br />
When the machine begins to start VE’s, it’s safe to leave the console and login via ssh. All virts should be set to auto start all the VEs after a crash. Further, most (newer) virts are set to “fastboot” it’s VE’s (to find out, do:<br />
grep -i fast /etc/sysconfig/vz <br />
and look for <tt>VZFASTBOOT=yes</tt>). If this was set prior to the machine’s crash (setting it after the machine boots will not have any effect until the vz service is restarted) it will start each ve as fast as possible, in serial, then go thru each VE (serially), shutting it down running a vzquota (disk usage) check, then bringing it back up. The benefit is that all VE’s are brought up quickly (within 15min or so depending on the #), the downside is a customer watching closely will notice 2 outages – 1st the machine crash, 2nd their quota check (which will be a much shorter downtime- on the order of a few minutes). <br />
<br />
Where “fastboot” is not set to yes (i.e on quar1), vz will start them consecutively, checking the quotas one at a time, and the 60th VE may not start until an hour or two later - this is not acceptable.<br />
<br />
The good news is, if you run vzctl start for a VE that is already started, you will simply get an error: <tt>VE is already started</tt>. Further, if you attempt to vzctl start a VE that is in the process of being started, you will simply get an error: unable to lock VE. So, there is no danger in simply running scripts to start smaller sets of VEs. If the system is not autostarting, then there is no issue, and even if it does, when it conflicts, one process (yours or the autostart) will lose, and just move on to the next one.<br />
<br />
A script has been written to assist with ve starts: [[#startvirt.pl|startvirt.pl]] which will start 6 ve’s at once until there are no more left. If startvirt.pl is used on a system where “fastboot” was on, it will circumvent the fastboot for ve’s started by startvirt.pl – they will go through the complete quota check before starting- therefore this is not advisable when a system has crashed. When a system is booted cleanly, and there's no need for vzquota checks, then startvirt.pl is safe and advisable to run.<br />
<br />
=== Make sure all containers are running ===<br />
You can quickly get a feel for how many ve’s are started by running:<br />
<br />
<pre>[root@virt4 log]# vs<br />
VEID 16066 exist mounted running<br />
VEID 16067 exist mounted running<br />
VEID 4102 exist mounted running<br />
VEID 4112 exist mounted running<br />
VEID 4116 exist mounted running<br />
VEID 4122 exist mounted running<br />
VEID 4123 exist mounted running<br />
VEID 4124 exist mounted running<br />
VEID 4132 exist mounted running<br />
VEID 4148 exist mounted running<br />
VEID 4151 exist mounted running<br />
VEID 4155 exist mounted running<br />
VEID 42 exist mounted running<br />
VEID 432 exist mounted running<br />
VEID 434 exist mounted running<br />
VEID 442 exist mounted running<br />
VEID 450 exist mounted running<br />
VEID 452 exist mounted running<br />
VEID 453 exist mounted running<br />
VEID 454 exist mounted running<br />
VEID 462 exist mounted running<br />
VEID 463 exist mounted running<br />
VEID 464 exist mounted running<br />
VEID 465 exist mounted running<br />
VEID 477 exist mounted running<br />
VEID 484 exist mounted running<br />
VEID 486 exist mounted running<br />
VEID 490 exist mounted running</pre><br />
<br />
So to see how many ve’s have started:<br />
<pre>[root@virt11 root]# vs | grep running | wc -l<br />
39</pre><br />
<br />
And to see how many haven’t:<br />
<pre>[root@virt11 root]# vs | grep down | wc -l<br />
0</pre><br />
<br />
And how many we should have running:<br />
<pre>[root@virt11 root]# vs | wc -l<br />
39</pre><br />
<br />
Another tool you can use to see which ve’s have started, among other things is [[#vzstat|vzstat]]. It will give you CPU, memory, and other stats on each ve and the overall system. It’s a good thing to watch as ve’s are starting (note the VENum parameter, it will tell you how many have started):<br />
<br />
<pre>4:37pm, up 3 days, 5:31, 1 user, load average: 1.57, 1.68, 1.79<br />
VENum 40, procs 1705: running 2, sleeping 1694, unint 0, zombie 9, stopped 0<br />
CPU [ OK ]: VEs 57%, VE0 0%, user 8%, sys 7%, idle 85%, lat(ms) 412/2<br />
Mem [ OK ]: total 6057MB, free 9MB/54MB (low/high), lat(ms) 0/0<br />
Swap [ OK ]: tot 6142MB, free 4953MB, in 0.000MB/s, out 0.000MB/s<br />
Net [ OK ]: tot: in 0.043MB/s 402pkt/s, out 0.382MB/s 4116pkt/s<br />
Disks [ OK ]: in 0.002MB/s, out 0.000MB/s<br />
<br />
VEID ST %VM %KM PROC CPU SOCK FCNT MLAT IP<br />
1 OK 1.0/17 0.0/0.4 0/32/256 0.0/0.5 39/1256 0 9 69.55.227.152<br />
21 OK 1.3/39 0.1/0.2 0/46/410 0.2/2.8 23/1860 0 6 69.55.239.60<br />
133 OK 3.1/39 0.1/0.3 1/34/410 6.3/2.8 98/1860 0 0 69.55.227.147<br />
263 OK 2.3/39 0.1/0.2 0/56/410 0.3/2.8 34/1860 0 1 69.55.237.74<br />
456 OK 17/39 0.1/0.2 0/100/410 0.1/2.8 48/1860 0 11 69.55.236.65<br />
476 OK 0.6/39 0.0/0.2 0/33/410 0.1/2.8 96/1860 0 10 69.55.227.151<br />
524 OK 1.8/39 0.1/0.2 0/33/410 0.0/2.8 28/1860 0 0 69.55.227.153<br />
594 OK 3.1/39 0.1/0.2 0/45/410 0.0/2.8 87/1860 0 1 69.55.239.40<br />
670 OK 7.7/39 0.2/0.3 0/98/410 0.0/2.8 64/1860 0 216 69.55.225.136<br />
691 OK 2.0/39 0.1/0.2 0/31/410 0.0/0.7 25/1860 0 1 69.55.234.96<br />
744 OK 0.1/17 0.0/0.5 0/10/410 0.0/0.7 7/1860 0 6 69.55.224.253<br />
755 OK 1.1/39 0.0/0.2 0/27/410 0.0/2.8 33/1860 0 0 192.168.1.4<br />
835 OK 1.1/39 0.0/0.2 0/19/410 0.0/2.8 5/1860 0 0 69.55.227.134<br />
856 OK 0.3/39 0.0/0.2 0/13/410 0.0/2.8 16/1860 0 0 69.55.227.137<br />
936 OK 3.2/52 0.2/0.4 0/75/410 0.2/0.7 69/1910 0 8 69.55.224.181<br />
1020 OK 3.9/39 0.1/0.2 0/60/410 0.1/0.7 55/1860 0 8 69.55.227.52<br />
1027 OK 0.3/39 0.0/0.2 0/14/410 0.0/2.8 17/1860 0 0 69.55.227.83<br />
1029 OK 1.9/39 0.1/0.2 0/48/410 0.2/2.8 25/1860 0 5 69.55.227.85<br />
1032 OK 12/39 0.1/0.4 0/80/410 0.0/2.8 41/1860 0 8 69.55.227.90</pre><br />
<br />
When you are all done, you will want to make sure that all the VEs really did get started, run vs one more time.<br />
<br />
Note the time all ve’s are back up and enter that into and save the crash log entry.<br />
<br />
Occasionally, a ve will not start automatically. The most common reason for a ve not to come up normally is the ve was at it’s disk limit before the crash, and will not start since they’re over the limit. To overcome this, set the disk space to current usage level (the system will give this to you when it fails to start), start the ve, then re-set the disk space back to the prior level. Lastly, contact the customer to let them know they’re out of disk (or allocate more disk if they're entitled to more).<br />
<br />
== Hitting performance barriers and fixing them ==<br />
<br />
There are multiple modes virtuozzo offers to allocate resources to a ve. We utilize 2: SLM and UBC parameters<br />
On our 4.x systems, we use all SLM – it’s simpler to manage and understand. There are a few systems on virt19/18 that may also use SLM. Everything else uses UBC. <br />
You can tell a SLM ve by:<br />
<br />
SLMMODE="all"<br />
<br />
in their conf file. <br />
<br />
TODO: detail SLM modes and parameters.<br />
<br />
If someone is in SLM mode and they hit memory resource limits, they simply need to upgrade to more memory.<br />
<br />
The following applies to everyone else (UBC).<br />
<br />
Customers will often email and say that they are getting out of memory errors - a common one is "cannot fork" ... basically, anytime you see something odd like this, it means they are hitting one of their limits that is in place in their conf file.<br />
<br />
The conf file, however, simply shows their limits - how do we know what they are currently at ?<br />
<br />
The answer is a file called v - this file contains the current status (and peaks) of their performance settings, and also counts how many times they have hit the barrier. The output of the file looks like this:<br />
<br />
<pre>764: kmemsize 384113 898185 8100000 8200000 0<br />
lockedpages 0 0 322 322 0<br />
privvmpages 1292 7108 610000 615000 0<br />
shmpages 270 528 33000 34500 0<br />
dummy 0 0 0 0 0<br />
numproc 8 23 410 415 0<br />
physpages 48 5624 0 2147483647 0<br />
vmguarpages 0 0 13019 2147483647 0<br />
oomguarpages 641 6389 13019 2147483647 0<br />
numtcpsock 3 21 1210 1215 0<br />
numflock 1 3 107 117 0<br />
numpty 0 2 19 19 0<br />
numsiginfo 0 4 274 274 0<br />
tcpsndbuf 0 80928 1800000 1900000 0 <br />
tcprcvbuf 0 108976 1800000 1900000 0<br />
othersockbuf 2224 37568 900000 950000 0<br />
dgramrcvbuf 0 4272 200000 200000 0<br />
numothersock 3 9 650 660 0<br />
dcachesize 53922 100320 786432 818029 0<br />
numfile 161 382 7500 7600 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
numiptent 4 4 155 155 0</pre><br />
<br />
The first column is the name of the counter in question - the same names we saw in the systems conf file. The second column is the _current_ value of that counter, the third column is the max that that counter has ever risen to, the fourth column is the soft limit, and the fifth column is the hard limit (which is the same as the numbers in that systems conf file).<br />
<br />
The sixth number is the failcount - how many times the current usage has risen to hit the barrier. It will increase as soon as the current usage hits the soft limit.<br />
<br />
The problem with /proc/user_beancounters is that it actually contains that set of data for every running VE - so you can't just cat /proc/user_beancounters - it is too long and you get info for every other running system.<br />
<br />
You can vzctl enter the system and run:<br />
<br />
vzctl enter 9999<br />
cat /proc/user_beancounters<br />
<br />
inside their system, and you will just see the stats for their particular system, but entering their system every time you want to see it is combersome.<br />
<br />
So, I wrote a simple script called "vzs" which simply greps for the VEID, and spits out the next 20 or so lines (however many lines there are in the output, I forget) after it. For instance:<br />
<br />
<pre># vzs 765:<br />
765: kmemsize 2007936 2562780 8100000 8200000 0<br />
lockedpages 0 8 322 322 0<br />
privvmpages 26925 71126 610000 615000 0<br />
shmpages 16654 16750 33000 34500 0<br />
dummy 0 0 0 0 0<br />
numproc 41 57 410 415 0<br />
physpages 1794 49160 0 2147483647 0<br />
vmguarpages 0 0 13019 2147483647 0<br />
oomguarpages 4780 51270 13019 2147483647 0<br />
numtcpsock 23 37 1210 1215 0<br />
numflock 17 39 107 117 0<br />
numpty 1 3 19 19 0<br />
numsiginfo 0 6 274 274 0<br />
tcpsndbuf 22240 333600 1800000 1900000 0<br />
tcprcvbuf 0 222656 1800000 1900000 0<br />
othersockbuf 104528 414944 900000 950000 0<br />
dgramrcvbuf 0 4448 200000 200000 0<br />
numothersock 73 105 650 660 0<br />
dcachesize 247038 309111 786432 818029 0<br />
numfile 904 1231 7500 7600 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
numiptent 4 4 155 155 0</pre><br />
<br />
That showed us just the portion of /proc/user_beancounters for system 765.<br />
<br />
When you run the vzs command, always add a : after the VEID.<br />
<br />
So, if a customer complains about some out of memory errors, or no more files, or no more ptys, or just has an unspecific complain about processes dying, etc., the very first thing you need to do is check their beancounters with vzs. Usually you will spot an item that has a high failcount and needs to be upped.<br />
<br />
At that point you could simply up the counter with `vzctl set`. Generally pick a number 10-20% higher than the old one, and make the hard limit slightly larger than the the soft limit. However our systems now come in several levels and those levels have more/different memory allocations. If someone is complaining about something other than a memory limit (pty, numiptent, numflock), it’s generally safe to increase it, at least to the same level as what’s in the /vzconf/4unlimited file on the newest virt. If someone is hitting a memory limit, first make sure they are given what they deserve:<br />
<br />
(refer to mgmt -> payments -> packages)<br />
<br />
To set those levels, you use the [[#setmem|setmem]] command. <br />
<br />
The alternate (DEPRECATED) method would be to use one of 3 commands:<br />
256 <veid><br />
300 <veid><br />
384 <veid><br />
512 <veid><br />
<br />
If the levels were not right (you’d run vzs <veid> before and after to see the effect) tell the customer they’ve been adjusted and be done with it. If the levels were right, tell the customer they must upgrade to a higher package, tell them how to see level (control panel) and that they can reboot their system to escape this lockup contidion.<br />
<br />
Customers can also complain that their site is totally unreachable, or complain that it is down ... if the underlying machine is up, and all seems well, you may notice in the beancounters that network-specific counters are failing - such as numtcpsock, tcpsndbuf or tcprcvbuf. This will keep them from talking on the network and make it seem like their system is down. Again, just up the limits and things should be fine.<br />
<br />
On virts 1-4, you should first look at the default settings for that item on a later virt, such as virt 8 - we have increased the defaults a lot since the early machines. So, if you are going to up a counter on virt2, instead of upping it by 10-20%, instead up it to the new default that you see on virt8.<br />
<br />
== Moving a VE to another virt (migrate/migrateonline) ==<br />
<br />
This will take a while to complete - and it is best to do this at night when the load is light on both machines.<br />
<br />
There are different methods for this, depending on which version of virtuozzo is installed on the src. and dst. virt. <br />
To check which version is running: <br />
[root@virt12 private]# cat /etc/virtuozzo-release<br />
Virtuozzo release 2.6.0<br />
<br />
Ok, let's say that the VE is 1212, and vital stats are:<br />
<pre>[root@virt12 sbin]# vc 1212<br />
VE_ROOT="/vz1/root/1212"<br />
VE_PRIVATE="/vz1/private/1212"<br />
OSTEMPLATE="fedora-core-2/20040903"<br />
IP_ADDRESS="69.55.229.84"<br />
TEMPLATES="devel-fc2/20040903 php-fc2/20040813 mysql-fc2/20040812 postgresql-fc2/20040813 mod_perl-fc2/20040812 mod_ssl-fc2/20040811 jre-fc2/20040823 jdk-fc2/20040823 mailman-fc2/20040823 analog-fc2/20040824 proftpd-fc2/20040818 tomcat-fc2/20040823 usermin-fc2/20040909 webmin-fc2/20040909 uw-imap-fc2/20040830 phpBB-fc2/20040831 spamassassin-fc2/20040910 PostNuke-fc2/20040824 sl-webalizer-fc2/20040<br />
818"<br />
<br />
[root@virt12 sbin]# vzctl exec 1212 df -h<br />
Filesystem Size Used Avail Use% Mounted on<br />
/dev/vzfs 4.0G 405M 3.7G 10% /</pre><br />
<br />
From this you can see that he’s using (and will minimally need free on the dst server) ~400MB, and he’s running on a Fedora 2 template, version 20040903. He’s also got a bunch of other templates installed. It’s is '''vital''' that '''all''' these templates exist on the dst system. To confirm that, on the dst system run:<br />
<br />
For < 3.0:<br />
<br />
<pre>[root@virt14 private]# vzpkgls | grep fc2<br />
devel-fc2 20040903<br />
PostNuke-fc2 20040824<br />
analog-fc2 20040824<br />
awstats-fc2 20040824<br />
bbClone-fc2 20040824<br />
jdk-fc2 20040823<br />
jre-fc2 20040823<br />
mailman-fc2 20040823<br />
mod_frontpage-fc2 20040816<br />
mod_perl-fc2 20040812<br />
mod_ssl-fc2 20040811<br />
mysql-fc2 20040812<br />
openwebmail-fc2 20040817<br />
php-fc2 20040813<br />
phpBB-fc2 20040831<br />
postgresql-fc2 20040813<br />
proftpd-fc2 20040818<br />
sl-webalizer-fc2 20040818<br />
spamassassin-fc2 20040910<br />
tomcat-fc2 20040823<br />
usermin-fc2 20040909<br />
uw-imap-fc2 20040830<br />
webmin-fc2 20040909<br />
[root@virt14 private]# vzpkgls | grep fedora<br />
fedora-core-1 20040121 20040818<br />
fedora-core-devel-1 20040121 20040818<br />
fedora-core-2 20040903<br />
[root@virt14 private]#</pre><br />
<br />
For these older systems, you can simply match up the date on the template. <br />
<br />
For >= 3.0:<br />
<br />
<pre>[root@virt19 /vz2/private]# vzpkg list<br />
centos-5-x86 2008-01-07 22:05:57<br />
centos-5-x86 devel<br />
centos-5-x86 jre<br />
centos-5-x86 jsdk<br />
centos-5-x86 mod_perl<br />
centos-5-x86 mod_ssl<br />
centos-5-x86 mysql<br />
centos-5-x86 php<br />
centos-5-x86 plesk9<br />
centos-5-x86 plesk9-antivirus<br />
centos-5-x86 plesk9-api<br />
centos-5-x86 plesk9-atmail<br />
centos-5-x86 plesk9-backup<br />
centos-5-x86 plesk9-horde<br />
centos-5-x86 plesk9-mailman<br />
centos-5-x86 plesk9-mod-bw<br />
centos-5-x86 plesk9-postfix<br />
centos-5-x86 plesk9-ppwse<br />
centos-5-x86 plesk9-psa-firewall<br />
centos-5-x86 plesk9-psa-vpn<br />
centos-5-x86 plesk9-psa-fileserver<br />
centos-5-x86 plesk9-qmail<br />
centos-5-x86 plesk9-sb-publish<br />
centos-5-x86 plesk9-vault<br />
centos-5-x86 plesk9-vault-most-popular<br />
centos-5-x86 plesk9-watchdog</pre><br />
<br />
On these newer systems, it's difficult to tell whether the template on the dst matches exactly the src. Just cause a centos-5-x86 is listed on both servers doesn't mean all the same packages are there on the dst. To truly know, you must perform a sample rsync:<br />
<br />
rsync -avn /vz/template/centos/5/x86/ root@10.1.4.61:/vz/template/centos/5/x86/<br />
<br />
if you see a ton of output from the dry run command, then clearly there are some differences. You may opt to let the rsync complete (without running in dry run mode) the only downside is you've now used up more space on the dst and also the centos template will be a mess with old and new data- it will be difficult if not impossible to undo (if someday we wanted to reclaim the space).<br />
<br />
If you choose to merge templates, you should closely inspect the dry run output. You should also take care to exclude anything in the /config directory. For example:<br />
<br />
rsync -av -e ssh --stats --exclude=x86/config /vz/template/ubuntu/10.04/ root@10.1.4.62:/vz/template/ubuntu/10.04/<br />
<br />
Which will avoid this directory and contents:<br />
<pre>[root@virt11 /vz2/private]# ls /vz/template/ubuntu/10.04/x86/config*<br />
app os</pre><br />
<br />
This is important to avoid since the config may differ on the destination and we are really only interested in making sure the pacakges are there, not overwriting a newer config with an older one.<br />
<br />
If the dst system was missing a template, you have 2 choices: <br />
# put the missing template on the dst system. 2 choices here: <br />
## Install the template from rpm (found under backup2: /mnt/data4/vzrpms/distro/) or <br />
## rsync over the template (found under /vz/template) - see above<br />
# put the ve on a system which has all the proper templates<br />
<br />
=== pre-seeding a migration ===<br />
<br />
When migrating a customer (or when doing many) depending on how much data you have to transfer, it can take some time. Further, it can be difficult to gauge when a migration will complete or how long it will take. To help speed up the process and get a better idea about how long it will take you can pre-transfer a customer's data to the destination server. If done correctly, vzmigrate will see the pre-transferred data and pick up where you left off, having much less to transfer (just changed/new files). <br />
<br />
We believe vzmigrate uses rsync to do it's transfer. Therefore not only can you use rsync to do a pre-seed, you can also run rsync to see what is causing a repeatedly-failing vzmigrate to fail. <br />
<br />
There's no magic to a pre-seed, you just need to make sure it's named correctly.<br />
<br />
Given:<br />
<br />
source: /vz1/private/1234<br />
<br />
and you want to migrate to /vz2 on the target system, your rsync would look like:<br />
<br />
rsync -av /vz1/private/1234/ root@x.x.x.x:/vz2/private/1234.migrated/<br />
<br />
After running that successful rsync, the ensuing migrateonline (or migrate) will take much less time to complete- depending on the # of files to be analyzed and the # of changed files. In any case, it'll be much much faster than had you just started the migration from scratch.<br />
<br />
Further, as we discuss elsewhere in this topic, a failed migration can be moved from <tt>/vz/private/1234</tt> to <tt>/vz/private/1234.migrated</tt> on the destination if you want to restart a failed migration. This should '''only''' be done if the migration failed and the CT is not running on the destination HN.<br />
<br />
=== migrateonline intructions: src >=3.x -> dst>=3.x ===<br />
<br />
A script called [[#migrateonline|migrateonline]] was written to handle this kind of move. It is basically a wrapper for <tt>vzmigrate</tt> – vzmigrate is a util to seamlessly- as no no reboot of the ve necessary- move a ve from one host to another. This wrapper was initially written cause vz virtuozzo version 2.6.0 has a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables, causing problems when the ve was started up on the dst system. [[#migrate|migrate]] mitigates that. Since it makes multiple ssh connections to the dst virt, it’s a good idea to put the pub key for the src virt in the authorized_keys file on the dst virt. In addition, migrateonline emails ve owners when their migration starts and stops. For this to happen they need to put email addresses (on a single line, space delimited) in a file on their system: /migrate_notify. If the optional target dir is not specified, the ve will be moved to the same private/root location as it was on the src virt. Note: <tt>migrate</tt> is equivalent to <tt>migrateonline</tt>, but will <tt>migrate</tt> a ve AND restart it in the process.<br />
<br />
<pre>[root@virt12 sbin]# migrateonline<br />
usage: /usr/local/sbin/migrateonline <ip of node migrating to> <veid> [target dir: vz | vz1 | vz2]<br />
[root@virt12 sbin]# migrateonline 10.1.4.64 1212 vz<br />
starting to migrate 1212 at Sat Mar 26 22:40:38 PST 2005<br />
Turning off offline_management<br />
Saved parameters for VE 1212<br />
migrating with no start on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
clearing the arp cache<br />
now going to 10.1.4.64 and clear cache and starting<br />
starting it<br />
Delete port redirection<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
finished migrating 1212 at Sat Mar 26 22 22:52:01 PST 2005<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine.<br />
<br />
If they had backups, use the mvbackups command to move their backups to the new server:<br />
<br />
mvbackups 1212 virt14 vz<br />
<br />
Rename the ve<br />
<br />
[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/migrated-1212<br />
<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/old-1212-migrated-20120404-noarchive<br />
<br />
Update the customer’s systems in mgmt to reflect the new path and server.<br />
<br />
<br />
IF migrateonline does not work, you can try again using simply migrate- this will result in a brief reboot for the ve.<br />
Before you try again, make sure of a few things:<br />
<br />
Depending on where in the migration died, there may be partial data on the dst system in 1 of 2 places:<br />
(given the example above)<br />
<br />
/vz/private/1212<br />
<br />
or <br />
<br />
/vz/private/1212.migrated<br />
<br />
before you run migrate again, you'll want to rename so that all data is in <br />
1212.migrated:<br />
<br />
mv /vz/private/1212 /vz/private/1212.migrated<br />
<br />
this way, it will pick up where it left off and transfer only new files.<br />
<br />
Likewise, if you want to speed up a migration, you can pre-seed the dst as follows:<br />
<br />
[root@virt12 sbin]# rsync -avSH /vz/private/1212/ root@10.1.4.64:/vz/private/1212.migrated/<br />
<br />
then when you run migrate or migrateonline, it will only need to move the changed files- the migration will complete quickly<br />
<br />
=== migrate manually (if migrate/online fails) ===<br />
<br />
Lets say for whatever reason the migration fails. You can always move someone manually:<br />
<br />
1. stop ve: <br><br />
v stop 1234<br />
<br />
2. copy over data<br><br />
rsync -avSH /vz/private/1234/ root@1.1.1.1:/vzX/private/1234/<br />
<br />
NOTE: if you've previously seeded the data (run rsync while the VE was up/running), and this is a subsequent rsync, make sure the last rsync you do (while the VE is not running, has the --delete option in the rsync)<br />
<br />
3. copy over conf<br><br />
scp /vzconf/1234.conf root@1.1.1.1:/vzconf<br />
<br />
4. on dst, edit the conf to reflect the right vzX dir<br><br />
vi /vzconf/1234.conf<br />
<br />
5. on src remove the IPs<br><br />
ipdel 1234 2.2.2.2 3.3.3.3<br />
<br />
6. on dst add IPs <br><br />
ipadd 1234 2.2.2.2 3.3.3.3<br />
<br />
7. on dst, start ve: <br><br />
v start 1324<br />
<br />
8. cancel, then archive ve on src per above instrs.<br />
<br />
=== migrate src=2.6.0 -> dst>=2.6.0, or mass-migration with customer notify ===<br />
<br />
A script called <tt>migrate</tt> was written to handle this kind of move. It is basically a wrapper for vzmigrate – vzmigrate is a util to seamlessly move a ve from one host to another. This wrapper was initially written cause vz virtuozzo version 2.6.0 has a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables, causing problems when the ve was started up on the dst system. migrate mitigates that. Since it makes multiple ssh connections to the dst virt, it’s a good idea to put the pub key for the src virt in the authorized_keys file on the dst virt. In addition, migrate emails ve owners when their migration starts and stops. For this to happen they need to put email addresses (on a single line, space delimited) in a file on their system: /migrate_notify. If the optional target dir is not specified, the ve will be moved to the same private/root location as it was on the src virt. Note: migrateonline is equivalent to migrate, but will migrate a ve from one 2.6 '''kernel''' machine to another 2.6 kernel machine without restarting the ve.<br />
<br />
<pre>[root@virt12 sbin]# migrate<br />
usage: /usr/local/sbin/migrate <ip of node migrating to> <veid> [target dir: vz | vz1 | vz2]<br />
[root@virt12 sbin]# migrate 10.1.4.64 1212 vz<br />
starting to migrate 1212 at Sat Mar 26 22:40:38 PST 2005<br />
Turning off offline_management<br />
Saved parameters for VE 1212<br />
migrating with no start on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
clearing the arp cache<br />
now going to 10.1.4.64 and clear cache and starting<br />
starting it<br />
Delete port redirection<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
finished migrating 1212 at Sat Mar 26 22 22:52:01 PST 2005<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine (backup2 or mail).<br />
Cancel the ve (first we have to rename things which migrate changed so cancelve will find them):<br />
<br />
[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/1212.conf<br />
<br />
On 2.6.1 you’ll also have to move the private area:<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/1212<br />
<br />
<pre>[root@virt12 sbin]# cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, [[#cancelve|cancelve]] would offer to remove them. You want to say '''no''' to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
To move everyone off a system, you’d do:<br />
for f in `vl`; do migrate <ip> $f; done<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
=== vzmigrate: src=2.6.1 -> dst>=2.6.0 ===<br />
<br />
This version of vzmigrate works properly with regard to handling ips. It will not notify ve owners of moves as in the above example. Other than that it’s essentially the same.<br />
<br />
<pre>[root@virt12 sbin]# vzmigrate 10.1.4.64 -r no 1212:1212:/vz/private/1212:/vz/root/1212<br />
migrating on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine (backup2 or mail).<br />
Cancel the ve (first we have to rename things which vzmigrate changed so cancelve will find them):<br />
<br />
<pre>[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/1212.conf<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/1212<br />
<br />
[root@virt12 sbin]# cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, <tt>cancelve</tt> would offer to remove them. You want to say no to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
=== src=2.5.x ===<br />
<br />
First, go to the private dir:<br />
<br />
cd /vz1/private/<br />
<br />
Stop the VE - make sure it stops totally cleanly.<br />
<br />
vzctl stop 1212<br />
<br />
Then you’d use vemove - a script written to copy over the config, create tarballs of the ve’s data on the destination virt, and cancel the ve on the source system (in this example we’re going to put a ve that was in /vz1/private on the src virt, in /vz/private on the dst virt):<br />
<br />
<pre>[root@virt12 sbin]# vemove<br />
ERROR: Usage: vemove veid target_ip target_path_dir<br />
[root@virt12 sbin]# vemove 1212 10.1.4.64 /vz/private/1212<br />
tar cfpP - 1212 --ignore-failed-read | (ssh -2 -c arcfour 10.1.4.64 "split - -b 1024m /vz/private/1212.tar" )<br />
scp /vzconf/1212.conf 10.1.4.64:/vzconf<br />
cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, cancelve would offer to remove them. You want to say no to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
When you are done, go to /vz/private on the dst virt you will have files like this:<br />
<br />
<pre>1212.taraa<br />
1212.tarab<br />
1212.tarac</pre><br />
<br />
Each one 1024m (or less, for the last one) in size.<br />
<br />
on the dst server and run:<br />
<br />
cat 1212.tar?? | tar xpPBf -<br />
<br />
and after 20 mins or so it will be totally untarred. Now since the conf<br />
file is already there, you can go ahead and start the system.<br />
<br />
vzctl start 1212<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
NOTE: you MUST tar the system up using the virtuozzo version of tar that<br />
is on all the virt systems, and further you MUST untar the tarball with<br />
the virtuozzo tar, using these options: `<tt>tar xpPBf -</tt>`<br />
<br />
If you tar up an entire VE and move it to a non-virtuozzo machine, that is<br />
ok, and you can untar it there with normal tar commands, but do not untar<br />
it and then repack it with a normal tar and expect it to work - you need<br />
to use virtuozzo tar commands on virtuozzo tarballs to make it work.<br />
<br />
The backups are sort of an exception, since we are just (usually)<br />
restoring user data that was created after we gave them the system, and<br />
therefore has nothing to do with magic symlinks or vz-rpms, etc.<br />
<br />
== Moving a VE on the same virt ==<br />
<br />
Easy way:<br><br />
Scenario 1: ve 123 is to be renamed 1231 and moved from vz1 to vz<br />
<br />
vzmlocal 123:1231:/vz/private/1231:/vz/root/1231<br />
<br />
Scenario 2: ve 123 is to be moved vz1 to vz<br />
<br />
vzmlocal 123:123:/vz/private/123:/vz/root/123<br />
<br />
vzmlocal will reboot the ve at the end of the move<br />
<br />
Manual/old way:<br />
<br />
1) <tt>vzctl stop 123</tt><br><br />
2) <tt>mv /vz1/private/123 /vz/private/.</tt><br><br />
(or cp -a if you want to copy)<br />
3) in <tt>/etc/sysconfig/vz-scripts/123.conf</tt> change value<br><br />
of '<tt>VE_PRIVATE</tt>' variable to point to a new private area location<br />
4) <tt>vzctl start 123</tt><br><br />
5) update backups if needed: <tt>mvbackups 123 virtX virt1 vz</tt><br><br />
6) update management scerens<br><br />
<br />
Notes: a) absolute path to private area is stored in quota file <tt>/var/vzquota/quota.123</tt> - so during first startup quota will be recalculated.<br><br />
b) if you're going to write some script to do a job, you MUST be sure that $VEID won't be expanded to '' in ve config file - ie. you need to escape '$'. Otherwise you might have:<br />
<br />
VE_PRIVATE="/vz/private/"<br />
<br />
in config, and 'vzctl destroy' for this VE ID '''will remove everything under /vz/private/ directory'''.<br />
<br />
== Adding a veth device to a VE ==<br />
<br />
Not totally sure what this is, but a customer asked for it and here's what we did (as instructed by vz support):<br />
<br />
<pre>v set 99 --netif_add eth99 --save<br />
ipdel 99 69.55.230.58<br />
v set 99 --ifname eth99 --ipadd 69.55.230.58 --save<br />
v set 99 --ifname eth99 --gateway 69.55.230.1 --save<br />
<br />
vznetcfg net new veth_net<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
net99 active eth0 veth77.77,veth99.99<br />
veth_net active<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
br0 bridge veth_net<br />
br99 bridge net99<br />
veth99.99 veth net99<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic net99 69.55.227.70/24<br />
<br />
vznetcfg br attach br0 eth0<br />
<br />
(will remove 99 from orig net and move to veth_net)<br />
vznetcfg net addif veth_net veth99.99<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
net99 active<br />
veth_net active eth0 veth77.77,veth99.99<br />
<br />
(delete the old crap)<br />
vznetcfg net del net99<br />
<br />
<br />
Then, to add another device in<br />
<br />
v set 77 --netif_add eth77 --save<br />
ipdel 77 69.55.230.78<br />
v set 77 --ifname eth77 --ipadd 69.55.230.78 --save<br />
v set 77 --ifname eth77 --gateway 69.55.230.1 --save<br />
v set 77 --save --ifname eth77 --network veth_net<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
veth77.77 veth<br />
br0 bridge veth_net<br />
veth99.99 veth veth_net<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic veth_net 69.55.227.70/24<br />
<br />
vznetcfg net addif veth_net veth77.77<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
veth77.77 veth veth_net<br />
br0 bridge veth_net<br />
veth99.99 veth veth_net<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic veth_net 69.55.227.70/24<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
veth_net active eth0 veth77.77,veth99.99<br />
<br />
another example<br />
<br />
v set 1182 --netif_add eth1182 --save<br />
ipdel 1182 69.55.236.217<br />
v set 1182 --ifname eth1182 --ipadd 69.55.236.217 --save<br />
v set 1182 --ifname eth1182 --gateway 69.55.236.1 --save<br />
vznetcfg net addif veth_net veth1182.1182<br />
v set 1182 --save --ifname eth1182 --network veth_net<br />
<br />
<br />
unused/not working commands:<br />
ifconfig veth99.0 0<br />
vznetcfg net list<br />
vznetcfg br new br99 net99<br />
vznetcfg br attach br99 eth0<br />
vznetcfg br show<br />
vznetcfg if list<br />
vznetcfg net addif net99 veth99.99<br />
vznetcfg net addif eth0 net99<br />
<br />
---------<br />
<br />
vznetcfg br new br1182 net1182<br />
<br />
vznetcfg br attach br99 eth0<br />
vznetcfg if list<br />
vznetcfg net addif net99 veth99.99<br />
vznetcfg net addif eth0 net99<br />
<br />
<br />
<br />
vznetcfg net addif eth0 net1182<br />
<br />
<br />
# vznetcfg net addif veth99.0 net99<br />
--- 8< ---<br />
<br />
vznetcfg net new net<br />
# vznetcfg net addif eth0 net99<br />
<br />
<br />
# vzctl set 99 --save --netif_add eth0 (at this stage veth99.0 interface have to appear<br />
on node)<br />
# vzctl set 99 --save --ifname eth0 --ipadd 69.55.230.58 (and probably few more arguments<br />
here - see 'man vzctl')<br />
# vznetcfg net addif veth99.0 net99</pre><br />
<br />
<br />
== Assigning/remove ip from a VE ==<br />
<br />
1. Add or remove ips:<br />
ipdel 1234 1.1.1.1 2.2.2.2<br />
ipadd 1234 1.1.1.1 2.2.2.2<br />
<br />
2. update Mgmt screens<br />
<br />
3. offer to update any DNS we do for them<br />
<br />
4. check to see if we had rules for old IP in firwall<br />
<br />
== Enabling tun device for a ve ==<br />
Note, there’s a command for this: [[#addtun|addtun]]<br />
<br />
For posterity:<br />
Make sure the tun.o module is already loaded before Virtuozzo is started: <br />
lsmod <br />
Allow the VPS to use the TUN/TAP device: <br />
vzctl set 101 --devices c:10:200:rw --save <br />
Create the corresponding device inside the VPS and set the proper permissions: <br />
vzctl exec 101 mkdir -p /dev/net <br />
vzctl exec 101 mknod /dev/net/tun c 10 200 <br />
vzctl exec 101 chmod 600 /dev/net/tun<br />
<br />
== Remaking a system (on same virt) ==<br />
<br />
1. [[#cancelve|cancelve]] (or v destroy x - ONLY if you're POSITIVE no data needs to be saved)<br />
<br />
2. [[#vemake|vemake]] using same veid<br />
<br />
3. [[#mvbackups|mvbackups]] or [[#vb|vb]] (if new mount point)<br />
<br />
4. update mgmt with new dir/ip <br />
<br />
5. update firewall if changed ip<br />
<br />
== Traffic accounting on linux ==<br />
<br />
DEPRECATED - all tracking is done via bwdb now. This is how we used to track traffic.<br />
<br />
TODO: update for diff versions of vz<br />
<br />
Unlike FreeBSD, where we have to add firewall count rules to the system to count the traffic, on virtuozzo counts the traffic for us. You an see the current traffic stats by running `vznetstat`:<br />
<br />
<pre># vznetstat<br />
VEID Net.Class Output(bytes) Input(bytes)<br />
-----------------------------------------------<br />
24218 1 484M 39M<br />
24245 1 463M 143M<br />
2451 1 2224M 265M<br />
2454 1 2616M 385M<br />
4149 1 125M 68M<br />
418 1 1560M 34M<br />
472 1 1219M 315M<br />
726 1 628M 317M<br />
763 1 223M 82M<br />
771 1 4234M 437M<br />
-----------------------------------------------<br />
Total: 13780M 2090M</pre><br />
<br />
As you can see the VEID is on a line with the in and out bytes. So, we simply run a cron job:<br />
<br />
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /root/vztrafdump.sh<br />
<br />
Just like we do on FreeBSD - this one goes through all the VEs in /vz/private and greps the line from vznetstat that matches them and dumps it in /jc_traffic_dump on their system. Then it does it again for all the VEs in /vz1/private. It is important to note that vznetstat runs only once, and the grepping is done from a temporary file that contains that output - we do this because running vznetstat once for each VE that we read out of /vz/private and /vz1/private would take way too long and be too intensive.<br />
<br />
You do not need to do anything to facilitate this other than make sure that that cron job is running - the vznetstat counters are always running, and any new VEs that are added to the system will be accounted for automatically.<br />
<br />
Traffic resetting no longer works with vz 2.6, so we disable the vztrafdump.sh on those virts.<br />
<br />
== Watchdog script ==<br />
<br />
On some of the older virts, we have a watchdog running that kills procs that are deemed bad per the following:<br />
<br />
/root/watchdog from quar1<br />
<br />
<pre>if echo $line | awk '{print $(NF-3)}' | grep [5-9]...<br />
then<br />
# 50-90%<br />
if echo $line | awk '{print $(NF-1)}' | grep "...:.."<br />
then<br />
# running for > 99min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
if echo $line | awk '{print $(NF-1)}' | grep "....m"<br />
then<br />
# running for > 1000min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
fi<br />
<br />
if echo $line | awk '{print $(NF-3)}' | grep [1-9]...<br />
then<br />
# running for 10-90 percent<br />
if echo $line | awk '{print $NF}' | egrep 'cfusion|counter|vchkpw'<br />
then<br />
<br />
if echo $line | awk '{print $(NF-1)}' | grep "[2-9]:.."<br />
then<br />
# between 2-9min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
elif echo $line | awk '{print $(NF-1)}' | grep "[0-9][0-9]:.."<br />
then<br />
# up to 99min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
fi<br />
fi</pre><br />
<br />
<br />
== Misc Linux Items ==<br />
<br />
We are overselling hard drive space ... when you configure a linux system with a certain amount of disk space (the default is 4gigs) you do not actually use up 4gigs of space on the system. The diskspace setting for a user is simply a cap, and they only use up as much space on the actual disk drive as they are actually using.<br />
<br />
When you create a new linux system, even though there are some 300 RPMs or so installed, if you run `df -k` you will see that the entire 4gig partition is empty - no space is being used. This is because the files in their system are "magic symlinks" to the template for their OS that is in /vz/template - however, any changes to any of those files will "disconnect" them and they will immediately begin using space in their system. Further, any new files uploaded (even if those new files overwrite existing files) will take up space on the partition.<br />
<br />
<br />
if you see this:<br />
<br />
<pre>[root@virt8 root]# vzctl stop 160 ; vzctl start 160<br />
VE is not running<br />
Starting VE ...<br />
VE is unmounted<br />
VE is mounted<br />
Adding IP address(es): 69.55.226.83 69.55.226.84<br />
bash ERROR: Can't change file /etc/sysconfig/network<br />
Deleting IP address(es): 69.55.226.83 69.55.226.84<br />
VE is unmounted<br />
[root@virt8 root]#</pre><br />
<br />
it probably means they no longer have /bin/bash - copy one in for them<br />
<br />
ALSO: another possibility is that they have removed the `ed` RPM from their system - it needs to be reinstalled into their system. But since their system is down, this is tricky ...<br />
<br />
VE startup scripts used by 'vzctl' want package 'ed' to be available inside VE. So if package 'ed' will be enabled in OS template config and OS template itself VE #827 is based on - this error should be fixed.<br />
<br />
yes, it is possible to add RPM to VE while it not running.<br />
Try to do following:<br />
<br />
<pre># cd /vz/template/<OS_template_with_ed_package>/<br />
# vzctl mount 827<br />
# rpm -Uvh --root /vz/root/827 --veid 827 ed-0.2-25.i386.vz.rpm</pre><br />
<br />
Usually theres an error, but its ok<br />
<br />
Note: replace 'ed-0.2-25.i386.vz.rpm' in last command with actual<br />
version of 'ed' package you have.<br />
<br />
----<br />
<br />
So how do I know what template the user has ? cat their conf file and it is listed in there. For example, if the conf file has:<br />
<br />
<pre>[root@virt12 sbin]# vc 1103<br />
…snip…<br />
OSTEMPLATE="debian-3.0/20030822"<br />
TEMPLATES="mod_perl-deb30/20030707 mod_ssl-deb30/20030703 mysql-deb30/20030707 proftpd-deb30/20030703 webmin-deb30/20030823 "<br />
…</pre><br />
<br />
then they are on debian 3.0, all of their system RPMs are in /vz/template/debian-3.0, and they are using version 20030822 of that debian 3.0 template. Also, they’ve also got additional packages installed (mod_perl, mod_ssl, etc). Those are also found under /vz/template<br />
<br />
----<br />
<br />
Edits needed to run java:<br />
<br />
When we first created the VEs, the default setting for privvmpages was 93000:94000 ... which was high enough that most people never had problems ... however, you can;t run java or jdk or tomcat or anything java related with that setting. We have found that by setting privvmpages to 610000:615000 that java runs just fine. That is now the default setting. It is exceedingly rare that anyone needs it higher than that, although we have seen it once or twice.<br />
<br />
Any problems with java at all - the first thing you need to do is see if the failcnt has raised for privvmpages.<br />
<br />
<pre># vzctl start 160<br />
Starting VE ...<br />
vzquota : (error) Quota on syscall for 160: Device or resource busy<br />
Running vzquota on failed for VE 160 [3]</pre><br />
<br />
This is because my pwd is _in_ their private directory - you can't start it until you move out<br />
<br />
People seem to have trouble with php if they are clueless newbies. Here are two common problems/solutions:<br />
<br />
no... but i figured it out myself. problem was the php.ini file that came<br />
vanilla with the account was not configured to work with apache (the<br />
ENGINE directive was set to off).<br />
<br />
everything else seems fine now.<br />
<br />
----<br />
<br />
the problem was in the php.ini file. I noticed that is wasnt showing<br />
the code when it was in an html file so I looked at the php.ini file<br />
and had to change it so it recognized <? tags aswell as <?php tags.<br />
<br />
Also, make sure added to httpd.conf<br />
AddType application/x-httpd-php .php<br />
<br />
----<br />
<br />
You can set the time zone:<br />
<br />
You can change the timezone by doing this:<br />
<br />
ln -sf /usr/share/zoneinfo/<zone> /etc/localtime<br />
<br />
where <zone> is the zone you want in the /usr/share/zoneinfo/ directory.<br />
<br />
----<br />
<br />
Failing shm_open calls:<br />
<br />
first, please check if /dev/shm is mounted inside VE.<br />
'cat /proc/mounts' command should show something like this:<br />
tmpfs /dev/shm tmpfs rw 0 0<br />
<br />
If /dev/shm is not mounted you have 2 ways to solve issue:<br />
1. execute following command inside VE (doesn't require VE reboot):<br />
mount -t tmpfs none /dev/shm<br />
2. add following string to /etc/fstab inside VE and reboot it:<br />
tmpfs /dev/shm tmpfs defaults 0 0<br />
<br />
----<br />
<br />
You can have a mounted but not running ve<br />
Just:<br />
vzctl mount <veid><br />
<br />
----<br />
<br />
When a debian sys can’t get on the network, and you try:<br />
<br />
<pre>vzctl set 1046 --ipadd 69.55.227.117<br />
Adding IP address(es): 69.55.227.117<br />
Failed to bring up lo.<br />
Failed to bring up venet0.</pre><br />
<br />
They probably removed iproute package, which must be the one from swsoft. To restore:<br />
<pre># dpkg -i --veid=1046 --admindir=/vz1/private/1046/root/var/lib/dpkg --instdir=/vz1/private/1046/root/ /vz/template/debian-3.0/iproute_20010824-8_i386.vz.deb<br />
(Reading database ... 16007 files and directories currently installed.)<br />
Preparing to replace iproute 20010824-8 (using .../iproute_20010824-8_i386.vz.deb) ...<br />
Unpacking replacement iproute ...<br />
Setting up iproute (20010824-8) ...</pre><br />
<br />
Then restart their ve<br />
<br />
----<br />
<br />
in a ve i do:<br />
<br />
<pre>cd /<br />
du -h .</pre><br />
<br />
and get: 483M .<br />
<br />
i do:<br />
<br />
<pre>bash-2.05a# df -h<br />
Filesystem Size Used Avail Use% Mounted on<br />
/dev/vzfs 4.0G 2.3G 1.7G 56% /</pre><br />
<br />
how can this be?<br />
<br />
Is it possible that quota file was corrupted somehow? Please try to: <br />
<pre>vzctl stop <VEID><br />
vzquota drop <VEID><br />
vzquota init <VEID><br />
vzctl start <VEID></pre><br />
<br />
----<br />
<br />
How to stop vz from starting after reboot:<br />
<br />
VIRTUOZZO=no <br />
in <br />
/etc/sysconfig/vz<br />
<br />
To start: <br />
service vz start<br />
(after setting VIRTUOZZO=yes in /etc/sysconfig/vz)<br />
<br />
service vz restart will do some kind of 'soft reboot' -- restart all<br />
VPSes and reload modules without rebooting the node<br />
<br />
if you need to shut down all VPSes really really fast, run killall -9 init<br />
<br />
----<br />
<br />
Postfix tip:<br />
<br />
You may want to tweak settings: default_process_limit=10<br />
<br />
= Virtuozzo VPS Management Tools =<br />
<br />
== vm ==<br />
<br />
TODO<br />
<br />
== cancelve ==<br />
cancelve <veid><br />
this will:<br />
* stop a ve<br />
* check for backups (offer to remove them from the backup server <br />
and the backup.config)<br />
* rename the private dir<br />
* check for PTR, provide the commands to reset to default<br />
* and rename the ve’s config<br />
* remind you to remove firewall rules<br />
* remind you to remove DNS entries<br />
<br />
== ipadd ==<br />
ipadd <veid> <ip> [ip] [ip]<br />
add’s ip(s) to a ve<br />
<br />
== ipdel ==<br />
ipdel <veid> <ip> [ip] [ip]<br />
removes ip(s) from a ve<br />
<br />
== vc ==<br />
vc <veid><br />
the equivalent of: <tt>cat /vzconf/<veid>.conf</tt><br />
<br />
== vl ==<br />
vl<br />
will displays a list of ve #’s, 1 per line. (ostensibly to use in a for loop)<br />
<br />
== vp ==<br />
vp <veid><br />
the equivalent of: <tt>vzps auxww –E <veid></tt><br />
<br />
== vpe ==<br />
DEPRECATED<br />
vpe <veid> <br />
this will allow you to do a vp when a ve is running out of control, the equivalent of (deprecated since vp operates outside the VPS): <br />
<pre>vzctl set <veid> --kmemsize 2100000:2200000<br />
vzctl exec <veid> ps auxw<br />
vzctl set <veid> --kmemsize (ve’s orig lvalue):(ve’s orig hvalue)</pre><br />
<br />
== vt ==<br />
vt <veid><br />
the equivalent of: <tt>vztop –E <veid></tt><br />
<br />
== vr ==<br />
vr <veid><br />
the equivalent of: <tt>vzctl stop <veid>; vzctl start <veid></tt><br />
You can run this even if the ve is down - the stop command will just fail<br />
<br />
== vs ==<br />
vs [veid]<br />
displays status (output of <tt>vzctl status <veid></tt>) on each ve configured on the system (as determined by <tt>/vzconf/*.conf</tt>)<br />
If passed an argument, gives the status for just that ve. <br />
A running system looks like:<br />
<tt>VEID 16066 exist mounted running</tt><br />
<br />
A ve which is not running (but does exist) looks like:<br />
<tt>VEID 9990 exist unmounted down</tt><br />
<br />
A ve which is not running and doesn’t exist looks like:<br />
<tt>VEID 421 deleted unmounted down</tt><br />
<br />
== vs2 ==<br />
vs2 [veid]<br />
this is similar to vs in that it displays status (output of <tt>vzctl status <veid></tt>) on each ve,<br />
but the difference is it’s list comes from doing an ls on the data dirs. This was meant to catch <br />
the rare case where a ve configured but exists. <br />
<br />
== vw ==<br />
vw [veid]<br />
displays the output of ‘<tt>w</tt>’ (the equivalent of <tt>vzctl exec <veid> w</tt>) for each configured ve (as determined by <tt>/vzconf/*.conf</tt>). Useful for determing which ve is contributing to a heavily-loaded system.<br />
If passed an argument, gives ‘<tt>w</tt>‘ output for just that ve. <br />
Ex:<br />
<pre>[root@virt2 etc]# vw<br />
134<br />
10:52pm up 79 days, 6:14, 0 users, load average: 0.02, 0.02, 0.00<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT<br />
16027<br />
2:52pm up 7 days, 19:54, 0 users, load average: 0.00, 0.00, 0.00<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT<br />
16055<br />
2:52pm up 79 days, 6:38, 0 users, load average: 0.00, 0.04, 0.07<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT</pre><br />
<br />
== vwe ==<br />
vwe [constraint]<br />
just like <tt>vw</tt>, but takes a constraint as an argument, only show’s ve’s with loads >= the constraint provided. If no constraint is provided, 1 is used by default<br />
<br />
== vzs ==<br />
vzs [veid]<br />
displays the beancounter status for all ve’s, or a particular ve if an argument is passed<br />
<br />
== ve ==<br />
ve <veid><br />
the equivalent of: <tt>vzctl enter <veid></tt><br />
<br />
== vx ==<br />
vx <veid> <command><br />
the equivalent of: <tt>/usr/sbin/vzctl exec <veid> <command></tt><br />
<br />
== dprocs ==<br />
dprocs [count]<br />
a script which outputs a continuous report (or a certain number of reports if an option is passed) of processes stuck in the D state and which VPS’s those procs belong to.<br />
<br />
== setmem ==<br />
setmem <VEID> <256|512|768|1024|2048><br />
adjusts the memory resources for the VE<br />
<br />
== afacheck.sh ==<br />
afacheck.sh<br />
displays the health/status of containers and mirrors on an adaptec card (currently quar1, tempvirt1-2, virt9, virt10)- all other are LSI<br />
<br />
== backup ==<br />
backup<br />
backup script called nightly to update virt scripts and do customer backups<br />
<br />
== checkload.pl ==<br />
checkload.pl<br />
this was intended to be setup as a cronjob to watch processes on a virt when the load <br />
rises above a certain level. Not currently in use.<br />
<br />
== findbackuppigs.pl ==<br />
findbackuppigs.pl<br />
looks for files larger than 50MB which customers have asked us to backup. Emails matches<br />
to linux@johncompanies.com<br />
<br />
== gatherlinux.pl ==<br />
gatherlinux.pl<br />
gathers up data about ve’s configured and writes to a file. Used for audits against the db<br />
<br />
== gb ==<br />
gb <search><br />
greps backup.config for the given search parameter<br />
<br />
== gbg ==<br />
gbg <search><br />
greps backup.config for the given search parameter and presents just the directories (for clean pasting)<br />
<br />
== linuxtrafficgather.pl ==<br />
linuxtrafficgather.pl [yy-mm]<br />
sends a traffic usage summary by ve to support@johncomapnies.com and payments@johncompanies.com.<br />
Optional arguments are year and month (must be in the past). If not passed, assumes last month. Relies on <br />
traffic logs created by netstatreset and netstatbackup<br />
<br />
== linuxtrafficwatch.pl ==<br />
linuxtrafficwatch.pl<br />
checks traffic usage and emails support@johncomapnies.com when a ve reaches the warning level (35G) and<br />
the limit (40G). Works on virtuozzo versions <= 2.6.x. We really aren’t using this anymore now that we have netflow.<br />
<br />
== linuxtrafficwatch2.pl ==<br />
linuxtrafficwatch2.pl<br />
checks traffic usage and emails support@johncomapnies.com when a ve reaches the warning level (35G) and<br />
the limit (40G). Works on virtuozzo version 2.6.x. We really aren’t using this anymore now that we have netflow.<br />
<br />
== load.pl ==<br />
load.pl<br />
feeds info to load mrtg - executed by inetd.<br />
<br />
== mb (linux) ==<br />
mb <mount|umount><br />
(nfs) mounts and umounts dirs to backup2. Shortcuts are mbm and mbu to mount and unmount. <br />
<br />
== migrate ==<br />
migrate <ip of node migrating to> <veid> <target_veid> [target dir: vz | vz1 | vz2]<br />
this is basically a wrapper for <tt>vzmigrate</tt> – vzmigrate is a util to seamlessly move a ve from one host to another. This wrapper was written cause vz virtuozzo version 2.6 had a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables. This script mitigates that. Since it makes multiple ssh connections to the target host, it’s a good idea to put the pub key for the src system in the authorized_keys file on the target host. In addition, it emails ve owners when their migration starts and stops (if they place email addresses in a file on their system: /migrate_notify). To move everyone off a system, you’d do:<br />
for f in `vl`; do migrate <ip> $f; done<br />
<br />
== migrateonline ==<br />
migrateonline <ip of node migrating to> <veid> <target_veid> [target dir: vz | vz1 | vz2]<br />
this is the same as migrate but will migrate a ve in <tt>–online</tt> mode which means it won’t be shut down at the end of the migration. This only works when migrating ve’s between 2 machines running a 2.6 kernel (currently tempvirt1-2. virt16-19, virt12). If you get an error that the machine you’re trying to migrate to has a different CPU or features, etc, then you have to edit the file and add the –f switch to the vzmigrate line- you can basically ignore this kind of warning (but never ignore a warning about missing templates on the destination node). NOTE: This edit (if made to migrateonline) will be overwritten by the base script during each night’s backup.<br />
<br />
== netstatbackup ==<br />
DEPRECATED<br />
netstatbackup <br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.5.x. <br />
<br />
== netstatbackup2 ==<br />
DEPRECATED<br />
netstatbackup2<br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.6.x. <br />
<br />
== netstatreset ==<br />
DEPRECATED<br />
netstatreset<br />
writes traffic count data to a logfile and resets counters to 0. Works on virtuozzo versions 2.5.x <br />
<br />
== netstatreset2 ==<br />
DEPRECATED<br />
netstatreset2<br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.6.x. <br />
<br />
== orphanedbackupwatchlinux ==<br />
orphanedbackupwatchlinux <br />
looks for directories on backup2 which aren’t configured in backup.config and offers to <br />
delete them<br />
<br />
== rsync.backup (linux) ==<br />
rsync.backup<br />
does customer backups (relies on backup.config)<br />
<br />
== startvirt.pl ==<br />
startvirt.pl<br />
forks off start ve commands – keeps 6 running at a time. This is not to be used on systems where fastboot is enabled as it circumvents the benefit of the fastboot. The script will occasionally not exit gracefully and will continue to use up CPU, so it should be watched. Also, don’t exit from the script till you’re sure all ve’s are started – if you do you need to start them manually and may have to free up locks. On some systems, the startvirt script doesn’t exit cleanly and you have to ^C out of it. Be careful though- doing so can leave some VE’s in an odd bootup state and you may need to ‘vr’ them manually. You should check to see which ve’s aren’t running and/or confirm all have started when ^C’ing out of startvirt.<br />
<br />
== taskdone (linux) ==<br />
taskdone<br />
when called will email support@johncompanies.com with the hostname of the machine from which it was <br />
executed as the subject<br />
<br />
== vb (linux) ==<br />
vb<br />
the equivalent of: <tt>vi /usr/local/sbin/backup.config</tt><br />
<br />
== vemakeXX ==<br />
DEPRECATED<br />
vemakerh9 <br />
ve create script for RH9 (see vemake)<br />
<br />
vemakedebian30 <br />
ve create script for debian 3.0 (Woody) (see vemake)<br />
<br />
vemakedebian31 <br />
ve create script for debian 3.1 (Sarge) (see vemake)<br />
<br />
vemakedebian40 <br />
ve create script for debian 4.0 (Etch) (see vemake)<br />
<br />
vemakefedora, vemakefedora2, vemakefedora4, vemakefedora5, vemakefedora6, vemakefedora7<br />
ve create script for fedora core 1, 2, 4, 5, 6, 7 respectively (see vemake)<br />
<br />
vemakecentos3, vemakecentos4<br />
ve create script for centos 3, 4 respectively (see vemake)<br />
<br />
vemakesuse, vemakesuse93, vemakesuse100<br />
ve create script for suse 9.2, 9.3, 10.0 respectively (see vemake)<br />
<br />
vemakeubuntu5, vemakeubuntu606, vemakeubuntu606 vemakeubuntu704<br />
ve create script for ubuntu 5.10, 6.06, 6.10, 7.04 respectively (see vemake)<br />
<br />
== vemove ==<br />
DEPRECATED<br />
vemove <veid> <target_ip> </vz/private/123><br />
this script simplifies the old way of moving ve’s from one system to another - in short moving a ve to or from a virt running virtuozzo < 2.6.x<br />
It’s the equivalent of:<br />
<tt>tar cfpP - <veid> --ignore-failed-read | (ssh -2 -c arcfour <target_ip> "split - -b 1024m </vz/private/123>.tar" )</tt>This should only be used if migrate/vzmigrate can’t be used. <br />
<br />
== vim.watchdog ==<br />
vim.watchdog <br />
looks for and kills procs matching “vi|vim|nano|pine|elm” that are running for a long time & consuming lots of cpu. Works on virtuozzo versions 2.5.x<br />
<br />
== vim.watchdog2 ==<br />
vim.watchdog2<br />
looks for and kills procs matching “vi|vim|nano|pine|elm” that are running for a long time & consuming lots of cpu.<br />
Works on virtuozzo versions 2.6.x.<br />
<br />
== vzmigrate ==<br />
vzmigrate <target_ip> -r no <veid>:[dst veid]:[dst /vzX/private/veid]:[dst /vzX/root/veid]<br />
(this is the raw command “wrapped” by migrate/migrateonline) this will seamlessly move a ve from one host to another. The ve will run for the duration of the migration till the very end when it’s shut down, ip moved and started up on the target system. The filesystem on the src will remain. This should be watched – occasionally the move will timeout and leave the system shut down. If target private and root aren’t specified it just puts it in /vz. Only works when both systems are running virtuozzo 2.6.x<br />
<br />
== vztrafdump.sh ==<br />
DEPRECATED<br />
vztrafdump.sh<br />
writes traffic usage info by ve to a file called jc_traffic_dump in each ve’s / dir. Works on virtuozzo versions <= 2.5.x. <br />
<br />
== vztrafdump2.sh ==<br />
DEPRECATED<br />
vztrafdump2.sh<br />
writes traffic usage info by ve to a file called jc_traffic_dump in each ve’s / dir. Works on virtuozzo versions 2.6.x. <br />
<br />
== addtun ==<br />
addtun <veid><br />
Add’s tun device to ve.<br />
<br />
== bwcap ==<br />
bwcap <veid> <kbps><br />
Ex: <tt>bwcap 1234 512</tt><br />
Caps a VE’s bandwidth to the amount given<br />
<br />
== setdisk ==<br />
setdisk <veid> <diskspace in GB><br />
Ex: <tt>setdisk 1234 5</tt><br />
Gives a VE’s a given amount of disk space<br />
<br />
== vdf ==<br />
vdf <veid> <br />
the equivalent of: <tt>vzctl exec <veid> df –h</tt><br />
<br />
== vdff ==<br />
vdff<br />
runs a (condensed) vdf for all ve’s in your pwd (must be run from /vz/privateN)<br />
<br />
== mvbackups ==<br />
mvbackups <veid> <target_machine> (virt1) <target_dir> (vz1)<br />
moves backups from one location to another on the backup server, and provides you with option to remove entries from current backup.config, and simple paste command to add the config to backup.config on the target server<br />
<br />
== checkquota ==<br />
checkquota<br />
for all the ve’s in the cwd (run from /vz/private, /vz1/private, etc) reports what vz quota says they’re using and what the actual usage is (as reported by du)<br />
<br />
== clearquota ==<br />
clearquota <veid><br />
Recalculates a ve’s quota, prints out the usage before and after. The equivalent of:<br />
<tt>vdf <veid>; v stop <veid>; vzquota drop <veid>; v start <veid>; vdf <veid></tt><br />
<br />
== dprocs ==<br />
dprocs<br />
Sometimes the server’s have a large number of processes get stuck in the D state- this script shows (every 3 secs) which VE’s have D procs, which procs<br />
are stuck and a running average of the top “offenders”<br />
<br />
== vzstat ==<br />
vstat<br />
sort of like top for VZ. sort VEs by CPU usage by pressing 'o' and then 'c' keys</div>Supporthttps://wiki.jcihosting.com/index.php?title=VPS_Management&diff=1016VPS Management2013-03-11T23:03:38Z<p>Support: /* Assigning/remove ip from a VE */</p>
<hr />
<div>= Common Problems =<br />
== Login to any machine without a password ==<br />
<br />
This is possible via the use of ssh keys. The process is thus:<br />
<br />
1. place the public key for your user (root@mail) in the /root/.ssh/authorized_keys file on the server you wish to login to<br />
cat /root/.ssh/id_dsa.pub<br />
(paste that into authorized_keys on the target server). If the file doesn't exist, create it.<br />
<br />
2. enable root login (usually only applies to FreeBSD). Edit the /etc/ssh/sshd_config on the target server and change:<br />
<tt>#PermitRootLogin no</tt><br />
to<br />
<tt>PermitRootLogin yes</tt><br />
<br />
3. Restart the sshd on the target machine. First, find the sshd process: <br />
jailps <hostname> | grep sshd <br />
or <br />
vp <VEID> | grep sshd<br />
<br />
Look for the process resembling:<br />
root 17296 0.0 0.0 5280 1036 ? Ss 2011 4:27 /usr/sbin/sshd <br />
(this is the sshd)<br />
<br />
Not:<br />
root 6270 0.5 0.0 6808 2536 ? Ss 14:33 0:00 sshd: root [priv]<br />
(this is an sshd child- someone already ssh'd in as root)<br />
<br />
Restart the sshd: <br />
kill -1 <PID><br />
<br />
Ex:<br />
kill -1 17296<br />
<br />
You may now ssh in.<br />
<br />
Once you're done, IF you enabled root login, you should repeat steps 2 and 3 to disable root logins.<br />
<br />
== Letting someone in who has locked themselves out (killed sshd, lost pwd) ==<br />
<br />
There are two ways people frequently lock themselves out - either they forget a password, or they kill off sshd somehow.<br />
<br />
These are actually both fairly easy to solve. First, let's say someone kills off their sshd, or somehow mangles /etc/ssh/sshd_config such that it no longer lets them in.<br />
<br />
Their email may be very short, or it may have all sorts of details about how you should fix sshd_config to let them in ... just ignore all of this. They can fix their own mangled sshd. Fixing this is very simple. First, edit the /etc/inetd.conf on their system and uncomment the telnet line:<br />
<br />
telnet stream tcp nowait root /usr/libexec/telnetd telnetd<br />
#telnet stream tcp6 nowait root /usr/libexec/telnetd telnetd<br />
<br />
(just leave the tcp6 version of telnet commented)<br />
<br />
Then, use jailps to list the processes on their system, and find their inetd process. Then simply:<br />
<br />
kill -HUP (pid)<br />
<br />
where (pid) is the PID of their inetd process. Now they have telnet running on their system and they can log in and do whatever they need to do.<br />
<br />
The only complications that could occur are:<br />
<br />
a) their firewall config on our firewall has port 23 blocked, in which case you will need to open that - will be covered in a different lesson.<br />
<br />
b) they are not running inetd, so you can't HUP it. If this happens, edit their /etc/rc.conf, add the inetd_enable="YES" line, and then kill<br />
their jail with /tmp/jailkill.pl - then restart their jail with the jail line from their quad/safe file. Easy.<br />
<br />
If they have forgotten a password,<br />
<br />
On 6.x+ you can reset their password with:<br />
jexec <jailID from jls> passwd root<br />
<br />
Note: the default password for 6.x jails is 8ico2987, for 4.x it is p455agfa<br />
<br />
On 4.x, you need to cd to their etc directory<br />
... for instance:<br />
<br />
cd /mnt/data2/198.78.65.136-col00261-DIR/etc<br />
<br />
and run:<br />
<br />
vipw -d .<br />
<br />
Then paste in these two lines (theres a paste with these):<br />
<br />
root:$1$krszPxhk$xkCepSnz3mIikT3vCtJCt0:0:0::0:0:Charlie &:/root:/bin/csh<br />
user:$1$Mx9p5Npk$QdMU6c8YQqp2FW2M3irEh/:1001:1001::0:0:User &:/home/user:/bin/sh<br />
<br />
overwriting the lines they already have for "user" and "root" - then just tell them that both user and root have been reset to the default password of p455agfa.<br />
<br />
For linux, just passwd inside shell or <br />
vzctl set <veid> --userpasswd root:p455agfa –save<br />
<br />
Starting in 2009 we began giving out randomized passwords for FreeBSD and Linux as the default password. That is stored with each system in Mgmt. You should look for and reset the password to that password in the event of a reset and refer the customer to use their original password from their welcome email- this way we don’t have to send the password again via email (in clear text).<br />
<br />
<br />
== sendmail can’t be contacted from ext ip (only locally) ==<br />
<br />
By default redhat puts this line in sendmail.mc:<br />
<br />
DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')<br />
<br />
which makes it only answer on localhost. Comment it out like:<br />
<br />
dnl DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')<br />
<br />
and then rebuild sendmail.cf with:<br />
<br />
m4 /etc/mail/sendmail.mc > /etc/sendmail.cf<br />
<br />
== virt doesn’t properly let go of ve’s ip(s) when moved to another system ==<br />
<br />
On virtuozzo 2.6 systems, it's been observed that when moving ips from one virt to another that sometimes the routing table will not get updated to reflect the removal of the ip addresses.<br />
<br />
A recent example was a customer that was moving to a new ve on a new virt and the ip addresses were traded between the two ve's. After the trade the two systems were not able to talk to each other. When looking at the routing table for the old system all the ip addresses were still in the routing table as being local, like so:<br />
<br />
<pre>netstat -rn | grep 69.55.225.149<br />
69.55.225.149 0.0.0.0 255.255.255.255 UH 40 0 0 venet0</pre><br />
<br />
This was preventing traffic to the other system from being routed properly.<br />
The solution is to manually delete the route:<br />
<br />
route delete 69.55.225.149 gw 0.0.0.0<br />
<br />
Supposedly, this was fixed in 2.6.1<br />
<br />
== sshd on FreeBSD 6.2 segfaults ==<br />
<br />
First try to reinstall ssh<br />
<br />
<pre>cd /usr/src/secure<br />
cd lib/libssh<br />
make depend && make all install<br />
cd ../../usr.sbin/sshd<br />
make depend && make all install<br />
cd ../../usr.bin/ssh<br />
make depend && make all install</pre><br />
<br />
Failing that, find the library that’s messed up:<br />
<br />
<pre>ldd /usr/sbin/sshd<br />
libssh.so.3 => /usr/lib/libssh.so.3 (0x280a3000) <br />
libutil.so.5 => /lib/libutil.so.5 (0x280d8000) <br />
libz.so.3 => /lib/libz.so.3 (0x280e4000) <br />
libwrap.so.4 => /usr/lib/libwrap.so.4 (0x280f5000) <br />
libpam.so.3 => /usr/lib/libpam.so.3 (0x280fc000) <br />
libbsm.so.1 => /usr/lib/libbsm.so.1 (0x28103000) <br />
libgssapi.so.8 => /usr/lib/libgssapi.so.8 (0x28112000) <br />
libkrb5.so.8 => /usr/lib/libkrb5.so.8 (0x28120000) <br />
libasn1.so.8 => /usr/lib/libasn1.so.8 (0x28154000) <br />
libcom_err.so.3 => /usr/lib/libcom_err.so.3 (0x28175000) <br />
libroken.so.8 => /usr/lib/libroken.so.8 (0x28177000) <br />
libcrypto.so.4 => /lib/libcrypto.so.4 (0x28183000) <br />
libcrypt.so.3 => /lib/libcrypt.so.3 (0x28276000) <br />
libc.so.6 => /lib/libc.so.6 (0x2828e000) <br />
libmd.so.3 => /lib/libmd.so.3 (0x28373000)</pre><br />
<br />
md5 them and compare to other jail hosts or jails running on host<br />
<br />
for libcrypto reinstall:<br />
<pre>/usr/src/crypto<br />
make depend && make all install</pre><br />
<br />
= FreeBSD VPS =<br />
<br />
== Starting jails: Quad/Safe Files ==<br />
<br />
FreeBSD customer systems do not start up automatically at boot time. When one of our freebsd machines boots up, it boots up, and does nothing else. To start jails, we put the commands to start each jail into a shell script(s) and run the script(s). Jail startup is something that needs to be actively monitored, which is why we don’t just run the script automatically. More on monitoring later.<br />
<br />
NOTE: >=7.x we have moved to 1 quad file: <tt>quad1</tt>. Startups are not done by running each quad, but rather [[#startalljails|startalljails]] which relies on the contents of <tt>quad1</tt>. The specifics of this are lower in this article. What follows here applies for pre 7.x systems.<br />
<br />
There are eight files in <tt>/usr/local/jail/rc.d</tt>:<br />
<br />
<pre>jail3# ls /usr/local/jail/rc.d/<br />
quad1 quad2 quad3 quad4 safe1 safe2 safe3 safe4<br />
jail3#</pre><br />
<br />
four quad files and four safe files.<br />
<br />
Each file contains an even number of system startup blocks (total number of jails divided by 4)<br />
<br />
The reason for this is, if we make one large script to startup all the systems at boot time, it will take too long - the first system in the script will start up right after system boot, which is great, but the last system may not start for another 20 minutes.<br />
<br />
Since there is no way to parralelize this during the startup procedure, we simply open four terminals (in screen window 9) and run each script, one in each terminal. This way they all run simultaneously, and the very last system in each startup script gets started in 1/4th the time it would if there was one large file<br />
<br />
The files are generally organized so that quad/safe 1&2 have only jails from disk 1, and quad/safe 3&4 have jails from disk 2. This helps ensure that only 2 fscks on any disk are going on at once. Further, they are balanced so that all quad/safe’s finish executing around the same time. We do this by making sure each quad/safe has a similar number of jails and represents a similar number of inodes (see js).<br />
<br />
The other, very important reason we do it this way, and this is the reason there are quad files and safe files, is that in the event of a system crash, every single vn-backed filesystem that was mounted at the time of system crash needs to be fsck'd. However, fsck'ing takes time, so if we shut the system down gracefully, we don't want to fsck.<br />
<br />
Therefore, we have two sets of scripts - the four quad scripts are identical to the four safe scripts except for the fact that the quad scripts contain fsck commands for each filesystem.<br />
<br />
So, if you shut a system down gracefully, start four terminals and run safe1 in window one, and safe2 in window 2, and so on.<br />
<br />
If you crash, start four terminals (or go to screen window 9) and run quad1 in window one, and quad2 in window 2, and so on.<br />
<br />
Here is a snip of (a 4.x version) quad2 from jail17:<br />
<br />
<pre>vnconfig /dev/vn16 /mnt/data2/69.55.228.7-col00820<br />
fsck -y /dev/vn16<br />
mount /dev/vn16c /mnt/data2/69.55.228.7-col00820-DIR<br />
chmod 0666 /mnt/data2/69.55.228.7-col00820-DIR/dev/null<br />
jail /mnt/data2/69.55.228.7-col00820-DIR mail1.phimail.com 69.55.228.7 /bin/sh /etc/rc<br />
<br />
# moved to data2 col00368<br />
#vnconfig /dev/vn28 /mnt/data2/69.55.236.132-col00368<br />
#fsck -y /dev/vn28<br />
#mount /dev/vn28c /mnt/data2/69.55.236.132-col00368-DIR<br />
#chmod 0666 /mnt/data2/69.55.236.132-col00368-DIR/dev/null<br />
#jail /mnt/data2/69.55.236.132-col00368-DIR limehouse.org 69.55.236.132 /bin/sh /etc/rc<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
vnconfig /dev/vn22 /mnt/data2/69.55.228.13-col01063<br />
fsck -y /dev/vn22<br />
mount /dev/vn22c /mnt/data2/69.55.228.13-col01063-DIR<br />
chmod 0666 /mnt/data2/69.55.228.13-col01063-DIR/dev/null<br />
jail /mnt/data2/69.55.228.13-col01063-DIR www.widestream.com.au 69.55.228.13 /bin/sh /etc/rc<br />
<br />
# cancelled col00106<br />
#vnconfig /dev/vn15 /mnt/data2/69.55.238.5-col00106<br />
#fsck -y /dev/vn15<br />
#mount /dev/vn15c /mnt/data2/69.55.238.5-col00106-DIR<br />
#chmod 0666 /mnt/data2/69.55.238.5-col00106-DIR/dev/null<br />
#jail /mnt/data2/69.55.238.5-col00106-DIR mail.azebu.net 69.55.238.5 /bin/sh /etc/rc</pre><br />
<br />
As you can see, two of the systems specified are commented out - presumably those customers cancelled, or were moved to new servers.<br />
<br />
As you can see, the vnconfig line is the simpler command line, not the longer one that was used when it was first configured. As you can see, all that is done is, vnconfig the filesystem, then fsck it, then mount it. The fourth command is the `jail` command used to start the system – but that will be covered later.<br />
<br />
Here is the safe2 file from jail17:<br />
<br />
<pre>vnconfig /dev/vn16 /mnt/data2/69.55.228.7-col00820<br />
mount /dev/vn16c /mnt/data2/69.55.228.7-col00820-DIR<br />
chmod 0666 /mnt/data2/69.55.228.7-col00820-DIR/dev/null<br />
jail /mnt/data2/69.55.228.7-col00820-DIR mail1.phimail.com 69.55.228.7 /bin/sh /etc/rc<br />
<br />
# moved to data2 col00368<br />
#vnconfig /dev/vn28 /mnt/data2/69.55.236.132-col00368<br />
#mount /dev/vn28c /mnt/data2/69.55.236.132-col00368-DIR<br />
#chmod 0666 /mnt/data2/69.55.236.132-col00368-DIR/dev/null<br />
#jail /mnt/data2/69.55.236.132-col00368-DIR limehouse.org 69.55.236.132 /bin/sh /etc/rc<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
vnconfig /dev/vn22 /mnt/data2/69.55.228.13-col01063<br />
mount /dev/vn22c /mnt/data2/69.55.228.13-col01063-DIR<br />
chmod 0666 /mnt/data2/69.55.228.13-col01063-DIR/dev/null<br />
jail /mnt/data2/69.55.228.13-col01063-DIR www.widestream.com.au 69.55.228.13 /bin/sh /etc/rc<br />
<br />
# cancelled col00106<br />
#vnconfig /dev/vn15 /mnt/data2/69.55.238.5-col00106<br />
#mount /dev/vn15c /mnt/data2/69.55.238.5-col00106-DIR<br />
#chmod 0666 /mnt/data2/69.55.238.5-col00106-DIR/dev/null<br />
#jail /mnt/data2/69.55.238.5-col00106-DIR mail.azebu.net 69.55.238.5 /bin/sh /etc/rc</pre><br />
<br />
As you can see, it is exactly the same, but it does not have the fsck lines.<br />
<br />
Take a look at the last entry - note that the file is named:<br />
<br />
/mnt/data2/69.55.238.5-col00106<br />
<br />
and the mount point is named:<br />
<br />
/mnt/data2/69.55.238.5-col00106-DIR<br />
<br />
This is the general format on all the FreeBSD systems. The file is always named:<br />
<br />
IP-custnumber<br />
<br />
and the directory is named:<br />
<br />
IP-custnumber-DIR<br />
<br />
If you run safe when you need a fsck, the mount will fail and jail will fail:<br />
<br />
# mount /dev/vn1c /mnt/data2/jails/65.248.2.131-ns1.kozubik.com-DIR<br />
mount: /dev/vn1c: Operation not permitted<br />
<br />
No reboot needed, just run the quad script<br />
<br />
Starting with 6.x jails, we added block delimiters to the quad/safe files, the block looks like:<br />
<br />
<pre>echo '## begin ##: nuie.solaris.mu'<br />
fsck -y /dev/concat/v30v31a<br />
mount /dev/concat/v30v31a /mnt/data1/69.55.228.218-col01441-DIR<br />
mount_devfs devfs /mnt/data1/69.55.228.218-col01441-DIR/dev<br />
devfs -m /mnt/data1/69.55.228.218-col01441-DIR/dev rule -s 3 applyset<br />
jail /mnt/data1/69.55.228.218-col01441-DIR nuie.solaris.mu 69.55.228.218 /bin/sh /etc/rc<br />
echo '## end ##: nuie.solaris.mu'</pre><br />
<br />
These are more than just informative when running quad/safe’s, the echo lines MUST be present for certain tools to work properly. So it’s important that any updates to the hostname also be updated on the 2 echo lines. For example, if you try to startjail a jail with a hostname which is on the jail line but not the echo lines, the command will return with host not found.<br />
<br />
=== FreeBSD 7.x+ notes ===<br />
<br />
Starting with the release of FreeBSD 7.x, we are doing jail startups in a slightly different way. First, thereis only 1 file: <tt>/usr/local/jail/rc.d/quad1</tt><br />
There are no other quads or corresponding safe files. The reason for this is twofold, 1. We can pass –C to fsck which will tell is to skip the fsck if the fs is clean (no more need for safe files), 2. We have a new startup script which can be launched multiple times, running in parallel to start jails, where quad1 is the master jail file. <br />
Quad1 could still be run as a shell script, but it would take a very long time for it to run completely so it’s not advisable; or you should break it down into smaller chunks (like quad1, quad2, quad3, etc)<br />
<br />
Here is a snip of (a 7.x version) quad1 from jail2:<br />
<br />
<pre>echo '## begin ##: projects.tw.com'<br />
mdconfig -a -t vnode -f /mnt/data1/69.55.230.46-col01213 -u 50<br />
fsck -Cy /dev/md50c<br />
mount /dev/md50c /mnt/data1/69.55.230.46-col01213-DIR<br />
mount -t devfs devfs /mnt/data1/69.55.230.46-col01213-DIR/dev<br />
devfs -m /mnt/data1/69.55.230.46-col01213-DIR/dev rule -s 3 applyset<br />
jail /mnt/data1/69.55.230.46-col01213-DIR projects.tw.com 69.55.230.46 /bin/sh /etc/rc<br />
echo '## end ##: projects.tw.com'</pre><br />
<br />
Cancelled jails are no longer commented out and stored in quad1, rather they’re moved to <tt>/usr/local/jail/rc.d/deprecated</tt> <br />
<br />
To start these jails, start the 4 ssh sessions as you would for a normal crash and then instead of running quad1-4, instead run startalljails in each window. IMPORTANT- before running startalljails you should make sure you ran preboot once as it will clear out all the lockfiles and enable startalljails to work properly.<br />
<br />
== Problems with the quad/safe files ==<br />
<br />
When you run the quad/safe files, there are two problems that can occur - either a particular system will hang during initialization, OR a system will spit out output to the screen, impeding your ability to do anything. Or both.<br />
<br />
First off, when you start a jail, you see output like this:<br />
<br />
<pre>Skipping disk checks ...<br />
adjkerntz[25285]: sysctl(put_wallclock): Operation not permitted<br />
Doing initial network setup:.<br />
ifconfig: ioctl (SIOCDIFADDR): permission denied<br />
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384<br />
Additional routing options: TCP keepalive=YESsysctl:<br />
net.inet.tcp.always_keepalive: Operation not permitted.<br />
Routing daemons:.<br />
Additional daemons: syslogd.<br />
Doing additional network setup:.<br />
Starting final network daemons:.<br />
ELF ldconfig path: /usr/lib /usr/lib/compat /usr/X11R6/lib /usr/local/lib<br />
a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout /usr/X11R6/lib/aout<br />
Starting standard daemons: inetd cron sshd sendmail sendmail-clientmqueue.<br />
Initial rc.i386 initialization:.<br />
Configuring syscons: blanktime.<br />
Additional ABI support:.<br />
Local package initialization:.<br />
Additional TCP options:.</pre><br />
<br />
Now, let's look at this line, near the end:<br />
<br />
Local package initialization:.<br />
<br />
This is where a list of daemons that are set to start at boot time willshow up. You might see something like:<br />
<br />
Local package initialization: mysqld apache sendmail sendmail-clientmqueue<br />
<br />
Or something like this:<br />
<br />
Local package initialization: postgres postfix apache<br />
<br />
The problem is that many systems (about 4-5 per machine) will hang on that line. Basically it will get to some of the way through the total daemons to be started:<br />
<br />
Local package initialization: mysqld apache<br />
<br />
and will just sit there. Forever.<br />
<br />
Fortunately, pressing ctrl-c will break out of it. Not only will it break out of it, but it will also continue on that same line and start the other daemons:<br />
<br />
Local package initialization: mysqld apache ^c sendmail-clientmqueue<br />
<br />
and then continue on to finish the startup, and then move to the next system to be started.<br />
<br />
So what does this mean? It means that if a machine crashes, and you start four screen-windows to run four quads or four safes, you need to periodically cycle between them and see if any systems are stuck at that point, causing their quad/safe file to hang. A good rule of thumb is, if you see a system at that point in the startup, give it another 100 seconds - if it is still at the exact same spot, hit ctrl-c. Its also a good idea to go back into the quad file (just before the first command in the jail startup block) and note that this jail tends to need a control-c or more time as follows:<br />
<pre>echo '### NOTE ### slow sendmail'<br />
echo '### NOTE ###: ^C @ Starting sendmail.'</pre><br />
<br />
'''NEVER''' hit ctrl-c repeatedly if you don't get an immediate response - that will cause the following jail’s startup commands to be aborted.<br />
<br />
A second problem that can occur is that a jail - maybe the first one in that particular quad/safe, maybe the last one, or maybe one in the middle, will start spitting out status or error messages from one of its init scripts. This is not a problem - basically, hit enter a few times and see if you get a prompt - if you do get a prompt, that means that the quad/safe script has already completed. Therefore it is safe to log out (and log out of the user that you su'd from) and then log back in (if necessary).<br />
<br />
The tricky thing is, if a system in the middle starts flooding with messages, and you hit enter a few times and don't get a prompt. Are you not getting a prompt because some subsequent system is hanging at the initialization, as we discussede above ? Or are you not getting a prompt because that quad file is currently running an fsck ? Usually you can tell by scrolling back in screen’s history to see what it was doing before you started getting the messages.<br />
<br />
If you don’t get clues from history, you have to use your judgement - instead of giving it 100 seconds to respond, perhaps give it 2-3 mins ... if you still get no response (no prompt) when you hit enter, hit ctrl-c. However, be aware that you might still be hitting ctrl-c in the middle of an fsck. This means you will get an error like "filesystem still marked dirty" and then the vnconfig for it will fail and so will the jail command, and the next system in the quad file will then start starting up.<br />
<br />
If this happens, just wait until the end of all the quad files have finished, and start that system manually.<br />
<br />
If things really get weird, like a screen flooded with errors, and you can't get a prompt, and ctrl-c does nothing, then you need to just eventually (give it ten mins or so) just kill that window with ctrl-p, then k, and then log in again and manually check which systems are now running and which aren't, and manually start up any that are not.<br />
<br />
Don't EVER risk running a particular quad/safe file a second time.<br />
If the quad/safe script gets executed twice, reboot the machine immediately.<br />
<br />
So, for all the above reasons, anytime a machine crashes and you run all the quads or all the safes, '''always''' check every jail afterwards to make sure it is running - even if you have no hangs or complications at all.<br />
Run this command:<br />
<br />
<tt>[[#jailpsall|jailpsall]]</tt> <br />
<br />
Note: [[#postboot|postboot]] also populates ipfw counts, so it '''should not be run multiple times''', use <tt>jailpsall</tt> for subsequent extensive ps’ing<br />
<br />
And make sure they all show as running. If one does not show as running, check its /etc/rc.conf file first to see if maybe it is using a different hostname first before starting it manually.<br />
<br />
One thing we have implemented to alleviate these startup hangs and noisy jails, is to put jail start blocks that are slow or hangy at the bottom of the safe/quad file. Further, for each bad jail we note in each quad/safe just before the start block something like:<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
<br />
That way we’ll be prepared to ^C when we see that message appear during the quad/safe startup process. If you observe a new, undocumented hang, '''after''' the quad/safe has finished, place a line similar to the above in the quad file, move the jail start block to the end of the file, then run [[#buildsafe|buildsafe]]<br />
<br />
<br />
== Recovering from a crash (FreeBSD) ==<br />
<br />
=== Diagnose whether you have a crash ===<br />
The most important thing is to get the machine and all jails back up as soon as possible. Note the time, you’ll need to create a crash log entry (Mgmt. -> Reference -> CrashLog). The first thing to do is head over to the [[Screen#Screen_Organization|serial console screen]] and see if there’s any kernel error messages output. Try to copy any messages (or just a sample of repeating messages) you see into the notes section of the crash log. If there are no messages, the machine may just be really busy- wait a bit (5-10min) to see if it comes back. If it's still pinging, odds are its very busy. Note, if you see messages about swap space exhausted, the server is obviously out of memory, however it may recover briefly enough for you to get a jtop in to see who's lauched a ton of procs (most likely) and then issue a quick jailkill to get it back under control.<br />
<br />
If it doesn't come back, or the messages indicate a fatal error, you will need to proceed with a power cycle (ctrl+alt+del will not work).<br />
<br />
=== Power cycle the server ===<br />
If this machine is not a Dell 2950 with a [[DRAC/RMM#DRAC|DRAC card]] (i.e. if you can’t ssh into the DRAC card (as root, using the standard root pass) and issue <br />
racadm serveraction hardreset<br />
then you will need someone at the data center power the macine off, wait 30 sec, then turn it back on. Make sure to re-attach via console:<br />
tip jailX<br />
immediately after power down. <br />
<br />
=== (Re)attach to the console ===<br />
Stay on the console the entire time during boot. As the BIOS posts- look out for the RAID card output- does everything look healthy? The output may be scrambled, look for "DEGRADED" or "FAILED". Once the OS starts booting you will be disconnected (dropped back to the shell on the console server) a couple times during the boot up. The reason you want to quickly re-attach is two-fold: 1. If you don’t reattach quickly then you won’t get any console output, 2. you want to be attached before the server ''potentially'' starts (an extensive) fsck. If you attach after the fsck begins, you’ll have seen no indication it started an fsck and the server will appear frozen during startup- no output, no response. <br />
<br />
IMPORTANT NOTE: on some older FreeBSD systems, there will be no output to the video (KVM) console as it boots up. The console output is redirected to the serial port ... so if a jail crashes, and you attach a kvm, the output during the bootup procedure will not be shown on the screen. However, when the bootup is done, you will get a login prompt on the screen and will be able to log in as normal. <tt>/boot/loader.conf</tt> is where serial console redirect output lives, so comment that if you want to catch output on kvm.<br />
On newer systems it sends most output to both locations. <br />
<br />
=== Assess the heath of the server ===<br />
Once the server boots up fully, you should be able to ssh in. Look around- make sure all the mounts are there and reporting the correct size/usage (i.e. /mnt/data1 /mnt/data2 /mnt/data3 - look in /etc/fstab to determine which mount points should be there), check to see if RAID mirrors are healthy. See [[RAID_Cards#Common_CLI_commands_.28megacli.29|megacli]], [[#aaccheck|aaccheck]]<br />
<br />
Before you start the jails, you need to run [[#preboot|preboot]]. This will do some assurance checks to make sure things are prepped to start the jails. Any issues that come out of preboot need to be addressed before starting jails.<br />
<br />
=== Start jails ===<br />
[[#Starting_jails:_Quad.2FSafe_Files|More on starting jails]]<br />
Customer jails (the VPSs) do not start up automatically at boot time. When a FreeBSD machines boots up, it boots up, and does nothing else. To start jails, we put the commands to start each jail into a shell script(s) and run the script(s). Jail startup is something that needs to be actively monitored, which is why we don’t just run the script automatically. <br />
<br />
In order to start jails, we run the quad files: quad1 quad2 quad3 and quad4 (on new systems there is only quad1). If the machine was cleanly rebooted- which wouldn't be the case if this was a crash, you may run the safe files (safe1 safe2 safe3 safe4) in lieu of quads. <br />
<br />
Open up 4 logins to the server (use the windows in [[Screen#Screen_Organization|a9]])<br />
In each of the 4 windows you will:<br />
<br />
If there is a [[#startalljails|startalljails]] script (and only quad1), run that command in each of the 4 windows. It will parse through the quad1 file and start each jail. Follow the instructions [[#Problems_with_the_quad.2Fsafe_files|here]] for monitoring startup. Note that you can be a little more lenient with jails that take awhile to start- startalljails will work around the slow jails and start the rest. As long as there aren't 4 jails which are "hung" during startup, the rest will get started eventually.<br />
-or-<br />
If there is no startalljails script, there will be multiple quad files. In each of the 4 windows, start each of the quads. i.e. start quad1 in window1, quad2 in window2 and so on. DO NOT start any quad twice. It will crash the server. If you accidentally do this, just jailkill all the jails which are in the quad and run the quad again. Follow the instructions here for monitoring quad startup.<br />
<br />
Note the time the last jail boots- this is what you will enter in the crash log.<br />
<br />
Save the crash log.<br />
<br />
=== Check to make sure all jails have started ===<br />
There's a simple script which will make sure all jails have started, and enter the ipfw counter rules: [[#postboot|postboot]] <br />
Run postboot, which will do a jailps on each jail it finds (excluding commented out jails) in the quad file(s). We're looking for 2 things:<br />
# systems spawning out of control or too many procs<br />
# jails which haven't started<br />
On 7.x and newer systems it will print out the problems (which jails haven't started) at the conclusion of postboot. <br />
On older systems you will need to watch closely to see if/when there's a problem, namely:<br />
<br />
[hostname] doesnt exist on this server<br />
<br />
When you get this message, it means one of 2 things:<br />
1. the jail really didn't start:<br />
When a jail doesn't start it usually boils down to a problem in the quad file. Perhaps the path name is wrong (data1 vs data2) or the name of the vn/mdfile is wrong. Once this is corrected, you will need to run the commands from the quad file manually, or you may use <tt>startjail <hostname></tt><br />
<br />
2. the customer has changed their hostname (and not told us) so their jail ''is'' running, just under a different hostname:<br />
On systems with jls, this is easy to rectify. First, get the customer info: <tt>g <hostname></tt><br />
Then look for the customer in jls: <tt>jls | grep <col0XXXX></tt><br />
From there you will see their new hostname- you should update that hostname in the quad file: don't forget to edit it on the <tt>## begin ##</tt> and <tt>## end ##</tt> lines, and in mgmt. <br />
On older systems without jls, this will be harder, you will need to look further to see their hostname- perhaps its in their /etc/rc.conf<br />
<br />
<br />
Once all jails are started, do some spot checks- try to ssh or browse to some customers, just to make sure things are really ok.<br />
<br />
== Adding disk to a 7.x/8.x jail ==<br />
or<br />
== Moving customer to a different drive (md) ==<br />
<br />
NOTE: this doesn’t apply to mx2 which uses gvinum. Use same procedure as 6.x<br />
NOTE: if you unmount before mdconfig, re-mdconfig (attach) then unmount then mdconfig -u again <br />
<br />
<br />
(parts to change/customize are <tt><span style="color:red">red</span></tt>)<br />
<br />
<br />
If someone wants more disk space, there’s a paste for it, send it to them (explains about downtime, etc).<br />
<br />
1. Figure out the space avail from <tt>js</tt>. Ideally, you want to put the customers new space on a different partition (and create the new md on the new partition). <br />
<br />
<br />
2. make a mental note of how much space they're currently using<br />
<br />
<br />
3. <tt>jailkill <hostname></tt> <br />
<br />
<br />
4. Umount it (including their devfs) but leave the md config’d (so if you use stopjail, you will have to re-mdconfig it)<br />
<br />
<br />
5. <tt>g <customerID></tt> to get the info (IP/cust#) needed to feed the new mdfile and mount name, and to see the current md device. <br />
<br />
<br />
6a. When there's enough room to place new system on an alternate, or the same drive:<br />
USE CAUTION not to overwrite (touch, mdconfig) existing md!!<br><br />
<tt>touch /mnt/data<span style="color:red">3/69.55.234.66-col01334</span><br><br />
mdconfig -a -t vnode -s 10g -f /mnt/data3/69.55.234.66-col01334 -u 97<br><br />
newfs /dev/md97</tt><br />
<br />
<br />
Optional- if new space is on a different drive, move the mount point directory AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
<tt>mount /dev/md<span style="color:red">97</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<br />
confirm you are mounted to /dev/md<span style="color:red">97</span> and space is correct:<br><br />
<tt>df .</tt> <br />
<br />
<br />
do the dump and pipe directly to restore:<br><br />
<tt>dump -0a -f - /dev/md<span style="color:red">1</span> | restore -r -f - <br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use <tt>df</tt> to confirm the restored data size matches the original usage figure<br />
<br />
<br />
md-unconfig old system:<br><br />
<tt>mdconfig -d -u <span style="color:red">1</span></tt><br />
<br />
<br />
archive old mdfile. <br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.237.26-col00241</span> /mnt/data<span style="color:red">1/old-col00241-mdfile-noarchive-20091211</span></tt><br />
<br />
<br />
edit quad (vq1) to point to new (/mnt/data<span style="color:red">3</span>) location AND new md number (md<span style="color:red">97</span>)<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
6b. When there's not enough room on an alternate partition, or on the same drive...but there is enough room if you were to remove the existing customer's space:<br />
<br />
<br />
mount backup nfs mounts:<br><br />
<tt>mbm</tt><br> <br />
(run <tt>df</tt> to confirm backup mounts are mounted)<br />
<br />
<br />
dump the customer to backup2 or backup1<br><br />
<tt>dump -0a -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> /dev/md<span style="color:red">1</span></tt><br><br />
(when complete WITHOUT errors, <tt>du</tt> the dump file to confirm it matches size, roughly, with usage)<br />
<br />
<br />
unconfigure and remove old mdfile<br><br />
<tt>mdconfig -d -u <span style="color:red">1</span><br><br />
rm /mnt/data<span style="color:red">1/69.55.237.26-col00241</span></tt><br><br />
(there should now be enough space to recreate your bigger system. If not, run sync a couple times)<br />
<br />
<br />
create the new system (ok to reuse old mdfile and md#):<br><br />
<tt>touch /mnt/data<span style="color:red">1/69.55.234.66-col01334</span><br><br />
mdconfig -a -t vnode -s <span style="color:red">10</span>g -f /mnt/data<span style="color:red">1/69.55.234.66-col01334</span> -u <span style="color:red">1</span><br><br />
newfs /dev/md<span style="color:red">1</span><br><br />
mount /dev/md<span style="color:red">1</span> /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> <br><br />
cd /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<br />
confirm you are mounted to /dev/md<span style="color:red">1</span> and space is correct:<br><br />
<tt>df .</tt> <br />
<br />
<br />
do the restore from the dumpfile on the backup server:<br><br />
<tt>restore -r -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> .<br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
umount nfs:<br><br />
<tt>mbu</tt><br />
<br />
<br />
If md# changed (or mount point), edit quad (<tt>vq1</tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new md number (md<span style="color:red">1</span>)<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
7. update disk (and dir if applicable) in mgmt screen<br />
<br />
<br />
8. update backup list AND move backups, if applicatble<br />
<br />
Ex: <tt>mvbackups <span style="color:red">69.55.237.26-col00241</span> jail<span style="color:red">9</span> data<span style="color:red">3</span></tt><br />
<br />
<br />
9. Optional: archive old mdfile<br />
<br />
<tt>mbm<br><br />
gzip -c old-col01588-mdfile-noarchive-20120329 > /deprecated/old-col01588-mdfile-noarchive-20120329.gz<br><br />
mbu<br><br />
rm old-col01588-mdfile-noarchive-20120329</tt><br />
<br />
== Adding disk to a 6.x jail (gvinum/gconcat) ==<br />
or<br />
== Moving customer to a different drive (gvinum/gconcat) ==<br />
<br />
(parts to change are <span style="color:red">highlighted</span>)<br />
<br />
<br />
If someone wants more disk space, there’s a paste for it, send it to them (explains about downtime, etc).<br />
<br />
<br />
1. Figure out the space avail from [[#js|js]]. Ideally, you want to put the customers new space on a different partition (and create the new md on the new partition). <br />
<br />
<br />
2. make a mental note of how much space they're currently using<br />
<br />
<br />
3. <tt>[[#stopjail|stopjail]] <hostname></tt> <br />
<br />
<br />
4. <tt>[[#g|g]] <customerID></tt> to get the info (IP/cust#) needed to feed the new mount name and existing volume/device. <br />
<br />
<br />
5a. When there's enough room to place new system on an alternate, or the same drive (using only UNUSED - including if it's in use by the system in question - gvinum volumes):<br />
<br />
<br />
Configure the new device:<br><br />
A. for a 2G system (single gvinum volume):<br><br />
<tt>bsdlabel -r -w /dev/gvinum/v<span style="color:red">123</span><br><br />
newfs /dev/gvinum/v<span style="color:red">123</span>a</tt><br><br />
<br />
-or- <br />
<br />
<br />
B. for a >2G system (create a gconcat volume):<br><br />
try to grab a contiguous block of gvinum volumes. gconcat volumes MAY NOT span drives (i.e. you cannot use a gvinum volume from data3 and a volume from data2 in the same gconcat volume). <br />
<br />
<tt>gconcat label <span style="color:red">v82-v84 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84</span><br><br />
bsdlabel -r -w /dev/concat/<span style="color:red">v82-v84</span><br><br />
newfs /dev/concat/<span style="color:red">v82-v84</span>a</tt><br />
<br />
<br />
Other valid gconcat examples:<br><br />
<tt>gconcat label v82-v84v109v112 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84 /dev/gvinum/v109 /dev/gvinum/v112<br><br />
gconcat label v82v83 /dev/gvinum/v82 /dev/gvinum/v83</tt> <br><br />
Note, long names will truncate: v144v145v148-v115 will truncate to v144v145v148-v1 (so you will refer to it as v144v145v148-v1 thereafter)<br />
<br />
<br />
Optional- if new volume is on a different drive, move the mount point directory (get the drive from js output) AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
<br />
confirm you are mounted to the device (<tt>/dev/gvinum/v<span style="color:red">123</span>a</tt> OR <tt>/dev/concat/<span style="color:red">v82-v84</span></tt>) and space is correct:<br><br />
A. <tt>mount /dev/gvinum/v<span style="color:red">123</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br><br />
-or-<br><br />
B. <tt>mount /dev/concat/<span style="color:red">v82-v84</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<tt>cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
df .</tt> <br />
<br />
<br />
do the dump and pipe directly to restore:<br><br />
<tt>dump -0a -f - /dev/gvinum/v<span style="color:red">1</span> | restore -r -f - <br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
edit quad (<tt>vq<span style="color:red">1</span></tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new volume (<tt>/dev/gvinum/v<span style="color:red">123</span>a</tt> or <tt>/dev/concat/<span style="color:red">v82-v84</span>a</tt>) , run <tt>buildsafe</tt><br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
5b. When there's not enough room on an alternate partition, or on the same drive...but there is enough room if you were to remove the existing customer's space (i.e. if you want/need to reuse the existing gvinum volumes and add on more):<br />
<br />
<br />
mount backup nfs mounts:<br><br />
<tt>mbm</tt> <br><br />
(run df to confirm backup mounts are mounted)<br />
<br />
<br />
dump the customer to backup2 or backup1<br><br />
<tt>dump -0a -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> /dev/<span style="color:red">gconcat/v106-v107</span></tt><br><br />
(when complete WITHOUT errors, du the dump file to confirm it matches size, roughly, with usage)<br />
<br />
<br />
unconfigure the old gconcat volume<br><br />
list member gvinum volumes:<br />
<br />
<tt>gconcat list <span style="color:red">v106v107</span></tt><br />
<br />
<br />
Output will resemble:<br><br />
<pre>Geom name: v106v107<br />
State: UP<br />
Status: Total=2, Online=2<br />
Type: AUTOMATIC<br />
ID: 3530663882<br />
Providers:<br />
1. Name: concat/v106v107<br />
Mediasize: 4294966272 (4.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e2<br />
Consumers:<br />
1. Name: gvinum/sd/v106.p0.s0<br />
Mediasize: 2147483648 (2.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e3<br />
Start: 0<br />
End: 2147483136<br />
2. Name: gvinum/sd/v107.p0.s0<br />
Mediasize: 2147483648 (2.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e3<br />
Start: 2147483136<br />
End: 4294966272</pre><br />
<br />
stop volume and clear members<br><br />
<tt>gconcat stop <span style="color:red">v106v107</span><br><br />
gconcat clear <span style="color:red">gvinum/sd/v106.p0.s0 gvinum/sd/v107.p0.s0</span></tt><br />
<br />
<br />
create new device- and its ok to reuse old/former members<br><br />
try to grab a contiguous block of gvinum volumes. gconcat volumes MAY NOT span drives (i.e. you cannot use a gvinum volume from data3 and a volume from data2 in the same gconcat volume). <br><br />
<tt>gconcat label <span style="color:red">v82-v84v106v107 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84 /dev/gvinum/v106 /dev/gvinum/v107</span> <br><br />
bsdlabel -r -w /dev/concat/<span style="color:red">v82-v84v106v107</span><br><br />
newfs /dev/concat/<span style="color:red">v82-v84v106v107</span>a</tt><br />
<br />
<br />
Optional- if new volume is on a different drive, move the mount point directory (get the drive from js output) AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
confirm you are mounted to the device (/dev/concat/<span style="color:red">v82-v84v106v107</span>) and space is correct:<br><br />
<tt>mount /dev/concat/<span style="color:red">v82-v84v106v107</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<tt>cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
df .</tt> <br />
<br />
do the restore from the dumpfile on the backup server:<br><br />
<tt>restore -r -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> .<br><br />
rm restoresymtable</tt><br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
edit quad (<tt>vq<span style="color:red">1</span></tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new volume (<tt>/dev/concat/<span style="color:red">v82-v84v106v107</span>a</tt>), run buildsafe<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
TODO: clean up/clear old gvin/gconcat vol<br />
<br />
<br />
6. update disk (and dir if applicable) in mgmt screen<br />
<br />
<br />
7. update backup list AND move backups, if applicatble<br />
<br />
<br />
Ex: <tt>mvbackups <span style="color:red">69.55.237.26-col00241</span> jail<span style="color:red">9</span> data<span style="color:red">3</span></tt><br />
<br />
<br />
<br />
DEPRECATED - steps to tack on a new gvin to existing gconcat- leads to corrupted fs<br />
bsdlabel -e /dev/concat/v82-v84<br />
<br />
To figure out new size of the c partition, multiply 4194304 by the # of 2G gvinum volumes and subtract the # of 2G volumes:<br />
10G: 4194304 * 5 – 5 = 20971515<br />
8G: 4194304 * 4 – 4 = 16777212<br />
6G: 4194304 * 3 – 3 = 12582909<br />
4G: 4194304 * 2 – 2 = 8388606<br />
<br />
To figure out the new size of the a partition, subtract 16 from the c partition:<br />
10G: 20971515 – 16 = 20971499<br />
8G: 16777212 – 16 = 16777196<br />
6G: 12582909 – 16 = 12582893<br />
4G: 8388606 – 16 = 8388590<br />
<br />
Orig:<br />
8 partitions:<br />
# size offset fstype [fsize bsize bps/cpg]<br />
a: 8388590 16 4.2BSD 2048 16384 28552<br />
c: 8388606 0 unused 0 0 # "raw" part, don't edit<br />
<br />
New:<br />
8 partitions:<br />
# size offset fstype [fsize bsize bps/cpg]<br />
a: 12582893 16 4.2BSD 2048 16384 28552<br />
c: 12582909 0 unused 0 0 # "raw" part, don't edit<br />
<br />
sync; sync<br />
<br />
growfs /dev/concat/v82-v84a<br />
<br />
fsck –fy /dev/concat/v82-v84a<br />
<br />
sync<br />
<br />
fsck –fy /dev/concat/v82-v84a<br />
<br />
(keep running fsck’s till NO errors)<br />
<br />
== Problems un-mounting - and with mount_null’s ==<br />
<br />
If you cannot unmount a filesystem, beacuse it says the filesystem is busy, it is because of three things:<br />
<br />
a) the jail is still running<br />
<br />
b) you are actually in that directory, even though the jail is stopped<br />
<br />
c) there are still dev, null_mount or linprocfs mount points mounted inside that directory.<br />
<br />
d) when trying to umount null_mounts that are really long and you get an error like “No such file or directory”, it’s an OS bug where the dir is truncated. No known fix<br />
<br />
e) there are still files open somewhere inside the dir. Use <tt>fstat | grep <cid></tt> to find the process that has files open<br />
<br />
f) Starting with 6.x, the jail mechanism does a poor job of keeping track of processes running in a jail and if it thinks there are still procs running, it will refuse to umount the disk. If this is happening you should see a low number in the #REF column when you run jls. In this case you ''can'' safely <tt>umount –f</tt> the mount. <br />
<br />
Please note -if you forcibly unmount a (4.x) filesystem that has null_mounts<br />
still mounted in it, the system '''will crash''' within 10-15 mins.<br />
<br />
== Misc jail Items ==<br />
<br />
We are overselling hard drive space on jail2, jail8, jail9, a couple jails on jail17, jail4, jail12 and jail18.<br />
Even though the vn file shows 4G size, it doesn’t actually occupy that amount of space on the disk. So be careful not to fill up drives where we’re overselling – use oversellcheck to confirm you’re not oversold by more than 10G.<br />
There are other truncated jails, they are generally noted in a the file on the root system: /root/truncated<br />
<br />
The act of moving a truncated vn to another system un-does the truncating- the truncated vn is filled with 0’s and it occupies physical disk space for which it’s configured. So, you should use dumpremote to preserve the truncation.<br />
<br />
= FreeBSD VPS Management Tools =<br />
<br />
These files are located in /usr/local/jail/rc.d and /usr/local/jail/bin<br />
<br />
== jailmake ==<br />
<br />
Applies to 7.x+ <br />
On older systems syntax differs, run jailmake once to see.<br />
<br />
Note: this procedure differs on mx2 which is 7.x but still uses gvinum<br />
<br />
# run js to figure out which md’s are in use, which disk has enough space, IP to put it on<br />
# use col00xxx for both hostnames if they don’t give you a hostname<br />
# copy over dir, ip and password to pending customer screen<br />
<br />
<tt>Usage: jailmake IP[,IP] CID disk[1|2|3] md# hostname shorthost ipfw# email [size in GB]</tt><br />
<br />
Ex: <br />
<br />
Jail2# jailmake 69.55.234.66 col01334 3 97 vps.bsd.it vps 1334 fb@bsd.it<br />
<br />
== jailps ==<br />
jailps [hostname]<br />
DEPRECATED FOR jps: displays processes belonging to/running inside a jail. The command<br />
takes one (optional) argument – the hostname of the jail you wish to query. If you don’t <br />
supply an argument, all processes on the machine are listed and grouped by jail. <br />
<br />
== jps ==<br />
jps [hostname]<br />
displays processes belonging to/running inside a jail. The command<br />
takes one (optional) argument – the hostname or ID of the jail you wish to query. <br />
<br />
== jailkill ==<br />
jailkill <hostname><br />
stops all process running in a jail.<br />
<br />
You can also run:<br />
jailkill <JID><br />
<br />
=== problems ===<br />
Occasionally you will hit an issue where jail will not kill off:<br />
<br />
<pre>jail9# jailkill www.domain.com<br />
www.domain.com .. killed: none<br />
jail9#</pre><br />
Because no processes are running under that hostname. You cannot use jailps.pl either:<br />
<br />
<pre>jail9# jailps www.domain.com<br />
www.domain.com doesn’t exist on this server<br />
jail9#</pre><br />
<br />
The reasons for this are usually:<br />
* the jail is no longer running<br />
<br />
* the jail's hostname has changed<br />
In this case, <br />
<br />
>=6.x: run a <tt>jls|grep <jail's IP></tt> to find the correct hostname, then update the quad file, then kill the jail.<br />
<br />
<6.x: the first step is to cat their /etc/rc.conf file to see if you can tell what they set the new hostname to. This very often works. For example:<br />
<br />
cat /mnt/data2/198.78.65.136-col00261-DIR/etc/rc.conf<br />
<br />
But maybe they set the hostname with the hostname command, and the original hostname is still in /etc/rc.conf.<br />
<br />
The welcome email clearly states that they should tell us if they change their hostname, so there is no problem in just emailing them and asking them what they set the new hostname to.<br />
<br />
Once you know the new hostname OR if a customer simply emails to inform you that they have set the hostname to something different, you need to edit the quad and safe files that their system is in to input the new hostname.<br />
<br />
However, if push comes to shove and you cannot find out the hostname from them or from their system, then you need to start doing some detective work.<br />
<br />
The easiest thing to do is run jailps looking for a hostname similar to their original hostname. Or you could get into the /bin/sh shell by running:<br />
<br />
/bin/sh<br />
<br />
and then looking at every hostname of every process:<br />
<br />
for f in `ls /proc` ; do cat /proc/$f/status ; done<br />
<br />
and scanning for a hostname that is either similar to their original hostname, or that you don't see in any of the quad safe files.<br />
<br />
This is very brute force though, and it is possible that catting every file in /proc is dangerous - I don't recommend it. A better thing would be to identify any processes that you know belong to this system – perhaps the reason you are trying to find this system is because they are running something bad - and just catting the status from only that PID.<br />
<br />
Somewhere there’s a jail where there may be 2 systems named www. Look at /etc/rc.conf and make sure they’re both really www. If they are, jailkill www, jailps www to make sure not running. Then immediately restart the other one, as the fqdn (as found from a rev nslookup)<br />
<br />
* on >=6.x the hostname may not yet be hashed:<br />
<pre>jail9 /# jls<br />
JID Hostname Path IP Address(es)<br />
1 bitnet.dgate.org /mnt/data1/69.55.232.50-col02094-DIR 69.55.232.50<br />
2 ns3.hctc.net /mnt/data1/69.55.234.52-col01925-DIR 69.55.234.52<br />
3 bsd1 /mnt/data1/69.55.232.44-col00155-DIR 69.55.232.44<br />
4 let2.bbag.org /mnt/data1/69.55.230.92-col00202-DIR 69.55.230.92<br />
5 post.org /mnt/data2/69.55.232.51-col02095-DIR 69.55.232.51 ...<br />
6 ns2 /mnt/data1/69.55.232.47-col01506-DIR 69.55.232.47 ...<br />
7 arlen.server.net /mnt/data1/69.55.232.52-col01171-DIR 69.55.232.52<br />
8 deskfood.com /mnt/data1/69.55.232.71-col00419-DIR 69.55.232.71<br />
9 mirage.confluentforms.com /mnt/data1/69.55.232.54-col02105-DIR 69.55.232.54 ...<br />
10 beachmember.com /mnt/data1/69.55.232.59-col02107-DIR 69.55.232.59<br />
11 www.agottem.com /mnt/data1/69.55.232.60-col02109-DIR 69.55.232.60<br />
12 sdhobbit.myglance.org /mnt/data1/69.55.236.82-col01708-DIR 69.55.236.82<br />
13 ns1.jnielsen.net /mnt/data1/69.55.234.48-col00204-DIR 69.55.234.48 ...<br />
14 ymt.rollingegg.net /mnt/data2/69.55.236.71-col01678-DIR 69.55.236.71<br />
15 verse.unixlore.net /mnt/data1/69.55.232.58-col02131-DIR 69.55.232.58<br />
16 smcc-mail.org /mnt/data2/69.55.232.68-col02144-DIR 69.55.232.68<br />
17 kasoutsuki.w4jdh.net /mnt/data2/69.55.232.46-col02147-DIR 69.55.232.46<br />
18 dili.thium.net /mnt/data2/69.55.232.80-col01901-DIR 69.55.232.80<br />
20 www.tekmarsis.com /mnt/data2/69.55.232.66-col02155-DIR 69.55.232.66<br />
21 vps.yoxel.net /mnt/data2/69.55.236.67-col01673-DIR 69.55.236.67<br />
22 smitty.twitalertz.com /mnt/data2/69.55.232.84-col02153-DIR 69.55.232.84<br />
23 deliver4.klatha.com /mnt/data2/69.55.232.67-col02160-DIR 69.55.232.67<br />
24 nideffer.com /mnt/data2/69.55.232.65-col00412-DIR 69.55.232.65<br />
25 usa.hanyuan.com /mnt/data2/69.55.232.57-col02163-DIR 69.55.232.57<br />
26 daifuku.ppbh.com /mnt/data2/69.55.236.91-col01720-DIR 69.55.236.91<br />
27 collins.greencape.net /mnt/data2/69.55.232.83-col01294-DIR 69.55.232.83<br />
28 ragebox.com /mnt/data2/69.55.230.104-col01278-DIR 69.55.230.104<br />
29 outside.mt.net /mnt/data2/69.55.232.72-col02166-DIR 69.55.232.72<br />
30 vps.payneful.ca /mnt/data2/69.55.234.98-col01999-DIR 69.55.234.98<br />
31 higgins /mnt/data2/69.55.232.87-col02165-DIR 69.55.232.87 ...<br />
32 ozymandius /mnt/data2/69.55.228.96-col01233-DIR 69.55.228.96<br />
33 trusted.realtors.org /mnt/data2/69.55.238.72-col02170-DIR 69.55.238.72<br />
34 jc1.flanderous.com /mnt/data2/69.55.239.22-col01504-DIR 69.55.239.22<br />
36 guppylog.com /mnt/data2/69.55.238.73-col00036-DIR 69.55.238.73<br />
40 haliohost.com /mnt/data2/69.55.234.41-col01916-DIR 69.55.234.41 ...<br />
41 satyr.jorge.cc /mnt/data1/69.55.232.70-col01963-DIR 69.55.232.70<br />
jail9 /# jailkill satyr.jorge.cc<br />
ERROR: jail_: jail "satyr,jorge,cc" not found<br />
</pre><br />
<br />
Note how it's saying <tt>satyr,jorge,cc</tt> is not found, and not <tt>satyr.jorge.cc</tt>. <br />
<br />
The jail subsystem tracks things using comma-delimited hostnames. That is created every few hours:<br />
<br />
jail9 /# crontab -l<br />
0 0,6,12,18 * * * /usr/local/jail/bin/sync_jail_names<br />
<br />
So if we run this manually:<br />
jail9 /# /usr/local/jail/bin/sync_jail_names<br />
<br />
Then kill the jail:<br />
jail9 /# jailkill satyr.jorge.cc<br />
successfully killed: satyr,jorge,cc</pre><br />
<br />
It worked.<br />
<br />
<br />
If you ever see this when trying to kill a jail:<br />
<br />
<pre># jailkill e-scribe.com<br />
killing JID: 6 hostname: e-scribe.com<br />
3 procs running<br />
3 procs running<br />
3 procs running<br />
3 procs running<br />
...</pre><br />
<br />
<tt>[[#jailkill|jailkill]]</tt> probably got lost trying to kill off the jail. Just ctrl-c the jailkill process, then run a jailps on the hostname, and <tt>kill -9</tt> any process which is still running. Keep running jailps and <tt>kill -9</tt> till all processes are gone.<br />
<br />
== jailpsall ==<br />
jailpsall<br />
will run a jailps on all jails configured in the quad files (this is different from<br />
jailps with no arguments as it won’t help you find a “hidden” system)<br />
<br />
== jailpsw ==<br />
jailpsw<br />
will run a jailps with an extra -w to provide wider output<br />
<br />
== jt (>=7.x) ==<br />
jt<br />
displays the top 20 processes on the server (the top 20 processes from top) and <br />
which jail owns them. This is very helpful for determining who is doing what when<br />
the server is very busy.<br />
<br />
== jtop (>=7.x) ==<br />
jtop<br />
a wrapper for top displaying processes on the server and which jail owns them. Constantly updates, like top. <br />
<br />
== jtop (<7.x) ==<br />
jtop<br />
displays the top 20 processes on the server (the top 20 processes from top) and <br />
which jail owns them. This is very helpful for determining who is doing what when<br />
the server is very busy.<br />
<br />
== stopjail ==<br />
stopjail <hostname> [1]<br />
this will jailkill, umount and vnconfig –u a jail. If passed an optional 2nd<br />
argument, it will not exit before umounting and un-vnconfig’ing in the event<br />
jailkill returns no processes killed. This is useful if you just want to umount<br />
and vnconfig –u a jail you’ve already killed. It is intelligent in that it won’t <br />
try to umount or vnconfig –u if it’s not necessary.<br />
<br />
== startjail ==<br />
startjail <hostname><br />
this will start vnconfig, mount (including linprocfs and null-mounts), and start a jail.<br />
Essentially, it reads the jail’s relevant block from the right quad file and executes it.<br />
It is intelligent in that it won’t try to mount or vnconfig if it’s not necessary.<br />
<br />
== jpid ==<br />
jpid <pid><br />
displays information about a process – including which jail owns it.<br />
It’s the equivalent of running cat /proc/<pid>/status<br />
<br />
== canceljail ==<br />
canceljail <hostname> [1]<br />
this will stop a jail (the equivalent of stopjail), check for backups (offer to remove them <br />
from the backup server and the backup.config), rename the vnfile, remove the dir, and <br />
edit quad/safe. If passed an optional 2nd argument, it will not exit upon failing to kill<br />
and processes owned by the jail. This is useful if you just want to cancel a jail which <br />
is already stopped.<br />
<br />
== jls ==<br />
jls [-v]<br />
Lists all jails running:<br />
<pre>JID #REF IP Address Hostname Path<br />
101 135 69.55.224.148 mail.pc9.org /mnt/data2/69.55.224.148-col01034-DIR</pre><br />
<br />
#REF is the number of references or procs(?) running<br />
<br />
Running with -v will give you all IPs assigned to each jail (7.2 up)<br />
<pre>JID #REF Hostname Path IP Address(es)<br />
101 139 mail.pc9.org /mnt/data2/69.55.224.148-col01034-DIR 69.55.224.14869.55.234.85</pre><br />
<br />
== startalljails ==<br />
startalljails<br />
7.2+ only. This will parse through quad1 and start all jails. It utilizes lockfiles so it won’t try to start a jail more than once- therefore multiple instances can be running in parallel without fear of starting a jail twice. If a jail startup gets stuck, you can ^C without fear of killing the script. IMPORTANT- before running startalljails you should make sure you ran preboot once as it will clear out all the lockfiles and enable startalljails to work properly.<br />
<br />
== aaccheck.sh ==<br />
aaccheck.sh<br />
displayes the output of container list and task list from aaccli<br />
<br />
== backup ==<br />
backup<br />
backup script called nightly to update jail scripts and do customer backups<br />
<br />
== buildsafe ==<br />
buildsafe<br />
creates safe files based on quads (automatically removing the fsck’s). This will destructively overwrite safe files<br />
<br />
== checkload.pl ==<br />
checkload.pl<br />
this was intended to be setup as a cronjob to watch processes on a jail when the load <br />
rises above a certain level. Not currently in use.<br />
<br />
== checkprio.pl ==<br />
checkprio.pl<br />
will look for any process (other than the current shell’s csh, sh, sshd procs) with a non-normal priority and normalize it<br />
<br />
== diskusagemon == <br />
diskusagemon <mount point> <1k blocks><br />
watches a mount point’s disk use, when it reaches the level specified in the 2nd argument,<br />
it exits. This is useful when doing a restore and you want to be paged as it’s nearing completion.<br />
Best used as: <tt>diskusagemon /asd/asd 1234; pagexxx</tt><br />
<br />
== dumprestore ==<br />
dumprestore <dumpfile><br />
this is a perl expect script which automatically enters ‘1’ and ‘y’. It seems to cause restore to fail<br />
to set owner permissions on large restores.<br />
<br />
== g ==<br />
g <search><br />
greps the quad/safe files for the given search parameter<br />
<br />
== gather.pl ==<br />
gather.pl<br />
gathers up data about jails configured and writes to a file. Used for audits against the db<br />
<br />
== gb ==<br />
gb <search><br />
greps backup.config for the given search parameter<br />
<br />
== gbg ==<br />
gbg <search><br />
greps backup.config for the given search parameter and presents just the directories (for clean pasting)<br />
<br />
== ipfwbackup ==<br />
ipfwbackup<br />
writes ipfw traffic count data to a logfile<br />
<br />
== ipfwreset ==<br />
ipfwreset<br />
writes ipfw traffic count data to a logfile and resets counters to 0<br />
<br />
== js ==<br />
js<br />
output varies by OS version, but generally provides information about the base jail:<br />
- which vn’s are in use<br />
- disk usage<br />
- info about the contents of quads<br />
- the # of inodes represented by the jails contained in the group (133.2 in the example below), and how many jails per data mount, as well as subtotals<br />
- ips bound to the base machine but not in use by a jail<br />
- free gvinum volumes, or unused vn’s or used md’s<br />
<br />
<pre>/usr/local/jail/rc.d/quad1:<br />
/mnt/data1 133.2 (1)<br />
/mnt/data2 1040.5 (7)<br />
total 1173.7 (8)<br />
/usr/local/jail/rc.d/quad2:<br />
/mnt/data1 983.4 (6)<br />
total 983.4 (6)<br />
/usr/local/jail/rc.d/quad3:<br />
/mnt/data1 693.4 (4)<br />
/mnt/data2 371.6 (3)<br />
total 1065 (7)<br />
/usr/local/jail/rc.d/quad4:<br />
/mnt/data1 466.6 (3)<br />
/mnt/data2 882.2 (5)<br />
total 1348.8 (8)<br />
/mnt/data1: 2276.6 (14)<br />
/mnt/data2: 2294.3 (15)<br />
<br />
Available IPs:<br />
69.55.230.11 69.55.230.13 69.55.228.200<br />
<br />
Available volumes:<br />
v78 /mnt/data2 2G<br />
v79 /mnt/data2 2G<br />
v80 /mnt/data2 2G</pre> <br />
<br />
== load.pl ==<br />
load.pl<br />
feeds info to load mrtg - executed by inetd.<br />
<br />
== makevirginjail ==<br />
makevirginjail<br />
Only on some systems, makes an empty jail (doesn't do restore step)<br />
<br />
== mb == <br />
mb <mount|umount><br />
(nfs) mounts and umounts dirs to backup2. Shortcuts are mbm and mbu to mount and unmount. <br />
<br />
== notify.sh ==<br />
notify.sh<br />
emails reboot@johncompanies.com – intended to be called at boot time to alert us to a machine which panics and reboots and isn’t caught by bb or castle.<br />
<br />
== orphanedbackupwatch ==<br />
orphanedbackupwatch<br />
looks for directories on backup2 which aren’t configured in backup.config and offers to delete them<br />
<br />
== postboot ==<br />
postboot<br />
to be run after a machine reboot and quad/safe’s are done executing. It will:<br />
* do chmod 666 on each jail’s /dev/null<br />
* add ipfw counts<br />
* run jailpsall (so you can see if a configured jail isn’t running)<br />
<br />
== preboot ==<br />
preboot<br />
to be run before running quad/safe – checks for misconfigurations: <br />
* a jail configured in a quad but not a safe<br />
* a jail is listed more than once in a quad<br />
* the ip assigned to a jail isn’t configured on the machine<br />
* alias numbering skips in the rc.conf (resulting in the above)<br />
* orphaned vnfile's that aren't mentioned in a quad/safe<br />
* ip mismatches between dir/vnfile name and the jail’s ip<br />
* dir/vnfiles's in quad/safe that don’t exist <br />
<br />
== quadanalyze.pl ==<br />
quadanalyze.pl<br />
called by js, produces the info (seen above with js explanation) about the contents of quad (inode count, # of jails, etc.)<br />
<br />
== rsync.backup ==<br />
rsync.backup<br />
does customer backups (relies on backup.config)<br />
<br />
== taskdone ==<br />
taskdone<br />
when called will email support@johncompanies.com with the hostname of the machine from which it was executed as the subject<br />
<br />
== topten ==<br />
topten<br />
summarizes the top 10 traffic users (called by ipfwreset)<br />
<br />
== trafficgather.pl ==<br />
trafficgather.pl [yy-mm]<br />
sends a traffic usage summary by jail to support@johncomapnies.com and payments@johncompanies.com. Optional arguments are year and month (must be in the past). If not passed, assumes last month. Relies on traffic logs created by ipfwreset and ipfwbackup<br />
<br />
== trafficwatch.pl ==<br />
trafficwatch.pl<br />
checks traffic usage and emails support@johncomapnies.com when a jail reaches the warning level (35G) and the limit (40G). We really aren’t using this anymore now that we have netflow.<br />
<br />
== trafstats ==<br />
trafstats<br />
writes ipfw traffic usage info by jail to a file called jc_traffic_dump in each jail’s / dir<br />
<br />
== truncate_jailmake ==<br />
truncate_jailmake<br />
a version of jailmake which creates truncated vnfiles.<br />
<br />
== vb ==<br />
vb<br />
the equivalent of: <tt>vi /usr/local/jail/bin/backup.config</tt><br />
<br />
== vs (freebsd) ==<br />
vs<n><br />
the equivalent of: <tt>vi /usr/local/jail/rc.d/safe<n></tt><br />
<br />
vq<n><br />
the equivalent of: vi /usr/local/jail/rc.d/quad<n><br />
<br />
== dumpremote ==<br />
dumpremote <user@machine> </remote/location/file-dump> <vnX><br />
ex: dumpremote user@10.1.4.117 /mnt/data3/remote.echoditto.com-dump 7<br />
this will dump a vn filesystem to a remote machine and location<br />
<br />
== oversellcheck ==<br />
oversellcheck<br />
displays how much a disk is oversold or undersold taking into account truncated vn files. Only for use on 4.x systems<br />
<br />
== mvbackups (freebsd) ==<br />
mvbackups <dir> (1.1.1.1-col00001-DIR) <target_machine> (jail1) <target_dir> (data1)<br />
moves backups from one location to another on the backup server, and provides you with option to remove entries from current backup.config, and simple paste command to add the config to backup.config on the target server<br />
<br />
== jailnice ==<br />
jailnice <hostname><br />
applies <tt>renice 19 [PID]</tt> and <tt>rtprio 31 –[PID]</tt> to each process in the given jail<br />
<br />
== dumpremoterestore ==<br />
dumpremoterestore <device> <ip of target machine> <dir on target machine><br />
ex: dumpremoterestore /dev/vn51 10.1.4.118 /mnt/data2/69.55.239.45-col00688-DIR<br />
dumps a device and restores it to a directory on a remote machine. Requires that you enable root ssh on the <br />
remote machine.<br />
<br />
== psj ==<br />
psj<br />
shows just the procs running on the base system – a ps auxw but without jail’d procs present<br />
<br />
== perc5iraidchk ==<br />
perc5iraidchk<br />
checks for degraded arrays on Dell 2950 systems with Perc5/6 controllers<br />
<br />
== perc4eraidchk ==<br />
perc4eraidchk<br />
checks for degraded arrays on Dell 2850 systems with Perc4e/Di controllers<br />
<br />
= Virtuozzo VPS =<br />
<br />
Note: We use VEID (Virtual Environment ID) and CTID (Container ID) interchangably. Similarly, VE and CT. They mean the same thing.<br />
VZPP = VirtuoZzo Power Panel (the control panel for each CT)<br />
<br />
All linux systems exist in /vz, /vz1 or /vz2 - since each linux machine holds roughly 60-90 customers, there will be roughly 30-45 in each partition.<br />
<br />
The actual filesystem of the system in question is in:<br />
<br />
/vz(1-2)/private/(VEID)<br />
<br />
Where VEID is the identifier for that system - an all-numeric string larger than 100.<br />
<br />
The actual mounted and running systems are in the corresponding:<br />
<br />
/vz(1-2)/root/(VEID)<br />
<br />
But we rarely interact with any system from this mount point.<br />
<br />
You should never need to touch the root portion of their system – however you can traverse their filesystem by going to <tt>/vz(1-2)/private/(VEID)/root</tt> (<tt>/vz(1-2)/private/(VEID)/fs/root</tt> on 4.x systems) the root of their filesystem is in that directory, and their entire system is underneath that.<br />
<br />
Every VE has a startup script in <tt>/etc/sysconfig/vz-scripts</tt> (which is symlinked as <tt>/vzconf</tt> on all systems) - the VE startup script is simply named <tt>(VEID).conf</tt> - it contains all the system parameters for that VE:<br />
<br />
<pre># Configuration file generated by vzsplit for 60 VE<br />
# on HN with total amount of physical mem 2011 Mb<br />
<br />
VERSION="2"<br />
CLASSID="2"<br />
<br />
ONBOOT="yes"<br />
<br />
KMEMSIZE="8100000:8200000"<br />
LOCKEDPAGES="322:322"<br />
PRIVVMPAGES="610000:615000"<br />
SHMPAGES="33000:34500"<br />
NUMPROC="410:415"<br />
PHYSPAGES="0:2147483647"<br />
VMGUARPAGES="13019:2147483647"<br />
OOMGUARPAGES="13019:2147483647"<br />
NUMTCPSOCK="1210:1215"<br />
NUMFLOCK="107:117"<br />
NUMPTY="19:19"<br />
NUMSIGINFO="274:274"<br />
TCPSNDBUF="1800000:1900000"<br />
TCPRCVBUF="1800000:1900000"<br />
OTHERSOCKBUF="900000:950000"<br />
DGRAMRCVBUF="200000:200000"<br />
NUMOTHERSOCK="650:660"<br />
DCACHE="786432:818029"<br />
NUMFILE="7500:7600"<br />
AVNUMPROC="51:51"<br />
IPTENTRIES="155:155"<br />
DISKSPACE="4194304:4613734"<br />
DISKINODES="400000:420000"<br />
CPUUNITS="1412"<br />
QUOTAUGIDLIMIT="2000"<br />
VE_ROOT="/vz1/root/636"<br />
VE_PRIVATE="/vz1/private/636"<br />
NAMESERVER="69.55.225.225 69.55.230.3"<br />
OSTEMPLATE="vzredhat-7.3/20030305"<br />
VE_TYPE="regular"<br />
IP_ADDRESS="69.55.225.229"<br />
HOSTNAME="textengine.net"</pre><br />
<br />
<br />
As you can see, the hostname is set here, the disk space is set here, the number of inodes, the number of files that can be open, the number of tcp sockets, etc. - all are set here.<br />
<br />
In fact, everything that can be set on this customer system is set in this conf file.<br />
<br />
<br />
All interaction with the customer system is done with the VEID. You start the system by running:<br />
<br />
vzctl start 999<br />
<br />
You stop it by running:<br />
<br />
vzctl stop 999<br />
<br />
You execute commands in it by running:<br />
<br />
vzctl exec 999 df -k<br />
<br />
You enter into it, via a root-shell backdoor with:<br />
<br />
vzctl enter 999<br />
<br />
and you set parameters for the system, while it is still running, with:<br />
<br />
vzctl set 999 --diskspace 6100000:6200000 --save<br />
<br />
<tt>vzctl</tt> is the most commonly used command - we have aliased <tt>v</tt> to <tt>vzctl</tt> since we use it so often. We’ll continue to use <tt>vzctl</tt> in our examples, but feel free to use just <tt>v</tt>.<br />
<br />
Let's say the user wants more diskspace. You can cat their conf file and see:<br />
<br />
DISKSPACE="4194304:4613734"<br />
<br />
So right now they have 4gigs of space. You can then change it to 6 with:<br />
<br />
vzctl set 999 --diskspace 6100000:6200000 --save<br />
<br />
IMPORTANT: all issuances of the vzctl set command need to end with <tt>–save</tt> - if they don't, the setting will be set, but it will not be saved to the conf file, and they will not have those settings next time they boot.<br />
<br />
All of the tunables in the conf file can be set with the vzctl set command. Note that in the conf file, and on the vzctl set command line, we always issue two numbers seperated by a colon - that is because we are setting the hard and soft limits. Always set the hard limit slightly above the soft limit, as you see it is in the conf file for all those settings.<br />
<br />
There are also things you can set with `<tt>vzctl set</tt>` that are not in the conf file as settings, per se. For instance, you can add IPs:<br />
<br />
vzctl set 999 --ipadd 10.10.10.10 --save<br />
<br />
or multiple IPs:<br />
<br />
vzctl set 999 --ipadd 10.10.10.10 --ipadd 10.10.20.30 --save<br />
<br />
or change the hostname:<br />
<br />
vzctl set 999 --hostname www.example.com --save<br />
<br />
You can even set the nameservers:<br />
<br />
vzctl set 999 --nameserver 198.78.66.4 --nameserver 198.78.70.180 --save<br />
<br />
Although you probably will never do that.<br />
<br />
You can disable a VPS from being started (by VZPP or reboot) (4.x):<br />
<br />
vzctl set 999 --disabled yes --save <br />
<br />
You can disable a VPS from being started (by VZPP or reboot) (<=3.x):<br />
<br />
vzctl set 999 --onboot=no --save <br />
<br />
You can disable a VPS from using his control panel:<br />
<br />
vzctl set 999 --offline_management=no --save <br />
<br />
You can suspend a VPS, so it can be resumed in the same state it was in when it was stopped (4.x):<br />
<br />
vzctl suspend 999<br />
<br />
and to resume it:<br />
<br />
vzctl resume 999<br />
<br />
to see who owns process:<br />
vzpid <PID><br />
<br />
to mount up an unmounted ve:<br />
vzctl mount 827<br />
<br />
<br />
To see network stats for CT's:<br />
<pre># vznetstat<br />
VEID Net.Class Output(bytes) Input(bytes)<br />
-----------------------------------------------<br />
24218 1 484M 39M<br />
24245 1 463M 143M<br />
2451 1 2224M 265M<br />
2454 1 2616M 385M<br />
4149 1 125M 68M<br />
418 1 1560M 34M<br />
472 1 1219M 315M<br />
726 1 628M 317M<br />
763 1 223M 82M<br />
771 1 4234M 437M<br />
-----------------------------------------------<br />
Total: 13780M 2090M</pre><br />
<br />
<br />
One thing that sometimes comes up on older systems that we created with smaller defaults is that the system would run out of inodes. The user will email and say they cannot create any more files or grow any files larger, but they will also say that they are not out of diskspace ... they are running:<br />
<br />
df -k<br />
<br />
and seeing how much space is free - and they are not out of space. They are most likely out of inodes - which they would see by running:<br />
<br />
df -i<br />
<br />
So, the first thing you should do is enter their system with:<br />
<br />
vzctl enter 999<br />
<br />
and run: <tt>df -i</tt><br />
<br />
to confirm your theory. Then exit their system. Then simply cat their conf file and see what their inodes are set to (probably 200000:200000, since that was the old default on the older systems) and run:<br />
<br />
vzctl set 999 --diskinodes 400000:400000 --save<br />
<br />
If they are not out of inodes, then a good possibility is that they have maxed out their numfile configuration variable, which controls how many files they can have in their system. The current default is 7500 (which nobody has ever hit), but the old default was as low as 2000, so you would run something like:<br />
<br />
vzctl set 999 --numfile 7500:7500 --save<br />
<br />
----<br />
<br />
You cannot start or stop a VE if your pwd is its private (/vz/private/999) or root (/vz/root/999) directories, or anywhere below them.<br />
<br />
<br />
== Recovering from a crash (linux) ==<br />
<br />
=== Diagnose whether you have a crash ===<br />
The most important thing is to get the machine and all ve’s back up as soon as possible. Note the time, you’ll need to create a crash log entry (Mgmt. -> Reference -> CrashLog). The first thing to do is head over to the [[Screen#Screen_Organization|serial console screen]] and see if there’s any kernel error messages output. Try to copy any messages (or just a sample of repeating messages) you see into the notes section of the crash log – these will also likely need to be sent to virtuozzo for interpretation. If the messages are spewing too fast, hit ^O + H to start a screen log dump which you can ob1182.pts-38.bb serve after the machine is rebooted. Additionally, if the machine is responsive, you can get a trace to send to virtuozzo by hooking up a kvm and entering these 3 sequences:<br />
<pre>alt+print screen+m<br />
alt+print screen+p<br />
alt+print screen+t</pre><br />
<br />
If there are no messages, the machine may just be really busy- wait a bit (5-10min) to see if it comes back. If it's still pinging, odds are its very busy. If it doesn't come back, or the messages indicate a fatal error, you will need to proceed with a power cycle (ctrl+alt+del will not work).<br />
<br />
=== Power cycle the server ===<br />
If this machine is not a Dell 2950 with a [[DRAC/RMM#DRAC|DRAC card]] (i.e. if you can’t ssh into the DRAC card and issue racadm serveraction hardreset, then you will need someone at the data center to power the macine off, wait 30 sec, then turn it back on. Make sure to re-attach via console (<tt>tip virtxx</tt>) immediately after power down. <br />
<br />
=== (Re)attach to the console ===<br />
Stay on the console the entire time during boot. As the BIOS posts- look out for the RAID card output- does everything look healthy? The output may be scrambled, look for "DEGRADED" or "FAILED". Once the OS starts booting you will be disconnected (dropped back to the shell on the console server) a couple times during the boot up. The reason you want to quickly re-attach is two-fold: 1. If you don’t reattach quickly then you won’t get any console output, 2. you want to be attached before the server ''potentially'' starts (an extensive) fsck. If you attach after the fsck begins, you’ll have seen no indication it started an fsck and the server will appear frozen during startup- no output, no response. <br />
<br />
=== Start containers/VE's/VPSs ===<br />
When the machine begins to start VE’s, it’s safe to leave the console and login via ssh. All virts should be set to auto start all the VEs after a crash. Further, most (newer) virts are set to “fastboot” it’s VE’s (to find out, do:<br />
grep -i fast /etc/sysconfig/vz <br />
and look for <tt>VZFASTBOOT=yes</tt>). If this was set prior to the machine’s crash (setting it after the machine boots will not have any effect until the vz service is restarted) it will start each ve as fast as possible, in serial, then go thru each VE (serially), shutting it down running a vzquota (disk usage) check, then bringing it back up. The benefit is that all VE’s are brought up quickly (within 15min or so depending on the #), the downside is a customer watching closely will notice 2 outages – 1st the machine crash, 2nd their quota check (which will be a much shorter downtime- on the order of a few minutes). <br />
<br />
Where “fastboot” is not set to yes (i.e on quar1), vz will start them consecutively, checking the quotas one at a time, and the 60th VE may not start until an hour or two later - this is not acceptable.<br />
<br />
The good news is, if you run vzctl start for a VE that is already started, you will simply get an error: <tt>VE is already started</tt>. Further, if you attempt to vzctl start a VE that is in the process of being started, you will simply get an error: unable to lock VE. So, there is no danger in simply running scripts to start smaller sets of VEs. If the system is not autostarting, then there is no issue, and even if it does, when it conflicts, one process (yours or the autostart) will lose, and just move on to the next one.<br />
<br />
A script has been written to assist with ve starts: [[#startvirt.pl|startvirt.pl]] which will start 6 ve’s at once until there are no more left. If startvirt.pl is used on a system where “fastboot” was on, it will circumvent the fastboot for ve’s started by startvirt.pl – they will go through the complete quota check before starting- therefore this is not advisable when a system has crashed. When a system is booted cleanly, and there's no need for vzquota checks, then startvirt.pl is safe and advisable to run.<br />
<br />
=== Make sure all containers are running ===<br />
You can quickly get a feel for how many ve’s are started by running:<br />
<br />
<pre>[root@virt4 log]# vs<br />
VEID 16066 exist mounted running<br />
VEID 16067 exist mounted running<br />
VEID 4102 exist mounted running<br />
VEID 4112 exist mounted running<br />
VEID 4116 exist mounted running<br />
VEID 4122 exist mounted running<br />
VEID 4123 exist mounted running<br />
VEID 4124 exist mounted running<br />
VEID 4132 exist mounted running<br />
VEID 4148 exist mounted running<br />
VEID 4151 exist mounted running<br />
VEID 4155 exist mounted running<br />
VEID 42 exist mounted running<br />
VEID 432 exist mounted running<br />
VEID 434 exist mounted running<br />
VEID 442 exist mounted running<br />
VEID 450 exist mounted running<br />
VEID 452 exist mounted running<br />
VEID 453 exist mounted running<br />
VEID 454 exist mounted running<br />
VEID 462 exist mounted running<br />
VEID 463 exist mounted running<br />
VEID 464 exist mounted running<br />
VEID 465 exist mounted running<br />
VEID 477 exist mounted running<br />
VEID 484 exist mounted running<br />
VEID 486 exist mounted running<br />
VEID 490 exist mounted running</pre><br />
<br />
So to see how many ve’s have started:<br />
<pre>[root@virt11 root]# vs | grep running | wc -l<br />
39</pre><br />
<br />
And to see how many haven’t:<br />
<pre>[root@virt11 root]# vs | grep down | wc -l<br />
0</pre><br />
<br />
And how many we should have running:<br />
<pre>[root@virt11 root]# vs | wc -l<br />
39</pre><br />
<br />
Another tool you can use to see which ve’s have started, among other things is [[#vzstat|vzstat]]. It will give you CPU, memory, and other stats on each ve and the overall system. It’s a good thing to watch as ve’s are starting (note the VENum parameter, it will tell you how many have started):<br />
<br />
<pre>4:37pm, up 3 days, 5:31, 1 user, load average: 1.57, 1.68, 1.79<br />
VENum 40, procs 1705: running 2, sleeping 1694, unint 0, zombie 9, stopped 0<br />
CPU [ OK ]: VEs 57%, VE0 0%, user 8%, sys 7%, idle 85%, lat(ms) 412/2<br />
Mem [ OK ]: total 6057MB, free 9MB/54MB (low/high), lat(ms) 0/0<br />
Swap [ OK ]: tot 6142MB, free 4953MB, in 0.000MB/s, out 0.000MB/s<br />
Net [ OK ]: tot: in 0.043MB/s 402pkt/s, out 0.382MB/s 4116pkt/s<br />
Disks [ OK ]: in 0.002MB/s, out 0.000MB/s<br />
<br />
VEID ST %VM %KM PROC CPU SOCK FCNT MLAT IP<br />
1 OK 1.0/17 0.0/0.4 0/32/256 0.0/0.5 39/1256 0 9 69.55.227.152<br />
21 OK 1.3/39 0.1/0.2 0/46/410 0.2/2.8 23/1860 0 6 69.55.239.60<br />
133 OK 3.1/39 0.1/0.3 1/34/410 6.3/2.8 98/1860 0 0 69.55.227.147<br />
263 OK 2.3/39 0.1/0.2 0/56/410 0.3/2.8 34/1860 0 1 69.55.237.74<br />
456 OK 17/39 0.1/0.2 0/100/410 0.1/2.8 48/1860 0 11 69.55.236.65<br />
476 OK 0.6/39 0.0/0.2 0/33/410 0.1/2.8 96/1860 0 10 69.55.227.151<br />
524 OK 1.8/39 0.1/0.2 0/33/410 0.0/2.8 28/1860 0 0 69.55.227.153<br />
594 OK 3.1/39 0.1/0.2 0/45/410 0.0/2.8 87/1860 0 1 69.55.239.40<br />
670 OK 7.7/39 0.2/0.3 0/98/410 0.0/2.8 64/1860 0 216 69.55.225.136<br />
691 OK 2.0/39 0.1/0.2 0/31/410 0.0/0.7 25/1860 0 1 69.55.234.96<br />
744 OK 0.1/17 0.0/0.5 0/10/410 0.0/0.7 7/1860 0 6 69.55.224.253<br />
755 OK 1.1/39 0.0/0.2 0/27/410 0.0/2.8 33/1860 0 0 192.168.1.4<br />
835 OK 1.1/39 0.0/0.2 0/19/410 0.0/2.8 5/1860 0 0 69.55.227.134<br />
856 OK 0.3/39 0.0/0.2 0/13/410 0.0/2.8 16/1860 0 0 69.55.227.137<br />
936 OK 3.2/52 0.2/0.4 0/75/410 0.2/0.7 69/1910 0 8 69.55.224.181<br />
1020 OK 3.9/39 0.1/0.2 0/60/410 0.1/0.7 55/1860 0 8 69.55.227.52<br />
1027 OK 0.3/39 0.0/0.2 0/14/410 0.0/2.8 17/1860 0 0 69.55.227.83<br />
1029 OK 1.9/39 0.1/0.2 0/48/410 0.2/2.8 25/1860 0 5 69.55.227.85<br />
1032 OK 12/39 0.1/0.4 0/80/410 0.0/2.8 41/1860 0 8 69.55.227.90</pre><br />
<br />
When you are all done, you will want to make sure that all the VEs really did get started, run vs one more time.<br />
<br />
Note the time all ve’s are back up and enter that into and save the crash log entry.<br />
<br />
Occasionally, a ve will not start automatically. The most common reason for a ve not to come up normally is the ve was at it’s disk limit before the crash, and will not start since they’re over the limit. To overcome this, set the disk space to current usage level (the system will give this to you when it fails to start), start the ve, then re-set the disk space back to the prior level. Lastly, contact the customer to let them know they’re out of disk (or allocate more disk if they're entitled to more).<br />
<br />
== Hitting performance barriers and fixing them ==<br />
<br />
There are multiple modes virtuozzo offers to allocate resources to a ve. We utilize 2: SLM and UBC parameters<br />
On our 4.x systems, we use all SLM – it’s simpler to manage and understand. There are a few systems on virt19/18 that may also use SLM. Everything else uses UBC. <br />
You can tell a SLM ve by:<br />
<br />
SLMMODE="all"<br />
<br />
in their conf file. <br />
<br />
TODO: detail SLM modes and parameters.<br />
<br />
If someone is in SLM mode and they hit memory resource limits, they simply need to upgrade to more memory.<br />
<br />
The following applies to everyone else (UBC).<br />
<br />
Customers will often email and say that they are getting out of memory errors - a common one is "cannot fork" ... basically, anytime you see something odd like this, it means they are hitting one of their limits that is in place in their conf file.<br />
<br />
The conf file, however, simply shows their limits - how do we know what they are currently at ?<br />
<br />
The answer is a file called v - this file contains the current status (and peaks) of their performance settings, and also counts how many times they have hit the barrier. The output of the file looks like this:<br />
<br />
<pre>764: kmemsize 384113 898185 8100000 8200000 0<br />
lockedpages 0 0 322 322 0<br />
privvmpages 1292 7108 610000 615000 0<br />
shmpages 270 528 33000 34500 0<br />
dummy 0 0 0 0 0<br />
numproc 8 23 410 415 0<br />
physpages 48 5624 0 2147483647 0<br />
vmguarpages 0 0 13019 2147483647 0<br />
oomguarpages 641 6389 13019 2147483647 0<br />
numtcpsock 3 21 1210 1215 0<br />
numflock 1 3 107 117 0<br />
numpty 0 2 19 19 0<br />
numsiginfo 0 4 274 274 0<br />
tcpsndbuf 0 80928 1800000 1900000 0 <br />
tcprcvbuf 0 108976 1800000 1900000 0<br />
othersockbuf 2224 37568 900000 950000 0<br />
dgramrcvbuf 0 4272 200000 200000 0<br />
numothersock 3 9 650 660 0<br />
dcachesize 53922 100320 786432 818029 0<br />
numfile 161 382 7500 7600 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
numiptent 4 4 155 155 0</pre><br />
<br />
The first column is the name of the counter in question - the same names we saw in the systems conf file. The second column is the _current_ value of that counter, the third column is the max that that counter has ever risen to, the fourth column is the soft limit, and the fifth column is the hard limit (which is the same as the numbers in that systems conf file).<br />
<br />
The sixth number is the failcount - how many times the current usage has risen to hit the barrier. It will increase as soon as the current usage hits the soft limit.<br />
<br />
The problem with /proc/user_beancounters is that it actually contains that set of data for every running VE - so you can't just cat /proc/user_beancounters - it is too long and you get info for every other running system.<br />
<br />
You can vzctl enter the system and run:<br />
<br />
vzctl enter 9999<br />
cat /proc/user_beancounters<br />
<br />
inside their system, and you will just see the stats for their particular system, but entering their system every time you want to see it is combersome.<br />
<br />
So, I wrote a simple script called "vzs" which simply greps for the VEID, and spits out the next 20 or so lines (however many lines there are in the output, I forget) after it. For instance:<br />
<br />
<pre># vzs 765:<br />
765: kmemsize 2007936 2562780 8100000 8200000 0<br />
lockedpages 0 8 322 322 0<br />
privvmpages 26925 71126 610000 615000 0<br />
shmpages 16654 16750 33000 34500 0<br />
dummy 0 0 0 0 0<br />
numproc 41 57 410 415 0<br />
physpages 1794 49160 0 2147483647 0<br />
vmguarpages 0 0 13019 2147483647 0<br />
oomguarpages 4780 51270 13019 2147483647 0<br />
numtcpsock 23 37 1210 1215 0<br />
numflock 17 39 107 117 0<br />
numpty 1 3 19 19 0<br />
numsiginfo 0 6 274 274 0<br />
tcpsndbuf 22240 333600 1800000 1900000 0<br />
tcprcvbuf 0 222656 1800000 1900000 0<br />
othersockbuf 104528 414944 900000 950000 0<br />
dgramrcvbuf 0 4448 200000 200000 0<br />
numothersock 73 105 650 660 0<br />
dcachesize 247038 309111 786432 818029 0<br />
numfile 904 1231 7500 7600 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
numiptent 4 4 155 155 0</pre><br />
<br />
That showed us just the portion of /proc/user_beancounters for system 765.<br />
<br />
When you run the vzs command, always add a : after the VEID.<br />
<br />
So, if a customer complains about some out of memory errors, or no more files, or no more ptys, or just has an unspecific complain about processes dying, etc., the very first thing you need to do is check their beancounters with vzs. Usually you will spot an item that has a high failcount and needs to be upped.<br />
<br />
At that point you could simply up the counter with `vzctl set`. Generally pick a number 10-20% higher than the old one, and make the hard limit slightly larger than the the soft limit. However our systems now come in several levels and those levels have more/different memory allocations. If someone is complaining about something other than a memory limit (pty, numiptent, numflock), it’s generally safe to increase it, at least to the same level as what’s in the /vzconf/4unlimited file on the newest virt. If someone is hitting a memory limit, first make sure they are given what they deserve:<br />
<br />
(refer to mgmt -> payments -> packages)<br />
<br />
To set those levels, you use the [[#setmem|setmem]] command. <br />
<br />
The alternate (DEPRECATED) method would be to use one of 3 commands:<br />
256 <veid><br />
300 <veid><br />
384 <veid><br />
512 <veid><br />
<br />
If the levels were not right (you’d run vzs <veid> before and after to see the effect) tell the customer they’ve been adjusted and be done with it. If the levels were right, tell the customer they must upgrade to a higher package, tell them how to see level (control panel) and that they can reboot their system to escape this lockup contidion.<br />
<br />
Customers can also complain that their site is totally unreachable, or complain that it is down ... if the underlying machine is up, and all seems well, you may notice in the beancounters that network-specific counters are failing - such as numtcpsock, tcpsndbuf or tcprcvbuf. This will keep them from talking on the network and make it seem like their system is down. Again, just up the limits and things should be fine.<br />
<br />
On virts 1-4, you should first look at the default settings for that item on a later virt, such as virt 8 - we have increased the defaults a lot since the early machines. So, if you are going to up a counter on virt2, instead of upping it by 10-20%, instead up it to the new default that you see on virt8.<br />
<br />
== Moving a VE to another virt (migrate/migrateonline) ==<br />
<br />
This will take a while to complete - and it is best to do this at night when the load is light on both machines.<br />
<br />
There are different methods for this, depending on which version of virtuozzo is installed on the src. and dst. virt. <br />
To check which version is running: <br />
[root@virt12 private]# cat /etc/virtuozzo-release<br />
Virtuozzo release 2.6.0<br />
<br />
Ok, let's say that the VE is 1212, and vital stats are:<br />
<pre>[root@virt12 sbin]# vc 1212<br />
VE_ROOT="/vz1/root/1212"<br />
VE_PRIVATE="/vz1/private/1212"<br />
OSTEMPLATE="fedora-core-2/20040903"<br />
IP_ADDRESS="69.55.229.84"<br />
TEMPLATES="devel-fc2/20040903 php-fc2/20040813 mysql-fc2/20040812 postgresql-fc2/20040813 mod_perl-fc2/20040812 mod_ssl-fc2/20040811 jre-fc2/20040823 jdk-fc2/20040823 mailman-fc2/20040823 analog-fc2/20040824 proftpd-fc2/20040818 tomcat-fc2/20040823 usermin-fc2/20040909 webmin-fc2/20040909 uw-imap-fc2/20040830 phpBB-fc2/20040831 spamassassin-fc2/20040910 PostNuke-fc2/20040824 sl-webalizer-fc2/20040<br />
818"<br />
<br />
[root@virt12 sbin]# vzctl exec 1212 df -h<br />
Filesystem Size Used Avail Use% Mounted on<br />
/dev/vzfs 4.0G 405M 3.7G 10% /</pre><br />
<br />
From this you can see that he’s using (and will minimally need free on the dst server) ~400MB, and he’s running on a Fedora 2 template, version 20040903. He’s also got a bunch of other templates installed. It’s is '''vital''' that '''all''' these templates exist on the dst system. To confirm that, on the dst system run:<br />
<br />
For < 3.0:<br />
<br />
<pre>[root@virt14 private]# vzpkgls | grep fc2<br />
devel-fc2 20040903<br />
PostNuke-fc2 20040824<br />
analog-fc2 20040824<br />
awstats-fc2 20040824<br />
bbClone-fc2 20040824<br />
jdk-fc2 20040823<br />
jre-fc2 20040823<br />
mailman-fc2 20040823<br />
mod_frontpage-fc2 20040816<br />
mod_perl-fc2 20040812<br />
mod_ssl-fc2 20040811<br />
mysql-fc2 20040812<br />
openwebmail-fc2 20040817<br />
php-fc2 20040813<br />
phpBB-fc2 20040831<br />
postgresql-fc2 20040813<br />
proftpd-fc2 20040818<br />
sl-webalizer-fc2 20040818<br />
spamassassin-fc2 20040910<br />
tomcat-fc2 20040823<br />
usermin-fc2 20040909<br />
uw-imap-fc2 20040830<br />
webmin-fc2 20040909<br />
[root@virt14 private]# vzpkgls | grep fedora<br />
fedora-core-1 20040121 20040818<br />
fedora-core-devel-1 20040121 20040818<br />
fedora-core-2 20040903<br />
[root@virt14 private]#</pre><br />
<br />
For these older systems, you can simply match up the date on the template. <br />
<br />
For >= 3.0:<br />
<br />
<pre>[root@virt19 /vz2/private]# vzpkg list<br />
centos-5-x86 2008-01-07 22:05:57<br />
centos-5-x86 devel<br />
centos-5-x86 jre<br />
centos-5-x86 jsdk<br />
centos-5-x86 mod_perl<br />
centos-5-x86 mod_ssl<br />
centos-5-x86 mysql<br />
centos-5-x86 php<br />
centos-5-x86 plesk9<br />
centos-5-x86 plesk9-antivirus<br />
centos-5-x86 plesk9-api<br />
centos-5-x86 plesk9-atmail<br />
centos-5-x86 plesk9-backup<br />
centos-5-x86 plesk9-horde<br />
centos-5-x86 plesk9-mailman<br />
centos-5-x86 plesk9-mod-bw<br />
centos-5-x86 plesk9-postfix<br />
centos-5-x86 plesk9-ppwse<br />
centos-5-x86 plesk9-psa-firewall<br />
centos-5-x86 plesk9-psa-vpn<br />
centos-5-x86 plesk9-psa-fileserver<br />
centos-5-x86 plesk9-qmail<br />
centos-5-x86 plesk9-sb-publish<br />
centos-5-x86 plesk9-vault<br />
centos-5-x86 plesk9-vault-most-popular<br />
centos-5-x86 plesk9-watchdog</pre><br />
<br />
On these newer systems, it's difficult to tell whether the template on the dst matches exactly the src. Just cause a centos-5-x86 is listed on both servers doesn't mean all the same packages are there on the dst. To truly know, you must perform a sample rsync:<br />
<br />
rsync -avn /vz/template/centos/5/x86/ root@10.1.4.61:/vz/template/centos/5/x86/<br />
<br />
if you see a ton of output from the dry run command, then clearly there are some differences. You may opt to let the rsync complete (without running in dry run mode) the only downside is you've now used up more space on the dst and also the centos template will be a mess with old and new data- it will be difficult if not impossible to undo (if someday we wanted to reclaim the space).<br />
<br />
If you choose to merge templates, you should closely inspect the dry run output. You should also take care to exclude anything in the /config directory. For example:<br />
<br />
rsync -av -e ssh --stats --exclude=x86/config /vz/template/ubuntu/10.04/ root@10.1.4.62:/vz/template/ubuntu/10.04/<br />
<br />
Which will avoid this directory and contents:<br />
<pre>[root@virt11 /vz2/private]# ls /vz/template/ubuntu/10.04/x86/config*<br />
app os</pre><br />
<br />
This is important to avoid since the config may differ on the destination and we are really only interested in making sure the pacakges are there, not overwriting a newer config with an older one.<br />
<br />
If the dst system was missing a template, you have 2 choices: <br />
# put the missing template on the dst system. 2 choices here: <br />
## Install the template from rpm (found under backup2: /mnt/data4/vzrpms/distro/) or <br />
## rsync over the template (found under /vz/template) - see above<br />
# put the ve on a system which has all the proper templates<br />
<br />
=== pre-seeding a migration ===<br />
<br />
When migrating a customer (or when doing many) depending on how much data you have to transfer, it can take some time. Further, it can be difficult to gauge when a migration will complete or how long it will take. To help speed up the process and get a better idea about how long it will take you can pre-transfer a customer's data to the destination server. If done correctly, vzmigrate will see the pre-transferred data and pick up where you left off, having much less to transfer (just changed/new files). <br />
<br />
We believe vzmigrate uses rsync to do it's transfer. Therefore not only can you use rsync to do a pre-seed, you can also run rsync to see what is causing a repeatedly-failing vzmigrate to fail. <br />
<br />
There's no magic to a pre-seed, you just need to make sure it's named correctly.<br />
<br />
Given:<br />
<br />
source: /vz1/private/1234<br />
<br />
and you want to migrate to /vz2 on the target system, your rsync would look like:<br />
<br />
rsync -av /vz1/private/1234/ root@x.x.x.x:/vz2/private/1234.migrated/<br />
<br />
After running that successful rsync, the ensuing migrateonline (or migrate) will take much less time to complete- depending on the # of files to be analyzed and the # of changed files. In any case, it'll be much much faster than had you just started the migration from scratch.<br />
<br />
Further, as we discuss elsewhere in this topic, a failed migration can be moved from <tt>/vz/private/1234</tt> to <tt>/vz/private/1234.migrated</tt> on the destination if you want to restart a failed migration. This should '''only''' be done if the migration failed and the CT is not running on the destination HN.<br />
<br />
=== migrateonline intructions: src >=3.x -> dst>=3.x ===<br />
<br />
A script called [[#migrateonline|migrateonline]] was written to handle this kind of move. It is basically a wrapper for <tt>vzmigrate</tt> – vzmigrate is a util to seamlessly- as no no reboot of the ve necessary- move a ve from one host to another. This wrapper was initially written cause vz virtuozzo version 2.6.0 has a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables, causing problems when the ve was started up on the dst system. [[#migrate|migrate]] mitigates that. Since it makes multiple ssh connections to the dst virt, it’s a good idea to put the pub key for the src virt in the authorized_keys file on the dst virt. In addition, migrateonline emails ve owners when their migration starts and stops. For this to happen they need to put email addresses (on a single line, space delimited) in a file on their system: /migrate_notify. If the optional target dir is not specified, the ve will be moved to the same private/root location as it was on the src virt. Note: <tt>migrate</tt> is equivalent to <tt>migrateonline</tt>, but will <tt>migrate</tt> a ve AND restart it in the process.<br />
<br />
<pre>[root@virt12 sbin]# migrateonline<br />
usage: /usr/local/sbin/migrateonline <ip of node migrating to> <veid> [target dir: vz | vz1 | vz2]<br />
[root@virt12 sbin]# migrateonline 10.1.4.64 1212 vz<br />
starting to migrate 1212 at Sat Mar 26 22:40:38 PST 2005<br />
Turning off offline_management<br />
Saved parameters for VE 1212<br />
migrating with no start on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
clearing the arp cache<br />
now going to 10.1.4.64 and clear cache and starting<br />
starting it<br />
Delete port redirection<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
finished migrating 1212 at Sat Mar 26 22 22:52:01 PST 2005<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine.<br />
<br />
If they had backups, use the mvbackups command to move their backups to the new server:<br />
<br />
mvbackups 1212 virt14 vz<br />
<br />
Rename the ve<br />
<br />
[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/migrated-1212<br />
<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/old-1212-migrated-20120404-noarchive<br />
<br />
Update the customer’s systems in mgmt to reflect the new path and server.<br />
<br />
<br />
IF migrateonline does not work, you can try again using simply migrate- this will result in a brief reboot for the ve.<br />
Before you try again, make sure of a few things:<br />
<br />
Depending on where in the migration died, there may be partial data on the dst system in 1 of 2 places:<br />
(given the example above)<br />
<br />
/vz/private/1212<br />
<br />
or <br />
<br />
/vz/private/1212.migrated<br />
<br />
before you run migrate again, you'll want to rename so that all data is in <br />
1212.migrated:<br />
<br />
mv /vz/private/1212 /vz/private/1212.migrated<br />
<br />
this way, it will pick up where it left off and transfer only new files.<br />
<br />
Likewise, if you want to speed up a migration, you can pre-seed the dst as follows:<br />
<br />
[root@virt12 sbin]# rsync -avSH /vz/private/1212/ root@10.1.4.64:/vz/private/1212.migrated/<br />
<br />
then when you run migrate or migrateonline, it will only need to move the changed files- the migration will complete quickly<br />
<br />
=== migrate manually (if migrate/online fails) ===<br />
<br />
Lets say for whatever reason the migration fails. You can always move someone manually:<br />
<br />
1. stop ve: <br><br />
v stop 1234<br />
<br />
2. copy over data<br><br />
rsync -avSH /vz/private/1234/ root@1.1.1.1:/vzX/private/1234/<br />
<br />
NOTE: if you've previously seeded the data (run rsync while the VE was up/running), and this is a subsequent rsync, make sure the last rsync you do (while the VE is not running, has the --delete option in the rsync)<br />
<br />
3. copy over conf<br><br />
scp /vzconf/1234.conf root@1.1.1.1:/vzconf<br />
<br />
4. on dst, edit the conf to reflect the right vzX dir<br><br />
vi /vzconf/1234.conf<br />
<br />
5. on src remove the IPs<br><br />
ipdel 1234 2.2.2.2 3.3.3.3<br />
<br />
6. on dst add IPs <br><br />
ipadd 1234 2.2.2.2 3.3.3.3<br />
<br />
7. on dst, start ve: <br><br />
v start 1324<br />
<br />
8. cancel, then archive ve on src per above instrs.<br />
<br />
=== migrate src=2.6.0 -> dst>=2.6.0, or mass-migration with customer notify ===<br />
<br />
A script called <tt>migrate</tt> was written to handle this kind of move. It is basically a wrapper for vzmigrate – vzmigrate is a util to seamlessly move a ve from one host to another. This wrapper was initially written cause vz virtuozzo version 2.6.0 has a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables, causing problems when the ve was started up on the dst system. migrate mitigates that. Since it makes multiple ssh connections to the dst virt, it’s a good idea to put the pub key for the src virt in the authorized_keys file on the dst virt. In addition, migrate emails ve owners when their migration starts and stops. For this to happen they need to put email addresses (on a single line, space delimited) in a file on their system: /migrate_notify. If the optional target dir is not specified, the ve will be moved to the same private/root location as it was on the src virt. Note: migrateonline is equivalent to migrate, but will migrate a ve from one 2.6 '''kernel''' machine to another 2.6 kernel machine without restarting the ve.<br />
<br />
<pre>[root@virt12 sbin]# migrate<br />
usage: /usr/local/sbin/migrate <ip of node migrating to> <veid> [target dir: vz | vz1 | vz2]<br />
[root@virt12 sbin]# migrate 10.1.4.64 1212 vz<br />
starting to migrate 1212 at Sat Mar 26 22:40:38 PST 2005<br />
Turning off offline_management<br />
Saved parameters for VE 1212<br />
migrating with no start on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
clearing the arp cache<br />
now going to 10.1.4.64 and clear cache and starting<br />
starting it<br />
Delete port redirection<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
finished migrating 1212 at Sat Mar 26 22 22:52:01 PST 2005<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine (backup2 or mail).<br />
Cancel the ve (first we have to rename things which migrate changed so cancelve will find them):<br />
<br />
[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/1212.conf<br />
<br />
On 2.6.1 you’ll also have to move the private area:<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/1212<br />
<br />
<pre>[root@virt12 sbin]# cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, [[#cancelve|cancelve]] would offer to remove them. You want to say '''no''' to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
To move everyone off a system, you’d do:<br />
for f in `vl`; do migrate <ip> $f; done<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
=== vzmigrate: src=2.6.1 -> dst>=2.6.0 ===<br />
<br />
This version of vzmigrate works properly with regard to handling ips. It will not notify ve owners of moves as in the above example. Other than that it’s essentially the same.<br />
<br />
<pre>[root@virt12 sbin]# vzmigrate 10.1.4.64 -r no 1212:1212:/vz/private/1212:/vz/root/1212<br />
migrating on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine (backup2 or mail).<br />
Cancel the ve (first we have to rename things which vzmigrate changed so cancelve will find them):<br />
<br />
<pre>[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/1212.conf<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/1212<br />
<br />
[root@virt12 sbin]# cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, <tt>cancelve</tt> would offer to remove them. You want to say no to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
=== src=2.5.x ===<br />
<br />
First, go to the private dir:<br />
<br />
cd /vz1/private/<br />
<br />
Stop the VE - make sure it stops totally cleanly.<br />
<br />
vzctl stop 1212<br />
<br />
Then you’d use vemove - a script written to copy over the config, create tarballs of the ve’s data on the destination virt, and cancel the ve on the source system (in this example we’re going to put a ve that was in /vz1/private on the src virt, in /vz/private on the dst virt):<br />
<br />
<pre>[root@virt12 sbin]# vemove<br />
ERROR: Usage: vemove veid target_ip target_path_dir<br />
[root@virt12 sbin]# vemove 1212 10.1.4.64 /vz/private/1212<br />
tar cfpP - 1212 --ignore-failed-read | (ssh -2 -c arcfour 10.1.4.64 "split - -b 1024m /vz/private/1212.tar" )<br />
scp /vzconf/1212.conf 10.1.4.64:/vzconf<br />
cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, cancelve would offer to remove them. You want to say no to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
When you are done, go to /vz/private on the dst virt you will have files like this:<br />
<br />
<pre>1212.taraa<br />
1212.tarab<br />
1212.tarac</pre><br />
<br />
Each one 1024m (or less, for the last one) in size.<br />
<br />
on the dst server and run:<br />
<br />
cat 1212.tar?? | tar xpPBf -<br />
<br />
and after 20 mins or so it will be totally untarred. Now since the conf<br />
file is already there, you can go ahead and start the system.<br />
<br />
vzctl start 1212<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
NOTE: you MUST tar the system up using the virtuozzo version of tar that<br />
is on all the virt systems, and further you MUST untar the tarball with<br />
the virtuozzo tar, using these options: `<tt>tar xpPBf -</tt>`<br />
<br />
If you tar up an entire VE and move it to a non-virtuozzo machine, that is<br />
ok, and you can untar it there with normal tar commands, but do not untar<br />
it and then repack it with a normal tar and expect it to work - you need<br />
to use virtuozzo tar commands on virtuozzo tarballs to make it work.<br />
<br />
The backups are sort of an exception, since we are just (usually)<br />
restoring user data that was created after we gave them the system, and<br />
therefore has nothing to do with magic symlinks or vz-rpms, etc.<br />
<br />
== Moving a VE on the same virt ==<br />
<br />
Easy way:<br><br />
Scenario 1: ve 123 is to be renamed 1231 and moved from vz1 to vz<br />
<br />
vzmlocal 123:1231:/vz/private/1231:/vz/root/1231<br />
<br />
Scenario 2: ve 123 is to be moved vz1 to vz<br />
<br />
vzmlocal 123:123:/vz/private/123:/vz/root/123<br />
<br />
vzmlocal will reboot the ve at the end of the move<br />
<br />
Manual/old way:<br />
<br />
1) <tt>vzctl stop 123</tt><br><br />
2) <tt>mv /vz1/private/123 /vz/private/.</tt><br><br />
(or cp -a if you want to copy)<br />
3) in <tt>/etc/sysconfig/vz-scripts/123.conf</tt> change value<br><br />
of '<tt>VE_PRIVATE</tt>' variable to point to a new private area location<br />
4) <tt>vzctl start 123</tt><br><br />
5) update backups if needed: <tt>mvbackups 123 virtX virt1 vz</tt><br><br />
6) update management scerens<br><br />
<br />
Notes: a) absolute path to private area is stored in quota file <tt>/var/vzquota/quota.123</tt> - so during first startup quota will be recalculated.<br><br />
b) if you're going to write some script to do a job, you MUST be sure that $VEID won't be expanded to '' in ve config file - ie. you need to escape '$'. Otherwise you might have:<br />
<br />
VE_PRIVATE="/vz/private/"<br />
<br />
in config, and 'vzctl destroy' for this VE ID '''will remove everything under /vz/private/ directory'''.<br />
<br />
== Adding a veth device to a VE ==<br />
<br />
Not totally sure what this is, but a customer asked for it and here's what we did (as instructed by vz support):<br />
<br />
<pre>v set 99 --netif_add eth99 --save<br />
ipdel 99 69.55.230.58<br />
v set 99 --ifname eth99 --ipadd 69.55.230.58 --save<br />
v set 99 --ifname eth99 --gateway 69.55.230.1 --save<br />
<br />
vznetcfg net new veth_net<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
net99 active eth0 veth77.77,veth99.99<br />
veth_net active<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
br0 bridge veth_net<br />
br99 bridge net99<br />
veth99.99 veth net99<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic net99 69.55.227.70/24<br />
<br />
vznetcfg br attach br0 eth0<br />
<br />
(will remove 99 from orig net and move to veth_net)<br />
vznetcfg net addif veth_net veth99.99<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
net99 active<br />
veth_net active eth0 veth77.77,veth99.99<br />
<br />
(delete the old crap)<br />
vznetcfg net del net99<br />
<br />
<br />
Then, to add another device in<br />
<br />
v set 77 --netif_add eth77 --save<br />
ipdel 77 69.55.230.78<br />
v set 77 --ifname eth77 --ipadd 69.55.230.78 --save<br />
v set 77 --ifname eth77 --gateway 69.55.230.1 --save<br />
v set 77 --save --ifname eth77 --network veth_net<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
veth77.77 veth<br />
br0 bridge veth_net<br />
veth99.99 veth veth_net<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic veth_net 69.55.227.70/24<br />
<br />
vznetcfg net addif veth_net veth77.77<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
veth77.77 veth veth_net<br />
br0 bridge veth_net<br />
veth99.99 veth veth_net<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic veth_net 69.55.227.70/24<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
veth_net active eth0 veth77.77,veth99.99<br />
<br />
another example<br />
<br />
v set 1182 --netif_add eth1182 --save<br />
ipdel 1182 69.55.236.217<br />
v set 1182 --ifname eth1182 --ipadd 69.55.236.217 --save<br />
v set 1182 --ifname eth1182 --gateway 69.55.236.1 --save<br />
vznetcfg net addif veth_net veth1182.1182<br />
v set 1182 --save --ifname eth1182 --network veth_net<br />
<br />
<br />
unused/not working commands:<br />
ifconfig veth99.0 0<br />
vznetcfg net list<br />
vznetcfg br new br99 net99<br />
vznetcfg br attach br99 eth0<br />
vznetcfg br show<br />
vznetcfg if list<br />
vznetcfg net addif net99 veth99.99<br />
vznetcfg net addif eth0 net99<br />
<br />
---------<br />
<br />
vznetcfg br new br1182 net1182<br />
<br />
vznetcfg br attach br99 eth0<br />
vznetcfg if list<br />
vznetcfg net addif net99 veth99.99<br />
vznetcfg net addif eth0 net99<br />
<br />
<br />
<br />
vznetcfg net addif eth0 net1182<br />
<br />
<br />
# vznetcfg net addif veth99.0 net99<br />
--- 8< ---<br />
<br />
vznetcfg net new net<br />
# vznetcfg net addif eth0 net99<br />
<br />
<br />
# vzctl set 99 --save --netif_add eth0 (at this stage veth99.0 interface have to appear<br />
on node)<br />
# vzctl set 99 --save --ifname eth0 --ipadd 69.55.230.58 (and probably few more arguments<br />
here - see 'man vzctl')<br />
# vznetcfg net addif veth99.0 net99</pre><br />
<br />
<br />
== Assigning/remove ip from a VE ==<br />
<br />
1. Add or remove ips:<br />
ipdel 1234 1.1.1.1 2.2.2.2<br />
ipadd 1234 1.1.1.1 2.2.2.2<br />
<br />
2. update Mgmt screens<br />
<br />
3. offer to update any DNS we do for them<br />
<br />
4. check to see if we had rules for old IP in firwall<br />
<br />
== Enabling tun device for a ve ==<br />
Note, there’s a command for this: [[#addtun|addtun]]<br />
<br />
For posterity:<br />
Make sure the tun.o module is already loaded before Virtuozzo is started: <br />
lsmod <br />
Allow the VPS to use the TUN/TAP device: <br />
vzctl set 101 --devices c:10:200:rw --save <br />
Create the corresponding device inside the VPS and set the proper permissions: <br />
vzctl exec 101 mkdir -p /dev/net <br />
vzctl exec 101 mknod /dev/net/tun c 10 200 <br />
vzctl exec 101 chmod 600 /dev/net/tun<br />
<br />
== Remaking a system (on same virt) ==<br />
<br />
1. [[#cancelve|cancelve]] (or v destroy x - ONLY if you're POSITIVE no data needs to be saved)<br />
2. [[#vemake|vemake]] using same veid<br />
3. [[#mvbackups|mvbackups]] or [[#vb|vb]] (if new mount point)<br />
4. update mgmt with new dir/ip <br />
5. update firewall if changed ip<br />
<br />
<br />
== Traffic accounting on linux ==<br />
<br />
DEPRECATED - all tracking is done via bwdb now. This is how we used to track traffic.<br />
<br />
TODO: update for diff versions of vz<br />
<br />
Unlike FreeBSD, where we have to add firewall count rules to the system to count the traffic, on virtuozzo counts the traffic for us. You an see the current traffic stats by running `vznetstat`:<br />
<br />
<pre># vznetstat<br />
VEID Net.Class Output(bytes) Input(bytes)<br />
-----------------------------------------------<br />
24218 1 484M 39M<br />
24245 1 463M 143M<br />
2451 1 2224M 265M<br />
2454 1 2616M 385M<br />
4149 1 125M 68M<br />
418 1 1560M 34M<br />
472 1 1219M 315M<br />
726 1 628M 317M<br />
763 1 223M 82M<br />
771 1 4234M 437M<br />
-----------------------------------------------<br />
Total: 13780M 2090M</pre><br />
<br />
As you can see the VEID is on a line with the in and out bytes. So, we simply run a cron job:<br />
<br />
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /root/vztrafdump.sh<br />
<br />
Just like we do on FreeBSD - this one goes through all the VEs in /vz/private and greps the line from vznetstat that matches them and dumps it in /jc_traffic_dump on their system. Then it does it again for all the VEs in /vz1/private. It is important to note that vznetstat runs only once, and the grepping is done from a temporary file that contains that output - we do this because running vznetstat once for each VE that we read out of /vz/private and /vz1/private would take way too long and be too intensive.<br />
<br />
You do not need to do anything to facilitate this other than make sure that that cron job is running - the vznetstat counters are always running, and any new VEs that are added to the system will be accounted for automatically.<br />
<br />
Traffic resetting no longer works with vz 2.6, so we disable the vztrafdump.sh on those virts.<br />
<br />
== Watchdog script ==<br />
<br />
On some of the older virts, we have a watchdog running that kills procs that are deemed bad per the following:<br />
<br />
/root/watchdog from quar1<br />
<br />
<pre>if echo $line | awk '{print $(NF-3)}' | grep [5-9]...<br />
then<br />
# 50-90%<br />
if echo $line | awk '{print $(NF-1)}' | grep "...:.."<br />
then<br />
# running for > 99min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
if echo $line | awk '{print $(NF-1)}' | grep "....m"<br />
then<br />
# running for > 1000min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
fi<br />
<br />
if echo $line | awk '{print $(NF-3)}' | grep [1-9]...<br />
then<br />
# running for 10-90 percent<br />
if echo $line | awk '{print $NF}' | egrep 'cfusion|counter|vchkpw'<br />
then<br />
<br />
if echo $line | awk '{print $(NF-1)}' | grep "[2-9]:.."<br />
then<br />
# between 2-9min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
elif echo $line | awk '{print $(NF-1)}' | grep "[0-9][0-9]:.."<br />
then<br />
# up to 99min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
fi<br />
fi</pre><br />
<br />
<br />
== Misc Linux Items ==<br />
<br />
We are overselling hard drive space ... when you configure a linux system with a certain amount of disk space (the default is 4gigs) you do not actually use up 4gigs of space on the system. The diskspace setting for a user is simply a cap, and they only use up as much space on the actual disk drive as they are actually using.<br />
<br />
When you create a new linux system, even though there are some 300 RPMs or so installed, if you run `df -k` you will see that the entire 4gig partition is empty - no space is being used. This is because the files in their system are "magic symlinks" to the template for their OS that is in /vz/template - however, any changes to any of those files will "disconnect" them and they will immediately begin using space in their system. Further, any new files uploaded (even if those new files overwrite existing files) will take up space on the partition.<br />
<br />
<br />
if you see this:<br />
<br />
<pre>[root@virt8 root]# vzctl stop 160 ; vzctl start 160<br />
VE is not running<br />
Starting VE ...<br />
VE is unmounted<br />
VE is mounted<br />
Adding IP address(es): 69.55.226.83 69.55.226.84<br />
bash ERROR: Can't change file /etc/sysconfig/network<br />
Deleting IP address(es): 69.55.226.83 69.55.226.84<br />
VE is unmounted<br />
[root@virt8 root]#</pre><br />
<br />
it probably means they no longer have /bin/bash - copy one in for them<br />
<br />
ALSO: another possibility is that they have removed the `ed` RPM from their system - it needs to be reinstalled into their system. But since their system is down, this is tricky ...<br />
<br />
VE startup scripts used by 'vzctl' want package 'ed' to be available inside VE. So if package 'ed' will be enabled in OS template config and OS template itself VE #827 is based on - this error should be fixed.<br />
<br />
yes, it is possible to add RPM to VE while it not running.<br />
Try to do following:<br />
<br />
<pre># cd /vz/template/<OS_template_with_ed_package>/<br />
# vzctl mount 827<br />
# rpm -Uvh --root /vz/root/827 --veid 827 ed-0.2-25.i386.vz.rpm</pre><br />
<br />
Usually theres an error, but its ok<br />
<br />
Note: replace 'ed-0.2-25.i386.vz.rpm' in last command with actual<br />
version of 'ed' package you have.<br />
<br />
----<br />
<br />
So how do I know what template the user has ? cat their conf file and it is listed in there. For example, if the conf file has:<br />
<br />
<pre>[root@virt12 sbin]# vc 1103<br />
…snip…<br />
OSTEMPLATE="debian-3.0/20030822"<br />
TEMPLATES="mod_perl-deb30/20030707 mod_ssl-deb30/20030703 mysql-deb30/20030707 proftpd-deb30/20030703 webmin-deb30/20030823 "<br />
…</pre><br />
<br />
then they are on debian 3.0, all of their system RPMs are in /vz/template/debian-3.0, and they are using version 20030822 of that debian 3.0 template. Also, they’ve also got additional packages installed (mod_perl, mod_ssl, etc). Those are also found under /vz/template<br />
<br />
----<br />
<br />
Edits needed to run java:<br />
<br />
When we first created the VEs, the default setting for privvmpages was 93000:94000 ... which was high enough that most people never had problems ... however, you can;t run java or jdk or tomcat or anything java related with that setting. We have found that by setting privvmpages to 610000:615000 that java runs just fine. That is now the default setting. It is exceedingly rare that anyone needs it higher than that, although we have seen it once or twice.<br />
<br />
Any problems with java at all - the first thing you need to do is see if the failcnt has raised for privvmpages.<br />
<br />
<pre># vzctl start 160<br />
Starting VE ...<br />
vzquota : (error) Quota on syscall for 160: Device or resource busy<br />
Running vzquota on failed for VE 160 [3]</pre><br />
<br />
This is because my pwd is _in_ their private directory - you can't start it until you move out<br />
<br />
People seem to have trouble with php if they are clueless newbies. Here are two common problems/solutions:<br />
<br />
no... but i figured it out myself. problem was the php.ini file that came<br />
vanilla with the account was not configured to work with apache (the<br />
ENGINE directive was set to off).<br />
<br />
everything else seems fine now.<br />
<br />
----<br />
<br />
the problem was in the php.ini file. I noticed that is wasnt showing<br />
the code when it was in an html file so I looked at the php.ini file<br />
and had to change it so it recognized <? tags aswell as <?php tags.<br />
<br />
Also, make sure added to httpd.conf<br />
AddType application/x-httpd-php .php<br />
<br />
----<br />
<br />
You can set the time zone:<br />
<br />
You can change the timezone by doing this:<br />
<br />
ln -sf /usr/share/zoneinfo/<zone> /etc/localtime<br />
<br />
where <zone> is the zone you want in the /usr/share/zoneinfo/ directory.<br />
<br />
----<br />
<br />
Failing shm_open calls:<br />
<br />
first, please check if /dev/shm is mounted inside VE.<br />
'cat /proc/mounts' command should show something like this:<br />
tmpfs /dev/shm tmpfs rw 0 0<br />
<br />
If /dev/shm is not mounted you have 2 ways to solve issue:<br />
1. execute following command inside VE (doesn't require VE reboot):<br />
mount -t tmpfs none /dev/shm<br />
2. add following string to /etc/fstab inside VE and reboot it:<br />
tmpfs /dev/shm tmpfs defaults 0 0<br />
<br />
----<br />
<br />
You can have a mounted but not running ve<br />
Just:<br />
vzctl mount <veid><br />
<br />
----<br />
<br />
When a debian sys can’t get on the network, and you try:<br />
<br />
<pre>vzctl set 1046 --ipadd 69.55.227.117<br />
Adding IP address(es): 69.55.227.117<br />
Failed to bring up lo.<br />
Failed to bring up venet0.</pre><br />
<br />
They probably removed iproute package, which must be the one from swsoft. To restore:<br />
<pre># dpkg -i --veid=1046 --admindir=/vz1/private/1046/root/var/lib/dpkg --instdir=/vz1/private/1046/root/ /vz/template/debian-3.0/iproute_20010824-8_i386.vz.deb<br />
(Reading database ... 16007 files and directories currently installed.)<br />
Preparing to replace iproute 20010824-8 (using .../iproute_20010824-8_i386.vz.deb) ...<br />
Unpacking replacement iproute ...<br />
Setting up iproute (20010824-8) ...</pre><br />
<br />
Then restart their ve<br />
<br />
----<br />
<br />
in a ve i do:<br />
<br />
<pre>cd /<br />
du -h .</pre><br />
<br />
and get: 483M .<br />
<br />
i do:<br />
<br />
<pre>bash-2.05a# df -h<br />
Filesystem Size Used Avail Use% Mounted on<br />
/dev/vzfs 4.0G 2.3G 1.7G 56% /</pre><br />
<br />
how can this be?<br />
<br />
Is it possible that quota file was corrupted somehow? Please try to: <br />
<pre>vzctl stop <VEID><br />
vzquota drop <VEID><br />
vzquota init <VEID><br />
vzctl start <VEID></pre><br />
<br />
----<br />
<br />
How to stop vz from starting after reboot:<br />
<br />
VIRTUOZZO=no <br />
in <br />
/etc/sysconfig/vz<br />
<br />
To start: <br />
service vz start<br />
(after setting VIRTUOZZO=yes in /etc/sysconfig/vz)<br />
<br />
service vz restart will do some kind of 'soft reboot' -- restart all<br />
VPSes and reload modules without rebooting the node<br />
<br />
if you need to shut down all VPSes really really fast, run killall -9 init<br />
<br />
----<br />
<br />
Postfix tip:<br />
<br />
You may want to tweak settings: default_process_limit=10<br />
<br />
= Virtuozzo VPS Management Tools =<br />
<br />
== vm ==<br />
<br />
TODO<br />
<br />
== cancelve ==<br />
cancelve <veid><br />
this will:<br />
* stop a ve<br />
* check for backups (offer to remove them from the backup server <br />
and the backup.config)<br />
* rename the private dir<br />
* check for PTR, provide the commands to reset to default<br />
* and rename the ve’s config<br />
* remind you to remove firewall rules<br />
* remind you to remove DNS entries<br />
<br />
== ipadd ==<br />
ipadd <veid> <ip> [ip] [ip]<br />
add’s ip(s) to a ve<br />
<br />
== ipdel ==<br />
ipdel <veid> <ip> [ip] [ip]<br />
removes ip(s) from a ve<br />
<br />
== vc ==<br />
vc <veid><br />
the equivalent of: <tt>cat /vzconf/<veid>.conf</tt><br />
<br />
== vl ==<br />
vl<br />
will displays a list of ve #’s, 1 per line. (ostensibly to use in a for loop)<br />
<br />
== vp ==<br />
vp <veid><br />
the equivalent of: <tt>vzps auxww –E <veid></tt><br />
<br />
== vpe ==<br />
DEPRECATED<br />
vpe <veid> <br />
this will allow you to do a vp when a ve is running out of control, the equivalent of (deprecated since vp operates outside the VPS): <br />
<pre>vzctl set <veid> --kmemsize 2100000:2200000<br />
vzctl exec <veid> ps auxw<br />
vzctl set <veid> --kmemsize (ve’s orig lvalue):(ve’s orig hvalue)</pre><br />
<br />
== vt ==<br />
vt <veid><br />
the equivalent of: <tt>vztop –E <veid></tt><br />
<br />
== vr ==<br />
vr <veid><br />
the equivalent of: <tt>vzctl stop <veid>; vzctl start <veid></tt><br />
You can run this even if the ve is down - the stop command will just fail<br />
<br />
== vs ==<br />
vs [veid]<br />
displays status (output of <tt>vzctl status <veid></tt>) on each ve configured on the system (as determined by <tt>/vzconf/*.conf</tt>)<br />
If passed an argument, gives the status for just that ve. <br />
A running system looks like:<br />
<tt>VEID 16066 exist mounted running</tt><br />
<br />
A ve which is not running (but does exist) looks like:<br />
<tt>VEID 9990 exist unmounted down</tt><br />
<br />
A ve which is not running and doesn’t exist looks like:<br />
<tt>VEID 421 deleted unmounted down</tt><br />
<br />
== vs2 ==<br />
vs2 [veid]<br />
this is similar to vs in that it displays status (output of <tt>vzctl status <veid></tt>) on each ve,<br />
but the difference is it’s list comes from doing an ls on the data dirs. This was meant to catch <br />
the rare case where a ve configured but exists. <br />
<br />
== vw ==<br />
vw [veid]<br />
displays the output of ‘<tt>w</tt>’ (the equivalent of <tt>vzctl exec <veid> w</tt>) for each configured ve (as determined by <tt>/vzconf/*.conf</tt>). Useful for determing which ve is contributing to a heavily-loaded system.<br />
If passed an argument, gives ‘<tt>w</tt>‘ output for just that ve. <br />
Ex:<br />
<pre>[root@virt2 etc]# vw<br />
134<br />
10:52pm up 79 days, 6:14, 0 users, load average: 0.02, 0.02, 0.00<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT<br />
16027<br />
2:52pm up 7 days, 19:54, 0 users, load average: 0.00, 0.00, 0.00<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT<br />
16055<br />
2:52pm up 79 days, 6:38, 0 users, load average: 0.00, 0.04, 0.07<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT</pre><br />
<br />
== vwe ==<br />
vwe [constraint]<br />
just like <tt>vw</tt>, but takes a constraint as an argument, only show’s ve’s with loads >= the constraint provided. If no constraint is provided, 1 is used by default<br />
<br />
== vzs ==<br />
vzs [veid]<br />
displays the beancounter status for all ve’s, or a particular ve if an argument is passed<br />
<br />
== ve ==<br />
ve <veid><br />
the equivalent of: <tt>vzctl enter <veid></tt><br />
<br />
== vx ==<br />
vx <veid> <command><br />
the equivalent of: <tt>/usr/sbin/vzctl exec <veid> <command></tt><br />
<br />
== dprocs ==<br />
dprocs [count]<br />
a script which outputs a continuous report (or a certain number of reports if an option is passed) of processes stuck in the D state and which VPS’s those procs belong to.<br />
<br />
== setmem ==<br />
setmem <VEID> <256|512|768|1024|2048><br />
adjusts the memory resources for the VE<br />
<br />
== afacheck.sh ==<br />
afacheck.sh<br />
displays the health/status of containers and mirrors on an adaptec card (currently quar1, tempvirt1-2, virt9, virt10)- all other are LSI<br />
<br />
== backup ==<br />
backup<br />
backup script called nightly to update virt scripts and do customer backups<br />
<br />
== checkload.pl ==<br />
checkload.pl<br />
this was intended to be setup as a cronjob to watch processes on a virt when the load <br />
rises above a certain level. Not currently in use.<br />
<br />
== findbackuppigs.pl ==<br />
findbackuppigs.pl<br />
looks for files larger than 50MB which customers have asked us to backup. Emails matches<br />
to linux@johncompanies.com<br />
<br />
== gatherlinux.pl ==<br />
gatherlinux.pl<br />
gathers up data about ve’s configured and writes to a file. Used for audits against the db<br />
<br />
== gb ==<br />
gb <search><br />
greps backup.config for the given search parameter<br />
<br />
== gbg ==<br />
gbg <search><br />
greps backup.config for the given search parameter and presents just the directories (for clean pasting)<br />
<br />
== linuxtrafficgather.pl ==<br />
linuxtrafficgather.pl [yy-mm]<br />
sends a traffic usage summary by ve to support@johncomapnies.com and payments@johncompanies.com.<br />
Optional arguments are year and month (must be in the past). If not passed, assumes last month. Relies on <br />
traffic logs created by netstatreset and netstatbackup<br />
<br />
== linuxtrafficwatch.pl ==<br />
linuxtrafficwatch.pl<br />
checks traffic usage and emails support@johncomapnies.com when a ve reaches the warning level (35G) and<br />
the limit (40G). Works on virtuozzo versions <= 2.6.x. We really aren’t using this anymore now that we have netflow.<br />
<br />
== linuxtrafficwatch2.pl ==<br />
linuxtrafficwatch2.pl<br />
checks traffic usage and emails support@johncomapnies.com when a ve reaches the warning level (35G) and<br />
the limit (40G). Works on virtuozzo version 2.6.x. We really aren’t using this anymore now that we have netflow.<br />
<br />
== load.pl ==<br />
load.pl<br />
feeds info to load mrtg - executed by inetd.<br />
<br />
== mb (linux) ==<br />
mb <mount|umount><br />
(nfs) mounts and umounts dirs to backup2. Shortcuts are mbm and mbu to mount and unmount. <br />
<br />
== migrate ==<br />
migrate <ip of node migrating to> <veid> <target_veid> [target dir: vz | vz1 | vz2]<br />
this is basically a wrapper for <tt>vzmigrate</tt> – vzmigrate is a util to seamlessly move a ve from one host to another. This wrapper was written cause vz virtuozzo version 2.6 had a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables. This script mitigates that. Since it makes multiple ssh connections to the target host, it’s a good idea to put the pub key for the src system in the authorized_keys file on the target host. In addition, it emails ve owners when their migration starts and stops (if they place email addresses in a file on their system: /migrate_notify). To move everyone off a system, you’d do:<br />
for f in `vl`; do migrate <ip> $f; done<br />
<br />
== migrateonline ==<br />
migrateonline <ip of node migrating to> <veid> <target_veid> [target dir: vz | vz1 | vz2]<br />
this is the same as migrate but will migrate a ve in <tt>–online</tt> mode which means it won’t be shut down at the end of the migration. This only works when migrating ve’s between 2 machines running a 2.6 kernel (currently tempvirt1-2. virt16-19, virt12). If you get an error that the machine you’re trying to migrate to has a different CPU or features, etc, then you have to edit the file and add the –f switch to the vzmigrate line- you can basically ignore this kind of warning (but never ignore a warning about missing templates on the destination node). NOTE: This edit (if made to migrateonline) will be overwritten by the base script during each night’s backup.<br />
<br />
== netstatbackup ==<br />
DEPRECATED<br />
netstatbackup <br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.5.x. <br />
<br />
== netstatbackup2 ==<br />
DEPRECATED<br />
netstatbackup2<br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.6.x. <br />
<br />
== netstatreset ==<br />
DEPRECATED<br />
netstatreset<br />
writes traffic count data to a logfile and resets counters to 0. Works on virtuozzo versions 2.5.x <br />
<br />
== netstatreset2 ==<br />
DEPRECATED<br />
netstatreset2<br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.6.x. <br />
<br />
== orphanedbackupwatchlinux ==<br />
orphanedbackupwatchlinux <br />
looks for directories on backup2 which aren’t configured in backup.config and offers to <br />
delete them<br />
<br />
== rsync.backup (linux) ==<br />
rsync.backup<br />
does customer backups (relies on backup.config)<br />
<br />
== startvirt.pl ==<br />
startvirt.pl<br />
forks off start ve commands – keeps 6 running at a time. This is not to be used on systems where fastboot is enabled as it circumvents the benefit of the fastboot. The script will occasionally not exit gracefully and will continue to use up CPU, so it should be watched. Also, don’t exit from the script till you’re sure all ve’s are started – if you do you need to start them manually and may have to free up locks. On some systems, the startvirt script doesn’t exit cleanly and you have to ^C out of it. Be careful though- doing so can leave some VE’s in an odd bootup state and you may need to ‘vr’ them manually. You should check to see which ve’s aren’t running and/or confirm all have started when ^C’ing out of startvirt.<br />
<br />
== taskdone (linux) ==<br />
taskdone<br />
when called will email support@johncompanies.com with the hostname of the machine from which it was <br />
executed as the subject<br />
<br />
== vb (linux) ==<br />
vb<br />
the equivalent of: <tt>vi /usr/local/sbin/backup.config</tt><br />
<br />
== vemakeXX ==<br />
DEPRECATED<br />
vemakerh9 <br />
ve create script for RH9 (see vemake)<br />
<br />
vemakedebian30 <br />
ve create script for debian 3.0 (Woody) (see vemake)<br />
<br />
vemakedebian31 <br />
ve create script for debian 3.1 (Sarge) (see vemake)<br />
<br />
vemakedebian40 <br />
ve create script for debian 4.0 (Etch) (see vemake)<br />
<br />
vemakefedora, vemakefedora2, vemakefedora4, vemakefedora5, vemakefedora6, vemakefedora7<br />
ve create script for fedora core 1, 2, 4, 5, 6, 7 respectively (see vemake)<br />
<br />
vemakecentos3, vemakecentos4<br />
ve create script for centos 3, 4 respectively (see vemake)<br />
<br />
vemakesuse, vemakesuse93, vemakesuse100<br />
ve create script for suse 9.2, 9.3, 10.0 respectively (see vemake)<br />
<br />
vemakeubuntu5, vemakeubuntu606, vemakeubuntu606 vemakeubuntu704<br />
ve create script for ubuntu 5.10, 6.06, 6.10, 7.04 respectively (see vemake)<br />
<br />
== vemove ==<br />
DEPRECATED<br />
vemove <veid> <target_ip> </vz/private/123><br />
this script simplifies the old way of moving ve’s from one system to another - in short moving a ve to or from a virt running virtuozzo < 2.6.x<br />
It’s the equivalent of:<br />
<tt>tar cfpP - <veid> --ignore-failed-read | (ssh -2 -c arcfour <target_ip> "split - -b 1024m </vz/private/123>.tar" )</tt>This should only be used if migrate/vzmigrate can’t be used. <br />
<br />
== vim.watchdog ==<br />
vim.watchdog <br />
looks for and kills procs matching “vi|vim|nano|pine|elm” that are running for a long time & consuming lots of cpu. Works on virtuozzo versions 2.5.x<br />
<br />
== vim.watchdog2 ==<br />
vim.watchdog2<br />
looks for and kills procs matching “vi|vim|nano|pine|elm” that are running for a long time & consuming lots of cpu.<br />
Works on virtuozzo versions 2.6.x.<br />
<br />
== vzmigrate ==<br />
vzmigrate <target_ip> -r no <veid>:[dst veid]:[dst /vzX/private/veid]:[dst /vzX/root/veid]<br />
(this is the raw command “wrapped” by migrate/migrateonline) this will seamlessly move a ve from one host to another. The ve will run for the duration of the migration till the very end when it’s shut down, ip moved and started up on the target system. The filesystem on the src will remain. This should be watched – occasionally the move will timeout and leave the system shut down. If target private and root aren’t specified it just puts it in /vz. Only works when both systems are running virtuozzo 2.6.x<br />
<br />
== vztrafdump.sh ==<br />
DEPRECATED<br />
vztrafdump.sh<br />
writes traffic usage info by ve to a file called jc_traffic_dump in each ve’s / dir. Works on virtuozzo versions <= 2.5.x. <br />
<br />
== vztrafdump2.sh ==<br />
DEPRECATED<br />
vztrafdump2.sh<br />
writes traffic usage info by ve to a file called jc_traffic_dump in each ve’s / dir. Works on virtuozzo versions 2.6.x. <br />
<br />
== addtun ==<br />
addtun <veid><br />
Add’s tun device to ve.<br />
<br />
== bwcap ==<br />
bwcap <veid> <kbps><br />
Ex: <tt>bwcap 1234 512</tt><br />
Caps a VE’s bandwidth to the amount given<br />
<br />
== setdisk ==<br />
setdisk <veid> <diskspace in GB><br />
Ex: <tt>setdisk 1234 5</tt><br />
Gives a VE’s a given amount of disk space<br />
<br />
== vdf ==<br />
vdf <veid> <br />
the equivalent of: <tt>vzctl exec <veid> df –h</tt><br />
<br />
== vdff ==<br />
vdff<br />
runs a (condensed) vdf for all ve’s in your pwd (must be run from /vz/privateN)<br />
<br />
== mvbackups ==<br />
mvbackups <veid> <target_machine> (virt1) <target_dir> (vz1)<br />
moves backups from one location to another on the backup server, and provides you with option to remove entries from current backup.config, and simple paste command to add the config to backup.config on the target server<br />
<br />
== checkquota ==<br />
checkquota<br />
for all the ve’s in the cwd (run from /vz/private, /vz1/private, etc) reports what vz quota says they’re using and what the actual usage is (as reported by du)<br />
<br />
== clearquota ==<br />
clearquota <veid><br />
Recalculates a ve’s quota, prints out the usage before and after. The equivalent of:<br />
<tt>vdf <veid>; v stop <veid>; vzquota drop <veid>; v start <veid>; vdf <veid></tt><br />
<br />
== dprocs ==<br />
dprocs<br />
Sometimes the server’s have a large number of processes get stuck in the D state- this script shows (every 3 secs) which VE’s have D procs, which procs<br />
are stuck and a running average of the top “offenders”<br />
<br />
== vzstat ==<br />
vstat<br />
sort of like top for VZ. sort VEs by CPU usage by pressing 'o' and then 'c' keys</div>Supporthttps://wiki.jcihosting.com/index.php?title=VPS_Management&diff=1015VPS Management2013-03-11T23:03:23Z<p>Support: /* Moving a VE on the same virt */</p>
<hr />
<div>= Common Problems =<br />
== Login to any machine without a password ==<br />
<br />
This is possible via the use of ssh keys. The process is thus:<br />
<br />
1. place the public key for your user (root@mail) in the /root/.ssh/authorized_keys file on the server you wish to login to<br />
cat /root/.ssh/id_dsa.pub<br />
(paste that into authorized_keys on the target server). If the file doesn't exist, create it.<br />
<br />
2. enable root login (usually only applies to FreeBSD). Edit the /etc/ssh/sshd_config on the target server and change:<br />
<tt>#PermitRootLogin no</tt><br />
to<br />
<tt>PermitRootLogin yes</tt><br />
<br />
3. Restart the sshd on the target machine. First, find the sshd process: <br />
jailps <hostname> | grep sshd <br />
or <br />
vp <VEID> | grep sshd<br />
<br />
Look for the process resembling:<br />
root 17296 0.0 0.0 5280 1036 ? Ss 2011 4:27 /usr/sbin/sshd <br />
(this is the sshd)<br />
<br />
Not:<br />
root 6270 0.5 0.0 6808 2536 ? Ss 14:33 0:00 sshd: root [priv]<br />
(this is an sshd child- someone already ssh'd in as root)<br />
<br />
Restart the sshd: <br />
kill -1 <PID><br />
<br />
Ex:<br />
kill -1 17296<br />
<br />
You may now ssh in.<br />
<br />
Once you're done, IF you enabled root login, you should repeat steps 2 and 3 to disable root logins.<br />
<br />
== Letting someone in who has locked themselves out (killed sshd, lost pwd) ==<br />
<br />
There are two ways people frequently lock themselves out - either they forget a password, or they kill off sshd somehow.<br />
<br />
These are actually both fairly easy to solve. First, let's say someone kills off their sshd, or somehow mangles /etc/ssh/sshd_config such that it no longer lets them in.<br />
<br />
Their email may be very short, or it may have all sorts of details about how you should fix sshd_config to let them in ... just ignore all of this. They can fix their own mangled sshd. Fixing this is very simple. First, edit the /etc/inetd.conf on their system and uncomment the telnet line:<br />
<br />
telnet stream tcp nowait root /usr/libexec/telnetd telnetd<br />
#telnet stream tcp6 nowait root /usr/libexec/telnetd telnetd<br />
<br />
(just leave the tcp6 version of telnet commented)<br />
<br />
Then, use jailps to list the processes on their system, and find their inetd process. Then simply:<br />
<br />
kill -HUP (pid)<br />
<br />
where (pid) is the PID of their inetd process. Now they have telnet running on their system and they can log in and do whatever they need to do.<br />
<br />
The only complications that could occur are:<br />
<br />
a) their firewall config on our firewall has port 23 blocked, in which case you will need to open that - will be covered in a different lesson.<br />
<br />
b) they are not running inetd, so you can't HUP it. If this happens, edit their /etc/rc.conf, add the inetd_enable="YES" line, and then kill<br />
their jail with /tmp/jailkill.pl - then restart their jail with the jail line from their quad/safe file. Easy.<br />
<br />
If they have forgotten a password,<br />
<br />
On 6.x+ you can reset their password with:<br />
jexec <jailID from jls> passwd root<br />
<br />
Note: the default password for 6.x jails is 8ico2987, for 4.x it is p455agfa<br />
<br />
On 4.x, you need to cd to their etc directory<br />
... for instance:<br />
<br />
cd /mnt/data2/198.78.65.136-col00261-DIR/etc<br />
<br />
and run:<br />
<br />
vipw -d .<br />
<br />
Then paste in these two lines (theres a paste with these):<br />
<br />
root:$1$krszPxhk$xkCepSnz3mIikT3vCtJCt0:0:0::0:0:Charlie &:/root:/bin/csh<br />
user:$1$Mx9p5Npk$QdMU6c8YQqp2FW2M3irEh/:1001:1001::0:0:User &:/home/user:/bin/sh<br />
<br />
overwriting the lines they already have for "user" and "root" - then just tell them that both user and root have been reset to the default password of p455agfa.<br />
<br />
For linux, just passwd inside shell or <br />
vzctl set <veid> --userpasswd root:p455agfa –save<br />
<br />
Starting in 2009 we began giving out randomized passwords for FreeBSD and Linux as the default password. That is stored with each system in Mgmt. You should look for and reset the password to that password in the event of a reset and refer the customer to use their original password from their welcome email- this way we don’t have to send the password again via email (in clear text).<br />
<br />
<br />
== sendmail can’t be contacted from ext ip (only locally) ==<br />
<br />
By default redhat puts this line in sendmail.mc:<br />
<br />
DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')<br />
<br />
which makes it only answer on localhost. Comment it out like:<br />
<br />
dnl DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')<br />
<br />
and then rebuild sendmail.cf with:<br />
<br />
m4 /etc/mail/sendmail.mc > /etc/sendmail.cf<br />
<br />
== virt doesn’t properly let go of ve’s ip(s) when moved to another system ==<br />
<br />
On virtuozzo 2.6 systems, it's been observed that when moving ips from one virt to another that sometimes the routing table will not get updated to reflect the removal of the ip addresses.<br />
<br />
A recent example was a customer that was moving to a new ve on a new virt and the ip addresses were traded between the two ve's. After the trade the two systems were not able to talk to each other. When looking at the routing table for the old system all the ip addresses were still in the routing table as being local, like so:<br />
<br />
<pre>netstat -rn | grep 69.55.225.149<br />
69.55.225.149 0.0.0.0 255.255.255.255 UH 40 0 0 venet0</pre><br />
<br />
This was preventing traffic to the other system from being routed properly.<br />
The solution is to manually delete the route:<br />
<br />
route delete 69.55.225.149 gw 0.0.0.0<br />
<br />
Supposedly, this was fixed in 2.6.1<br />
<br />
== sshd on FreeBSD 6.2 segfaults ==<br />
<br />
First try to reinstall ssh<br />
<br />
<pre>cd /usr/src/secure<br />
cd lib/libssh<br />
make depend && make all install<br />
cd ../../usr.sbin/sshd<br />
make depend && make all install<br />
cd ../../usr.bin/ssh<br />
make depend && make all install</pre><br />
<br />
Failing that, find the library that’s messed up:<br />
<br />
<pre>ldd /usr/sbin/sshd<br />
libssh.so.3 => /usr/lib/libssh.so.3 (0x280a3000) <br />
libutil.so.5 => /lib/libutil.so.5 (0x280d8000) <br />
libz.so.3 => /lib/libz.so.3 (0x280e4000) <br />
libwrap.so.4 => /usr/lib/libwrap.so.4 (0x280f5000) <br />
libpam.so.3 => /usr/lib/libpam.so.3 (0x280fc000) <br />
libbsm.so.1 => /usr/lib/libbsm.so.1 (0x28103000) <br />
libgssapi.so.8 => /usr/lib/libgssapi.so.8 (0x28112000) <br />
libkrb5.so.8 => /usr/lib/libkrb5.so.8 (0x28120000) <br />
libasn1.so.8 => /usr/lib/libasn1.so.8 (0x28154000) <br />
libcom_err.so.3 => /usr/lib/libcom_err.so.3 (0x28175000) <br />
libroken.so.8 => /usr/lib/libroken.so.8 (0x28177000) <br />
libcrypto.so.4 => /lib/libcrypto.so.4 (0x28183000) <br />
libcrypt.so.3 => /lib/libcrypt.so.3 (0x28276000) <br />
libc.so.6 => /lib/libc.so.6 (0x2828e000) <br />
libmd.so.3 => /lib/libmd.so.3 (0x28373000)</pre><br />
<br />
md5 them and compare to other jail hosts or jails running on host<br />
<br />
for libcrypto reinstall:<br />
<pre>/usr/src/crypto<br />
make depend && make all install</pre><br />
<br />
= FreeBSD VPS =<br />
<br />
== Starting jails: Quad/Safe Files ==<br />
<br />
FreeBSD customer systems do not start up automatically at boot time. When one of our freebsd machines boots up, it boots up, and does nothing else. To start jails, we put the commands to start each jail into a shell script(s) and run the script(s). Jail startup is something that needs to be actively monitored, which is why we don’t just run the script automatically. More on monitoring later.<br />
<br />
NOTE: >=7.x we have moved to 1 quad file: <tt>quad1</tt>. Startups are not done by running each quad, but rather [[#startalljails|startalljails]] which relies on the contents of <tt>quad1</tt>. The specifics of this are lower in this article. What follows here applies for pre 7.x systems.<br />
<br />
There are eight files in <tt>/usr/local/jail/rc.d</tt>:<br />
<br />
<pre>jail3# ls /usr/local/jail/rc.d/<br />
quad1 quad2 quad3 quad4 safe1 safe2 safe3 safe4<br />
jail3#</pre><br />
<br />
four quad files and four safe files.<br />
<br />
Each file contains an even number of system startup blocks (total number of jails divided by 4)<br />
<br />
The reason for this is, if we make one large script to startup all the systems at boot time, it will take too long - the first system in the script will start up right after system boot, which is great, but the last system may not start for another 20 minutes.<br />
<br />
Since there is no way to parralelize this during the startup procedure, we simply open four terminals (in screen window 9) and run each script, one in each terminal. This way they all run simultaneously, and the very last system in each startup script gets started in 1/4th the time it would if there was one large file<br />
<br />
The files are generally organized so that quad/safe 1&2 have only jails from disk 1, and quad/safe 3&4 have jails from disk 2. This helps ensure that only 2 fscks on any disk are going on at once. Further, they are balanced so that all quad/safe’s finish executing around the same time. We do this by making sure each quad/safe has a similar number of jails and represents a similar number of inodes (see js).<br />
<br />
The other, very important reason we do it this way, and this is the reason there are quad files and safe files, is that in the event of a system crash, every single vn-backed filesystem that was mounted at the time of system crash needs to be fsck'd. However, fsck'ing takes time, so if we shut the system down gracefully, we don't want to fsck.<br />
<br />
Therefore, we have two sets of scripts - the four quad scripts are identical to the four safe scripts except for the fact that the quad scripts contain fsck commands for each filesystem.<br />
<br />
So, if you shut a system down gracefully, start four terminals and run safe1 in window one, and safe2 in window 2, and so on.<br />
<br />
If you crash, start four terminals (or go to screen window 9) and run quad1 in window one, and quad2 in window 2, and so on.<br />
<br />
Here is a snip of (a 4.x version) quad2 from jail17:<br />
<br />
<pre>vnconfig /dev/vn16 /mnt/data2/69.55.228.7-col00820<br />
fsck -y /dev/vn16<br />
mount /dev/vn16c /mnt/data2/69.55.228.7-col00820-DIR<br />
chmod 0666 /mnt/data2/69.55.228.7-col00820-DIR/dev/null<br />
jail /mnt/data2/69.55.228.7-col00820-DIR mail1.phimail.com 69.55.228.7 /bin/sh /etc/rc<br />
<br />
# moved to data2 col00368<br />
#vnconfig /dev/vn28 /mnt/data2/69.55.236.132-col00368<br />
#fsck -y /dev/vn28<br />
#mount /dev/vn28c /mnt/data2/69.55.236.132-col00368-DIR<br />
#chmod 0666 /mnt/data2/69.55.236.132-col00368-DIR/dev/null<br />
#jail /mnt/data2/69.55.236.132-col00368-DIR limehouse.org 69.55.236.132 /bin/sh /etc/rc<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
vnconfig /dev/vn22 /mnt/data2/69.55.228.13-col01063<br />
fsck -y /dev/vn22<br />
mount /dev/vn22c /mnt/data2/69.55.228.13-col01063-DIR<br />
chmod 0666 /mnt/data2/69.55.228.13-col01063-DIR/dev/null<br />
jail /mnt/data2/69.55.228.13-col01063-DIR www.widestream.com.au 69.55.228.13 /bin/sh /etc/rc<br />
<br />
# cancelled col00106<br />
#vnconfig /dev/vn15 /mnt/data2/69.55.238.5-col00106<br />
#fsck -y /dev/vn15<br />
#mount /dev/vn15c /mnt/data2/69.55.238.5-col00106-DIR<br />
#chmod 0666 /mnt/data2/69.55.238.5-col00106-DIR/dev/null<br />
#jail /mnt/data2/69.55.238.5-col00106-DIR mail.azebu.net 69.55.238.5 /bin/sh /etc/rc</pre><br />
<br />
As you can see, two of the systems specified are commented out - presumably those customers cancelled, or were moved to new servers.<br />
<br />
As you can see, the vnconfig line is the simpler command line, not the longer one that was used when it was first configured. As you can see, all that is done is, vnconfig the filesystem, then fsck it, then mount it. The fourth command is the `jail` command used to start the system – but that will be covered later.<br />
<br />
Here is the safe2 file from jail17:<br />
<br />
<pre>vnconfig /dev/vn16 /mnt/data2/69.55.228.7-col00820<br />
mount /dev/vn16c /mnt/data2/69.55.228.7-col00820-DIR<br />
chmod 0666 /mnt/data2/69.55.228.7-col00820-DIR/dev/null<br />
jail /mnt/data2/69.55.228.7-col00820-DIR mail1.phimail.com 69.55.228.7 /bin/sh /etc/rc<br />
<br />
# moved to data2 col00368<br />
#vnconfig /dev/vn28 /mnt/data2/69.55.236.132-col00368<br />
#mount /dev/vn28c /mnt/data2/69.55.236.132-col00368-DIR<br />
#chmod 0666 /mnt/data2/69.55.236.132-col00368-DIR/dev/null<br />
#jail /mnt/data2/69.55.236.132-col00368-DIR limehouse.org 69.55.236.132 /bin/sh /etc/rc<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
vnconfig /dev/vn22 /mnt/data2/69.55.228.13-col01063<br />
mount /dev/vn22c /mnt/data2/69.55.228.13-col01063-DIR<br />
chmod 0666 /mnt/data2/69.55.228.13-col01063-DIR/dev/null<br />
jail /mnt/data2/69.55.228.13-col01063-DIR www.widestream.com.au 69.55.228.13 /bin/sh /etc/rc<br />
<br />
# cancelled col00106<br />
#vnconfig /dev/vn15 /mnt/data2/69.55.238.5-col00106<br />
#mount /dev/vn15c /mnt/data2/69.55.238.5-col00106-DIR<br />
#chmod 0666 /mnt/data2/69.55.238.5-col00106-DIR/dev/null<br />
#jail /mnt/data2/69.55.238.5-col00106-DIR mail.azebu.net 69.55.238.5 /bin/sh /etc/rc</pre><br />
<br />
As you can see, it is exactly the same, but it does not have the fsck lines.<br />
<br />
Take a look at the last entry - note that the file is named:<br />
<br />
/mnt/data2/69.55.238.5-col00106<br />
<br />
and the mount point is named:<br />
<br />
/mnt/data2/69.55.238.5-col00106-DIR<br />
<br />
This is the general format on all the FreeBSD systems. The file is always named:<br />
<br />
IP-custnumber<br />
<br />
and the directory is named:<br />
<br />
IP-custnumber-DIR<br />
<br />
If you run safe when you need a fsck, the mount will fail and jail will fail:<br />
<br />
# mount /dev/vn1c /mnt/data2/jails/65.248.2.131-ns1.kozubik.com-DIR<br />
mount: /dev/vn1c: Operation not permitted<br />
<br />
No reboot needed, just run the quad script<br />
<br />
Starting with 6.x jails, we added block delimiters to the quad/safe files, the block looks like:<br />
<br />
<pre>echo '## begin ##: nuie.solaris.mu'<br />
fsck -y /dev/concat/v30v31a<br />
mount /dev/concat/v30v31a /mnt/data1/69.55.228.218-col01441-DIR<br />
mount_devfs devfs /mnt/data1/69.55.228.218-col01441-DIR/dev<br />
devfs -m /mnt/data1/69.55.228.218-col01441-DIR/dev rule -s 3 applyset<br />
jail /mnt/data1/69.55.228.218-col01441-DIR nuie.solaris.mu 69.55.228.218 /bin/sh /etc/rc<br />
echo '## end ##: nuie.solaris.mu'</pre><br />
<br />
These are more than just informative when running quad/safe’s, the echo lines MUST be present for certain tools to work properly. So it’s important that any updates to the hostname also be updated on the 2 echo lines. For example, if you try to startjail a jail with a hostname which is on the jail line but not the echo lines, the command will return with host not found.<br />
<br />
=== FreeBSD 7.x+ notes ===<br />
<br />
Starting with the release of FreeBSD 7.x, we are doing jail startups in a slightly different way. First, thereis only 1 file: <tt>/usr/local/jail/rc.d/quad1</tt><br />
There are no other quads or corresponding safe files. The reason for this is twofold, 1. We can pass –C to fsck which will tell is to skip the fsck if the fs is clean (no more need for safe files), 2. We have a new startup script which can be launched multiple times, running in parallel to start jails, where quad1 is the master jail file. <br />
Quad1 could still be run as a shell script, but it would take a very long time for it to run completely so it’s not advisable; or you should break it down into smaller chunks (like quad1, quad2, quad3, etc)<br />
<br />
Here is a snip of (a 7.x version) quad1 from jail2:<br />
<br />
<pre>echo '## begin ##: projects.tw.com'<br />
mdconfig -a -t vnode -f /mnt/data1/69.55.230.46-col01213 -u 50<br />
fsck -Cy /dev/md50c<br />
mount /dev/md50c /mnt/data1/69.55.230.46-col01213-DIR<br />
mount -t devfs devfs /mnt/data1/69.55.230.46-col01213-DIR/dev<br />
devfs -m /mnt/data1/69.55.230.46-col01213-DIR/dev rule -s 3 applyset<br />
jail /mnt/data1/69.55.230.46-col01213-DIR projects.tw.com 69.55.230.46 /bin/sh /etc/rc<br />
echo '## end ##: projects.tw.com'</pre><br />
<br />
Cancelled jails are no longer commented out and stored in quad1, rather they’re moved to <tt>/usr/local/jail/rc.d/deprecated</tt> <br />
<br />
To start these jails, start the 4 ssh sessions as you would for a normal crash and then instead of running quad1-4, instead run startalljails in each window. IMPORTANT- before running startalljails you should make sure you ran preboot once as it will clear out all the lockfiles and enable startalljails to work properly.<br />
<br />
== Problems with the quad/safe files ==<br />
<br />
When you run the quad/safe files, there are two problems that can occur - either a particular system will hang during initialization, OR a system will spit out output to the screen, impeding your ability to do anything. Or both.<br />
<br />
First off, when you start a jail, you see output like this:<br />
<br />
<pre>Skipping disk checks ...<br />
adjkerntz[25285]: sysctl(put_wallclock): Operation not permitted<br />
Doing initial network setup:.<br />
ifconfig: ioctl (SIOCDIFADDR): permission denied<br />
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384<br />
Additional routing options: TCP keepalive=YESsysctl:<br />
net.inet.tcp.always_keepalive: Operation not permitted.<br />
Routing daemons:.<br />
Additional daemons: syslogd.<br />
Doing additional network setup:.<br />
Starting final network daemons:.<br />
ELF ldconfig path: /usr/lib /usr/lib/compat /usr/X11R6/lib /usr/local/lib<br />
a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout /usr/X11R6/lib/aout<br />
Starting standard daemons: inetd cron sshd sendmail sendmail-clientmqueue.<br />
Initial rc.i386 initialization:.<br />
Configuring syscons: blanktime.<br />
Additional ABI support:.<br />
Local package initialization:.<br />
Additional TCP options:.</pre><br />
<br />
Now, let's look at this line, near the end:<br />
<br />
Local package initialization:.<br />
<br />
This is where a list of daemons that are set to start at boot time willshow up. You might see something like:<br />
<br />
Local package initialization: mysqld apache sendmail sendmail-clientmqueue<br />
<br />
Or something like this:<br />
<br />
Local package initialization: postgres postfix apache<br />
<br />
The problem is that many systems (about 4-5 per machine) will hang on that line. Basically it will get to some of the way through the total daemons to be started:<br />
<br />
Local package initialization: mysqld apache<br />
<br />
and will just sit there. Forever.<br />
<br />
Fortunately, pressing ctrl-c will break out of it. Not only will it break out of it, but it will also continue on that same line and start the other daemons:<br />
<br />
Local package initialization: mysqld apache ^c sendmail-clientmqueue<br />
<br />
and then continue on to finish the startup, and then move to the next system to be started.<br />
<br />
So what does this mean? It means that if a machine crashes, and you start four screen-windows to run four quads or four safes, you need to periodically cycle between them and see if any systems are stuck at that point, causing their quad/safe file to hang. A good rule of thumb is, if you see a system at that point in the startup, give it another 100 seconds - if it is still at the exact same spot, hit ctrl-c. Its also a good idea to go back into the quad file (just before the first command in the jail startup block) and note that this jail tends to need a control-c or more time as follows:<br />
<pre>echo '### NOTE ### slow sendmail'<br />
echo '### NOTE ###: ^C @ Starting sendmail.'</pre><br />
<br />
'''NEVER''' hit ctrl-c repeatedly if you don't get an immediate response - that will cause the following jail’s startup commands to be aborted.<br />
<br />
A second problem that can occur is that a jail - maybe the first one in that particular quad/safe, maybe the last one, or maybe one in the middle, will start spitting out status or error messages from one of its init scripts. This is not a problem - basically, hit enter a few times and see if you get a prompt - if you do get a prompt, that means that the quad/safe script has already completed. Therefore it is safe to log out (and log out of the user that you su'd from) and then log back in (if necessary).<br />
<br />
The tricky thing is, if a system in the middle starts flooding with messages, and you hit enter a few times and don't get a prompt. Are you not getting a prompt because some subsequent system is hanging at the initialization, as we discussede above ? Or are you not getting a prompt because that quad file is currently running an fsck ? Usually you can tell by scrolling back in screen’s history to see what it was doing before you started getting the messages.<br />
<br />
If you don’t get clues from history, you have to use your judgement - instead of giving it 100 seconds to respond, perhaps give it 2-3 mins ... if you still get no response (no prompt) when you hit enter, hit ctrl-c. However, be aware that you might still be hitting ctrl-c in the middle of an fsck. This means you will get an error like "filesystem still marked dirty" and then the vnconfig for it will fail and so will the jail command, and the next system in the quad file will then start starting up.<br />
<br />
If this happens, just wait until the end of all the quad files have finished, and start that system manually.<br />
<br />
If things really get weird, like a screen flooded with errors, and you can't get a prompt, and ctrl-c does nothing, then you need to just eventually (give it ten mins or so) just kill that window with ctrl-p, then k, and then log in again and manually check which systems are now running and which aren't, and manually start up any that are not.<br />
<br />
Don't EVER risk running a particular quad/safe file a second time.<br />
If the quad/safe script gets executed twice, reboot the machine immediately.<br />
<br />
So, for all the above reasons, anytime a machine crashes and you run all the quads or all the safes, '''always''' check every jail afterwards to make sure it is running - even if you have no hangs or complications at all.<br />
Run this command:<br />
<br />
<tt>[[#jailpsall|jailpsall]]</tt> <br />
<br />
Note: [[#postboot|postboot]] also populates ipfw counts, so it '''should not be run multiple times''', use <tt>jailpsall</tt> for subsequent extensive ps’ing<br />
<br />
And make sure they all show as running. If one does not show as running, check its /etc/rc.conf file first to see if maybe it is using a different hostname first before starting it manually.<br />
<br />
One thing we have implemented to alleviate these startup hangs and noisy jails, is to put jail start blocks that are slow or hangy at the bottom of the safe/quad file. Further, for each bad jail we note in each quad/safe just before the start block something like:<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
<br />
That way we’ll be prepared to ^C when we see that message appear during the quad/safe startup process. If you observe a new, undocumented hang, '''after''' the quad/safe has finished, place a line similar to the above in the quad file, move the jail start block to the end of the file, then run [[#buildsafe|buildsafe]]<br />
<br />
<br />
== Recovering from a crash (FreeBSD) ==<br />
<br />
=== Diagnose whether you have a crash ===<br />
The most important thing is to get the machine and all jails back up as soon as possible. Note the time, you’ll need to create a crash log entry (Mgmt. -> Reference -> CrashLog). The first thing to do is head over to the [[Screen#Screen_Organization|serial console screen]] and see if there’s any kernel error messages output. Try to copy any messages (or just a sample of repeating messages) you see into the notes section of the crash log. If there are no messages, the machine may just be really busy- wait a bit (5-10min) to see if it comes back. If it's still pinging, odds are its very busy. Note, if you see messages about swap space exhausted, the server is obviously out of memory, however it may recover briefly enough for you to get a jtop in to see who's lauched a ton of procs (most likely) and then issue a quick jailkill to get it back under control.<br />
<br />
If it doesn't come back, or the messages indicate a fatal error, you will need to proceed with a power cycle (ctrl+alt+del will not work).<br />
<br />
=== Power cycle the server ===<br />
If this machine is not a Dell 2950 with a [[DRAC/RMM#DRAC|DRAC card]] (i.e. if you can’t ssh into the DRAC card (as root, using the standard root pass) and issue <br />
racadm serveraction hardreset<br />
then you will need someone at the data center power the macine off, wait 30 sec, then turn it back on. Make sure to re-attach via console:<br />
tip jailX<br />
immediately after power down. <br />
<br />
=== (Re)attach to the console ===<br />
Stay on the console the entire time during boot. As the BIOS posts- look out for the RAID card output- does everything look healthy? The output may be scrambled, look for "DEGRADED" or "FAILED". Once the OS starts booting you will be disconnected (dropped back to the shell on the console server) a couple times during the boot up. The reason you want to quickly re-attach is two-fold: 1. If you don’t reattach quickly then you won’t get any console output, 2. you want to be attached before the server ''potentially'' starts (an extensive) fsck. If you attach after the fsck begins, you’ll have seen no indication it started an fsck and the server will appear frozen during startup- no output, no response. <br />
<br />
IMPORTANT NOTE: on some older FreeBSD systems, there will be no output to the video (KVM) console as it boots up. The console output is redirected to the serial port ... so if a jail crashes, and you attach a kvm, the output during the bootup procedure will not be shown on the screen. However, when the bootup is done, you will get a login prompt on the screen and will be able to log in as normal. <tt>/boot/loader.conf</tt> is where serial console redirect output lives, so comment that if you want to catch output on kvm.<br />
On newer systems it sends most output to both locations. <br />
<br />
=== Assess the heath of the server ===<br />
Once the server boots up fully, you should be able to ssh in. Look around- make sure all the mounts are there and reporting the correct size/usage (i.e. /mnt/data1 /mnt/data2 /mnt/data3 - look in /etc/fstab to determine which mount points should be there), check to see if RAID mirrors are healthy. See [[RAID_Cards#Common_CLI_commands_.28megacli.29|megacli]], [[#aaccheck|aaccheck]]<br />
<br />
Before you start the jails, you need to run [[#preboot|preboot]]. This will do some assurance checks to make sure things are prepped to start the jails. Any issues that come out of preboot need to be addressed before starting jails.<br />
<br />
=== Start jails ===<br />
[[#Starting_jails:_Quad.2FSafe_Files|More on starting jails]]<br />
Customer jails (the VPSs) do not start up automatically at boot time. When a FreeBSD machines boots up, it boots up, and does nothing else. To start jails, we put the commands to start each jail into a shell script(s) and run the script(s). Jail startup is something that needs to be actively monitored, which is why we don’t just run the script automatically. <br />
<br />
In order to start jails, we run the quad files: quad1 quad2 quad3 and quad4 (on new systems there is only quad1). If the machine was cleanly rebooted- which wouldn't be the case if this was a crash, you may run the safe files (safe1 safe2 safe3 safe4) in lieu of quads. <br />
<br />
Open up 4 logins to the server (use the windows in [[Screen#Screen_Organization|a9]])<br />
In each of the 4 windows you will:<br />
<br />
If there is a [[#startalljails|startalljails]] script (and only quad1), run that command in each of the 4 windows. It will parse through the quad1 file and start each jail. Follow the instructions [[#Problems_with_the_quad.2Fsafe_files|here]] for monitoring startup. Note that you can be a little more lenient with jails that take awhile to start- startalljails will work around the slow jails and start the rest. As long as there aren't 4 jails which are "hung" during startup, the rest will get started eventually.<br />
-or-<br />
If there is no startalljails script, there will be multiple quad files. In each of the 4 windows, start each of the quads. i.e. start quad1 in window1, quad2 in window2 and so on. DO NOT start any quad twice. It will crash the server. If you accidentally do this, just jailkill all the jails which are in the quad and run the quad again. Follow the instructions here for monitoring quad startup.<br />
<br />
Note the time the last jail boots- this is what you will enter in the crash log.<br />
<br />
Save the crash log.<br />
<br />
=== Check to make sure all jails have started ===<br />
There's a simple script which will make sure all jails have started, and enter the ipfw counter rules: [[#postboot|postboot]] <br />
Run postboot, which will do a jailps on each jail it finds (excluding commented out jails) in the quad file(s). We're looking for 2 things:<br />
# systems spawning out of control or too many procs<br />
# jails which haven't started<br />
On 7.x and newer systems it will print out the problems (which jails haven't started) at the conclusion of postboot. <br />
On older systems you will need to watch closely to see if/when there's a problem, namely:<br />
<br />
[hostname] doesnt exist on this server<br />
<br />
When you get this message, it means one of 2 things:<br />
1. the jail really didn't start:<br />
When a jail doesn't start it usually boils down to a problem in the quad file. Perhaps the path name is wrong (data1 vs data2) or the name of the vn/mdfile is wrong. Once this is corrected, you will need to run the commands from the quad file manually, or you may use <tt>startjail <hostname></tt><br />
<br />
2. the customer has changed their hostname (and not told us) so their jail ''is'' running, just under a different hostname:<br />
On systems with jls, this is easy to rectify. First, get the customer info: <tt>g <hostname></tt><br />
Then look for the customer in jls: <tt>jls | grep <col0XXXX></tt><br />
From there you will see their new hostname- you should update that hostname in the quad file: don't forget to edit it on the <tt>## begin ##</tt> and <tt>## end ##</tt> lines, and in mgmt. <br />
On older systems without jls, this will be harder, you will need to look further to see their hostname- perhaps its in their /etc/rc.conf<br />
<br />
<br />
Once all jails are started, do some spot checks- try to ssh or browse to some customers, just to make sure things are really ok.<br />
<br />
== Adding disk to a 7.x/8.x jail ==<br />
or<br />
== Moving customer to a different drive (md) ==<br />
<br />
NOTE: this doesn’t apply to mx2 which uses gvinum. Use same procedure as 6.x<br />
NOTE: if you unmount before mdconfig, re-mdconfig (attach) then unmount then mdconfig -u again <br />
<br />
<br />
(parts to change/customize are <tt><span style="color:red">red</span></tt>)<br />
<br />
<br />
If someone wants more disk space, there’s a paste for it, send it to them (explains about downtime, etc).<br />
<br />
1. Figure out the space avail from <tt>js</tt>. Ideally, you want to put the customers new space on a different partition (and create the new md on the new partition). <br />
<br />
<br />
2. make a mental note of how much space they're currently using<br />
<br />
<br />
3. <tt>jailkill <hostname></tt> <br />
<br />
<br />
4. Umount it (including their devfs) but leave the md config’d (so if you use stopjail, you will have to re-mdconfig it)<br />
<br />
<br />
5. <tt>g <customerID></tt> to get the info (IP/cust#) needed to feed the new mdfile and mount name, and to see the current md device. <br />
<br />
<br />
6a. When there's enough room to place new system on an alternate, or the same drive:<br />
USE CAUTION not to overwrite (touch, mdconfig) existing md!!<br><br />
<tt>touch /mnt/data<span style="color:red">3/69.55.234.66-col01334</span><br><br />
mdconfig -a -t vnode -s 10g -f /mnt/data3/69.55.234.66-col01334 -u 97<br><br />
newfs /dev/md97</tt><br />
<br />
<br />
Optional- if new space is on a different drive, move the mount point directory AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
<tt>mount /dev/md<span style="color:red">97</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<br />
confirm you are mounted to /dev/md<span style="color:red">97</span> and space is correct:<br><br />
<tt>df .</tt> <br />
<br />
<br />
do the dump and pipe directly to restore:<br><br />
<tt>dump -0a -f - /dev/md<span style="color:red">1</span> | restore -r -f - <br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use <tt>df</tt> to confirm the restored data size matches the original usage figure<br />
<br />
<br />
md-unconfig old system:<br><br />
<tt>mdconfig -d -u <span style="color:red">1</span></tt><br />
<br />
<br />
archive old mdfile. <br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.237.26-col00241</span> /mnt/data<span style="color:red">1/old-col00241-mdfile-noarchive-20091211</span></tt><br />
<br />
<br />
edit quad (vq1) to point to new (/mnt/data<span style="color:red">3</span>) location AND new md number (md<span style="color:red">97</span>)<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
6b. When there's not enough room on an alternate partition, or on the same drive...but there is enough room if you were to remove the existing customer's space:<br />
<br />
<br />
mount backup nfs mounts:<br><br />
<tt>mbm</tt><br> <br />
(run <tt>df</tt> to confirm backup mounts are mounted)<br />
<br />
<br />
dump the customer to backup2 or backup1<br><br />
<tt>dump -0a -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> /dev/md<span style="color:red">1</span></tt><br><br />
(when complete WITHOUT errors, <tt>du</tt> the dump file to confirm it matches size, roughly, with usage)<br />
<br />
<br />
unconfigure and remove old mdfile<br><br />
<tt>mdconfig -d -u <span style="color:red">1</span><br><br />
rm /mnt/data<span style="color:red">1/69.55.237.26-col00241</span></tt><br><br />
(there should now be enough space to recreate your bigger system. If not, run sync a couple times)<br />
<br />
<br />
create the new system (ok to reuse old mdfile and md#):<br><br />
<tt>touch /mnt/data<span style="color:red">1/69.55.234.66-col01334</span><br><br />
mdconfig -a -t vnode -s <span style="color:red">10</span>g -f /mnt/data<span style="color:red">1/69.55.234.66-col01334</span> -u <span style="color:red">1</span><br><br />
newfs /dev/md<span style="color:red">1</span><br><br />
mount /dev/md<span style="color:red">1</span> /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> <br><br />
cd /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<br />
confirm you are mounted to /dev/md<span style="color:red">1</span> and space is correct:<br><br />
<tt>df .</tt> <br />
<br />
<br />
do the restore from the dumpfile on the backup server:<br><br />
<tt>restore -r -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> .<br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
umount nfs:<br><br />
<tt>mbu</tt><br />
<br />
<br />
If md# changed (or mount point), edit quad (<tt>vq1</tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new md number (md<span style="color:red">1</span>)<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
7. update disk (and dir if applicable) in mgmt screen<br />
<br />
<br />
8. update backup list AND move backups, if applicatble<br />
<br />
Ex: <tt>mvbackups <span style="color:red">69.55.237.26-col00241</span> jail<span style="color:red">9</span> data<span style="color:red">3</span></tt><br />
<br />
<br />
9. Optional: archive old mdfile<br />
<br />
<tt>mbm<br><br />
gzip -c old-col01588-mdfile-noarchive-20120329 > /deprecated/old-col01588-mdfile-noarchive-20120329.gz<br><br />
mbu<br><br />
rm old-col01588-mdfile-noarchive-20120329</tt><br />
<br />
== Adding disk to a 6.x jail (gvinum/gconcat) ==<br />
or<br />
== Moving customer to a different drive (gvinum/gconcat) ==<br />
<br />
(parts to change are <span style="color:red">highlighted</span>)<br />
<br />
<br />
If someone wants more disk space, there’s a paste for it, send it to them (explains about downtime, etc).<br />
<br />
<br />
1. Figure out the space avail from [[#js|js]]. Ideally, you want to put the customers new space on a different partition (and create the new md on the new partition). <br />
<br />
<br />
2. make a mental note of how much space they're currently using<br />
<br />
<br />
3. <tt>[[#stopjail|stopjail]] <hostname></tt> <br />
<br />
<br />
4. <tt>[[#g|g]] <customerID></tt> to get the info (IP/cust#) needed to feed the new mount name and existing volume/device. <br />
<br />
<br />
5a. When there's enough room to place new system on an alternate, or the same drive (using only UNUSED - including if it's in use by the system in question - gvinum volumes):<br />
<br />
<br />
Configure the new device:<br><br />
A. for a 2G system (single gvinum volume):<br><br />
<tt>bsdlabel -r -w /dev/gvinum/v<span style="color:red">123</span><br><br />
newfs /dev/gvinum/v<span style="color:red">123</span>a</tt><br><br />
<br />
-or- <br />
<br />
<br />
B. for a >2G system (create a gconcat volume):<br><br />
try to grab a contiguous block of gvinum volumes. gconcat volumes MAY NOT span drives (i.e. you cannot use a gvinum volume from data3 and a volume from data2 in the same gconcat volume). <br />
<br />
<tt>gconcat label <span style="color:red">v82-v84 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84</span><br><br />
bsdlabel -r -w /dev/concat/<span style="color:red">v82-v84</span><br><br />
newfs /dev/concat/<span style="color:red">v82-v84</span>a</tt><br />
<br />
<br />
Other valid gconcat examples:<br><br />
<tt>gconcat label v82-v84v109v112 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84 /dev/gvinum/v109 /dev/gvinum/v112<br><br />
gconcat label v82v83 /dev/gvinum/v82 /dev/gvinum/v83</tt> <br><br />
Note, long names will truncate: v144v145v148-v115 will truncate to v144v145v148-v1 (so you will refer to it as v144v145v148-v1 thereafter)<br />
<br />
<br />
Optional- if new volume is on a different drive, move the mount point directory (get the drive from js output) AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
<br />
confirm you are mounted to the device (<tt>/dev/gvinum/v<span style="color:red">123</span>a</tt> OR <tt>/dev/concat/<span style="color:red">v82-v84</span></tt>) and space is correct:<br><br />
A. <tt>mount /dev/gvinum/v<span style="color:red">123</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br><br />
-or-<br><br />
B. <tt>mount /dev/concat/<span style="color:red">v82-v84</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<tt>cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
df .</tt> <br />
<br />
<br />
do the dump and pipe directly to restore:<br><br />
<tt>dump -0a -f - /dev/gvinum/v<span style="color:red">1</span> | restore -r -f - <br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
edit quad (<tt>vq<span style="color:red">1</span></tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new volume (<tt>/dev/gvinum/v<span style="color:red">123</span>a</tt> or <tt>/dev/concat/<span style="color:red">v82-v84</span>a</tt>) , run <tt>buildsafe</tt><br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
5b. When there's not enough room on an alternate partition, or on the same drive...but there is enough room if you were to remove the existing customer's space (i.e. if you want/need to reuse the existing gvinum volumes and add on more):<br />
<br />
<br />
mount backup nfs mounts:<br><br />
<tt>mbm</tt> <br><br />
(run df to confirm backup mounts are mounted)<br />
<br />
<br />
dump the customer to backup2 or backup1<br><br />
<tt>dump -0a -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> /dev/<span style="color:red">gconcat/v106-v107</span></tt><br><br />
(when complete WITHOUT errors, du the dump file to confirm it matches size, roughly, with usage)<br />
<br />
<br />
unconfigure the old gconcat volume<br><br />
list member gvinum volumes:<br />
<br />
<tt>gconcat list <span style="color:red">v106v107</span></tt><br />
<br />
<br />
Output will resemble:<br><br />
<pre>Geom name: v106v107<br />
State: UP<br />
Status: Total=2, Online=2<br />
Type: AUTOMATIC<br />
ID: 3530663882<br />
Providers:<br />
1. Name: concat/v106v107<br />
Mediasize: 4294966272 (4.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e2<br />
Consumers:<br />
1. Name: gvinum/sd/v106.p0.s0<br />
Mediasize: 2147483648 (2.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e3<br />
Start: 0<br />
End: 2147483136<br />
2. Name: gvinum/sd/v107.p0.s0<br />
Mediasize: 2147483648 (2.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e3<br />
Start: 2147483136<br />
End: 4294966272</pre><br />
<br />
stop volume and clear members<br><br />
<tt>gconcat stop <span style="color:red">v106v107</span><br><br />
gconcat clear <span style="color:red">gvinum/sd/v106.p0.s0 gvinum/sd/v107.p0.s0</span></tt><br />
<br />
<br />
create new device- and its ok to reuse old/former members<br><br />
try to grab a contiguous block of gvinum volumes. gconcat volumes MAY NOT span drives (i.e. you cannot use a gvinum volume from data3 and a volume from data2 in the same gconcat volume). <br><br />
<tt>gconcat label <span style="color:red">v82-v84v106v107 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84 /dev/gvinum/v106 /dev/gvinum/v107</span> <br><br />
bsdlabel -r -w /dev/concat/<span style="color:red">v82-v84v106v107</span><br><br />
newfs /dev/concat/<span style="color:red">v82-v84v106v107</span>a</tt><br />
<br />
<br />
Optional- if new volume is on a different drive, move the mount point directory (get the drive from js output) AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
confirm you are mounted to the device (/dev/concat/<span style="color:red">v82-v84v106v107</span>) and space is correct:<br><br />
<tt>mount /dev/concat/<span style="color:red">v82-v84v106v107</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<tt>cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
df .</tt> <br />
<br />
do the restore from the dumpfile on the backup server:<br><br />
<tt>restore -r -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> .<br><br />
rm restoresymtable</tt><br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
edit quad (<tt>vq<span style="color:red">1</span></tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new volume (<tt>/dev/concat/<span style="color:red">v82-v84v106v107</span>a</tt>), run buildsafe<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
TODO: clean up/clear old gvin/gconcat vol<br />
<br />
<br />
6. update disk (and dir if applicable) in mgmt screen<br />
<br />
<br />
7. update backup list AND move backups, if applicatble<br />
<br />
<br />
Ex: <tt>mvbackups <span style="color:red">69.55.237.26-col00241</span> jail<span style="color:red">9</span> data<span style="color:red">3</span></tt><br />
<br />
<br />
<br />
DEPRECATED - steps to tack on a new gvin to existing gconcat- leads to corrupted fs<br />
bsdlabel -e /dev/concat/v82-v84<br />
<br />
To figure out new size of the c partition, multiply 4194304 by the # of 2G gvinum volumes and subtract the # of 2G volumes:<br />
10G: 4194304 * 5 – 5 = 20971515<br />
8G: 4194304 * 4 – 4 = 16777212<br />
6G: 4194304 * 3 – 3 = 12582909<br />
4G: 4194304 * 2 – 2 = 8388606<br />
<br />
To figure out the new size of the a partition, subtract 16 from the c partition:<br />
10G: 20971515 – 16 = 20971499<br />
8G: 16777212 – 16 = 16777196<br />
6G: 12582909 – 16 = 12582893<br />
4G: 8388606 – 16 = 8388590<br />
<br />
Orig:<br />
8 partitions:<br />
# size offset fstype [fsize bsize bps/cpg]<br />
a: 8388590 16 4.2BSD 2048 16384 28552<br />
c: 8388606 0 unused 0 0 # "raw" part, don't edit<br />
<br />
New:<br />
8 partitions:<br />
# size offset fstype [fsize bsize bps/cpg]<br />
a: 12582893 16 4.2BSD 2048 16384 28552<br />
c: 12582909 0 unused 0 0 # "raw" part, don't edit<br />
<br />
sync; sync<br />
<br />
growfs /dev/concat/v82-v84a<br />
<br />
fsck –fy /dev/concat/v82-v84a<br />
<br />
sync<br />
<br />
fsck –fy /dev/concat/v82-v84a<br />
<br />
(keep running fsck’s till NO errors)<br />
<br />
== Problems un-mounting - and with mount_null’s ==<br />
<br />
If you cannot unmount a filesystem, beacuse it says the filesystem is busy, it is because of three things:<br />
<br />
a) the jail is still running<br />
<br />
b) you are actually in that directory, even though the jail is stopped<br />
<br />
c) there are still dev, null_mount or linprocfs mount points mounted inside that directory.<br />
<br />
d) when trying to umount null_mounts that are really long and you get an error like “No such file or directory”, it’s an OS bug where the dir is truncated. No known fix<br />
<br />
e) there are still files open somewhere inside the dir. Use <tt>fstat | grep <cid></tt> to find the process that has files open<br />
<br />
f) Starting with 6.x, the jail mechanism does a poor job of keeping track of processes running in a jail and if it thinks there are still procs running, it will refuse to umount the disk. If this is happening you should see a low number in the #REF column when you run jls. In this case you ''can'' safely <tt>umount –f</tt> the mount. <br />
<br />
Please note -if you forcibly unmount a (4.x) filesystem that has null_mounts<br />
still mounted in it, the system '''will crash''' within 10-15 mins.<br />
<br />
== Misc jail Items ==<br />
<br />
We are overselling hard drive space on jail2, jail8, jail9, a couple jails on jail17, jail4, jail12 and jail18.<br />
Even though the vn file shows 4G size, it doesn’t actually occupy that amount of space on the disk. So be careful not to fill up drives where we’re overselling – use oversellcheck to confirm you’re not oversold by more than 10G.<br />
There are other truncated jails, they are generally noted in a the file on the root system: /root/truncated<br />
<br />
The act of moving a truncated vn to another system un-does the truncating- the truncated vn is filled with 0’s and it occupies physical disk space for which it’s configured. So, you should use dumpremote to preserve the truncation.<br />
<br />
= FreeBSD VPS Management Tools =<br />
<br />
These files are located in /usr/local/jail/rc.d and /usr/local/jail/bin<br />
<br />
== jailmake ==<br />
<br />
Applies to 7.x+ <br />
On older systems syntax differs, run jailmake once to see.<br />
<br />
Note: this procedure differs on mx2 which is 7.x but still uses gvinum<br />
<br />
# run js to figure out which md’s are in use, which disk has enough space, IP to put it on<br />
# use col00xxx for both hostnames if they don’t give you a hostname<br />
# copy over dir, ip and password to pending customer screen<br />
<br />
<tt>Usage: jailmake IP[,IP] CID disk[1|2|3] md# hostname shorthost ipfw# email [size in GB]</tt><br />
<br />
Ex: <br />
<br />
Jail2# jailmake 69.55.234.66 col01334 3 97 vps.bsd.it vps 1334 fb@bsd.it<br />
<br />
== jailps ==<br />
jailps [hostname]<br />
DEPRECATED FOR jps: displays processes belonging to/running inside a jail. The command<br />
takes one (optional) argument – the hostname of the jail you wish to query. If you don’t <br />
supply an argument, all processes on the machine are listed and grouped by jail. <br />
<br />
== jps ==<br />
jps [hostname]<br />
displays processes belonging to/running inside a jail. The command<br />
takes one (optional) argument – the hostname or ID of the jail you wish to query. <br />
<br />
== jailkill ==<br />
jailkill <hostname><br />
stops all process running in a jail.<br />
<br />
You can also run:<br />
jailkill <JID><br />
<br />
=== problems ===<br />
Occasionally you will hit an issue where jail will not kill off:<br />
<br />
<pre>jail9# jailkill www.domain.com<br />
www.domain.com .. killed: none<br />
jail9#</pre><br />
Because no processes are running under that hostname. You cannot use jailps.pl either:<br />
<br />
<pre>jail9# jailps www.domain.com<br />
www.domain.com doesn’t exist on this server<br />
jail9#</pre><br />
<br />
The reasons for this are usually:<br />
* the jail is no longer running<br />
<br />
* the jail's hostname has changed<br />
In this case, <br />
<br />
>=6.x: run a <tt>jls|grep <jail's IP></tt> to find the correct hostname, then update the quad file, then kill the jail.<br />
<br />
<6.x: the first step is to cat their /etc/rc.conf file to see if you can tell what they set the new hostname to. This very often works. For example:<br />
<br />
cat /mnt/data2/198.78.65.136-col00261-DIR/etc/rc.conf<br />
<br />
But maybe they set the hostname with the hostname command, and the original hostname is still in /etc/rc.conf.<br />
<br />
The welcome email clearly states that they should tell us if they change their hostname, so there is no problem in just emailing them and asking them what they set the new hostname to.<br />
<br />
Once you know the new hostname OR if a customer simply emails to inform you that they have set the hostname to something different, you need to edit the quad and safe files that their system is in to input the new hostname.<br />
<br />
However, if push comes to shove and you cannot find out the hostname from them or from their system, then you need to start doing some detective work.<br />
<br />
The easiest thing to do is run jailps looking for a hostname similar to their original hostname. Or you could get into the /bin/sh shell by running:<br />
<br />
/bin/sh<br />
<br />
and then looking at every hostname of every process:<br />
<br />
for f in `ls /proc` ; do cat /proc/$f/status ; done<br />
<br />
and scanning for a hostname that is either similar to their original hostname, or that you don't see in any of the quad safe files.<br />
<br />
This is very brute force though, and it is possible that catting every file in /proc is dangerous - I don't recommend it. A better thing would be to identify any processes that you know belong to this system – perhaps the reason you are trying to find this system is because they are running something bad - and just catting the status from only that PID.<br />
<br />
Somewhere there’s a jail where there may be 2 systems named www. Look at /etc/rc.conf and make sure they’re both really www. If they are, jailkill www, jailps www to make sure not running. Then immediately restart the other one, as the fqdn (as found from a rev nslookup)<br />
<br />
* on >=6.x the hostname may not yet be hashed:<br />
<pre>jail9 /# jls<br />
JID Hostname Path IP Address(es)<br />
1 bitnet.dgate.org /mnt/data1/69.55.232.50-col02094-DIR 69.55.232.50<br />
2 ns3.hctc.net /mnt/data1/69.55.234.52-col01925-DIR 69.55.234.52<br />
3 bsd1 /mnt/data1/69.55.232.44-col00155-DIR 69.55.232.44<br />
4 let2.bbag.org /mnt/data1/69.55.230.92-col00202-DIR 69.55.230.92<br />
5 post.org /mnt/data2/69.55.232.51-col02095-DIR 69.55.232.51 ...<br />
6 ns2 /mnt/data1/69.55.232.47-col01506-DIR 69.55.232.47 ...<br />
7 arlen.server.net /mnt/data1/69.55.232.52-col01171-DIR 69.55.232.52<br />
8 deskfood.com /mnt/data1/69.55.232.71-col00419-DIR 69.55.232.71<br />
9 mirage.confluentforms.com /mnt/data1/69.55.232.54-col02105-DIR 69.55.232.54 ...<br />
10 beachmember.com /mnt/data1/69.55.232.59-col02107-DIR 69.55.232.59<br />
11 www.agottem.com /mnt/data1/69.55.232.60-col02109-DIR 69.55.232.60<br />
12 sdhobbit.myglance.org /mnt/data1/69.55.236.82-col01708-DIR 69.55.236.82<br />
13 ns1.jnielsen.net /mnt/data1/69.55.234.48-col00204-DIR 69.55.234.48 ...<br />
14 ymt.rollingegg.net /mnt/data2/69.55.236.71-col01678-DIR 69.55.236.71<br />
15 verse.unixlore.net /mnt/data1/69.55.232.58-col02131-DIR 69.55.232.58<br />
16 smcc-mail.org /mnt/data2/69.55.232.68-col02144-DIR 69.55.232.68<br />
17 kasoutsuki.w4jdh.net /mnt/data2/69.55.232.46-col02147-DIR 69.55.232.46<br />
18 dili.thium.net /mnt/data2/69.55.232.80-col01901-DIR 69.55.232.80<br />
20 www.tekmarsis.com /mnt/data2/69.55.232.66-col02155-DIR 69.55.232.66<br />
21 vps.yoxel.net /mnt/data2/69.55.236.67-col01673-DIR 69.55.236.67<br />
22 smitty.twitalertz.com /mnt/data2/69.55.232.84-col02153-DIR 69.55.232.84<br />
23 deliver4.klatha.com /mnt/data2/69.55.232.67-col02160-DIR 69.55.232.67<br />
24 nideffer.com /mnt/data2/69.55.232.65-col00412-DIR 69.55.232.65<br />
25 usa.hanyuan.com /mnt/data2/69.55.232.57-col02163-DIR 69.55.232.57<br />
26 daifuku.ppbh.com /mnt/data2/69.55.236.91-col01720-DIR 69.55.236.91<br />
27 collins.greencape.net /mnt/data2/69.55.232.83-col01294-DIR 69.55.232.83<br />
28 ragebox.com /mnt/data2/69.55.230.104-col01278-DIR 69.55.230.104<br />
29 outside.mt.net /mnt/data2/69.55.232.72-col02166-DIR 69.55.232.72<br />
30 vps.payneful.ca /mnt/data2/69.55.234.98-col01999-DIR 69.55.234.98<br />
31 higgins /mnt/data2/69.55.232.87-col02165-DIR 69.55.232.87 ...<br />
32 ozymandius /mnt/data2/69.55.228.96-col01233-DIR 69.55.228.96<br />
33 trusted.realtors.org /mnt/data2/69.55.238.72-col02170-DIR 69.55.238.72<br />
34 jc1.flanderous.com /mnt/data2/69.55.239.22-col01504-DIR 69.55.239.22<br />
36 guppylog.com /mnt/data2/69.55.238.73-col00036-DIR 69.55.238.73<br />
40 haliohost.com /mnt/data2/69.55.234.41-col01916-DIR 69.55.234.41 ...<br />
41 satyr.jorge.cc /mnt/data1/69.55.232.70-col01963-DIR 69.55.232.70<br />
jail9 /# jailkill satyr.jorge.cc<br />
ERROR: jail_: jail "satyr,jorge,cc" not found<br />
</pre><br />
<br />
Note how it's saying <tt>satyr,jorge,cc</tt> is not found, and not <tt>satyr.jorge.cc</tt>. <br />
<br />
The jail subsystem tracks things using comma-delimited hostnames. That is created every few hours:<br />
<br />
jail9 /# crontab -l<br />
0 0,6,12,18 * * * /usr/local/jail/bin/sync_jail_names<br />
<br />
So if we run this manually:<br />
jail9 /# /usr/local/jail/bin/sync_jail_names<br />
<br />
Then kill the jail:<br />
jail9 /# jailkill satyr.jorge.cc<br />
successfully killed: satyr,jorge,cc</pre><br />
<br />
It worked.<br />
<br />
<br />
If you ever see this when trying to kill a jail:<br />
<br />
<pre># jailkill e-scribe.com<br />
killing JID: 6 hostname: e-scribe.com<br />
3 procs running<br />
3 procs running<br />
3 procs running<br />
3 procs running<br />
...</pre><br />
<br />
<tt>[[#jailkill|jailkill]]</tt> probably got lost trying to kill off the jail. Just ctrl-c the jailkill process, then run a jailps on the hostname, and <tt>kill -9</tt> any process which is still running. Keep running jailps and <tt>kill -9</tt> till all processes are gone.<br />
<br />
== jailpsall ==<br />
jailpsall<br />
will run a jailps on all jails configured in the quad files (this is different from<br />
jailps with no arguments as it won’t help you find a “hidden” system)<br />
<br />
== jailpsw ==<br />
jailpsw<br />
will run a jailps with an extra -w to provide wider output<br />
<br />
== jt (>=7.x) ==<br />
jt<br />
displays the top 20 processes on the server (the top 20 processes from top) and <br />
which jail owns them. This is very helpful for determining who is doing what when<br />
the server is very busy.<br />
<br />
== jtop (>=7.x) ==<br />
jtop<br />
a wrapper for top displaying processes on the server and which jail owns them. Constantly updates, like top. <br />
<br />
== jtop (<7.x) ==<br />
jtop<br />
displays the top 20 processes on the server (the top 20 processes from top) and <br />
which jail owns them. This is very helpful for determining who is doing what when<br />
the server is very busy.<br />
<br />
== stopjail ==<br />
stopjail <hostname> [1]<br />
this will jailkill, umount and vnconfig –u a jail. If passed an optional 2nd<br />
argument, it will not exit before umounting and un-vnconfig’ing in the event<br />
jailkill returns no processes killed. This is useful if you just want to umount<br />
and vnconfig –u a jail you’ve already killed. It is intelligent in that it won’t <br />
try to umount or vnconfig –u if it’s not necessary.<br />
<br />
== startjail ==<br />
startjail <hostname><br />
this will start vnconfig, mount (including linprocfs and null-mounts), and start a jail.<br />
Essentially, it reads the jail’s relevant block from the right quad file and executes it.<br />
It is intelligent in that it won’t try to mount or vnconfig if it’s not necessary.<br />
<br />
== jpid ==<br />
jpid <pid><br />
displays information about a process – including which jail owns it.<br />
It’s the equivalent of running cat /proc/<pid>/status<br />
<br />
== canceljail ==<br />
canceljail <hostname> [1]<br />
this will stop a jail (the equivalent of stopjail), check for backups (offer to remove them <br />
from the backup server and the backup.config), rename the vnfile, remove the dir, and <br />
edit quad/safe. If passed an optional 2nd argument, it will not exit upon failing to kill<br />
and processes owned by the jail. This is useful if you just want to cancel a jail which <br />
is already stopped.<br />
<br />
== jls ==<br />
jls [-v]<br />
Lists all jails running:<br />
<pre>JID #REF IP Address Hostname Path<br />
101 135 69.55.224.148 mail.pc9.org /mnt/data2/69.55.224.148-col01034-DIR</pre><br />
<br />
#REF is the number of references or procs(?) running<br />
<br />
Running with -v will give you all IPs assigned to each jail (7.2 up)<br />
<pre>JID #REF Hostname Path IP Address(es)<br />
101 139 mail.pc9.org /mnt/data2/69.55.224.148-col01034-DIR 69.55.224.14869.55.234.85</pre><br />
<br />
== startalljails ==<br />
startalljails<br />
7.2+ only. This will parse through quad1 and start all jails. It utilizes lockfiles so it won’t try to start a jail more than once- therefore multiple instances can be running in parallel without fear of starting a jail twice. If a jail startup gets stuck, you can ^C without fear of killing the script. IMPORTANT- before running startalljails you should make sure you ran preboot once as it will clear out all the lockfiles and enable startalljails to work properly.<br />
<br />
== aaccheck.sh ==<br />
aaccheck.sh<br />
displayes the output of container list and task list from aaccli<br />
<br />
== backup ==<br />
backup<br />
backup script called nightly to update jail scripts and do customer backups<br />
<br />
== buildsafe ==<br />
buildsafe<br />
creates safe files based on quads (automatically removing the fsck’s). This will destructively overwrite safe files<br />
<br />
== checkload.pl ==<br />
checkload.pl<br />
this was intended to be setup as a cronjob to watch processes on a jail when the load <br />
rises above a certain level. Not currently in use.<br />
<br />
== checkprio.pl ==<br />
checkprio.pl<br />
will look for any process (other than the current shell’s csh, sh, sshd procs) with a non-normal priority and normalize it<br />
<br />
== diskusagemon == <br />
diskusagemon <mount point> <1k blocks><br />
watches a mount point’s disk use, when it reaches the level specified in the 2nd argument,<br />
it exits. This is useful when doing a restore and you want to be paged as it’s nearing completion.<br />
Best used as: <tt>diskusagemon /asd/asd 1234; pagexxx</tt><br />
<br />
== dumprestore ==<br />
dumprestore <dumpfile><br />
this is a perl expect script which automatically enters ‘1’ and ‘y’. It seems to cause restore to fail<br />
to set owner permissions on large restores.<br />
<br />
== g ==<br />
g <search><br />
greps the quad/safe files for the given search parameter<br />
<br />
== gather.pl ==<br />
gather.pl<br />
gathers up data about jails configured and writes to a file. Used for audits against the db<br />
<br />
== gb ==<br />
gb <search><br />
greps backup.config for the given search parameter<br />
<br />
== gbg ==<br />
gbg <search><br />
greps backup.config for the given search parameter and presents just the directories (for clean pasting)<br />
<br />
== ipfwbackup ==<br />
ipfwbackup<br />
writes ipfw traffic count data to a logfile<br />
<br />
== ipfwreset ==<br />
ipfwreset<br />
writes ipfw traffic count data to a logfile and resets counters to 0<br />
<br />
== js ==<br />
js<br />
output varies by OS version, but generally provides information about the base jail:<br />
- which vn’s are in use<br />
- disk usage<br />
- info about the contents of quads<br />
- the # of inodes represented by the jails contained in the group (133.2 in the example below), and how many jails per data mount, as well as subtotals<br />
- ips bound to the base machine but not in use by a jail<br />
- free gvinum volumes, or unused vn’s or used md’s<br />
<br />
<pre>/usr/local/jail/rc.d/quad1:<br />
/mnt/data1 133.2 (1)<br />
/mnt/data2 1040.5 (7)<br />
total 1173.7 (8)<br />
/usr/local/jail/rc.d/quad2:<br />
/mnt/data1 983.4 (6)<br />
total 983.4 (6)<br />
/usr/local/jail/rc.d/quad3:<br />
/mnt/data1 693.4 (4)<br />
/mnt/data2 371.6 (3)<br />
total 1065 (7)<br />
/usr/local/jail/rc.d/quad4:<br />
/mnt/data1 466.6 (3)<br />
/mnt/data2 882.2 (5)<br />
total 1348.8 (8)<br />
/mnt/data1: 2276.6 (14)<br />
/mnt/data2: 2294.3 (15)<br />
<br />
Available IPs:<br />
69.55.230.11 69.55.230.13 69.55.228.200<br />
<br />
Available volumes:<br />
v78 /mnt/data2 2G<br />
v79 /mnt/data2 2G<br />
v80 /mnt/data2 2G</pre> <br />
<br />
== load.pl ==<br />
load.pl<br />
feeds info to load mrtg - executed by inetd.<br />
<br />
== makevirginjail ==<br />
makevirginjail<br />
Only on some systems, makes an empty jail (doesn't do restore step)<br />
<br />
== mb == <br />
mb <mount|umount><br />
(nfs) mounts and umounts dirs to backup2. Shortcuts are mbm and mbu to mount and unmount. <br />
<br />
== notify.sh ==<br />
notify.sh<br />
emails reboot@johncompanies.com – intended to be called at boot time to alert us to a machine which panics and reboots and isn’t caught by bb or castle.<br />
<br />
== orphanedbackupwatch ==<br />
orphanedbackupwatch<br />
looks for directories on backup2 which aren’t configured in backup.config and offers to delete them<br />
<br />
== postboot ==<br />
postboot<br />
to be run after a machine reboot and quad/safe’s are done executing. It will:<br />
* do chmod 666 on each jail’s /dev/null<br />
* add ipfw counts<br />
* run jailpsall (so you can see if a configured jail isn’t running)<br />
<br />
== preboot ==<br />
preboot<br />
to be run before running quad/safe – checks for misconfigurations: <br />
* a jail configured in a quad but not a safe<br />
* a jail is listed more than once in a quad<br />
* the ip assigned to a jail isn’t configured on the machine<br />
* alias numbering skips in the rc.conf (resulting in the above)<br />
* orphaned vnfile's that aren't mentioned in a quad/safe<br />
* ip mismatches between dir/vnfile name and the jail’s ip<br />
* dir/vnfiles's in quad/safe that don’t exist <br />
<br />
== quadanalyze.pl ==<br />
quadanalyze.pl<br />
called by js, produces the info (seen above with js explanation) about the contents of quad (inode count, # of jails, etc.)<br />
<br />
== rsync.backup ==<br />
rsync.backup<br />
does customer backups (relies on backup.config)<br />
<br />
== taskdone ==<br />
taskdone<br />
when called will email support@johncompanies.com with the hostname of the machine from which it was executed as the subject<br />
<br />
== topten ==<br />
topten<br />
summarizes the top 10 traffic users (called by ipfwreset)<br />
<br />
== trafficgather.pl ==<br />
trafficgather.pl [yy-mm]<br />
sends a traffic usage summary by jail to support@johncomapnies.com and payments@johncompanies.com. Optional arguments are year and month (must be in the past). If not passed, assumes last month. Relies on traffic logs created by ipfwreset and ipfwbackup<br />
<br />
== trafficwatch.pl ==<br />
trafficwatch.pl<br />
checks traffic usage and emails support@johncomapnies.com when a jail reaches the warning level (35G) and the limit (40G). We really aren’t using this anymore now that we have netflow.<br />
<br />
== trafstats ==<br />
trafstats<br />
writes ipfw traffic usage info by jail to a file called jc_traffic_dump in each jail’s / dir<br />
<br />
== truncate_jailmake ==<br />
truncate_jailmake<br />
a version of jailmake which creates truncated vnfiles.<br />
<br />
== vb ==<br />
vb<br />
the equivalent of: <tt>vi /usr/local/jail/bin/backup.config</tt><br />
<br />
== vs (freebsd) ==<br />
vs<n><br />
the equivalent of: <tt>vi /usr/local/jail/rc.d/safe<n></tt><br />
<br />
vq<n><br />
the equivalent of: vi /usr/local/jail/rc.d/quad<n><br />
<br />
== dumpremote ==<br />
dumpremote <user@machine> </remote/location/file-dump> <vnX><br />
ex: dumpremote user@10.1.4.117 /mnt/data3/remote.echoditto.com-dump 7<br />
this will dump a vn filesystem to a remote machine and location<br />
<br />
== oversellcheck ==<br />
oversellcheck<br />
displays how much a disk is oversold or undersold taking into account truncated vn files. Only for use on 4.x systems<br />
<br />
== mvbackups (freebsd) ==<br />
mvbackups <dir> (1.1.1.1-col00001-DIR) <target_machine> (jail1) <target_dir> (data1)<br />
moves backups from one location to another on the backup server, and provides you with option to remove entries from current backup.config, and simple paste command to add the config to backup.config on the target server<br />
<br />
== jailnice ==<br />
jailnice <hostname><br />
applies <tt>renice 19 [PID]</tt> and <tt>rtprio 31 –[PID]</tt> to each process in the given jail<br />
<br />
== dumpremoterestore ==<br />
dumpremoterestore <device> <ip of target machine> <dir on target machine><br />
ex: dumpremoterestore /dev/vn51 10.1.4.118 /mnt/data2/69.55.239.45-col00688-DIR<br />
dumps a device and restores it to a directory on a remote machine. Requires that you enable root ssh on the <br />
remote machine.<br />
<br />
== psj ==<br />
psj<br />
shows just the procs running on the base system – a ps auxw but without jail’d procs present<br />
<br />
== perc5iraidchk ==<br />
perc5iraidchk<br />
checks for degraded arrays on Dell 2950 systems with Perc5/6 controllers<br />
<br />
== perc4eraidchk ==<br />
perc4eraidchk<br />
checks for degraded arrays on Dell 2850 systems with Perc4e/Di controllers<br />
<br />
= Virtuozzo VPS =<br />
<br />
Note: We use VEID (Virtual Environment ID) and CTID (Container ID) interchangably. Similarly, VE and CT. They mean the same thing.<br />
VZPP = VirtuoZzo Power Panel (the control panel for each CT)<br />
<br />
All linux systems exist in /vz, /vz1 or /vz2 - since each linux machine holds roughly 60-90 customers, there will be roughly 30-45 in each partition.<br />
<br />
The actual filesystem of the system in question is in:<br />
<br />
/vz(1-2)/private/(VEID)<br />
<br />
Where VEID is the identifier for that system - an all-numeric string larger than 100.<br />
<br />
The actual mounted and running systems are in the corresponding:<br />
<br />
/vz(1-2)/root/(VEID)<br />
<br />
But we rarely interact with any system from this mount point.<br />
<br />
You should never need to touch the root portion of their system – however you can traverse their filesystem by going to <tt>/vz(1-2)/private/(VEID)/root</tt> (<tt>/vz(1-2)/private/(VEID)/fs/root</tt> on 4.x systems) the root of their filesystem is in that directory, and their entire system is underneath that.<br />
<br />
Every VE has a startup script in <tt>/etc/sysconfig/vz-scripts</tt> (which is symlinked as <tt>/vzconf</tt> on all systems) - the VE startup script is simply named <tt>(VEID).conf</tt> - it contains all the system parameters for that VE:<br />
<br />
<pre># Configuration file generated by vzsplit for 60 VE<br />
# on HN with total amount of physical mem 2011 Mb<br />
<br />
VERSION="2"<br />
CLASSID="2"<br />
<br />
ONBOOT="yes"<br />
<br />
KMEMSIZE="8100000:8200000"<br />
LOCKEDPAGES="322:322"<br />
PRIVVMPAGES="610000:615000"<br />
SHMPAGES="33000:34500"<br />
NUMPROC="410:415"<br />
PHYSPAGES="0:2147483647"<br />
VMGUARPAGES="13019:2147483647"<br />
OOMGUARPAGES="13019:2147483647"<br />
NUMTCPSOCK="1210:1215"<br />
NUMFLOCK="107:117"<br />
NUMPTY="19:19"<br />
NUMSIGINFO="274:274"<br />
TCPSNDBUF="1800000:1900000"<br />
TCPRCVBUF="1800000:1900000"<br />
OTHERSOCKBUF="900000:950000"<br />
DGRAMRCVBUF="200000:200000"<br />
NUMOTHERSOCK="650:660"<br />
DCACHE="786432:818029"<br />
NUMFILE="7500:7600"<br />
AVNUMPROC="51:51"<br />
IPTENTRIES="155:155"<br />
DISKSPACE="4194304:4613734"<br />
DISKINODES="400000:420000"<br />
CPUUNITS="1412"<br />
QUOTAUGIDLIMIT="2000"<br />
VE_ROOT="/vz1/root/636"<br />
VE_PRIVATE="/vz1/private/636"<br />
NAMESERVER="69.55.225.225 69.55.230.3"<br />
OSTEMPLATE="vzredhat-7.3/20030305"<br />
VE_TYPE="regular"<br />
IP_ADDRESS="69.55.225.229"<br />
HOSTNAME="textengine.net"</pre><br />
<br />
<br />
As you can see, the hostname is set here, the disk space is set here, the number of inodes, the number of files that can be open, the number of tcp sockets, etc. - all are set here.<br />
<br />
In fact, everything that can be set on this customer system is set in this conf file.<br />
<br />
<br />
All interaction with the customer system is done with the VEID. You start the system by running:<br />
<br />
vzctl start 999<br />
<br />
You stop it by running:<br />
<br />
vzctl stop 999<br />
<br />
You execute commands in it by running:<br />
<br />
vzctl exec 999 df -k<br />
<br />
You enter into it, via a root-shell backdoor with:<br />
<br />
vzctl enter 999<br />
<br />
and you set parameters for the system, while it is still running, with:<br />
<br />
vzctl set 999 --diskspace 6100000:6200000 --save<br />
<br />
<tt>vzctl</tt> is the most commonly used command - we have aliased <tt>v</tt> to <tt>vzctl</tt> since we use it so often. We’ll continue to use <tt>vzctl</tt> in our examples, but feel free to use just <tt>v</tt>.<br />
<br />
Let's say the user wants more diskspace. You can cat their conf file and see:<br />
<br />
DISKSPACE="4194304:4613734"<br />
<br />
So right now they have 4gigs of space. You can then change it to 6 with:<br />
<br />
vzctl set 999 --diskspace 6100000:6200000 --save<br />
<br />
IMPORTANT: all issuances of the vzctl set command need to end with <tt>–save</tt> - if they don't, the setting will be set, but it will not be saved to the conf file, and they will not have those settings next time they boot.<br />
<br />
All of the tunables in the conf file can be set with the vzctl set command. Note that in the conf file, and on the vzctl set command line, we always issue two numbers seperated by a colon - that is because we are setting the hard and soft limits. Always set the hard limit slightly above the soft limit, as you see it is in the conf file for all those settings.<br />
<br />
There are also things you can set with `<tt>vzctl set</tt>` that are not in the conf file as settings, per se. For instance, you can add IPs:<br />
<br />
vzctl set 999 --ipadd 10.10.10.10 --save<br />
<br />
or multiple IPs:<br />
<br />
vzctl set 999 --ipadd 10.10.10.10 --ipadd 10.10.20.30 --save<br />
<br />
or change the hostname:<br />
<br />
vzctl set 999 --hostname www.example.com --save<br />
<br />
You can even set the nameservers:<br />
<br />
vzctl set 999 --nameserver 198.78.66.4 --nameserver 198.78.70.180 --save<br />
<br />
Although you probably will never do that.<br />
<br />
You can disable a VPS from being started (by VZPP or reboot) (4.x):<br />
<br />
vzctl set 999 --disabled yes --save <br />
<br />
You can disable a VPS from being started (by VZPP or reboot) (<=3.x):<br />
<br />
vzctl set 999 --onboot=no --save <br />
<br />
You can disable a VPS from using his control panel:<br />
<br />
vzctl set 999 --offline_management=no --save <br />
<br />
You can suspend a VPS, so it can be resumed in the same state it was in when it was stopped (4.x):<br />
<br />
vzctl suspend 999<br />
<br />
and to resume it:<br />
<br />
vzctl resume 999<br />
<br />
to see who owns process:<br />
vzpid <PID><br />
<br />
to mount up an unmounted ve:<br />
vzctl mount 827<br />
<br />
<br />
To see network stats for CT's:<br />
<pre># vznetstat<br />
VEID Net.Class Output(bytes) Input(bytes)<br />
-----------------------------------------------<br />
24218 1 484M 39M<br />
24245 1 463M 143M<br />
2451 1 2224M 265M<br />
2454 1 2616M 385M<br />
4149 1 125M 68M<br />
418 1 1560M 34M<br />
472 1 1219M 315M<br />
726 1 628M 317M<br />
763 1 223M 82M<br />
771 1 4234M 437M<br />
-----------------------------------------------<br />
Total: 13780M 2090M</pre><br />
<br />
<br />
One thing that sometimes comes up on older systems that we created with smaller defaults is that the system would run out of inodes. The user will email and say they cannot create any more files or grow any files larger, but they will also say that they are not out of diskspace ... they are running:<br />
<br />
df -k<br />
<br />
and seeing how much space is free - and they are not out of space. They are most likely out of inodes - which they would see by running:<br />
<br />
df -i<br />
<br />
So, the first thing you should do is enter their system with:<br />
<br />
vzctl enter 999<br />
<br />
and run: <tt>df -i</tt><br />
<br />
to confirm your theory. Then exit their system. Then simply cat their conf file and see what their inodes are set to (probably 200000:200000, since that was the old default on the older systems) and run:<br />
<br />
vzctl set 999 --diskinodes 400000:400000 --save<br />
<br />
If they are not out of inodes, then a good possibility is that they have maxed out their numfile configuration variable, which controls how many files they can have in their system. The current default is 7500 (which nobody has ever hit), but the old default was as low as 2000, so you would run something like:<br />
<br />
vzctl set 999 --numfile 7500:7500 --save<br />
<br />
----<br />
<br />
You cannot start or stop a VE if your pwd is its private (/vz/private/999) or root (/vz/root/999) directories, or anywhere below them.<br />
<br />
<br />
== Recovering from a crash (linux) ==<br />
<br />
=== Diagnose whether you have a crash ===<br />
The most important thing is to get the machine and all ve’s back up as soon as possible. Note the time, you’ll need to create a crash log entry (Mgmt. -> Reference -> CrashLog). The first thing to do is head over to the [[Screen#Screen_Organization|serial console screen]] and see if there’s any kernel error messages output. Try to copy any messages (or just a sample of repeating messages) you see into the notes section of the crash log – these will also likely need to be sent to virtuozzo for interpretation. If the messages are spewing too fast, hit ^O + H to start a screen log dump which you can ob1182.pts-38.bb serve after the machine is rebooted. Additionally, if the machine is responsive, you can get a trace to send to virtuozzo by hooking up a kvm and entering these 3 sequences:<br />
<pre>alt+print screen+m<br />
alt+print screen+p<br />
alt+print screen+t</pre><br />
<br />
If there are no messages, the machine may just be really busy- wait a bit (5-10min) to see if it comes back. If it's still pinging, odds are its very busy. If it doesn't come back, or the messages indicate a fatal error, you will need to proceed with a power cycle (ctrl+alt+del will not work).<br />
<br />
=== Power cycle the server ===<br />
If this machine is not a Dell 2950 with a [[DRAC/RMM#DRAC|DRAC card]] (i.e. if you can’t ssh into the DRAC card and issue racadm serveraction hardreset, then you will need someone at the data center to power the macine off, wait 30 sec, then turn it back on. Make sure to re-attach via console (<tt>tip virtxx</tt>) immediately after power down. <br />
<br />
=== (Re)attach to the console ===<br />
Stay on the console the entire time during boot. As the BIOS posts- look out for the RAID card output- does everything look healthy? The output may be scrambled, look for "DEGRADED" or "FAILED". Once the OS starts booting you will be disconnected (dropped back to the shell on the console server) a couple times during the boot up. The reason you want to quickly re-attach is two-fold: 1. If you don’t reattach quickly then you won’t get any console output, 2. you want to be attached before the server ''potentially'' starts (an extensive) fsck. If you attach after the fsck begins, you’ll have seen no indication it started an fsck and the server will appear frozen during startup- no output, no response. <br />
<br />
=== Start containers/VE's/VPSs ===<br />
When the machine begins to start VE’s, it’s safe to leave the console and login via ssh. All virts should be set to auto start all the VEs after a crash. Further, most (newer) virts are set to “fastboot” it’s VE’s (to find out, do:<br />
grep -i fast /etc/sysconfig/vz <br />
and look for <tt>VZFASTBOOT=yes</tt>). If this was set prior to the machine’s crash (setting it after the machine boots will not have any effect until the vz service is restarted) it will start each ve as fast as possible, in serial, then go thru each VE (serially), shutting it down running a vzquota (disk usage) check, then bringing it back up. The benefit is that all VE’s are brought up quickly (within 15min or so depending on the #), the downside is a customer watching closely will notice 2 outages – 1st the machine crash, 2nd their quota check (which will be a much shorter downtime- on the order of a few minutes). <br />
<br />
Where “fastboot” is not set to yes (i.e on quar1), vz will start them consecutively, checking the quotas one at a time, and the 60th VE may not start until an hour or two later - this is not acceptable.<br />
<br />
The good news is, if you run vzctl start for a VE that is already started, you will simply get an error: <tt>VE is already started</tt>. Further, if you attempt to vzctl start a VE that is in the process of being started, you will simply get an error: unable to lock VE. So, there is no danger in simply running scripts to start smaller sets of VEs. If the system is not autostarting, then there is no issue, and even if it does, when it conflicts, one process (yours or the autostart) will lose, and just move on to the next one.<br />
<br />
A script has been written to assist with ve starts: [[#startvirt.pl|startvirt.pl]] which will start 6 ve’s at once until there are no more left. If startvirt.pl is used on a system where “fastboot” was on, it will circumvent the fastboot for ve’s started by startvirt.pl – they will go through the complete quota check before starting- therefore this is not advisable when a system has crashed. When a system is booted cleanly, and there's no need for vzquota checks, then startvirt.pl is safe and advisable to run.<br />
<br />
=== Make sure all containers are running ===<br />
You can quickly get a feel for how many ve’s are started by running:<br />
<br />
<pre>[root@virt4 log]# vs<br />
VEID 16066 exist mounted running<br />
VEID 16067 exist mounted running<br />
VEID 4102 exist mounted running<br />
VEID 4112 exist mounted running<br />
VEID 4116 exist mounted running<br />
VEID 4122 exist mounted running<br />
VEID 4123 exist mounted running<br />
VEID 4124 exist mounted running<br />
VEID 4132 exist mounted running<br />
VEID 4148 exist mounted running<br />
VEID 4151 exist mounted running<br />
VEID 4155 exist mounted running<br />
VEID 42 exist mounted running<br />
VEID 432 exist mounted running<br />
VEID 434 exist mounted running<br />
VEID 442 exist mounted running<br />
VEID 450 exist mounted running<br />
VEID 452 exist mounted running<br />
VEID 453 exist mounted running<br />
VEID 454 exist mounted running<br />
VEID 462 exist mounted running<br />
VEID 463 exist mounted running<br />
VEID 464 exist mounted running<br />
VEID 465 exist mounted running<br />
VEID 477 exist mounted running<br />
VEID 484 exist mounted running<br />
VEID 486 exist mounted running<br />
VEID 490 exist mounted running</pre><br />
<br />
So to see how many ve’s have started:<br />
<pre>[root@virt11 root]# vs | grep running | wc -l<br />
39</pre><br />
<br />
And to see how many haven’t:<br />
<pre>[root@virt11 root]# vs | grep down | wc -l<br />
0</pre><br />
<br />
And how many we should have running:<br />
<pre>[root@virt11 root]# vs | wc -l<br />
39</pre><br />
<br />
Another tool you can use to see which ve’s have started, among other things is [[#vzstat|vzstat]]. It will give you CPU, memory, and other stats on each ve and the overall system. It’s a good thing to watch as ve’s are starting (note the VENum parameter, it will tell you how many have started):<br />
<br />
<pre>4:37pm, up 3 days, 5:31, 1 user, load average: 1.57, 1.68, 1.79<br />
VENum 40, procs 1705: running 2, sleeping 1694, unint 0, zombie 9, stopped 0<br />
CPU [ OK ]: VEs 57%, VE0 0%, user 8%, sys 7%, idle 85%, lat(ms) 412/2<br />
Mem [ OK ]: total 6057MB, free 9MB/54MB (low/high), lat(ms) 0/0<br />
Swap [ OK ]: tot 6142MB, free 4953MB, in 0.000MB/s, out 0.000MB/s<br />
Net [ OK ]: tot: in 0.043MB/s 402pkt/s, out 0.382MB/s 4116pkt/s<br />
Disks [ OK ]: in 0.002MB/s, out 0.000MB/s<br />
<br />
VEID ST %VM %KM PROC CPU SOCK FCNT MLAT IP<br />
1 OK 1.0/17 0.0/0.4 0/32/256 0.0/0.5 39/1256 0 9 69.55.227.152<br />
21 OK 1.3/39 0.1/0.2 0/46/410 0.2/2.8 23/1860 0 6 69.55.239.60<br />
133 OK 3.1/39 0.1/0.3 1/34/410 6.3/2.8 98/1860 0 0 69.55.227.147<br />
263 OK 2.3/39 0.1/0.2 0/56/410 0.3/2.8 34/1860 0 1 69.55.237.74<br />
456 OK 17/39 0.1/0.2 0/100/410 0.1/2.8 48/1860 0 11 69.55.236.65<br />
476 OK 0.6/39 0.0/0.2 0/33/410 0.1/2.8 96/1860 0 10 69.55.227.151<br />
524 OK 1.8/39 0.1/0.2 0/33/410 0.0/2.8 28/1860 0 0 69.55.227.153<br />
594 OK 3.1/39 0.1/0.2 0/45/410 0.0/2.8 87/1860 0 1 69.55.239.40<br />
670 OK 7.7/39 0.2/0.3 0/98/410 0.0/2.8 64/1860 0 216 69.55.225.136<br />
691 OK 2.0/39 0.1/0.2 0/31/410 0.0/0.7 25/1860 0 1 69.55.234.96<br />
744 OK 0.1/17 0.0/0.5 0/10/410 0.0/0.7 7/1860 0 6 69.55.224.253<br />
755 OK 1.1/39 0.0/0.2 0/27/410 0.0/2.8 33/1860 0 0 192.168.1.4<br />
835 OK 1.1/39 0.0/0.2 0/19/410 0.0/2.8 5/1860 0 0 69.55.227.134<br />
856 OK 0.3/39 0.0/0.2 0/13/410 0.0/2.8 16/1860 0 0 69.55.227.137<br />
936 OK 3.2/52 0.2/0.4 0/75/410 0.2/0.7 69/1910 0 8 69.55.224.181<br />
1020 OK 3.9/39 0.1/0.2 0/60/410 0.1/0.7 55/1860 0 8 69.55.227.52<br />
1027 OK 0.3/39 0.0/0.2 0/14/410 0.0/2.8 17/1860 0 0 69.55.227.83<br />
1029 OK 1.9/39 0.1/0.2 0/48/410 0.2/2.8 25/1860 0 5 69.55.227.85<br />
1032 OK 12/39 0.1/0.4 0/80/410 0.0/2.8 41/1860 0 8 69.55.227.90</pre><br />
<br />
When you are all done, you will want to make sure that all the VEs really did get started, run vs one more time.<br />
<br />
Note the time all ve’s are back up and enter that into and save the crash log entry.<br />
<br />
Occasionally, a ve will not start automatically. The most common reason for a ve not to come up normally is the ve was at it’s disk limit before the crash, and will not start since they’re over the limit. To overcome this, set the disk space to current usage level (the system will give this to you when it fails to start), start the ve, then re-set the disk space back to the prior level. Lastly, contact the customer to let them know they’re out of disk (or allocate more disk if they're entitled to more).<br />
<br />
== Hitting performance barriers and fixing them ==<br />
<br />
There are multiple modes virtuozzo offers to allocate resources to a ve. We utilize 2: SLM and UBC parameters<br />
On our 4.x systems, we use all SLM – it’s simpler to manage and understand. There are a few systems on virt19/18 that may also use SLM. Everything else uses UBC. <br />
You can tell a SLM ve by:<br />
<br />
SLMMODE="all"<br />
<br />
in their conf file. <br />
<br />
TODO: detail SLM modes and parameters.<br />
<br />
If someone is in SLM mode and they hit memory resource limits, they simply need to upgrade to more memory.<br />
<br />
The following applies to everyone else (UBC).<br />
<br />
Customers will often email and say that they are getting out of memory errors - a common one is "cannot fork" ... basically, anytime you see something odd like this, it means they are hitting one of their limits that is in place in their conf file.<br />
<br />
The conf file, however, simply shows their limits - how do we know what they are currently at ?<br />
<br />
The answer is a file called v - this file contains the current status (and peaks) of their performance settings, and also counts how many times they have hit the barrier. The output of the file looks like this:<br />
<br />
<pre>764: kmemsize 384113 898185 8100000 8200000 0<br />
lockedpages 0 0 322 322 0<br />
privvmpages 1292 7108 610000 615000 0<br />
shmpages 270 528 33000 34500 0<br />
dummy 0 0 0 0 0<br />
numproc 8 23 410 415 0<br />
physpages 48 5624 0 2147483647 0<br />
vmguarpages 0 0 13019 2147483647 0<br />
oomguarpages 641 6389 13019 2147483647 0<br />
numtcpsock 3 21 1210 1215 0<br />
numflock 1 3 107 117 0<br />
numpty 0 2 19 19 0<br />
numsiginfo 0 4 274 274 0<br />
tcpsndbuf 0 80928 1800000 1900000 0 <br />
tcprcvbuf 0 108976 1800000 1900000 0<br />
othersockbuf 2224 37568 900000 950000 0<br />
dgramrcvbuf 0 4272 200000 200000 0<br />
numothersock 3 9 650 660 0<br />
dcachesize 53922 100320 786432 818029 0<br />
numfile 161 382 7500 7600 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
numiptent 4 4 155 155 0</pre><br />
<br />
The first column is the name of the counter in question - the same names we saw in the systems conf file. The second column is the _current_ value of that counter, the third column is the max that that counter has ever risen to, the fourth column is the soft limit, and the fifth column is the hard limit (which is the same as the numbers in that systems conf file).<br />
<br />
The sixth number is the failcount - how many times the current usage has risen to hit the barrier. It will increase as soon as the current usage hits the soft limit.<br />
<br />
The problem with /proc/user_beancounters is that it actually contains that set of data for every running VE - so you can't just cat /proc/user_beancounters - it is too long and you get info for every other running system.<br />
<br />
You can vzctl enter the system and run:<br />
<br />
vzctl enter 9999<br />
cat /proc/user_beancounters<br />
<br />
inside their system, and you will just see the stats for their particular system, but entering their system every time you want to see it is combersome.<br />
<br />
So, I wrote a simple script called "vzs" which simply greps for the VEID, and spits out the next 20 or so lines (however many lines there are in the output, I forget) after it. For instance:<br />
<br />
<pre># vzs 765:<br />
765: kmemsize 2007936 2562780 8100000 8200000 0<br />
lockedpages 0 8 322 322 0<br />
privvmpages 26925 71126 610000 615000 0<br />
shmpages 16654 16750 33000 34500 0<br />
dummy 0 0 0 0 0<br />
numproc 41 57 410 415 0<br />
physpages 1794 49160 0 2147483647 0<br />
vmguarpages 0 0 13019 2147483647 0<br />
oomguarpages 4780 51270 13019 2147483647 0<br />
numtcpsock 23 37 1210 1215 0<br />
numflock 17 39 107 117 0<br />
numpty 1 3 19 19 0<br />
numsiginfo 0 6 274 274 0<br />
tcpsndbuf 22240 333600 1800000 1900000 0<br />
tcprcvbuf 0 222656 1800000 1900000 0<br />
othersockbuf 104528 414944 900000 950000 0<br />
dgramrcvbuf 0 4448 200000 200000 0<br />
numothersock 73 105 650 660 0<br />
dcachesize 247038 309111 786432 818029 0<br />
numfile 904 1231 7500 7600 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
numiptent 4 4 155 155 0</pre><br />
<br />
That showed us just the portion of /proc/user_beancounters for system 765.<br />
<br />
When you run the vzs command, always add a : after the VEID.<br />
<br />
So, if a customer complains about some out of memory errors, or no more files, or no more ptys, or just has an unspecific complain about processes dying, etc., the very first thing you need to do is check their beancounters with vzs. Usually you will spot an item that has a high failcount and needs to be upped.<br />
<br />
At that point you could simply up the counter with `vzctl set`. Generally pick a number 10-20% higher than the old one, and make the hard limit slightly larger than the the soft limit. However our systems now come in several levels and those levels have more/different memory allocations. If someone is complaining about something other than a memory limit (pty, numiptent, numflock), it’s generally safe to increase it, at least to the same level as what’s in the /vzconf/4unlimited file on the newest virt. If someone is hitting a memory limit, first make sure they are given what they deserve:<br />
<br />
(refer to mgmt -> payments -> packages)<br />
<br />
To set those levels, you use the [[#setmem|setmem]] command. <br />
<br />
The alternate (DEPRECATED) method would be to use one of 3 commands:<br />
256 <veid><br />
300 <veid><br />
384 <veid><br />
512 <veid><br />
<br />
If the levels were not right (you’d run vzs <veid> before and after to see the effect) tell the customer they’ve been adjusted and be done with it. If the levels were right, tell the customer they must upgrade to a higher package, tell them how to see level (control panel) and that they can reboot their system to escape this lockup contidion.<br />
<br />
Customers can also complain that their site is totally unreachable, or complain that it is down ... if the underlying machine is up, and all seems well, you may notice in the beancounters that network-specific counters are failing - such as numtcpsock, tcpsndbuf or tcprcvbuf. This will keep them from talking on the network and make it seem like their system is down. Again, just up the limits and things should be fine.<br />
<br />
On virts 1-4, you should first look at the default settings for that item on a later virt, such as virt 8 - we have increased the defaults a lot since the early machines. So, if you are going to up a counter on virt2, instead of upping it by 10-20%, instead up it to the new default that you see on virt8.<br />
<br />
== Moving a VE to another virt (migrate/migrateonline) ==<br />
<br />
This will take a while to complete - and it is best to do this at night when the load is light on both machines.<br />
<br />
There are different methods for this, depending on which version of virtuozzo is installed on the src. and dst. virt. <br />
To check which version is running: <br />
[root@virt12 private]# cat /etc/virtuozzo-release<br />
Virtuozzo release 2.6.0<br />
<br />
Ok, let's say that the VE is 1212, and vital stats are:<br />
<pre>[root@virt12 sbin]# vc 1212<br />
VE_ROOT="/vz1/root/1212"<br />
VE_PRIVATE="/vz1/private/1212"<br />
OSTEMPLATE="fedora-core-2/20040903"<br />
IP_ADDRESS="69.55.229.84"<br />
TEMPLATES="devel-fc2/20040903 php-fc2/20040813 mysql-fc2/20040812 postgresql-fc2/20040813 mod_perl-fc2/20040812 mod_ssl-fc2/20040811 jre-fc2/20040823 jdk-fc2/20040823 mailman-fc2/20040823 analog-fc2/20040824 proftpd-fc2/20040818 tomcat-fc2/20040823 usermin-fc2/20040909 webmin-fc2/20040909 uw-imap-fc2/20040830 phpBB-fc2/20040831 spamassassin-fc2/20040910 PostNuke-fc2/20040824 sl-webalizer-fc2/20040<br />
818"<br />
<br />
[root@virt12 sbin]# vzctl exec 1212 df -h<br />
Filesystem Size Used Avail Use% Mounted on<br />
/dev/vzfs 4.0G 405M 3.7G 10% /</pre><br />
<br />
From this you can see that he’s using (and will minimally need free on the dst server) ~400MB, and he’s running on a Fedora 2 template, version 20040903. He’s also got a bunch of other templates installed. It’s is '''vital''' that '''all''' these templates exist on the dst system. To confirm that, on the dst system run:<br />
<br />
For < 3.0:<br />
<br />
<pre>[root@virt14 private]# vzpkgls | grep fc2<br />
devel-fc2 20040903<br />
PostNuke-fc2 20040824<br />
analog-fc2 20040824<br />
awstats-fc2 20040824<br />
bbClone-fc2 20040824<br />
jdk-fc2 20040823<br />
jre-fc2 20040823<br />
mailman-fc2 20040823<br />
mod_frontpage-fc2 20040816<br />
mod_perl-fc2 20040812<br />
mod_ssl-fc2 20040811<br />
mysql-fc2 20040812<br />
openwebmail-fc2 20040817<br />
php-fc2 20040813<br />
phpBB-fc2 20040831<br />
postgresql-fc2 20040813<br />
proftpd-fc2 20040818<br />
sl-webalizer-fc2 20040818<br />
spamassassin-fc2 20040910<br />
tomcat-fc2 20040823<br />
usermin-fc2 20040909<br />
uw-imap-fc2 20040830<br />
webmin-fc2 20040909<br />
[root@virt14 private]# vzpkgls | grep fedora<br />
fedora-core-1 20040121 20040818<br />
fedora-core-devel-1 20040121 20040818<br />
fedora-core-2 20040903<br />
[root@virt14 private]#</pre><br />
<br />
For these older systems, you can simply match up the date on the template. <br />
<br />
For >= 3.0:<br />
<br />
<pre>[root@virt19 /vz2/private]# vzpkg list<br />
centos-5-x86 2008-01-07 22:05:57<br />
centos-5-x86 devel<br />
centos-5-x86 jre<br />
centos-5-x86 jsdk<br />
centos-5-x86 mod_perl<br />
centos-5-x86 mod_ssl<br />
centos-5-x86 mysql<br />
centos-5-x86 php<br />
centos-5-x86 plesk9<br />
centos-5-x86 plesk9-antivirus<br />
centos-5-x86 plesk9-api<br />
centos-5-x86 plesk9-atmail<br />
centos-5-x86 plesk9-backup<br />
centos-5-x86 plesk9-horde<br />
centos-5-x86 plesk9-mailman<br />
centos-5-x86 plesk9-mod-bw<br />
centos-5-x86 plesk9-postfix<br />
centos-5-x86 plesk9-ppwse<br />
centos-5-x86 plesk9-psa-firewall<br />
centos-5-x86 plesk9-psa-vpn<br />
centos-5-x86 plesk9-psa-fileserver<br />
centos-5-x86 plesk9-qmail<br />
centos-5-x86 plesk9-sb-publish<br />
centos-5-x86 plesk9-vault<br />
centos-5-x86 plesk9-vault-most-popular<br />
centos-5-x86 plesk9-watchdog</pre><br />
<br />
On these newer systems, it's difficult to tell whether the template on the dst matches exactly the src. Just cause a centos-5-x86 is listed on both servers doesn't mean all the same packages are there on the dst. To truly know, you must perform a sample rsync:<br />
<br />
rsync -avn /vz/template/centos/5/x86/ root@10.1.4.61:/vz/template/centos/5/x86/<br />
<br />
if you see a ton of output from the dry run command, then clearly there are some differences. You may opt to let the rsync complete (without running in dry run mode) the only downside is you've now used up more space on the dst and also the centos template will be a mess with old and new data- it will be difficult if not impossible to undo (if someday we wanted to reclaim the space).<br />
<br />
If you choose to merge templates, you should closely inspect the dry run output. You should also take care to exclude anything in the /config directory. For example:<br />
<br />
rsync -av -e ssh --stats --exclude=x86/config /vz/template/ubuntu/10.04/ root@10.1.4.62:/vz/template/ubuntu/10.04/<br />
<br />
Which will avoid this directory and contents:<br />
<pre>[root@virt11 /vz2/private]# ls /vz/template/ubuntu/10.04/x86/config*<br />
app os</pre><br />
<br />
This is important to avoid since the config may differ on the destination and we are really only interested in making sure the pacakges are there, not overwriting a newer config with an older one.<br />
<br />
If the dst system was missing a template, you have 2 choices: <br />
# put the missing template on the dst system. 2 choices here: <br />
## Install the template from rpm (found under backup2: /mnt/data4/vzrpms/distro/) or <br />
## rsync over the template (found under /vz/template) - see above<br />
# put the ve on a system which has all the proper templates<br />
<br />
=== pre-seeding a migration ===<br />
<br />
When migrating a customer (or when doing many) depending on how much data you have to transfer, it can take some time. Further, it can be difficult to gauge when a migration will complete or how long it will take. To help speed up the process and get a better idea about how long it will take you can pre-transfer a customer's data to the destination server. If done correctly, vzmigrate will see the pre-transferred data and pick up where you left off, having much less to transfer (just changed/new files). <br />
<br />
We believe vzmigrate uses rsync to do it's transfer. Therefore not only can you use rsync to do a pre-seed, you can also run rsync to see what is causing a repeatedly-failing vzmigrate to fail. <br />
<br />
There's no magic to a pre-seed, you just need to make sure it's named correctly.<br />
<br />
Given:<br />
<br />
source: /vz1/private/1234<br />
<br />
and you want to migrate to /vz2 on the target system, your rsync would look like:<br />
<br />
rsync -av /vz1/private/1234/ root@x.x.x.x:/vz2/private/1234.migrated/<br />
<br />
After running that successful rsync, the ensuing migrateonline (or migrate) will take much less time to complete- depending on the # of files to be analyzed and the # of changed files. In any case, it'll be much much faster than had you just started the migration from scratch.<br />
<br />
Further, as we discuss elsewhere in this topic, a failed migration can be moved from <tt>/vz/private/1234</tt> to <tt>/vz/private/1234.migrated</tt> on the destination if you want to restart a failed migration. This should '''only''' be done if the migration failed and the CT is not running on the destination HN.<br />
<br />
=== migrateonline intructions: src >=3.x -> dst>=3.x ===<br />
<br />
A script called [[#migrateonline|migrateonline]] was written to handle this kind of move. It is basically a wrapper for <tt>vzmigrate</tt> – vzmigrate is a util to seamlessly- as no no reboot of the ve necessary- move a ve from one host to another. This wrapper was initially written cause vz virtuozzo version 2.6.0 has a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables, causing problems when the ve was started up on the dst system. [[#migrate|migrate]] mitigates that. Since it makes multiple ssh connections to the dst virt, it’s a good idea to put the pub key for the src virt in the authorized_keys file on the dst virt. In addition, migrateonline emails ve owners when their migration starts and stops. For this to happen they need to put email addresses (on a single line, space delimited) in a file on their system: /migrate_notify. If the optional target dir is not specified, the ve will be moved to the same private/root location as it was on the src virt. Note: <tt>migrate</tt> is equivalent to <tt>migrateonline</tt>, but will <tt>migrate</tt> a ve AND restart it in the process.<br />
<br />
<pre>[root@virt12 sbin]# migrateonline<br />
usage: /usr/local/sbin/migrateonline <ip of node migrating to> <veid> [target dir: vz | vz1 | vz2]<br />
[root@virt12 sbin]# migrateonline 10.1.4.64 1212 vz<br />
starting to migrate 1212 at Sat Mar 26 22:40:38 PST 2005<br />
Turning off offline_management<br />
Saved parameters for VE 1212<br />
migrating with no start on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
clearing the arp cache<br />
now going to 10.1.4.64 and clear cache and starting<br />
starting it<br />
Delete port redirection<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
finished migrating 1212 at Sat Mar 26 22 22:52:01 PST 2005<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine.<br />
<br />
If they had backups, use the mvbackups command to move their backups to the new server:<br />
<br />
mvbackups 1212 virt14 vz<br />
<br />
Rename the ve<br />
<br />
[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/migrated-1212<br />
<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/old-1212-migrated-20120404-noarchive<br />
<br />
Update the customer’s systems in mgmt to reflect the new path and server.<br />
<br />
<br />
IF migrateonline does not work, you can try again using simply migrate- this will result in a brief reboot for the ve.<br />
Before you try again, make sure of a few things:<br />
<br />
Depending on where in the migration died, there may be partial data on the dst system in 1 of 2 places:<br />
(given the example above)<br />
<br />
/vz/private/1212<br />
<br />
or <br />
<br />
/vz/private/1212.migrated<br />
<br />
before you run migrate again, you'll want to rename so that all data is in <br />
1212.migrated:<br />
<br />
mv /vz/private/1212 /vz/private/1212.migrated<br />
<br />
this way, it will pick up where it left off and transfer only new files.<br />
<br />
Likewise, if you want to speed up a migration, you can pre-seed the dst as follows:<br />
<br />
[root@virt12 sbin]# rsync -avSH /vz/private/1212/ root@10.1.4.64:/vz/private/1212.migrated/<br />
<br />
then when you run migrate or migrateonline, it will only need to move the changed files- the migration will complete quickly<br />
<br />
=== migrate manually (if migrate/online fails) ===<br />
<br />
Lets say for whatever reason the migration fails. You can always move someone manually:<br />
<br />
1. stop ve: <br><br />
v stop 1234<br />
<br />
2. copy over data<br><br />
rsync -avSH /vz/private/1234/ root@1.1.1.1:/vzX/private/1234/<br />
<br />
NOTE: if you've previously seeded the data (run rsync while the VE was up/running), and this is a subsequent rsync, make sure the last rsync you do (while the VE is not running, has the --delete option in the rsync)<br />
<br />
3. copy over conf<br><br />
scp /vzconf/1234.conf root@1.1.1.1:/vzconf<br />
<br />
4. on dst, edit the conf to reflect the right vzX dir<br><br />
vi /vzconf/1234.conf<br />
<br />
5. on src remove the IPs<br><br />
ipdel 1234 2.2.2.2 3.3.3.3<br />
<br />
6. on dst add IPs <br><br />
ipadd 1234 2.2.2.2 3.3.3.3<br />
<br />
7. on dst, start ve: <br><br />
v start 1324<br />
<br />
8. cancel, then archive ve on src per above instrs.<br />
<br />
=== migrate src=2.6.0 -> dst>=2.6.0, or mass-migration with customer notify ===<br />
<br />
A script called <tt>migrate</tt> was written to handle this kind of move. It is basically a wrapper for vzmigrate – vzmigrate is a util to seamlessly move a ve from one host to another. This wrapper was initially written cause vz virtuozzo version 2.6.0 has a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables, causing problems when the ve was started up on the dst system. migrate mitigates that. Since it makes multiple ssh connections to the dst virt, it’s a good idea to put the pub key for the src virt in the authorized_keys file on the dst virt. In addition, migrate emails ve owners when their migration starts and stops. For this to happen they need to put email addresses (on a single line, space delimited) in a file on their system: /migrate_notify. If the optional target dir is not specified, the ve will be moved to the same private/root location as it was on the src virt. Note: migrateonline is equivalent to migrate, but will migrate a ve from one 2.6 '''kernel''' machine to another 2.6 kernel machine without restarting the ve.<br />
<br />
<pre>[root@virt12 sbin]# migrate<br />
usage: /usr/local/sbin/migrate <ip of node migrating to> <veid> [target dir: vz | vz1 | vz2]<br />
[root@virt12 sbin]# migrate 10.1.4.64 1212 vz<br />
starting to migrate 1212 at Sat Mar 26 22:40:38 PST 2005<br />
Turning off offline_management<br />
Saved parameters for VE 1212<br />
migrating with no start on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
clearing the arp cache<br />
now going to 10.1.4.64 and clear cache and starting<br />
starting it<br />
Delete port redirection<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
finished migrating 1212 at Sat Mar 26 22 22:52:01 PST 2005<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine (backup2 or mail).<br />
Cancel the ve (first we have to rename things which migrate changed so cancelve will find them):<br />
<br />
[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/1212.conf<br />
<br />
On 2.6.1 you’ll also have to move the private area:<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/1212<br />
<br />
<pre>[root@virt12 sbin]# cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, [[#cancelve|cancelve]] would offer to remove them. You want to say '''no''' to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
To move everyone off a system, you’d do:<br />
for f in `vl`; do migrate <ip> $f; done<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
=== vzmigrate: src=2.6.1 -> dst>=2.6.0 ===<br />
<br />
This version of vzmigrate works properly with regard to handling ips. It will not notify ve owners of moves as in the above example. Other than that it’s essentially the same.<br />
<br />
<pre>[root@virt12 sbin]# vzmigrate 10.1.4.64 -r no 1212:1212:/vz/private/1212:/vz/root/1212<br />
migrating on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine (backup2 or mail).<br />
Cancel the ve (first we have to rename things which vzmigrate changed so cancelve will find them):<br />
<br />
<pre>[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/1212.conf<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/1212<br />
<br />
[root@virt12 sbin]# cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, <tt>cancelve</tt> would offer to remove them. You want to say no to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
=== src=2.5.x ===<br />
<br />
First, go to the private dir:<br />
<br />
cd /vz1/private/<br />
<br />
Stop the VE - make sure it stops totally cleanly.<br />
<br />
vzctl stop 1212<br />
<br />
Then you’d use vemove - a script written to copy over the config, create tarballs of the ve’s data on the destination virt, and cancel the ve on the source system (in this example we’re going to put a ve that was in /vz1/private on the src virt, in /vz/private on the dst virt):<br />
<br />
<pre>[root@virt12 sbin]# vemove<br />
ERROR: Usage: vemove veid target_ip target_path_dir<br />
[root@virt12 sbin]# vemove 1212 10.1.4.64 /vz/private/1212<br />
tar cfpP - 1212 --ignore-failed-read | (ssh -2 -c arcfour 10.1.4.64 "split - -b 1024m /vz/private/1212.tar" )<br />
scp /vzconf/1212.conf 10.1.4.64:/vzconf<br />
cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, cancelve would offer to remove them. You want to say no to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
When you are done, go to /vz/private on the dst virt you will have files like this:<br />
<br />
<pre>1212.taraa<br />
1212.tarab<br />
1212.tarac</pre><br />
<br />
Each one 1024m (or less, for the last one) in size.<br />
<br />
on the dst server and run:<br />
<br />
cat 1212.tar?? | tar xpPBf -<br />
<br />
and after 20 mins or so it will be totally untarred. Now since the conf<br />
file is already there, you can go ahead and start the system.<br />
<br />
vzctl start 1212<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
NOTE: you MUST tar the system up using the virtuozzo version of tar that<br />
is on all the virt systems, and further you MUST untar the tarball with<br />
the virtuozzo tar, using these options: `<tt>tar xpPBf -</tt>`<br />
<br />
If you tar up an entire VE and move it to a non-virtuozzo machine, that is<br />
ok, and you can untar it there with normal tar commands, but do not untar<br />
it and then repack it with a normal tar and expect it to work - you need<br />
to use virtuozzo tar commands on virtuozzo tarballs to make it work.<br />
<br />
The backups are sort of an exception, since we are just (usually)<br />
restoring user data that was created after we gave them the system, and<br />
therefore has nothing to do with magic symlinks or vz-rpms, etc.<br />
<br />
== Moving a VE on the same virt ==<br />
<br />
Easy way:<br><br />
Scenario 1: ve 123 is to be renamed 1231 and moved from vz1 to vz<br />
<br />
vzmlocal 123:1231:/vz/private/1231:/vz/root/1231<br />
<br />
Scenario 2: ve 123 is to be moved vz1 to vz<br />
<br />
vzmlocal 123:123:/vz/private/123:/vz/root/123<br />
<br />
vzmlocal will reboot the ve at the end of the move<br />
<br />
Manual/old way:<br />
<br />
1) <tt>vzctl stop 123</tt><br><br />
2) <tt>mv /vz1/private/123 /vz/private/.</tt><br><br />
(or cp -a if you want to copy)<br />
3) in <tt>/etc/sysconfig/vz-scripts/123.conf</tt> change value<br><br />
of '<tt>VE_PRIVATE</tt>' variable to point to a new private area location<br />
4) <tt>vzctl start 123</tt><br><br />
5) update backups if needed: <tt>mvbackups 123 virtX virt1 vz</tt><br><br />
6) update management scerens<br><br />
<br />
Notes: a) absolute path to private area is stored in quota file <tt>/var/vzquota/quota.123</tt> - so during first startup quota will be recalculated.<br><br />
b) if you're going to write some script to do a job, you MUST be sure that $VEID won't be expanded to '' in ve config file - ie. you need to escape '$'. Otherwise you might have:<br />
<br />
VE_PRIVATE="/vz/private/"<br />
<br />
in config, and 'vzctl destroy' for this VE ID '''will remove everything under /vz/private/ directory'''.<br />
<br />
== Adding a veth device to a VE ==<br />
<br />
Not totally sure what this is, but a customer asked for it and here's what we did (as instructed by vz support):<br />
<br />
<pre>v set 99 --netif_add eth99 --save<br />
ipdel 99 69.55.230.58<br />
v set 99 --ifname eth99 --ipadd 69.55.230.58 --save<br />
v set 99 --ifname eth99 --gateway 69.55.230.1 --save<br />
<br />
vznetcfg net new veth_net<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
net99 active eth0 veth77.77,veth99.99<br />
veth_net active<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
br0 bridge veth_net<br />
br99 bridge net99<br />
veth99.99 veth net99<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic net99 69.55.227.70/24<br />
<br />
vznetcfg br attach br0 eth0<br />
<br />
(will remove 99 from orig net and move to veth_net)<br />
vznetcfg net addif veth_net veth99.99<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
net99 active<br />
veth_net active eth0 veth77.77,veth99.99<br />
<br />
(delete the old crap)<br />
vznetcfg net del net99<br />
<br />
<br />
Then, to add another device in<br />
<br />
v set 77 --netif_add eth77 --save<br />
ipdel 77 69.55.230.78<br />
v set 77 --ifname eth77 --ipadd 69.55.230.78 --save<br />
v set 77 --ifname eth77 --gateway 69.55.230.1 --save<br />
v set 77 --save --ifname eth77 --network veth_net<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
veth77.77 veth<br />
br0 bridge veth_net<br />
veth99.99 veth veth_net<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic veth_net 69.55.227.70/24<br />
<br />
vznetcfg net addif veth_net veth77.77<br />
<br />
# vznetcfg if list<br />
Name Type Network ID Addresses<br />
veth77.77 veth veth_net<br />
br0 bridge veth_net<br />
veth99.99 veth veth_net<br />
eth1 nic 10.1.4.62/24<br />
eth0 nic veth_net 69.55.227.70/24<br />
<br />
# vznetcfg net list<br />
Network ID Status Master Interface Slave Interfaces<br />
veth_net active eth0 veth77.77,veth99.99<br />
<br />
another example<br />
<br />
v set 1182 --netif_add eth1182 --save<br />
ipdel 1182 69.55.236.217<br />
v set 1182 --ifname eth1182 --ipadd 69.55.236.217 --save<br />
v set 1182 --ifname eth1182 --gateway 69.55.236.1 --save<br />
vznetcfg net addif veth_net veth1182.1182<br />
v set 1182 --save --ifname eth1182 --network veth_net<br />
<br />
<br />
unused/not working commands:<br />
ifconfig veth99.0 0<br />
vznetcfg net list<br />
vznetcfg br new br99 net99<br />
vznetcfg br attach br99 eth0<br />
vznetcfg br show<br />
vznetcfg if list<br />
vznetcfg net addif net99 veth99.99<br />
vznetcfg net addif eth0 net99<br />
<br />
---------<br />
<br />
vznetcfg br new br1182 net1182<br />
<br />
vznetcfg br attach br99 eth0<br />
vznetcfg if list<br />
vznetcfg net addif net99 veth99.99<br />
vznetcfg net addif eth0 net99<br />
<br />
<br />
<br />
vznetcfg net addif eth0 net1182<br />
<br />
<br />
# vznetcfg net addif veth99.0 net99<br />
--- 8< ---<br />
<br />
vznetcfg net new net<br />
# vznetcfg net addif eth0 net99<br />
<br />
<br />
# vzctl set 99 --save --netif_add eth0 (at this stage veth99.0 interface have to appear<br />
on node)<br />
# vzctl set 99 --save --ifname eth0 --ipadd 69.55.230.58 (and probably few more arguments<br />
here - see 'man vzctl')<br />
# vznetcfg net addif veth99.0 net99</pre><br />
<br />
<br />
== Assigning/remove ip from a VE ==<br />
<br />
1. Add or remove ips:<br />
ipdel 1234 1.1.1.1 2.2.2.2<br />
ipadd 1234 1.1.1.1 2.2.2.2<br />
2. update Mgmt screens<br />
3. offer to update any DNS we do for them<br />
4. check to see if we had rules for old IP in firwall<br />
<br />
<br />
== Enabling tun device for a ve ==<br />
Note, there’s a command for this: [[#addtun|addtun]]<br />
<br />
For posterity:<br />
Make sure the tun.o module is already loaded before Virtuozzo is started: <br />
lsmod <br />
Allow the VPS to use the TUN/TAP device: <br />
vzctl set 101 --devices c:10:200:rw --save <br />
Create the corresponding device inside the VPS and set the proper permissions: <br />
vzctl exec 101 mkdir -p /dev/net <br />
vzctl exec 101 mknod /dev/net/tun c 10 200 <br />
vzctl exec 101 chmod 600 /dev/net/tun<br />
<br />
== Remaking a system (on same virt) ==<br />
<br />
1. [[#cancelve|cancelve]] (or v destroy x - ONLY if you're POSITIVE no data needs to be saved)<br />
2. [[#vemake|vemake]] using same veid<br />
3. [[#mvbackups|mvbackups]] or [[#vb|vb]] (if new mount point)<br />
4. update mgmt with new dir/ip <br />
5. update firewall if changed ip<br />
<br />
<br />
== Traffic accounting on linux ==<br />
<br />
DEPRECATED - all tracking is done via bwdb now. This is how we used to track traffic.<br />
<br />
TODO: update for diff versions of vz<br />
<br />
Unlike FreeBSD, where we have to add firewall count rules to the system to count the traffic, on virtuozzo counts the traffic for us. You an see the current traffic stats by running `vznetstat`:<br />
<br />
<pre># vznetstat<br />
VEID Net.Class Output(bytes) Input(bytes)<br />
-----------------------------------------------<br />
24218 1 484M 39M<br />
24245 1 463M 143M<br />
2451 1 2224M 265M<br />
2454 1 2616M 385M<br />
4149 1 125M 68M<br />
418 1 1560M 34M<br />
472 1 1219M 315M<br />
726 1 628M 317M<br />
763 1 223M 82M<br />
771 1 4234M 437M<br />
-----------------------------------------------<br />
Total: 13780M 2090M</pre><br />
<br />
As you can see the VEID is on a line with the in and out bytes. So, we simply run a cron job:<br />
<br />
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /root/vztrafdump.sh<br />
<br />
Just like we do on FreeBSD - this one goes through all the VEs in /vz/private and greps the line from vznetstat that matches them and dumps it in /jc_traffic_dump on their system. Then it does it again for all the VEs in /vz1/private. It is important to note that vznetstat runs only once, and the grepping is done from a temporary file that contains that output - we do this because running vznetstat once for each VE that we read out of /vz/private and /vz1/private would take way too long and be too intensive.<br />
<br />
You do not need to do anything to facilitate this other than make sure that that cron job is running - the vznetstat counters are always running, and any new VEs that are added to the system will be accounted for automatically.<br />
<br />
Traffic resetting no longer works with vz 2.6, so we disable the vztrafdump.sh on those virts.<br />
<br />
== Watchdog script ==<br />
<br />
On some of the older virts, we have a watchdog running that kills procs that are deemed bad per the following:<br />
<br />
/root/watchdog from quar1<br />
<br />
<pre>if echo $line | awk '{print $(NF-3)}' | grep [5-9]...<br />
then<br />
# 50-90%<br />
if echo $line | awk '{print $(NF-1)}' | grep "...:.."<br />
then<br />
# running for > 99min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
if echo $line | awk '{print $(NF-1)}' | grep "....m"<br />
then<br />
# running for > 1000min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
fi<br />
<br />
if echo $line | awk '{print $(NF-3)}' | grep [1-9]...<br />
then<br />
# running for 10-90 percent<br />
if echo $line | awk '{print $NF}' | egrep 'cfusion|counter|vchkpw'<br />
then<br />
<br />
if echo $line | awk '{print $(NF-1)}' | grep "[2-9]:.."<br />
then<br />
# between 2-9min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
elif echo $line | awk '{print $(NF-1)}' | grep "[0-9][0-9]:.."<br />
then<br />
# up to 99min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
fi<br />
fi</pre><br />
<br />
<br />
== Misc Linux Items ==<br />
<br />
We are overselling hard drive space ... when you configure a linux system with a certain amount of disk space (the default is 4gigs) you do not actually use up 4gigs of space on the system. The diskspace setting for a user is simply a cap, and they only use up as much space on the actual disk drive as they are actually using.<br />
<br />
When you create a new linux system, even though there are some 300 RPMs or so installed, if you run `df -k` you will see that the entire 4gig partition is empty - no space is being used. This is because the files in their system are "magic symlinks" to the template for their OS that is in /vz/template - however, any changes to any of those files will "disconnect" them and they will immediately begin using space in their system. Further, any new files uploaded (even if those new files overwrite existing files) will take up space on the partition.<br />
<br />
<br />
if you see this:<br />
<br />
<pre>[root@virt8 root]# vzctl stop 160 ; vzctl start 160<br />
VE is not running<br />
Starting VE ...<br />
VE is unmounted<br />
VE is mounted<br />
Adding IP address(es): 69.55.226.83 69.55.226.84<br />
bash ERROR: Can't change file /etc/sysconfig/network<br />
Deleting IP address(es): 69.55.226.83 69.55.226.84<br />
VE is unmounted<br />
[root@virt8 root]#</pre><br />
<br />
it probably means they no longer have /bin/bash - copy one in for them<br />
<br />
ALSO: another possibility is that they have removed the `ed` RPM from their system - it needs to be reinstalled into their system. But since their system is down, this is tricky ...<br />
<br />
VE startup scripts used by 'vzctl' want package 'ed' to be available inside VE. So if package 'ed' will be enabled in OS template config and OS template itself VE #827 is based on - this error should be fixed.<br />
<br />
yes, it is possible to add RPM to VE while it not running.<br />
Try to do following:<br />
<br />
<pre># cd /vz/template/<OS_template_with_ed_package>/<br />
# vzctl mount 827<br />
# rpm -Uvh --root /vz/root/827 --veid 827 ed-0.2-25.i386.vz.rpm</pre><br />
<br />
Usually theres an error, but its ok<br />
<br />
Note: replace 'ed-0.2-25.i386.vz.rpm' in last command with actual<br />
version of 'ed' package you have.<br />
<br />
----<br />
<br />
So how do I know what template the user has ? cat their conf file and it is listed in there. For example, if the conf file has:<br />
<br />
<pre>[root@virt12 sbin]# vc 1103<br />
…snip…<br />
OSTEMPLATE="debian-3.0/20030822"<br />
TEMPLATES="mod_perl-deb30/20030707 mod_ssl-deb30/20030703 mysql-deb30/20030707 proftpd-deb30/20030703 webmin-deb30/20030823 "<br />
…</pre><br />
<br />
then they are on debian 3.0, all of their system RPMs are in /vz/template/debian-3.0, and they are using version 20030822 of that debian 3.0 template. Also, they’ve also got additional packages installed (mod_perl, mod_ssl, etc). Those are also found under /vz/template<br />
<br />
----<br />
<br />
Edits needed to run java:<br />
<br />
When we first created the VEs, the default setting for privvmpages was 93000:94000 ... which was high enough that most people never had problems ... however, you can;t run java or jdk or tomcat or anything java related with that setting. We have found that by setting privvmpages to 610000:615000 that java runs just fine. That is now the default setting. It is exceedingly rare that anyone needs it higher than that, although we have seen it once or twice.<br />
<br />
Any problems with java at all - the first thing you need to do is see if the failcnt has raised for privvmpages.<br />
<br />
<pre># vzctl start 160<br />
Starting VE ...<br />
vzquota : (error) Quota on syscall for 160: Device or resource busy<br />
Running vzquota on failed for VE 160 [3]</pre><br />
<br />
This is because my pwd is _in_ their private directory - you can't start it until you move out<br />
<br />
People seem to have trouble with php if they are clueless newbies. Here are two common problems/solutions:<br />
<br />
no... but i figured it out myself. problem was the php.ini file that came<br />
vanilla with the account was not configured to work with apache (the<br />
ENGINE directive was set to off).<br />
<br />
everything else seems fine now.<br />
<br />
----<br />
<br />
the problem was in the php.ini file. I noticed that is wasnt showing<br />
the code when it was in an html file so I looked at the php.ini file<br />
and had to change it so it recognized <? tags aswell as <?php tags.<br />
<br />
Also, make sure added to httpd.conf<br />
AddType application/x-httpd-php .php<br />
<br />
----<br />
<br />
You can set the time zone:<br />
<br />
You can change the timezone by doing this:<br />
<br />
ln -sf /usr/share/zoneinfo/<zone> /etc/localtime<br />
<br />
where <zone> is the zone you want in the /usr/share/zoneinfo/ directory.<br />
<br />
----<br />
<br />
Failing shm_open calls:<br />
<br />
first, please check if /dev/shm is mounted inside VE.<br />
'cat /proc/mounts' command should show something like this:<br />
tmpfs /dev/shm tmpfs rw 0 0<br />
<br />
If /dev/shm is not mounted you have 2 ways to solve issue:<br />
1. execute following command inside VE (doesn't require VE reboot):<br />
mount -t tmpfs none /dev/shm<br />
2. add following string to /etc/fstab inside VE and reboot it:<br />
tmpfs /dev/shm tmpfs defaults 0 0<br />
<br />
----<br />
<br />
You can have a mounted but not running ve<br />
Just:<br />
vzctl mount <veid><br />
<br />
----<br />
<br />
When a debian sys can’t get on the network, and you try:<br />
<br />
<pre>vzctl set 1046 --ipadd 69.55.227.117<br />
Adding IP address(es): 69.55.227.117<br />
Failed to bring up lo.<br />
Failed to bring up venet0.</pre><br />
<br />
They probably removed iproute package, which must be the one from swsoft. To restore:<br />
<pre># dpkg -i --veid=1046 --admindir=/vz1/private/1046/root/var/lib/dpkg --instdir=/vz1/private/1046/root/ /vz/template/debian-3.0/iproute_20010824-8_i386.vz.deb<br />
(Reading database ... 16007 files and directories currently installed.)<br />
Preparing to replace iproute 20010824-8 (using .../iproute_20010824-8_i386.vz.deb) ...<br />
Unpacking replacement iproute ...<br />
Setting up iproute (20010824-8) ...</pre><br />
<br />
Then restart their ve<br />
<br />
----<br />
<br />
in a ve i do:<br />
<br />
<pre>cd /<br />
du -h .</pre><br />
<br />
and get: 483M .<br />
<br />
i do:<br />
<br />
<pre>bash-2.05a# df -h<br />
Filesystem Size Used Avail Use% Mounted on<br />
/dev/vzfs 4.0G 2.3G 1.7G 56% /</pre><br />
<br />
how can this be?<br />
<br />
Is it possible that quota file was corrupted somehow? Please try to: <br />
<pre>vzctl stop <VEID><br />
vzquota drop <VEID><br />
vzquota init <VEID><br />
vzctl start <VEID></pre><br />
<br />
----<br />
<br />
How to stop vz from starting after reboot:<br />
<br />
VIRTUOZZO=no <br />
in <br />
/etc/sysconfig/vz<br />
<br />
To start: <br />
service vz start<br />
(after setting VIRTUOZZO=yes in /etc/sysconfig/vz)<br />
<br />
service vz restart will do some kind of 'soft reboot' -- restart all<br />
VPSes and reload modules without rebooting the node<br />
<br />
if you need to shut down all VPSes really really fast, run killall -9 init<br />
<br />
----<br />
<br />
Postfix tip:<br />
<br />
You may want to tweak settings: default_process_limit=10<br />
<br />
= Virtuozzo VPS Management Tools =<br />
<br />
== vm ==<br />
<br />
TODO<br />
<br />
== cancelve ==<br />
cancelve <veid><br />
this will:<br />
* stop a ve<br />
* check for backups (offer to remove them from the backup server <br />
and the backup.config)<br />
* rename the private dir<br />
* check for PTR, provide the commands to reset to default<br />
* and rename the ve’s config<br />
* remind you to remove firewall rules<br />
* remind you to remove DNS entries<br />
<br />
== ipadd ==<br />
ipadd <veid> <ip> [ip] [ip]<br />
add’s ip(s) to a ve<br />
<br />
== ipdel ==<br />
ipdel <veid> <ip> [ip] [ip]<br />
removes ip(s) from a ve<br />
<br />
== vc ==<br />
vc <veid><br />
the equivalent of: <tt>cat /vzconf/<veid>.conf</tt><br />
<br />
== vl ==<br />
vl<br />
will displays a list of ve #’s, 1 per line. (ostensibly to use in a for loop)<br />
<br />
== vp ==<br />
vp <veid><br />
the equivalent of: <tt>vzps auxww –E <veid></tt><br />
<br />
== vpe ==<br />
DEPRECATED<br />
vpe <veid> <br />
this will allow you to do a vp when a ve is running out of control, the equivalent of (deprecated since vp operates outside the VPS): <br />
<pre>vzctl set <veid> --kmemsize 2100000:2200000<br />
vzctl exec <veid> ps auxw<br />
vzctl set <veid> --kmemsize (ve’s orig lvalue):(ve’s orig hvalue)</pre><br />
<br />
== vt ==<br />
vt <veid><br />
the equivalent of: <tt>vztop –E <veid></tt><br />
<br />
== vr ==<br />
vr <veid><br />
the equivalent of: <tt>vzctl stop <veid>; vzctl start <veid></tt><br />
You can run this even if the ve is down - the stop command will just fail<br />
<br />
== vs ==<br />
vs [veid]<br />
displays status (output of <tt>vzctl status <veid></tt>) on each ve configured on the system (as determined by <tt>/vzconf/*.conf</tt>)<br />
If passed an argument, gives the status for just that ve. <br />
A running system looks like:<br />
<tt>VEID 16066 exist mounted running</tt><br />
<br />
A ve which is not running (but does exist) looks like:<br />
<tt>VEID 9990 exist unmounted down</tt><br />
<br />
A ve which is not running and doesn’t exist looks like:<br />
<tt>VEID 421 deleted unmounted down</tt><br />
<br />
== vs2 ==<br />
vs2 [veid]<br />
this is similar to vs in that it displays status (output of <tt>vzctl status <veid></tt>) on each ve,<br />
but the difference is it’s list comes from doing an ls on the data dirs. This was meant to catch <br />
the rare case where a ve configured but exists. <br />
<br />
== vw ==<br />
vw [veid]<br />
displays the output of ‘<tt>w</tt>’ (the equivalent of <tt>vzctl exec <veid> w</tt>) for each configured ve (as determined by <tt>/vzconf/*.conf</tt>). Useful for determing which ve is contributing to a heavily-loaded system.<br />
If passed an argument, gives ‘<tt>w</tt>‘ output for just that ve. <br />
Ex:<br />
<pre>[root@virt2 etc]# vw<br />
134<br />
10:52pm up 79 days, 6:14, 0 users, load average: 0.02, 0.02, 0.00<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT<br />
16027<br />
2:52pm up 7 days, 19:54, 0 users, load average: 0.00, 0.00, 0.00<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT<br />
16055<br />
2:52pm up 79 days, 6:38, 0 users, load average: 0.00, 0.04, 0.07<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT</pre><br />
<br />
== vwe ==<br />
vwe [constraint]<br />
just like <tt>vw</tt>, but takes a constraint as an argument, only show’s ve’s with loads >= the constraint provided. If no constraint is provided, 1 is used by default<br />
<br />
== vzs ==<br />
vzs [veid]<br />
displays the beancounter status for all ve’s, or a particular ve if an argument is passed<br />
<br />
== ve ==<br />
ve <veid><br />
the equivalent of: <tt>vzctl enter <veid></tt><br />
<br />
== vx ==<br />
vx <veid> <command><br />
the equivalent of: <tt>/usr/sbin/vzctl exec <veid> <command></tt><br />
<br />
== dprocs ==<br />
dprocs [count]<br />
a script which outputs a continuous report (or a certain number of reports if an option is passed) of processes stuck in the D state and which VPS’s those procs belong to.<br />
<br />
== setmem ==<br />
setmem <VEID> <256|512|768|1024|2048><br />
adjusts the memory resources for the VE<br />
<br />
== afacheck.sh ==<br />
afacheck.sh<br />
displays the health/status of containers and mirrors on an adaptec card (currently quar1, tempvirt1-2, virt9, virt10)- all other are LSI<br />
<br />
== backup ==<br />
backup<br />
backup script called nightly to update virt scripts and do customer backups<br />
<br />
== checkload.pl ==<br />
checkload.pl<br />
this was intended to be setup as a cronjob to watch processes on a virt when the load <br />
rises above a certain level. Not currently in use.<br />
<br />
== findbackuppigs.pl ==<br />
findbackuppigs.pl<br />
looks for files larger than 50MB which customers have asked us to backup. Emails matches<br />
to linux@johncompanies.com<br />
<br />
== gatherlinux.pl ==<br />
gatherlinux.pl<br />
gathers up data about ve’s configured and writes to a file. Used for audits against the db<br />
<br />
== gb ==<br />
gb <search><br />
greps backup.config for the given search parameter<br />
<br />
== gbg ==<br />
gbg <search><br />
greps backup.config for the given search parameter and presents just the directories (for clean pasting)<br />
<br />
== linuxtrafficgather.pl ==<br />
linuxtrafficgather.pl [yy-mm]<br />
sends a traffic usage summary by ve to support@johncomapnies.com and payments@johncompanies.com.<br />
Optional arguments are year and month (must be in the past). If not passed, assumes last month. Relies on <br />
traffic logs created by netstatreset and netstatbackup<br />
<br />
== linuxtrafficwatch.pl ==<br />
linuxtrafficwatch.pl<br />
checks traffic usage and emails support@johncomapnies.com when a ve reaches the warning level (35G) and<br />
the limit (40G). Works on virtuozzo versions <= 2.6.x. We really aren’t using this anymore now that we have netflow.<br />
<br />
== linuxtrafficwatch2.pl ==<br />
linuxtrafficwatch2.pl<br />
checks traffic usage and emails support@johncomapnies.com when a ve reaches the warning level (35G) and<br />
the limit (40G). Works on virtuozzo version 2.6.x. We really aren’t using this anymore now that we have netflow.<br />
<br />
== load.pl ==<br />
load.pl<br />
feeds info to load mrtg - executed by inetd.<br />
<br />
== mb (linux) ==<br />
mb <mount|umount><br />
(nfs) mounts and umounts dirs to backup2. Shortcuts are mbm and mbu to mount and unmount. <br />
<br />
== migrate ==<br />
migrate <ip of node migrating to> <veid> <target_veid> [target dir: vz | vz1 | vz2]<br />
this is basically a wrapper for <tt>vzmigrate</tt> – vzmigrate is a util to seamlessly move a ve from one host to another. This wrapper was written cause vz virtuozzo version 2.6 had a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables. This script mitigates that. Since it makes multiple ssh connections to the target host, it’s a good idea to put the pub key for the src system in the authorized_keys file on the target host. In addition, it emails ve owners when their migration starts and stops (if they place email addresses in a file on their system: /migrate_notify). To move everyone off a system, you’d do:<br />
for f in `vl`; do migrate <ip> $f; done<br />
<br />
== migrateonline ==<br />
migrateonline <ip of node migrating to> <veid> <target_veid> [target dir: vz | vz1 | vz2]<br />
this is the same as migrate but will migrate a ve in <tt>–online</tt> mode which means it won’t be shut down at the end of the migration. This only works when migrating ve’s between 2 machines running a 2.6 kernel (currently tempvirt1-2. virt16-19, virt12). If you get an error that the machine you’re trying to migrate to has a different CPU or features, etc, then you have to edit the file and add the –f switch to the vzmigrate line- you can basically ignore this kind of warning (but never ignore a warning about missing templates on the destination node). NOTE: This edit (if made to migrateonline) will be overwritten by the base script during each night’s backup.<br />
<br />
== netstatbackup ==<br />
DEPRECATED<br />
netstatbackup <br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.5.x. <br />
<br />
== netstatbackup2 ==<br />
DEPRECATED<br />
netstatbackup2<br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.6.x. <br />
<br />
== netstatreset ==<br />
DEPRECATED<br />
netstatreset<br />
writes traffic count data to a logfile and resets counters to 0. Works on virtuozzo versions 2.5.x <br />
<br />
== netstatreset2 ==<br />
DEPRECATED<br />
netstatreset2<br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.6.x. <br />
<br />
== orphanedbackupwatchlinux ==<br />
orphanedbackupwatchlinux <br />
looks for directories on backup2 which aren’t configured in backup.config and offers to <br />
delete them<br />
<br />
== rsync.backup (linux) ==<br />
rsync.backup<br />
does customer backups (relies on backup.config)<br />
<br />
== startvirt.pl ==<br />
startvirt.pl<br />
forks off start ve commands – keeps 6 running at a time. This is not to be used on systems where fastboot is enabled as it circumvents the benefit of the fastboot. The script will occasionally not exit gracefully and will continue to use up CPU, so it should be watched. Also, don’t exit from the script till you’re sure all ve’s are started – if you do you need to start them manually and may have to free up locks. On some systems, the startvirt script doesn’t exit cleanly and you have to ^C out of it. Be careful though- doing so can leave some VE’s in an odd bootup state and you may need to ‘vr’ them manually. You should check to see which ve’s aren’t running and/or confirm all have started when ^C’ing out of startvirt.<br />
<br />
== taskdone (linux) ==<br />
taskdone<br />
when called will email support@johncompanies.com with the hostname of the machine from which it was <br />
executed as the subject<br />
<br />
== vb (linux) ==<br />
vb<br />
the equivalent of: <tt>vi /usr/local/sbin/backup.config</tt><br />
<br />
== vemakeXX ==<br />
DEPRECATED<br />
vemakerh9 <br />
ve create script for RH9 (see vemake)<br />
<br />
vemakedebian30 <br />
ve create script for debian 3.0 (Woody) (see vemake)<br />
<br />
vemakedebian31 <br />
ve create script for debian 3.1 (Sarge) (see vemake)<br />
<br />
vemakedebian40 <br />
ve create script for debian 4.0 (Etch) (see vemake)<br />
<br />
vemakefedora, vemakefedora2, vemakefedora4, vemakefedora5, vemakefedora6, vemakefedora7<br />
ve create script for fedora core 1, 2, 4, 5, 6, 7 respectively (see vemake)<br />
<br />
vemakecentos3, vemakecentos4<br />
ve create script for centos 3, 4 respectively (see vemake)<br />
<br />
vemakesuse, vemakesuse93, vemakesuse100<br />
ve create script for suse 9.2, 9.3, 10.0 respectively (see vemake)<br />
<br />
vemakeubuntu5, vemakeubuntu606, vemakeubuntu606 vemakeubuntu704<br />
ve create script for ubuntu 5.10, 6.06, 6.10, 7.04 respectively (see vemake)<br />
<br />
== vemove ==<br />
DEPRECATED<br />
vemove <veid> <target_ip> </vz/private/123><br />
this script simplifies the old way of moving ve’s from one system to another - in short moving a ve to or from a virt running virtuozzo < 2.6.x<br />
It’s the equivalent of:<br />
<tt>tar cfpP - <veid> --ignore-failed-read | (ssh -2 -c arcfour <target_ip> "split - -b 1024m </vz/private/123>.tar" )</tt>This should only be used if migrate/vzmigrate can’t be used. <br />
<br />
== vim.watchdog ==<br />
vim.watchdog <br />
looks for and kills procs matching “vi|vim|nano|pine|elm” that are running for a long time & consuming lots of cpu. Works on virtuozzo versions 2.5.x<br />
<br />
== vim.watchdog2 ==<br />
vim.watchdog2<br />
looks for and kills procs matching “vi|vim|nano|pine|elm” that are running for a long time & consuming lots of cpu.<br />
Works on virtuozzo versions 2.6.x.<br />
<br />
== vzmigrate ==<br />
vzmigrate <target_ip> -r no <veid>:[dst veid]:[dst /vzX/private/veid]:[dst /vzX/root/veid]<br />
(this is the raw command “wrapped” by migrate/migrateonline) this will seamlessly move a ve from one host to another. The ve will run for the duration of the migration till the very end when it’s shut down, ip moved and started up on the target system. The filesystem on the src will remain. This should be watched – occasionally the move will timeout and leave the system shut down. If target private and root aren’t specified it just puts it in /vz. Only works when both systems are running virtuozzo 2.6.x<br />
<br />
== vztrafdump.sh ==<br />
DEPRECATED<br />
vztrafdump.sh<br />
writes traffic usage info by ve to a file called jc_traffic_dump in each ve’s / dir. Works on virtuozzo versions <= 2.5.x. <br />
<br />
== vztrafdump2.sh ==<br />
DEPRECATED<br />
vztrafdump2.sh<br />
writes traffic usage info by ve to a file called jc_traffic_dump in each ve’s / dir. Works on virtuozzo versions 2.6.x. <br />
<br />
== addtun ==<br />
addtun <veid><br />
Add’s tun device to ve.<br />
<br />
== bwcap ==<br />
bwcap <veid> <kbps><br />
Ex: <tt>bwcap 1234 512</tt><br />
Caps a VE’s bandwidth to the amount given<br />
<br />
== setdisk ==<br />
setdisk <veid> <diskspace in GB><br />
Ex: <tt>setdisk 1234 5</tt><br />
Gives a VE’s a given amount of disk space<br />
<br />
== vdf ==<br />
vdf <veid> <br />
the equivalent of: <tt>vzctl exec <veid> df –h</tt><br />
<br />
== vdff ==<br />
vdff<br />
runs a (condensed) vdf for all ve’s in your pwd (must be run from /vz/privateN)<br />
<br />
== mvbackups ==<br />
mvbackups <veid> <target_machine> (virt1) <target_dir> (vz1)<br />
moves backups from one location to another on the backup server, and provides you with option to remove entries from current backup.config, and simple paste command to add the config to backup.config on the target server<br />
<br />
== checkquota ==<br />
checkquota<br />
for all the ve’s in the cwd (run from /vz/private, /vz1/private, etc) reports what vz quota says they’re using and what the actual usage is (as reported by du)<br />
<br />
== clearquota ==<br />
clearquota <veid><br />
Recalculates a ve’s quota, prints out the usage before and after. The equivalent of:<br />
<tt>vdf <veid>; v stop <veid>; vzquota drop <veid>; v start <veid>; vdf <veid></tt><br />
<br />
== dprocs ==<br />
dprocs<br />
Sometimes the server’s have a large number of processes get stuck in the D state- this script shows (every 3 secs) which VE’s have D procs, which procs<br />
are stuck and a running average of the top “offenders”<br />
<br />
== vzstat ==<br />
vstat<br />
sort of like top for VZ. sort VEs by CPU usage by pressing 'o' and then 'c' keys</div>Supporthttps://wiki.jcihosting.com/index.php?title=VPS_Management&diff=1014VPS Management2013-03-11T23:00:34Z<p>Support: /* Traffic accounting on linux */</p>
<hr />
<div>= Common Problems =<br />
== Login to any machine without a password ==<br />
<br />
This is possible via the use of ssh keys. The process is thus:<br />
<br />
1. place the public key for your user (root@mail) in the /root/.ssh/authorized_keys file on the server you wish to login to<br />
cat /root/.ssh/id_dsa.pub<br />
(paste that into authorized_keys on the target server). If the file doesn't exist, create it.<br />
<br />
2. enable root login (usually only applies to FreeBSD). Edit the /etc/ssh/sshd_config on the target server and change:<br />
<tt>#PermitRootLogin no</tt><br />
to<br />
<tt>PermitRootLogin yes</tt><br />
<br />
3. Restart the sshd on the target machine. First, find the sshd process: <br />
jailps <hostname> | grep sshd <br />
or <br />
vp <VEID> | grep sshd<br />
<br />
Look for the process resembling:<br />
root 17296 0.0 0.0 5280 1036 ? Ss 2011 4:27 /usr/sbin/sshd <br />
(this is the sshd)<br />
<br />
Not:<br />
root 6270 0.5 0.0 6808 2536 ? Ss 14:33 0:00 sshd: root [priv]<br />
(this is an sshd child- someone already ssh'd in as root)<br />
<br />
Restart the sshd: <br />
kill -1 <PID><br />
<br />
Ex:<br />
kill -1 17296<br />
<br />
You may now ssh in.<br />
<br />
Once you're done, IF you enabled root login, you should repeat steps 2 and 3 to disable root logins.<br />
<br />
== Letting someone in who has locked themselves out (killed sshd, lost pwd) ==<br />
<br />
There are two ways people frequently lock themselves out - either they forget a password, or they kill off sshd somehow.<br />
<br />
These are actually both fairly easy to solve. First, let's say someone kills off their sshd, or somehow mangles /etc/ssh/sshd_config such that it no longer lets them in.<br />
<br />
Their email may be very short, or it may have all sorts of details about how you should fix sshd_config to let them in ... just ignore all of this. They can fix their own mangled sshd. Fixing this is very simple. First, edit the /etc/inetd.conf on their system and uncomment the telnet line:<br />
<br />
telnet stream tcp nowait root /usr/libexec/telnetd telnetd<br />
#telnet stream tcp6 nowait root /usr/libexec/telnetd telnetd<br />
<br />
(just leave the tcp6 version of telnet commented)<br />
<br />
Then, use jailps to list the processes on their system, and find their inetd process. Then simply:<br />
<br />
kill -HUP (pid)<br />
<br />
where (pid) is the PID of their inetd process. Now they have telnet running on their system and they can log in and do whatever they need to do.<br />
<br />
The only complications that could occur are:<br />
<br />
a) their firewall config on our firewall has port 23 blocked, in which case you will need to open that - will be covered in a different lesson.<br />
<br />
b) they are not running inetd, so you can't HUP it. If this happens, edit their /etc/rc.conf, add the inetd_enable="YES" line, and then kill<br />
their jail with /tmp/jailkill.pl - then restart their jail with the jail line from their quad/safe file. Easy.<br />
<br />
If they have forgotten a password,<br />
<br />
On 6.x+ you can reset their password with:<br />
jexec <jailID from jls> passwd root<br />
<br />
Note: the default password for 6.x jails is 8ico2987, for 4.x it is p455agfa<br />
<br />
On 4.x, you need to cd to their etc directory<br />
... for instance:<br />
<br />
cd /mnt/data2/198.78.65.136-col00261-DIR/etc<br />
<br />
and run:<br />
<br />
vipw -d .<br />
<br />
Then paste in these two lines (theres a paste with these):<br />
<br />
root:$1$krszPxhk$xkCepSnz3mIikT3vCtJCt0:0:0::0:0:Charlie &:/root:/bin/csh<br />
user:$1$Mx9p5Npk$QdMU6c8YQqp2FW2M3irEh/:1001:1001::0:0:User &:/home/user:/bin/sh<br />
<br />
overwriting the lines they already have for "user" and "root" - then just tell them that both user and root have been reset to the default password of p455agfa.<br />
<br />
For linux, just passwd inside shell or <br />
vzctl set <veid> --userpasswd root:p455agfa –save<br />
<br />
Starting in 2009 we began giving out randomized passwords for FreeBSD and Linux as the default password. That is stored with each system in Mgmt. You should look for and reset the password to that password in the event of a reset and refer the customer to use their original password from their welcome email- this way we don’t have to send the password again via email (in clear text).<br />
<br />
<br />
== sendmail can’t be contacted from ext ip (only locally) ==<br />
<br />
By default redhat puts this line in sendmail.mc:<br />
<br />
DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')<br />
<br />
which makes it only answer on localhost. Comment it out like:<br />
<br />
dnl DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')<br />
<br />
and then rebuild sendmail.cf with:<br />
<br />
m4 /etc/mail/sendmail.mc > /etc/sendmail.cf<br />
<br />
== virt doesn’t properly let go of ve’s ip(s) when moved to another system ==<br />
<br />
On virtuozzo 2.6 systems, it's been observed that when moving ips from one virt to another that sometimes the routing table will not get updated to reflect the removal of the ip addresses.<br />
<br />
A recent example was a customer that was moving to a new ve on a new virt and the ip addresses were traded between the two ve's. After the trade the two systems were not able to talk to each other. When looking at the routing table for the old system all the ip addresses were still in the routing table as being local, like so:<br />
<br />
<pre>netstat -rn | grep 69.55.225.149<br />
69.55.225.149 0.0.0.0 255.255.255.255 UH 40 0 0 venet0</pre><br />
<br />
This was preventing traffic to the other system from being routed properly.<br />
The solution is to manually delete the route:<br />
<br />
route delete 69.55.225.149 gw 0.0.0.0<br />
<br />
Supposedly, this was fixed in 2.6.1<br />
<br />
== sshd on FreeBSD 6.2 segfaults ==<br />
<br />
First try to reinstall ssh<br />
<br />
<pre>cd /usr/src/secure<br />
cd lib/libssh<br />
make depend && make all install<br />
cd ../../usr.sbin/sshd<br />
make depend && make all install<br />
cd ../../usr.bin/ssh<br />
make depend && make all install</pre><br />
<br />
Failing that, find the library that’s messed up:<br />
<br />
<pre>ldd /usr/sbin/sshd<br />
libssh.so.3 => /usr/lib/libssh.so.3 (0x280a3000) <br />
libutil.so.5 => /lib/libutil.so.5 (0x280d8000) <br />
libz.so.3 => /lib/libz.so.3 (0x280e4000) <br />
libwrap.so.4 => /usr/lib/libwrap.so.4 (0x280f5000) <br />
libpam.so.3 => /usr/lib/libpam.so.3 (0x280fc000) <br />
libbsm.so.1 => /usr/lib/libbsm.so.1 (0x28103000) <br />
libgssapi.so.8 => /usr/lib/libgssapi.so.8 (0x28112000) <br />
libkrb5.so.8 => /usr/lib/libkrb5.so.8 (0x28120000) <br />
libasn1.so.8 => /usr/lib/libasn1.so.8 (0x28154000) <br />
libcom_err.so.3 => /usr/lib/libcom_err.so.3 (0x28175000) <br />
libroken.so.8 => /usr/lib/libroken.so.8 (0x28177000) <br />
libcrypto.so.4 => /lib/libcrypto.so.4 (0x28183000) <br />
libcrypt.so.3 => /lib/libcrypt.so.3 (0x28276000) <br />
libc.so.6 => /lib/libc.so.6 (0x2828e000) <br />
libmd.so.3 => /lib/libmd.so.3 (0x28373000)</pre><br />
<br />
md5 them and compare to other jail hosts or jails running on host<br />
<br />
for libcrypto reinstall:<br />
<pre>/usr/src/crypto<br />
make depend && make all install</pre><br />
<br />
= FreeBSD VPS =<br />
<br />
== Starting jails: Quad/Safe Files ==<br />
<br />
FreeBSD customer systems do not start up automatically at boot time. When one of our freebsd machines boots up, it boots up, and does nothing else. To start jails, we put the commands to start each jail into a shell script(s) and run the script(s). Jail startup is something that needs to be actively monitored, which is why we don’t just run the script automatically. More on monitoring later.<br />
<br />
NOTE: >=7.x we have moved to 1 quad file: <tt>quad1</tt>. Startups are not done by running each quad, but rather [[#startalljails|startalljails]] which relies on the contents of <tt>quad1</tt>. The specifics of this are lower in this article. What follows here applies for pre 7.x systems.<br />
<br />
There are eight files in <tt>/usr/local/jail/rc.d</tt>:<br />
<br />
<pre>jail3# ls /usr/local/jail/rc.d/<br />
quad1 quad2 quad3 quad4 safe1 safe2 safe3 safe4<br />
jail3#</pre><br />
<br />
four quad files and four safe files.<br />
<br />
Each file contains an even number of system startup blocks (total number of jails divided by 4)<br />
<br />
The reason for this is, if we make one large script to startup all the systems at boot time, it will take too long - the first system in the script will start up right after system boot, which is great, but the last system may not start for another 20 minutes.<br />
<br />
Since there is no way to parralelize this during the startup procedure, we simply open four terminals (in screen window 9) and run each script, one in each terminal. This way they all run simultaneously, and the very last system in each startup script gets started in 1/4th the time it would if there was one large file<br />
<br />
The files are generally organized so that quad/safe 1&2 have only jails from disk 1, and quad/safe 3&4 have jails from disk 2. This helps ensure that only 2 fscks on any disk are going on at once. Further, they are balanced so that all quad/safe’s finish executing around the same time. We do this by making sure each quad/safe has a similar number of jails and represents a similar number of inodes (see js).<br />
<br />
The other, very important reason we do it this way, and this is the reason there are quad files and safe files, is that in the event of a system crash, every single vn-backed filesystem that was mounted at the time of system crash needs to be fsck'd. However, fsck'ing takes time, so if we shut the system down gracefully, we don't want to fsck.<br />
<br />
Therefore, we have two sets of scripts - the four quad scripts are identical to the four safe scripts except for the fact that the quad scripts contain fsck commands for each filesystem.<br />
<br />
So, if you shut a system down gracefully, start four terminals and run safe1 in window one, and safe2 in window 2, and so on.<br />
<br />
If you crash, start four terminals (or go to screen window 9) and run quad1 in window one, and quad2 in window 2, and so on.<br />
<br />
Here is a snip of (a 4.x version) quad2 from jail17:<br />
<br />
<pre>vnconfig /dev/vn16 /mnt/data2/69.55.228.7-col00820<br />
fsck -y /dev/vn16<br />
mount /dev/vn16c /mnt/data2/69.55.228.7-col00820-DIR<br />
chmod 0666 /mnt/data2/69.55.228.7-col00820-DIR/dev/null<br />
jail /mnt/data2/69.55.228.7-col00820-DIR mail1.phimail.com 69.55.228.7 /bin/sh /etc/rc<br />
<br />
# moved to data2 col00368<br />
#vnconfig /dev/vn28 /mnt/data2/69.55.236.132-col00368<br />
#fsck -y /dev/vn28<br />
#mount /dev/vn28c /mnt/data2/69.55.236.132-col00368-DIR<br />
#chmod 0666 /mnt/data2/69.55.236.132-col00368-DIR/dev/null<br />
#jail /mnt/data2/69.55.236.132-col00368-DIR limehouse.org 69.55.236.132 /bin/sh /etc/rc<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
vnconfig /dev/vn22 /mnt/data2/69.55.228.13-col01063<br />
fsck -y /dev/vn22<br />
mount /dev/vn22c /mnt/data2/69.55.228.13-col01063-DIR<br />
chmod 0666 /mnt/data2/69.55.228.13-col01063-DIR/dev/null<br />
jail /mnt/data2/69.55.228.13-col01063-DIR www.widestream.com.au 69.55.228.13 /bin/sh /etc/rc<br />
<br />
# cancelled col00106<br />
#vnconfig /dev/vn15 /mnt/data2/69.55.238.5-col00106<br />
#fsck -y /dev/vn15<br />
#mount /dev/vn15c /mnt/data2/69.55.238.5-col00106-DIR<br />
#chmod 0666 /mnt/data2/69.55.238.5-col00106-DIR/dev/null<br />
#jail /mnt/data2/69.55.238.5-col00106-DIR mail.azebu.net 69.55.238.5 /bin/sh /etc/rc</pre><br />
<br />
As you can see, two of the systems specified are commented out - presumably those customers cancelled, or were moved to new servers.<br />
<br />
As you can see, the vnconfig line is the simpler command line, not the longer one that was used when it was first configured. As you can see, all that is done is, vnconfig the filesystem, then fsck it, then mount it. The fourth command is the `jail` command used to start the system – but that will be covered later.<br />
<br />
Here is the safe2 file from jail17:<br />
<br />
<pre>vnconfig /dev/vn16 /mnt/data2/69.55.228.7-col00820<br />
mount /dev/vn16c /mnt/data2/69.55.228.7-col00820-DIR<br />
chmod 0666 /mnt/data2/69.55.228.7-col00820-DIR/dev/null<br />
jail /mnt/data2/69.55.228.7-col00820-DIR mail1.phimail.com 69.55.228.7 /bin/sh /etc/rc<br />
<br />
# moved to data2 col00368<br />
#vnconfig /dev/vn28 /mnt/data2/69.55.236.132-col00368<br />
#mount /dev/vn28c /mnt/data2/69.55.236.132-col00368-DIR<br />
#chmod 0666 /mnt/data2/69.55.236.132-col00368-DIR/dev/null<br />
#jail /mnt/data2/69.55.236.132-col00368-DIR limehouse.org 69.55.236.132 /bin/sh /etc/rc<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
vnconfig /dev/vn22 /mnt/data2/69.55.228.13-col01063<br />
mount /dev/vn22c /mnt/data2/69.55.228.13-col01063-DIR<br />
chmod 0666 /mnt/data2/69.55.228.13-col01063-DIR/dev/null<br />
jail /mnt/data2/69.55.228.13-col01063-DIR www.widestream.com.au 69.55.228.13 /bin/sh /etc/rc<br />
<br />
# cancelled col00106<br />
#vnconfig /dev/vn15 /mnt/data2/69.55.238.5-col00106<br />
#mount /dev/vn15c /mnt/data2/69.55.238.5-col00106-DIR<br />
#chmod 0666 /mnt/data2/69.55.238.5-col00106-DIR/dev/null<br />
#jail /mnt/data2/69.55.238.5-col00106-DIR mail.azebu.net 69.55.238.5 /bin/sh /etc/rc</pre><br />
<br />
As you can see, it is exactly the same, but it does not have the fsck lines.<br />
<br />
Take a look at the last entry - note that the file is named:<br />
<br />
/mnt/data2/69.55.238.5-col00106<br />
<br />
and the mount point is named:<br />
<br />
/mnt/data2/69.55.238.5-col00106-DIR<br />
<br />
This is the general format on all the FreeBSD systems. The file is always named:<br />
<br />
IP-custnumber<br />
<br />
and the directory is named:<br />
<br />
IP-custnumber-DIR<br />
<br />
If you run safe when you need a fsck, the mount will fail and jail will fail:<br />
<br />
# mount /dev/vn1c /mnt/data2/jails/65.248.2.131-ns1.kozubik.com-DIR<br />
mount: /dev/vn1c: Operation not permitted<br />
<br />
No reboot needed, just run the quad script<br />
<br />
Starting with 6.x jails, we added block delimiters to the quad/safe files, the block looks like:<br />
<br />
<pre>echo '## begin ##: nuie.solaris.mu'<br />
fsck -y /dev/concat/v30v31a<br />
mount /dev/concat/v30v31a /mnt/data1/69.55.228.218-col01441-DIR<br />
mount_devfs devfs /mnt/data1/69.55.228.218-col01441-DIR/dev<br />
devfs -m /mnt/data1/69.55.228.218-col01441-DIR/dev rule -s 3 applyset<br />
jail /mnt/data1/69.55.228.218-col01441-DIR nuie.solaris.mu 69.55.228.218 /bin/sh /etc/rc<br />
echo '## end ##: nuie.solaris.mu'</pre><br />
<br />
These are more than just informative when running quad/safe’s, the echo lines MUST be present for certain tools to work properly. So it’s important that any updates to the hostname also be updated on the 2 echo lines. For example, if you try to startjail a jail with a hostname which is on the jail line but not the echo lines, the command will return with host not found.<br />
<br />
=== FreeBSD 7.x+ notes ===<br />
<br />
Starting with the release of FreeBSD 7.x, we are doing jail startups in a slightly different way. First, thereis only 1 file: <tt>/usr/local/jail/rc.d/quad1</tt><br />
There are no other quads or corresponding safe files. The reason for this is twofold, 1. We can pass –C to fsck which will tell is to skip the fsck if the fs is clean (no more need for safe files), 2. We have a new startup script which can be launched multiple times, running in parallel to start jails, where quad1 is the master jail file. <br />
Quad1 could still be run as a shell script, but it would take a very long time for it to run completely so it’s not advisable; or you should break it down into smaller chunks (like quad1, quad2, quad3, etc)<br />
<br />
Here is a snip of (a 7.x version) quad1 from jail2:<br />
<br />
<pre>echo '## begin ##: projects.tw.com'<br />
mdconfig -a -t vnode -f /mnt/data1/69.55.230.46-col01213 -u 50<br />
fsck -Cy /dev/md50c<br />
mount /dev/md50c /mnt/data1/69.55.230.46-col01213-DIR<br />
mount -t devfs devfs /mnt/data1/69.55.230.46-col01213-DIR/dev<br />
devfs -m /mnt/data1/69.55.230.46-col01213-DIR/dev rule -s 3 applyset<br />
jail /mnt/data1/69.55.230.46-col01213-DIR projects.tw.com 69.55.230.46 /bin/sh /etc/rc<br />
echo '## end ##: projects.tw.com'</pre><br />
<br />
Cancelled jails are no longer commented out and stored in quad1, rather they’re moved to <tt>/usr/local/jail/rc.d/deprecated</tt> <br />
<br />
To start these jails, start the 4 ssh sessions as you would for a normal crash and then instead of running quad1-4, instead run startalljails in each window. IMPORTANT- before running startalljails you should make sure you ran preboot once as it will clear out all the lockfiles and enable startalljails to work properly.<br />
<br />
== Problems with the quad/safe files ==<br />
<br />
When you run the quad/safe files, there are two problems that can occur - either a particular system will hang during initialization, OR a system will spit out output to the screen, impeding your ability to do anything. Or both.<br />
<br />
First off, when you start a jail, you see output like this:<br />
<br />
<pre>Skipping disk checks ...<br />
adjkerntz[25285]: sysctl(put_wallclock): Operation not permitted<br />
Doing initial network setup:.<br />
ifconfig: ioctl (SIOCDIFADDR): permission denied<br />
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384<br />
Additional routing options: TCP keepalive=YESsysctl:<br />
net.inet.tcp.always_keepalive: Operation not permitted.<br />
Routing daemons:.<br />
Additional daemons: syslogd.<br />
Doing additional network setup:.<br />
Starting final network daemons:.<br />
ELF ldconfig path: /usr/lib /usr/lib/compat /usr/X11R6/lib /usr/local/lib<br />
a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout /usr/X11R6/lib/aout<br />
Starting standard daemons: inetd cron sshd sendmail sendmail-clientmqueue.<br />
Initial rc.i386 initialization:.<br />
Configuring syscons: blanktime.<br />
Additional ABI support:.<br />
Local package initialization:.<br />
Additional TCP options:.</pre><br />
<br />
Now, let's look at this line, near the end:<br />
<br />
Local package initialization:.<br />
<br />
This is where a list of daemons that are set to start at boot time willshow up. You might see something like:<br />
<br />
Local package initialization: mysqld apache sendmail sendmail-clientmqueue<br />
<br />
Or something like this:<br />
<br />
Local package initialization: postgres postfix apache<br />
<br />
The problem is that many systems (about 4-5 per machine) will hang on that line. Basically it will get to some of the way through the total daemons to be started:<br />
<br />
Local package initialization: mysqld apache<br />
<br />
and will just sit there. Forever.<br />
<br />
Fortunately, pressing ctrl-c will break out of it. Not only will it break out of it, but it will also continue on that same line and start the other daemons:<br />
<br />
Local package initialization: mysqld apache ^c sendmail-clientmqueue<br />
<br />
and then continue on to finish the startup, and then move to the next system to be started.<br />
<br />
So what does this mean? It means that if a machine crashes, and you start four screen-windows to run four quads or four safes, you need to periodically cycle between them and see if any systems are stuck at that point, causing their quad/safe file to hang. A good rule of thumb is, if you see a system at that point in the startup, give it another 100 seconds - if it is still at the exact same spot, hit ctrl-c. Its also a good idea to go back into the quad file (just before the first command in the jail startup block) and note that this jail tends to need a control-c or more time as follows:<br />
<pre>echo '### NOTE ### slow sendmail'<br />
echo '### NOTE ###: ^C @ Starting sendmail.'</pre><br />
<br />
'''NEVER''' hit ctrl-c repeatedly if you don't get an immediate response - that will cause the following jail’s startup commands to be aborted.<br />
<br />
A second problem that can occur is that a jail - maybe the first one in that particular quad/safe, maybe the last one, or maybe one in the middle, will start spitting out status or error messages from one of its init scripts. This is not a problem - basically, hit enter a few times and see if you get a prompt - if you do get a prompt, that means that the quad/safe script has already completed. Therefore it is safe to log out (and log out of the user that you su'd from) and then log back in (if necessary).<br />
<br />
The tricky thing is, if a system in the middle starts flooding with messages, and you hit enter a few times and don't get a prompt. Are you not getting a prompt because some subsequent system is hanging at the initialization, as we discussede above ? Or are you not getting a prompt because that quad file is currently running an fsck ? Usually you can tell by scrolling back in screen’s history to see what it was doing before you started getting the messages.<br />
<br />
If you don’t get clues from history, you have to use your judgement - instead of giving it 100 seconds to respond, perhaps give it 2-3 mins ... if you still get no response (no prompt) when you hit enter, hit ctrl-c. However, be aware that you might still be hitting ctrl-c in the middle of an fsck. This means you will get an error like "filesystem still marked dirty" and then the vnconfig for it will fail and so will the jail command, and the next system in the quad file will then start starting up.<br />
<br />
If this happens, just wait until the end of all the quad files have finished, and start that system manually.<br />
<br />
If things really get weird, like a screen flooded with errors, and you can't get a prompt, and ctrl-c does nothing, then you need to just eventually (give it ten mins or so) just kill that window with ctrl-p, then k, and then log in again and manually check which systems are now running and which aren't, and manually start up any that are not.<br />
<br />
Don't EVER risk running a particular quad/safe file a second time.<br />
If the quad/safe script gets executed twice, reboot the machine immediately.<br />
<br />
So, for all the above reasons, anytime a machine crashes and you run all the quads or all the safes, '''always''' check every jail afterwards to make sure it is running - even if you have no hangs or complications at all.<br />
Run this command:<br />
<br />
<tt>[[#jailpsall|jailpsall]]</tt> <br />
<br />
Note: [[#postboot|postboot]] also populates ipfw counts, so it '''should not be run multiple times''', use <tt>jailpsall</tt> for subsequent extensive ps’ing<br />
<br />
And make sure they all show as running. If one does not show as running, check its /etc/rc.conf file first to see if maybe it is using a different hostname first before starting it manually.<br />
<br />
One thing we have implemented to alleviate these startup hangs and noisy jails, is to put jail start blocks that are slow or hangy at the bottom of the safe/quad file. Further, for each bad jail we note in each quad/safe just before the start block something like:<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
<br />
That way we’ll be prepared to ^C when we see that message appear during the quad/safe startup process. If you observe a new, undocumented hang, '''after''' the quad/safe has finished, place a line similar to the above in the quad file, move the jail start block to the end of the file, then run [[#buildsafe|buildsafe]]<br />
<br />
<br />
== Recovering from a crash (FreeBSD) ==<br />
<br />
=== Diagnose whether you have a crash ===<br />
The most important thing is to get the machine and all jails back up as soon as possible. Note the time, you’ll need to create a crash log entry (Mgmt. -> Reference -> CrashLog). The first thing to do is head over to the [[Screen#Screen_Organization|serial console screen]] and see if there’s any kernel error messages output. Try to copy any messages (or just a sample of repeating messages) you see into the notes section of the crash log. If there are no messages, the machine may just be really busy- wait a bit (5-10min) to see if it comes back. If it's still pinging, odds are its very busy. Note, if you see messages about swap space exhausted, the server is obviously out of memory, however it may recover briefly enough for you to get a jtop in to see who's lauched a ton of procs (most likely) and then issue a quick jailkill to get it back under control.<br />
<br />
If it doesn't come back, or the messages indicate a fatal error, you will need to proceed with a power cycle (ctrl+alt+del will not work).<br />
<br />
=== Power cycle the server ===<br />
If this machine is not a Dell 2950 with a [[DRAC/RMM#DRAC|DRAC card]] (i.e. if you can’t ssh into the DRAC card (as root, using the standard root pass) and issue <br />
racadm serveraction hardreset<br />
then you will need someone at the data center power the macine off, wait 30 sec, then turn it back on. Make sure to re-attach via console:<br />
tip jailX<br />
immediately after power down. <br />
<br />
=== (Re)attach to the console ===<br />
Stay on the console the entire time during boot. As the BIOS posts- look out for the RAID card output- does everything look healthy? The output may be scrambled, look for "DEGRADED" or "FAILED". Once the OS starts booting you will be disconnected (dropped back to the shell on the console server) a couple times during the boot up. The reason you want to quickly re-attach is two-fold: 1. If you don’t reattach quickly then you won’t get any console output, 2. you want to be attached before the server ''potentially'' starts (an extensive) fsck. If you attach after the fsck begins, you’ll have seen no indication it started an fsck and the server will appear frozen during startup- no output, no response. <br />
<br />
IMPORTANT NOTE: on some older FreeBSD systems, there will be no output to the video (KVM) console as it boots up. The console output is redirected to the serial port ... so if a jail crashes, and you attach a kvm, the output during the bootup procedure will not be shown on the screen. However, when the bootup is done, you will get a login prompt on the screen and will be able to log in as normal. <tt>/boot/loader.conf</tt> is where serial console redirect output lives, so comment that if you want to catch output on kvm.<br />
On newer systems it sends most output to both locations. <br />
<br />
=== Assess the heath of the server ===<br />
Once the server boots up fully, you should be able to ssh in. Look around- make sure all the mounts are there and reporting the correct size/usage (i.e. /mnt/data1 /mnt/data2 /mnt/data3 - look in /etc/fstab to determine which mount points should be there), check to see if RAID mirrors are healthy. See [[RAID_Cards#Common_CLI_commands_.28megacli.29|megacli]], [[#aaccheck|aaccheck]]<br />
<br />
Before you start the jails, you need to run [[#preboot|preboot]]. This will do some assurance checks to make sure things are prepped to start the jails. Any issues that come out of preboot need to be addressed before starting jails.<br />
<br />
=== Start jails ===<br />
[[#Starting_jails:_Quad.2FSafe_Files|More on starting jails]]<br />
Customer jails (the VPSs) do not start up automatically at boot time. When a FreeBSD machines boots up, it boots up, and does nothing else. To start jails, we put the commands to start each jail into a shell script(s) and run the script(s). Jail startup is something that needs to be actively monitored, which is why we don’t just run the script automatically. <br />
<br />
In order to start jails, we run the quad files: quad1 quad2 quad3 and quad4 (on new systems there is only quad1). If the machine was cleanly rebooted- which wouldn't be the case if this was a crash, you may run the safe files (safe1 safe2 safe3 safe4) in lieu of quads. <br />
<br />
Open up 4 logins to the server (use the windows in [[Screen#Screen_Organization|a9]])<br />
In each of the 4 windows you will:<br />
<br />
If there is a [[#startalljails|startalljails]] script (and only quad1), run that command in each of the 4 windows. It will parse through the quad1 file and start each jail. Follow the instructions [[#Problems_with_the_quad.2Fsafe_files|here]] for monitoring startup. Note that you can be a little more lenient with jails that take awhile to start- startalljails will work around the slow jails and start the rest. As long as there aren't 4 jails which are "hung" during startup, the rest will get started eventually.<br />
-or-<br />
If there is no startalljails script, there will be multiple quad files. In each of the 4 windows, start each of the quads. i.e. start quad1 in window1, quad2 in window2 and so on. DO NOT start any quad twice. It will crash the server. If you accidentally do this, just jailkill all the jails which are in the quad and run the quad again. Follow the instructions here for monitoring quad startup.<br />
<br />
Note the time the last jail boots- this is what you will enter in the crash log.<br />
<br />
Save the crash log.<br />
<br />
=== Check to make sure all jails have started ===<br />
There's a simple script which will make sure all jails have started, and enter the ipfw counter rules: [[#postboot|postboot]] <br />
Run postboot, which will do a jailps on each jail it finds (excluding commented out jails) in the quad file(s). We're looking for 2 things:<br />
# systems spawning out of control or too many procs<br />
# jails which haven't started<br />
On 7.x and newer systems it will print out the problems (which jails haven't started) at the conclusion of postboot. <br />
On older systems you will need to watch closely to see if/when there's a problem, namely:<br />
<br />
[hostname] doesnt exist on this server<br />
<br />
When you get this message, it means one of 2 things:<br />
1. the jail really didn't start:<br />
When a jail doesn't start it usually boils down to a problem in the quad file. Perhaps the path name is wrong (data1 vs data2) or the name of the vn/mdfile is wrong. Once this is corrected, you will need to run the commands from the quad file manually, or you may use <tt>startjail <hostname></tt><br />
<br />
2. the customer has changed their hostname (and not told us) so their jail ''is'' running, just under a different hostname:<br />
On systems with jls, this is easy to rectify. First, get the customer info: <tt>g <hostname></tt><br />
Then look for the customer in jls: <tt>jls | grep <col0XXXX></tt><br />
From there you will see their new hostname- you should update that hostname in the quad file: don't forget to edit it on the <tt>## begin ##</tt> and <tt>## end ##</tt> lines, and in mgmt. <br />
On older systems without jls, this will be harder, you will need to look further to see their hostname- perhaps its in their /etc/rc.conf<br />
<br />
<br />
Once all jails are started, do some spot checks- try to ssh or browse to some customers, just to make sure things are really ok.<br />
<br />
== Adding disk to a 7.x/8.x jail ==<br />
or<br />
== Moving customer to a different drive (md) ==<br />
<br />
NOTE: this doesn’t apply to mx2 which uses gvinum. Use same procedure as 6.x<br />
NOTE: if you unmount before mdconfig, re-mdconfig (attach) then unmount then mdconfig -u again <br />
<br />
<br />
(parts to change/customize are <tt><span style="color:red">red</span></tt>)<br />
<br />
<br />
If someone wants more disk space, there’s a paste for it, send it to them (explains about downtime, etc).<br />
<br />
1. Figure out the space avail from <tt>js</tt>. Ideally, you want to put the customers new space on a different partition (and create the new md on the new partition). <br />
<br />
<br />
2. make a mental note of how much space they're currently using<br />
<br />
<br />
3. <tt>jailkill <hostname></tt> <br />
<br />
<br />
4. Umount it (including their devfs) but leave the md config’d (so if you use stopjail, you will have to re-mdconfig it)<br />
<br />
<br />
5. <tt>g <customerID></tt> to get the info (IP/cust#) needed to feed the new mdfile and mount name, and to see the current md device. <br />
<br />
<br />
6a. When there's enough room to place new system on an alternate, or the same drive:<br />
USE CAUTION not to overwrite (touch, mdconfig) existing md!!<br><br />
<tt>touch /mnt/data<span style="color:red">3/69.55.234.66-col01334</span><br><br />
mdconfig -a -t vnode -s 10g -f /mnt/data3/69.55.234.66-col01334 -u 97<br><br />
newfs /dev/md97</tt><br />
<br />
<br />
Optional- if new space is on a different drive, move the mount point directory AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
<tt>mount /dev/md<span style="color:red">97</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<br />
confirm you are mounted to /dev/md<span style="color:red">97</span> and space is correct:<br><br />
<tt>df .</tt> <br />
<br />
<br />
do the dump and pipe directly to restore:<br><br />
<tt>dump -0a -f - /dev/md<span style="color:red">1</span> | restore -r -f - <br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use <tt>df</tt> to confirm the restored data size matches the original usage figure<br />
<br />
<br />
md-unconfig old system:<br><br />
<tt>mdconfig -d -u <span style="color:red">1</span></tt><br />
<br />
<br />
archive old mdfile. <br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.237.26-col00241</span> /mnt/data<span style="color:red">1/old-col00241-mdfile-noarchive-20091211</span></tt><br />
<br />
<br />
edit quad (vq1) to point to new (/mnt/data<span style="color:red">3</span>) location AND new md number (md<span style="color:red">97</span>)<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
6b. When there's not enough room on an alternate partition, or on the same drive...but there is enough room if you were to remove the existing customer's space:<br />
<br />
<br />
mount backup nfs mounts:<br><br />
<tt>mbm</tt><br> <br />
(run <tt>df</tt> to confirm backup mounts are mounted)<br />
<br />
<br />
dump the customer to backup2 or backup1<br><br />
<tt>dump -0a -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> /dev/md<span style="color:red">1</span></tt><br><br />
(when complete WITHOUT errors, <tt>du</tt> the dump file to confirm it matches size, roughly, with usage)<br />
<br />
<br />
unconfigure and remove old mdfile<br><br />
<tt>mdconfig -d -u <span style="color:red">1</span><br><br />
rm /mnt/data<span style="color:red">1/69.55.237.26-col00241</span></tt><br><br />
(there should now be enough space to recreate your bigger system. If not, run sync a couple times)<br />
<br />
<br />
create the new system (ok to reuse old mdfile and md#):<br><br />
<tt>touch /mnt/data<span style="color:red">1/69.55.234.66-col01334</span><br><br />
mdconfig -a -t vnode -s <span style="color:red">10</span>g -f /mnt/data<span style="color:red">1/69.55.234.66-col01334</span> -u <span style="color:red">1</span><br><br />
newfs /dev/md<span style="color:red">1</span><br><br />
mount /dev/md<span style="color:red">1</span> /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> <br><br />
cd /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<br />
confirm you are mounted to /dev/md<span style="color:red">1</span> and space is correct:<br><br />
<tt>df .</tt> <br />
<br />
<br />
do the restore from the dumpfile on the backup server:<br><br />
<tt>restore -r -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> .<br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
umount nfs:<br><br />
<tt>mbu</tt><br />
<br />
<br />
If md# changed (or mount point), edit quad (<tt>vq1</tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new md number (md<span style="color:red">1</span>)<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
7. update disk (and dir if applicable) in mgmt screen<br />
<br />
<br />
8. update backup list AND move backups, if applicatble<br />
<br />
Ex: <tt>mvbackups <span style="color:red">69.55.237.26-col00241</span> jail<span style="color:red">9</span> data<span style="color:red">3</span></tt><br />
<br />
<br />
9. Optional: archive old mdfile<br />
<br />
<tt>mbm<br><br />
gzip -c old-col01588-mdfile-noarchive-20120329 > /deprecated/old-col01588-mdfile-noarchive-20120329.gz<br><br />
mbu<br><br />
rm old-col01588-mdfile-noarchive-20120329</tt><br />
<br />
== Adding disk to a 6.x jail (gvinum/gconcat) ==<br />
or<br />
== Moving customer to a different drive (gvinum/gconcat) ==<br />
<br />
(parts to change are <span style="color:red">highlighted</span>)<br />
<br />
<br />
If someone wants more disk space, there’s a paste for it, send it to them (explains about downtime, etc).<br />
<br />
<br />
1. Figure out the space avail from [[#js|js]]. Ideally, you want to put the customers new space on a different partition (and create the new md on the new partition). <br />
<br />
<br />
2. make a mental note of how much space they're currently using<br />
<br />
<br />
3. <tt>[[#stopjail|stopjail]] <hostname></tt> <br />
<br />
<br />
4. <tt>[[#g|g]] <customerID></tt> to get the info (IP/cust#) needed to feed the new mount name and existing volume/device. <br />
<br />
<br />
5a. When there's enough room to place new system on an alternate, or the same drive (using only UNUSED - including if it's in use by the system in question - gvinum volumes):<br />
<br />
<br />
Configure the new device:<br><br />
A. for a 2G system (single gvinum volume):<br><br />
<tt>bsdlabel -r -w /dev/gvinum/v<span style="color:red">123</span><br><br />
newfs /dev/gvinum/v<span style="color:red">123</span>a</tt><br><br />
<br />
-or- <br />
<br />
<br />
B. for a >2G system (create a gconcat volume):<br><br />
try to grab a contiguous block of gvinum volumes. gconcat volumes MAY NOT span drives (i.e. you cannot use a gvinum volume from data3 and a volume from data2 in the same gconcat volume). <br />
<br />
<tt>gconcat label <span style="color:red">v82-v84 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84</span><br><br />
bsdlabel -r -w /dev/concat/<span style="color:red">v82-v84</span><br><br />
newfs /dev/concat/<span style="color:red">v82-v84</span>a</tt><br />
<br />
<br />
Other valid gconcat examples:<br><br />
<tt>gconcat label v82-v84v109v112 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84 /dev/gvinum/v109 /dev/gvinum/v112<br><br />
gconcat label v82v83 /dev/gvinum/v82 /dev/gvinum/v83</tt> <br><br />
Note, long names will truncate: v144v145v148-v115 will truncate to v144v145v148-v1 (so you will refer to it as v144v145v148-v1 thereafter)<br />
<br />
<br />
Optional- if new volume is on a different drive, move the mount point directory (get the drive from js output) AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
<br />
confirm you are mounted to the device (<tt>/dev/gvinum/v<span style="color:red">123</span>a</tt> OR <tt>/dev/concat/<span style="color:red">v82-v84</span></tt>) and space is correct:<br><br />
A. <tt>mount /dev/gvinum/v<span style="color:red">123</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br><br />
-or-<br><br />
B. <tt>mount /dev/concat/<span style="color:red">v82-v84</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<tt>cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
df .</tt> <br />
<br />
<br />
do the dump and pipe directly to restore:<br><br />
<tt>dump -0a -f - /dev/gvinum/v<span style="color:red">1</span> | restore -r -f - <br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
edit quad (<tt>vq<span style="color:red">1</span></tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new volume (<tt>/dev/gvinum/v<span style="color:red">123</span>a</tt> or <tt>/dev/concat/<span style="color:red">v82-v84</span>a</tt>) , run <tt>buildsafe</tt><br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
5b. When there's not enough room on an alternate partition, or on the same drive...but there is enough room if you were to remove the existing customer's space (i.e. if you want/need to reuse the existing gvinum volumes and add on more):<br />
<br />
<br />
mount backup nfs mounts:<br><br />
<tt>mbm</tt> <br><br />
(run df to confirm backup mounts are mounted)<br />
<br />
<br />
dump the customer to backup2 or backup1<br><br />
<tt>dump -0a -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> /dev/<span style="color:red">gconcat/v106-v107</span></tt><br><br />
(when complete WITHOUT errors, du the dump file to confirm it matches size, roughly, with usage)<br />
<br />
<br />
unconfigure the old gconcat volume<br><br />
list member gvinum volumes:<br />
<br />
<tt>gconcat list <span style="color:red">v106v107</span></tt><br />
<br />
<br />
Output will resemble:<br><br />
<pre>Geom name: v106v107<br />
State: UP<br />
Status: Total=2, Online=2<br />
Type: AUTOMATIC<br />
ID: 3530663882<br />
Providers:<br />
1. Name: concat/v106v107<br />
Mediasize: 4294966272 (4.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e2<br />
Consumers:<br />
1. Name: gvinum/sd/v106.p0.s0<br />
Mediasize: 2147483648 (2.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e3<br />
Start: 0<br />
End: 2147483136<br />
2. Name: gvinum/sd/v107.p0.s0<br />
Mediasize: 2147483648 (2.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e3<br />
Start: 2147483136<br />
End: 4294966272</pre><br />
<br />
stop volume and clear members<br><br />
<tt>gconcat stop <span style="color:red">v106v107</span><br><br />
gconcat clear <span style="color:red">gvinum/sd/v106.p0.s0 gvinum/sd/v107.p0.s0</span></tt><br />
<br />
<br />
create new device- and its ok to reuse old/former members<br><br />
try to grab a contiguous block of gvinum volumes. gconcat volumes MAY NOT span drives (i.e. you cannot use a gvinum volume from data3 and a volume from data2 in the same gconcat volume). <br><br />
<tt>gconcat label <span style="color:red">v82-v84v106v107 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84 /dev/gvinum/v106 /dev/gvinum/v107</span> <br><br />
bsdlabel -r -w /dev/concat/<span style="color:red">v82-v84v106v107</span><br><br />
newfs /dev/concat/<span style="color:red">v82-v84v106v107</span>a</tt><br />
<br />
<br />
Optional- if new volume is on a different drive, move the mount point directory (get the drive from js output) AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
confirm you are mounted to the device (/dev/concat/<span style="color:red">v82-v84v106v107</span>) and space is correct:<br><br />
<tt>mount /dev/concat/<span style="color:red">v82-v84v106v107</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<tt>cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
df .</tt> <br />
<br />
do the restore from the dumpfile on the backup server:<br><br />
<tt>restore -r -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> .<br><br />
rm restoresymtable</tt><br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
edit quad (<tt>vq<span style="color:red">1</span></tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new volume (<tt>/dev/concat/<span style="color:red">v82-v84v106v107</span>a</tt>), run buildsafe<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
TODO: clean up/clear old gvin/gconcat vol<br />
<br />
<br />
6. update disk (and dir if applicable) in mgmt screen<br />
<br />
<br />
7. update backup list AND move backups, if applicatble<br />
<br />
<br />
Ex: <tt>mvbackups <span style="color:red">69.55.237.26-col00241</span> jail<span style="color:red">9</span> data<span style="color:red">3</span></tt><br />
<br />
<br />
<br />
DEPRECATED - steps to tack on a new gvin to existing gconcat- leads to corrupted fs<br />
bsdlabel -e /dev/concat/v82-v84<br />
<br />
To figure out new size of the c partition, multiply 4194304 by the # of 2G gvinum volumes and subtract the # of 2G volumes:<br />
10G: 4194304 * 5 – 5 = 20971515<br />
8G: 4194304 * 4 – 4 = 16777212<br />
6G: 4194304 * 3 – 3 = 12582909<br />
4G: 4194304 * 2 – 2 = 8388606<br />
<br />
To figure out the new size of the a partition, subtract 16 from the c partition:<br />
10G: 20971515 – 16 = 20971499<br />
8G: 16777212 – 16 = 16777196<br />
6G: 12582909 – 16 = 12582893<br />
4G: 8388606 – 16 = 8388590<br />
<br />
Orig:<br />
8 partitions:<br />
# size offset fstype [fsize bsize bps/cpg]<br />
a: 8388590 16 4.2BSD 2048 16384 28552<br />
c: 8388606 0 unused 0 0 # "raw" part, don't edit<br />
<br />
New:<br />
8 partitions:<br />
# size offset fstype [fsize bsize bps/cpg]<br />
a: 12582893 16 4.2BSD 2048 16384 28552<br />
c: 12582909 0 unused 0 0 # "raw" part, don't edit<br />
<br />
sync; sync<br />
<br />
growfs /dev/concat/v82-v84a<br />
<br />
fsck –fy /dev/concat/v82-v84a<br />
<br />
sync<br />
<br />
fsck –fy /dev/concat/v82-v84a<br />
<br />
(keep running fsck’s till NO errors)<br />
<br />
== Problems un-mounting - and with mount_null’s ==<br />
<br />
If you cannot unmount a filesystem, beacuse it says the filesystem is busy, it is because of three things:<br />
<br />
a) the jail is still running<br />
<br />
b) you are actually in that directory, even though the jail is stopped<br />
<br />
c) there are still dev, null_mount or linprocfs mount points mounted inside that directory.<br />
<br />
d) when trying to umount null_mounts that are really long and you get an error like “No such file or directory”, it’s an OS bug where the dir is truncated. No known fix<br />
<br />
e) there are still files open somewhere inside the dir. Use <tt>fstat | grep <cid></tt> to find the process that has files open<br />
<br />
f) Starting with 6.x, the jail mechanism does a poor job of keeping track of processes running in a jail and if it thinks there are still procs running, it will refuse to umount the disk. If this is happening you should see a low number in the #REF column when you run jls. In this case you ''can'' safely <tt>umount –f</tt> the mount. <br />
<br />
Please note -if you forcibly unmount a (4.x) filesystem that has null_mounts<br />
still mounted in it, the system '''will crash''' within 10-15 mins.<br />
<br />
== Misc jail Items ==<br />
<br />
We are overselling hard drive space on jail2, jail8, jail9, a couple jails on jail17, jail4, jail12 and jail18.<br />
Even though the vn file shows 4G size, it doesn’t actually occupy that amount of space on the disk. So be careful not to fill up drives where we’re overselling – use oversellcheck to confirm you’re not oversold by more than 10G.<br />
There are other truncated jails, they are generally noted in a the file on the root system: /root/truncated<br />
<br />
The act of moving a truncated vn to another system un-does the truncating- the truncated vn is filled with 0’s and it occupies physical disk space for which it’s configured. So, you should use dumpremote to preserve the truncation.<br />
<br />
= FreeBSD VPS Management Tools =<br />
<br />
These files are located in /usr/local/jail/rc.d and /usr/local/jail/bin<br />
<br />
== jailmake ==<br />
<br />
Applies to 7.x+ <br />
On older systems syntax differs, run jailmake once to see.<br />
<br />
Note: this procedure differs on mx2 which is 7.x but still uses gvinum<br />
<br />
# run js to figure out which md’s are in use, which disk has enough space, IP to put it on<br />
# use col00xxx for both hostnames if they don’t give you a hostname<br />
# copy over dir, ip and password to pending customer screen<br />
<br />
<tt>Usage: jailmake IP[,IP] CID disk[1|2|3] md# hostname shorthost ipfw# email [size in GB]</tt><br />
<br />
Ex: <br />
<br />
Jail2# jailmake 69.55.234.66 col01334 3 97 vps.bsd.it vps 1334 fb@bsd.it<br />
<br />
== jailps ==<br />
jailps [hostname]<br />
DEPRECATED FOR jps: displays processes belonging to/running inside a jail. The command<br />
takes one (optional) argument – the hostname of the jail you wish to query. If you don’t <br />
supply an argument, all processes on the machine are listed and grouped by jail. <br />
<br />
== jps ==<br />
jps [hostname]<br />
displays processes belonging to/running inside a jail. The command<br />
takes one (optional) argument – the hostname or ID of the jail you wish to query. <br />
<br />
== jailkill ==<br />
jailkill <hostname><br />
stops all process running in a jail.<br />
<br />
You can also run:<br />
jailkill <JID><br />
<br />
=== problems ===<br />
Occasionally you will hit an issue where jail will not kill off:<br />
<br />
<pre>jail9# jailkill www.domain.com<br />
www.domain.com .. killed: none<br />
jail9#</pre><br />
Because no processes are running under that hostname. You cannot use jailps.pl either:<br />
<br />
<pre>jail9# jailps www.domain.com<br />
www.domain.com doesn’t exist on this server<br />
jail9#</pre><br />
<br />
The reasons for this are usually:<br />
* the jail is no longer running<br />
<br />
* the jail's hostname has changed<br />
In this case, <br />
<br />
>=6.x: run a <tt>jls|grep <jail's IP></tt> to find the correct hostname, then update the quad file, then kill the jail.<br />
<br />
<6.x: the first step is to cat their /etc/rc.conf file to see if you can tell what they set the new hostname to. This very often works. For example:<br />
<br />
cat /mnt/data2/198.78.65.136-col00261-DIR/etc/rc.conf<br />
<br />
But maybe they set the hostname with the hostname command, and the original hostname is still in /etc/rc.conf.<br />
<br />
The welcome email clearly states that they should tell us if they change their hostname, so there is no problem in just emailing them and asking them what they set the new hostname to.<br />
<br />
Once you know the new hostname OR if a customer simply emails to inform you that they have set the hostname to something different, you need to edit the quad and safe files that their system is in to input the new hostname.<br />
<br />
However, if push comes to shove and you cannot find out the hostname from them or from their system, then you need to start doing some detective work.<br />
<br />
The easiest thing to do is run jailps looking for a hostname similar to their original hostname. Or you could get into the /bin/sh shell by running:<br />
<br />
/bin/sh<br />
<br />
and then looking at every hostname of every process:<br />
<br />
for f in `ls /proc` ; do cat /proc/$f/status ; done<br />
<br />
and scanning for a hostname that is either similar to their original hostname, or that you don't see in any of the quad safe files.<br />
<br />
This is very brute force though, and it is possible that catting every file in /proc is dangerous - I don't recommend it. A better thing would be to identify any processes that you know belong to this system – perhaps the reason you are trying to find this system is because they are running something bad - and just catting the status from only that PID.<br />
<br />
Somewhere there’s a jail where there may be 2 systems named www. Look at /etc/rc.conf and make sure they’re both really www. If they are, jailkill www, jailps www to make sure not running. Then immediately restart the other one, as the fqdn (as found from a rev nslookup)<br />
<br />
* on >=6.x the hostname may not yet be hashed:<br />
<pre>jail9 /# jls<br />
JID Hostname Path IP Address(es)<br />
1 bitnet.dgate.org /mnt/data1/69.55.232.50-col02094-DIR 69.55.232.50<br />
2 ns3.hctc.net /mnt/data1/69.55.234.52-col01925-DIR 69.55.234.52<br />
3 bsd1 /mnt/data1/69.55.232.44-col00155-DIR 69.55.232.44<br />
4 let2.bbag.org /mnt/data1/69.55.230.92-col00202-DIR 69.55.230.92<br />
5 post.org /mnt/data2/69.55.232.51-col02095-DIR 69.55.232.51 ...<br />
6 ns2 /mnt/data1/69.55.232.47-col01506-DIR 69.55.232.47 ...<br />
7 arlen.server.net /mnt/data1/69.55.232.52-col01171-DIR 69.55.232.52<br />
8 deskfood.com /mnt/data1/69.55.232.71-col00419-DIR 69.55.232.71<br />
9 mirage.confluentforms.com /mnt/data1/69.55.232.54-col02105-DIR 69.55.232.54 ...<br />
10 beachmember.com /mnt/data1/69.55.232.59-col02107-DIR 69.55.232.59<br />
11 www.agottem.com /mnt/data1/69.55.232.60-col02109-DIR 69.55.232.60<br />
12 sdhobbit.myglance.org /mnt/data1/69.55.236.82-col01708-DIR 69.55.236.82<br />
13 ns1.jnielsen.net /mnt/data1/69.55.234.48-col00204-DIR 69.55.234.48 ...<br />
14 ymt.rollingegg.net /mnt/data2/69.55.236.71-col01678-DIR 69.55.236.71<br />
15 verse.unixlore.net /mnt/data1/69.55.232.58-col02131-DIR 69.55.232.58<br />
16 smcc-mail.org /mnt/data2/69.55.232.68-col02144-DIR 69.55.232.68<br />
17 kasoutsuki.w4jdh.net /mnt/data2/69.55.232.46-col02147-DIR 69.55.232.46<br />
18 dili.thium.net /mnt/data2/69.55.232.80-col01901-DIR 69.55.232.80<br />
20 www.tekmarsis.com /mnt/data2/69.55.232.66-col02155-DIR 69.55.232.66<br />
21 vps.yoxel.net /mnt/data2/69.55.236.67-col01673-DIR 69.55.236.67<br />
22 smitty.twitalertz.com /mnt/data2/69.55.232.84-col02153-DIR 69.55.232.84<br />
23 deliver4.klatha.com /mnt/data2/69.55.232.67-col02160-DIR 69.55.232.67<br />
24 nideffer.com /mnt/data2/69.55.232.65-col00412-DIR 69.55.232.65<br />
25 usa.hanyuan.com /mnt/data2/69.55.232.57-col02163-DIR 69.55.232.57<br />
26 daifuku.ppbh.com /mnt/data2/69.55.236.91-col01720-DIR 69.55.236.91<br />
27 collins.greencape.net /mnt/data2/69.55.232.83-col01294-DIR 69.55.232.83<br />
28 ragebox.com /mnt/data2/69.55.230.104-col01278-DIR 69.55.230.104<br />
29 outside.mt.net /mnt/data2/69.55.232.72-col02166-DIR 69.55.232.72<br />
30 vps.payneful.ca /mnt/data2/69.55.234.98-col01999-DIR 69.55.234.98<br />
31 higgins /mnt/data2/69.55.232.87-col02165-DIR 69.55.232.87 ...<br />
32 ozymandius /mnt/data2/69.55.228.96-col01233-DIR 69.55.228.96<br />
33 trusted.realtors.org /mnt/data2/69.55.238.72-col02170-DIR 69.55.238.72<br />
34 jc1.flanderous.com /mnt/data2/69.55.239.22-col01504-DIR 69.55.239.22<br />
36 guppylog.com /mnt/data2/69.55.238.73-col00036-DIR 69.55.238.73<br />
40 haliohost.com /mnt/data2/69.55.234.41-col01916-DIR 69.55.234.41 ...<br />
41 satyr.jorge.cc /mnt/data1/69.55.232.70-col01963-DIR 69.55.232.70<br />
jail9 /# jailkill satyr.jorge.cc<br />
ERROR: jail_: jail "satyr,jorge,cc" not found<br />
</pre><br />
<br />
Note how it's saying <tt>satyr,jorge,cc</tt> is not found, and not <tt>satyr.jorge.cc</tt>. <br />
<br />
The jail subsystem tracks things using comma-delimited hostnames. That is created every few hours:<br />
<br />
jail9 /# crontab -l<br />
0 0,6,12,18 * * * /usr/local/jail/bin/sync_jail_names<br />
<br />
So if we run this manually:<br />
jail9 /# /usr/local/jail/bin/sync_jail_names<br />
<br />
Then kill the jail:<br />
jail9 /# jailkill satyr.jorge.cc<br />
successfully killed: satyr,jorge,cc</pre><br />
<br />
It worked.<br />
<br />
<br />
If you ever see this when trying to kill a jail:<br />
<br />
<pre># jailkill e-scribe.com<br />
killing JID: 6 hostname: e-scribe.com<br />
3 procs running<br />
3 procs running<br />
3 procs running<br />
3 procs running<br />
...</pre><br />
<br />
<tt>[[#jailkill|jailkill]]</tt> probably got lost trying to kill off the jail. Just ctrl-c the jailkill process, then run a jailps on the hostname, and <tt>kill -9</tt> any process which is still running. Keep running jailps and <tt>kill -9</tt> till all processes are gone.<br />
<br />
== jailpsall ==<br />
jailpsall<br />
will run a jailps on all jails configured in the quad files (this is different from<br />
jailps with no arguments as it won’t help you find a “hidden” system)<br />
<br />
== jailpsw ==<br />
jailpsw<br />
will run a jailps with an extra -w to provide wider output<br />
<br />
== jt (>=7.x) ==<br />
jt<br />
displays the top 20 processes on the server (the top 20 processes from top) and <br />
which jail owns them. This is very helpful for determining who is doing what when<br />
the server is very busy.<br />
<br />
== jtop (>=7.x) ==<br />
jtop<br />
a wrapper for top displaying processes on the server and which jail owns them. Constantly updates, like top. <br />
<br />
== jtop (<7.x) ==<br />
jtop<br />
displays the top 20 processes on the server (the top 20 processes from top) and <br />
which jail owns them. This is very helpful for determining who is doing what when<br />
the server is very busy.<br />
<br />
== stopjail ==<br />
stopjail <hostname> [1]<br />
this will jailkill, umount and vnconfig –u a jail. If passed an optional 2nd<br />
argument, it will not exit before umounting and un-vnconfig’ing in the event<br />
jailkill returns no processes killed. This is useful if you just want to umount<br />
and vnconfig –u a jail you’ve already killed. It is intelligent in that it won’t <br />
try to umount or vnconfig –u if it’s not necessary.<br />
<br />
== startjail ==<br />
startjail <hostname><br />
this will start vnconfig, mount (including linprocfs and null-mounts), and start a jail.<br />
Essentially, it reads the jail’s relevant block from the right quad file and executes it.<br />
It is intelligent in that it won’t try to mount or vnconfig if it’s not necessary.<br />
<br />
== jpid ==<br />
jpid <pid><br />
displays information about a process – including which jail owns it.<br />
It’s the equivalent of running cat /proc/<pid>/status<br />
<br />
== canceljail ==<br />
canceljail <hostname> [1]<br />
this will stop a jail (the equivalent of stopjail), check for backups (offer to remove them <br />
from the backup server and the backup.config), rename the vnfile, remove the dir, and <br />
edit quad/safe. If passed an optional 2nd argument, it will not exit upon failing to kill<br />
and processes owned by the jail. This is useful if you just want to cancel a jail which <br />
is already stopped.<br />
<br />
== jls ==<br />
jls [-v]<br />
Lists all jails running:<br />
<pre>JID #REF IP Address Hostname Path<br />
101 135 69.55.224.148 mail.pc9.org /mnt/data2/69.55.224.148-col01034-DIR</pre><br />
<br />
#REF is the number of references or procs(?) running<br />
<br />
Running with -v will give you all IPs assigned to each jail (7.2 up)<br />
<pre>JID #REF Hostname Path IP Address(es)<br />
101 139 mail.pc9.org /mnt/data2/69.55.224.148-col01034-DIR 69.55.224.14869.55.234.85</pre><br />
<br />
== startalljails ==<br />
startalljails<br />
7.2+ only. This will parse through quad1 and start all jails. It utilizes lockfiles so it won’t try to start a jail more than once- therefore multiple instances can be running in parallel without fear of starting a jail twice. If a jail startup gets stuck, you can ^C without fear of killing the script. IMPORTANT- before running startalljails you should make sure you ran preboot once as it will clear out all the lockfiles and enable startalljails to work properly.<br />
<br />
== aaccheck.sh ==<br />
aaccheck.sh<br />
displayes the output of container list and task list from aaccli<br />
<br />
== backup ==<br />
backup<br />
backup script called nightly to update jail scripts and do customer backups<br />
<br />
== buildsafe ==<br />
buildsafe<br />
creates safe files based on quads (automatically removing the fsck’s). This will destructively overwrite safe files<br />
<br />
== checkload.pl ==<br />
checkload.pl<br />
this was intended to be setup as a cronjob to watch processes on a jail when the load <br />
rises above a certain level. Not currently in use.<br />
<br />
== checkprio.pl ==<br />
checkprio.pl<br />
will look for any process (other than the current shell’s csh, sh, sshd procs) with a non-normal priority and normalize it<br />
<br />
== diskusagemon == <br />
diskusagemon <mount point> <1k blocks><br />
watches a mount point’s disk use, when it reaches the level specified in the 2nd argument,<br />
it exits. This is useful when doing a restore and you want to be paged as it’s nearing completion.<br />
Best used as: <tt>diskusagemon /asd/asd 1234; pagexxx</tt><br />
<br />
== dumprestore ==<br />
dumprestore <dumpfile><br />
this is a perl expect script which automatically enters ‘1’ and ‘y’. It seems to cause restore to fail<br />
to set owner permissions on large restores.<br />
<br />
== g ==<br />
g <search><br />
greps the quad/safe files for the given search parameter<br />
<br />
== gather.pl ==<br />
gather.pl<br />
gathers up data about jails configured and writes to a file. Used for audits against the db<br />
<br />
== gb ==<br />
gb <search><br />
greps backup.config for the given search parameter<br />
<br />
== gbg ==<br />
gbg <search><br />
greps backup.config for the given search parameter and presents just the directories (for clean pasting)<br />
<br />
== ipfwbackup ==<br />
ipfwbackup<br />
writes ipfw traffic count data to a logfile<br />
<br />
== ipfwreset ==<br />
ipfwreset<br />
writes ipfw traffic count data to a logfile and resets counters to 0<br />
<br />
== js ==<br />
js<br />
output varies by OS version, but generally provides information about the base jail:<br />
- which vn’s are in use<br />
- disk usage<br />
- info about the contents of quads<br />
- the # of inodes represented by the jails contained in the group (133.2 in the example below), and how many jails per data mount, as well as subtotals<br />
- ips bound to the base machine but not in use by a jail<br />
- free gvinum volumes, or unused vn’s or used md’s<br />
<br />
<pre>/usr/local/jail/rc.d/quad1:<br />
/mnt/data1 133.2 (1)<br />
/mnt/data2 1040.5 (7)<br />
total 1173.7 (8)<br />
/usr/local/jail/rc.d/quad2:<br />
/mnt/data1 983.4 (6)<br />
total 983.4 (6)<br />
/usr/local/jail/rc.d/quad3:<br />
/mnt/data1 693.4 (4)<br />
/mnt/data2 371.6 (3)<br />
total 1065 (7)<br />
/usr/local/jail/rc.d/quad4:<br />
/mnt/data1 466.6 (3)<br />
/mnt/data2 882.2 (5)<br />
total 1348.8 (8)<br />
/mnt/data1: 2276.6 (14)<br />
/mnt/data2: 2294.3 (15)<br />
<br />
Available IPs:<br />
69.55.230.11 69.55.230.13 69.55.228.200<br />
<br />
Available volumes:<br />
v78 /mnt/data2 2G<br />
v79 /mnt/data2 2G<br />
v80 /mnt/data2 2G</pre> <br />
<br />
== load.pl ==<br />
load.pl<br />
feeds info to load mrtg - executed by inetd.<br />
<br />
== makevirginjail ==<br />
makevirginjail<br />
Only on some systems, makes an empty jail (doesn't do restore step)<br />
<br />
== mb == <br />
mb <mount|umount><br />
(nfs) mounts and umounts dirs to backup2. Shortcuts are mbm and mbu to mount and unmount. <br />
<br />
== notify.sh ==<br />
notify.sh<br />
emails reboot@johncompanies.com – intended to be called at boot time to alert us to a machine which panics and reboots and isn’t caught by bb or castle.<br />
<br />
== orphanedbackupwatch ==<br />
orphanedbackupwatch<br />
looks for directories on backup2 which aren’t configured in backup.config and offers to delete them<br />
<br />
== postboot ==<br />
postboot<br />
to be run after a machine reboot and quad/safe’s are done executing. It will:<br />
* do chmod 666 on each jail’s /dev/null<br />
* add ipfw counts<br />
* run jailpsall (so you can see if a configured jail isn’t running)<br />
<br />
== preboot ==<br />
preboot<br />
to be run before running quad/safe – checks for misconfigurations: <br />
* a jail configured in a quad but not a safe<br />
* a jail is listed more than once in a quad<br />
* the ip assigned to a jail isn’t configured on the machine<br />
* alias numbering skips in the rc.conf (resulting in the above)<br />
* orphaned vnfile's that aren't mentioned in a quad/safe<br />
* ip mismatches between dir/vnfile name and the jail’s ip<br />
* dir/vnfiles's in quad/safe that don’t exist <br />
<br />
== quadanalyze.pl ==<br />
quadanalyze.pl<br />
called by js, produces the info (seen above with js explanation) about the contents of quad (inode count, # of jails, etc.)<br />
<br />
== rsync.backup ==<br />
rsync.backup<br />
does customer backups (relies on backup.config)<br />
<br />
== taskdone ==<br />
taskdone<br />
when called will email support@johncompanies.com with the hostname of the machine from which it was executed as the subject<br />
<br />
== topten ==<br />
topten<br />
summarizes the top 10 traffic users (called by ipfwreset)<br />
<br />
== trafficgather.pl ==<br />
trafficgather.pl [yy-mm]<br />
sends a traffic usage summary by jail to support@johncomapnies.com and payments@johncompanies.com. Optional arguments are year and month (must be in the past). If not passed, assumes last month. Relies on traffic logs created by ipfwreset and ipfwbackup<br />
<br />
== trafficwatch.pl ==<br />
trafficwatch.pl<br />
checks traffic usage and emails support@johncomapnies.com when a jail reaches the warning level (35G) and the limit (40G). We really aren’t using this anymore now that we have netflow.<br />
<br />
== trafstats ==<br />
trafstats<br />
writes ipfw traffic usage info by jail to a file called jc_traffic_dump in each jail’s / dir<br />
<br />
== truncate_jailmake ==<br />
truncate_jailmake<br />
a version of jailmake which creates truncated vnfiles.<br />
<br />
== vb ==<br />
vb<br />
the equivalent of: <tt>vi /usr/local/jail/bin/backup.config</tt><br />
<br />
== vs (freebsd) ==<br />
vs<n><br />
the equivalent of: <tt>vi /usr/local/jail/rc.d/safe<n></tt><br />
<br />
vq<n><br />
the equivalent of: vi /usr/local/jail/rc.d/quad<n><br />
<br />
== dumpremote ==<br />
dumpremote <user@machine> </remote/location/file-dump> <vnX><br />
ex: dumpremote user@10.1.4.117 /mnt/data3/remote.echoditto.com-dump 7<br />
this will dump a vn filesystem to a remote machine and location<br />
<br />
== oversellcheck ==<br />
oversellcheck<br />
displays how much a disk is oversold or undersold taking into account truncated vn files. Only for use on 4.x systems<br />
<br />
== mvbackups (freebsd) ==<br />
mvbackups <dir> (1.1.1.1-col00001-DIR) <target_machine> (jail1) <target_dir> (data1)<br />
moves backups from one location to another on the backup server, and provides you with option to remove entries from current backup.config, and simple paste command to add the config to backup.config on the target server<br />
<br />
== jailnice ==<br />
jailnice <hostname><br />
applies <tt>renice 19 [PID]</tt> and <tt>rtprio 31 –[PID]</tt> to each process in the given jail<br />
<br />
== dumpremoterestore ==<br />
dumpremoterestore <device> <ip of target machine> <dir on target machine><br />
ex: dumpremoterestore /dev/vn51 10.1.4.118 /mnt/data2/69.55.239.45-col00688-DIR<br />
dumps a device and restores it to a directory on a remote machine. Requires that you enable root ssh on the <br />
remote machine.<br />
<br />
== psj ==<br />
psj<br />
shows just the procs running on the base system – a ps auxw but without jail’d procs present<br />
<br />
== perc5iraidchk ==<br />
perc5iraidchk<br />
checks for degraded arrays on Dell 2950 systems with Perc5/6 controllers<br />
<br />
== perc4eraidchk ==<br />
perc4eraidchk<br />
checks for degraded arrays on Dell 2850 systems with Perc4e/Di controllers<br />
<br />
= Virtuozzo VPS =<br />
<br />
Note: We use VEID (Virtual Environment ID) and CTID (Container ID) interchangably. Similarly, VE and CT. They mean the same thing.<br />
VZPP = VirtuoZzo Power Panel (the control panel for each CT)<br />
<br />
All linux systems exist in /vz, /vz1 or /vz2 - since each linux machine holds roughly 60-90 customers, there will be roughly 30-45 in each partition.<br />
<br />
The actual filesystem of the system in question is in:<br />
<br />
/vz(1-2)/private/(VEID)<br />
<br />
Where VEID is the identifier for that system - an all-numeric string larger than 100.<br />
<br />
The actual mounted and running systems are in the corresponding:<br />
<br />
/vz(1-2)/root/(VEID)<br />
<br />
But we rarely interact with any system from this mount point.<br />
<br />
You should never need to touch the root portion of their system – however you can traverse their filesystem by going to <tt>/vz(1-2)/private/(VEID)/root</tt> (<tt>/vz(1-2)/private/(VEID)/fs/root</tt> on 4.x systems) the root of their filesystem is in that directory, and their entire system is underneath that.<br />
<br />
Every VE has a startup script in <tt>/etc/sysconfig/vz-scripts</tt> (which is symlinked as <tt>/vzconf</tt> on all systems) - the VE startup script is simply named <tt>(VEID).conf</tt> - it contains all the system parameters for that VE:<br />
<br />
<pre># Configuration file generated by vzsplit for 60 VE<br />
# on HN with total amount of physical mem 2011 Mb<br />
<br />
VERSION="2"<br />
CLASSID="2"<br />
<br />
ONBOOT="yes"<br />
<br />
KMEMSIZE="8100000:8200000"<br />
LOCKEDPAGES="322:322"<br />
PRIVVMPAGES="610000:615000"<br />
SHMPAGES="33000:34500"<br />
NUMPROC="410:415"<br />
PHYSPAGES="0:2147483647"<br />
VMGUARPAGES="13019:2147483647"<br />
OOMGUARPAGES="13019:2147483647"<br />
NUMTCPSOCK="1210:1215"<br />
NUMFLOCK="107:117"<br />
NUMPTY="19:19"<br />
NUMSIGINFO="274:274"<br />
TCPSNDBUF="1800000:1900000"<br />
TCPRCVBUF="1800000:1900000"<br />
OTHERSOCKBUF="900000:950000"<br />
DGRAMRCVBUF="200000:200000"<br />
NUMOTHERSOCK="650:660"<br />
DCACHE="786432:818029"<br />
NUMFILE="7500:7600"<br />
AVNUMPROC="51:51"<br />
IPTENTRIES="155:155"<br />
DISKSPACE="4194304:4613734"<br />
DISKINODES="400000:420000"<br />
CPUUNITS="1412"<br />
QUOTAUGIDLIMIT="2000"<br />
VE_ROOT="/vz1/root/636"<br />
VE_PRIVATE="/vz1/private/636"<br />
NAMESERVER="69.55.225.225 69.55.230.3"<br />
OSTEMPLATE="vzredhat-7.3/20030305"<br />
VE_TYPE="regular"<br />
IP_ADDRESS="69.55.225.229"<br />
HOSTNAME="textengine.net"</pre><br />
<br />
<br />
As you can see, the hostname is set here, the disk space is set here, the number of inodes, the number of files that can be open, the number of tcp sockets, etc. - all are set here.<br />
<br />
In fact, everything that can be set on this customer system is set in this conf file.<br />
<br />
<br />
All interaction with the customer system is done with the VEID. You start the system by running:<br />
<br />
vzctl start 999<br />
<br />
You stop it by running:<br />
<br />
vzctl stop 999<br />
<br />
You execute commands in it by running:<br />
<br />
vzctl exec 999 df -k<br />
<br />
You enter into it, via a root-shell backdoor with:<br />
<br />
vzctl enter 999<br />
<br />
and you set parameters for the system, while it is still running, with:<br />
<br />
vzctl set 999 --diskspace 6100000:6200000 --save<br />
<br />
<tt>vzctl</tt> is the most commonly used command - we have aliased <tt>v</tt> to <tt>vzctl</tt> since we use it so often. We’ll continue to use <tt>vzctl</tt> in our examples, but feel free to use just <tt>v</tt>.<br />
<br />
Let's say the user wants more diskspace. You can cat their conf file and see:<br />
<br />
DISKSPACE="4194304:4613734"<br />
<br />
So right now they have 4gigs of space. You can then change it to 6 with:<br />
<br />
vzctl set 999 --diskspace 6100000:6200000 --save<br />
<br />
IMPORTANT: all issuances of the vzctl set command need to end with <tt>–save</tt> - if they don't, the setting will be set, but it will not be saved to the conf file, and they will not have those settings next time they boot.<br />
<br />
All of the tunables in the conf file can be set with the vzctl set command. Note that in the conf file, and on the vzctl set command line, we always issue two numbers seperated by a colon - that is because we are setting the hard and soft limits. Always set the hard limit slightly above the soft limit, as you see it is in the conf file for all those settings.<br />
<br />
There are also things you can set with `<tt>vzctl set</tt>` that are not in the conf file as settings, per se. For instance, you can add IPs:<br />
<br />
vzctl set 999 --ipadd 10.10.10.10 --save<br />
<br />
or multiple IPs:<br />
<br />
vzctl set 999 --ipadd 10.10.10.10 --ipadd 10.10.20.30 --save<br />
<br />
or change the hostname:<br />
<br />
vzctl set 999 --hostname www.example.com --save<br />
<br />
You can even set the nameservers:<br />
<br />
vzctl set 999 --nameserver 198.78.66.4 --nameserver 198.78.70.180 --save<br />
<br />
Although you probably will never do that.<br />
<br />
You can disable a VPS from being started (by VZPP or reboot) (4.x):<br />
<br />
vzctl set 999 --disabled yes --save <br />
<br />
You can disable a VPS from being started (by VZPP or reboot) (<=3.x):<br />
<br />
vzctl set 999 --onboot=no --save <br />
<br />
You can disable a VPS from using his control panel:<br />
<br />
vzctl set 999 --offline_management=no --save <br />
<br />
You can suspend a VPS, so it can be resumed in the same state it was in when it was stopped (4.x):<br />
<br />
vzctl suspend 999<br />
<br />
and to resume it:<br />
<br />
vzctl resume 999<br />
<br />
to see who owns process:<br />
vzpid <PID><br />
<br />
to mount up an unmounted ve:<br />
vzctl mount 827<br />
<br />
<br />
To see network stats for CT's:<br />
<pre># vznetstat<br />
VEID Net.Class Output(bytes) Input(bytes)<br />
-----------------------------------------------<br />
24218 1 484M 39M<br />
24245 1 463M 143M<br />
2451 1 2224M 265M<br />
2454 1 2616M 385M<br />
4149 1 125M 68M<br />
418 1 1560M 34M<br />
472 1 1219M 315M<br />
726 1 628M 317M<br />
763 1 223M 82M<br />
771 1 4234M 437M<br />
-----------------------------------------------<br />
Total: 13780M 2090M</pre><br />
<br />
<br />
One thing that sometimes comes up on older systems that we created with smaller defaults is that the system would run out of inodes. The user will email and say they cannot create any more files or grow any files larger, but they will also say that they are not out of diskspace ... they are running:<br />
<br />
df -k<br />
<br />
and seeing how much space is free - and they are not out of space. They are most likely out of inodes - which they would see by running:<br />
<br />
df -i<br />
<br />
So, the first thing you should do is enter their system with:<br />
<br />
vzctl enter 999<br />
<br />
and run: <tt>df -i</tt><br />
<br />
to confirm your theory. Then exit their system. Then simply cat their conf file and see what their inodes are set to (probably 200000:200000, since that was the old default on the older systems) and run:<br />
<br />
vzctl set 999 --diskinodes 400000:400000 --save<br />
<br />
If they are not out of inodes, then a good possibility is that they have maxed out their numfile configuration variable, which controls how many files they can have in their system. The current default is 7500 (which nobody has ever hit), but the old default was as low as 2000, so you would run something like:<br />
<br />
vzctl set 999 --numfile 7500:7500 --save<br />
<br />
----<br />
<br />
You cannot start or stop a VE if your pwd is its private (/vz/private/999) or root (/vz/root/999) directories, or anywhere below them.<br />
<br />
<br />
== Recovering from a crash (linux) ==<br />
<br />
=== Diagnose whether you have a crash ===<br />
The most important thing is to get the machine and all ve’s back up as soon as possible. Note the time, you’ll need to create a crash log entry (Mgmt. -> Reference -> CrashLog). The first thing to do is head over to the [[Screen#Screen_Organization|serial console screen]] and see if there’s any kernel error messages output. Try to copy any messages (or just a sample of repeating messages) you see into the notes section of the crash log – these will also likely need to be sent to virtuozzo for interpretation. If the messages are spewing too fast, hit ^O + H to start a screen log dump which you can ob1182.pts-38.bb serve after the machine is rebooted. Additionally, if the machine is responsive, you can get a trace to send to virtuozzo by hooking up a kvm and entering these 3 sequences:<br />
<pre>alt+print screen+m<br />
alt+print screen+p<br />
alt+print screen+t</pre><br />
<br />
If there are no messages, the machine may just be really busy- wait a bit (5-10min) to see if it comes back. If it's still pinging, odds are its very busy. If it doesn't come back, or the messages indicate a fatal error, you will need to proceed with a power cycle (ctrl+alt+del will not work).<br />
<br />
=== Power cycle the server ===<br />
If this machine is not a Dell 2950 with a [[DRAC/RMM#DRAC|DRAC card]] (i.e. if you can’t ssh into the DRAC card and issue racadm serveraction hardreset, then you will need someone at the data center to power the macine off, wait 30 sec, then turn it back on. Make sure to re-attach via console (<tt>tip virtxx</tt>) immediately after power down. <br />
<br />
=== (Re)attach to the console ===<br />
Stay on the console the entire time during boot. As the BIOS posts- look out for the RAID card output- does everything look healthy? The output may be scrambled, look for "DEGRADED" or "FAILED". Once the OS starts booting you will be disconnected (dropped back to the shell on the console server) a couple times during the boot up. The reason you want to quickly re-attach is two-fold: 1. If you don’t reattach quickly then you won’t get any console output, 2. you want to be attached before the server ''potentially'' starts (an extensive) fsck. If you attach after the fsck begins, you’ll have seen no indication it started an fsck and the server will appear frozen during startup- no output, no response. <br />
<br />
=== Start containers/VE's/VPSs ===<br />
When the machine begins to start VE’s, it’s safe to leave the console and login via ssh. All virts should be set to auto start all the VEs after a crash. Further, most (newer) virts are set to “fastboot” it’s VE’s (to find out, do:<br />
grep -i fast /etc/sysconfig/vz <br />
and look for <tt>VZFASTBOOT=yes</tt>). If this was set prior to the machine’s crash (setting it after the machine boots will not have any effect until the vz service is restarted) it will start each ve as fast as possible, in serial, then go thru each VE (serially), shutting it down running a vzquota (disk usage) check, then bringing it back up. The benefit is that all VE’s are brought up quickly (within 15min or so depending on the #), the downside is a customer watching closely will notice 2 outages – 1st the machine crash, 2nd their quota check (which will be a much shorter downtime- on the order of a few minutes). <br />
<br />
Where “fastboot” is not set to yes (i.e on quar1), vz will start them consecutively, checking the quotas one at a time, and the 60th VE may not start until an hour or two later - this is not acceptable.<br />
<br />
The good news is, if you run vzctl start for a VE that is already started, you will simply get an error: <tt>VE is already started</tt>. Further, if you attempt to vzctl start a VE that is in the process of being started, you will simply get an error: unable to lock VE. So, there is no danger in simply running scripts to start smaller sets of VEs. If the system is not autostarting, then there is no issue, and even if it does, when it conflicts, one process (yours or the autostart) will lose, and just move on to the next one.<br />
<br />
A script has been written to assist with ve starts: [[#startvirt.pl|startvirt.pl]] which will start 6 ve’s at once until there are no more left. If startvirt.pl is used on a system where “fastboot” was on, it will circumvent the fastboot for ve’s started by startvirt.pl – they will go through the complete quota check before starting- therefore this is not advisable when a system has crashed. When a system is booted cleanly, and there's no need for vzquota checks, then startvirt.pl is safe and advisable to run.<br />
<br />
=== Make sure all containers are running ===<br />
You can quickly get a feel for how many ve’s are started by running:<br />
<br />
<pre>[root@virt4 log]# vs<br />
VEID 16066 exist mounted running<br />
VEID 16067 exist mounted running<br />
VEID 4102 exist mounted running<br />
VEID 4112 exist mounted running<br />
VEID 4116 exist mounted running<br />
VEID 4122 exist mounted running<br />
VEID 4123 exist mounted running<br />
VEID 4124 exist mounted running<br />
VEID 4132 exist mounted running<br />
VEID 4148 exist mounted running<br />
VEID 4151 exist mounted running<br />
VEID 4155 exist mounted running<br />
VEID 42 exist mounted running<br />
VEID 432 exist mounted running<br />
VEID 434 exist mounted running<br />
VEID 442 exist mounted running<br />
VEID 450 exist mounted running<br />
VEID 452 exist mounted running<br />
VEID 453 exist mounted running<br />
VEID 454 exist mounted running<br />
VEID 462 exist mounted running<br />
VEID 463 exist mounted running<br />
VEID 464 exist mounted running<br />
VEID 465 exist mounted running<br />
VEID 477 exist mounted running<br />
VEID 484 exist mounted running<br />
VEID 486 exist mounted running<br />
VEID 490 exist mounted running</pre><br />
<br />
So to see how many ve’s have started:<br />
<pre>[root@virt11 root]# vs | grep running | wc -l<br />
39</pre><br />
<br />
And to see how many haven’t:<br />
<pre>[root@virt11 root]# vs | grep down | wc -l<br />
0</pre><br />
<br />
And how many we should have running:<br />
<pre>[root@virt11 root]# vs | wc -l<br />
39</pre><br />
<br />
Another tool you can use to see which ve’s have started, among other things is [[#vzstat|vzstat]]. It will give you CPU, memory, and other stats on each ve and the overall system. It’s a good thing to watch as ve’s are starting (note the VENum parameter, it will tell you how many have started):<br />
<br />
<pre>4:37pm, up 3 days, 5:31, 1 user, load average: 1.57, 1.68, 1.79<br />
VENum 40, procs 1705: running 2, sleeping 1694, unint 0, zombie 9, stopped 0<br />
CPU [ OK ]: VEs 57%, VE0 0%, user 8%, sys 7%, idle 85%, lat(ms) 412/2<br />
Mem [ OK ]: total 6057MB, free 9MB/54MB (low/high), lat(ms) 0/0<br />
Swap [ OK ]: tot 6142MB, free 4953MB, in 0.000MB/s, out 0.000MB/s<br />
Net [ OK ]: tot: in 0.043MB/s 402pkt/s, out 0.382MB/s 4116pkt/s<br />
Disks [ OK ]: in 0.002MB/s, out 0.000MB/s<br />
<br />
VEID ST %VM %KM PROC CPU SOCK FCNT MLAT IP<br />
1 OK 1.0/17 0.0/0.4 0/32/256 0.0/0.5 39/1256 0 9 69.55.227.152<br />
21 OK 1.3/39 0.1/0.2 0/46/410 0.2/2.8 23/1860 0 6 69.55.239.60<br />
133 OK 3.1/39 0.1/0.3 1/34/410 6.3/2.8 98/1860 0 0 69.55.227.147<br />
263 OK 2.3/39 0.1/0.2 0/56/410 0.3/2.8 34/1860 0 1 69.55.237.74<br />
456 OK 17/39 0.1/0.2 0/100/410 0.1/2.8 48/1860 0 11 69.55.236.65<br />
476 OK 0.6/39 0.0/0.2 0/33/410 0.1/2.8 96/1860 0 10 69.55.227.151<br />
524 OK 1.8/39 0.1/0.2 0/33/410 0.0/2.8 28/1860 0 0 69.55.227.153<br />
594 OK 3.1/39 0.1/0.2 0/45/410 0.0/2.8 87/1860 0 1 69.55.239.40<br />
670 OK 7.7/39 0.2/0.3 0/98/410 0.0/2.8 64/1860 0 216 69.55.225.136<br />
691 OK 2.0/39 0.1/0.2 0/31/410 0.0/0.7 25/1860 0 1 69.55.234.96<br />
744 OK 0.1/17 0.0/0.5 0/10/410 0.0/0.7 7/1860 0 6 69.55.224.253<br />
755 OK 1.1/39 0.0/0.2 0/27/410 0.0/2.8 33/1860 0 0 192.168.1.4<br />
835 OK 1.1/39 0.0/0.2 0/19/410 0.0/2.8 5/1860 0 0 69.55.227.134<br />
856 OK 0.3/39 0.0/0.2 0/13/410 0.0/2.8 16/1860 0 0 69.55.227.137<br />
936 OK 3.2/52 0.2/0.4 0/75/410 0.2/0.7 69/1910 0 8 69.55.224.181<br />
1020 OK 3.9/39 0.1/0.2 0/60/410 0.1/0.7 55/1860 0 8 69.55.227.52<br />
1027 OK 0.3/39 0.0/0.2 0/14/410 0.0/2.8 17/1860 0 0 69.55.227.83<br />
1029 OK 1.9/39 0.1/0.2 0/48/410 0.2/2.8 25/1860 0 5 69.55.227.85<br />
1032 OK 12/39 0.1/0.4 0/80/410 0.0/2.8 41/1860 0 8 69.55.227.90</pre><br />
<br />
When you are all done, you will want to make sure that all the VEs really did get started, run vs one more time.<br />
<br />
Note the time all ve’s are back up and enter that into and save the crash log entry.<br />
<br />
Occasionally, a ve will not start automatically. The most common reason for a ve not to come up normally is the ve was at it’s disk limit before the crash, and will not start since they’re over the limit. To overcome this, set the disk space to current usage level (the system will give this to you when it fails to start), start the ve, then re-set the disk space back to the prior level. Lastly, contact the customer to let them know they’re out of disk (or allocate more disk if they're entitled to more).<br />
<br />
== Hitting performance barriers and fixing them ==<br />
<br />
There are multiple modes virtuozzo offers to allocate resources to a ve. We utilize 2: SLM and UBC parameters<br />
On our 4.x systems, we use all SLM – it’s simpler to manage and understand. There are a few systems on virt19/18 that may also use SLM. Everything else uses UBC. <br />
You can tell a SLM ve by:<br />
<br />
SLMMODE="all"<br />
<br />
in their conf file. <br />
<br />
TODO: detail SLM modes and parameters.<br />
<br />
If someone is in SLM mode and they hit memory resource limits, they simply need to upgrade to more memory.<br />
<br />
The following applies to everyone else (UBC).<br />
<br />
Customers will often email and say that they are getting out of memory errors - a common one is "cannot fork" ... basically, anytime you see something odd like this, it means they are hitting one of their limits that is in place in their conf file.<br />
<br />
The conf file, however, simply shows their limits - how do we know what they are currently at ?<br />
<br />
The answer is a file called v - this file contains the current status (and peaks) of their performance settings, and also counts how many times they have hit the barrier. The output of the file looks like this:<br />
<br />
<pre>764: kmemsize 384113 898185 8100000 8200000 0<br />
lockedpages 0 0 322 322 0<br />
privvmpages 1292 7108 610000 615000 0<br />
shmpages 270 528 33000 34500 0<br />
dummy 0 0 0 0 0<br />
numproc 8 23 410 415 0<br />
physpages 48 5624 0 2147483647 0<br />
vmguarpages 0 0 13019 2147483647 0<br />
oomguarpages 641 6389 13019 2147483647 0<br />
numtcpsock 3 21 1210 1215 0<br />
numflock 1 3 107 117 0<br />
numpty 0 2 19 19 0<br />
numsiginfo 0 4 274 274 0<br />
tcpsndbuf 0 80928 1800000 1900000 0 <br />
tcprcvbuf 0 108976 1800000 1900000 0<br />
othersockbuf 2224 37568 900000 950000 0<br />
dgramrcvbuf 0 4272 200000 200000 0<br />
numothersock 3 9 650 660 0<br />
dcachesize 53922 100320 786432 818029 0<br />
numfile 161 382 7500 7600 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
numiptent 4 4 155 155 0</pre><br />
<br />
The first column is the name of the counter in question - the same names we saw in the systems conf file. The second column is the _current_ value of that counter, the third column is the max that that counter has ever risen to, the fourth column is the soft limit, and the fifth column is the hard limit (which is the same as the numbers in that systems conf file).<br />
<br />
The sixth number is the failcount - how many times the current usage has risen to hit the barrier. It will increase as soon as the current usage hits the soft limit.<br />
<br />
The problem with /proc/user_beancounters is that it actually contains that set of data for every running VE - so you can't just cat /proc/user_beancounters - it is too long and you get info for every other running system.<br />
<br />
You can vzctl enter the system and run:<br />
<br />
vzctl enter 9999<br />
cat /proc/user_beancounters<br />
<br />
inside their system, and you will just see the stats for their particular system, but entering their system every time you want to see it is combersome.<br />
<br />
So, I wrote a simple script called "vzs" which simply greps for the VEID, and spits out the next 20 or so lines (however many lines there are in the output, I forget) after it. For instance:<br />
<br />
<pre># vzs 765:<br />
765: kmemsize 2007936 2562780 8100000 8200000 0<br />
lockedpages 0 8 322 322 0<br />
privvmpages 26925 71126 610000 615000 0<br />
shmpages 16654 16750 33000 34500 0<br />
dummy 0 0 0 0 0<br />
numproc 41 57 410 415 0<br />
physpages 1794 49160 0 2147483647 0<br />
vmguarpages 0 0 13019 2147483647 0<br />
oomguarpages 4780 51270 13019 2147483647 0<br />
numtcpsock 23 37 1210 1215 0<br />
numflock 17 39 107 117 0<br />
numpty 1 3 19 19 0<br />
numsiginfo 0 6 274 274 0<br />
tcpsndbuf 22240 333600 1800000 1900000 0<br />
tcprcvbuf 0 222656 1800000 1900000 0<br />
othersockbuf 104528 414944 900000 950000 0<br />
dgramrcvbuf 0 4448 200000 200000 0<br />
numothersock 73 105 650 660 0<br />
dcachesize 247038 309111 786432 818029 0<br />
numfile 904 1231 7500 7600 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
numiptent 4 4 155 155 0</pre><br />
<br />
That showed us just the portion of /proc/user_beancounters for system 765.<br />
<br />
When you run the vzs command, always add a : after the VEID.<br />
<br />
So, if a customer complains about some out of memory errors, or no more files, or no more ptys, or just has an unspecific complain about processes dying, etc., the very first thing you need to do is check their beancounters with vzs. Usually you will spot an item that has a high failcount and needs to be upped.<br />
<br />
At that point you could simply up the counter with `vzctl set`. Generally pick a number 10-20% higher than the old one, and make the hard limit slightly larger than the the soft limit. However our systems now come in several levels and those levels have more/different memory allocations. If someone is complaining about something other than a memory limit (pty, numiptent, numflock), it’s generally safe to increase it, at least to the same level as what’s in the /vzconf/4unlimited file on the newest virt. If someone is hitting a memory limit, first make sure they are given what they deserve:<br />
<br />
(refer to mgmt -> payments -> packages)<br />
<br />
To set those levels, you use the [[#setmem|setmem]] command. <br />
<br />
The alternate (DEPRECATED) method would be to use one of 3 commands:<br />
256 <veid><br />
300 <veid><br />
384 <veid><br />
512 <veid><br />
<br />
If the levels were not right (you’d run vzs <veid> before and after to see the effect) tell the customer they’ve been adjusted and be done with it. If the levels were right, tell the customer they must upgrade to a higher package, tell them how to see level (control panel) and that they can reboot their system to escape this lockup contidion.<br />
<br />
Customers can also complain that their site is totally unreachable, or complain that it is down ... if the underlying machine is up, and all seems well, you may notice in the beancounters that network-specific counters are failing - such as numtcpsock, tcpsndbuf or tcprcvbuf. This will keep them from talking on the network and make it seem like their system is down. Again, just up the limits and things should be fine.<br />
<br />
On virts 1-4, you should first look at the default settings for that item on a later virt, such as virt 8 - we have increased the defaults a lot since the early machines. So, if you are going to up a counter on virt2, instead of upping it by 10-20%, instead up it to the new default that you see on virt8.<br />
<br />
== Moving a VE to another virt (migrate/migrateonline) ==<br />
<br />
This will take a while to complete - and it is best to do this at night when the load is light on both machines.<br />
<br />
There are different methods for this, depending on which version of virtuozzo is installed on the src. and dst. virt. <br />
To check which version is running: <br />
[root@virt12 private]# cat /etc/virtuozzo-release<br />
Virtuozzo release 2.6.0<br />
<br />
Ok, let's say that the VE is 1212, and vital stats are:<br />
<pre>[root@virt12 sbin]# vc 1212<br />
VE_ROOT="/vz1/root/1212"<br />
VE_PRIVATE="/vz1/private/1212"<br />
OSTEMPLATE="fedora-core-2/20040903"<br />
IP_ADDRESS="69.55.229.84"<br />
TEMPLATES="devel-fc2/20040903 php-fc2/20040813 mysql-fc2/20040812 postgresql-fc2/20040813 mod_perl-fc2/20040812 mod_ssl-fc2/20040811 jre-fc2/20040823 jdk-fc2/20040823 mailman-fc2/20040823 analog-fc2/20040824 proftpd-fc2/20040818 tomcat-fc2/20040823 usermin-fc2/20040909 webmin-fc2/20040909 uw-imap-fc2/20040830 phpBB-fc2/20040831 spamassassin-fc2/20040910 PostNuke-fc2/20040824 sl-webalizer-fc2/20040<br />
818"<br />
<br />
[root@virt12 sbin]# vzctl exec 1212 df -h<br />
Filesystem Size Used Avail Use% Mounted on<br />
/dev/vzfs 4.0G 405M 3.7G 10% /</pre><br />
<br />
From this you can see that he’s using (and will minimally need free on the dst server) ~400MB, and he’s running on a Fedora 2 template, version 20040903. He’s also got a bunch of other templates installed. It’s is '''vital''' that '''all''' these templates exist on the dst system. To confirm that, on the dst system run:<br />
<br />
For < 3.0:<br />
<br />
<pre>[root@virt14 private]# vzpkgls | grep fc2<br />
devel-fc2 20040903<br />
PostNuke-fc2 20040824<br />
analog-fc2 20040824<br />
awstats-fc2 20040824<br />
bbClone-fc2 20040824<br />
jdk-fc2 20040823<br />
jre-fc2 20040823<br />
mailman-fc2 20040823<br />
mod_frontpage-fc2 20040816<br />
mod_perl-fc2 20040812<br />
mod_ssl-fc2 20040811<br />
mysql-fc2 20040812<br />
openwebmail-fc2 20040817<br />
php-fc2 20040813<br />
phpBB-fc2 20040831<br />
postgresql-fc2 20040813<br />
proftpd-fc2 20040818<br />
sl-webalizer-fc2 20040818<br />
spamassassin-fc2 20040910<br />
tomcat-fc2 20040823<br />
usermin-fc2 20040909<br />
uw-imap-fc2 20040830<br />
webmin-fc2 20040909<br />
[root@virt14 private]# vzpkgls | grep fedora<br />
fedora-core-1 20040121 20040818<br />
fedora-core-devel-1 20040121 20040818<br />
fedora-core-2 20040903<br />
[root@virt14 private]#</pre><br />
<br />
For these older systems, you can simply match up the date on the template. <br />
<br />
For >= 3.0:<br />
<br />
<pre>[root@virt19 /vz2/private]# vzpkg list<br />
centos-5-x86 2008-01-07 22:05:57<br />
centos-5-x86 devel<br />
centos-5-x86 jre<br />
centos-5-x86 jsdk<br />
centos-5-x86 mod_perl<br />
centos-5-x86 mod_ssl<br />
centos-5-x86 mysql<br />
centos-5-x86 php<br />
centos-5-x86 plesk9<br />
centos-5-x86 plesk9-antivirus<br />
centos-5-x86 plesk9-api<br />
centos-5-x86 plesk9-atmail<br />
centos-5-x86 plesk9-backup<br />
centos-5-x86 plesk9-horde<br />
centos-5-x86 plesk9-mailman<br />
centos-5-x86 plesk9-mod-bw<br />
centos-5-x86 plesk9-postfix<br />
centos-5-x86 plesk9-ppwse<br />
centos-5-x86 plesk9-psa-firewall<br />
centos-5-x86 plesk9-psa-vpn<br />
centos-5-x86 plesk9-psa-fileserver<br />
centos-5-x86 plesk9-qmail<br />
centos-5-x86 plesk9-sb-publish<br />
centos-5-x86 plesk9-vault<br />
centos-5-x86 plesk9-vault-most-popular<br />
centos-5-x86 plesk9-watchdog</pre><br />
<br />
On these newer systems, it's difficult to tell whether the template on the dst matches exactly the src. Just cause a centos-5-x86 is listed on both servers doesn't mean all the same packages are there on the dst. To truly know, you must perform a sample rsync:<br />
<br />
rsync -avn /vz/template/centos/5/x86/ root@10.1.4.61:/vz/template/centos/5/x86/<br />
<br />
if you see a ton of output from the dry run command, then clearly there are some differences. You may opt to let the rsync complete (without running in dry run mode) the only downside is you've now used up more space on the dst and also the centos template will be a mess with old and new data- it will be difficult if not impossible to undo (if someday we wanted to reclaim the space).<br />
<br />
If you choose to merge templates, you should closely inspect the dry run output. You should also take care to exclude anything in the /config directory. For example:<br />
<br />
rsync -av -e ssh --stats --exclude=x86/config /vz/template/ubuntu/10.04/ root@10.1.4.62:/vz/template/ubuntu/10.04/<br />
<br />
Which will avoid this directory and contents:<br />
<pre>[root@virt11 /vz2/private]# ls /vz/template/ubuntu/10.04/x86/config*<br />
app os</pre><br />
<br />
This is important to avoid since the config may differ on the destination and we are really only interested in making sure the pacakges are there, not overwriting a newer config with an older one.<br />
<br />
If the dst system was missing a template, you have 2 choices: <br />
# put the missing template on the dst system. 2 choices here: <br />
## Install the template from rpm (found under backup2: /mnt/data4/vzrpms/distro/) or <br />
## rsync over the template (found under /vz/template) - see above<br />
# put the ve on a system which has all the proper templates<br />
<br />
=== pre-seeding a migration ===<br />
<br />
When migrating a customer (or when doing many) depending on how much data you have to transfer, it can take some time. Further, it can be difficult to gauge when a migration will complete or how long it will take. To help speed up the process and get a better idea about how long it will take you can pre-transfer a customer's data to the destination server. If done correctly, vzmigrate will see the pre-transferred data and pick up where you left off, having much less to transfer (just changed/new files). <br />
<br />
We believe vzmigrate uses rsync to do it's transfer. Therefore not only can you use rsync to do a pre-seed, you can also run rsync to see what is causing a repeatedly-failing vzmigrate to fail. <br />
<br />
There's no magic to a pre-seed, you just need to make sure it's named correctly.<br />
<br />
Given:<br />
<br />
source: /vz1/private/1234<br />
<br />
and you want to migrate to /vz2 on the target system, your rsync would look like:<br />
<br />
rsync -av /vz1/private/1234/ root@x.x.x.x:/vz2/private/1234.migrated/<br />
<br />
After running that successful rsync, the ensuing migrateonline (or migrate) will take much less time to complete- depending on the # of files to be analyzed and the # of changed files. In any case, it'll be much much faster than had you just started the migration from scratch.<br />
<br />
Further, as we discuss elsewhere in this topic, a failed migration can be moved from <tt>/vz/private/1234</tt> to <tt>/vz/private/1234.migrated</tt> on the destination if you want to restart a failed migration. This should '''only''' be done if the migration failed and the CT is not running on the destination HN.<br />
<br />
=== migrateonline intructions: src >=3.x -> dst>=3.x ===<br />
<br />
A script called [[#migrateonline|migrateonline]] was written to handle this kind of move. It is basically a wrapper for <tt>vzmigrate</tt> – vzmigrate is a util to seamlessly- as no no reboot of the ve necessary- move a ve from one host to another. This wrapper was initially written cause vz virtuozzo version 2.6.0 has a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables, causing problems when the ve was started up on the dst system. [[#migrate|migrate]] mitigates that. Since it makes multiple ssh connections to the dst virt, it’s a good idea to put the pub key for the src virt in the authorized_keys file on the dst virt. In addition, migrateonline emails ve owners when their migration starts and stops. For this to happen they need to put email addresses (on a single line, space delimited) in a file on their system: /migrate_notify. If the optional target dir is not specified, the ve will be moved to the same private/root location as it was on the src virt. Note: <tt>migrate</tt> is equivalent to <tt>migrateonline</tt>, but will <tt>migrate</tt> a ve AND restart it in the process.<br />
<br />
<pre>[root@virt12 sbin]# migrateonline<br />
usage: /usr/local/sbin/migrateonline <ip of node migrating to> <veid> [target dir: vz | vz1 | vz2]<br />
[root@virt12 sbin]# migrateonline 10.1.4.64 1212 vz<br />
starting to migrate 1212 at Sat Mar 26 22:40:38 PST 2005<br />
Turning off offline_management<br />
Saved parameters for VE 1212<br />
migrating with no start on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
clearing the arp cache<br />
now going to 10.1.4.64 and clear cache and starting<br />
starting it<br />
Delete port redirection<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
finished migrating 1212 at Sat Mar 26 22 22:52:01 PST 2005<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine.<br />
<br />
If they had backups, use the mvbackups command to move their backups to the new server:<br />
<br />
mvbackups 1212 virt14 vz<br />
<br />
Rename the ve<br />
<br />
[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/migrated-1212<br />
<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/old-1212-migrated-20120404-noarchive<br />
<br />
Update the customer’s systems in mgmt to reflect the new path and server.<br />
<br />
<br />
IF migrateonline does not work, you can try again using simply migrate- this will result in a brief reboot for the ve.<br />
Before you try again, make sure of a few things:<br />
<br />
Depending on where in the migration died, there may be partial data on the dst system in 1 of 2 places:<br />
(given the example above)<br />
<br />
/vz/private/1212<br />
<br />
or <br />
<br />
/vz/private/1212.migrated<br />
<br />
before you run migrate again, you'll want to rename so that all data is in <br />
1212.migrated:<br />
<br />
mv /vz/private/1212 /vz/private/1212.migrated<br />
<br />
this way, it will pick up where it left off and transfer only new files.<br />
<br />
Likewise, if you want to speed up a migration, you can pre-seed the dst as follows:<br />
<br />
[root@virt12 sbin]# rsync -avSH /vz/private/1212/ root@10.1.4.64:/vz/private/1212.migrated/<br />
<br />
then when you run migrate or migrateonline, it will only need to move the changed files- the migration will complete quickly<br />
<br />
=== migrate manually (if migrate/online fails) ===<br />
<br />
Lets say for whatever reason the migration fails. You can always move someone manually:<br />
<br />
1. stop ve: <br><br />
v stop 1234<br />
<br />
2. copy over data<br><br />
rsync -avSH /vz/private/1234/ root@1.1.1.1:/vzX/private/1234/<br />
<br />
NOTE: if you've previously seeded the data (run rsync while the VE was up/running), and this is a subsequent rsync, make sure the last rsync you do (while the VE is not running, has the --delete option in the rsync)<br />
<br />
3. copy over conf<br><br />
scp /vzconf/1234.conf root@1.1.1.1:/vzconf<br />
<br />
4. on dst, edit the conf to reflect the right vzX dir<br><br />
vi /vzconf/1234.conf<br />
<br />
5. on src remove the IPs<br><br />
ipdel 1234 2.2.2.2 3.3.3.3<br />
<br />
6. on dst add IPs <br><br />
ipadd 1234 2.2.2.2 3.3.3.3<br />
<br />
7. on dst, start ve: <br><br />
v start 1324<br />
<br />
8. cancel, then archive ve on src per above instrs.<br />
<br />
=== migrate src=2.6.0 -> dst>=2.6.0, or mass-migration with customer notify ===<br />
<br />
A script called <tt>migrate</tt> was written to handle this kind of move. It is basically a wrapper for vzmigrate – vzmigrate is a util to seamlessly move a ve from one host to another. This wrapper was initially written cause vz virtuozzo version 2.6.0 has a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables, causing problems when the ve was started up on the dst system. migrate mitigates that. Since it makes multiple ssh connections to the dst virt, it’s a good idea to put the pub key for the src virt in the authorized_keys file on the dst virt. In addition, migrate emails ve owners when their migration starts and stops. For this to happen they need to put email addresses (on a single line, space delimited) in a file on their system: /migrate_notify. If the optional target dir is not specified, the ve will be moved to the same private/root location as it was on the src virt. Note: migrateonline is equivalent to migrate, but will migrate a ve from one 2.6 '''kernel''' machine to another 2.6 kernel machine without restarting the ve.<br />
<br />
<pre>[root@virt12 sbin]# migrate<br />
usage: /usr/local/sbin/migrate <ip of node migrating to> <veid> [target dir: vz | vz1 | vz2]<br />
[root@virt12 sbin]# migrate 10.1.4.64 1212 vz<br />
starting to migrate 1212 at Sat Mar 26 22:40:38 PST 2005<br />
Turning off offline_management<br />
Saved parameters for VE 1212<br />
migrating with no start on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
clearing the arp cache<br />
now going to 10.1.4.64 and clear cache and starting<br />
starting it<br />
Delete port redirection<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
finished migrating 1212 at Sat Mar 26 22 22:52:01 PST 2005<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine (backup2 or mail).<br />
Cancel the ve (first we have to rename things which migrate changed so cancelve will find them):<br />
<br />
[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/1212.conf<br />
<br />
On 2.6.1 you’ll also have to move the private area:<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/1212<br />
<br />
<pre>[root@virt12 sbin]# cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, [[#cancelve|cancelve]] would offer to remove them. You want to say '''no''' to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
To move everyone off a system, you’d do:<br />
for f in `vl`; do migrate <ip> $f; done<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
=== vzmigrate: src=2.6.1 -> dst>=2.6.0 ===<br />
<br />
This version of vzmigrate works properly with regard to handling ips. It will not notify ve owners of moves as in the above example. Other than that it’s essentially the same.<br />
<br />
<pre>[root@virt12 sbin]# vzmigrate 10.1.4.64 -r no 1212:1212:/vz/private/1212:/vz/root/1212<br />
migrating on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine (backup2 or mail).<br />
Cancel the ve (first we have to rename things which vzmigrate changed so cancelve will find them):<br />
<br />
<pre>[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/1212.conf<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/1212<br />
<br />
[root@virt12 sbin]# cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, <tt>cancelve</tt> would offer to remove them. You want to say no to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
=== src=2.5.x ===<br />
<br />
First, go to the private dir:<br />
<br />
cd /vz1/private/<br />
<br />
Stop the VE - make sure it stops totally cleanly.<br />
<br />
vzctl stop 1212<br />
<br />
Then you’d use vemove - a script written to copy over the config, create tarballs of the ve’s data on the destination virt, and cancel the ve on the source system (in this example we’re going to put a ve that was in /vz1/private on the src virt, in /vz/private on the dst virt):<br />
<br />
<pre>[root@virt12 sbin]# vemove<br />
ERROR: Usage: vemove veid target_ip target_path_dir<br />
[root@virt12 sbin]# vemove 1212 10.1.4.64 /vz/private/1212<br />
tar cfpP - 1212 --ignore-failed-read | (ssh -2 -c arcfour 10.1.4.64 "split - -b 1024m /vz/private/1212.tar" )<br />
scp /vzconf/1212.conf 10.1.4.64:/vzconf<br />
cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, cancelve would offer to remove them. You want to say no to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
When you are done, go to /vz/private on the dst virt you will have files like this:<br />
<br />
<pre>1212.taraa<br />
1212.tarab<br />
1212.tarac</pre><br />
<br />
Each one 1024m (or less, for the last one) in size.<br />
<br />
on the dst server and run:<br />
<br />
cat 1212.tar?? | tar xpPBf -<br />
<br />
and after 20 mins or so it will be totally untarred. Now since the conf<br />
file is already there, you can go ahead and start the system.<br />
<br />
vzctl start 1212<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
NOTE: you MUST tar the system up using the virtuozzo version of tar that<br />
is on all the virt systems, and further you MUST untar the tarball with<br />
the virtuozzo tar, using these options: `<tt>tar xpPBf -</tt>`<br />
<br />
If you tar up an entire VE and move it to a non-virtuozzo machine, that is<br />
ok, and you can untar it there with normal tar commands, but do not untar<br />
it and then repack it with a normal tar and expect it to work - you need<br />
to use virtuozzo tar commands on virtuozzo tarballs to make it work.<br />
<br />
The backups are sort of an exception, since we are just (usually)<br />
restoring user data that was created after we gave them the system, and<br />
therefore has nothing to do with magic symlinks or vz-rpms, etc.<br />
<br />
== Moving a VE on the same virt ==<br />
<br />
Easy way:<br><br />
Scenario 1: ve 123 is to be renamed 1231 and moved from vz1 to vz<br />
<br />
vzmlocal 123:1231:/vz/private/1231:/vz/root/1231<br />
<br />
Scenario 2: ve 123 is to be moved vz1 to vz<br />
<br />
vzmlocal 123:123:/vz/private/123:/vz/root/123<br />
<br />
vzmlocal will reboot the ve at the end of the move<br />
<br />
Manual/old way:<br />
<br />
1) <tt>vzctl stop 123</tt><br><br />
2) <tt>mv /vz1/private/123 /vz/private/.</tt><br><br />
(or cp -a if you want to copy)<br />
3) in <tt>/etc/sysconfig/vz-scripts/123.conf</tt> change value<br><br />
of '<tt>VE_PRIVATE</tt>' variable to point to a new private area location<br />
4) <tt>vzctl start 123</tt><br><br />
5) update backups if needed: <tt>mvbackups 123 virtX virt1 vz</tt><br><br />
6) update management scerens<br><br />
<br />
Notes: a) absolute path to private area is stored in quota file <tt>/var/vzquota/quota.123</tt> - so during first startup quota will be recalculated.<br><br />
b) if you're going to write some script to do a job, you MUST be sure that $VEID won't be expanded to '' in ve config file - ie. you need to escape '$'. Otherwise you might have:<br />
<br />
VE_PRIVATE="/vz/private/"<br />
<br />
in config, and 'vzctl destroy' for this VE ID '''will remove everything under /vz/private/ directory'''.<br />
<br />
== Remaking a system (on same virt) ==<br />
<br />
1. [[#cancelve|cancelve]] (or v destroy x - ONLY if you're POSITIVE no data needs to be saved)<br />
2. [[#vemake|vemake]] using same veid<br />
3. [[#mvbackups|mvbackups]] or [[#vb|vb]] (if new mount point)<br />
4. update mgmt with new dir/ip <br />
5. update firewall if changed ip<br />
<br />
<br />
== Traffic accounting on linux ==<br />
<br />
DEPRECATED - all tracking is done via bwdb now. This is how we used to track traffic.<br />
<br />
TODO: update for diff versions of vz<br />
<br />
Unlike FreeBSD, where we have to add firewall count rules to the system to count the traffic, on virtuozzo counts the traffic for us. You an see the current traffic stats by running `vznetstat`:<br />
<br />
<pre># vznetstat<br />
VEID Net.Class Output(bytes) Input(bytes)<br />
-----------------------------------------------<br />
24218 1 484M 39M<br />
24245 1 463M 143M<br />
2451 1 2224M 265M<br />
2454 1 2616M 385M<br />
4149 1 125M 68M<br />
418 1 1560M 34M<br />
472 1 1219M 315M<br />
726 1 628M 317M<br />
763 1 223M 82M<br />
771 1 4234M 437M<br />
-----------------------------------------------<br />
Total: 13780M 2090M</pre><br />
<br />
As you can see the VEID is on a line with the in and out bytes. So, we simply run a cron job:<br />
<br />
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /root/vztrafdump.sh<br />
<br />
Just like we do on FreeBSD - this one goes through all the VEs in /vz/private and greps the line from vznetstat that matches them and dumps it in /jc_traffic_dump on their system. Then it does it again for all the VEs in /vz1/private. It is important to note that vznetstat runs only once, and the grepping is done from a temporary file that contains that output - we do this because running vznetstat once for each VE that we read out of /vz/private and /vz1/private would take way too long and be too intensive.<br />
<br />
You do not need to do anything to facilitate this other than make sure that that cron job is running - the vznetstat counters are always running, and any new VEs that are added to the system will be accounted for automatically.<br />
<br />
Traffic resetting no longer works with vz 2.6, so we disable the vztrafdump.sh on those virts.<br />
<br />
== Watchdog script ==<br />
<br />
On some of the older virts, we have a watchdog running that kills procs that are deemed bad per the following:<br />
<br />
/root/watchdog from quar1<br />
<br />
<pre>if echo $line | awk '{print $(NF-3)}' | grep [5-9]...<br />
then<br />
# 50-90%<br />
if echo $line | awk '{print $(NF-1)}' | grep "...:.."<br />
then<br />
# running for > 99min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
if echo $line | awk '{print $(NF-1)}' | grep "....m"<br />
then<br />
# running for > 1000min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
fi<br />
<br />
if echo $line | awk '{print $(NF-3)}' | grep [1-9]...<br />
then<br />
# running for 10-90 percent<br />
if echo $line | awk '{print $NF}' | egrep 'cfusion|counter|vchkpw'<br />
then<br />
<br />
if echo $line | awk '{print $(NF-1)}' | grep "[2-9]:.."<br />
then<br />
# between 2-9min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
elif echo $line | awk '{print $(NF-1)}' | grep "[0-9][0-9]:.."<br />
then<br />
# up to 99min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
fi<br />
fi</pre><br />
<br />
<br />
== Misc Linux Items ==<br />
<br />
We are overselling hard drive space ... when you configure a linux system with a certain amount of disk space (the default is 4gigs) you do not actually use up 4gigs of space on the system. The diskspace setting for a user is simply a cap, and they only use up as much space on the actual disk drive as they are actually using.<br />
<br />
When you create a new linux system, even though there are some 300 RPMs or so installed, if you run `df -k` you will see that the entire 4gig partition is empty - no space is being used. This is because the files in their system are "magic symlinks" to the template for their OS that is in /vz/template - however, any changes to any of those files will "disconnect" them and they will immediately begin using space in their system. Further, any new files uploaded (even if those new files overwrite existing files) will take up space on the partition.<br />
<br />
<br />
if you see this:<br />
<br />
<pre>[root@virt8 root]# vzctl stop 160 ; vzctl start 160<br />
VE is not running<br />
Starting VE ...<br />
VE is unmounted<br />
VE is mounted<br />
Adding IP address(es): 69.55.226.83 69.55.226.84<br />
bash ERROR: Can't change file /etc/sysconfig/network<br />
Deleting IP address(es): 69.55.226.83 69.55.226.84<br />
VE is unmounted<br />
[root@virt8 root]#</pre><br />
<br />
it probably means they no longer have /bin/bash - copy one in for them<br />
<br />
ALSO: another possibility is that they have removed the `ed` RPM from their system - it needs to be reinstalled into their system. But since their system is down, this is tricky ...<br />
<br />
VE startup scripts used by 'vzctl' want package 'ed' to be available inside VE. So if package 'ed' will be enabled in OS template config and OS template itself VE #827 is based on - this error should be fixed.<br />
<br />
yes, it is possible to add RPM to VE while it not running.<br />
Try to do following:<br />
<br />
<pre># cd /vz/template/<OS_template_with_ed_package>/<br />
# vzctl mount 827<br />
# rpm -Uvh --root /vz/root/827 --veid 827 ed-0.2-25.i386.vz.rpm</pre><br />
<br />
Usually theres an error, but its ok<br />
<br />
Note: replace 'ed-0.2-25.i386.vz.rpm' in last command with actual<br />
version of 'ed' package you have.<br />
<br />
----<br />
<br />
So how do I know what template the user has ? cat their conf file and it is listed in there. For example, if the conf file has:<br />
<br />
<pre>[root@virt12 sbin]# vc 1103<br />
…snip…<br />
OSTEMPLATE="debian-3.0/20030822"<br />
TEMPLATES="mod_perl-deb30/20030707 mod_ssl-deb30/20030703 mysql-deb30/20030707 proftpd-deb30/20030703 webmin-deb30/20030823 "<br />
…</pre><br />
<br />
then they are on debian 3.0, all of their system RPMs are in /vz/template/debian-3.0, and they are using version 20030822 of that debian 3.0 template. Also, they’ve also got additional packages installed (mod_perl, mod_ssl, etc). Those are also found under /vz/template<br />
<br />
----<br />
<br />
Edits needed to run java:<br />
<br />
When we first created the VEs, the default setting for privvmpages was 93000:94000 ... which was high enough that most people never had problems ... however, you can;t run java or jdk or tomcat or anything java related with that setting. We have found that by setting privvmpages to 610000:615000 that java runs just fine. That is now the default setting. It is exceedingly rare that anyone needs it higher than that, although we have seen it once or twice.<br />
<br />
Any problems with java at all - the first thing you need to do is see if the failcnt has raised for privvmpages.<br />
<br />
<pre># vzctl start 160<br />
Starting VE ...<br />
vzquota : (error) Quota on syscall for 160: Device or resource busy<br />
Running vzquota on failed for VE 160 [3]</pre><br />
<br />
This is because my pwd is _in_ their private directory - you can't start it until you move out<br />
<br />
People seem to have trouble with php if they are clueless newbies. Here are two common problems/solutions:<br />
<br />
no... but i figured it out myself. problem was the php.ini file that came<br />
vanilla with the account was not configured to work with apache (the<br />
ENGINE directive was set to off).<br />
<br />
everything else seems fine now.<br />
<br />
----<br />
<br />
the problem was in the php.ini file. I noticed that is wasnt showing<br />
the code when it was in an html file so I looked at the php.ini file<br />
and had to change it so it recognized <? tags aswell as <?php tags.<br />
<br />
Also, make sure added to httpd.conf<br />
AddType application/x-httpd-php .php<br />
<br />
----<br />
<br />
You can set the time zone:<br />
<br />
You can change the timezone by doing this:<br />
<br />
ln -sf /usr/share/zoneinfo/<zone> /etc/localtime<br />
<br />
where <zone> is the zone you want in the /usr/share/zoneinfo/ directory.<br />
<br />
----<br />
<br />
Failing shm_open calls:<br />
<br />
first, please check if /dev/shm is mounted inside VE.<br />
'cat /proc/mounts' command should show something like this:<br />
tmpfs /dev/shm tmpfs rw 0 0<br />
<br />
If /dev/shm is not mounted you have 2 ways to solve issue:<br />
1. execute following command inside VE (doesn't require VE reboot):<br />
mount -t tmpfs none /dev/shm<br />
2. add following string to /etc/fstab inside VE and reboot it:<br />
tmpfs /dev/shm tmpfs defaults 0 0<br />
<br />
----<br />
<br />
You can have a mounted but not running ve<br />
Just:<br />
vzctl mount <veid><br />
<br />
----<br />
<br />
When a debian sys can’t get on the network, and you try:<br />
<br />
<pre>vzctl set 1046 --ipadd 69.55.227.117<br />
Adding IP address(es): 69.55.227.117<br />
Failed to bring up lo.<br />
Failed to bring up venet0.</pre><br />
<br />
They probably removed iproute package, which must be the one from swsoft. To restore:<br />
<pre># dpkg -i --veid=1046 --admindir=/vz1/private/1046/root/var/lib/dpkg --instdir=/vz1/private/1046/root/ /vz/template/debian-3.0/iproute_20010824-8_i386.vz.deb<br />
(Reading database ... 16007 files and directories currently installed.)<br />
Preparing to replace iproute 20010824-8 (using .../iproute_20010824-8_i386.vz.deb) ...<br />
Unpacking replacement iproute ...<br />
Setting up iproute (20010824-8) ...</pre><br />
<br />
Then restart their ve<br />
<br />
----<br />
<br />
in a ve i do:<br />
<br />
<pre>cd /<br />
du -h .</pre><br />
<br />
and get: 483M .<br />
<br />
i do:<br />
<br />
<pre>bash-2.05a# df -h<br />
Filesystem Size Used Avail Use% Mounted on<br />
/dev/vzfs 4.0G 2.3G 1.7G 56% /</pre><br />
<br />
how can this be?<br />
<br />
Is it possible that quota file was corrupted somehow? Please try to: <br />
<pre>vzctl stop <VEID><br />
vzquota drop <VEID><br />
vzquota init <VEID><br />
vzctl start <VEID></pre><br />
<br />
----<br />
<br />
How to stop vz from starting after reboot:<br />
<br />
VIRTUOZZO=no <br />
in <br />
/etc/sysconfig/vz<br />
<br />
To start: <br />
service vz start<br />
(after setting VIRTUOZZO=yes in /etc/sysconfig/vz)<br />
<br />
service vz restart will do some kind of 'soft reboot' -- restart all<br />
VPSes and reload modules without rebooting the node<br />
<br />
if you need to shut down all VPSes really really fast, run killall -9 init<br />
<br />
----<br />
<br />
Postfix tip:<br />
<br />
You may want to tweak settings: default_process_limit=10<br />
<br />
= Virtuozzo VPS Management Tools =<br />
<br />
== vm ==<br />
<br />
TODO<br />
<br />
== cancelve ==<br />
cancelve <veid><br />
this will:<br />
* stop a ve<br />
* check for backups (offer to remove them from the backup server <br />
and the backup.config)<br />
* rename the private dir<br />
* check for PTR, provide the commands to reset to default<br />
* and rename the ve’s config<br />
* remind you to remove firewall rules<br />
* remind you to remove DNS entries<br />
<br />
== ipadd ==<br />
ipadd <veid> <ip> [ip] [ip]<br />
add’s ip(s) to a ve<br />
<br />
== ipdel ==<br />
ipdel <veid> <ip> [ip] [ip]<br />
removes ip(s) from a ve<br />
<br />
== vc ==<br />
vc <veid><br />
the equivalent of: <tt>cat /vzconf/<veid>.conf</tt><br />
<br />
== vl ==<br />
vl<br />
will displays a list of ve #’s, 1 per line. (ostensibly to use in a for loop)<br />
<br />
== vp ==<br />
vp <veid><br />
the equivalent of: <tt>vzps auxww –E <veid></tt><br />
<br />
== vpe ==<br />
DEPRECATED<br />
vpe <veid> <br />
this will allow you to do a vp when a ve is running out of control, the equivalent of (deprecated since vp operates outside the VPS): <br />
<pre>vzctl set <veid> --kmemsize 2100000:2200000<br />
vzctl exec <veid> ps auxw<br />
vzctl set <veid> --kmemsize (ve’s orig lvalue):(ve’s orig hvalue)</pre><br />
<br />
== vt ==<br />
vt <veid><br />
the equivalent of: <tt>vztop –E <veid></tt><br />
<br />
== vr ==<br />
vr <veid><br />
the equivalent of: <tt>vzctl stop <veid>; vzctl start <veid></tt><br />
You can run this even if the ve is down - the stop command will just fail<br />
<br />
== vs ==<br />
vs [veid]<br />
displays status (output of <tt>vzctl status <veid></tt>) on each ve configured on the system (as determined by <tt>/vzconf/*.conf</tt>)<br />
If passed an argument, gives the status for just that ve. <br />
A running system looks like:<br />
<tt>VEID 16066 exist mounted running</tt><br />
<br />
A ve which is not running (but does exist) looks like:<br />
<tt>VEID 9990 exist unmounted down</tt><br />
<br />
A ve which is not running and doesn’t exist looks like:<br />
<tt>VEID 421 deleted unmounted down</tt><br />
<br />
== vs2 ==<br />
vs2 [veid]<br />
this is similar to vs in that it displays status (output of <tt>vzctl status <veid></tt>) on each ve,<br />
but the difference is it’s list comes from doing an ls on the data dirs. This was meant to catch <br />
the rare case where a ve configured but exists. <br />
<br />
== vw ==<br />
vw [veid]<br />
displays the output of ‘<tt>w</tt>’ (the equivalent of <tt>vzctl exec <veid> w</tt>) for each configured ve (as determined by <tt>/vzconf/*.conf</tt>). Useful for determing which ve is contributing to a heavily-loaded system.<br />
If passed an argument, gives ‘<tt>w</tt>‘ output for just that ve. <br />
Ex:<br />
<pre>[root@virt2 etc]# vw<br />
134<br />
10:52pm up 79 days, 6:14, 0 users, load average: 0.02, 0.02, 0.00<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT<br />
16027<br />
2:52pm up 7 days, 19:54, 0 users, load average: 0.00, 0.00, 0.00<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT<br />
16055<br />
2:52pm up 79 days, 6:38, 0 users, load average: 0.00, 0.04, 0.07<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT</pre><br />
<br />
== vwe ==<br />
vwe [constraint]<br />
just like <tt>vw</tt>, but takes a constraint as an argument, only show’s ve’s with loads >= the constraint provided. If no constraint is provided, 1 is used by default<br />
<br />
== vzs ==<br />
vzs [veid]<br />
displays the beancounter status for all ve’s, or a particular ve if an argument is passed<br />
<br />
== ve ==<br />
ve <veid><br />
the equivalent of: <tt>vzctl enter <veid></tt><br />
<br />
== vx ==<br />
vx <veid> <command><br />
the equivalent of: <tt>/usr/sbin/vzctl exec <veid> <command></tt><br />
<br />
== dprocs ==<br />
dprocs [count]<br />
a script which outputs a continuous report (or a certain number of reports if an option is passed) of processes stuck in the D state and which VPS’s those procs belong to.<br />
<br />
== setmem ==<br />
setmem <VEID> <256|512|768|1024|2048><br />
adjusts the memory resources for the VE<br />
<br />
== afacheck.sh ==<br />
afacheck.sh<br />
displays the health/status of containers and mirrors on an adaptec card (currently quar1, tempvirt1-2, virt9, virt10)- all other are LSI<br />
<br />
== backup ==<br />
backup<br />
backup script called nightly to update virt scripts and do customer backups<br />
<br />
== checkload.pl ==<br />
checkload.pl<br />
this was intended to be setup as a cronjob to watch processes on a virt when the load <br />
rises above a certain level. Not currently in use.<br />
<br />
== findbackuppigs.pl ==<br />
findbackuppigs.pl<br />
looks for files larger than 50MB which customers have asked us to backup. Emails matches<br />
to linux@johncompanies.com<br />
<br />
== gatherlinux.pl ==<br />
gatherlinux.pl<br />
gathers up data about ve’s configured and writes to a file. Used for audits against the db<br />
<br />
== gb ==<br />
gb <search><br />
greps backup.config for the given search parameter<br />
<br />
== gbg ==<br />
gbg <search><br />
greps backup.config for the given search parameter and presents just the directories (for clean pasting)<br />
<br />
== linuxtrafficgather.pl ==<br />
linuxtrafficgather.pl [yy-mm]<br />
sends a traffic usage summary by ve to support@johncomapnies.com and payments@johncompanies.com.<br />
Optional arguments are year and month (must be in the past). If not passed, assumes last month. Relies on <br />
traffic logs created by netstatreset and netstatbackup<br />
<br />
== linuxtrafficwatch.pl ==<br />
linuxtrafficwatch.pl<br />
checks traffic usage and emails support@johncomapnies.com when a ve reaches the warning level (35G) and<br />
the limit (40G). Works on virtuozzo versions <= 2.6.x. We really aren’t using this anymore now that we have netflow.<br />
<br />
== linuxtrafficwatch2.pl ==<br />
linuxtrafficwatch2.pl<br />
checks traffic usage and emails support@johncomapnies.com when a ve reaches the warning level (35G) and<br />
the limit (40G). Works on virtuozzo version 2.6.x. We really aren’t using this anymore now that we have netflow.<br />
<br />
== load.pl ==<br />
load.pl<br />
feeds info to load mrtg - executed by inetd.<br />
<br />
== mb (linux) ==<br />
mb <mount|umount><br />
(nfs) mounts and umounts dirs to backup2. Shortcuts are mbm and mbu to mount and unmount. <br />
<br />
== migrate ==<br />
migrate <ip of node migrating to> <veid> <target_veid> [target dir: vz | vz1 | vz2]<br />
this is basically a wrapper for <tt>vzmigrate</tt> – vzmigrate is a util to seamlessly move a ve from one host to another. This wrapper was written cause vz virtuozzo version 2.6 had a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables. This script mitigates that. Since it makes multiple ssh connections to the target host, it’s a good idea to put the pub key for the src system in the authorized_keys file on the target host. In addition, it emails ve owners when their migration starts and stops (if they place email addresses in a file on their system: /migrate_notify). To move everyone off a system, you’d do:<br />
for f in `vl`; do migrate <ip> $f; done<br />
<br />
== migrateonline ==<br />
migrateonline <ip of node migrating to> <veid> <target_veid> [target dir: vz | vz1 | vz2]<br />
this is the same as migrate but will migrate a ve in <tt>–online</tt> mode which means it won’t be shut down at the end of the migration. This only works when migrating ve’s between 2 machines running a 2.6 kernel (currently tempvirt1-2. virt16-19, virt12). If you get an error that the machine you’re trying to migrate to has a different CPU or features, etc, then you have to edit the file and add the –f switch to the vzmigrate line- you can basically ignore this kind of warning (but never ignore a warning about missing templates on the destination node). NOTE: This edit (if made to migrateonline) will be overwritten by the base script during each night’s backup.<br />
<br />
== netstatbackup ==<br />
DEPRECATED<br />
netstatbackup <br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.5.x. <br />
<br />
== netstatbackup2 ==<br />
DEPRECATED<br />
netstatbackup2<br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.6.x. <br />
<br />
== netstatreset ==<br />
DEPRECATED<br />
netstatreset<br />
writes traffic count data to a logfile and resets counters to 0. Works on virtuozzo versions 2.5.x <br />
<br />
== netstatreset2 ==<br />
DEPRECATED<br />
netstatreset2<br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.6.x. <br />
<br />
== orphanedbackupwatchlinux ==<br />
orphanedbackupwatchlinux <br />
looks for directories on backup2 which aren’t configured in backup.config and offers to <br />
delete them<br />
<br />
== rsync.backup (linux) ==<br />
rsync.backup<br />
does customer backups (relies on backup.config)<br />
<br />
== startvirt.pl ==<br />
startvirt.pl<br />
forks off start ve commands – keeps 6 running at a time. This is not to be used on systems where fastboot is enabled as it circumvents the benefit of the fastboot. The script will occasionally not exit gracefully and will continue to use up CPU, so it should be watched. Also, don’t exit from the script till you’re sure all ve’s are started – if you do you need to start them manually and may have to free up locks. On some systems, the startvirt script doesn’t exit cleanly and you have to ^C out of it. Be careful though- doing so can leave some VE’s in an odd bootup state and you may need to ‘vr’ them manually. You should check to see which ve’s aren’t running and/or confirm all have started when ^C’ing out of startvirt.<br />
<br />
== taskdone (linux) ==<br />
taskdone<br />
when called will email support@johncompanies.com with the hostname of the machine from which it was <br />
executed as the subject<br />
<br />
== vb (linux) ==<br />
vb<br />
the equivalent of: <tt>vi /usr/local/sbin/backup.config</tt><br />
<br />
== vemakeXX ==<br />
DEPRECATED<br />
vemakerh9 <br />
ve create script for RH9 (see vemake)<br />
<br />
vemakedebian30 <br />
ve create script for debian 3.0 (Woody) (see vemake)<br />
<br />
vemakedebian31 <br />
ve create script for debian 3.1 (Sarge) (see vemake)<br />
<br />
vemakedebian40 <br />
ve create script for debian 4.0 (Etch) (see vemake)<br />
<br />
vemakefedora, vemakefedora2, vemakefedora4, vemakefedora5, vemakefedora6, vemakefedora7<br />
ve create script for fedora core 1, 2, 4, 5, 6, 7 respectively (see vemake)<br />
<br />
vemakecentos3, vemakecentos4<br />
ve create script for centos 3, 4 respectively (see vemake)<br />
<br />
vemakesuse, vemakesuse93, vemakesuse100<br />
ve create script for suse 9.2, 9.3, 10.0 respectively (see vemake)<br />
<br />
vemakeubuntu5, vemakeubuntu606, vemakeubuntu606 vemakeubuntu704<br />
ve create script for ubuntu 5.10, 6.06, 6.10, 7.04 respectively (see vemake)<br />
<br />
== vemove ==<br />
DEPRECATED<br />
vemove <veid> <target_ip> </vz/private/123><br />
this script simplifies the old way of moving ve’s from one system to another - in short moving a ve to or from a virt running virtuozzo < 2.6.x<br />
It’s the equivalent of:<br />
<tt>tar cfpP - <veid> --ignore-failed-read | (ssh -2 -c arcfour <target_ip> "split - -b 1024m </vz/private/123>.tar" )</tt>This should only be used if migrate/vzmigrate can’t be used. <br />
<br />
== vim.watchdog ==<br />
vim.watchdog <br />
looks for and kills procs matching “vi|vim|nano|pine|elm” that are running for a long time & consuming lots of cpu. Works on virtuozzo versions 2.5.x<br />
<br />
== vim.watchdog2 ==<br />
vim.watchdog2<br />
looks for and kills procs matching “vi|vim|nano|pine|elm” that are running for a long time & consuming lots of cpu.<br />
Works on virtuozzo versions 2.6.x.<br />
<br />
== vzmigrate ==<br />
vzmigrate <target_ip> -r no <veid>:[dst veid]:[dst /vzX/private/veid]:[dst /vzX/root/veid]<br />
(this is the raw command “wrapped” by migrate/migrateonline) this will seamlessly move a ve from one host to another. The ve will run for the duration of the migration till the very end when it’s shut down, ip moved and started up on the target system. The filesystem on the src will remain. This should be watched – occasionally the move will timeout and leave the system shut down. If target private and root aren’t specified it just puts it in /vz. Only works when both systems are running virtuozzo 2.6.x<br />
<br />
== vztrafdump.sh ==<br />
DEPRECATED<br />
vztrafdump.sh<br />
writes traffic usage info by ve to a file called jc_traffic_dump in each ve’s / dir. Works on virtuozzo versions <= 2.5.x. <br />
<br />
== vztrafdump2.sh ==<br />
DEPRECATED<br />
vztrafdump2.sh<br />
writes traffic usage info by ve to a file called jc_traffic_dump in each ve’s / dir. Works on virtuozzo versions 2.6.x. <br />
<br />
== addtun ==<br />
addtun <veid><br />
Add’s tun device to ve.<br />
<br />
== bwcap ==<br />
bwcap <veid> <kbps><br />
Ex: <tt>bwcap 1234 512</tt><br />
Caps a VE’s bandwidth to the amount given<br />
<br />
== setdisk ==<br />
setdisk <veid> <diskspace in GB><br />
Ex: <tt>setdisk 1234 5</tt><br />
Gives a VE’s a given amount of disk space<br />
<br />
== vdf ==<br />
vdf <veid> <br />
the equivalent of: <tt>vzctl exec <veid> df –h</tt><br />
<br />
== vdff ==<br />
vdff<br />
runs a (condensed) vdf for all ve’s in your pwd (must be run from /vz/privateN)<br />
<br />
== mvbackups ==<br />
mvbackups <veid> <target_machine> (virt1) <target_dir> (vz1)<br />
moves backups from one location to another on the backup server, and provides you with option to remove entries from current backup.config, and simple paste command to add the config to backup.config on the target server<br />
<br />
== checkquota ==<br />
checkquota<br />
for all the ve’s in the cwd (run from /vz/private, /vz1/private, etc) reports what vz quota says they’re using and what the actual usage is (as reported by du)<br />
<br />
== clearquota ==<br />
clearquota <veid><br />
Recalculates a ve’s quota, prints out the usage before and after. The equivalent of:<br />
<tt>vdf <veid>; v stop <veid>; vzquota drop <veid>; v start <veid>; vdf <veid></tt><br />
<br />
== dprocs ==<br />
dprocs<br />
Sometimes the server’s have a large number of processes get stuck in the D state- this script shows (every 3 secs) which VE’s have D procs, which procs<br />
are stuck and a running average of the top “offenders”<br />
<br />
== vzstat ==<br />
vstat<br />
sort of like top for VZ. sort VEs by CPU usage by pressing 'o' and then 'c' keys</div>Supporthttps://wiki.jcihosting.com/index.php?title=VPS_Management&diff=1013VPS Management2013-03-11T22:55:57Z<p>Support: /* Instructions for src=2.5.x */</p>
<hr />
<div>= Common Problems =<br />
== Login to any machine without a password ==<br />
<br />
This is possible via the use of ssh keys. The process is thus:<br />
<br />
1. place the public key for your user (root@mail) in the /root/.ssh/authorized_keys file on the server you wish to login to<br />
cat /root/.ssh/id_dsa.pub<br />
(paste that into authorized_keys on the target server). If the file doesn't exist, create it.<br />
<br />
2. enable root login (usually only applies to FreeBSD). Edit the /etc/ssh/sshd_config on the target server and change:<br />
<tt>#PermitRootLogin no</tt><br />
to<br />
<tt>PermitRootLogin yes</tt><br />
<br />
3. Restart the sshd on the target machine. First, find the sshd process: <br />
jailps <hostname> | grep sshd <br />
or <br />
vp <VEID> | grep sshd<br />
<br />
Look for the process resembling:<br />
root 17296 0.0 0.0 5280 1036 ? Ss 2011 4:27 /usr/sbin/sshd <br />
(this is the sshd)<br />
<br />
Not:<br />
root 6270 0.5 0.0 6808 2536 ? Ss 14:33 0:00 sshd: root [priv]<br />
(this is an sshd child- someone already ssh'd in as root)<br />
<br />
Restart the sshd: <br />
kill -1 <PID><br />
<br />
Ex:<br />
kill -1 17296<br />
<br />
You may now ssh in.<br />
<br />
Once you're done, IF you enabled root login, you should repeat steps 2 and 3 to disable root logins.<br />
<br />
== Letting someone in who has locked themselves out (killed sshd, lost pwd) ==<br />
<br />
There are two ways people frequently lock themselves out - either they forget a password, or they kill off sshd somehow.<br />
<br />
These are actually both fairly easy to solve. First, let's say someone kills off their sshd, or somehow mangles /etc/ssh/sshd_config such that it no longer lets them in.<br />
<br />
Their email may be very short, or it may have all sorts of details about how you should fix sshd_config to let them in ... just ignore all of this. They can fix their own mangled sshd. Fixing this is very simple. First, edit the /etc/inetd.conf on their system and uncomment the telnet line:<br />
<br />
telnet stream tcp nowait root /usr/libexec/telnetd telnetd<br />
#telnet stream tcp6 nowait root /usr/libexec/telnetd telnetd<br />
<br />
(just leave the tcp6 version of telnet commented)<br />
<br />
Then, use jailps to list the processes on their system, and find their inetd process. Then simply:<br />
<br />
kill -HUP (pid)<br />
<br />
where (pid) is the PID of their inetd process. Now they have telnet running on their system and they can log in and do whatever they need to do.<br />
<br />
The only complications that could occur are:<br />
<br />
a) their firewall config on our firewall has port 23 blocked, in which case you will need to open that - will be covered in a different lesson.<br />
<br />
b) they are not running inetd, so you can't HUP it. If this happens, edit their /etc/rc.conf, add the inetd_enable="YES" line, and then kill<br />
their jail with /tmp/jailkill.pl - then restart their jail with the jail line from their quad/safe file. Easy.<br />
<br />
If they have forgotten a password,<br />
<br />
On 6.x+ you can reset their password with:<br />
jexec <jailID from jls> passwd root<br />
<br />
Note: the default password for 6.x jails is 8ico2987, for 4.x it is p455agfa<br />
<br />
On 4.x, you need to cd to their etc directory<br />
... for instance:<br />
<br />
cd /mnt/data2/198.78.65.136-col00261-DIR/etc<br />
<br />
and run:<br />
<br />
vipw -d .<br />
<br />
Then paste in these two lines (theres a paste with these):<br />
<br />
root:$1$krszPxhk$xkCepSnz3mIikT3vCtJCt0:0:0::0:0:Charlie &:/root:/bin/csh<br />
user:$1$Mx9p5Npk$QdMU6c8YQqp2FW2M3irEh/:1001:1001::0:0:User &:/home/user:/bin/sh<br />
<br />
overwriting the lines they already have for "user" and "root" - then just tell them that both user and root have been reset to the default password of p455agfa.<br />
<br />
For linux, just passwd inside shell or <br />
vzctl set <veid> --userpasswd root:p455agfa –save<br />
<br />
Starting in 2009 we began giving out randomized passwords for FreeBSD and Linux as the default password. That is stored with each system in Mgmt. You should look for and reset the password to that password in the event of a reset and refer the customer to use their original password from their welcome email- this way we don’t have to send the password again via email (in clear text).<br />
<br />
<br />
== sendmail can’t be contacted from ext ip (only locally) ==<br />
<br />
By default redhat puts this line in sendmail.mc:<br />
<br />
DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')<br />
<br />
which makes it only answer on localhost. Comment it out like:<br />
<br />
dnl DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')<br />
<br />
and then rebuild sendmail.cf with:<br />
<br />
m4 /etc/mail/sendmail.mc > /etc/sendmail.cf<br />
<br />
== virt doesn’t properly let go of ve’s ip(s) when moved to another system ==<br />
<br />
On virtuozzo 2.6 systems, it's been observed that when moving ips from one virt to another that sometimes the routing table will not get updated to reflect the removal of the ip addresses.<br />
<br />
A recent example was a customer that was moving to a new ve on a new virt and the ip addresses were traded between the two ve's. After the trade the two systems were not able to talk to each other. When looking at the routing table for the old system all the ip addresses were still in the routing table as being local, like so:<br />
<br />
<pre>netstat -rn | grep 69.55.225.149<br />
69.55.225.149 0.0.0.0 255.255.255.255 UH 40 0 0 venet0</pre><br />
<br />
This was preventing traffic to the other system from being routed properly.<br />
The solution is to manually delete the route:<br />
<br />
route delete 69.55.225.149 gw 0.0.0.0<br />
<br />
Supposedly, this was fixed in 2.6.1<br />
<br />
== sshd on FreeBSD 6.2 segfaults ==<br />
<br />
First try to reinstall ssh<br />
<br />
<pre>cd /usr/src/secure<br />
cd lib/libssh<br />
make depend && make all install<br />
cd ../../usr.sbin/sshd<br />
make depend && make all install<br />
cd ../../usr.bin/ssh<br />
make depend && make all install</pre><br />
<br />
Failing that, find the library that’s messed up:<br />
<br />
<pre>ldd /usr/sbin/sshd<br />
libssh.so.3 => /usr/lib/libssh.so.3 (0x280a3000) <br />
libutil.so.5 => /lib/libutil.so.5 (0x280d8000) <br />
libz.so.3 => /lib/libz.so.3 (0x280e4000) <br />
libwrap.so.4 => /usr/lib/libwrap.so.4 (0x280f5000) <br />
libpam.so.3 => /usr/lib/libpam.so.3 (0x280fc000) <br />
libbsm.so.1 => /usr/lib/libbsm.so.1 (0x28103000) <br />
libgssapi.so.8 => /usr/lib/libgssapi.so.8 (0x28112000) <br />
libkrb5.so.8 => /usr/lib/libkrb5.so.8 (0x28120000) <br />
libasn1.so.8 => /usr/lib/libasn1.so.8 (0x28154000) <br />
libcom_err.so.3 => /usr/lib/libcom_err.so.3 (0x28175000) <br />
libroken.so.8 => /usr/lib/libroken.so.8 (0x28177000) <br />
libcrypto.so.4 => /lib/libcrypto.so.4 (0x28183000) <br />
libcrypt.so.3 => /lib/libcrypt.so.3 (0x28276000) <br />
libc.so.6 => /lib/libc.so.6 (0x2828e000) <br />
libmd.so.3 => /lib/libmd.so.3 (0x28373000)</pre><br />
<br />
md5 them and compare to other jail hosts or jails running on host<br />
<br />
for libcrypto reinstall:<br />
<pre>/usr/src/crypto<br />
make depend && make all install</pre><br />
<br />
= FreeBSD VPS =<br />
<br />
== Starting jails: Quad/Safe Files ==<br />
<br />
FreeBSD customer systems do not start up automatically at boot time. When one of our freebsd machines boots up, it boots up, and does nothing else. To start jails, we put the commands to start each jail into a shell script(s) and run the script(s). Jail startup is something that needs to be actively monitored, which is why we don’t just run the script automatically. More on monitoring later.<br />
<br />
NOTE: >=7.x we have moved to 1 quad file: <tt>quad1</tt>. Startups are not done by running each quad, but rather [[#startalljails|startalljails]] which relies on the contents of <tt>quad1</tt>. The specifics of this are lower in this article. What follows here applies for pre 7.x systems.<br />
<br />
There are eight files in <tt>/usr/local/jail/rc.d</tt>:<br />
<br />
<pre>jail3# ls /usr/local/jail/rc.d/<br />
quad1 quad2 quad3 quad4 safe1 safe2 safe3 safe4<br />
jail3#</pre><br />
<br />
four quad files and four safe files.<br />
<br />
Each file contains an even number of system startup blocks (total number of jails divided by 4)<br />
<br />
The reason for this is, if we make one large script to startup all the systems at boot time, it will take too long - the first system in the script will start up right after system boot, which is great, but the last system may not start for another 20 minutes.<br />
<br />
Since there is no way to parralelize this during the startup procedure, we simply open four terminals (in screen window 9) and run each script, one in each terminal. This way they all run simultaneously, and the very last system in each startup script gets started in 1/4th the time it would if there was one large file<br />
<br />
The files are generally organized so that quad/safe 1&2 have only jails from disk 1, and quad/safe 3&4 have jails from disk 2. This helps ensure that only 2 fscks on any disk are going on at once. Further, they are balanced so that all quad/safe’s finish executing around the same time. We do this by making sure each quad/safe has a similar number of jails and represents a similar number of inodes (see js).<br />
<br />
The other, very important reason we do it this way, and this is the reason there are quad files and safe files, is that in the event of a system crash, every single vn-backed filesystem that was mounted at the time of system crash needs to be fsck'd. However, fsck'ing takes time, so if we shut the system down gracefully, we don't want to fsck.<br />
<br />
Therefore, we have two sets of scripts - the four quad scripts are identical to the four safe scripts except for the fact that the quad scripts contain fsck commands for each filesystem.<br />
<br />
So, if you shut a system down gracefully, start four terminals and run safe1 in window one, and safe2 in window 2, and so on.<br />
<br />
If you crash, start four terminals (or go to screen window 9) and run quad1 in window one, and quad2 in window 2, and so on.<br />
<br />
Here is a snip of (a 4.x version) quad2 from jail17:<br />
<br />
<pre>vnconfig /dev/vn16 /mnt/data2/69.55.228.7-col00820<br />
fsck -y /dev/vn16<br />
mount /dev/vn16c /mnt/data2/69.55.228.7-col00820-DIR<br />
chmod 0666 /mnt/data2/69.55.228.7-col00820-DIR/dev/null<br />
jail /mnt/data2/69.55.228.7-col00820-DIR mail1.phimail.com 69.55.228.7 /bin/sh /etc/rc<br />
<br />
# moved to data2 col00368<br />
#vnconfig /dev/vn28 /mnt/data2/69.55.236.132-col00368<br />
#fsck -y /dev/vn28<br />
#mount /dev/vn28c /mnt/data2/69.55.236.132-col00368-DIR<br />
#chmod 0666 /mnt/data2/69.55.236.132-col00368-DIR/dev/null<br />
#jail /mnt/data2/69.55.236.132-col00368-DIR limehouse.org 69.55.236.132 /bin/sh /etc/rc<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
vnconfig /dev/vn22 /mnt/data2/69.55.228.13-col01063<br />
fsck -y /dev/vn22<br />
mount /dev/vn22c /mnt/data2/69.55.228.13-col01063-DIR<br />
chmod 0666 /mnt/data2/69.55.228.13-col01063-DIR/dev/null<br />
jail /mnt/data2/69.55.228.13-col01063-DIR www.widestream.com.au 69.55.228.13 /bin/sh /etc/rc<br />
<br />
# cancelled col00106<br />
#vnconfig /dev/vn15 /mnt/data2/69.55.238.5-col00106<br />
#fsck -y /dev/vn15<br />
#mount /dev/vn15c /mnt/data2/69.55.238.5-col00106-DIR<br />
#chmod 0666 /mnt/data2/69.55.238.5-col00106-DIR/dev/null<br />
#jail /mnt/data2/69.55.238.5-col00106-DIR mail.azebu.net 69.55.238.5 /bin/sh /etc/rc</pre><br />
<br />
As you can see, two of the systems specified are commented out - presumably those customers cancelled, or were moved to new servers.<br />
<br />
As you can see, the vnconfig line is the simpler command line, not the longer one that was used when it was first configured. As you can see, all that is done is, vnconfig the filesystem, then fsck it, then mount it. The fourth command is the `jail` command used to start the system – but that will be covered later.<br />
<br />
Here is the safe2 file from jail17:<br />
<br />
<pre>vnconfig /dev/vn16 /mnt/data2/69.55.228.7-col00820<br />
mount /dev/vn16c /mnt/data2/69.55.228.7-col00820-DIR<br />
chmod 0666 /mnt/data2/69.55.228.7-col00820-DIR/dev/null<br />
jail /mnt/data2/69.55.228.7-col00820-DIR mail1.phimail.com 69.55.228.7 /bin/sh /etc/rc<br />
<br />
# moved to data2 col00368<br />
#vnconfig /dev/vn28 /mnt/data2/69.55.236.132-col00368<br />
#mount /dev/vn28c /mnt/data2/69.55.236.132-col00368-DIR<br />
#chmod 0666 /mnt/data2/69.55.236.132-col00368-DIR/dev/null<br />
#jail /mnt/data2/69.55.236.132-col00368-DIR limehouse.org 69.55.236.132 /bin/sh /etc/rc<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
vnconfig /dev/vn22 /mnt/data2/69.55.228.13-col01063<br />
mount /dev/vn22c /mnt/data2/69.55.228.13-col01063-DIR<br />
chmod 0666 /mnt/data2/69.55.228.13-col01063-DIR/dev/null<br />
jail /mnt/data2/69.55.228.13-col01063-DIR www.widestream.com.au 69.55.228.13 /bin/sh /etc/rc<br />
<br />
# cancelled col00106<br />
#vnconfig /dev/vn15 /mnt/data2/69.55.238.5-col00106<br />
#mount /dev/vn15c /mnt/data2/69.55.238.5-col00106-DIR<br />
#chmod 0666 /mnt/data2/69.55.238.5-col00106-DIR/dev/null<br />
#jail /mnt/data2/69.55.238.5-col00106-DIR mail.azebu.net 69.55.238.5 /bin/sh /etc/rc</pre><br />
<br />
As you can see, it is exactly the same, but it does not have the fsck lines.<br />
<br />
Take a look at the last entry - note that the file is named:<br />
<br />
/mnt/data2/69.55.238.5-col00106<br />
<br />
and the mount point is named:<br />
<br />
/mnt/data2/69.55.238.5-col00106-DIR<br />
<br />
This is the general format on all the FreeBSD systems. The file is always named:<br />
<br />
IP-custnumber<br />
<br />
and the directory is named:<br />
<br />
IP-custnumber-DIR<br />
<br />
If you run safe when you need a fsck, the mount will fail and jail will fail:<br />
<br />
# mount /dev/vn1c /mnt/data2/jails/65.248.2.131-ns1.kozubik.com-DIR<br />
mount: /dev/vn1c: Operation not permitted<br />
<br />
No reboot needed, just run the quad script<br />
<br />
Starting with 6.x jails, we added block delimiters to the quad/safe files, the block looks like:<br />
<br />
<pre>echo '## begin ##: nuie.solaris.mu'<br />
fsck -y /dev/concat/v30v31a<br />
mount /dev/concat/v30v31a /mnt/data1/69.55.228.218-col01441-DIR<br />
mount_devfs devfs /mnt/data1/69.55.228.218-col01441-DIR/dev<br />
devfs -m /mnt/data1/69.55.228.218-col01441-DIR/dev rule -s 3 applyset<br />
jail /mnt/data1/69.55.228.218-col01441-DIR nuie.solaris.mu 69.55.228.218 /bin/sh /etc/rc<br />
echo '## end ##: nuie.solaris.mu'</pre><br />
<br />
These are more than just informative when running quad/safe’s, the echo lines MUST be present for certain tools to work properly. So it’s important that any updates to the hostname also be updated on the 2 echo lines. For example, if you try to startjail a jail with a hostname which is on the jail line but not the echo lines, the command will return with host not found.<br />
<br />
=== FreeBSD 7.x+ notes ===<br />
<br />
Starting with the release of FreeBSD 7.x, we are doing jail startups in a slightly different way. First, thereis only 1 file: <tt>/usr/local/jail/rc.d/quad1</tt><br />
There are no other quads or corresponding safe files. The reason for this is twofold, 1. We can pass –C to fsck which will tell is to skip the fsck if the fs is clean (no more need for safe files), 2. We have a new startup script which can be launched multiple times, running in parallel to start jails, where quad1 is the master jail file. <br />
Quad1 could still be run as a shell script, but it would take a very long time for it to run completely so it’s not advisable; or you should break it down into smaller chunks (like quad1, quad2, quad3, etc)<br />
<br />
Here is a snip of (a 7.x version) quad1 from jail2:<br />
<br />
<pre>echo '## begin ##: projects.tw.com'<br />
mdconfig -a -t vnode -f /mnt/data1/69.55.230.46-col01213 -u 50<br />
fsck -Cy /dev/md50c<br />
mount /dev/md50c /mnt/data1/69.55.230.46-col01213-DIR<br />
mount -t devfs devfs /mnt/data1/69.55.230.46-col01213-DIR/dev<br />
devfs -m /mnt/data1/69.55.230.46-col01213-DIR/dev rule -s 3 applyset<br />
jail /mnt/data1/69.55.230.46-col01213-DIR projects.tw.com 69.55.230.46 /bin/sh /etc/rc<br />
echo '## end ##: projects.tw.com'</pre><br />
<br />
Cancelled jails are no longer commented out and stored in quad1, rather they’re moved to <tt>/usr/local/jail/rc.d/deprecated</tt> <br />
<br />
To start these jails, start the 4 ssh sessions as you would for a normal crash and then instead of running quad1-4, instead run startalljails in each window. IMPORTANT- before running startalljails you should make sure you ran preboot once as it will clear out all the lockfiles and enable startalljails to work properly.<br />
<br />
== Problems with the quad/safe files ==<br />
<br />
When you run the quad/safe files, there are two problems that can occur - either a particular system will hang during initialization, OR a system will spit out output to the screen, impeding your ability to do anything. Or both.<br />
<br />
First off, when you start a jail, you see output like this:<br />
<br />
<pre>Skipping disk checks ...<br />
adjkerntz[25285]: sysctl(put_wallclock): Operation not permitted<br />
Doing initial network setup:.<br />
ifconfig: ioctl (SIOCDIFADDR): permission denied<br />
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384<br />
Additional routing options: TCP keepalive=YESsysctl:<br />
net.inet.tcp.always_keepalive: Operation not permitted.<br />
Routing daemons:.<br />
Additional daemons: syslogd.<br />
Doing additional network setup:.<br />
Starting final network daemons:.<br />
ELF ldconfig path: /usr/lib /usr/lib/compat /usr/X11R6/lib /usr/local/lib<br />
a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout /usr/X11R6/lib/aout<br />
Starting standard daemons: inetd cron sshd sendmail sendmail-clientmqueue.<br />
Initial rc.i386 initialization:.<br />
Configuring syscons: blanktime.<br />
Additional ABI support:.<br />
Local package initialization:.<br />
Additional TCP options:.</pre><br />
<br />
Now, let's look at this line, near the end:<br />
<br />
Local package initialization:.<br />
<br />
This is where a list of daemons that are set to start at boot time willshow up. You might see something like:<br />
<br />
Local package initialization: mysqld apache sendmail sendmail-clientmqueue<br />
<br />
Or something like this:<br />
<br />
Local package initialization: postgres postfix apache<br />
<br />
The problem is that many systems (about 4-5 per machine) will hang on that line. Basically it will get to some of the way through the total daemons to be started:<br />
<br />
Local package initialization: mysqld apache<br />
<br />
and will just sit there. Forever.<br />
<br />
Fortunately, pressing ctrl-c will break out of it. Not only will it break out of it, but it will also continue on that same line and start the other daemons:<br />
<br />
Local package initialization: mysqld apache ^c sendmail-clientmqueue<br />
<br />
and then continue on to finish the startup, and then move to the next system to be started.<br />
<br />
So what does this mean? It means that if a machine crashes, and you start four screen-windows to run four quads or four safes, you need to periodically cycle between them and see if any systems are stuck at that point, causing their quad/safe file to hang. A good rule of thumb is, if you see a system at that point in the startup, give it another 100 seconds - if it is still at the exact same spot, hit ctrl-c. Its also a good idea to go back into the quad file (just before the first command in the jail startup block) and note that this jail tends to need a control-c or more time as follows:<br />
<pre>echo '### NOTE ### slow sendmail'<br />
echo '### NOTE ###: ^C @ Starting sendmail.'</pre><br />
<br />
'''NEVER''' hit ctrl-c repeatedly if you don't get an immediate response - that will cause the following jail’s startup commands to be aborted.<br />
<br />
A second problem that can occur is that a jail - maybe the first one in that particular quad/safe, maybe the last one, or maybe one in the middle, will start spitting out status or error messages from one of its init scripts. This is not a problem - basically, hit enter a few times and see if you get a prompt - if you do get a prompt, that means that the quad/safe script has already completed. Therefore it is safe to log out (and log out of the user that you su'd from) and then log back in (if necessary).<br />
<br />
The tricky thing is, if a system in the middle starts flooding with messages, and you hit enter a few times and don't get a prompt. Are you not getting a prompt because some subsequent system is hanging at the initialization, as we discussede above ? Or are you not getting a prompt because that quad file is currently running an fsck ? Usually you can tell by scrolling back in screen’s history to see what it was doing before you started getting the messages.<br />
<br />
If you don’t get clues from history, you have to use your judgement - instead of giving it 100 seconds to respond, perhaps give it 2-3 mins ... if you still get no response (no prompt) when you hit enter, hit ctrl-c. However, be aware that you might still be hitting ctrl-c in the middle of an fsck. This means you will get an error like "filesystem still marked dirty" and then the vnconfig for it will fail and so will the jail command, and the next system in the quad file will then start starting up.<br />
<br />
If this happens, just wait until the end of all the quad files have finished, and start that system manually.<br />
<br />
If things really get weird, like a screen flooded with errors, and you can't get a prompt, and ctrl-c does nothing, then you need to just eventually (give it ten mins or so) just kill that window with ctrl-p, then k, and then log in again and manually check which systems are now running and which aren't, and manually start up any that are not.<br />
<br />
Don't EVER risk running a particular quad/safe file a second time.<br />
If the quad/safe script gets executed twice, reboot the machine immediately.<br />
<br />
So, for all the above reasons, anytime a machine crashes and you run all the quads or all the safes, '''always''' check every jail afterwards to make sure it is running - even if you have no hangs or complications at all.<br />
Run this command:<br />
<br />
<tt>[[#jailpsall|jailpsall]]</tt> <br />
<br />
Note: [[#postboot|postboot]] also populates ipfw counts, so it '''should not be run multiple times''', use <tt>jailpsall</tt> for subsequent extensive ps’ing<br />
<br />
And make sure they all show as running. If one does not show as running, check its /etc/rc.conf file first to see if maybe it is using a different hostname first before starting it manually.<br />
<br />
One thing we have implemented to alleviate these startup hangs and noisy jails, is to put jail start blocks that are slow or hangy at the bottom of the safe/quad file. Further, for each bad jail we note in each quad/safe just before the start block something like:<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
<br />
That way we’ll be prepared to ^C when we see that message appear during the quad/safe startup process. If you observe a new, undocumented hang, '''after''' the quad/safe has finished, place a line similar to the above in the quad file, move the jail start block to the end of the file, then run [[#buildsafe|buildsafe]]<br />
<br />
<br />
== Recovering from a crash (FreeBSD) ==<br />
<br />
=== Diagnose whether you have a crash ===<br />
The most important thing is to get the machine and all jails back up as soon as possible. Note the time, you’ll need to create a crash log entry (Mgmt. -> Reference -> CrashLog). The first thing to do is head over to the [[Screen#Screen_Organization|serial console screen]] and see if there’s any kernel error messages output. Try to copy any messages (or just a sample of repeating messages) you see into the notes section of the crash log. If there are no messages, the machine may just be really busy- wait a bit (5-10min) to see if it comes back. If it's still pinging, odds are its very busy. Note, if you see messages about swap space exhausted, the server is obviously out of memory, however it may recover briefly enough for you to get a jtop in to see who's lauched a ton of procs (most likely) and then issue a quick jailkill to get it back under control.<br />
<br />
If it doesn't come back, or the messages indicate a fatal error, you will need to proceed with a power cycle (ctrl+alt+del will not work).<br />
<br />
=== Power cycle the server ===<br />
If this machine is not a Dell 2950 with a [[DRAC/RMM#DRAC|DRAC card]] (i.e. if you can’t ssh into the DRAC card (as root, using the standard root pass) and issue <br />
racadm serveraction hardreset<br />
then you will need someone at the data center power the macine off, wait 30 sec, then turn it back on. Make sure to re-attach via console:<br />
tip jailX<br />
immediately after power down. <br />
<br />
=== (Re)attach to the console ===<br />
Stay on the console the entire time during boot. As the BIOS posts- look out for the RAID card output- does everything look healthy? The output may be scrambled, look for "DEGRADED" or "FAILED". Once the OS starts booting you will be disconnected (dropped back to the shell on the console server) a couple times during the boot up. The reason you want to quickly re-attach is two-fold: 1. If you don’t reattach quickly then you won’t get any console output, 2. you want to be attached before the server ''potentially'' starts (an extensive) fsck. If you attach after the fsck begins, you’ll have seen no indication it started an fsck and the server will appear frozen during startup- no output, no response. <br />
<br />
IMPORTANT NOTE: on some older FreeBSD systems, there will be no output to the video (KVM) console as it boots up. The console output is redirected to the serial port ... so if a jail crashes, and you attach a kvm, the output during the bootup procedure will not be shown on the screen. However, when the bootup is done, you will get a login prompt on the screen and will be able to log in as normal. <tt>/boot/loader.conf</tt> is where serial console redirect output lives, so comment that if you want to catch output on kvm.<br />
On newer systems it sends most output to both locations. <br />
<br />
=== Assess the heath of the server ===<br />
Once the server boots up fully, you should be able to ssh in. Look around- make sure all the mounts are there and reporting the correct size/usage (i.e. /mnt/data1 /mnt/data2 /mnt/data3 - look in /etc/fstab to determine which mount points should be there), check to see if RAID mirrors are healthy. See [[RAID_Cards#Common_CLI_commands_.28megacli.29|megacli]], [[#aaccheck|aaccheck]]<br />
<br />
Before you start the jails, you need to run [[#preboot|preboot]]. This will do some assurance checks to make sure things are prepped to start the jails. Any issues that come out of preboot need to be addressed before starting jails.<br />
<br />
=== Start jails ===<br />
[[#Starting_jails:_Quad.2FSafe_Files|More on starting jails]]<br />
Customer jails (the VPSs) do not start up automatically at boot time. When a FreeBSD machines boots up, it boots up, and does nothing else. To start jails, we put the commands to start each jail into a shell script(s) and run the script(s). Jail startup is something that needs to be actively monitored, which is why we don’t just run the script automatically. <br />
<br />
In order to start jails, we run the quad files: quad1 quad2 quad3 and quad4 (on new systems there is only quad1). If the machine was cleanly rebooted- which wouldn't be the case if this was a crash, you may run the safe files (safe1 safe2 safe3 safe4) in lieu of quads. <br />
<br />
Open up 4 logins to the server (use the windows in [[Screen#Screen_Organization|a9]])<br />
In each of the 4 windows you will:<br />
<br />
If there is a [[#startalljails|startalljails]] script (and only quad1), run that command in each of the 4 windows. It will parse through the quad1 file and start each jail. Follow the instructions [[#Problems_with_the_quad.2Fsafe_files|here]] for monitoring startup. Note that you can be a little more lenient with jails that take awhile to start- startalljails will work around the slow jails and start the rest. As long as there aren't 4 jails which are "hung" during startup, the rest will get started eventually.<br />
-or-<br />
If there is no startalljails script, there will be multiple quad files. In each of the 4 windows, start each of the quads. i.e. start quad1 in window1, quad2 in window2 and so on. DO NOT start any quad twice. It will crash the server. If you accidentally do this, just jailkill all the jails which are in the quad and run the quad again. Follow the instructions here for monitoring quad startup.<br />
<br />
Note the time the last jail boots- this is what you will enter in the crash log.<br />
<br />
Save the crash log.<br />
<br />
=== Check to make sure all jails have started ===<br />
There's a simple script which will make sure all jails have started, and enter the ipfw counter rules: [[#postboot|postboot]] <br />
Run postboot, which will do a jailps on each jail it finds (excluding commented out jails) in the quad file(s). We're looking for 2 things:<br />
# systems spawning out of control or too many procs<br />
# jails which haven't started<br />
On 7.x and newer systems it will print out the problems (which jails haven't started) at the conclusion of postboot. <br />
On older systems you will need to watch closely to see if/when there's a problem, namely:<br />
<br />
[hostname] doesnt exist on this server<br />
<br />
When you get this message, it means one of 2 things:<br />
1. the jail really didn't start:<br />
When a jail doesn't start it usually boils down to a problem in the quad file. Perhaps the path name is wrong (data1 vs data2) or the name of the vn/mdfile is wrong. Once this is corrected, you will need to run the commands from the quad file manually, or you may use <tt>startjail <hostname></tt><br />
<br />
2. the customer has changed their hostname (and not told us) so their jail ''is'' running, just under a different hostname:<br />
On systems with jls, this is easy to rectify. First, get the customer info: <tt>g <hostname></tt><br />
Then look for the customer in jls: <tt>jls | grep <col0XXXX></tt><br />
From there you will see their new hostname- you should update that hostname in the quad file: don't forget to edit it on the <tt>## begin ##</tt> and <tt>## end ##</tt> lines, and in mgmt. <br />
On older systems without jls, this will be harder, you will need to look further to see their hostname- perhaps its in their /etc/rc.conf<br />
<br />
<br />
Once all jails are started, do some spot checks- try to ssh or browse to some customers, just to make sure things are really ok.<br />
<br />
== Adding disk to a 7.x/8.x jail ==<br />
or<br />
== Moving customer to a different drive (md) ==<br />
<br />
NOTE: this doesn’t apply to mx2 which uses gvinum. Use same procedure as 6.x<br />
NOTE: if you unmount before mdconfig, re-mdconfig (attach) then unmount then mdconfig -u again <br />
<br />
<br />
(parts to change/customize are <tt><span style="color:red">red</span></tt>)<br />
<br />
<br />
If someone wants more disk space, there’s a paste for it, send it to them (explains about downtime, etc).<br />
<br />
1. Figure out the space avail from <tt>js</tt>. Ideally, you want to put the customers new space on a different partition (and create the new md on the new partition). <br />
<br />
<br />
2. make a mental note of how much space they're currently using<br />
<br />
<br />
3. <tt>jailkill <hostname></tt> <br />
<br />
<br />
4. Umount it (including their devfs) but leave the md config’d (so if you use stopjail, you will have to re-mdconfig it)<br />
<br />
<br />
5. <tt>g <customerID></tt> to get the info (IP/cust#) needed to feed the new mdfile and mount name, and to see the current md device. <br />
<br />
<br />
6a. When there's enough room to place new system on an alternate, or the same drive:<br />
USE CAUTION not to overwrite (touch, mdconfig) existing md!!<br><br />
<tt>touch /mnt/data<span style="color:red">3/69.55.234.66-col01334</span><br><br />
mdconfig -a -t vnode -s 10g -f /mnt/data3/69.55.234.66-col01334 -u 97<br><br />
newfs /dev/md97</tt><br />
<br />
<br />
Optional- if new space is on a different drive, move the mount point directory AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
<tt>mount /dev/md<span style="color:red">97</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<br />
confirm you are mounted to /dev/md<span style="color:red">97</span> and space is correct:<br><br />
<tt>df .</tt> <br />
<br />
<br />
do the dump and pipe directly to restore:<br><br />
<tt>dump -0a -f - /dev/md<span style="color:red">1</span> | restore -r -f - <br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use <tt>df</tt> to confirm the restored data size matches the original usage figure<br />
<br />
<br />
md-unconfig old system:<br><br />
<tt>mdconfig -d -u <span style="color:red">1</span></tt><br />
<br />
<br />
archive old mdfile. <br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.237.26-col00241</span> /mnt/data<span style="color:red">1/old-col00241-mdfile-noarchive-20091211</span></tt><br />
<br />
<br />
edit quad (vq1) to point to new (/mnt/data<span style="color:red">3</span>) location AND new md number (md<span style="color:red">97</span>)<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
6b. When there's not enough room on an alternate partition, or on the same drive...but there is enough room if you were to remove the existing customer's space:<br />
<br />
<br />
mount backup nfs mounts:<br><br />
<tt>mbm</tt><br> <br />
(run <tt>df</tt> to confirm backup mounts are mounted)<br />
<br />
<br />
dump the customer to backup2 or backup1<br><br />
<tt>dump -0a -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> /dev/md<span style="color:red">1</span></tt><br><br />
(when complete WITHOUT errors, <tt>du</tt> the dump file to confirm it matches size, roughly, with usage)<br />
<br />
<br />
unconfigure and remove old mdfile<br><br />
<tt>mdconfig -d -u <span style="color:red">1</span><br><br />
rm /mnt/data<span style="color:red">1/69.55.237.26-col00241</span></tt><br><br />
(there should now be enough space to recreate your bigger system. If not, run sync a couple times)<br />
<br />
<br />
create the new system (ok to reuse old mdfile and md#):<br><br />
<tt>touch /mnt/data<span style="color:red">1/69.55.234.66-col01334</span><br><br />
mdconfig -a -t vnode -s <span style="color:red">10</span>g -f /mnt/data<span style="color:red">1/69.55.234.66-col01334</span> -u <span style="color:red">1</span><br><br />
newfs /dev/md<span style="color:red">1</span><br><br />
mount /dev/md<span style="color:red">1</span> /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> <br><br />
cd /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<br />
confirm you are mounted to /dev/md<span style="color:red">1</span> and space is correct:<br><br />
<tt>df .</tt> <br />
<br />
<br />
do the restore from the dumpfile on the backup server:<br><br />
<tt>restore -r -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> .<br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
umount nfs:<br><br />
<tt>mbu</tt><br />
<br />
<br />
If md# changed (or mount point), edit quad (<tt>vq1</tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new md number (md<span style="color:red">1</span>)<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
7. update disk (and dir if applicable) in mgmt screen<br />
<br />
<br />
8. update backup list AND move backups, if applicatble<br />
<br />
Ex: <tt>mvbackups <span style="color:red">69.55.237.26-col00241</span> jail<span style="color:red">9</span> data<span style="color:red">3</span></tt><br />
<br />
<br />
9. Optional: archive old mdfile<br />
<br />
<tt>mbm<br><br />
gzip -c old-col01588-mdfile-noarchive-20120329 > /deprecated/old-col01588-mdfile-noarchive-20120329.gz<br><br />
mbu<br><br />
rm old-col01588-mdfile-noarchive-20120329</tt><br />
<br />
== Adding disk to a 6.x jail (gvinum/gconcat) ==<br />
or<br />
== Moving customer to a different drive (gvinum/gconcat) ==<br />
<br />
(parts to change are <span style="color:red">highlighted</span>)<br />
<br />
<br />
If someone wants more disk space, there’s a paste for it, send it to them (explains about downtime, etc).<br />
<br />
<br />
1. Figure out the space avail from [[#js|js]]. Ideally, you want to put the customers new space on a different partition (and create the new md on the new partition). <br />
<br />
<br />
2. make a mental note of how much space they're currently using<br />
<br />
<br />
3. <tt>[[#stopjail|stopjail]] <hostname></tt> <br />
<br />
<br />
4. <tt>[[#g|g]] <customerID></tt> to get the info (IP/cust#) needed to feed the new mount name and existing volume/device. <br />
<br />
<br />
5a. When there's enough room to place new system on an alternate, or the same drive (using only UNUSED - including if it's in use by the system in question - gvinum volumes):<br />
<br />
<br />
Configure the new device:<br><br />
A. for a 2G system (single gvinum volume):<br><br />
<tt>bsdlabel -r -w /dev/gvinum/v<span style="color:red">123</span><br><br />
newfs /dev/gvinum/v<span style="color:red">123</span>a</tt><br><br />
<br />
-or- <br />
<br />
<br />
B. for a >2G system (create a gconcat volume):<br><br />
try to grab a contiguous block of gvinum volumes. gconcat volumes MAY NOT span drives (i.e. you cannot use a gvinum volume from data3 and a volume from data2 in the same gconcat volume). <br />
<br />
<tt>gconcat label <span style="color:red">v82-v84 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84</span><br><br />
bsdlabel -r -w /dev/concat/<span style="color:red">v82-v84</span><br><br />
newfs /dev/concat/<span style="color:red">v82-v84</span>a</tt><br />
<br />
<br />
Other valid gconcat examples:<br><br />
<tt>gconcat label v82-v84v109v112 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84 /dev/gvinum/v109 /dev/gvinum/v112<br><br />
gconcat label v82v83 /dev/gvinum/v82 /dev/gvinum/v83</tt> <br><br />
Note, long names will truncate: v144v145v148-v115 will truncate to v144v145v148-v1 (so you will refer to it as v144v145v148-v1 thereafter)<br />
<br />
<br />
Optional- if new volume is on a different drive, move the mount point directory (get the drive from js output) AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
<br />
confirm you are mounted to the device (<tt>/dev/gvinum/v<span style="color:red">123</span>a</tt> OR <tt>/dev/concat/<span style="color:red">v82-v84</span></tt>) and space is correct:<br><br />
A. <tt>mount /dev/gvinum/v<span style="color:red">123</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br><br />
-or-<br><br />
B. <tt>mount /dev/concat/<span style="color:red">v82-v84</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<tt>cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
df .</tt> <br />
<br />
<br />
do the dump and pipe directly to restore:<br><br />
<tt>dump -0a -f - /dev/gvinum/v<span style="color:red">1</span> | restore -r -f - <br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
edit quad (<tt>vq<span style="color:red">1</span></tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new volume (<tt>/dev/gvinum/v<span style="color:red">123</span>a</tt> or <tt>/dev/concat/<span style="color:red">v82-v84</span>a</tt>) , run <tt>buildsafe</tt><br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
5b. When there's not enough room on an alternate partition, or on the same drive...but there is enough room if you were to remove the existing customer's space (i.e. if you want/need to reuse the existing gvinum volumes and add on more):<br />
<br />
<br />
mount backup nfs mounts:<br><br />
<tt>mbm</tt> <br><br />
(run df to confirm backup mounts are mounted)<br />
<br />
<br />
dump the customer to backup2 or backup1<br><br />
<tt>dump -0a -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> /dev/<span style="color:red">gconcat/v106-v107</span></tt><br><br />
(when complete WITHOUT errors, du the dump file to confirm it matches size, roughly, with usage)<br />
<br />
<br />
unconfigure the old gconcat volume<br><br />
list member gvinum volumes:<br />
<br />
<tt>gconcat list <span style="color:red">v106v107</span></tt><br />
<br />
<br />
Output will resemble:<br><br />
<pre>Geom name: v106v107<br />
State: UP<br />
Status: Total=2, Online=2<br />
Type: AUTOMATIC<br />
ID: 3530663882<br />
Providers:<br />
1. Name: concat/v106v107<br />
Mediasize: 4294966272 (4.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e2<br />
Consumers:<br />
1. Name: gvinum/sd/v106.p0.s0<br />
Mediasize: 2147483648 (2.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e3<br />
Start: 0<br />
End: 2147483136<br />
2. Name: gvinum/sd/v107.p0.s0<br />
Mediasize: 2147483648 (2.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e3<br />
Start: 2147483136<br />
End: 4294966272</pre><br />
<br />
stop volume and clear members<br><br />
<tt>gconcat stop <span style="color:red">v106v107</span><br><br />
gconcat clear <span style="color:red">gvinum/sd/v106.p0.s0 gvinum/sd/v107.p0.s0</span></tt><br />
<br />
<br />
create new device- and its ok to reuse old/former members<br><br />
try to grab a contiguous block of gvinum volumes. gconcat volumes MAY NOT span drives (i.e. you cannot use a gvinum volume from data3 and a volume from data2 in the same gconcat volume). <br><br />
<tt>gconcat label <span style="color:red">v82-v84v106v107 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84 /dev/gvinum/v106 /dev/gvinum/v107</span> <br><br />
bsdlabel -r -w /dev/concat/<span style="color:red">v82-v84v106v107</span><br><br />
newfs /dev/concat/<span style="color:red">v82-v84v106v107</span>a</tt><br />
<br />
<br />
Optional- if new volume is on a different drive, move the mount point directory (get the drive from js output) AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
confirm you are mounted to the device (/dev/concat/<span style="color:red">v82-v84v106v107</span>) and space is correct:<br><br />
<tt>mount /dev/concat/<span style="color:red">v82-v84v106v107</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<tt>cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
df .</tt> <br />
<br />
do the restore from the dumpfile on the backup server:<br><br />
<tt>restore -r -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> .<br><br />
rm restoresymtable</tt><br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
edit quad (<tt>vq<span style="color:red">1</span></tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new volume (<tt>/dev/concat/<span style="color:red">v82-v84v106v107</span>a</tt>), run buildsafe<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
TODO: clean up/clear old gvin/gconcat vol<br />
<br />
<br />
6. update disk (and dir if applicable) in mgmt screen<br />
<br />
<br />
7. update backup list AND move backups, if applicatble<br />
<br />
<br />
Ex: <tt>mvbackups <span style="color:red">69.55.237.26-col00241</span> jail<span style="color:red">9</span> data<span style="color:red">3</span></tt><br />
<br />
<br />
<br />
DEPRECATED - steps to tack on a new gvin to existing gconcat- leads to corrupted fs<br />
bsdlabel -e /dev/concat/v82-v84<br />
<br />
To figure out new size of the c partition, multiply 4194304 by the # of 2G gvinum volumes and subtract the # of 2G volumes:<br />
10G: 4194304 * 5 – 5 = 20971515<br />
8G: 4194304 * 4 – 4 = 16777212<br />
6G: 4194304 * 3 – 3 = 12582909<br />
4G: 4194304 * 2 – 2 = 8388606<br />
<br />
To figure out the new size of the a partition, subtract 16 from the c partition:<br />
10G: 20971515 – 16 = 20971499<br />
8G: 16777212 – 16 = 16777196<br />
6G: 12582909 – 16 = 12582893<br />
4G: 8388606 – 16 = 8388590<br />
<br />
Orig:<br />
8 partitions:<br />
# size offset fstype [fsize bsize bps/cpg]<br />
a: 8388590 16 4.2BSD 2048 16384 28552<br />
c: 8388606 0 unused 0 0 # "raw" part, don't edit<br />
<br />
New:<br />
8 partitions:<br />
# size offset fstype [fsize bsize bps/cpg]<br />
a: 12582893 16 4.2BSD 2048 16384 28552<br />
c: 12582909 0 unused 0 0 # "raw" part, don't edit<br />
<br />
sync; sync<br />
<br />
growfs /dev/concat/v82-v84a<br />
<br />
fsck –fy /dev/concat/v82-v84a<br />
<br />
sync<br />
<br />
fsck –fy /dev/concat/v82-v84a<br />
<br />
(keep running fsck’s till NO errors)<br />
<br />
== Problems un-mounting - and with mount_null’s ==<br />
<br />
If you cannot unmount a filesystem, beacuse it says the filesystem is busy, it is because of three things:<br />
<br />
a) the jail is still running<br />
<br />
b) you are actually in that directory, even though the jail is stopped<br />
<br />
c) there are still dev, null_mount or linprocfs mount points mounted inside that directory.<br />
<br />
d) when trying to umount null_mounts that are really long and you get an error like “No such file or directory”, it’s an OS bug where the dir is truncated. No known fix<br />
<br />
e) there are still files open somewhere inside the dir. Use <tt>fstat | grep <cid></tt> to find the process that has files open<br />
<br />
f) Starting with 6.x, the jail mechanism does a poor job of keeping track of processes running in a jail and if it thinks there are still procs running, it will refuse to umount the disk. If this is happening you should see a low number in the #REF column when you run jls. In this case you ''can'' safely <tt>umount –f</tt> the mount. <br />
<br />
Please note -if you forcibly unmount a (4.x) filesystem that has null_mounts<br />
still mounted in it, the system '''will crash''' within 10-15 mins.<br />
<br />
== Misc jail Items ==<br />
<br />
We are overselling hard drive space on jail2, jail8, jail9, a couple jails on jail17, jail4, jail12 and jail18.<br />
Even though the vn file shows 4G size, it doesn’t actually occupy that amount of space on the disk. So be careful not to fill up drives where we’re overselling – use oversellcheck to confirm you’re not oversold by more than 10G.<br />
There are other truncated jails, they are generally noted in a the file on the root system: /root/truncated<br />
<br />
The act of moving a truncated vn to another system un-does the truncating- the truncated vn is filled with 0’s and it occupies physical disk space for which it’s configured. So, you should use dumpremote to preserve the truncation.<br />
<br />
= FreeBSD VPS Management Tools =<br />
<br />
These files are located in /usr/local/jail/rc.d and /usr/local/jail/bin<br />
<br />
== jailmake ==<br />
<br />
Applies to 7.x+ <br />
On older systems syntax differs, run jailmake once to see.<br />
<br />
Note: this procedure differs on mx2 which is 7.x but still uses gvinum<br />
<br />
# run js to figure out which md’s are in use, which disk has enough space, IP to put it on<br />
# use col00xxx for both hostnames if they don’t give you a hostname<br />
# copy over dir, ip and password to pending customer screen<br />
<br />
<tt>Usage: jailmake IP[,IP] CID disk[1|2|3] md# hostname shorthost ipfw# email [size in GB]</tt><br />
<br />
Ex: <br />
<br />
Jail2# jailmake 69.55.234.66 col01334 3 97 vps.bsd.it vps 1334 fb@bsd.it<br />
<br />
== jailps ==<br />
jailps [hostname]<br />
DEPRECATED FOR jps: displays processes belonging to/running inside a jail. The command<br />
takes one (optional) argument – the hostname of the jail you wish to query. If you don’t <br />
supply an argument, all processes on the machine are listed and grouped by jail. <br />
<br />
== jps ==<br />
jps [hostname]<br />
displays processes belonging to/running inside a jail. The command<br />
takes one (optional) argument – the hostname or ID of the jail you wish to query. <br />
<br />
== jailkill ==<br />
jailkill <hostname><br />
stops all process running in a jail.<br />
<br />
You can also run:<br />
jailkill <JID><br />
<br />
=== problems ===<br />
Occasionally you will hit an issue where jail will not kill off:<br />
<br />
<pre>jail9# jailkill www.domain.com<br />
www.domain.com .. killed: none<br />
jail9#</pre><br />
Because no processes are running under that hostname. You cannot use jailps.pl either:<br />
<br />
<pre>jail9# jailps www.domain.com<br />
www.domain.com doesn’t exist on this server<br />
jail9#</pre><br />
<br />
The reasons for this are usually:<br />
* the jail is no longer running<br />
<br />
* the jail's hostname has changed<br />
In this case, <br />
<br />
>=6.x: run a <tt>jls|grep <jail's IP></tt> to find the correct hostname, then update the quad file, then kill the jail.<br />
<br />
<6.x: the first step is to cat their /etc/rc.conf file to see if you can tell what they set the new hostname to. This very often works. For example:<br />
<br />
cat /mnt/data2/198.78.65.136-col00261-DIR/etc/rc.conf<br />
<br />
But maybe they set the hostname with the hostname command, and the original hostname is still in /etc/rc.conf.<br />
<br />
The welcome email clearly states that they should tell us if they change their hostname, so there is no problem in just emailing them and asking them what they set the new hostname to.<br />
<br />
Once you know the new hostname OR if a customer simply emails to inform you that they have set the hostname to something different, you need to edit the quad and safe files that their system is in to input the new hostname.<br />
<br />
However, if push comes to shove and you cannot find out the hostname from them or from their system, then you need to start doing some detective work.<br />
<br />
The easiest thing to do is run jailps looking for a hostname similar to their original hostname. Or you could get into the /bin/sh shell by running:<br />
<br />
/bin/sh<br />
<br />
and then looking at every hostname of every process:<br />
<br />
for f in `ls /proc` ; do cat /proc/$f/status ; done<br />
<br />
and scanning for a hostname that is either similar to their original hostname, or that you don't see in any of the quad safe files.<br />
<br />
This is very brute force though, and it is possible that catting every file in /proc is dangerous - I don't recommend it. A better thing would be to identify any processes that you know belong to this system – perhaps the reason you are trying to find this system is because they are running something bad - and just catting the status from only that PID.<br />
<br />
Somewhere there’s a jail where there may be 2 systems named www. Look at /etc/rc.conf and make sure they’re both really www. If they are, jailkill www, jailps www to make sure not running. Then immediately restart the other one, as the fqdn (as found from a rev nslookup)<br />
<br />
* on >=6.x the hostname may not yet be hashed:<br />
<pre>jail9 /# jls<br />
JID Hostname Path IP Address(es)<br />
1 bitnet.dgate.org /mnt/data1/69.55.232.50-col02094-DIR 69.55.232.50<br />
2 ns3.hctc.net /mnt/data1/69.55.234.52-col01925-DIR 69.55.234.52<br />
3 bsd1 /mnt/data1/69.55.232.44-col00155-DIR 69.55.232.44<br />
4 let2.bbag.org /mnt/data1/69.55.230.92-col00202-DIR 69.55.230.92<br />
5 post.org /mnt/data2/69.55.232.51-col02095-DIR 69.55.232.51 ...<br />
6 ns2 /mnt/data1/69.55.232.47-col01506-DIR 69.55.232.47 ...<br />
7 arlen.server.net /mnt/data1/69.55.232.52-col01171-DIR 69.55.232.52<br />
8 deskfood.com /mnt/data1/69.55.232.71-col00419-DIR 69.55.232.71<br />
9 mirage.confluentforms.com /mnt/data1/69.55.232.54-col02105-DIR 69.55.232.54 ...<br />
10 beachmember.com /mnt/data1/69.55.232.59-col02107-DIR 69.55.232.59<br />
11 www.agottem.com /mnt/data1/69.55.232.60-col02109-DIR 69.55.232.60<br />
12 sdhobbit.myglance.org /mnt/data1/69.55.236.82-col01708-DIR 69.55.236.82<br />
13 ns1.jnielsen.net /mnt/data1/69.55.234.48-col00204-DIR 69.55.234.48 ...<br />
14 ymt.rollingegg.net /mnt/data2/69.55.236.71-col01678-DIR 69.55.236.71<br />
15 verse.unixlore.net /mnt/data1/69.55.232.58-col02131-DIR 69.55.232.58<br />
16 smcc-mail.org /mnt/data2/69.55.232.68-col02144-DIR 69.55.232.68<br />
17 kasoutsuki.w4jdh.net /mnt/data2/69.55.232.46-col02147-DIR 69.55.232.46<br />
18 dili.thium.net /mnt/data2/69.55.232.80-col01901-DIR 69.55.232.80<br />
20 www.tekmarsis.com /mnt/data2/69.55.232.66-col02155-DIR 69.55.232.66<br />
21 vps.yoxel.net /mnt/data2/69.55.236.67-col01673-DIR 69.55.236.67<br />
22 smitty.twitalertz.com /mnt/data2/69.55.232.84-col02153-DIR 69.55.232.84<br />
23 deliver4.klatha.com /mnt/data2/69.55.232.67-col02160-DIR 69.55.232.67<br />
24 nideffer.com /mnt/data2/69.55.232.65-col00412-DIR 69.55.232.65<br />
25 usa.hanyuan.com /mnt/data2/69.55.232.57-col02163-DIR 69.55.232.57<br />
26 daifuku.ppbh.com /mnt/data2/69.55.236.91-col01720-DIR 69.55.236.91<br />
27 collins.greencape.net /mnt/data2/69.55.232.83-col01294-DIR 69.55.232.83<br />
28 ragebox.com /mnt/data2/69.55.230.104-col01278-DIR 69.55.230.104<br />
29 outside.mt.net /mnt/data2/69.55.232.72-col02166-DIR 69.55.232.72<br />
30 vps.payneful.ca /mnt/data2/69.55.234.98-col01999-DIR 69.55.234.98<br />
31 higgins /mnt/data2/69.55.232.87-col02165-DIR 69.55.232.87 ...<br />
32 ozymandius /mnt/data2/69.55.228.96-col01233-DIR 69.55.228.96<br />
33 trusted.realtors.org /mnt/data2/69.55.238.72-col02170-DIR 69.55.238.72<br />
34 jc1.flanderous.com /mnt/data2/69.55.239.22-col01504-DIR 69.55.239.22<br />
36 guppylog.com /mnt/data2/69.55.238.73-col00036-DIR 69.55.238.73<br />
40 haliohost.com /mnt/data2/69.55.234.41-col01916-DIR 69.55.234.41 ...<br />
41 satyr.jorge.cc /mnt/data1/69.55.232.70-col01963-DIR 69.55.232.70<br />
jail9 /# jailkill satyr.jorge.cc<br />
ERROR: jail_: jail "satyr,jorge,cc" not found<br />
</pre><br />
<br />
Note how it's saying <tt>satyr,jorge,cc</tt> is not found, and not <tt>satyr.jorge.cc</tt>. <br />
<br />
The jail subsystem tracks things using comma-delimited hostnames. That is created every few hours:<br />
<br />
jail9 /# crontab -l<br />
0 0,6,12,18 * * * /usr/local/jail/bin/sync_jail_names<br />
<br />
So if we run this manually:<br />
jail9 /# /usr/local/jail/bin/sync_jail_names<br />
<br />
Then kill the jail:<br />
jail9 /# jailkill satyr.jorge.cc<br />
successfully killed: satyr,jorge,cc</pre><br />
<br />
It worked.<br />
<br />
<br />
If you ever see this when trying to kill a jail:<br />
<br />
<pre># jailkill e-scribe.com<br />
killing JID: 6 hostname: e-scribe.com<br />
3 procs running<br />
3 procs running<br />
3 procs running<br />
3 procs running<br />
...</pre><br />
<br />
<tt>[[#jailkill|jailkill]]</tt> probably got lost trying to kill off the jail. Just ctrl-c the jailkill process, then run a jailps on the hostname, and <tt>kill -9</tt> any process which is still running. Keep running jailps and <tt>kill -9</tt> till all processes are gone.<br />
<br />
== jailpsall ==<br />
jailpsall<br />
will run a jailps on all jails configured in the quad files (this is different from<br />
jailps with no arguments as it won’t help you find a “hidden” system)<br />
<br />
== jailpsw ==<br />
jailpsw<br />
will run a jailps with an extra -w to provide wider output<br />
<br />
== jt (>=7.x) ==<br />
jt<br />
displays the top 20 processes on the server (the top 20 processes from top) and <br />
which jail owns them. This is very helpful for determining who is doing what when<br />
the server is very busy.<br />
<br />
== jtop (>=7.x) ==<br />
jtop<br />
a wrapper for top displaying processes on the server and which jail owns them. Constantly updates, like top. <br />
<br />
== jtop (<7.x) ==<br />
jtop<br />
displays the top 20 processes on the server (the top 20 processes from top) and <br />
which jail owns them. This is very helpful for determining who is doing what when<br />
the server is very busy.<br />
<br />
== stopjail ==<br />
stopjail <hostname> [1]<br />
this will jailkill, umount and vnconfig –u a jail. If passed an optional 2nd<br />
argument, it will not exit before umounting and un-vnconfig’ing in the event<br />
jailkill returns no processes killed. This is useful if you just want to umount<br />
and vnconfig –u a jail you’ve already killed. It is intelligent in that it won’t <br />
try to umount or vnconfig –u if it’s not necessary.<br />
<br />
== startjail ==<br />
startjail <hostname><br />
this will start vnconfig, mount (including linprocfs and null-mounts), and start a jail.<br />
Essentially, it reads the jail’s relevant block from the right quad file and executes it.<br />
It is intelligent in that it won’t try to mount or vnconfig if it’s not necessary.<br />
<br />
== jpid ==<br />
jpid <pid><br />
displays information about a process – including which jail owns it.<br />
It’s the equivalent of running cat /proc/<pid>/status<br />
<br />
== canceljail ==<br />
canceljail <hostname> [1]<br />
this will stop a jail (the equivalent of stopjail), check for backups (offer to remove them <br />
from the backup server and the backup.config), rename the vnfile, remove the dir, and <br />
edit quad/safe. If passed an optional 2nd argument, it will not exit upon failing to kill<br />
and processes owned by the jail. This is useful if you just want to cancel a jail which <br />
is already stopped.<br />
<br />
== jls ==<br />
jls [-v]<br />
Lists all jails running:<br />
<pre>JID #REF IP Address Hostname Path<br />
101 135 69.55.224.148 mail.pc9.org /mnt/data2/69.55.224.148-col01034-DIR</pre><br />
<br />
#REF is the number of references or procs(?) running<br />
<br />
Running with -v will give you all IPs assigned to each jail (7.2 up)<br />
<pre>JID #REF Hostname Path IP Address(es)<br />
101 139 mail.pc9.org /mnt/data2/69.55.224.148-col01034-DIR 69.55.224.14869.55.234.85</pre><br />
<br />
== startalljails ==<br />
startalljails<br />
7.2+ only. This will parse through quad1 and start all jails. It utilizes lockfiles so it won’t try to start a jail more than once- therefore multiple instances can be running in parallel without fear of starting a jail twice. If a jail startup gets stuck, you can ^C without fear of killing the script. IMPORTANT- before running startalljails you should make sure you ran preboot once as it will clear out all the lockfiles and enable startalljails to work properly.<br />
<br />
== aaccheck.sh ==<br />
aaccheck.sh<br />
displayes the output of container list and task list from aaccli<br />
<br />
== backup ==<br />
backup<br />
backup script called nightly to update jail scripts and do customer backups<br />
<br />
== buildsafe ==<br />
buildsafe<br />
creates safe files based on quads (automatically removing the fsck’s). This will destructively overwrite safe files<br />
<br />
== checkload.pl ==<br />
checkload.pl<br />
this was intended to be setup as a cronjob to watch processes on a jail when the load <br />
rises above a certain level. Not currently in use.<br />
<br />
== checkprio.pl ==<br />
checkprio.pl<br />
will look for any process (other than the current shell’s csh, sh, sshd procs) with a non-normal priority and normalize it<br />
<br />
== diskusagemon == <br />
diskusagemon <mount point> <1k blocks><br />
watches a mount point’s disk use, when it reaches the level specified in the 2nd argument,<br />
it exits. This is useful when doing a restore and you want to be paged as it’s nearing completion.<br />
Best used as: <tt>diskusagemon /asd/asd 1234; pagexxx</tt><br />
<br />
== dumprestore ==<br />
dumprestore <dumpfile><br />
this is a perl expect script which automatically enters ‘1’ and ‘y’. It seems to cause restore to fail<br />
to set owner permissions on large restores.<br />
<br />
== g ==<br />
g <search><br />
greps the quad/safe files for the given search parameter<br />
<br />
== gather.pl ==<br />
gather.pl<br />
gathers up data about jails configured and writes to a file. Used for audits against the db<br />
<br />
== gb ==<br />
gb <search><br />
greps backup.config for the given search parameter<br />
<br />
== gbg ==<br />
gbg <search><br />
greps backup.config for the given search parameter and presents just the directories (for clean pasting)<br />
<br />
== ipfwbackup ==<br />
ipfwbackup<br />
writes ipfw traffic count data to a logfile<br />
<br />
== ipfwreset ==<br />
ipfwreset<br />
writes ipfw traffic count data to a logfile and resets counters to 0<br />
<br />
== js ==<br />
js<br />
output varies by OS version, but generally provides information about the base jail:<br />
- which vn’s are in use<br />
- disk usage<br />
- info about the contents of quads<br />
- the # of inodes represented by the jails contained in the group (133.2 in the example below), and how many jails per data mount, as well as subtotals<br />
- ips bound to the base machine but not in use by a jail<br />
- free gvinum volumes, or unused vn’s or used md’s<br />
<br />
<pre>/usr/local/jail/rc.d/quad1:<br />
/mnt/data1 133.2 (1)<br />
/mnt/data2 1040.5 (7)<br />
total 1173.7 (8)<br />
/usr/local/jail/rc.d/quad2:<br />
/mnt/data1 983.4 (6)<br />
total 983.4 (6)<br />
/usr/local/jail/rc.d/quad3:<br />
/mnt/data1 693.4 (4)<br />
/mnt/data2 371.6 (3)<br />
total 1065 (7)<br />
/usr/local/jail/rc.d/quad4:<br />
/mnt/data1 466.6 (3)<br />
/mnt/data2 882.2 (5)<br />
total 1348.8 (8)<br />
/mnt/data1: 2276.6 (14)<br />
/mnt/data2: 2294.3 (15)<br />
<br />
Available IPs:<br />
69.55.230.11 69.55.230.13 69.55.228.200<br />
<br />
Available volumes:<br />
v78 /mnt/data2 2G<br />
v79 /mnt/data2 2G<br />
v80 /mnt/data2 2G</pre> <br />
<br />
== load.pl ==<br />
load.pl<br />
feeds info to load mrtg - executed by inetd.<br />
<br />
== makevirginjail ==<br />
makevirginjail<br />
Only on some systems, makes an empty jail (doesn't do restore step)<br />
<br />
== mb == <br />
mb <mount|umount><br />
(nfs) mounts and umounts dirs to backup2. Shortcuts are mbm and mbu to mount and unmount. <br />
<br />
== notify.sh ==<br />
notify.sh<br />
emails reboot@johncompanies.com – intended to be called at boot time to alert us to a machine which panics and reboots and isn’t caught by bb or castle.<br />
<br />
== orphanedbackupwatch ==<br />
orphanedbackupwatch<br />
looks for directories on backup2 which aren’t configured in backup.config and offers to delete them<br />
<br />
== postboot ==<br />
postboot<br />
to be run after a machine reboot and quad/safe’s are done executing. It will:<br />
* do chmod 666 on each jail’s /dev/null<br />
* add ipfw counts<br />
* run jailpsall (so you can see if a configured jail isn’t running)<br />
<br />
== preboot ==<br />
preboot<br />
to be run before running quad/safe – checks for misconfigurations: <br />
* a jail configured in a quad but not a safe<br />
* a jail is listed more than once in a quad<br />
* the ip assigned to a jail isn’t configured on the machine<br />
* alias numbering skips in the rc.conf (resulting in the above)<br />
* orphaned vnfile's that aren't mentioned in a quad/safe<br />
* ip mismatches between dir/vnfile name and the jail’s ip<br />
* dir/vnfiles's in quad/safe that don’t exist <br />
<br />
== quadanalyze.pl ==<br />
quadanalyze.pl<br />
called by js, produces the info (seen above with js explanation) about the contents of quad (inode count, # of jails, etc.)<br />
<br />
== rsync.backup ==<br />
rsync.backup<br />
does customer backups (relies on backup.config)<br />
<br />
== taskdone ==<br />
taskdone<br />
when called will email support@johncompanies.com with the hostname of the machine from which it was executed as the subject<br />
<br />
== topten ==<br />
topten<br />
summarizes the top 10 traffic users (called by ipfwreset)<br />
<br />
== trafficgather.pl ==<br />
trafficgather.pl [yy-mm]<br />
sends a traffic usage summary by jail to support@johncomapnies.com and payments@johncompanies.com. Optional arguments are year and month (must be in the past). If not passed, assumes last month. Relies on traffic logs created by ipfwreset and ipfwbackup<br />
<br />
== trafficwatch.pl ==<br />
trafficwatch.pl<br />
checks traffic usage and emails support@johncomapnies.com when a jail reaches the warning level (35G) and the limit (40G). We really aren’t using this anymore now that we have netflow.<br />
<br />
== trafstats ==<br />
trafstats<br />
writes ipfw traffic usage info by jail to a file called jc_traffic_dump in each jail’s / dir<br />
<br />
== truncate_jailmake ==<br />
truncate_jailmake<br />
a version of jailmake which creates truncated vnfiles.<br />
<br />
== vb ==<br />
vb<br />
the equivalent of: <tt>vi /usr/local/jail/bin/backup.config</tt><br />
<br />
== vs (freebsd) ==<br />
vs<n><br />
the equivalent of: <tt>vi /usr/local/jail/rc.d/safe<n></tt><br />
<br />
vq<n><br />
the equivalent of: vi /usr/local/jail/rc.d/quad<n><br />
<br />
== dumpremote ==<br />
dumpremote <user@machine> </remote/location/file-dump> <vnX><br />
ex: dumpremote user@10.1.4.117 /mnt/data3/remote.echoditto.com-dump 7<br />
this will dump a vn filesystem to a remote machine and location<br />
<br />
== oversellcheck ==<br />
oversellcheck<br />
displays how much a disk is oversold or undersold taking into account truncated vn files. Only for use on 4.x systems<br />
<br />
== mvbackups (freebsd) ==<br />
mvbackups <dir> (1.1.1.1-col00001-DIR) <target_machine> (jail1) <target_dir> (data1)<br />
moves backups from one location to another on the backup server, and provides you with option to remove entries from current backup.config, and simple paste command to add the config to backup.config on the target server<br />
<br />
== jailnice ==<br />
jailnice <hostname><br />
applies <tt>renice 19 [PID]</tt> and <tt>rtprio 31 –[PID]</tt> to each process in the given jail<br />
<br />
== dumpremoterestore ==<br />
dumpremoterestore <device> <ip of target machine> <dir on target machine><br />
ex: dumpremoterestore /dev/vn51 10.1.4.118 /mnt/data2/69.55.239.45-col00688-DIR<br />
dumps a device and restores it to a directory on a remote machine. Requires that you enable root ssh on the <br />
remote machine.<br />
<br />
== psj ==<br />
psj<br />
shows just the procs running on the base system – a ps auxw but without jail’d procs present<br />
<br />
== perc5iraidchk ==<br />
perc5iraidchk<br />
checks for degraded arrays on Dell 2950 systems with Perc5/6 controllers<br />
<br />
== perc4eraidchk ==<br />
perc4eraidchk<br />
checks for degraded arrays on Dell 2850 systems with Perc4e/Di controllers<br />
<br />
= Virtuozzo VPS =<br />
<br />
Note: We use VEID (Virtual Environment ID) and CTID (Container ID) interchangably. Similarly, VE and CT. They mean the same thing.<br />
VZPP = VirtuoZzo Power Panel (the control panel for each CT)<br />
<br />
All linux systems exist in /vz, /vz1 or /vz2 - since each linux machine holds roughly 60-90 customers, there will be roughly 30-45 in each partition.<br />
<br />
The actual filesystem of the system in question is in:<br />
<br />
/vz(1-2)/private/(VEID)<br />
<br />
Where VEID is the identifier for that system - an all-numeric string larger than 100.<br />
<br />
The actual mounted and running systems are in the corresponding:<br />
<br />
/vz(1-2)/root/(VEID)<br />
<br />
But we rarely interact with any system from this mount point.<br />
<br />
You should never need to touch the root portion of their system – however you can traverse their filesystem by going to <tt>/vz(1-2)/private/(VEID)/root</tt> (<tt>/vz(1-2)/private/(VEID)/fs/root</tt> on 4.x systems) the root of their filesystem is in that directory, and their entire system is underneath that.<br />
<br />
Every VE has a startup script in <tt>/etc/sysconfig/vz-scripts</tt> (which is symlinked as <tt>/vzconf</tt> on all systems) - the VE startup script is simply named <tt>(VEID).conf</tt> - it contains all the system parameters for that VE:<br />
<br />
<pre># Configuration file generated by vzsplit for 60 VE<br />
# on HN with total amount of physical mem 2011 Mb<br />
<br />
VERSION="2"<br />
CLASSID="2"<br />
<br />
ONBOOT="yes"<br />
<br />
KMEMSIZE="8100000:8200000"<br />
LOCKEDPAGES="322:322"<br />
PRIVVMPAGES="610000:615000"<br />
SHMPAGES="33000:34500"<br />
NUMPROC="410:415"<br />
PHYSPAGES="0:2147483647"<br />
VMGUARPAGES="13019:2147483647"<br />
OOMGUARPAGES="13019:2147483647"<br />
NUMTCPSOCK="1210:1215"<br />
NUMFLOCK="107:117"<br />
NUMPTY="19:19"<br />
NUMSIGINFO="274:274"<br />
TCPSNDBUF="1800000:1900000"<br />
TCPRCVBUF="1800000:1900000"<br />
OTHERSOCKBUF="900000:950000"<br />
DGRAMRCVBUF="200000:200000"<br />
NUMOTHERSOCK="650:660"<br />
DCACHE="786432:818029"<br />
NUMFILE="7500:7600"<br />
AVNUMPROC="51:51"<br />
IPTENTRIES="155:155"<br />
DISKSPACE="4194304:4613734"<br />
DISKINODES="400000:420000"<br />
CPUUNITS="1412"<br />
QUOTAUGIDLIMIT="2000"<br />
VE_ROOT="/vz1/root/636"<br />
VE_PRIVATE="/vz1/private/636"<br />
NAMESERVER="69.55.225.225 69.55.230.3"<br />
OSTEMPLATE="vzredhat-7.3/20030305"<br />
VE_TYPE="regular"<br />
IP_ADDRESS="69.55.225.229"<br />
HOSTNAME="textengine.net"</pre><br />
<br />
<br />
As you can see, the hostname is set here, the disk space is set here, the number of inodes, the number of files that can be open, the number of tcp sockets, etc. - all are set here.<br />
<br />
In fact, everything that can be set on this customer system is set in this conf file.<br />
<br />
<br />
All interaction with the customer system is done with the VEID. You start the system by running:<br />
<br />
vzctl start 999<br />
<br />
You stop it by running:<br />
<br />
vzctl stop 999<br />
<br />
You execute commands in it by running:<br />
<br />
vzctl exec 999 df -k<br />
<br />
You enter into it, via a root-shell backdoor with:<br />
<br />
vzctl enter 999<br />
<br />
and you set parameters for the system, while it is still running, with:<br />
<br />
vzctl set 999 --diskspace 6100000:6200000 --save<br />
<br />
<tt>vzctl</tt> is the most commonly used command - we have aliased <tt>v</tt> to <tt>vzctl</tt> since we use it so often. We’ll continue to use <tt>vzctl</tt> in our examples, but feel free to use just <tt>v</tt>.<br />
<br />
Let's say the user wants more diskspace. You can cat their conf file and see:<br />
<br />
DISKSPACE="4194304:4613734"<br />
<br />
So right now they have 4gigs of space. You can then change it to 6 with:<br />
<br />
vzctl set 999 --diskspace 6100000:6200000 --save<br />
<br />
IMPORTANT: all issuances of the vzctl set command need to end with <tt>–save</tt> - if they don't, the setting will be set, but it will not be saved to the conf file, and they will not have those settings next time they boot.<br />
<br />
All of the tunables in the conf file can be set with the vzctl set command. Note that in the conf file, and on the vzctl set command line, we always issue two numbers seperated by a colon - that is because we are setting the hard and soft limits. Always set the hard limit slightly above the soft limit, as you see it is in the conf file for all those settings.<br />
<br />
There are also things you can set with `<tt>vzctl set</tt>` that are not in the conf file as settings, per se. For instance, you can add IPs:<br />
<br />
vzctl set 999 --ipadd 10.10.10.10 --save<br />
<br />
or multiple IPs:<br />
<br />
vzctl set 999 --ipadd 10.10.10.10 --ipadd 10.10.20.30 --save<br />
<br />
or change the hostname:<br />
<br />
vzctl set 999 --hostname www.example.com --save<br />
<br />
You can even set the nameservers:<br />
<br />
vzctl set 999 --nameserver 198.78.66.4 --nameserver 198.78.70.180 --save<br />
<br />
Although you probably will never do that.<br />
<br />
You can disable a VPS from being started (by VZPP or reboot) (4.x):<br />
<br />
vzctl set 999 --disabled yes --save <br />
<br />
You can disable a VPS from being started (by VZPP or reboot) (<=3.x):<br />
<br />
vzctl set 999 --onboot=no --save <br />
<br />
You can disable a VPS from using his control panel:<br />
<br />
vzctl set 999 --offline_management=no --save <br />
<br />
You can suspend a VPS, so it can be resumed in the same state it was in when it was stopped (4.x):<br />
<br />
vzctl suspend 999<br />
<br />
and to resume it:<br />
<br />
vzctl resume 999<br />
<br />
to see who owns process:<br />
vzpid <PID><br />
<br />
to mount up an unmounted ve:<br />
vzctl mount 827<br />
<br />
<br />
To see network stats for CT's:<br />
<pre># vznetstat<br />
VEID Net.Class Output(bytes) Input(bytes)<br />
-----------------------------------------------<br />
24218 1 484M 39M<br />
24245 1 463M 143M<br />
2451 1 2224M 265M<br />
2454 1 2616M 385M<br />
4149 1 125M 68M<br />
418 1 1560M 34M<br />
472 1 1219M 315M<br />
726 1 628M 317M<br />
763 1 223M 82M<br />
771 1 4234M 437M<br />
-----------------------------------------------<br />
Total: 13780M 2090M</pre><br />
<br />
<br />
One thing that sometimes comes up on older systems that we created with smaller defaults is that the system would run out of inodes. The user will email and say they cannot create any more files or grow any files larger, but they will also say that they are not out of diskspace ... they are running:<br />
<br />
df -k<br />
<br />
and seeing how much space is free - and they are not out of space. They are most likely out of inodes - which they would see by running:<br />
<br />
df -i<br />
<br />
So, the first thing you should do is enter their system with:<br />
<br />
vzctl enter 999<br />
<br />
and run: <tt>df -i</tt><br />
<br />
to confirm your theory. Then exit their system. Then simply cat their conf file and see what their inodes are set to (probably 200000:200000, since that was the old default on the older systems) and run:<br />
<br />
vzctl set 999 --diskinodes 400000:400000 --save<br />
<br />
If they are not out of inodes, then a good possibility is that they have maxed out their numfile configuration variable, which controls how many files they can have in their system. The current default is 7500 (which nobody has ever hit), but the old default was as low as 2000, so you would run something like:<br />
<br />
vzctl set 999 --numfile 7500:7500 --save<br />
<br />
----<br />
<br />
You cannot start or stop a VE if your pwd is its private (/vz/private/999) or root (/vz/root/999) directories, or anywhere below them.<br />
<br />
<br />
== Recovering from a crash (linux) ==<br />
<br />
=== Diagnose whether you have a crash ===<br />
The most important thing is to get the machine and all ve’s back up as soon as possible. Note the time, you’ll need to create a crash log entry (Mgmt. -> Reference -> CrashLog). The first thing to do is head over to the [[Screen#Screen_Organization|serial console screen]] and see if there’s any kernel error messages output. Try to copy any messages (or just a sample of repeating messages) you see into the notes section of the crash log – these will also likely need to be sent to virtuozzo for interpretation. If the messages are spewing too fast, hit ^O + H to start a screen log dump which you can ob1182.pts-38.bb serve after the machine is rebooted. Additionally, if the machine is responsive, you can get a trace to send to virtuozzo by hooking up a kvm and entering these 3 sequences:<br />
<pre>alt+print screen+m<br />
alt+print screen+p<br />
alt+print screen+t</pre><br />
<br />
If there are no messages, the machine may just be really busy- wait a bit (5-10min) to see if it comes back. If it's still pinging, odds are its very busy. If it doesn't come back, or the messages indicate a fatal error, you will need to proceed with a power cycle (ctrl+alt+del will not work).<br />
<br />
=== Power cycle the server ===<br />
If this machine is not a Dell 2950 with a [[DRAC/RMM#DRAC|DRAC card]] (i.e. if you can’t ssh into the DRAC card and issue racadm serveraction hardreset, then you will need someone at the data center to power the macine off, wait 30 sec, then turn it back on. Make sure to re-attach via console (<tt>tip virtxx</tt>) immediately after power down. <br />
<br />
=== (Re)attach to the console ===<br />
Stay on the console the entire time during boot. As the BIOS posts- look out for the RAID card output- does everything look healthy? The output may be scrambled, look for "DEGRADED" or "FAILED". Once the OS starts booting you will be disconnected (dropped back to the shell on the console server) a couple times during the boot up. The reason you want to quickly re-attach is two-fold: 1. If you don’t reattach quickly then you won’t get any console output, 2. you want to be attached before the server ''potentially'' starts (an extensive) fsck. If you attach after the fsck begins, you’ll have seen no indication it started an fsck and the server will appear frozen during startup- no output, no response. <br />
<br />
=== Start containers/VE's/VPSs ===<br />
When the machine begins to start VE’s, it’s safe to leave the console and login via ssh. All virts should be set to auto start all the VEs after a crash. Further, most (newer) virts are set to “fastboot” it’s VE’s (to find out, do:<br />
grep -i fast /etc/sysconfig/vz <br />
and look for <tt>VZFASTBOOT=yes</tt>). If this was set prior to the machine’s crash (setting it after the machine boots will not have any effect until the vz service is restarted) it will start each ve as fast as possible, in serial, then go thru each VE (serially), shutting it down running a vzquota (disk usage) check, then bringing it back up. The benefit is that all VE’s are brought up quickly (within 15min or so depending on the #), the downside is a customer watching closely will notice 2 outages – 1st the machine crash, 2nd their quota check (which will be a much shorter downtime- on the order of a few minutes). <br />
<br />
Where “fastboot” is not set to yes (i.e on quar1), vz will start them consecutively, checking the quotas one at a time, and the 60th VE may not start until an hour or two later - this is not acceptable.<br />
<br />
The good news is, if you run vzctl start for a VE that is already started, you will simply get an error: <tt>VE is already started</tt>. Further, if you attempt to vzctl start a VE that is in the process of being started, you will simply get an error: unable to lock VE. So, there is no danger in simply running scripts to start smaller sets of VEs. If the system is not autostarting, then there is no issue, and even if it does, when it conflicts, one process (yours or the autostart) will lose, and just move on to the next one.<br />
<br />
A script has been written to assist with ve starts: [[#startvirt.pl|startvirt.pl]] which will start 6 ve’s at once until there are no more left. If startvirt.pl is used on a system where “fastboot” was on, it will circumvent the fastboot for ve’s started by startvirt.pl – they will go through the complete quota check before starting- therefore this is not advisable when a system has crashed. When a system is booted cleanly, and there's no need for vzquota checks, then startvirt.pl is safe and advisable to run.<br />
<br />
=== Make sure all containers are running ===<br />
You can quickly get a feel for how many ve’s are started by running:<br />
<br />
<pre>[root@virt4 log]# vs<br />
VEID 16066 exist mounted running<br />
VEID 16067 exist mounted running<br />
VEID 4102 exist mounted running<br />
VEID 4112 exist mounted running<br />
VEID 4116 exist mounted running<br />
VEID 4122 exist mounted running<br />
VEID 4123 exist mounted running<br />
VEID 4124 exist mounted running<br />
VEID 4132 exist mounted running<br />
VEID 4148 exist mounted running<br />
VEID 4151 exist mounted running<br />
VEID 4155 exist mounted running<br />
VEID 42 exist mounted running<br />
VEID 432 exist mounted running<br />
VEID 434 exist mounted running<br />
VEID 442 exist mounted running<br />
VEID 450 exist mounted running<br />
VEID 452 exist mounted running<br />
VEID 453 exist mounted running<br />
VEID 454 exist mounted running<br />
VEID 462 exist mounted running<br />
VEID 463 exist mounted running<br />
VEID 464 exist mounted running<br />
VEID 465 exist mounted running<br />
VEID 477 exist mounted running<br />
VEID 484 exist mounted running<br />
VEID 486 exist mounted running<br />
VEID 490 exist mounted running</pre><br />
<br />
So to see how many ve’s have started:<br />
<pre>[root@virt11 root]# vs | grep running | wc -l<br />
39</pre><br />
<br />
And to see how many haven’t:<br />
<pre>[root@virt11 root]# vs | grep down | wc -l<br />
0</pre><br />
<br />
And how many we should have running:<br />
<pre>[root@virt11 root]# vs | wc -l<br />
39</pre><br />
<br />
Another tool you can use to see which ve’s have started, among other things is [[#vzstat|vzstat]]. It will give you CPU, memory, and other stats on each ve and the overall system. It’s a good thing to watch as ve’s are starting (note the VENum parameter, it will tell you how many have started):<br />
<br />
<pre>4:37pm, up 3 days, 5:31, 1 user, load average: 1.57, 1.68, 1.79<br />
VENum 40, procs 1705: running 2, sleeping 1694, unint 0, zombie 9, stopped 0<br />
CPU [ OK ]: VEs 57%, VE0 0%, user 8%, sys 7%, idle 85%, lat(ms) 412/2<br />
Mem [ OK ]: total 6057MB, free 9MB/54MB (low/high), lat(ms) 0/0<br />
Swap [ OK ]: tot 6142MB, free 4953MB, in 0.000MB/s, out 0.000MB/s<br />
Net [ OK ]: tot: in 0.043MB/s 402pkt/s, out 0.382MB/s 4116pkt/s<br />
Disks [ OK ]: in 0.002MB/s, out 0.000MB/s<br />
<br />
VEID ST %VM %KM PROC CPU SOCK FCNT MLAT IP<br />
1 OK 1.0/17 0.0/0.4 0/32/256 0.0/0.5 39/1256 0 9 69.55.227.152<br />
21 OK 1.3/39 0.1/0.2 0/46/410 0.2/2.8 23/1860 0 6 69.55.239.60<br />
133 OK 3.1/39 0.1/0.3 1/34/410 6.3/2.8 98/1860 0 0 69.55.227.147<br />
263 OK 2.3/39 0.1/0.2 0/56/410 0.3/2.8 34/1860 0 1 69.55.237.74<br />
456 OK 17/39 0.1/0.2 0/100/410 0.1/2.8 48/1860 0 11 69.55.236.65<br />
476 OK 0.6/39 0.0/0.2 0/33/410 0.1/2.8 96/1860 0 10 69.55.227.151<br />
524 OK 1.8/39 0.1/0.2 0/33/410 0.0/2.8 28/1860 0 0 69.55.227.153<br />
594 OK 3.1/39 0.1/0.2 0/45/410 0.0/2.8 87/1860 0 1 69.55.239.40<br />
670 OK 7.7/39 0.2/0.3 0/98/410 0.0/2.8 64/1860 0 216 69.55.225.136<br />
691 OK 2.0/39 0.1/0.2 0/31/410 0.0/0.7 25/1860 0 1 69.55.234.96<br />
744 OK 0.1/17 0.0/0.5 0/10/410 0.0/0.7 7/1860 0 6 69.55.224.253<br />
755 OK 1.1/39 0.0/0.2 0/27/410 0.0/2.8 33/1860 0 0 192.168.1.4<br />
835 OK 1.1/39 0.0/0.2 0/19/410 0.0/2.8 5/1860 0 0 69.55.227.134<br />
856 OK 0.3/39 0.0/0.2 0/13/410 0.0/2.8 16/1860 0 0 69.55.227.137<br />
936 OK 3.2/52 0.2/0.4 0/75/410 0.2/0.7 69/1910 0 8 69.55.224.181<br />
1020 OK 3.9/39 0.1/0.2 0/60/410 0.1/0.7 55/1860 0 8 69.55.227.52<br />
1027 OK 0.3/39 0.0/0.2 0/14/410 0.0/2.8 17/1860 0 0 69.55.227.83<br />
1029 OK 1.9/39 0.1/0.2 0/48/410 0.2/2.8 25/1860 0 5 69.55.227.85<br />
1032 OK 12/39 0.1/0.4 0/80/410 0.0/2.8 41/1860 0 8 69.55.227.90</pre><br />
<br />
When you are all done, you will want to make sure that all the VEs really did get started, run vs one more time.<br />
<br />
Note the time all ve’s are back up and enter that into and save the crash log entry.<br />
<br />
Occasionally, a ve will not start automatically. The most common reason for a ve not to come up normally is the ve was at it’s disk limit before the crash, and will not start since they’re over the limit. To overcome this, set the disk space to current usage level (the system will give this to you when it fails to start), start the ve, then re-set the disk space back to the prior level. Lastly, contact the customer to let them know they’re out of disk (or allocate more disk if they're entitled to more).<br />
<br />
== Hitting performance barriers and fixing them ==<br />
<br />
There are multiple modes virtuozzo offers to allocate resources to a ve. We utilize 2: SLM and UBC parameters<br />
On our 4.x systems, we use all SLM – it’s simpler to manage and understand. There are a few systems on virt19/18 that may also use SLM. Everything else uses UBC. <br />
You can tell a SLM ve by:<br />
<br />
SLMMODE="all"<br />
<br />
in their conf file. <br />
<br />
TODO: detail SLM modes and parameters.<br />
<br />
If someone is in SLM mode and they hit memory resource limits, they simply need to upgrade to more memory.<br />
<br />
The following applies to everyone else (UBC).<br />
<br />
Customers will often email and say that they are getting out of memory errors - a common one is "cannot fork" ... basically, anytime you see something odd like this, it means they are hitting one of their limits that is in place in their conf file.<br />
<br />
The conf file, however, simply shows their limits - how do we know what they are currently at ?<br />
<br />
The answer is a file called v - this file contains the current status (and peaks) of their performance settings, and also counts how many times they have hit the barrier. The output of the file looks like this:<br />
<br />
<pre>764: kmemsize 384113 898185 8100000 8200000 0<br />
lockedpages 0 0 322 322 0<br />
privvmpages 1292 7108 610000 615000 0<br />
shmpages 270 528 33000 34500 0<br />
dummy 0 0 0 0 0<br />
numproc 8 23 410 415 0<br />
physpages 48 5624 0 2147483647 0<br />
vmguarpages 0 0 13019 2147483647 0<br />
oomguarpages 641 6389 13019 2147483647 0<br />
numtcpsock 3 21 1210 1215 0<br />
numflock 1 3 107 117 0<br />
numpty 0 2 19 19 0<br />
numsiginfo 0 4 274 274 0<br />
tcpsndbuf 0 80928 1800000 1900000 0 <br />
tcprcvbuf 0 108976 1800000 1900000 0<br />
othersockbuf 2224 37568 900000 950000 0<br />
dgramrcvbuf 0 4272 200000 200000 0<br />
numothersock 3 9 650 660 0<br />
dcachesize 53922 100320 786432 818029 0<br />
numfile 161 382 7500 7600 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
numiptent 4 4 155 155 0</pre><br />
<br />
The first column is the name of the counter in question - the same names we saw in the systems conf file. The second column is the _current_ value of that counter, the third column is the max that that counter has ever risen to, the fourth column is the soft limit, and the fifth column is the hard limit (which is the same as the numbers in that systems conf file).<br />
<br />
The sixth number is the failcount - how many times the current usage has risen to hit the barrier. It will increase as soon as the current usage hits the soft limit.<br />
<br />
The problem with /proc/user_beancounters is that it actually contains that set of data for every running VE - so you can't just cat /proc/user_beancounters - it is too long and you get info for every other running system.<br />
<br />
You can vzctl enter the system and run:<br />
<br />
vzctl enter 9999<br />
cat /proc/user_beancounters<br />
<br />
inside their system, and you will just see the stats for their particular system, but entering their system every time you want to see it is combersome.<br />
<br />
So, I wrote a simple script called "vzs" which simply greps for the VEID, and spits out the next 20 or so lines (however many lines there are in the output, I forget) after it. For instance:<br />
<br />
<pre># vzs 765:<br />
765: kmemsize 2007936 2562780 8100000 8200000 0<br />
lockedpages 0 8 322 322 0<br />
privvmpages 26925 71126 610000 615000 0<br />
shmpages 16654 16750 33000 34500 0<br />
dummy 0 0 0 0 0<br />
numproc 41 57 410 415 0<br />
physpages 1794 49160 0 2147483647 0<br />
vmguarpages 0 0 13019 2147483647 0<br />
oomguarpages 4780 51270 13019 2147483647 0<br />
numtcpsock 23 37 1210 1215 0<br />
numflock 17 39 107 117 0<br />
numpty 1 3 19 19 0<br />
numsiginfo 0 6 274 274 0<br />
tcpsndbuf 22240 333600 1800000 1900000 0<br />
tcprcvbuf 0 222656 1800000 1900000 0<br />
othersockbuf 104528 414944 900000 950000 0<br />
dgramrcvbuf 0 4448 200000 200000 0<br />
numothersock 73 105 650 660 0<br />
dcachesize 247038 309111 786432 818029 0<br />
numfile 904 1231 7500 7600 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
numiptent 4 4 155 155 0</pre><br />
<br />
That showed us just the portion of /proc/user_beancounters for system 765.<br />
<br />
When you run the vzs command, always add a : after the VEID.<br />
<br />
So, if a customer complains about some out of memory errors, or no more files, or no more ptys, or just has an unspecific complain about processes dying, etc., the very first thing you need to do is check their beancounters with vzs. Usually you will spot an item that has a high failcount and needs to be upped.<br />
<br />
At that point you could simply up the counter with `vzctl set`. Generally pick a number 10-20% higher than the old one, and make the hard limit slightly larger than the the soft limit. However our systems now come in several levels and those levels have more/different memory allocations. If someone is complaining about something other than a memory limit (pty, numiptent, numflock), it’s generally safe to increase it, at least to the same level as what’s in the /vzconf/4unlimited file on the newest virt. If someone is hitting a memory limit, first make sure they are given what they deserve:<br />
<br />
(refer to mgmt -> payments -> packages)<br />
<br />
To set those levels, you use the [[#setmem|setmem]] command. <br />
<br />
The alternate (DEPRECATED) method would be to use one of 3 commands:<br />
256 <veid><br />
300 <veid><br />
384 <veid><br />
512 <veid><br />
<br />
If the levels were not right (you’d run vzs <veid> before and after to see the effect) tell the customer they’ve been adjusted and be done with it. If the levels were right, tell the customer they must upgrade to a higher package, tell them how to see level (control panel) and that they can reboot their system to escape this lockup contidion.<br />
<br />
Customers can also complain that their site is totally unreachable, or complain that it is down ... if the underlying machine is up, and all seems well, you may notice in the beancounters that network-specific counters are failing - such as numtcpsock, tcpsndbuf or tcprcvbuf. This will keep them from talking on the network and make it seem like their system is down. Again, just up the limits and things should be fine.<br />
<br />
On virts 1-4, you should first look at the default settings for that item on a later virt, such as virt 8 - we have increased the defaults a lot since the early machines. So, if you are going to up a counter on virt2, instead of upping it by 10-20%, instead up it to the new default that you see on virt8.<br />
<br />
== Moving a VE to another virt (migrate/migrateonline) ==<br />
<br />
This will take a while to complete - and it is best to do this at night when the load is light on both machines.<br />
<br />
There are different methods for this, depending on which version of virtuozzo is installed on the src. and dst. virt. <br />
To check which version is running: <br />
[root@virt12 private]# cat /etc/virtuozzo-release<br />
Virtuozzo release 2.6.0<br />
<br />
Ok, let's say that the VE is 1212, and vital stats are:<br />
<pre>[root@virt12 sbin]# vc 1212<br />
VE_ROOT="/vz1/root/1212"<br />
VE_PRIVATE="/vz1/private/1212"<br />
OSTEMPLATE="fedora-core-2/20040903"<br />
IP_ADDRESS="69.55.229.84"<br />
TEMPLATES="devel-fc2/20040903 php-fc2/20040813 mysql-fc2/20040812 postgresql-fc2/20040813 mod_perl-fc2/20040812 mod_ssl-fc2/20040811 jre-fc2/20040823 jdk-fc2/20040823 mailman-fc2/20040823 analog-fc2/20040824 proftpd-fc2/20040818 tomcat-fc2/20040823 usermin-fc2/20040909 webmin-fc2/20040909 uw-imap-fc2/20040830 phpBB-fc2/20040831 spamassassin-fc2/20040910 PostNuke-fc2/20040824 sl-webalizer-fc2/20040<br />
818"<br />
<br />
[root@virt12 sbin]# vzctl exec 1212 df -h<br />
Filesystem Size Used Avail Use% Mounted on<br />
/dev/vzfs 4.0G 405M 3.7G 10% /</pre><br />
<br />
From this you can see that he’s using (and will minimally need free on the dst server) ~400MB, and he’s running on a Fedora 2 template, version 20040903. He’s also got a bunch of other templates installed. It’s is '''vital''' that '''all''' these templates exist on the dst system. To confirm that, on the dst system run:<br />
<br />
For < 3.0:<br />
<br />
<pre>[root@virt14 private]# vzpkgls | grep fc2<br />
devel-fc2 20040903<br />
PostNuke-fc2 20040824<br />
analog-fc2 20040824<br />
awstats-fc2 20040824<br />
bbClone-fc2 20040824<br />
jdk-fc2 20040823<br />
jre-fc2 20040823<br />
mailman-fc2 20040823<br />
mod_frontpage-fc2 20040816<br />
mod_perl-fc2 20040812<br />
mod_ssl-fc2 20040811<br />
mysql-fc2 20040812<br />
openwebmail-fc2 20040817<br />
php-fc2 20040813<br />
phpBB-fc2 20040831<br />
postgresql-fc2 20040813<br />
proftpd-fc2 20040818<br />
sl-webalizer-fc2 20040818<br />
spamassassin-fc2 20040910<br />
tomcat-fc2 20040823<br />
usermin-fc2 20040909<br />
uw-imap-fc2 20040830<br />
webmin-fc2 20040909<br />
[root@virt14 private]# vzpkgls | grep fedora<br />
fedora-core-1 20040121 20040818<br />
fedora-core-devel-1 20040121 20040818<br />
fedora-core-2 20040903<br />
[root@virt14 private]#</pre><br />
<br />
For these older systems, you can simply match up the date on the template. <br />
<br />
For >= 3.0:<br />
<br />
<pre>[root@virt19 /vz2/private]# vzpkg list<br />
centos-5-x86 2008-01-07 22:05:57<br />
centos-5-x86 devel<br />
centos-5-x86 jre<br />
centos-5-x86 jsdk<br />
centos-5-x86 mod_perl<br />
centos-5-x86 mod_ssl<br />
centos-5-x86 mysql<br />
centos-5-x86 php<br />
centos-5-x86 plesk9<br />
centos-5-x86 plesk9-antivirus<br />
centos-5-x86 plesk9-api<br />
centos-5-x86 plesk9-atmail<br />
centos-5-x86 plesk9-backup<br />
centos-5-x86 plesk9-horde<br />
centos-5-x86 plesk9-mailman<br />
centos-5-x86 plesk9-mod-bw<br />
centos-5-x86 plesk9-postfix<br />
centos-5-x86 plesk9-ppwse<br />
centos-5-x86 plesk9-psa-firewall<br />
centos-5-x86 plesk9-psa-vpn<br />
centos-5-x86 plesk9-psa-fileserver<br />
centos-5-x86 plesk9-qmail<br />
centos-5-x86 plesk9-sb-publish<br />
centos-5-x86 plesk9-vault<br />
centos-5-x86 plesk9-vault-most-popular<br />
centos-5-x86 plesk9-watchdog</pre><br />
<br />
On these newer systems, it's difficult to tell whether the template on the dst matches exactly the src. Just cause a centos-5-x86 is listed on both servers doesn't mean all the same packages are there on the dst. To truly know, you must perform a sample rsync:<br />
<br />
rsync -avn /vz/template/centos/5/x86/ root@10.1.4.61:/vz/template/centos/5/x86/<br />
<br />
if you see a ton of output from the dry run command, then clearly there are some differences. You may opt to let the rsync complete (without running in dry run mode) the only downside is you've now used up more space on the dst and also the centos template will be a mess with old and new data- it will be difficult if not impossible to undo (if someday we wanted to reclaim the space).<br />
<br />
If you choose to merge templates, you should closely inspect the dry run output. You should also take care to exclude anything in the /config directory. For example:<br />
<br />
rsync -av -e ssh --stats --exclude=x86/config /vz/template/ubuntu/10.04/ root@10.1.4.62:/vz/template/ubuntu/10.04/<br />
<br />
Which will avoid this directory and contents:<br />
<pre>[root@virt11 /vz2/private]# ls /vz/template/ubuntu/10.04/x86/config*<br />
app os</pre><br />
<br />
This is important to avoid since the config may differ on the destination and we are really only interested in making sure the pacakges are there, not overwriting a newer config with an older one.<br />
<br />
If the dst system was missing a template, you have 2 choices: <br />
# put the missing template on the dst system. 2 choices here: <br />
## Install the template from rpm (found under backup2: /mnt/data4/vzrpms/distro/) or <br />
## rsync over the template (found under /vz/template) - see above<br />
# put the ve on a system which has all the proper templates<br />
<br />
=== pre-seeding a migration ===<br />
<br />
When migrating a customer (or when doing many) depending on how much data you have to transfer, it can take some time. Further, it can be difficult to gauge when a migration will complete or how long it will take. To help speed up the process and get a better idea about how long it will take you can pre-transfer a customer's data to the destination server. If done correctly, vzmigrate will see the pre-transferred data and pick up where you left off, having much less to transfer (just changed/new files). <br />
<br />
We believe vzmigrate uses rsync to do it's transfer. Therefore not only can you use rsync to do a pre-seed, you can also run rsync to see what is causing a repeatedly-failing vzmigrate to fail. <br />
<br />
There's no magic to a pre-seed, you just need to make sure it's named correctly.<br />
<br />
Given:<br />
<br />
source: /vz1/private/1234<br />
<br />
and you want to migrate to /vz2 on the target system, your rsync would look like:<br />
<br />
rsync -av /vz1/private/1234/ root@x.x.x.x:/vz2/private/1234.migrated/<br />
<br />
After running that successful rsync, the ensuing migrateonline (or migrate) will take much less time to complete- depending on the # of files to be analyzed and the # of changed files. In any case, it'll be much much faster than had you just started the migration from scratch.<br />
<br />
Further, as we discuss elsewhere in this topic, a failed migration can be moved from <tt>/vz/private/1234</tt> to <tt>/vz/private/1234.migrated</tt> on the destination if you want to restart a failed migration. This should '''only''' be done if the migration failed and the CT is not running on the destination HN.<br />
<br />
=== migrateonline intructions: src >=3.x -> dst>=3.x ===<br />
<br />
A script called [[#migrateonline|migrateonline]] was written to handle this kind of move. It is basically a wrapper for <tt>vzmigrate</tt> – vzmigrate is a util to seamlessly- as no no reboot of the ve necessary- move a ve from one host to another. This wrapper was initially written cause vz virtuozzo version 2.6.0 has a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables, causing problems when the ve was started up on the dst system. [[#migrate|migrate]] mitigates that. Since it makes multiple ssh connections to the dst virt, it’s a good idea to put the pub key for the src virt in the authorized_keys file on the dst virt. In addition, migrateonline emails ve owners when their migration starts and stops. For this to happen they need to put email addresses (on a single line, space delimited) in a file on their system: /migrate_notify. If the optional target dir is not specified, the ve will be moved to the same private/root location as it was on the src virt. Note: <tt>migrate</tt> is equivalent to <tt>migrateonline</tt>, but will <tt>migrate</tt> a ve AND restart it in the process.<br />
<br />
<pre>[root@virt12 sbin]# migrateonline<br />
usage: /usr/local/sbin/migrateonline <ip of node migrating to> <veid> [target dir: vz | vz1 | vz2]<br />
[root@virt12 sbin]# migrateonline 10.1.4.64 1212 vz<br />
starting to migrate 1212 at Sat Mar 26 22:40:38 PST 2005<br />
Turning off offline_management<br />
Saved parameters for VE 1212<br />
migrating with no start on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
clearing the arp cache<br />
now going to 10.1.4.64 and clear cache and starting<br />
starting it<br />
Delete port redirection<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
finished migrating 1212 at Sat Mar 26 22 22:52:01 PST 2005<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine.<br />
<br />
If they had backups, use the mvbackups command to move their backups to the new server:<br />
<br />
mvbackups 1212 virt14 vz<br />
<br />
Rename the ve<br />
<br />
[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/migrated-1212<br />
<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/old-1212-migrated-20120404-noarchive<br />
<br />
Update the customer’s systems in mgmt to reflect the new path and server.<br />
<br />
<br />
IF migrateonline does not work, you can try again using simply migrate- this will result in a brief reboot for the ve.<br />
Before you try again, make sure of a few things:<br />
<br />
Depending on where in the migration died, there may be partial data on the dst system in 1 of 2 places:<br />
(given the example above)<br />
<br />
/vz/private/1212<br />
<br />
or <br />
<br />
/vz/private/1212.migrated<br />
<br />
before you run migrate again, you'll want to rename so that all data is in <br />
1212.migrated:<br />
<br />
mv /vz/private/1212 /vz/private/1212.migrated<br />
<br />
this way, it will pick up where it left off and transfer only new files.<br />
<br />
Likewise, if you want to speed up a migration, you can pre-seed the dst as follows:<br />
<br />
[root@virt12 sbin]# rsync -avSH /vz/private/1212/ root@10.1.4.64:/vz/private/1212.migrated/<br />
<br />
then when you run migrate or migrateonline, it will only need to move the changed files- the migration will complete quickly<br />
<br />
=== migrate manually (if migrate/online fails) ===<br />
<br />
Lets say for whatever reason the migration fails. You can always move someone manually:<br />
<br />
1. stop ve: <br><br />
v stop 1234<br />
<br />
2. copy over data<br><br />
rsync -avSH /vz/private/1234/ root@1.1.1.1:/vzX/private/1234/<br />
<br />
NOTE: if you've previously seeded the data (run rsync while the VE was up/running), and this is a subsequent rsync, make sure the last rsync you do (while the VE is not running, has the --delete option in the rsync)<br />
<br />
3. copy over conf<br><br />
scp /vzconf/1234.conf root@1.1.1.1:/vzconf<br />
<br />
4. on dst, edit the conf to reflect the right vzX dir<br><br />
vi /vzconf/1234.conf<br />
<br />
5. on src remove the IPs<br><br />
ipdel 1234 2.2.2.2 3.3.3.3<br />
<br />
6. on dst add IPs <br><br />
ipadd 1234 2.2.2.2 3.3.3.3<br />
<br />
7. on dst, start ve: <br><br />
v start 1324<br />
<br />
8. cancel, then archive ve on src per above instrs.<br />
<br />
=== migrate src=2.6.0 -> dst>=2.6.0, or mass-migration with customer notify ===<br />
<br />
A script called <tt>migrate</tt> was written to handle this kind of move. It is basically a wrapper for vzmigrate – vzmigrate is a util to seamlessly move a ve from one host to another. This wrapper was initially written cause vz virtuozzo version 2.6.0 has a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables, causing problems when the ve was started up on the dst system. migrate mitigates that. Since it makes multiple ssh connections to the dst virt, it’s a good idea to put the pub key for the src virt in the authorized_keys file on the dst virt. In addition, migrate emails ve owners when their migration starts and stops. For this to happen they need to put email addresses (on a single line, space delimited) in a file on their system: /migrate_notify. If the optional target dir is not specified, the ve will be moved to the same private/root location as it was on the src virt. Note: migrateonline is equivalent to migrate, but will migrate a ve from one 2.6 '''kernel''' machine to another 2.6 kernel machine without restarting the ve.<br />
<br />
<pre>[root@virt12 sbin]# migrate<br />
usage: /usr/local/sbin/migrate <ip of node migrating to> <veid> [target dir: vz | vz1 | vz2]<br />
[root@virt12 sbin]# migrate 10.1.4.64 1212 vz<br />
starting to migrate 1212 at Sat Mar 26 22:40:38 PST 2005<br />
Turning off offline_management<br />
Saved parameters for VE 1212<br />
migrating with no start on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
clearing the arp cache<br />
now going to 10.1.4.64 and clear cache and starting<br />
starting it<br />
Delete port redirection<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
finished migrating 1212 at Sat Mar 26 22 22:52:01 PST 2005<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine (backup2 or mail).<br />
Cancel the ve (first we have to rename things which migrate changed so cancelve will find them):<br />
<br />
[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/1212.conf<br />
<br />
On 2.6.1 you’ll also have to move the private area:<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/1212<br />
<br />
<pre>[root@virt12 sbin]# cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, [[#cancelve|cancelve]] would offer to remove them. You want to say '''no''' to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
To move everyone off a system, you’d do:<br />
for f in `vl`; do migrate <ip> $f; done<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
=== vzmigrate: src=2.6.1 -> dst>=2.6.0 ===<br />
<br />
This version of vzmigrate works properly with regard to handling ips. It will not notify ve owners of moves as in the above example. Other than that it’s essentially the same.<br />
<br />
<pre>[root@virt12 sbin]# vzmigrate 10.1.4.64 -r no 1212:1212:/vz/private/1212:/vz/root/1212<br />
migrating on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine (backup2 or mail).<br />
Cancel the ve (first we have to rename things which vzmigrate changed so cancelve will find them):<br />
<br />
<pre>[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/1212.conf<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/1212<br />
<br />
[root@virt12 sbin]# cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, <tt>cancelve</tt> would offer to remove them. You want to say no to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
=== src=2.5.x ===<br />
<br />
First, go to the private dir:<br />
<br />
cd /vz1/private/<br />
<br />
Stop the VE - make sure it stops totally cleanly.<br />
<br />
vzctl stop 1212<br />
<br />
Then you’d use vemove - a script written to copy over the config, create tarballs of the ve’s data on the destination virt, and cancel the ve on the source system (in this example we’re going to put a ve that was in /vz1/private on the src virt, in /vz/private on the dst virt):<br />
<br />
<pre>[root@virt12 sbin]# vemove<br />
ERROR: Usage: vemove veid target_ip target_path_dir<br />
[root@virt12 sbin]# vemove 1212 10.1.4.64 /vz/private/1212<br />
tar cfpP - 1212 --ignore-failed-read | (ssh -2 -c arcfour 10.1.4.64 "split - -b 1024m /vz/private/1212.tar" )<br />
scp /vzconf/1212.conf 10.1.4.64:/vzconf<br />
cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, cancelve would offer to remove them. You want to say no to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
When you are done, go to /vz/private on the dst virt you will have files like this:<br />
<br />
<pre>1212.taraa<br />
1212.tarab<br />
1212.tarac</pre><br />
<br />
Each one 1024m (or less, for the last one) in size.<br />
<br />
on the dst server and run:<br />
<br />
cat 1212.tar?? | tar xpPBf -<br />
<br />
and after 20 mins or so it will be totally untarred. Now since the conf<br />
file is already there, you can go ahead and start the system.<br />
<br />
vzctl start 1212<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
NOTE: you MUST tar the system up using the virtuozzo version of tar that<br />
is on all the virt systems, and further you MUST untar the tarball with<br />
the virtuozzo tar, using these options: `<tt>tar xpPBf -</tt>`<br />
<br />
If you tar up an entire VE and move it to a non-virtuozzo machine, that is<br />
ok, and you can untar it there with normal tar commands, but do not untar<br />
it and then repack it with a normal tar and expect it to work - you need<br />
to use virtuozzo tar commands on virtuozzo tarballs to make it work.<br />
<br />
The backups are sort of an exception, since we are just (usually)<br />
restoring user data that was created after we gave them the system, and<br />
therefore has nothing to do with magic symlinks or vz-rpms, etc.<br />
<br />
== Traffic accounting on linux ==<br />
<br />
DEPRECATED - all tracking is done via bwdb now. This is how we used to track traffic.<br />
<br />
TODO: update for diff versions of vz<br />
<br />
Unlike FreeBSD, where we have to add firewall count rules to the system to count the traffic, on virtuozzo counts the traffic for us. You an see the current traffic stats by running `vznetstat`:<br />
<br />
<pre># vznetstat<br />
VEID Net.Class Output(bytes) Input(bytes)<br />
-----------------------------------------------<br />
24218 1 484M 39M<br />
24245 1 463M 143M<br />
2451 1 2224M 265M<br />
2454 1 2616M 385M<br />
4149 1 125M 68M<br />
418 1 1560M 34M<br />
472 1 1219M 315M<br />
726 1 628M 317M<br />
763 1 223M 82M<br />
771 1 4234M 437M<br />
-----------------------------------------------<br />
Total: 13780M 2090M</pre><br />
<br />
As you can see the VEID is on a line with the in and out bytes. So, we simply run a cron job:<br />
<br />
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /root/vztrafdump.sh<br />
<br />
Just like we do on FreeBSD - this one goes through all the VEs in /vz/private and greps the line from vznetstat that matches them and dumps it in /jc_traffic_dump on their system. Then it does it again for all the VEs in /vz1/private. It is important to note that vznetstat runs only once, and the grepping is done from a temporary file that contains that output - we do this because running vznetstat once for each VE that we read out of /vz/private and /vz1/private would take way too long and be too intensive.<br />
<br />
You do not need to do anything to facilitate this other than make sure that that cron job is running - the vznetstat counters are always running, and any new VEs that are added to the system will be accounted for automatically.<br />
<br />
Traffic resetting no longer works with vz 2.6, so we disable the vztrafdump.sh on those virts.<br />
<br />
== Watchdog script ==<br />
<br />
On some of the older virts, we have a watchdog running that kills procs that are deemed bad per the following:<br />
<br />
/root/watchdog from quar1<br />
<br />
<pre>if echo $line | awk '{print $(NF-3)}' | grep [5-9]...<br />
then<br />
# 50-90%<br />
if echo $line | awk '{print $(NF-1)}' | grep "...:.."<br />
then<br />
# running for > 99min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
if echo $line | awk '{print $(NF-1)}' | grep "....m"<br />
then<br />
# running for > 1000min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
fi<br />
<br />
if echo $line | awk '{print $(NF-3)}' | grep [1-9]...<br />
then<br />
# running for 10-90 percent<br />
if echo $line | awk '{print $NF}' | egrep 'cfusion|counter|vchkpw'<br />
then<br />
<br />
if echo $line | awk '{print $(NF-1)}' | grep "[2-9]:.."<br />
then<br />
# between 2-9min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
elif echo $line | awk '{print $(NF-1)}' | grep "[0-9][0-9]:.."<br />
then<br />
# up to 99min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
fi<br />
fi</pre><br />
<br />
<br />
== Misc Linux Items ==<br />
<br />
We are overselling hard drive space ... when you configure a linux system with a certain amount of disk space (the default is 4gigs) you do not actually use up 4gigs of space on the system. The diskspace setting for a user is simply a cap, and they only use up as much space on the actual disk drive as they are actually using.<br />
<br />
When you create a new linux system, even though there are some 300 RPMs or so installed, if you run `df -k` you will see that the entire 4gig partition is empty - no space is being used. This is because the files in their system are "magic symlinks" to the template for their OS that is in /vz/template - however, any changes to any of those files will "disconnect" them and they will immediately begin using space in their system. Further, any new files uploaded (even if those new files overwrite existing files) will take up space on the partition.<br />
<br />
<br />
if you see this:<br />
<br />
<pre>[root@virt8 root]# vzctl stop 160 ; vzctl start 160<br />
VE is not running<br />
Starting VE ...<br />
VE is unmounted<br />
VE is mounted<br />
Adding IP address(es): 69.55.226.83 69.55.226.84<br />
bash ERROR: Can't change file /etc/sysconfig/network<br />
Deleting IP address(es): 69.55.226.83 69.55.226.84<br />
VE is unmounted<br />
[root@virt8 root]#</pre><br />
<br />
it probably means they no longer have /bin/bash - copy one in for them<br />
<br />
ALSO: another possibility is that they have removed the `ed` RPM from their system - it needs to be reinstalled into their system. But since their system is down, this is tricky ...<br />
<br />
VE startup scripts used by 'vzctl' want package 'ed' to be available inside VE. So if package 'ed' will be enabled in OS template config and OS template itself VE #827 is based on - this error should be fixed.<br />
<br />
yes, it is possible to add RPM to VE while it not running.<br />
Try to do following:<br />
<br />
<pre># cd /vz/template/<OS_template_with_ed_package>/<br />
# vzctl mount 827<br />
# rpm -Uvh --root /vz/root/827 --veid 827 ed-0.2-25.i386.vz.rpm</pre><br />
<br />
Usually theres an error, but its ok<br />
<br />
Note: replace 'ed-0.2-25.i386.vz.rpm' in last command with actual<br />
version of 'ed' package you have.<br />
<br />
----<br />
<br />
So how do I know what template the user has ? cat their conf file and it is listed in there. For example, if the conf file has:<br />
<br />
<pre>[root@virt12 sbin]# vc 1103<br />
…snip…<br />
OSTEMPLATE="debian-3.0/20030822"<br />
TEMPLATES="mod_perl-deb30/20030707 mod_ssl-deb30/20030703 mysql-deb30/20030707 proftpd-deb30/20030703 webmin-deb30/20030823 "<br />
…</pre><br />
<br />
then they are on debian 3.0, all of their system RPMs are in /vz/template/debian-3.0, and they are using version 20030822 of that debian 3.0 template. Also, they’ve also got additional packages installed (mod_perl, mod_ssl, etc). Those are also found under /vz/template<br />
<br />
----<br />
<br />
Edits needed to run java:<br />
<br />
When we first created the VEs, the default setting for privvmpages was 93000:94000 ... which was high enough that most people never had problems ... however, you can;t run java or jdk or tomcat or anything java related with that setting. We have found that by setting privvmpages to 610000:615000 that java runs just fine. That is now the default setting. It is exceedingly rare that anyone needs it higher than that, although we have seen it once or twice.<br />
<br />
Any problems with java at all - the first thing you need to do is see if the failcnt has raised for privvmpages.<br />
<br />
<pre># vzctl start 160<br />
Starting VE ...<br />
vzquota : (error) Quota on syscall for 160: Device or resource busy<br />
Running vzquota on failed for VE 160 [3]</pre><br />
<br />
This is because my pwd is _in_ their private directory - you can't start it until you move out<br />
<br />
People seem to have trouble with php if they are clueless newbies. Here are two common problems/solutions:<br />
<br />
no... but i figured it out myself. problem was the php.ini file that came<br />
vanilla with the account was not configured to work with apache (the<br />
ENGINE directive was set to off).<br />
<br />
everything else seems fine now.<br />
<br />
----<br />
<br />
the problem was in the php.ini file. I noticed that is wasnt showing<br />
the code when it was in an html file so I looked at the php.ini file<br />
and had to change it so it recognized <? tags aswell as <?php tags.<br />
<br />
Also, make sure added to httpd.conf<br />
AddType application/x-httpd-php .php<br />
<br />
----<br />
<br />
You can set the time zone:<br />
<br />
You can change the timezone by doing this:<br />
<br />
ln -sf /usr/share/zoneinfo/<zone> /etc/localtime<br />
<br />
where <zone> is the zone you want in the /usr/share/zoneinfo/ directory.<br />
<br />
----<br />
<br />
Failing shm_open calls:<br />
<br />
first, please check if /dev/shm is mounted inside VE.<br />
'cat /proc/mounts' command should show something like this:<br />
tmpfs /dev/shm tmpfs rw 0 0<br />
<br />
If /dev/shm is not mounted you have 2 ways to solve issue:<br />
1. execute following command inside VE (doesn't require VE reboot):<br />
mount -t tmpfs none /dev/shm<br />
2. add following string to /etc/fstab inside VE and reboot it:<br />
tmpfs /dev/shm tmpfs defaults 0 0<br />
<br />
----<br />
<br />
You can have a mounted but not running ve<br />
Just:<br />
vzctl mount <veid><br />
<br />
----<br />
<br />
When a debian sys can’t get on the network, and you try:<br />
<br />
<pre>vzctl set 1046 --ipadd 69.55.227.117<br />
Adding IP address(es): 69.55.227.117<br />
Failed to bring up lo.<br />
Failed to bring up venet0.</pre><br />
<br />
They probably removed iproute package, which must be the one from swsoft. To restore:<br />
<pre># dpkg -i --veid=1046 --admindir=/vz1/private/1046/root/var/lib/dpkg --instdir=/vz1/private/1046/root/ /vz/template/debian-3.0/iproute_20010824-8_i386.vz.deb<br />
(Reading database ... 16007 files and directories currently installed.)<br />
Preparing to replace iproute 20010824-8 (using .../iproute_20010824-8_i386.vz.deb) ...<br />
Unpacking replacement iproute ...<br />
Setting up iproute (20010824-8) ...</pre><br />
<br />
Then restart their ve<br />
<br />
----<br />
<br />
in a ve i do:<br />
<br />
<pre>cd /<br />
du -h .</pre><br />
<br />
and get: 483M .<br />
<br />
i do:<br />
<br />
<pre>bash-2.05a# df -h<br />
Filesystem Size Used Avail Use% Mounted on<br />
/dev/vzfs 4.0G 2.3G 1.7G 56% /</pre><br />
<br />
how can this be?<br />
<br />
Is it possible that quota file was corrupted somehow? Please try to: <br />
<pre>vzctl stop <VEID><br />
vzquota drop <VEID><br />
vzquota init <VEID><br />
vzctl start <VEID></pre><br />
<br />
----<br />
<br />
How to stop vz from starting after reboot:<br />
<br />
VIRTUOZZO=no <br />
in <br />
/etc/sysconfig/vz<br />
<br />
To start: <br />
service vz start<br />
(after setting VIRTUOZZO=yes in /etc/sysconfig/vz)<br />
<br />
service vz restart will do some kind of 'soft reboot' -- restart all<br />
VPSes and reload modules without rebooting the node<br />
<br />
if you need to shut down all VPSes really really fast, run killall -9 init<br />
<br />
----<br />
<br />
Postfix tip:<br />
<br />
You may want to tweak settings: default_process_limit=10<br />
<br />
= Virtuozzo VPS Management Tools =<br />
<br />
== vm ==<br />
<br />
TODO<br />
<br />
== cancelve ==<br />
cancelve <veid><br />
this will:<br />
* stop a ve<br />
* check for backups (offer to remove them from the backup server <br />
and the backup.config)<br />
* rename the private dir<br />
* check for PTR, provide the commands to reset to default<br />
* and rename the ve’s config<br />
* remind you to remove firewall rules<br />
* remind you to remove DNS entries<br />
<br />
== ipadd ==<br />
ipadd <veid> <ip> [ip] [ip]<br />
add’s ip(s) to a ve<br />
<br />
== ipdel ==<br />
ipdel <veid> <ip> [ip] [ip]<br />
removes ip(s) from a ve<br />
<br />
== vc ==<br />
vc <veid><br />
the equivalent of: <tt>cat /vzconf/<veid>.conf</tt><br />
<br />
== vl ==<br />
vl<br />
will displays a list of ve #’s, 1 per line. (ostensibly to use in a for loop)<br />
<br />
== vp ==<br />
vp <veid><br />
the equivalent of: <tt>vzps auxww –E <veid></tt><br />
<br />
== vpe ==<br />
DEPRECATED<br />
vpe <veid> <br />
this will allow you to do a vp when a ve is running out of control, the equivalent of (deprecated since vp operates outside the VPS): <br />
<pre>vzctl set <veid> --kmemsize 2100000:2200000<br />
vzctl exec <veid> ps auxw<br />
vzctl set <veid> --kmemsize (ve’s orig lvalue):(ve’s orig hvalue)</pre><br />
<br />
== vt ==<br />
vt <veid><br />
the equivalent of: <tt>vztop –E <veid></tt><br />
<br />
== vr ==<br />
vr <veid><br />
the equivalent of: <tt>vzctl stop <veid>; vzctl start <veid></tt><br />
You can run this even if the ve is down - the stop command will just fail<br />
<br />
== vs ==<br />
vs [veid]<br />
displays status (output of <tt>vzctl status <veid></tt>) on each ve configured on the system (as determined by <tt>/vzconf/*.conf</tt>)<br />
If passed an argument, gives the status for just that ve. <br />
A running system looks like:<br />
<tt>VEID 16066 exist mounted running</tt><br />
<br />
A ve which is not running (but does exist) looks like:<br />
<tt>VEID 9990 exist unmounted down</tt><br />
<br />
A ve which is not running and doesn’t exist looks like:<br />
<tt>VEID 421 deleted unmounted down</tt><br />
<br />
== vs2 ==<br />
vs2 [veid]<br />
this is similar to vs in that it displays status (output of <tt>vzctl status <veid></tt>) on each ve,<br />
but the difference is it’s list comes from doing an ls on the data dirs. This was meant to catch <br />
the rare case where a ve configured but exists. <br />
<br />
== vw ==<br />
vw [veid]<br />
displays the output of ‘<tt>w</tt>’ (the equivalent of <tt>vzctl exec <veid> w</tt>) for each configured ve (as determined by <tt>/vzconf/*.conf</tt>). Useful for determing which ve is contributing to a heavily-loaded system.<br />
If passed an argument, gives ‘<tt>w</tt>‘ output for just that ve. <br />
Ex:<br />
<pre>[root@virt2 etc]# vw<br />
134<br />
10:52pm up 79 days, 6:14, 0 users, load average: 0.02, 0.02, 0.00<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT<br />
16027<br />
2:52pm up 7 days, 19:54, 0 users, load average: 0.00, 0.00, 0.00<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT<br />
16055<br />
2:52pm up 79 days, 6:38, 0 users, load average: 0.00, 0.04, 0.07<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT</pre><br />
<br />
== vwe ==<br />
vwe [constraint]<br />
just like <tt>vw</tt>, but takes a constraint as an argument, only show’s ve’s with loads >= the constraint provided. If no constraint is provided, 1 is used by default<br />
<br />
== vzs ==<br />
vzs [veid]<br />
displays the beancounter status for all ve’s, or a particular ve if an argument is passed<br />
<br />
== ve ==<br />
ve <veid><br />
the equivalent of: <tt>vzctl enter <veid></tt><br />
<br />
== vx ==<br />
vx <veid> <command><br />
the equivalent of: <tt>/usr/sbin/vzctl exec <veid> <command></tt><br />
<br />
== dprocs ==<br />
dprocs [count]<br />
a script which outputs a continuous report (or a certain number of reports if an option is passed) of processes stuck in the D state and which VPS’s those procs belong to.<br />
<br />
== setmem ==<br />
setmem <VEID> <256|512|768|1024|2048><br />
adjusts the memory resources for the VE<br />
<br />
== afacheck.sh ==<br />
afacheck.sh<br />
displays the health/status of containers and mirrors on an adaptec card (currently quar1, tempvirt1-2, virt9, virt10)- all other are LSI<br />
<br />
== backup ==<br />
backup<br />
backup script called nightly to update virt scripts and do customer backups<br />
<br />
== checkload.pl ==<br />
checkload.pl<br />
this was intended to be setup as a cronjob to watch processes on a virt when the load <br />
rises above a certain level. Not currently in use.<br />
<br />
== findbackuppigs.pl ==<br />
findbackuppigs.pl<br />
looks for files larger than 50MB which customers have asked us to backup. Emails matches<br />
to linux@johncompanies.com<br />
<br />
== gatherlinux.pl ==<br />
gatherlinux.pl<br />
gathers up data about ve’s configured and writes to a file. Used for audits against the db<br />
<br />
== gb ==<br />
gb <search><br />
greps backup.config for the given search parameter<br />
<br />
== gbg ==<br />
gbg <search><br />
greps backup.config for the given search parameter and presents just the directories (for clean pasting)<br />
<br />
== linuxtrafficgather.pl ==<br />
linuxtrafficgather.pl [yy-mm]<br />
sends a traffic usage summary by ve to support@johncomapnies.com and payments@johncompanies.com.<br />
Optional arguments are year and month (must be in the past). If not passed, assumes last month. Relies on <br />
traffic logs created by netstatreset and netstatbackup<br />
<br />
== linuxtrafficwatch.pl ==<br />
linuxtrafficwatch.pl<br />
checks traffic usage and emails support@johncomapnies.com when a ve reaches the warning level (35G) and<br />
the limit (40G). Works on virtuozzo versions <= 2.6.x. We really aren’t using this anymore now that we have netflow.<br />
<br />
== linuxtrafficwatch2.pl ==<br />
linuxtrafficwatch2.pl<br />
checks traffic usage and emails support@johncomapnies.com when a ve reaches the warning level (35G) and<br />
the limit (40G). Works on virtuozzo version 2.6.x. We really aren’t using this anymore now that we have netflow.<br />
<br />
== load.pl ==<br />
load.pl<br />
feeds info to load mrtg - executed by inetd.<br />
<br />
== mb (linux) ==<br />
mb <mount|umount><br />
(nfs) mounts and umounts dirs to backup2. Shortcuts are mbm and mbu to mount and unmount. <br />
<br />
== migrate ==<br />
migrate <ip of node migrating to> <veid> <target_veid> [target dir: vz | vz1 | vz2]<br />
this is basically a wrapper for <tt>vzmigrate</tt> – vzmigrate is a util to seamlessly move a ve from one host to another. This wrapper was written cause vz virtuozzo version 2.6 had a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables. This script mitigates that. Since it makes multiple ssh connections to the target host, it’s a good idea to put the pub key for the src system in the authorized_keys file on the target host. In addition, it emails ve owners when their migration starts and stops (if they place email addresses in a file on their system: /migrate_notify). To move everyone off a system, you’d do:<br />
for f in `vl`; do migrate <ip> $f; done<br />
<br />
== migrateonline ==<br />
migrateonline <ip of node migrating to> <veid> <target_veid> [target dir: vz | vz1 | vz2]<br />
this is the same as migrate but will migrate a ve in <tt>–online</tt> mode which means it won’t be shut down at the end of the migration. This only works when migrating ve’s between 2 machines running a 2.6 kernel (currently tempvirt1-2. virt16-19, virt12). If you get an error that the machine you’re trying to migrate to has a different CPU or features, etc, then you have to edit the file and add the –f switch to the vzmigrate line- you can basically ignore this kind of warning (but never ignore a warning about missing templates on the destination node). NOTE: This edit (if made to migrateonline) will be overwritten by the base script during each night’s backup.<br />
<br />
== netstatbackup ==<br />
DEPRECATED<br />
netstatbackup <br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.5.x. <br />
<br />
== netstatbackup2 ==<br />
DEPRECATED<br />
netstatbackup2<br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.6.x. <br />
<br />
== netstatreset ==<br />
DEPRECATED<br />
netstatreset<br />
writes traffic count data to a logfile and resets counters to 0. Works on virtuozzo versions 2.5.x <br />
<br />
== netstatreset2 ==<br />
DEPRECATED<br />
netstatreset2<br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.6.x. <br />
<br />
== orphanedbackupwatchlinux ==<br />
orphanedbackupwatchlinux <br />
looks for directories on backup2 which aren’t configured in backup.config and offers to <br />
delete them<br />
<br />
== rsync.backup (linux) ==<br />
rsync.backup<br />
does customer backups (relies on backup.config)<br />
<br />
== startvirt.pl ==<br />
startvirt.pl<br />
forks off start ve commands – keeps 6 running at a time. This is not to be used on systems where fastboot is enabled as it circumvents the benefit of the fastboot. The script will occasionally not exit gracefully and will continue to use up CPU, so it should be watched. Also, don’t exit from the script till you’re sure all ve’s are started – if you do you need to start them manually and may have to free up locks. On some systems, the startvirt script doesn’t exit cleanly and you have to ^C out of it. Be careful though- doing so can leave some VE’s in an odd bootup state and you may need to ‘vr’ them manually. You should check to see which ve’s aren’t running and/or confirm all have started when ^C’ing out of startvirt.<br />
<br />
== taskdone (linux) ==<br />
taskdone<br />
when called will email support@johncompanies.com with the hostname of the machine from which it was <br />
executed as the subject<br />
<br />
== vb (linux) ==<br />
vb<br />
the equivalent of: <tt>vi /usr/local/sbin/backup.config</tt><br />
<br />
== vemakeXX ==<br />
DEPRECATED<br />
vemakerh9 <br />
ve create script for RH9 (see vemake)<br />
<br />
vemakedebian30 <br />
ve create script for debian 3.0 (Woody) (see vemake)<br />
<br />
vemakedebian31 <br />
ve create script for debian 3.1 (Sarge) (see vemake)<br />
<br />
vemakedebian40 <br />
ve create script for debian 4.0 (Etch) (see vemake)<br />
<br />
vemakefedora, vemakefedora2, vemakefedora4, vemakefedora5, vemakefedora6, vemakefedora7<br />
ve create script for fedora core 1, 2, 4, 5, 6, 7 respectively (see vemake)<br />
<br />
vemakecentos3, vemakecentos4<br />
ve create script for centos 3, 4 respectively (see vemake)<br />
<br />
vemakesuse, vemakesuse93, vemakesuse100<br />
ve create script for suse 9.2, 9.3, 10.0 respectively (see vemake)<br />
<br />
vemakeubuntu5, vemakeubuntu606, vemakeubuntu606 vemakeubuntu704<br />
ve create script for ubuntu 5.10, 6.06, 6.10, 7.04 respectively (see vemake)<br />
<br />
== vemove ==<br />
DEPRECATED<br />
vemove <veid> <target_ip> </vz/private/123><br />
this script simplifies the old way of moving ve’s from one system to another - in short moving a ve to or from a virt running virtuozzo < 2.6.x<br />
It’s the equivalent of:<br />
<tt>tar cfpP - <veid> --ignore-failed-read | (ssh -2 -c arcfour <target_ip> "split - -b 1024m </vz/private/123>.tar" )</tt>This should only be used if migrate/vzmigrate can’t be used. <br />
<br />
== vim.watchdog ==<br />
vim.watchdog <br />
looks for and kills procs matching “vi|vim|nano|pine|elm” that are running for a long time & consuming lots of cpu. Works on virtuozzo versions 2.5.x<br />
<br />
== vim.watchdog2 ==<br />
vim.watchdog2<br />
looks for and kills procs matching “vi|vim|nano|pine|elm” that are running for a long time & consuming lots of cpu.<br />
Works on virtuozzo versions 2.6.x.<br />
<br />
== vzmigrate ==<br />
vzmigrate <target_ip> -r no <veid>:[dst veid]:[dst /vzX/private/veid]:[dst /vzX/root/veid]<br />
(this is the raw command “wrapped” by migrate/migrateonline) this will seamlessly move a ve from one host to another. The ve will run for the duration of the migration till the very end when it’s shut down, ip moved and started up on the target system. The filesystem on the src will remain. This should be watched – occasionally the move will timeout and leave the system shut down. If target private and root aren’t specified it just puts it in /vz. Only works when both systems are running virtuozzo 2.6.x<br />
<br />
== vztrafdump.sh ==<br />
DEPRECATED<br />
vztrafdump.sh<br />
writes traffic usage info by ve to a file called jc_traffic_dump in each ve’s / dir. Works on virtuozzo versions <= 2.5.x. <br />
<br />
== vztrafdump2.sh ==<br />
DEPRECATED<br />
vztrafdump2.sh<br />
writes traffic usage info by ve to a file called jc_traffic_dump in each ve’s / dir. Works on virtuozzo versions 2.6.x. <br />
<br />
== addtun ==<br />
addtun <veid><br />
Add’s tun device to ve.<br />
<br />
== bwcap ==<br />
bwcap <veid> <kbps><br />
Ex: <tt>bwcap 1234 512</tt><br />
Caps a VE’s bandwidth to the amount given<br />
<br />
== setdisk ==<br />
setdisk <veid> <diskspace in GB><br />
Ex: <tt>setdisk 1234 5</tt><br />
Gives a VE’s a given amount of disk space<br />
<br />
== vdf ==<br />
vdf <veid> <br />
the equivalent of: <tt>vzctl exec <veid> df –h</tt><br />
<br />
== vdff ==<br />
vdff<br />
runs a (condensed) vdf for all ve’s in your pwd (must be run from /vz/privateN)<br />
<br />
== mvbackups ==<br />
mvbackups <veid> <target_machine> (virt1) <target_dir> (vz1)<br />
moves backups from one location to another on the backup server, and provides you with option to remove entries from current backup.config, and simple paste command to add the config to backup.config on the target server<br />
<br />
== checkquota ==<br />
checkquota<br />
for all the ve’s in the cwd (run from /vz/private, /vz1/private, etc) reports what vz quota says they’re using and what the actual usage is (as reported by du)<br />
<br />
== clearquota ==<br />
clearquota <veid><br />
Recalculates a ve’s quota, prints out the usage before and after. The equivalent of:<br />
<tt>vdf <veid>; v stop <veid>; vzquota drop <veid>; v start <veid>; vdf <veid></tt><br />
<br />
== dprocs ==<br />
dprocs<br />
Sometimes the server’s have a large number of processes get stuck in the D state- this script shows (every 3 secs) which VE’s have D procs, which procs<br />
are stuck and a running average of the top “offenders”<br />
<br />
== vzstat ==<br />
vstat<br />
sort of like top for VZ. sort VEs by CPU usage by pressing 'o' and then 'c' keys</div>Supporthttps://wiki.jcihosting.com/index.php?title=VPS_Management&diff=1012VPS Management2013-03-11T22:48:32Z<p>Support: /* Moving a VE to another virt (migrate/migrateonline) */</p>
<hr />
<div>= Common Problems =<br />
== Login to any machine without a password ==<br />
<br />
This is possible via the use of ssh keys. The process is thus:<br />
<br />
1. place the public key for your user (root@mail) in the /root/.ssh/authorized_keys file on the server you wish to login to<br />
cat /root/.ssh/id_dsa.pub<br />
(paste that into authorized_keys on the target server). If the file doesn't exist, create it.<br />
<br />
2. enable root login (usually only applies to FreeBSD). Edit the /etc/ssh/sshd_config on the target server and change:<br />
<tt>#PermitRootLogin no</tt><br />
to<br />
<tt>PermitRootLogin yes</tt><br />
<br />
3. Restart the sshd on the target machine. First, find the sshd process: <br />
jailps <hostname> | grep sshd <br />
or <br />
vp <VEID> | grep sshd<br />
<br />
Look for the process resembling:<br />
root 17296 0.0 0.0 5280 1036 ? Ss 2011 4:27 /usr/sbin/sshd <br />
(this is the sshd)<br />
<br />
Not:<br />
root 6270 0.5 0.0 6808 2536 ? Ss 14:33 0:00 sshd: root [priv]<br />
(this is an sshd child- someone already ssh'd in as root)<br />
<br />
Restart the sshd: <br />
kill -1 <PID><br />
<br />
Ex:<br />
kill -1 17296<br />
<br />
You may now ssh in.<br />
<br />
Once you're done, IF you enabled root login, you should repeat steps 2 and 3 to disable root logins.<br />
<br />
== Letting someone in who has locked themselves out (killed sshd, lost pwd) ==<br />
<br />
There are two ways people frequently lock themselves out - either they forget a password, or they kill off sshd somehow.<br />
<br />
These are actually both fairly easy to solve. First, let's say someone kills off their sshd, or somehow mangles /etc/ssh/sshd_config such that it no longer lets them in.<br />
<br />
Their email may be very short, or it may have all sorts of details about how you should fix sshd_config to let them in ... just ignore all of this. They can fix their own mangled sshd. Fixing this is very simple. First, edit the /etc/inetd.conf on their system and uncomment the telnet line:<br />
<br />
telnet stream tcp nowait root /usr/libexec/telnetd telnetd<br />
#telnet stream tcp6 nowait root /usr/libexec/telnetd telnetd<br />
<br />
(just leave the tcp6 version of telnet commented)<br />
<br />
Then, use jailps to list the processes on their system, and find their inetd process. Then simply:<br />
<br />
kill -HUP (pid)<br />
<br />
where (pid) is the PID of their inetd process. Now they have telnet running on their system and they can log in and do whatever they need to do.<br />
<br />
The only complications that could occur are:<br />
<br />
a) their firewall config on our firewall has port 23 blocked, in which case you will need to open that - will be covered in a different lesson.<br />
<br />
b) they are not running inetd, so you can't HUP it. If this happens, edit their /etc/rc.conf, add the inetd_enable="YES" line, and then kill<br />
their jail with /tmp/jailkill.pl - then restart their jail with the jail line from their quad/safe file. Easy.<br />
<br />
If they have forgotten a password,<br />
<br />
On 6.x+ you can reset their password with:<br />
jexec <jailID from jls> passwd root<br />
<br />
Note: the default password for 6.x jails is 8ico2987, for 4.x it is p455agfa<br />
<br />
On 4.x, you need to cd to their etc directory<br />
... for instance:<br />
<br />
cd /mnt/data2/198.78.65.136-col00261-DIR/etc<br />
<br />
and run:<br />
<br />
vipw -d .<br />
<br />
Then paste in these two lines (theres a paste with these):<br />
<br />
root:$1$krszPxhk$xkCepSnz3mIikT3vCtJCt0:0:0::0:0:Charlie &:/root:/bin/csh<br />
user:$1$Mx9p5Npk$QdMU6c8YQqp2FW2M3irEh/:1001:1001::0:0:User &:/home/user:/bin/sh<br />
<br />
overwriting the lines they already have for "user" and "root" - then just tell them that both user and root have been reset to the default password of p455agfa.<br />
<br />
For linux, just passwd inside shell or <br />
vzctl set <veid> --userpasswd root:p455agfa –save<br />
<br />
Starting in 2009 we began giving out randomized passwords for FreeBSD and Linux as the default password. That is stored with each system in Mgmt. You should look for and reset the password to that password in the event of a reset and refer the customer to use their original password from their welcome email- this way we don’t have to send the password again via email (in clear text).<br />
<br />
<br />
== sendmail can’t be contacted from ext ip (only locally) ==<br />
<br />
By default redhat puts this line in sendmail.mc:<br />
<br />
DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')<br />
<br />
which makes it only answer on localhost. Comment it out like:<br />
<br />
dnl DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')<br />
<br />
and then rebuild sendmail.cf with:<br />
<br />
m4 /etc/mail/sendmail.mc > /etc/sendmail.cf<br />
<br />
== virt doesn’t properly let go of ve’s ip(s) when moved to another system ==<br />
<br />
On virtuozzo 2.6 systems, it's been observed that when moving ips from one virt to another that sometimes the routing table will not get updated to reflect the removal of the ip addresses.<br />
<br />
A recent example was a customer that was moving to a new ve on a new virt and the ip addresses were traded between the two ve's. After the trade the two systems were not able to talk to each other. When looking at the routing table for the old system all the ip addresses were still in the routing table as being local, like so:<br />
<br />
<pre>netstat -rn | grep 69.55.225.149<br />
69.55.225.149 0.0.0.0 255.255.255.255 UH 40 0 0 venet0</pre><br />
<br />
This was preventing traffic to the other system from being routed properly.<br />
The solution is to manually delete the route:<br />
<br />
route delete 69.55.225.149 gw 0.0.0.0<br />
<br />
Supposedly, this was fixed in 2.6.1<br />
<br />
== sshd on FreeBSD 6.2 segfaults ==<br />
<br />
First try to reinstall ssh<br />
<br />
<pre>cd /usr/src/secure<br />
cd lib/libssh<br />
make depend && make all install<br />
cd ../../usr.sbin/sshd<br />
make depend && make all install<br />
cd ../../usr.bin/ssh<br />
make depend && make all install</pre><br />
<br />
Failing that, find the library that’s messed up:<br />
<br />
<pre>ldd /usr/sbin/sshd<br />
libssh.so.3 => /usr/lib/libssh.so.3 (0x280a3000) <br />
libutil.so.5 => /lib/libutil.so.5 (0x280d8000) <br />
libz.so.3 => /lib/libz.so.3 (0x280e4000) <br />
libwrap.so.4 => /usr/lib/libwrap.so.4 (0x280f5000) <br />
libpam.so.3 => /usr/lib/libpam.so.3 (0x280fc000) <br />
libbsm.so.1 => /usr/lib/libbsm.so.1 (0x28103000) <br />
libgssapi.so.8 => /usr/lib/libgssapi.so.8 (0x28112000) <br />
libkrb5.so.8 => /usr/lib/libkrb5.so.8 (0x28120000) <br />
libasn1.so.8 => /usr/lib/libasn1.so.8 (0x28154000) <br />
libcom_err.so.3 => /usr/lib/libcom_err.so.3 (0x28175000) <br />
libroken.so.8 => /usr/lib/libroken.so.8 (0x28177000) <br />
libcrypto.so.4 => /lib/libcrypto.so.4 (0x28183000) <br />
libcrypt.so.3 => /lib/libcrypt.so.3 (0x28276000) <br />
libc.so.6 => /lib/libc.so.6 (0x2828e000) <br />
libmd.so.3 => /lib/libmd.so.3 (0x28373000)</pre><br />
<br />
md5 them and compare to other jail hosts or jails running on host<br />
<br />
for libcrypto reinstall:<br />
<pre>/usr/src/crypto<br />
make depend && make all install</pre><br />
<br />
= FreeBSD VPS =<br />
<br />
== Starting jails: Quad/Safe Files ==<br />
<br />
FreeBSD customer systems do not start up automatically at boot time. When one of our freebsd machines boots up, it boots up, and does nothing else. To start jails, we put the commands to start each jail into a shell script(s) and run the script(s). Jail startup is something that needs to be actively monitored, which is why we don’t just run the script automatically. More on monitoring later.<br />
<br />
NOTE: >=7.x we have moved to 1 quad file: <tt>quad1</tt>. Startups are not done by running each quad, but rather [[#startalljails|startalljails]] which relies on the contents of <tt>quad1</tt>. The specifics of this are lower in this article. What follows here applies for pre 7.x systems.<br />
<br />
There are eight files in <tt>/usr/local/jail/rc.d</tt>:<br />
<br />
<pre>jail3# ls /usr/local/jail/rc.d/<br />
quad1 quad2 quad3 quad4 safe1 safe2 safe3 safe4<br />
jail3#</pre><br />
<br />
four quad files and four safe files.<br />
<br />
Each file contains an even number of system startup blocks (total number of jails divided by 4)<br />
<br />
The reason for this is, if we make one large script to startup all the systems at boot time, it will take too long - the first system in the script will start up right after system boot, which is great, but the last system may not start for another 20 minutes.<br />
<br />
Since there is no way to parralelize this during the startup procedure, we simply open four terminals (in screen window 9) and run each script, one in each terminal. This way they all run simultaneously, and the very last system in each startup script gets started in 1/4th the time it would if there was one large file<br />
<br />
The files are generally organized so that quad/safe 1&2 have only jails from disk 1, and quad/safe 3&4 have jails from disk 2. This helps ensure that only 2 fscks on any disk are going on at once. Further, they are balanced so that all quad/safe’s finish executing around the same time. We do this by making sure each quad/safe has a similar number of jails and represents a similar number of inodes (see js).<br />
<br />
The other, very important reason we do it this way, and this is the reason there are quad files and safe files, is that in the event of a system crash, every single vn-backed filesystem that was mounted at the time of system crash needs to be fsck'd. However, fsck'ing takes time, so if we shut the system down gracefully, we don't want to fsck.<br />
<br />
Therefore, we have two sets of scripts - the four quad scripts are identical to the four safe scripts except for the fact that the quad scripts contain fsck commands for each filesystem.<br />
<br />
So, if you shut a system down gracefully, start four terminals and run safe1 in window one, and safe2 in window 2, and so on.<br />
<br />
If you crash, start four terminals (or go to screen window 9) and run quad1 in window one, and quad2 in window 2, and so on.<br />
<br />
Here is a snip of (a 4.x version) quad2 from jail17:<br />
<br />
<pre>vnconfig /dev/vn16 /mnt/data2/69.55.228.7-col00820<br />
fsck -y /dev/vn16<br />
mount /dev/vn16c /mnt/data2/69.55.228.7-col00820-DIR<br />
chmod 0666 /mnt/data2/69.55.228.7-col00820-DIR/dev/null<br />
jail /mnt/data2/69.55.228.7-col00820-DIR mail1.phimail.com 69.55.228.7 /bin/sh /etc/rc<br />
<br />
# moved to data2 col00368<br />
#vnconfig /dev/vn28 /mnt/data2/69.55.236.132-col00368<br />
#fsck -y /dev/vn28<br />
#mount /dev/vn28c /mnt/data2/69.55.236.132-col00368-DIR<br />
#chmod 0666 /mnt/data2/69.55.236.132-col00368-DIR/dev/null<br />
#jail /mnt/data2/69.55.236.132-col00368-DIR limehouse.org 69.55.236.132 /bin/sh /etc/rc<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
vnconfig /dev/vn22 /mnt/data2/69.55.228.13-col01063<br />
fsck -y /dev/vn22<br />
mount /dev/vn22c /mnt/data2/69.55.228.13-col01063-DIR<br />
chmod 0666 /mnt/data2/69.55.228.13-col01063-DIR/dev/null<br />
jail /mnt/data2/69.55.228.13-col01063-DIR www.widestream.com.au 69.55.228.13 /bin/sh /etc/rc<br />
<br />
# cancelled col00106<br />
#vnconfig /dev/vn15 /mnt/data2/69.55.238.5-col00106<br />
#fsck -y /dev/vn15<br />
#mount /dev/vn15c /mnt/data2/69.55.238.5-col00106-DIR<br />
#chmod 0666 /mnt/data2/69.55.238.5-col00106-DIR/dev/null<br />
#jail /mnt/data2/69.55.238.5-col00106-DIR mail.azebu.net 69.55.238.5 /bin/sh /etc/rc</pre><br />
<br />
As you can see, two of the systems specified are commented out - presumably those customers cancelled, or were moved to new servers.<br />
<br />
As you can see, the vnconfig line is the simpler command line, not the longer one that was used when it was first configured. As you can see, all that is done is, vnconfig the filesystem, then fsck it, then mount it. The fourth command is the `jail` command used to start the system – but that will be covered later.<br />
<br />
Here is the safe2 file from jail17:<br />
<br />
<pre>vnconfig /dev/vn16 /mnt/data2/69.55.228.7-col00820<br />
mount /dev/vn16c /mnt/data2/69.55.228.7-col00820-DIR<br />
chmod 0666 /mnt/data2/69.55.228.7-col00820-DIR/dev/null<br />
jail /mnt/data2/69.55.228.7-col00820-DIR mail1.phimail.com 69.55.228.7 /bin/sh /etc/rc<br />
<br />
# moved to data2 col00368<br />
#vnconfig /dev/vn28 /mnt/data2/69.55.236.132-col00368<br />
#mount /dev/vn28c /mnt/data2/69.55.236.132-col00368-DIR<br />
#chmod 0666 /mnt/data2/69.55.236.132-col00368-DIR/dev/null<br />
#jail /mnt/data2/69.55.236.132-col00368-DIR limehouse.org 69.55.236.132 /bin/sh /etc/rc<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
vnconfig /dev/vn22 /mnt/data2/69.55.228.13-col01063<br />
mount /dev/vn22c /mnt/data2/69.55.228.13-col01063-DIR<br />
chmod 0666 /mnt/data2/69.55.228.13-col01063-DIR/dev/null<br />
jail /mnt/data2/69.55.228.13-col01063-DIR www.widestream.com.au 69.55.228.13 /bin/sh /etc/rc<br />
<br />
# cancelled col00106<br />
#vnconfig /dev/vn15 /mnt/data2/69.55.238.5-col00106<br />
#mount /dev/vn15c /mnt/data2/69.55.238.5-col00106-DIR<br />
#chmod 0666 /mnt/data2/69.55.238.5-col00106-DIR/dev/null<br />
#jail /mnt/data2/69.55.238.5-col00106-DIR mail.azebu.net 69.55.238.5 /bin/sh /etc/rc</pre><br />
<br />
As you can see, it is exactly the same, but it does not have the fsck lines.<br />
<br />
Take a look at the last entry - note that the file is named:<br />
<br />
/mnt/data2/69.55.238.5-col00106<br />
<br />
and the mount point is named:<br />
<br />
/mnt/data2/69.55.238.5-col00106-DIR<br />
<br />
This is the general format on all the FreeBSD systems. The file is always named:<br />
<br />
IP-custnumber<br />
<br />
and the directory is named:<br />
<br />
IP-custnumber-DIR<br />
<br />
If you run safe when you need a fsck, the mount will fail and jail will fail:<br />
<br />
# mount /dev/vn1c /mnt/data2/jails/65.248.2.131-ns1.kozubik.com-DIR<br />
mount: /dev/vn1c: Operation not permitted<br />
<br />
No reboot needed, just run the quad script<br />
<br />
Starting with 6.x jails, we added block delimiters to the quad/safe files, the block looks like:<br />
<br />
<pre>echo '## begin ##: nuie.solaris.mu'<br />
fsck -y /dev/concat/v30v31a<br />
mount /dev/concat/v30v31a /mnt/data1/69.55.228.218-col01441-DIR<br />
mount_devfs devfs /mnt/data1/69.55.228.218-col01441-DIR/dev<br />
devfs -m /mnt/data1/69.55.228.218-col01441-DIR/dev rule -s 3 applyset<br />
jail /mnt/data1/69.55.228.218-col01441-DIR nuie.solaris.mu 69.55.228.218 /bin/sh /etc/rc<br />
echo '## end ##: nuie.solaris.mu'</pre><br />
<br />
These are more than just informative when running quad/safe’s, the echo lines MUST be present for certain tools to work properly. So it’s important that any updates to the hostname also be updated on the 2 echo lines. For example, if you try to startjail a jail with a hostname which is on the jail line but not the echo lines, the command will return with host not found.<br />
<br />
=== FreeBSD 7.x+ notes ===<br />
<br />
Starting with the release of FreeBSD 7.x, we are doing jail startups in a slightly different way. First, thereis only 1 file: <tt>/usr/local/jail/rc.d/quad1</tt><br />
There are no other quads or corresponding safe files. The reason for this is twofold, 1. We can pass –C to fsck which will tell is to skip the fsck if the fs is clean (no more need for safe files), 2. We have a new startup script which can be launched multiple times, running in parallel to start jails, where quad1 is the master jail file. <br />
Quad1 could still be run as a shell script, but it would take a very long time for it to run completely so it’s not advisable; or you should break it down into smaller chunks (like quad1, quad2, quad3, etc)<br />
<br />
Here is a snip of (a 7.x version) quad1 from jail2:<br />
<br />
<pre>echo '## begin ##: projects.tw.com'<br />
mdconfig -a -t vnode -f /mnt/data1/69.55.230.46-col01213 -u 50<br />
fsck -Cy /dev/md50c<br />
mount /dev/md50c /mnt/data1/69.55.230.46-col01213-DIR<br />
mount -t devfs devfs /mnt/data1/69.55.230.46-col01213-DIR/dev<br />
devfs -m /mnt/data1/69.55.230.46-col01213-DIR/dev rule -s 3 applyset<br />
jail /mnt/data1/69.55.230.46-col01213-DIR projects.tw.com 69.55.230.46 /bin/sh /etc/rc<br />
echo '## end ##: projects.tw.com'</pre><br />
<br />
Cancelled jails are no longer commented out and stored in quad1, rather they’re moved to <tt>/usr/local/jail/rc.d/deprecated</tt> <br />
<br />
To start these jails, start the 4 ssh sessions as you would for a normal crash and then instead of running quad1-4, instead run startalljails in each window. IMPORTANT- before running startalljails you should make sure you ran preboot once as it will clear out all the lockfiles and enable startalljails to work properly.<br />
<br />
== Problems with the quad/safe files ==<br />
<br />
When you run the quad/safe files, there are two problems that can occur - either a particular system will hang during initialization, OR a system will spit out output to the screen, impeding your ability to do anything. Or both.<br />
<br />
First off, when you start a jail, you see output like this:<br />
<br />
<pre>Skipping disk checks ...<br />
adjkerntz[25285]: sysctl(put_wallclock): Operation not permitted<br />
Doing initial network setup:.<br />
ifconfig: ioctl (SIOCDIFADDR): permission denied<br />
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384<br />
Additional routing options: TCP keepalive=YESsysctl:<br />
net.inet.tcp.always_keepalive: Operation not permitted.<br />
Routing daemons:.<br />
Additional daemons: syslogd.<br />
Doing additional network setup:.<br />
Starting final network daemons:.<br />
ELF ldconfig path: /usr/lib /usr/lib/compat /usr/X11R6/lib /usr/local/lib<br />
a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout /usr/X11R6/lib/aout<br />
Starting standard daemons: inetd cron sshd sendmail sendmail-clientmqueue.<br />
Initial rc.i386 initialization:.<br />
Configuring syscons: blanktime.<br />
Additional ABI support:.<br />
Local package initialization:.<br />
Additional TCP options:.</pre><br />
<br />
Now, let's look at this line, near the end:<br />
<br />
Local package initialization:.<br />
<br />
This is where a list of daemons that are set to start at boot time willshow up. You might see something like:<br />
<br />
Local package initialization: mysqld apache sendmail sendmail-clientmqueue<br />
<br />
Or something like this:<br />
<br />
Local package initialization: postgres postfix apache<br />
<br />
The problem is that many systems (about 4-5 per machine) will hang on that line. Basically it will get to some of the way through the total daemons to be started:<br />
<br />
Local package initialization: mysqld apache<br />
<br />
and will just sit there. Forever.<br />
<br />
Fortunately, pressing ctrl-c will break out of it. Not only will it break out of it, but it will also continue on that same line and start the other daemons:<br />
<br />
Local package initialization: mysqld apache ^c sendmail-clientmqueue<br />
<br />
and then continue on to finish the startup, and then move to the next system to be started.<br />
<br />
So what does this mean? It means that if a machine crashes, and you start four screen-windows to run four quads or four safes, you need to periodically cycle between them and see if any systems are stuck at that point, causing their quad/safe file to hang. A good rule of thumb is, if you see a system at that point in the startup, give it another 100 seconds - if it is still at the exact same spot, hit ctrl-c. Its also a good idea to go back into the quad file (just before the first command in the jail startup block) and note that this jail tends to need a control-c or more time as follows:<br />
<pre>echo '### NOTE ### slow sendmail'<br />
echo '### NOTE ###: ^C @ Starting sendmail.'</pre><br />
<br />
'''NEVER''' hit ctrl-c repeatedly if you don't get an immediate response - that will cause the following jail’s startup commands to be aborted.<br />
<br />
A second problem that can occur is that a jail - maybe the first one in that particular quad/safe, maybe the last one, or maybe one in the middle, will start spitting out status or error messages from one of its init scripts. This is not a problem - basically, hit enter a few times and see if you get a prompt - if you do get a prompt, that means that the quad/safe script has already completed. Therefore it is safe to log out (and log out of the user that you su'd from) and then log back in (if necessary).<br />
<br />
The tricky thing is, if a system in the middle starts flooding with messages, and you hit enter a few times and don't get a prompt. Are you not getting a prompt because some subsequent system is hanging at the initialization, as we discussede above ? Or are you not getting a prompt because that quad file is currently running an fsck ? Usually you can tell by scrolling back in screen’s history to see what it was doing before you started getting the messages.<br />
<br />
If you don’t get clues from history, you have to use your judgement - instead of giving it 100 seconds to respond, perhaps give it 2-3 mins ... if you still get no response (no prompt) when you hit enter, hit ctrl-c. However, be aware that you might still be hitting ctrl-c in the middle of an fsck. This means you will get an error like "filesystem still marked dirty" and then the vnconfig for it will fail and so will the jail command, and the next system in the quad file will then start starting up.<br />
<br />
If this happens, just wait until the end of all the quad files have finished, and start that system manually.<br />
<br />
If things really get weird, like a screen flooded with errors, and you can't get a prompt, and ctrl-c does nothing, then you need to just eventually (give it ten mins or so) just kill that window with ctrl-p, then k, and then log in again and manually check which systems are now running and which aren't, and manually start up any that are not.<br />
<br />
Don't EVER risk running a particular quad/safe file a second time.<br />
If the quad/safe script gets executed twice, reboot the machine immediately.<br />
<br />
So, for all the above reasons, anytime a machine crashes and you run all the quads or all the safes, '''always''' check every jail afterwards to make sure it is running - even if you have no hangs or complications at all.<br />
Run this command:<br />
<br />
<tt>[[#jailpsall|jailpsall]]</tt> <br />
<br />
Note: [[#postboot|postboot]] also populates ipfw counts, so it '''should not be run multiple times''', use <tt>jailpsall</tt> for subsequent extensive ps’ing<br />
<br />
And make sure they all show as running. If one does not show as running, check its /etc/rc.conf file first to see if maybe it is using a different hostname first before starting it manually.<br />
<br />
One thing we have implemented to alleviate these startup hangs and noisy jails, is to put jail start blocks that are slow or hangy at the bottom of the safe/quad file. Further, for each bad jail we note in each quad/safe just before the start block something like:<br />
<br />
echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’<br />
<br />
That way we’ll be prepared to ^C when we see that message appear during the quad/safe startup process. If you observe a new, undocumented hang, '''after''' the quad/safe has finished, place a line similar to the above in the quad file, move the jail start block to the end of the file, then run [[#buildsafe|buildsafe]]<br />
<br />
<br />
== Recovering from a crash (FreeBSD) ==<br />
<br />
=== Diagnose whether you have a crash ===<br />
The most important thing is to get the machine and all jails back up as soon as possible. Note the time, you’ll need to create a crash log entry (Mgmt. -> Reference -> CrashLog). The first thing to do is head over to the [[Screen#Screen_Organization|serial console screen]] and see if there’s any kernel error messages output. Try to copy any messages (or just a sample of repeating messages) you see into the notes section of the crash log. If there are no messages, the machine may just be really busy- wait a bit (5-10min) to see if it comes back. If it's still pinging, odds are its very busy. Note, if you see messages about swap space exhausted, the server is obviously out of memory, however it may recover briefly enough for you to get a jtop in to see who's lauched a ton of procs (most likely) and then issue a quick jailkill to get it back under control.<br />
<br />
If it doesn't come back, or the messages indicate a fatal error, you will need to proceed with a power cycle (ctrl+alt+del will not work).<br />
<br />
=== Power cycle the server ===<br />
If this machine is not a Dell 2950 with a [[DRAC/RMM#DRAC|DRAC card]] (i.e. if you can’t ssh into the DRAC card (as root, using the standard root pass) and issue <br />
racadm serveraction hardreset<br />
then you will need someone at the data center power the macine off, wait 30 sec, then turn it back on. Make sure to re-attach via console:<br />
tip jailX<br />
immediately after power down. <br />
<br />
=== (Re)attach to the console ===<br />
Stay on the console the entire time during boot. As the BIOS posts- look out for the RAID card output- does everything look healthy? The output may be scrambled, look for "DEGRADED" or "FAILED". Once the OS starts booting you will be disconnected (dropped back to the shell on the console server) a couple times during the boot up. The reason you want to quickly re-attach is two-fold: 1. If you don’t reattach quickly then you won’t get any console output, 2. you want to be attached before the server ''potentially'' starts (an extensive) fsck. If you attach after the fsck begins, you’ll have seen no indication it started an fsck and the server will appear frozen during startup- no output, no response. <br />
<br />
IMPORTANT NOTE: on some older FreeBSD systems, there will be no output to the video (KVM) console as it boots up. The console output is redirected to the serial port ... so if a jail crashes, and you attach a kvm, the output during the bootup procedure will not be shown on the screen. However, when the bootup is done, you will get a login prompt on the screen and will be able to log in as normal. <tt>/boot/loader.conf</tt> is where serial console redirect output lives, so comment that if you want to catch output on kvm.<br />
On newer systems it sends most output to both locations. <br />
<br />
=== Assess the heath of the server ===<br />
Once the server boots up fully, you should be able to ssh in. Look around- make sure all the mounts are there and reporting the correct size/usage (i.e. /mnt/data1 /mnt/data2 /mnt/data3 - look in /etc/fstab to determine which mount points should be there), check to see if RAID mirrors are healthy. See [[RAID_Cards#Common_CLI_commands_.28megacli.29|megacli]], [[#aaccheck|aaccheck]]<br />
<br />
Before you start the jails, you need to run [[#preboot|preboot]]. This will do some assurance checks to make sure things are prepped to start the jails. Any issues that come out of preboot need to be addressed before starting jails.<br />
<br />
=== Start jails ===<br />
[[#Starting_jails:_Quad.2FSafe_Files|More on starting jails]]<br />
Customer jails (the VPSs) do not start up automatically at boot time. When a FreeBSD machines boots up, it boots up, and does nothing else. To start jails, we put the commands to start each jail into a shell script(s) and run the script(s). Jail startup is something that needs to be actively monitored, which is why we don’t just run the script automatically. <br />
<br />
In order to start jails, we run the quad files: quad1 quad2 quad3 and quad4 (on new systems there is only quad1). If the machine was cleanly rebooted- which wouldn't be the case if this was a crash, you may run the safe files (safe1 safe2 safe3 safe4) in lieu of quads. <br />
<br />
Open up 4 logins to the server (use the windows in [[Screen#Screen_Organization|a9]])<br />
In each of the 4 windows you will:<br />
<br />
If there is a [[#startalljails|startalljails]] script (and only quad1), run that command in each of the 4 windows. It will parse through the quad1 file and start each jail. Follow the instructions [[#Problems_with_the_quad.2Fsafe_files|here]] for monitoring startup. Note that you can be a little more lenient with jails that take awhile to start- startalljails will work around the slow jails and start the rest. As long as there aren't 4 jails which are "hung" during startup, the rest will get started eventually.<br />
-or-<br />
If there is no startalljails script, there will be multiple quad files. In each of the 4 windows, start each of the quads. i.e. start quad1 in window1, quad2 in window2 and so on. DO NOT start any quad twice. It will crash the server. If you accidentally do this, just jailkill all the jails which are in the quad and run the quad again. Follow the instructions here for monitoring quad startup.<br />
<br />
Note the time the last jail boots- this is what you will enter in the crash log.<br />
<br />
Save the crash log.<br />
<br />
=== Check to make sure all jails have started ===<br />
There's a simple script which will make sure all jails have started, and enter the ipfw counter rules: [[#postboot|postboot]] <br />
Run postboot, which will do a jailps on each jail it finds (excluding commented out jails) in the quad file(s). We're looking for 2 things:<br />
# systems spawning out of control or too many procs<br />
# jails which haven't started<br />
On 7.x and newer systems it will print out the problems (which jails haven't started) at the conclusion of postboot. <br />
On older systems you will need to watch closely to see if/when there's a problem, namely:<br />
<br />
[hostname] doesnt exist on this server<br />
<br />
When you get this message, it means one of 2 things:<br />
1. the jail really didn't start:<br />
When a jail doesn't start it usually boils down to a problem in the quad file. Perhaps the path name is wrong (data1 vs data2) or the name of the vn/mdfile is wrong. Once this is corrected, you will need to run the commands from the quad file manually, or you may use <tt>startjail <hostname></tt><br />
<br />
2. the customer has changed their hostname (and not told us) so their jail ''is'' running, just under a different hostname:<br />
On systems with jls, this is easy to rectify. First, get the customer info: <tt>g <hostname></tt><br />
Then look for the customer in jls: <tt>jls | grep <col0XXXX></tt><br />
From there you will see their new hostname- you should update that hostname in the quad file: don't forget to edit it on the <tt>## begin ##</tt> and <tt>## end ##</tt> lines, and in mgmt. <br />
On older systems without jls, this will be harder, you will need to look further to see their hostname- perhaps its in their /etc/rc.conf<br />
<br />
<br />
Once all jails are started, do some spot checks- try to ssh or browse to some customers, just to make sure things are really ok.<br />
<br />
== Adding disk to a 7.x/8.x jail ==<br />
or<br />
== Moving customer to a different drive (md) ==<br />
<br />
NOTE: this doesn’t apply to mx2 which uses gvinum. Use same procedure as 6.x<br />
NOTE: if you unmount before mdconfig, re-mdconfig (attach) then unmount then mdconfig -u again <br />
<br />
<br />
(parts to change/customize are <tt><span style="color:red">red</span></tt>)<br />
<br />
<br />
If someone wants more disk space, there’s a paste for it, send it to them (explains about downtime, etc).<br />
<br />
1. Figure out the space avail from <tt>js</tt>. Ideally, you want to put the customers new space on a different partition (and create the new md on the new partition). <br />
<br />
<br />
2. make a mental note of how much space they're currently using<br />
<br />
<br />
3. <tt>jailkill <hostname></tt> <br />
<br />
<br />
4. Umount it (including their devfs) but leave the md config’d (so if you use stopjail, you will have to re-mdconfig it)<br />
<br />
<br />
5. <tt>g <customerID></tt> to get the info (IP/cust#) needed to feed the new mdfile and mount name, and to see the current md device. <br />
<br />
<br />
6a. When there's enough room to place new system on an alternate, or the same drive:<br />
USE CAUTION not to overwrite (touch, mdconfig) existing md!!<br><br />
<tt>touch /mnt/data<span style="color:red">3/69.55.234.66-col01334</span><br><br />
mdconfig -a -t vnode -s 10g -f /mnt/data3/69.55.234.66-col01334 -u 97<br><br />
newfs /dev/md97</tt><br />
<br />
<br />
Optional- if new space is on a different drive, move the mount point directory AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
<tt>mount /dev/md<span style="color:red">97</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<br />
confirm you are mounted to /dev/md<span style="color:red">97</span> and space is correct:<br><br />
<tt>df .</tt> <br />
<br />
<br />
do the dump and pipe directly to restore:<br><br />
<tt>dump -0a -f - /dev/md<span style="color:red">1</span> | restore -r -f - <br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use <tt>df</tt> to confirm the restored data size matches the original usage figure<br />
<br />
<br />
md-unconfig old system:<br><br />
<tt>mdconfig -d -u <span style="color:red">1</span></tt><br />
<br />
<br />
archive old mdfile. <br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.237.26-col00241</span> /mnt/data<span style="color:red">1/old-col00241-mdfile-noarchive-20091211</span></tt><br />
<br />
<br />
edit quad (vq1) to point to new (/mnt/data<span style="color:red">3</span>) location AND new md number (md<span style="color:red">97</span>)<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
6b. When there's not enough room on an alternate partition, or on the same drive...but there is enough room if you were to remove the existing customer's space:<br />
<br />
<br />
mount backup nfs mounts:<br><br />
<tt>mbm</tt><br> <br />
(run <tt>df</tt> to confirm backup mounts are mounted)<br />
<br />
<br />
dump the customer to backup2 or backup1<br><br />
<tt>dump -0a -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> /dev/md<span style="color:red">1</span></tt><br><br />
(when complete WITHOUT errors, <tt>du</tt> the dump file to confirm it matches size, roughly, with usage)<br />
<br />
<br />
unconfigure and remove old mdfile<br><br />
<tt>mdconfig -d -u <span style="color:red">1</span><br><br />
rm /mnt/data<span style="color:red">1/69.55.237.26-col00241</span></tt><br><br />
(there should now be enough space to recreate your bigger system. If not, run sync a couple times)<br />
<br />
<br />
create the new system (ok to reuse old mdfile and md#):<br><br />
<tt>touch /mnt/data<span style="color:red">1/69.55.234.66-col01334</span><br><br />
mdconfig -a -t vnode -s <span style="color:red">10</span>g -f /mnt/data<span style="color:red">1/69.55.234.66-col01334</span> -u <span style="color:red">1</span><br><br />
newfs /dev/md<span style="color:red">1</span><br><br />
mount /dev/md<span style="color:red">1</span> /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> <br><br />
cd /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<br />
confirm you are mounted to /dev/md<span style="color:red">1</span> and space is correct:<br><br />
<tt>df .</tt> <br />
<br />
<br />
do the restore from the dumpfile on the backup server:<br><br />
<tt>restore -r -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> .<br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
umount nfs:<br><br />
<tt>mbu</tt><br />
<br />
<br />
If md# changed (or mount point), edit quad (<tt>vq1</tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new md number (md<span style="color:red">1</span>)<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
7. update disk (and dir if applicable) in mgmt screen<br />
<br />
<br />
8. update backup list AND move backups, if applicatble<br />
<br />
Ex: <tt>mvbackups <span style="color:red">69.55.237.26-col00241</span> jail<span style="color:red">9</span> data<span style="color:red">3</span></tt><br />
<br />
<br />
9. Optional: archive old mdfile<br />
<br />
<tt>mbm<br><br />
gzip -c old-col01588-mdfile-noarchive-20120329 > /deprecated/old-col01588-mdfile-noarchive-20120329.gz<br><br />
mbu<br><br />
rm old-col01588-mdfile-noarchive-20120329</tt><br />
<br />
== Adding disk to a 6.x jail (gvinum/gconcat) ==<br />
or<br />
== Moving customer to a different drive (gvinum/gconcat) ==<br />
<br />
(parts to change are <span style="color:red">highlighted</span>)<br />
<br />
<br />
If someone wants more disk space, there’s a paste for it, send it to them (explains about downtime, etc).<br />
<br />
<br />
1. Figure out the space avail from [[#js|js]]. Ideally, you want to put the customers new space on a different partition (and create the new md on the new partition). <br />
<br />
<br />
2. make a mental note of how much space they're currently using<br />
<br />
<br />
3. <tt>[[#stopjail|stopjail]] <hostname></tt> <br />
<br />
<br />
4. <tt>[[#g|g]] <customerID></tt> to get the info (IP/cust#) needed to feed the new mount name and existing volume/device. <br />
<br />
<br />
5a. When there's enough room to place new system on an alternate, or the same drive (using only UNUSED - including if it's in use by the system in question - gvinum volumes):<br />
<br />
<br />
Configure the new device:<br><br />
A. for a 2G system (single gvinum volume):<br><br />
<tt>bsdlabel -r -w /dev/gvinum/v<span style="color:red">123</span><br><br />
newfs /dev/gvinum/v<span style="color:red">123</span>a</tt><br><br />
<br />
-or- <br />
<br />
<br />
B. for a >2G system (create a gconcat volume):<br><br />
try to grab a contiguous block of gvinum volumes. gconcat volumes MAY NOT span drives (i.e. you cannot use a gvinum volume from data3 and a volume from data2 in the same gconcat volume). <br />
<br />
<tt>gconcat label <span style="color:red">v82-v84 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84</span><br><br />
bsdlabel -r -w /dev/concat/<span style="color:red">v82-v84</span><br><br />
newfs /dev/concat/<span style="color:red">v82-v84</span>a</tt><br />
<br />
<br />
Other valid gconcat examples:<br><br />
<tt>gconcat label v82-v84v109v112 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84 /dev/gvinum/v109 /dev/gvinum/v112<br><br />
gconcat label v82v83 /dev/gvinum/v82 /dev/gvinum/v83</tt> <br><br />
Note, long names will truncate: v144v145v148-v115 will truncate to v144v145v148-v1 (so you will refer to it as v144v145v148-v1 thereafter)<br />
<br />
<br />
Optional- if new volume is on a different drive, move the mount point directory (get the drive from js output) AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
<br />
confirm you are mounted to the device (<tt>/dev/gvinum/v<span style="color:red">123</span>a</tt> OR <tt>/dev/concat/<span style="color:red">v82-v84</span></tt>) and space is correct:<br><br />
A. <tt>mount /dev/gvinum/v<span style="color:red">123</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br><br />
-or-<br><br />
B. <tt>mount /dev/concat/<span style="color:red">v82-v84</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<tt>cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
df .</tt> <br />
<br />
<br />
do the dump and pipe directly to restore:<br><br />
<tt>dump -0a -f - /dev/gvinum/v<span style="color:red">1</span> | restore -r -f - <br><br />
rm restoresymtable</tt><br />
<br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
edit quad (<tt>vq<span style="color:red">1</span></tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new volume (<tt>/dev/gvinum/v<span style="color:red">123</span>a</tt> or <tt>/dev/concat/<span style="color:red">v82-v84</span>a</tt>) , run <tt>buildsafe</tt><br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
5b. When there's not enough room on an alternate partition, or on the same drive...but there is enough room if you were to remove the existing customer's space (i.e. if you want/need to reuse the existing gvinum volumes and add on more):<br />
<br />
<br />
mount backup nfs mounts:<br><br />
<tt>mbm</tt> <br><br />
(run df to confirm backup mounts are mounted)<br />
<br />
<br />
dump the customer to backup2 or backup1<br><br />
<tt>dump -0a -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> /dev/<span style="color:red">gconcat/v106-v107</span></tt><br><br />
(when complete WITHOUT errors, du the dump file to confirm it matches size, roughly, with usage)<br />
<br />
<br />
unconfigure the old gconcat volume<br><br />
list member gvinum volumes:<br />
<br />
<tt>gconcat list <span style="color:red">v106v107</span></tt><br />
<br />
<br />
Output will resemble:<br><br />
<pre>Geom name: v106v107<br />
State: UP<br />
Status: Total=2, Online=2<br />
Type: AUTOMATIC<br />
ID: 3530663882<br />
Providers:<br />
1. Name: concat/v106v107<br />
Mediasize: 4294966272 (4.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e2<br />
Consumers:<br />
1. Name: gvinum/sd/v106.p0.s0<br />
Mediasize: 2147483648 (2.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e3<br />
Start: 0<br />
End: 2147483136<br />
2. Name: gvinum/sd/v107.p0.s0<br />
Mediasize: 2147483648 (2.0G)<br />
Sectorsize: 512<br />
Mode: r1w1e3<br />
Start: 2147483136<br />
End: 4294966272</pre><br />
<br />
stop volume and clear members<br><br />
<tt>gconcat stop <span style="color:red">v106v107</span><br><br />
gconcat clear <span style="color:red">gvinum/sd/v106.p0.s0 gvinum/sd/v107.p0.s0</span></tt><br />
<br />
<br />
create new device- and its ok to reuse old/former members<br><br />
try to grab a contiguous block of gvinum volumes. gconcat volumes MAY NOT span drives (i.e. you cannot use a gvinum volume from data3 and a volume from data2 in the same gconcat volume). <br><br />
<tt>gconcat label <span style="color:red">v82-v84v106v107 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84 /dev/gvinum/v106 /dev/gvinum/v107</span> <br><br />
bsdlabel -r -w /dev/concat/<span style="color:red">v82-v84v106v107</span><br><br />
newfs /dev/concat/<span style="color:red">v82-v84v106v107</span>a</tt><br />
<br />
<br />
Optional- if new volume is on a different drive, move the mount point directory (get the drive from js output) AND use that directory in the mount and cd commands below:<br><br />
<tt>mv /mnt/data<span style="color:red">1/69.55.234.66-col01334-DIR</span> /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt><br />
<br />
confirm you are mounted to the device (/dev/concat/<span style="color:red">v82-v84v106v107</span>) and space is correct:<br><br />
<tt>mount /dev/concat/<span style="color:red">v82-v84v106v107</span>a /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span></tt> <br />
<br />
<tt>cd /mnt/data<span style="color:red">3/69.55.234.66-col01334-DIR</span> <br><br />
df .</tt> <br />
<br />
do the restore from the dumpfile on the backup server:<br><br />
<tt>restore -r -f /backup<span style="color:red">4/col00241.20120329.noarchive.dump</span> .<br><br />
rm restoresymtable</tt><br />
<br />
when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure<br />
<br />
<br />
edit quad (<tt>vq<span style="color:red">1</span></tt>) to point to new (/mnt/data<span style="color:red">3</span>) location AND new volume (<tt>/dev/concat/<span style="color:red">v82-v84v106v107</span>a</tt>), run buildsafe<br />
<br />
<br />
restart the jail:<br><br />
<tt>startjail <hostname></tt><br />
<br />
<br />
TODO: clean up/clear old gvin/gconcat vol<br />
<br />
<br />
6. update disk (and dir if applicable) in mgmt screen<br />
<br />
<br />
7. update backup list AND move backups, if applicatble<br />
<br />
<br />
Ex: <tt>mvbackups <span style="color:red">69.55.237.26-col00241</span> jail<span style="color:red">9</span> data<span style="color:red">3</span></tt><br />
<br />
<br />
<br />
DEPRECATED - steps to tack on a new gvin to existing gconcat- leads to corrupted fs<br />
bsdlabel -e /dev/concat/v82-v84<br />
<br />
To figure out new size of the c partition, multiply 4194304 by the # of 2G gvinum volumes and subtract the # of 2G volumes:<br />
10G: 4194304 * 5 – 5 = 20971515<br />
8G: 4194304 * 4 – 4 = 16777212<br />
6G: 4194304 * 3 – 3 = 12582909<br />
4G: 4194304 * 2 – 2 = 8388606<br />
<br />
To figure out the new size of the a partition, subtract 16 from the c partition:<br />
10G: 20971515 – 16 = 20971499<br />
8G: 16777212 – 16 = 16777196<br />
6G: 12582909 – 16 = 12582893<br />
4G: 8388606 – 16 = 8388590<br />
<br />
Orig:<br />
8 partitions:<br />
# size offset fstype [fsize bsize bps/cpg]<br />
a: 8388590 16 4.2BSD 2048 16384 28552<br />
c: 8388606 0 unused 0 0 # "raw" part, don't edit<br />
<br />
New:<br />
8 partitions:<br />
# size offset fstype [fsize bsize bps/cpg]<br />
a: 12582893 16 4.2BSD 2048 16384 28552<br />
c: 12582909 0 unused 0 0 # "raw" part, don't edit<br />
<br />
sync; sync<br />
<br />
growfs /dev/concat/v82-v84a<br />
<br />
fsck –fy /dev/concat/v82-v84a<br />
<br />
sync<br />
<br />
fsck –fy /dev/concat/v82-v84a<br />
<br />
(keep running fsck’s till NO errors)<br />
<br />
== Problems un-mounting - and with mount_null’s ==<br />
<br />
If you cannot unmount a filesystem, beacuse it says the filesystem is busy, it is because of three things:<br />
<br />
a) the jail is still running<br />
<br />
b) you are actually in that directory, even though the jail is stopped<br />
<br />
c) there are still dev, null_mount or linprocfs mount points mounted inside that directory.<br />
<br />
d) when trying to umount null_mounts that are really long and you get an error like “No such file or directory”, it’s an OS bug where the dir is truncated. No known fix<br />
<br />
e) there are still files open somewhere inside the dir. Use <tt>fstat | grep <cid></tt> to find the process that has files open<br />
<br />
f) Starting with 6.x, the jail mechanism does a poor job of keeping track of processes running in a jail and if it thinks there are still procs running, it will refuse to umount the disk. If this is happening you should see a low number in the #REF column when you run jls. In this case you ''can'' safely <tt>umount –f</tt> the mount. <br />
<br />
Please note -if you forcibly unmount a (4.x) filesystem that has null_mounts<br />
still mounted in it, the system '''will crash''' within 10-15 mins.<br />
<br />
== Misc jail Items ==<br />
<br />
We are overselling hard drive space on jail2, jail8, jail9, a couple jails on jail17, jail4, jail12 and jail18.<br />
Even though the vn file shows 4G size, it doesn’t actually occupy that amount of space on the disk. So be careful not to fill up drives where we’re overselling – use oversellcheck to confirm you’re not oversold by more than 10G.<br />
There are other truncated jails, they are generally noted in a the file on the root system: /root/truncated<br />
<br />
The act of moving a truncated vn to another system un-does the truncating- the truncated vn is filled with 0’s and it occupies physical disk space for which it’s configured. So, you should use dumpremote to preserve the truncation.<br />
<br />
= FreeBSD VPS Management Tools =<br />
<br />
These files are located in /usr/local/jail/rc.d and /usr/local/jail/bin<br />
<br />
== jailmake ==<br />
<br />
Applies to 7.x+ <br />
On older systems syntax differs, run jailmake once to see.<br />
<br />
Note: this procedure differs on mx2 which is 7.x but still uses gvinum<br />
<br />
# run js to figure out which md’s are in use, which disk has enough space, IP to put it on<br />
# use col00xxx for both hostnames if they don’t give you a hostname<br />
# copy over dir, ip and password to pending customer screen<br />
<br />
<tt>Usage: jailmake IP[,IP] CID disk[1|2|3] md# hostname shorthost ipfw# email [size in GB]</tt><br />
<br />
Ex: <br />
<br />
Jail2# jailmake 69.55.234.66 col01334 3 97 vps.bsd.it vps 1334 fb@bsd.it<br />
<br />
== jailps ==<br />
jailps [hostname]<br />
DEPRECATED FOR jps: displays processes belonging to/running inside a jail. The command<br />
takes one (optional) argument – the hostname of the jail you wish to query. If you don’t <br />
supply an argument, all processes on the machine are listed and grouped by jail. <br />
<br />
== jps ==<br />
jps [hostname]<br />
displays processes belonging to/running inside a jail. The command<br />
takes one (optional) argument – the hostname or ID of the jail you wish to query. <br />
<br />
== jailkill ==<br />
jailkill <hostname><br />
stops all process running in a jail.<br />
<br />
You can also run:<br />
jailkill <JID><br />
<br />
=== problems ===<br />
Occasionally you will hit an issue where jail will not kill off:<br />
<br />
<pre>jail9# jailkill www.domain.com<br />
www.domain.com .. killed: none<br />
jail9#</pre><br />
Because no processes are running under that hostname. You cannot use jailps.pl either:<br />
<br />
<pre>jail9# jailps www.domain.com<br />
www.domain.com doesn’t exist on this server<br />
jail9#</pre><br />
<br />
The reasons for this are usually:<br />
* the jail is no longer running<br />
<br />
* the jail's hostname has changed<br />
In this case, <br />
<br />
>=6.x: run a <tt>jls|grep <jail's IP></tt> to find the correct hostname, then update the quad file, then kill the jail.<br />
<br />
<6.x: the first step is to cat their /etc/rc.conf file to see if you can tell what they set the new hostname to. This very often works. For example:<br />
<br />
cat /mnt/data2/198.78.65.136-col00261-DIR/etc/rc.conf<br />
<br />
But maybe they set the hostname with the hostname command, and the original hostname is still in /etc/rc.conf.<br />
<br />
The welcome email clearly states that they should tell us if they change their hostname, so there is no problem in just emailing them and asking them what they set the new hostname to.<br />
<br />
Once you know the new hostname OR if a customer simply emails to inform you that they have set the hostname to something different, you need to edit the quad and safe files that their system is in to input the new hostname.<br />
<br />
However, if push comes to shove and you cannot find out the hostname from them or from their system, then you need to start doing some detective work.<br />
<br />
The easiest thing to do is run jailps looking for a hostname similar to their original hostname. Or you could get into the /bin/sh shell by running:<br />
<br />
/bin/sh<br />
<br />
and then looking at every hostname of every process:<br />
<br />
for f in `ls /proc` ; do cat /proc/$f/status ; done<br />
<br />
and scanning for a hostname that is either similar to their original hostname, or that you don't see in any of the quad safe files.<br />
<br />
This is very brute force though, and it is possible that catting every file in /proc is dangerous - I don't recommend it. A better thing would be to identify any processes that you know belong to this system – perhaps the reason you are trying to find this system is because they are running something bad - and just catting the status from only that PID.<br />
<br />
Somewhere there’s a jail where there may be 2 systems named www. Look at /etc/rc.conf and make sure they’re both really www. If they are, jailkill www, jailps www to make sure not running. Then immediately restart the other one, as the fqdn (as found from a rev nslookup)<br />
<br />
* on >=6.x the hostname may not yet be hashed:<br />
<pre>jail9 /# jls<br />
JID Hostname Path IP Address(es)<br />
1 bitnet.dgate.org /mnt/data1/69.55.232.50-col02094-DIR 69.55.232.50<br />
2 ns3.hctc.net /mnt/data1/69.55.234.52-col01925-DIR 69.55.234.52<br />
3 bsd1 /mnt/data1/69.55.232.44-col00155-DIR 69.55.232.44<br />
4 let2.bbag.org /mnt/data1/69.55.230.92-col00202-DIR 69.55.230.92<br />
5 post.org /mnt/data2/69.55.232.51-col02095-DIR 69.55.232.51 ...<br />
6 ns2 /mnt/data1/69.55.232.47-col01506-DIR 69.55.232.47 ...<br />
7 arlen.server.net /mnt/data1/69.55.232.52-col01171-DIR 69.55.232.52<br />
8 deskfood.com /mnt/data1/69.55.232.71-col00419-DIR 69.55.232.71<br />
9 mirage.confluentforms.com /mnt/data1/69.55.232.54-col02105-DIR 69.55.232.54 ...<br />
10 beachmember.com /mnt/data1/69.55.232.59-col02107-DIR 69.55.232.59<br />
11 www.agottem.com /mnt/data1/69.55.232.60-col02109-DIR 69.55.232.60<br />
12 sdhobbit.myglance.org /mnt/data1/69.55.236.82-col01708-DIR 69.55.236.82<br />
13 ns1.jnielsen.net /mnt/data1/69.55.234.48-col00204-DIR 69.55.234.48 ...<br />
14 ymt.rollingegg.net /mnt/data2/69.55.236.71-col01678-DIR 69.55.236.71<br />
15 verse.unixlore.net /mnt/data1/69.55.232.58-col02131-DIR 69.55.232.58<br />
16 smcc-mail.org /mnt/data2/69.55.232.68-col02144-DIR 69.55.232.68<br />
17 kasoutsuki.w4jdh.net /mnt/data2/69.55.232.46-col02147-DIR 69.55.232.46<br />
18 dili.thium.net /mnt/data2/69.55.232.80-col01901-DIR 69.55.232.80<br />
20 www.tekmarsis.com /mnt/data2/69.55.232.66-col02155-DIR 69.55.232.66<br />
21 vps.yoxel.net /mnt/data2/69.55.236.67-col01673-DIR 69.55.236.67<br />
22 smitty.twitalertz.com /mnt/data2/69.55.232.84-col02153-DIR 69.55.232.84<br />
23 deliver4.klatha.com /mnt/data2/69.55.232.67-col02160-DIR 69.55.232.67<br />
24 nideffer.com /mnt/data2/69.55.232.65-col00412-DIR 69.55.232.65<br />
25 usa.hanyuan.com /mnt/data2/69.55.232.57-col02163-DIR 69.55.232.57<br />
26 daifuku.ppbh.com /mnt/data2/69.55.236.91-col01720-DIR 69.55.236.91<br />
27 collins.greencape.net /mnt/data2/69.55.232.83-col01294-DIR 69.55.232.83<br />
28 ragebox.com /mnt/data2/69.55.230.104-col01278-DIR 69.55.230.104<br />
29 outside.mt.net /mnt/data2/69.55.232.72-col02166-DIR 69.55.232.72<br />
30 vps.payneful.ca /mnt/data2/69.55.234.98-col01999-DIR 69.55.234.98<br />
31 higgins /mnt/data2/69.55.232.87-col02165-DIR 69.55.232.87 ...<br />
32 ozymandius /mnt/data2/69.55.228.96-col01233-DIR 69.55.228.96<br />
33 trusted.realtors.org /mnt/data2/69.55.238.72-col02170-DIR 69.55.238.72<br />
34 jc1.flanderous.com /mnt/data2/69.55.239.22-col01504-DIR 69.55.239.22<br />
36 guppylog.com /mnt/data2/69.55.238.73-col00036-DIR 69.55.238.73<br />
40 haliohost.com /mnt/data2/69.55.234.41-col01916-DIR 69.55.234.41 ...<br />
41 satyr.jorge.cc /mnt/data1/69.55.232.70-col01963-DIR 69.55.232.70<br />
jail9 /# jailkill satyr.jorge.cc<br />
ERROR: jail_: jail "satyr,jorge,cc" not found<br />
</pre><br />
<br />
Note how it's saying <tt>satyr,jorge,cc</tt> is not found, and not <tt>satyr.jorge.cc</tt>. <br />
<br />
The jail subsystem tracks things using comma-delimited hostnames. That is created every few hours:<br />
<br />
jail9 /# crontab -l<br />
0 0,6,12,18 * * * /usr/local/jail/bin/sync_jail_names<br />
<br />
So if we run this manually:<br />
jail9 /# /usr/local/jail/bin/sync_jail_names<br />
<br />
Then kill the jail:<br />
jail9 /# jailkill satyr.jorge.cc<br />
successfully killed: satyr,jorge,cc</pre><br />
<br />
It worked.<br />
<br />
<br />
If you ever see this when trying to kill a jail:<br />
<br />
<pre># jailkill e-scribe.com<br />
killing JID: 6 hostname: e-scribe.com<br />
3 procs running<br />
3 procs running<br />
3 procs running<br />
3 procs running<br />
...</pre><br />
<br />
<tt>[[#jailkill|jailkill]]</tt> probably got lost trying to kill off the jail. Just ctrl-c the jailkill process, then run a jailps on the hostname, and <tt>kill -9</tt> any process which is still running. Keep running jailps and <tt>kill -9</tt> till all processes are gone.<br />
<br />
== jailpsall ==<br />
jailpsall<br />
will run a jailps on all jails configured in the quad files (this is different from<br />
jailps with no arguments as it won’t help you find a “hidden” system)<br />
<br />
== jailpsw ==<br />
jailpsw<br />
will run a jailps with an extra -w to provide wider output<br />
<br />
== jt (>=7.x) ==<br />
jt<br />
displays the top 20 processes on the server (the top 20 processes from top) and <br />
which jail owns them. This is very helpful for determining who is doing what when<br />
the server is very busy.<br />
<br />
== jtop (>=7.x) ==<br />
jtop<br />
a wrapper for top displaying processes on the server and which jail owns them. Constantly updates, like top. <br />
<br />
== jtop (<7.x) ==<br />
jtop<br />
displays the top 20 processes on the server (the top 20 processes from top) and <br />
which jail owns them. This is very helpful for determining who is doing what when<br />
the server is very busy.<br />
<br />
== stopjail ==<br />
stopjail <hostname> [1]<br />
this will jailkill, umount and vnconfig –u a jail. If passed an optional 2nd<br />
argument, it will not exit before umounting and un-vnconfig’ing in the event<br />
jailkill returns no processes killed. This is useful if you just want to umount<br />
and vnconfig –u a jail you’ve already killed. It is intelligent in that it won’t <br />
try to umount or vnconfig –u if it’s not necessary.<br />
<br />
== startjail ==<br />
startjail <hostname><br />
this will start vnconfig, mount (including linprocfs and null-mounts), and start a jail.<br />
Essentially, it reads the jail’s relevant block from the right quad file and executes it.<br />
It is intelligent in that it won’t try to mount or vnconfig if it’s not necessary.<br />
<br />
== jpid ==<br />
jpid <pid><br />
displays information about a process – including which jail owns it.<br />
It’s the equivalent of running cat /proc/<pid>/status<br />
<br />
== canceljail ==<br />
canceljail <hostname> [1]<br />
this will stop a jail (the equivalent of stopjail), check for backups (offer to remove them <br />
from the backup server and the backup.config), rename the vnfile, remove the dir, and <br />
edit quad/safe. If passed an optional 2nd argument, it will not exit upon failing to kill<br />
and processes owned by the jail. This is useful if you just want to cancel a jail which <br />
is already stopped.<br />
<br />
== jls ==<br />
jls [-v]<br />
Lists all jails running:<br />
<pre>JID #REF IP Address Hostname Path<br />
101 135 69.55.224.148 mail.pc9.org /mnt/data2/69.55.224.148-col01034-DIR</pre><br />
<br />
#REF is the number of references or procs(?) running<br />
<br />
Running with -v will give you all IPs assigned to each jail (7.2 up)<br />
<pre>JID #REF Hostname Path IP Address(es)<br />
101 139 mail.pc9.org /mnt/data2/69.55.224.148-col01034-DIR 69.55.224.14869.55.234.85</pre><br />
<br />
== startalljails ==<br />
startalljails<br />
7.2+ only. This will parse through quad1 and start all jails. It utilizes lockfiles so it won’t try to start a jail more than once- therefore multiple instances can be running in parallel without fear of starting a jail twice. If a jail startup gets stuck, you can ^C without fear of killing the script. IMPORTANT- before running startalljails you should make sure you ran preboot once as it will clear out all the lockfiles and enable startalljails to work properly.<br />
<br />
== aaccheck.sh ==<br />
aaccheck.sh<br />
displayes the output of container list and task list from aaccli<br />
<br />
== backup ==<br />
backup<br />
backup script called nightly to update jail scripts and do customer backups<br />
<br />
== buildsafe ==<br />
buildsafe<br />
creates safe files based on quads (automatically removing the fsck’s). This will destructively overwrite safe files<br />
<br />
== checkload.pl ==<br />
checkload.pl<br />
this was intended to be setup as a cronjob to watch processes on a jail when the load <br />
rises above a certain level. Not currently in use.<br />
<br />
== checkprio.pl ==<br />
checkprio.pl<br />
will look for any process (other than the current shell’s csh, sh, sshd procs) with a non-normal priority and normalize it<br />
<br />
== diskusagemon == <br />
diskusagemon <mount point> <1k blocks><br />
watches a mount point’s disk use, when it reaches the level specified in the 2nd argument,<br />
it exits. This is useful when doing a restore and you want to be paged as it’s nearing completion.<br />
Best used as: <tt>diskusagemon /asd/asd 1234; pagexxx</tt><br />
<br />
== dumprestore ==<br />
dumprestore <dumpfile><br />
this is a perl expect script which automatically enters ‘1’ and ‘y’. It seems to cause restore to fail<br />
to set owner permissions on large restores.<br />
<br />
== g ==<br />
g <search><br />
greps the quad/safe files for the given search parameter<br />
<br />
== gather.pl ==<br />
gather.pl<br />
gathers up data about jails configured and writes to a file. Used for audits against the db<br />
<br />
== gb ==<br />
gb <search><br />
greps backup.config for the given search parameter<br />
<br />
== gbg ==<br />
gbg <search><br />
greps backup.config for the given search parameter and presents just the directories (for clean pasting)<br />
<br />
== ipfwbackup ==<br />
ipfwbackup<br />
writes ipfw traffic count data to a logfile<br />
<br />
== ipfwreset ==<br />
ipfwreset<br />
writes ipfw traffic count data to a logfile and resets counters to 0<br />
<br />
== js ==<br />
js<br />
output varies by OS version, but generally provides information about the base jail:<br />
- which vn’s are in use<br />
- disk usage<br />
- info about the contents of quads<br />
- the # of inodes represented by the jails contained in the group (133.2 in the example below), and how many jails per data mount, as well as subtotals<br />
- ips bound to the base machine but not in use by a jail<br />
- free gvinum volumes, or unused vn’s or used md’s<br />
<br />
<pre>/usr/local/jail/rc.d/quad1:<br />
/mnt/data1 133.2 (1)<br />
/mnt/data2 1040.5 (7)<br />
total 1173.7 (8)<br />
/usr/local/jail/rc.d/quad2:<br />
/mnt/data1 983.4 (6)<br />
total 983.4 (6)<br />
/usr/local/jail/rc.d/quad3:<br />
/mnt/data1 693.4 (4)<br />
/mnt/data2 371.6 (3)<br />
total 1065 (7)<br />
/usr/local/jail/rc.d/quad4:<br />
/mnt/data1 466.6 (3)<br />
/mnt/data2 882.2 (5)<br />
total 1348.8 (8)<br />
/mnt/data1: 2276.6 (14)<br />
/mnt/data2: 2294.3 (15)<br />
<br />
Available IPs:<br />
69.55.230.11 69.55.230.13 69.55.228.200<br />
<br />
Available volumes:<br />
v78 /mnt/data2 2G<br />
v79 /mnt/data2 2G<br />
v80 /mnt/data2 2G</pre> <br />
<br />
== load.pl ==<br />
load.pl<br />
feeds info to load mrtg - executed by inetd.<br />
<br />
== makevirginjail ==<br />
makevirginjail<br />
Only on some systems, makes an empty jail (doesn't do restore step)<br />
<br />
== mb == <br />
mb <mount|umount><br />
(nfs) mounts and umounts dirs to backup2. Shortcuts are mbm and mbu to mount and unmount. <br />
<br />
== notify.sh ==<br />
notify.sh<br />
emails reboot@johncompanies.com – intended to be called at boot time to alert us to a machine which panics and reboots and isn’t caught by bb or castle.<br />
<br />
== orphanedbackupwatch ==<br />
orphanedbackupwatch<br />
looks for directories on backup2 which aren’t configured in backup.config and offers to delete them<br />
<br />
== postboot ==<br />
postboot<br />
to be run after a machine reboot and quad/safe’s are done executing. It will:<br />
* do chmod 666 on each jail’s /dev/null<br />
* add ipfw counts<br />
* run jailpsall (so you can see if a configured jail isn’t running)<br />
<br />
== preboot ==<br />
preboot<br />
to be run before running quad/safe – checks for misconfigurations: <br />
* a jail configured in a quad but not a safe<br />
* a jail is listed more than once in a quad<br />
* the ip assigned to a jail isn’t configured on the machine<br />
* alias numbering skips in the rc.conf (resulting in the above)<br />
* orphaned vnfile's that aren't mentioned in a quad/safe<br />
* ip mismatches between dir/vnfile name and the jail’s ip<br />
* dir/vnfiles's in quad/safe that don’t exist <br />
<br />
== quadanalyze.pl ==<br />
quadanalyze.pl<br />
called by js, produces the info (seen above with js explanation) about the contents of quad (inode count, # of jails, etc.)<br />
<br />
== rsync.backup ==<br />
rsync.backup<br />
does customer backups (relies on backup.config)<br />
<br />
== taskdone ==<br />
taskdone<br />
when called will email support@johncompanies.com with the hostname of the machine from which it was executed as the subject<br />
<br />
== topten ==<br />
topten<br />
summarizes the top 10 traffic users (called by ipfwreset)<br />
<br />
== trafficgather.pl ==<br />
trafficgather.pl [yy-mm]<br />
sends a traffic usage summary by jail to support@johncomapnies.com and payments@johncompanies.com. Optional arguments are year and month (must be in the past). If not passed, assumes last month. Relies on traffic logs created by ipfwreset and ipfwbackup<br />
<br />
== trafficwatch.pl ==<br />
trafficwatch.pl<br />
checks traffic usage and emails support@johncomapnies.com when a jail reaches the warning level (35G) and the limit (40G). We really aren’t using this anymore now that we have netflow.<br />
<br />
== trafstats ==<br />
trafstats<br />
writes ipfw traffic usage info by jail to a file called jc_traffic_dump in each jail’s / dir<br />
<br />
== truncate_jailmake ==<br />
truncate_jailmake<br />
a version of jailmake which creates truncated vnfiles.<br />
<br />
== vb ==<br />
vb<br />
the equivalent of: <tt>vi /usr/local/jail/bin/backup.config</tt><br />
<br />
== vs (freebsd) ==<br />
vs<n><br />
the equivalent of: <tt>vi /usr/local/jail/rc.d/safe<n></tt><br />
<br />
vq<n><br />
the equivalent of: vi /usr/local/jail/rc.d/quad<n><br />
<br />
== dumpremote ==<br />
dumpremote <user@machine> </remote/location/file-dump> <vnX><br />
ex: dumpremote user@10.1.4.117 /mnt/data3/remote.echoditto.com-dump 7<br />
this will dump a vn filesystem to a remote machine and location<br />
<br />
== oversellcheck ==<br />
oversellcheck<br />
displays how much a disk is oversold or undersold taking into account truncated vn files. Only for use on 4.x systems<br />
<br />
== mvbackups (freebsd) ==<br />
mvbackups <dir> (1.1.1.1-col00001-DIR) <target_machine> (jail1) <target_dir> (data1)<br />
moves backups from one location to another on the backup server, and provides you with option to remove entries from current backup.config, and simple paste command to add the config to backup.config on the target server<br />
<br />
== jailnice ==<br />
jailnice <hostname><br />
applies <tt>renice 19 [PID]</tt> and <tt>rtprio 31 –[PID]</tt> to each process in the given jail<br />
<br />
== dumpremoterestore ==<br />
dumpremoterestore <device> <ip of target machine> <dir on target machine><br />
ex: dumpremoterestore /dev/vn51 10.1.4.118 /mnt/data2/69.55.239.45-col00688-DIR<br />
dumps a device and restores it to a directory on a remote machine. Requires that you enable root ssh on the <br />
remote machine.<br />
<br />
== psj ==<br />
psj<br />
shows just the procs running on the base system – a ps auxw but without jail’d procs present<br />
<br />
== perc5iraidchk ==<br />
perc5iraidchk<br />
checks for degraded arrays on Dell 2950 systems with Perc5/6 controllers<br />
<br />
== perc4eraidchk ==<br />
perc4eraidchk<br />
checks for degraded arrays on Dell 2850 systems with Perc4e/Di controllers<br />
<br />
= Virtuozzo VPS =<br />
<br />
Note: We use VEID (Virtual Environment ID) and CTID (Container ID) interchangably. Similarly, VE and CT. They mean the same thing.<br />
VZPP = VirtuoZzo Power Panel (the control panel for each CT)<br />
<br />
All linux systems exist in /vz, /vz1 or /vz2 - since each linux machine holds roughly 60-90 customers, there will be roughly 30-45 in each partition.<br />
<br />
The actual filesystem of the system in question is in:<br />
<br />
/vz(1-2)/private/(VEID)<br />
<br />
Where VEID is the identifier for that system - an all-numeric string larger than 100.<br />
<br />
The actual mounted and running systems are in the corresponding:<br />
<br />
/vz(1-2)/root/(VEID)<br />
<br />
But we rarely interact with any system from this mount point.<br />
<br />
You should never need to touch the root portion of their system – however you can traverse their filesystem by going to <tt>/vz(1-2)/private/(VEID)/root</tt> (<tt>/vz(1-2)/private/(VEID)/fs/root</tt> on 4.x systems) the root of their filesystem is in that directory, and their entire system is underneath that.<br />
<br />
Every VE has a startup script in <tt>/etc/sysconfig/vz-scripts</tt> (which is symlinked as <tt>/vzconf</tt> on all systems) - the VE startup script is simply named <tt>(VEID).conf</tt> - it contains all the system parameters for that VE:<br />
<br />
<pre># Configuration file generated by vzsplit for 60 VE<br />
# on HN with total amount of physical mem 2011 Mb<br />
<br />
VERSION="2"<br />
CLASSID="2"<br />
<br />
ONBOOT="yes"<br />
<br />
KMEMSIZE="8100000:8200000"<br />
LOCKEDPAGES="322:322"<br />
PRIVVMPAGES="610000:615000"<br />
SHMPAGES="33000:34500"<br />
NUMPROC="410:415"<br />
PHYSPAGES="0:2147483647"<br />
VMGUARPAGES="13019:2147483647"<br />
OOMGUARPAGES="13019:2147483647"<br />
NUMTCPSOCK="1210:1215"<br />
NUMFLOCK="107:117"<br />
NUMPTY="19:19"<br />
NUMSIGINFO="274:274"<br />
TCPSNDBUF="1800000:1900000"<br />
TCPRCVBUF="1800000:1900000"<br />
OTHERSOCKBUF="900000:950000"<br />
DGRAMRCVBUF="200000:200000"<br />
NUMOTHERSOCK="650:660"<br />
DCACHE="786432:818029"<br />
NUMFILE="7500:7600"<br />
AVNUMPROC="51:51"<br />
IPTENTRIES="155:155"<br />
DISKSPACE="4194304:4613734"<br />
DISKINODES="400000:420000"<br />
CPUUNITS="1412"<br />
QUOTAUGIDLIMIT="2000"<br />
VE_ROOT="/vz1/root/636"<br />
VE_PRIVATE="/vz1/private/636"<br />
NAMESERVER="69.55.225.225 69.55.230.3"<br />
OSTEMPLATE="vzredhat-7.3/20030305"<br />
VE_TYPE="regular"<br />
IP_ADDRESS="69.55.225.229"<br />
HOSTNAME="textengine.net"</pre><br />
<br />
<br />
As you can see, the hostname is set here, the disk space is set here, the number of inodes, the number of files that can be open, the number of tcp sockets, etc. - all are set here.<br />
<br />
In fact, everything that can be set on this customer system is set in this conf file.<br />
<br />
<br />
All interaction with the customer system is done with the VEID. You start the system by running:<br />
<br />
vzctl start 999<br />
<br />
You stop it by running:<br />
<br />
vzctl stop 999<br />
<br />
You execute commands in it by running:<br />
<br />
vzctl exec 999 df -k<br />
<br />
You enter into it, via a root-shell backdoor with:<br />
<br />
vzctl enter 999<br />
<br />
and you set parameters for the system, while it is still running, with:<br />
<br />
vzctl set 999 --diskspace 6100000:6200000 --save<br />
<br />
<tt>vzctl</tt> is the most commonly used command - we have aliased <tt>v</tt> to <tt>vzctl</tt> since we use it so often. We’ll continue to use <tt>vzctl</tt> in our examples, but feel free to use just <tt>v</tt>.<br />
<br />
Let's say the user wants more diskspace. You can cat their conf file and see:<br />
<br />
DISKSPACE="4194304:4613734"<br />
<br />
So right now they have 4gigs of space. You can then change it to 6 with:<br />
<br />
vzctl set 999 --diskspace 6100000:6200000 --save<br />
<br />
IMPORTANT: all issuances of the vzctl set command need to end with <tt>–save</tt> - if they don't, the setting will be set, but it will not be saved to the conf file, and they will not have those settings next time they boot.<br />
<br />
All of the tunables in the conf file can be set with the vzctl set command. Note that in the conf file, and on the vzctl set command line, we always issue two numbers seperated by a colon - that is because we are setting the hard and soft limits. Always set the hard limit slightly above the soft limit, as you see it is in the conf file for all those settings.<br />
<br />
There are also things you can set with `<tt>vzctl set</tt>` that are not in the conf file as settings, per se. For instance, you can add IPs:<br />
<br />
vzctl set 999 --ipadd 10.10.10.10 --save<br />
<br />
or multiple IPs:<br />
<br />
vzctl set 999 --ipadd 10.10.10.10 --ipadd 10.10.20.30 --save<br />
<br />
or change the hostname:<br />
<br />
vzctl set 999 --hostname www.example.com --save<br />
<br />
You can even set the nameservers:<br />
<br />
vzctl set 999 --nameserver 198.78.66.4 --nameserver 198.78.70.180 --save<br />
<br />
Although you probably will never do that.<br />
<br />
You can disable a VPS from being started (by VZPP or reboot) (4.x):<br />
<br />
vzctl set 999 --disabled yes --save <br />
<br />
You can disable a VPS from being started (by VZPP or reboot) (<=3.x):<br />
<br />
vzctl set 999 --onboot=no --save <br />
<br />
You can disable a VPS from using his control panel:<br />
<br />
vzctl set 999 --offline_management=no --save <br />
<br />
You can suspend a VPS, so it can be resumed in the same state it was in when it was stopped (4.x):<br />
<br />
vzctl suspend 999<br />
<br />
and to resume it:<br />
<br />
vzctl resume 999<br />
<br />
to see who owns process:<br />
vzpid <PID><br />
<br />
to mount up an unmounted ve:<br />
vzctl mount 827<br />
<br />
<br />
To see network stats for CT's:<br />
<pre># vznetstat<br />
VEID Net.Class Output(bytes) Input(bytes)<br />
-----------------------------------------------<br />
24218 1 484M 39M<br />
24245 1 463M 143M<br />
2451 1 2224M 265M<br />
2454 1 2616M 385M<br />
4149 1 125M 68M<br />
418 1 1560M 34M<br />
472 1 1219M 315M<br />
726 1 628M 317M<br />
763 1 223M 82M<br />
771 1 4234M 437M<br />
-----------------------------------------------<br />
Total: 13780M 2090M</pre><br />
<br />
<br />
One thing that sometimes comes up on older systems that we created with smaller defaults is that the system would run out of inodes. The user will email and say they cannot create any more files or grow any files larger, but they will also say that they are not out of diskspace ... they are running:<br />
<br />
df -k<br />
<br />
and seeing how much space is free - and they are not out of space. They are most likely out of inodes - which they would see by running:<br />
<br />
df -i<br />
<br />
So, the first thing you should do is enter their system with:<br />
<br />
vzctl enter 999<br />
<br />
and run: <tt>df -i</tt><br />
<br />
to confirm your theory. Then exit their system. Then simply cat their conf file and see what their inodes are set to (probably 200000:200000, since that was the old default on the older systems) and run:<br />
<br />
vzctl set 999 --diskinodes 400000:400000 --save<br />
<br />
If they are not out of inodes, then a good possibility is that they have maxed out their numfile configuration variable, which controls how many files they can have in their system. The current default is 7500 (which nobody has ever hit), but the old default was as low as 2000, so you would run something like:<br />
<br />
vzctl set 999 --numfile 7500:7500 --save<br />
<br />
----<br />
<br />
You cannot start or stop a VE if your pwd is its private (/vz/private/999) or root (/vz/root/999) directories, or anywhere below them.<br />
<br />
<br />
== Recovering from a crash (linux) ==<br />
<br />
=== Diagnose whether you have a crash ===<br />
The most important thing is to get the machine and all ve’s back up as soon as possible. Note the time, you’ll need to create a crash log entry (Mgmt. -> Reference -> CrashLog). The first thing to do is head over to the [[Screen#Screen_Organization|serial console screen]] and see if there’s any kernel error messages output. Try to copy any messages (or just a sample of repeating messages) you see into the notes section of the crash log – these will also likely need to be sent to virtuozzo for interpretation. If the messages are spewing too fast, hit ^O + H to start a screen log dump which you can ob1182.pts-38.bb serve after the machine is rebooted. Additionally, if the machine is responsive, you can get a trace to send to virtuozzo by hooking up a kvm and entering these 3 sequences:<br />
<pre>alt+print screen+m<br />
alt+print screen+p<br />
alt+print screen+t</pre><br />
<br />
If there are no messages, the machine may just be really busy- wait a bit (5-10min) to see if it comes back. If it's still pinging, odds are its very busy. If it doesn't come back, or the messages indicate a fatal error, you will need to proceed with a power cycle (ctrl+alt+del will not work).<br />
<br />
=== Power cycle the server ===<br />
If this machine is not a Dell 2950 with a [[DRAC/RMM#DRAC|DRAC card]] (i.e. if you can’t ssh into the DRAC card and issue racadm serveraction hardreset, then you will need someone at the data center to power the macine off, wait 30 sec, then turn it back on. Make sure to re-attach via console (<tt>tip virtxx</tt>) immediately after power down. <br />
<br />
=== (Re)attach to the console ===<br />
Stay on the console the entire time during boot. As the BIOS posts- look out for the RAID card output- does everything look healthy? The output may be scrambled, look for "DEGRADED" or "FAILED". Once the OS starts booting you will be disconnected (dropped back to the shell on the console server) a couple times during the boot up. The reason you want to quickly re-attach is two-fold: 1. If you don’t reattach quickly then you won’t get any console output, 2. you want to be attached before the server ''potentially'' starts (an extensive) fsck. If you attach after the fsck begins, you’ll have seen no indication it started an fsck and the server will appear frozen during startup- no output, no response. <br />
<br />
=== Start containers/VE's/VPSs ===<br />
When the machine begins to start VE’s, it’s safe to leave the console and login via ssh. All virts should be set to auto start all the VEs after a crash. Further, most (newer) virts are set to “fastboot” it’s VE’s (to find out, do:<br />
grep -i fast /etc/sysconfig/vz <br />
and look for <tt>VZFASTBOOT=yes</tt>). If this was set prior to the machine’s crash (setting it after the machine boots will not have any effect until the vz service is restarted) it will start each ve as fast as possible, in serial, then go thru each VE (serially), shutting it down running a vzquota (disk usage) check, then bringing it back up. The benefit is that all VE’s are brought up quickly (within 15min or so depending on the #), the downside is a customer watching closely will notice 2 outages – 1st the machine crash, 2nd their quota check (which will be a much shorter downtime- on the order of a few minutes). <br />
<br />
Where “fastboot” is not set to yes (i.e on quar1), vz will start them consecutively, checking the quotas one at a time, and the 60th VE may not start until an hour or two later - this is not acceptable.<br />
<br />
The good news is, if you run vzctl start for a VE that is already started, you will simply get an error: <tt>VE is already started</tt>. Further, if you attempt to vzctl start a VE that is in the process of being started, you will simply get an error: unable to lock VE. So, there is no danger in simply running scripts to start smaller sets of VEs. If the system is not autostarting, then there is no issue, and even if it does, when it conflicts, one process (yours or the autostart) will lose, and just move on to the next one.<br />
<br />
A script has been written to assist with ve starts: [[#startvirt.pl|startvirt.pl]] which will start 6 ve’s at once until there are no more left. If startvirt.pl is used on a system where “fastboot” was on, it will circumvent the fastboot for ve’s started by startvirt.pl – they will go through the complete quota check before starting- therefore this is not advisable when a system has crashed. When a system is booted cleanly, and there's no need for vzquota checks, then startvirt.pl is safe and advisable to run.<br />
<br />
=== Make sure all containers are running ===<br />
You can quickly get a feel for how many ve’s are started by running:<br />
<br />
<pre>[root@virt4 log]# vs<br />
VEID 16066 exist mounted running<br />
VEID 16067 exist mounted running<br />
VEID 4102 exist mounted running<br />
VEID 4112 exist mounted running<br />
VEID 4116 exist mounted running<br />
VEID 4122 exist mounted running<br />
VEID 4123 exist mounted running<br />
VEID 4124 exist mounted running<br />
VEID 4132 exist mounted running<br />
VEID 4148 exist mounted running<br />
VEID 4151 exist mounted running<br />
VEID 4155 exist mounted running<br />
VEID 42 exist mounted running<br />
VEID 432 exist mounted running<br />
VEID 434 exist mounted running<br />
VEID 442 exist mounted running<br />
VEID 450 exist mounted running<br />
VEID 452 exist mounted running<br />
VEID 453 exist mounted running<br />
VEID 454 exist mounted running<br />
VEID 462 exist mounted running<br />
VEID 463 exist mounted running<br />
VEID 464 exist mounted running<br />
VEID 465 exist mounted running<br />
VEID 477 exist mounted running<br />
VEID 484 exist mounted running<br />
VEID 486 exist mounted running<br />
VEID 490 exist mounted running</pre><br />
<br />
So to see how many ve’s have started:<br />
<pre>[root@virt11 root]# vs | grep running | wc -l<br />
39</pre><br />
<br />
And to see how many haven’t:<br />
<pre>[root@virt11 root]# vs | grep down | wc -l<br />
0</pre><br />
<br />
And how many we should have running:<br />
<pre>[root@virt11 root]# vs | wc -l<br />
39</pre><br />
<br />
Another tool you can use to see which ve’s have started, among other things is [[#vzstat|vzstat]]. It will give you CPU, memory, and other stats on each ve and the overall system. It’s a good thing to watch as ve’s are starting (note the VENum parameter, it will tell you how many have started):<br />
<br />
<pre>4:37pm, up 3 days, 5:31, 1 user, load average: 1.57, 1.68, 1.79<br />
VENum 40, procs 1705: running 2, sleeping 1694, unint 0, zombie 9, stopped 0<br />
CPU [ OK ]: VEs 57%, VE0 0%, user 8%, sys 7%, idle 85%, lat(ms) 412/2<br />
Mem [ OK ]: total 6057MB, free 9MB/54MB (low/high), lat(ms) 0/0<br />
Swap [ OK ]: tot 6142MB, free 4953MB, in 0.000MB/s, out 0.000MB/s<br />
Net [ OK ]: tot: in 0.043MB/s 402pkt/s, out 0.382MB/s 4116pkt/s<br />
Disks [ OK ]: in 0.002MB/s, out 0.000MB/s<br />
<br />
VEID ST %VM %KM PROC CPU SOCK FCNT MLAT IP<br />
1 OK 1.0/17 0.0/0.4 0/32/256 0.0/0.5 39/1256 0 9 69.55.227.152<br />
21 OK 1.3/39 0.1/0.2 0/46/410 0.2/2.8 23/1860 0 6 69.55.239.60<br />
133 OK 3.1/39 0.1/0.3 1/34/410 6.3/2.8 98/1860 0 0 69.55.227.147<br />
263 OK 2.3/39 0.1/0.2 0/56/410 0.3/2.8 34/1860 0 1 69.55.237.74<br />
456 OK 17/39 0.1/0.2 0/100/410 0.1/2.8 48/1860 0 11 69.55.236.65<br />
476 OK 0.6/39 0.0/0.2 0/33/410 0.1/2.8 96/1860 0 10 69.55.227.151<br />
524 OK 1.8/39 0.1/0.2 0/33/410 0.0/2.8 28/1860 0 0 69.55.227.153<br />
594 OK 3.1/39 0.1/0.2 0/45/410 0.0/2.8 87/1860 0 1 69.55.239.40<br />
670 OK 7.7/39 0.2/0.3 0/98/410 0.0/2.8 64/1860 0 216 69.55.225.136<br />
691 OK 2.0/39 0.1/0.2 0/31/410 0.0/0.7 25/1860 0 1 69.55.234.96<br />
744 OK 0.1/17 0.0/0.5 0/10/410 0.0/0.7 7/1860 0 6 69.55.224.253<br />
755 OK 1.1/39 0.0/0.2 0/27/410 0.0/2.8 33/1860 0 0 192.168.1.4<br />
835 OK 1.1/39 0.0/0.2 0/19/410 0.0/2.8 5/1860 0 0 69.55.227.134<br />
856 OK 0.3/39 0.0/0.2 0/13/410 0.0/2.8 16/1860 0 0 69.55.227.137<br />
936 OK 3.2/52 0.2/0.4 0/75/410 0.2/0.7 69/1910 0 8 69.55.224.181<br />
1020 OK 3.9/39 0.1/0.2 0/60/410 0.1/0.7 55/1860 0 8 69.55.227.52<br />
1027 OK 0.3/39 0.0/0.2 0/14/410 0.0/2.8 17/1860 0 0 69.55.227.83<br />
1029 OK 1.9/39 0.1/0.2 0/48/410 0.2/2.8 25/1860 0 5 69.55.227.85<br />
1032 OK 12/39 0.1/0.4 0/80/410 0.0/2.8 41/1860 0 8 69.55.227.90</pre><br />
<br />
When you are all done, you will want to make sure that all the VEs really did get started, run vs one more time.<br />
<br />
Note the time all ve’s are back up and enter that into and save the crash log entry.<br />
<br />
Occasionally, a ve will not start automatically. The most common reason for a ve not to come up normally is the ve was at it’s disk limit before the crash, and will not start since they’re over the limit. To overcome this, set the disk space to current usage level (the system will give this to you when it fails to start), start the ve, then re-set the disk space back to the prior level. Lastly, contact the customer to let them know they’re out of disk (or allocate more disk if they're entitled to more).<br />
<br />
== Hitting performance barriers and fixing them ==<br />
<br />
There are multiple modes virtuozzo offers to allocate resources to a ve. We utilize 2: SLM and UBC parameters<br />
On our 4.x systems, we use all SLM – it’s simpler to manage and understand. There are a few systems on virt19/18 that may also use SLM. Everything else uses UBC. <br />
You can tell a SLM ve by:<br />
<br />
SLMMODE="all"<br />
<br />
in their conf file. <br />
<br />
TODO: detail SLM modes and parameters.<br />
<br />
If someone is in SLM mode and they hit memory resource limits, they simply need to upgrade to more memory.<br />
<br />
The following applies to everyone else (UBC).<br />
<br />
Customers will often email and say that they are getting out of memory errors - a common one is "cannot fork" ... basically, anytime you see something odd like this, it means they are hitting one of their limits that is in place in their conf file.<br />
<br />
The conf file, however, simply shows their limits - how do we know what they are currently at ?<br />
<br />
The answer is a file called v - this file contains the current status (and peaks) of their performance settings, and also counts how many times they have hit the barrier. The output of the file looks like this:<br />
<br />
<pre>764: kmemsize 384113 898185 8100000 8200000 0<br />
lockedpages 0 0 322 322 0<br />
privvmpages 1292 7108 610000 615000 0<br />
shmpages 270 528 33000 34500 0<br />
dummy 0 0 0 0 0<br />
numproc 8 23 410 415 0<br />
physpages 48 5624 0 2147483647 0<br />
vmguarpages 0 0 13019 2147483647 0<br />
oomguarpages 641 6389 13019 2147483647 0<br />
numtcpsock 3 21 1210 1215 0<br />
numflock 1 3 107 117 0<br />
numpty 0 2 19 19 0<br />
numsiginfo 0 4 274 274 0<br />
tcpsndbuf 0 80928 1800000 1900000 0 <br />
tcprcvbuf 0 108976 1800000 1900000 0<br />
othersockbuf 2224 37568 900000 950000 0<br />
dgramrcvbuf 0 4272 200000 200000 0<br />
numothersock 3 9 650 660 0<br />
dcachesize 53922 100320 786432 818029 0<br />
numfile 161 382 7500 7600 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
numiptent 4 4 155 155 0</pre><br />
<br />
The first column is the name of the counter in question - the same names we saw in the systems conf file. The second column is the _current_ value of that counter, the third column is the max that that counter has ever risen to, the fourth column is the soft limit, and the fifth column is the hard limit (which is the same as the numbers in that systems conf file).<br />
<br />
The sixth number is the failcount - how many times the current usage has risen to hit the barrier. It will increase as soon as the current usage hits the soft limit.<br />
<br />
The problem with /proc/user_beancounters is that it actually contains that set of data for every running VE - so you can't just cat /proc/user_beancounters - it is too long and you get info for every other running system.<br />
<br />
You can vzctl enter the system and run:<br />
<br />
vzctl enter 9999<br />
cat /proc/user_beancounters<br />
<br />
inside their system, and you will just see the stats for their particular system, but entering their system every time you want to see it is combersome.<br />
<br />
So, I wrote a simple script called "vzs" which simply greps for the VEID, and spits out the next 20 or so lines (however many lines there are in the output, I forget) after it. For instance:<br />
<br />
<pre># vzs 765:<br />
765: kmemsize 2007936 2562780 8100000 8200000 0<br />
lockedpages 0 8 322 322 0<br />
privvmpages 26925 71126 610000 615000 0<br />
shmpages 16654 16750 33000 34500 0<br />
dummy 0 0 0 0 0<br />
numproc 41 57 410 415 0<br />
physpages 1794 49160 0 2147483647 0<br />
vmguarpages 0 0 13019 2147483647 0<br />
oomguarpages 4780 51270 13019 2147483647 0<br />
numtcpsock 23 37 1210 1215 0<br />
numflock 17 39 107 117 0<br />
numpty 1 3 19 19 0<br />
numsiginfo 0 6 274 274 0<br />
tcpsndbuf 22240 333600 1800000 1900000 0<br />
tcprcvbuf 0 222656 1800000 1900000 0<br />
othersockbuf 104528 414944 900000 950000 0<br />
dgramrcvbuf 0 4448 200000 200000 0<br />
numothersock 73 105 650 660 0<br />
dcachesize 247038 309111 786432 818029 0<br />
numfile 904 1231 7500 7600 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
dummy 0 0 0 0 0<br />
numiptent 4 4 155 155 0</pre><br />
<br />
That showed us just the portion of /proc/user_beancounters for system 765.<br />
<br />
When you run the vzs command, always add a : after the VEID.<br />
<br />
So, if a customer complains about some out of memory errors, or no more files, or no more ptys, or just has an unspecific complain about processes dying, etc., the very first thing you need to do is check their beancounters with vzs. Usually you will spot an item that has a high failcount and needs to be upped.<br />
<br />
At that point you could simply up the counter with `vzctl set`. Generally pick a number 10-20% higher than the old one, and make the hard limit slightly larger than the the soft limit. However our systems now come in several levels and those levels have more/different memory allocations. If someone is complaining about something other than a memory limit (pty, numiptent, numflock), it’s generally safe to increase it, at least to the same level as what’s in the /vzconf/4unlimited file on the newest virt. If someone is hitting a memory limit, first make sure they are given what they deserve:<br />
<br />
(refer to mgmt -> payments -> packages)<br />
<br />
To set those levels, you use the [[#setmem|setmem]] command. <br />
<br />
The alternate (DEPRECATED) method would be to use one of 3 commands:<br />
256 <veid><br />
300 <veid><br />
384 <veid><br />
512 <veid><br />
<br />
If the levels were not right (you’d run vzs <veid> before and after to see the effect) tell the customer they’ve been adjusted and be done with it. If the levels were right, tell the customer they must upgrade to a higher package, tell them how to see level (control panel) and that they can reboot their system to escape this lockup contidion.<br />
<br />
Customers can also complain that their site is totally unreachable, or complain that it is down ... if the underlying machine is up, and all seems well, you may notice in the beancounters that network-specific counters are failing - such as numtcpsock, tcpsndbuf or tcprcvbuf. This will keep them from talking on the network and make it seem like their system is down. Again, just up the limits and things should be fine.<br />
<br />
On virts 1-4, you should first look at the default settings for that item on a later virt, such as virt 8 - we have increased the defaults a lot since the early machines. So, if you are going to up a counter on virt2, instead of upping it by 10-20%, instead up it to the new default that you see on virt8.<br />
<br />
== Moving a VE to another virt (migrate/migrateonline) ==<br />
<br />
This will take a while to complete - and it is best to do this at night when the load is light on both machines.<br />
<br />
There are different methods for this, depending on which version of virtuozzo is installed on the src. and dst. virt. <br />
To check which version is running: <br />
[root@virt12 private]# cat /etc/virtuozzo-release<br />
Virtuozzo release 2.6.0<br />
<br />
Ok, let's say that the VE is 1212, and vital stats are:<br />
<pre>[root@virt12 sbin]# vc 1212<br />
VE_ROOT="/vz1/root/1212"<br />
VE_PRIVATE="/vz1/private/1212"<br />
OSTEMPLATE="fedora-core-2/20040903"<br />
IP_ADDRESS="69.55.229.84"<br />
TEMPLATES="devel-fc2/20040903 php-fc2/20040813 mysql-fc2/20040812 postgresql-fc2/20040813 mod_perl-fc2/20040812 mod_ssl-fc2/20040811 jre-fc2/20040823 jdk-fc2/20040823 mailman-fc2/20040823 analog-fc2/20040824 proftpd-fc2/20040818 tomcat-fc2/20040823 usermin-fc2/20040909 webmin-fc2/20040909 uw-imap-fc2/20040830 phpBB-fc2/20040831 spamassassin-fc2/20040910 PostNuke-fc2/20040824 sl-webalizer-fc2/20040<br />
818"<br />
<br />
[root@virt12 sbin]# vzctl exec 1212 df -h<br />
Filesystem Size Used Avail Use% Mounted on<br />
/dev/vzfs 4.0G 405M 3.7G 10% /</pre><br />
<br />
From this you can see that he’s using (and will minimally need free on the dst server) ~400MB, and he’s running on a Fedora 2 template, version 20040903. He’s also got a bunch of other templates installed. It’s is '''vital''' that '''all''' these templates exist on the dst system. To confirm that, on the dst system run:<br />
<br />
For < 3.0:<br />
<br />
<pre>[root@virt14 private]# vzpkgls | grep fc2<br />
devel-fc2 20040903<br />
PostNuke-fc2 20040824<br />
analog-fc2 20040824<br />
awstats-fc2 20040824<br />
bbClone-fc2 20040824<br />
jdk-fc2 20040823<br />
jre-fc2 20040823<br />
mailman-fc2 20040823<br />
mod_frontpage-fc2 20040816<br />
mod_perl-fc2 20040812<br />
mod_ssl-fc2 20040811<br />
mysql-fc2 20040812<br />
openwebmail-fc2 20040817<br />
php-fc2 20040813<br />
phpBB-fc2 20040831<br />
postgresql-fc2 20040813<br />
proftpd-fc2 20040818<br />
sl-webalizer-fc2 20040818<br />
spamassassin-fc2 20040910<br />
tomcat-fc2 20040823<br />
usermin-fc2 20040909<br />
uw-imap-fc2 20040830<br />
webmin-fc2 20040909<br />
[root@virt14 private]# vzpkgls | grep fedora<br />
fedora-core-1 20040121 20040818<br />
fedora-core-devel-1 20040121 20040818<br />
fedora-core-2 20040903<br />
[root@virt14 private]#</pre><br />
<br />
For these older systems, you can simply match up the date on the template. <br />
<br />
For >= 3.0:<br />
<br />
<pre>[root@virt19 /vz2/private]# vzpkg list<br />
centos-5-x86 2008-01-07 22:05:57<br />
centos-5-x86 devel<br />
centos-5-x86 jre<br />
centos-5-x86 jsdk<br />
centos-5-x86 mod_perl<br />
centos-5-x86 mod_ssl<br />
centos-5-x86 mysql<br />
centos-5-x86 php<br />
centos-5-x86 plesk9<br />
centos-5-x86 plesk9-antivirus<br />
centos-5-x86 plesk9-api<br />
centos-5-x86 plesk9-atmail<br />
centos-5-x86 plesk9-backup<br />
centos-5-x86 plesk9-horde<br />
centos-5-x86 plesk9-mailman<br />
centos-5-x86 plesk9-mod-bw<br />
centos-5-x86 plesk9-postfix<br />
centos-5-x86 plesk9-ppwse<br />
centos-5-x86 plesk9-psa-firewall<br />
centos-5-x86 plesk9-psa-vpn<br />
centos-5-x86 plesk9-psa-fileserver<br />
centos-5-x86 plesk9-qmail<br />
centos-5-x86 plesk9-sb-publish<br />
centos-5-x86 plesk9-vault<br />
centos-5-x86 plesk9-vault-most-popular<br />
centos-5-x86 plesk9-watchdog</pre><br />
<br />
On these newer systems, it's difficult to tell whether the template on the dst matches exactly the src. Just cause a centos-5-x86 is listed on both servers doesn't mean all the same packages are there on the dst. To truly know, you must perform a sample rsync:<br />
<br />
rsync -avn /vz/template/centos/5/x86/ root@10.1.4.61:/vz/template/centos/5/x86/<br />
<br />
if you see a ton of output from the dry run command, then clearly there are some differences. You may opt to let the rsync complete (without running in dry run mode) the only downside is you've now used up more space on the dst and also the centos template will be a mess with old and new data- it will be difficult if not impossible to undo (if someday we wanted to reclaim the space).<br />
<br />
If you choose to merge templates, you should closely inspect the dry run output. You should also take care to exclude anything in the /config directory. For example:<br />
<br />
rsync -av -e ssh --stats --exclude=x86/config /vz/template/ubuntu/10.04/ root@10.1.4.62:/vz/template/ubuntu/10.04/<br />
<br />
Which will avoid this directory and contents:<br />
<pre>[root@virt11 /vz2/private]# ls /vz/template/ubuntu/10.04/x86/config*<br />
app os</pre><br />
<br />
This is important to avoid since the config may differ on the destination and we are really only interested in making sure the pacakges are there, not overwriting a newer config with an older one.<br />
<br />
If the dst system was missing a template, you have 2 choices: <br />
# put the missing template on the dst system. 2 choices here: <br />
## Install the template from rpm (found under backup2: /mnt/data4/vzrpms/distro/) or <br />
## rsync over the template (found under /vz/template) - see above<br />
# put the ve on a system which has all the proper templates<br />
<br />
=== pre-seeding a migration ===<br />
<br />
When migrating a customer (or when doing many) depending on how much data you have to transfer, it can take some time. Further, it can be difficult to gauge when a migration will complete or how long it will take. To help speed up the process and get a better idea about how long it will take you can pre-transfer a customer's data to the destination server. If done correctly, vzmigrate will see the pre-transferred data and pick up where you left off, having much less to transfer (just changed/new files). <br />
<br />
We believe vzmigrate uses rsync to do it's transfer. Therefore not only can you use rsync to do a pre-seed, you can also run rsync to see what is causing a repeatedly-failing vzmigrate to fail. <br />
<br />
There's no magic to a pre-seed, you just need to make sure it's named correctly.<br />
<br />
Given:<br />
<br />
source: /vz1/private/1234<br />
<br />
and you want to migrate to /vz2 on the target system, your rsync would look like:<br />
<br />
rsync -av /vz1/private/1234/ root@x.x.x.x:/vz2/private/1234.migrated/<br />
<br />
After running that successful rsync, the ensuing migrateonline (or migrate) will take much less time to complete- depending on the # of files to be analyzed and the # of changed files. In any case, it'll be much much faster than had you just started the migration from scratch.<br />
<br />
Further, as we discuss elsewhere in this topic, a failed migration can be moved from <tt>/vz/private/1234</tt> to <tt>/vz/private/1234.migrated</tt> on the destination if you want to restart a failed migration. This should '''only''' be done if the migration failed and the CT is not running on the destination HN.<br />
<br />
=== migrateonline intructions: src >=3.x -> dst>=3.x ===<br />
<br />
A script called [[#migrateonline|migrateonline]] was written to handle this kind of move. It is basically a wrapper for <tt>vzmigrate</tt> – vzmigrate is a util to seamlessly- as no no reboot of the ve necessary- move a ve from one host to another. This wrapper was initially written cause vz virtuozzo version 2.6.0 has a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables, causing problems when the ve was started up on the dst system. [[#migrate|migrate]] mitigates that. Since it makes multiple ssh connections to the dst virt, it’s a good idea to put the pub key for the src virt in the authorized_keys file on the dst virt. In addition, migrateonline emails ve owners when their migration starts and stops. For this to happen they need to put email addresses (on a single line, space delimited) in a file on their system: /migrate_notify. If the optional target dir is not specified, the ve will be moved to the same private/root location as it was on the src virt. Note: <tt>migrate</tt> is equivalent to <tt>migrateonline</tt>, but will <tt>migrate</tt> a ve AND restart it in the process.<br />
<br />
<pre>[root@virt12 sbin]# migrateonline<br />
usage: /usr/local/sbin/migrateonline <ip of node migrating to> <veid> [target dir: vz | vz1 | vz2]<br />
[root@virt12 sbin]# migrateonline 10.1.4.64 1212 vz<br />
starting to migrate 1212 at Sat Mar 26 22:40:38 PST 2005<br />
Turning off offline_management<br />
Saved parameters for VE 1212<br />
migrating with no start on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
clearing the arp cache<br />
now going to 10.1.4.64 and clear cache and starting<br />
starting it<br />
Delete port redirection<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
finished migrating 1212 at Sat Mar 26 22 22:52:01 PST 2005<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine.<br />
<br />
If they had backups, use the mvbackups command to move their backups to the new server:<br />
<br />
mvbackups 1212 virt14 vz<br />
<br />
Rename the ve<br />
<br />
[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/migrated-1212<br />
<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/old-1212-migrated-20120404-noarchive<br />
<br />
Update the customer’s systems in mgmt to reflect the new path and server.<br />
<br />
<br />
IF migrateonline does not work, you can try again using simply migrate- this will result in a brief reboot for the ve.<br />
Before you try again, make sure of a few things:<br />
<br />
Depending on where in the migration died, there may be partial data on the dst system in 1 of 2 places:<br />
(given the example above)<br />
<br />
/vz/private/1212<br />
<br />
or <br />
<br />
/vz/private/1212.migrated<br />
<br />
before you run migrate again, you'll want to rename so that all data is in <br />
1212.migrated:<br />
<br />
mv /vz/private/1212 /vz/private/1212.migrated<br />
<br />
this way, it will pick up where it left off and transfer only new files.<br />
<br />
Likewise, if you want to speed up a migration, you can pre-seed the dst as follows:<br />
<br />
[root@virt12 sbin]# rsync -avSH /vz/private/1212/ root@10.1.4.64:/vz/private/1212.migrated/<br />
<br />
then when you run migrate or migrateonline, it will only need to move the changed files- the migration will complete quickly<br />
<br />
=== migrate manually (if migrate/online fails) ===<br />
<br />
Lets say for whatever reason the migration fails. You can always move someone manually:<br />
<br />
1. stop ve: <br><br />
v stop 1234<br />
<br />
2. copy over data<br><br />
rsync -avSH /vz/private/1234/ root@1.1.1.1:/vzX/private/1234/<br />
<br />
NOTE: if you've previously seeded the data (run rsync while the VE was up/running), and this is a subsequent rsync, make sure the last rsync you do (while the VE is not running, has the --delete option in the rsync)<br />
<br />
3. copy over conf<br><br />
scp /vzconf/1234.conf root@1.1.1.1:/vzconf<br />
<br />
4. on dst, edit the conf to reflect the right vzX dir<br><br />
vi /vzconf/1234.conf<br />
<br />
5. on src remove the IPs<br><br />
ipdel 1234 2.2.2.2 3.3.3.3<br />
<br />
6. on dst add IPs <br><br />
ipadd 1234 2.2.2.2 3.3.3.3<br />
<br />
7. on dst, start ve: <br><br />
v start 1324<br />
<br />
8. cancel, then archive ve on src per above instrs.<br />
<br />
=== migrate src=2.6.0 -> dst>=2.6.0, or mass-migration with customer notify ===<br />
<br />
A script called <tt>migrate</tt> was written to handle this kind of move. It is basically a wrapper for vzmigrate – vzmigrate is a util to seamlessly move a ve from one host to another. This wrapper was initially written cause vz virtuozzo version 2.6.0 has a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables, causing problems when the ve was started up on the dst system. migrate mitigates that. Since it makes multiple ssh connections to the dst virt, it’s a good idea to put the pub key for the src virt in the authorized_keys file on the dst virt. In addition, migrate emails ve owners when their migration starts and stops. For this to happen they need to put email addresses (on a single line, space delimited) in a file on their system: /migrate_notify. If the optional target dir is not specified, the ve will be moved to the same private/root location as it was on the src virt. Note: migrateonline is equivalent to migrate, but will migrate a ve from one 2.6 '''kernel''' machine to another 2.6 kernel machine without restarting the ve.<br />
<br />
<pre>[root@virt12 sbin]# migrate<br />
usage: /usr/local/sbin/migrate <ip of node migrating to> <veid> [target dir: vz | vz1 | vz2]<br />
[root@virt12 sbin]# migrate 10.1.4.64 1212 vz<br />
starting to migrate 1212 at Sat Mar 26 22:40:38 PST 2005<br />
Turning off offline_management<br />
Saved parameters for VE 1212<br />
migrating with no start on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
clearing the arp cache<br />
now going to 10.1.4.64 and clear cache and starting<br />
starting it<br />
Delete port redirection<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
finished migrating 1212 at Sat Mar 26 22 22:52:01 PST 2005<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine (backup2 or mail).<br />
Cancel the ve (first we have to rename things which migrate changed so cancelve will find them):<br />
<br />
[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/1212.conf<br />
<br />
On 2.6.1 you’ll also have to move the private area:<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/1212<br />
<br />
<pre>[root@virt12 sbin]# cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, [[#cancelve|cancelve]] would offer to remove them. You want to say '''no''' to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
To move everyone off a system, you’d do:<br />
for f in `vl`; do migrate <ip> $f; done<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
=== vzmigrate: src=2.6.1 -> dst>=2.6.0 ===<br />
<br />
This version of vzmigrate works properly with regard to handling ips. It will not notify ve owners of moves as in the above example. Other than that it’s essentially the same.<br />
<br />
<pre>[root@virt12 sbin]# vzmigrate 10.1.4.64 -r no 1212:1212:/vz/private/1212:/vz/root/1212<br />
migrating on 10.1.4.64<br />
Connection to destination HN (10.1.4.64) is successfully established<br />
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...<br />
Syncing private area '/vz1/private/1212'<br />
- 100% |*************************************************|<br />
done<br />
Successfully completed<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es) to pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
Starting VE ...<br />
VE is mounted<br />
Adding port redirection to VE(1): 4643 8443<br />
Adding IP address(es): 69.55.229.84<br />
Hostname for VE set: fourmajor.com<br />
File resolv.conf was modified<br />
VE start in progress...<br />
[root@virt12 sbin]#</pre><br />
<br />
Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine (backup2 or mail).<br />
Cancel the ve (first we have to rename things which vzmigrate changed so cancelve will find them):<br />
<br />
<pre>[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/1212.conf<br />
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/1212<br />
<br />
[root@virt12 sbin]# cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, <tt>cancelve</tt> would offer to remove them. You want to say no to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
=== Instructions for src=2.5.x ===<br />
<br />
First, go to the private dir:<br />
<br />
cd /vz1/private/<br />
<br />
Stop the VE - make sure it stops totally cleanly.<br />
<br />
vzctl stop 1212<br />
<br />
Then you’d use vemove - a script written to copy over the config, create tarballs of the ve’s data on the destination virt, and cancel the ve on the source system (in this example we’re going to put a ve that was in /vz1/private on the src virt, in /vz/private on the dst virt):<br />
<br />
<pre>[root@virt12 sbin]# vemove<br />
ERROR: Usage: vemove veid target_ip target_path_dir<br />
[root@virt12 sbin]# vemove 1212 10.1.4.64 /vz/private/1212<br />
tar cfpP - 1212 --ignore-failed-read | (ssh -2 -c arcfour 10.1.4.64 "split - -b 1024m /vz/private/1212.tar" )<br />
scp /vzconf/1212.conf 10.1.4.64:/vzconf<br />
cancelve 1212<br />
v stop 1212<br />
v set 1212 --offline_management=no --save<br />
Delete port redirection<br />
Deleting IP address(es) from pool: 69.55.229.84<br />
Saved parameters for VE 1212<br />
mv /vzconf/1212.conf /vzconf/deprecated-1212<br />
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414<br />
don't forget to remove firewall rules and domains!<br />
[root@virt12 sbin]#</pre><br />
<br />
Note: if the system had backups, cancelve would offer to remove them. You want to say no to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.<br />
Then go to backup2 and move the dirs. So you’d do something like this:<br />
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/<br />
<br />
We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. <br />
If moving to the same drive, you can safely preserve hardlinks and move all files with:<br />
sh<br />
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done<br />
<br />
When you are done, go to /vz/private on the dst virt you will have files like this:<br />
<br />
<pre>1212.taraa<br />
1212.tarab<br />
1212.tarac</pre><br />
<br />
Each one 1024m (or less, for the last one) in size.<br />
<br />
on the dst server and run:<br />
<br />
cat 1212.tar?? | tar xpPBf -<br />
<br />
and after 20 mins or so it will be totally untarred. Now since the conf<br />
file is already there, you can go ahead and start the system.<br />
<br />
vzctl start 1212<br />
<br />
Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.<br />
<br />
NOTE: you MUST tar the system up using the virtuozzo version of tar that<br />
is on all the virt systems, and further you MUST untar the tarball with<br />
the virtuozzo tar, using these options: `<tt>tar xpPBf -</tt>`<br />
<br />
If you tar up an entire VE and move it to a non-virtuozzo machine, that is<br />
ok, and you can untar it there with normal tar commands, but do not untar<br />
it and then repack it with a normal tar and expect it to work - you need<br />
to use virtuozzo tar commands on virtuozzo tarballs to make it work.<br />
<br />
The backups are sort of an exception, since we are just (usually)<br />
restoring user data that was created after we gave them the system, and<br />
therefore has nothing to do with magic symlinks or vz-rpms, etc.<br />
<br />
== Traffic accounting on linux ==<br />
<br />
DEPRECATED - all tracking is done via bwdb now. This is how we used to track traffic.<br />
<br />
TODO: update for diff versions of vz<br />
<br />
Unlike FreeBSD, where we have to add firewall count rules to the system to count the traffic, on virtuozzo counts the traffic for us. You an see the current traffic stats by running `vznetstat`:<br />
<br />
<pre># vznetstat<br />
VEID Net.Class Output(bytes) Input(bytes)<br />
-----------------------------------------------<br />
24218 1 484M 39M<br />
24245 1 463M 143M<br />
2451 1 2224M 265M<br />
2454 1 2616M 385M<br />
4149 1 125M 68M<br />
418 1 1560M 34M<br />
472 1 1219M 315M<br />
726 1 628M 317M<br />
763 1 223M 82M<br />
771 1 4234M 437M<br />
-----------------------------------------------<br />
Total: 13780M 2090M</pre><br />
<br />
As you can see the VEID is on a line with the in and out bytes. So, we simply run a cron job:<br />
<br />
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /root/vztrafdump.sh<br />
<br />
Just like we do on FreeBSD - this one goes through all the VEs in /vz/private and greps the line from vznetstat that matches them and dumps it in /jc_traffic_dump on their system. Then it does it again for all the VEs in /vz1/private. It is important to note that vznetstat runs only once, and the grepping is done from a temporary file that contains that output - we do this because running vznetstat once for each VE that we read out of /vz/private and /vz1/private would take way too long and be too intensive.<br />
<br />
You do not need to do anything to facilitate this other than make sure that that cron job is running - the vznetstat counters are always running, and any new VEs that are added to the system will be accounted for automatically.<br />
<br />
Traffic resetting no longer works with vz 2.6, so we disable the vztrafdump.sh on those virts.<br />
<br />
== Watchdog script ==<br />
<br />
On some of the older virts, we have a watchdog running that kills procs that are deemed bad per the following:<br />
<br />
/root/watchdog from quar1<br />
<br />
<pre>if echo $line | awk '{print $(NF-3)}' | grep [5-9]...<br />
then<br />
# 50-90%<br />
if echo $line | awk '{print $(NF-1)}' | grep "...:.."<br />
then<br />
# running for > 99min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
if echo $line | awk '{print $(NF-1)}' | grep "....m"<br />
then<br />
# running for > 1000min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
fi<br />
<br />
if echo $line | awk '{print $(NF-3)}' | grep [1-9]...<br />
then<br />
# running for 10-90 percent<br />
if echo $line | awk '{print $NF}' | egrep 'cfusion|counter|vchkpw'<br />
then<br />
<br />
if echo $line | awk '{print $(NF-1)}' | grep "[2-9]:.."<br />
then<br />
# between 2-9min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
elif echo $line | awk '{print $(NF-1)}' | grep "[0-9][0-9]:.."<br />
then<br />
# up to 99min<br />
echo $line >> /root/wd.log<br />
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log<br />
kill -9 $pid<br />
<br />
fi<br />
fi<br />
fi</pre><br />
<br />
<br />
== Misc Linux Items ==<br />
<br />
We are overselling hard drive space ... when you configure a linux system with a certain amount of disk space (the default is 4gigs) you do not actually use up 4gigs of space on the system. The diskspace setting for a user is simply a cap, and they only use up as much space on the actual disk drive as they are actually using.<br />
<br />
When you create a new linux system, even though there are some 300 RPMs or so installed, if you run `df -k` you will see that the entire 4gig partition is empty - no space is being used. This is because the files in their system are "magic symlinks" to the template for their OS that is in /vz/template - however, any changes to any of those files will "disconnect" them and they will immediately begin using space in their system. Further, any new files uploaded (even if those new files overwrite existing files) will take up space on the partition.<br />
<br />
<br />
if you see this:<br />
<br />
<pre>[root@virt8 root]# vzctl stop 160 ; vzctl start 160<br />
VE is not running<br />
Starting VE ...<br />
VE is unmounted<br />
VE is mounted<br />
Adding IP address(es): 69.55.226.83 69.55.226.84<br />
bash ERROR: Can't change file /etc/sysconfig/network<br />
Deleting IP address(es): 69.55.226.83 69.55.226.84<br />
VE is unmounted<br />
[root@virt8 root]#</pre><br />
<br />
it probably means they no longer have /bin/bash - copy one in for them<br />
<br />
ALSO: another possibility is that they have removed the `ed` RPM from their system - it needs to be reinstalled into their system. But since their system is down, this is tricky ...<br />
<br />
VE startup scripts used by 'vzctl' want package 'ed' to be available inside VE. So if package 'ed' will be enabled in OS template config and OS template itself VE #827 is based on - this error should be fixed.<br />
<br />
yes, it is possible to add RPM to VE while it not running.<br />
Try to do following:<br />
<br />
<pre># cd /vz/template/<OS_template_with_ed_package>/<br />
# vzctl mount 827<br />
# rpm -Uvh --root /vz/root/827 --veid 827 ed-0.2-25.i386.vz.rpm</pre><br />
<br />
Usually theres an error, but its ok<br />
<br />
Note: replace 'ed-0.2-25.i386.vz.rpm' in last command with actual<br />
version of 'ed' package you have.<br />
<br />
----<br />
<br />
So how do I know what template the user has ? cat their conf file and it is listed in there. For example, if the conf file has:<br />
<br />
<pre>[root@virt12 sbin]# vc 1103<br />
…snip…<br />
OSTEMPLATE="debian-3.0/20030822"<br />
TEMPLATES="mod_perl-deb30/20030707 mod_ssl-deb30/20030703 mysql-deb30/20030707 proftpd-deb30/20030703 webmin-deb30/20030823 "<br />
…</pre><br />
<br />
then they are on debian 3.0, all of their system RPMs are in /vz/template/debian-3.0, and they are using version 20030822 of that debian 3.0 template. Also, they’ve also got additional packages installed (mod_perl, mod_ssl, etc). Those are also found under /vz/template<br />
<br />
----<br />
<br />
Edits needed to run java:<br />
<br />
When we first created the VEs, the default setting for privvmpages was 93000:94000 ... which was high enough that most people never had problems ... however, you can;t run java or jdk or tomcat or anything java related with that setting. We have found that by setting privvmpages to 610000:615000 that java runs just fine. That is now the default setting. It is exceedingly rare that anyone needs it higher than that, although we have seen it once or twice.<br />
<br />
Any problems with java at all - the first thing you need to do is see if the failcnt has raised for privvmpages.<br />
<br />
<pre># vzctl start 160<br />
Starting VE ...<br />
vzquota : (error) Quota on syscall for 160: Device or resource busy<br />
Running vzquota on failed for VE 160 [3]</pre><br />
<br />
This is because my pwd is _in_ their private directory - you can't start it until you move out<br />
<br />
People seem to have trouble with php if they are clueless newbies. Here are two common problems/solutions:<br />
<br />
no... but i figured it out myself. problem was the php.ini file that came<br />
vanilla with the account was not configured to work with apache (the<br />
ENGINE directive was set to off).<br />
<br />
everything else seems fine now.<br />
<br />
----<br />
<br />
the problem was in the php.ini file. I noticed that is wasnt showing<br />
the code when it was in an html file so I looked at the php.ini file<br />
and had to change it so it recognized <? tags aswell as <?php tags.<br />
<br />
Also, make sure added to httpd.conf<br />
AddType application/x-httpd-php .php<br />
<br />
----<br />
<br />
You can set the time zone:<br />
<br />
You can change the timezone by doing this:<br />
<br />
ln -sf /usr/share/zoneinfo/<zone> /etc/localtime<br />
<br />
where <zone> is the zone you want in the /usr/share/zoneinfo/ directory.<br />
<br />
----<br />
<br />
Failing shm_open calls:<br />
<br />
first, please check if /dev/shm is mounted inside VE.<br />
'cat /proc/mounts' command should show something like this:<br />
tmpfs /dev/shm tmpfs rw 0 0<br />
<br />
If /dev/shm is not mounted you have 2 ways to solve issue:<br />
1. execute following command inside VE (doesn't require VE reboot):<br />
mount -t tmpfs none /dev/shm<br />
2. add following string to /etc/fstab inside VE and reboot it:<br />
tmpfs /dev/shm tmpfs defaults 0 0<br />
<br />
----<br />
<br />
You can have a mounted but not running ve<br />
Just:<br />
vzctl mount <veid><br />
<br />
----<br />
<br />
When a debian sys can’t get on the network, and you try:<br />
<br />
<pre>vzctl set 1046 --ipadd 69.55.227.117<br />
Adding IP address(es): 69.55.227.117<br />
Failed to bring up lo.<br />
Failed to bring up venet0.</pre><br />
<br />
They probably removed iproute package, which must be the one from swsoft. To restore:<br />
<pre># dpkg -i --veid=1046 --admindir=/vz1/private/1046/root/var/lib/dpkg --instdir=/vz1/private/1046/root/ /vz/template/debian-3.0/iproute_20010824-8_i386.vz.deb<br />
(Reading database ... 16007 files and directories currently installed.)<br />
Preparing to replace iproute 20010824-8 (using .../iproute_20010824-8_i386.vz.deb) ...<br />
Unpacking replacement iproute ...<br />
Setting up iproute (20010824-8) ...</pre><br />
<br />
Then restart their ve<br />
<br />
----<br />
<br />
in a ve i do:<br />
<br />
<pre>cd /<br />
du -h .</pre><br />
<br />
and get: 483M .<br />
<br />
i do:<br />
<br />
<pre>bash-2.05a# df -h<br />
Filesystem Size Used Avail Use% Mounted on<br />
/dev/vzfs 4.0G 2.3G 1.7G 56% /</pre><br />
<br />
how can this be?<br />
<br />
Is it possible that quota file was corrupted somehow? Please try to: <br />
<pre>vzctl stop <VEID><br />
vzquota drop <VEID><br />
vzquota init <VEID><br />
vzctl start <VEID></pre><br />
<br />
----<br />
<br />
How to stop vz from starting after reboot:<br />
<br />
VIRTUOZZO=no <br />
in <br />
/etc/sysconfig/vz<br />
<br />
To start: <br />
service vz start<br />
(after setting VIRTUOZZO=yes in /etc/sysconfig/vz)<br />
<br />
service vz restart will do some kind of 'soft reboot' -- restart all<br />
VPSes and reload modules without rebooting the node<br />
<br />
if you need to shut down all VPSes really really fast, run killall -9 init<br />
<br />
----<br />
<br />
Postfix tip:<br />
<br />
You may want to tweak settings: default_process_limit=10<br />
<br />
= Virtuozzo VPS Management Tools =<br />
<br />
== vm ==<br />
<br />
TODO<br />
<br />
== cancelve ==<br />
cancelve <veid><br />
this will:<br />
* stop a ve<br />
* check for backups (offer to remove them from the backup server <br />
and the backup.config)<br />
* rename the private dir<br />
* check for PTR, provide the commands to reset to default<br />
* and rename the ve’s config<br />
* remind you to remove firewall rules<br />
* remind you to remove DNS entries<br />
<br />
== ipadd ==<br />
ipadd <veid> <ip> [ip] [ip]<br />
add’s ip(s) to a ve<br />
<br />
== ipdel ==<br />
ipdel <veid> <ip> [ip] [ip]<br />
removes ip(s) from a ve<br />
<br />
== vc ==<br />
vc <veid><br />
the equivalent of: <tt>cat /vzconf/<veid>.conf</tt><br />
<br />
== vl ==<br />
vl<br />
will displays a list of ve #’s, 1 per line. (ostensibly to use in a for loop)<br />
<br />
== vp ==<br />
vp <veid><br />
the equivalent of: <tt>vzps auxww –E <veid></tt><br />
<br />
== vpe ==<br />
DEPRECATED<br />
vpe <veid> <br />
this will allow you to do a vp when a ve is running out of control, the equivalent of (deprecated since vp operates outside the VPS): <br />
<pre>vzctl set <veid> --kmemsize 2100000:2200000<br />
vzctl exec <veid> ps auxw<br />
vzctl set <veid> --kmemsize (ve’s orig lvalue):(ve’s orig hvalue)</pre><br />
<br />
== vt ==<br />
vt <veid><br />
the equivalent of: <tt>vztop –E <veid></tt><br />
<br />
== vr ==<br />
vr <veid><br />
the equivalent of: <tt>vzctl stop <veid>; vzctl start <veid></tt><br />
You can run this even if the ve is down - the stop command will just fail<br />
<br />
== vs ==<br />
vs [veid]<br />
displays status (output of <tt>vzctl status <veid></tt>) on each ve configured on the system (as determined by <tt>/vzconf/*.conf</tt>)<br />
If passed an argument, gives the status for just that ve. <br />
A running system looks like:<br />
<tt>VEID 16066 exist mounted running</tt><br />
<br />
A ve which is not running (but does exist) looks like:<br />
<tt>VEID 9990 exist unmounted down</tt><br />
<br />
A ve which is not running and doesn’t exist looks like:<br />
<tt>VEID 421 deleted unmounted down</tt><br />
<br />
== vs2 ==<br />
vs2 [veid]<br />
this is similar to vs in that it displays status (output of <tt>vzctl status <veid></tt>) on each ve,<br />
but the difference is it’s list comes from doing an ls on the data dirs. This was meant to catch <br />
the rare case where a ve configured but exists. <br />
<br />
== vw ==<br />
vw [veid]<br />
displays the output of ‘<tt>w</tt>’ (the equivalent of <tt>vzctl exec <veid> w</tt>) for each configured ve (as determined by <tt>/vzconf/*.conf</tt>). Useful for determing which ve is contributing to a heavily-loaded system.<br />
If passed an argument, gives ‘<tt>w</tt>‘ output for just that ve. <br />
Ex:<br />
<pre>[root@virt2 etc]# vw<br />
134<br />
10:52pm up 79 days, 6:14, 0 users, load average: 0.02, 0.02, 0.00<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT<br />
16027<br />
2:52pm up 7 days, 19:54, 0 users, load average: 0.00, 0.00, 0.00<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT<br />
16055<br />
2:52pm up 79 days, 6:38, 0 users, load average: 0.00, 0.04, 0.07<br />
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT</pre><br />
<br />
== vwe ==<br />
vwe [constraint]<br />
just like <tt>vw</tt>, but takes a constraint as an argument, only show’s ve’s with loads >= the constraint provided. If no constraint is provided, 1 is used by default<br />
<br />
== vzs ==<br />
vzs [veid]<br />
displays the beancounter status for all ve’s, or a particular ve if an argument is passed<br />
<br />
== ve ==<br />
ve <veid><br />
the equivalent of: <tt>vzctl enter <veid></tt><br />
<br />
== vx ==<br />
vx <veid> <command><br />
the equivalent of: <tt>/usr/sbin/vzctl exec <veid> <command></tt><br />
<br />
== dprocs ==<br />
dprocs [count]<br />
a script which outputs a continuous report (or a certain number of reports if an option is passed) of processes stuck in the D state and which VPS’s those procs belong to.<br />
<br />
== setmem ==<br />
setmem <VEID> <256|512|768|1024|2048><br />
adjusts the memory resources for the VE<br />
<br />
== afacheck.sh ==<br />
afacheck.sh<br />
displays the health/status of containers and mirrors on an adaptec card (currently quar1, tempvirt1-2, virt9, virt10)- all other are LSI<br />
<br />
== backup ==<br />
backup<br />
backup script called nightly to update virt scripts and do customer backups<br />
<br />
== checkload.pl ==<br />
checkload.pl<br />
this was intended to be setup as a cronjob to watch processes on a virt when the load <br />
rises above a certain level. Not currently in use.<br />
<br />
== findbackuppigs.pl ==<br />
findbackuppigs.pl<br />
looks for files larger than 50MB which customers have asked us to backup. Emails matches<br />
to linux@johncompanies.com<br />
<br />
== gatherlinux.pl ==<br />
gatherlinux.pl<br />
gathers up data about ve’s configured and writes to a file. Used for audits against the db<br />
<br />
== gb ==<br />
gb <search><br />
greps backup.config for the given search parameter<br />
<br />
== gbg ==<br />
gbg <search><br />
greps backup.config for the given search parameter and presents just the directories (for clean pasting)<br />
<br />
== linuxtrafficgather.pl ==<br />
linuxtrafficgather.pl [yy-mm]<br />
sends a traffic usage summary by ve to support@johncomapnies.com and payments@johncompanies.com.<br />
Optional arguments are year and month (must be in the past). If not passed, assumes last month. Relies on <br />
traffic logs created by netstatreset and netstatbackup<br />
<br />
== linuxtrafficwatch.pl ==<br />
linuxtrafficwatch.pl<br />
checks traffic usage and emails support@johncomapnies.com when a ve reaches the warning level (35G) and<br />
the limit (40G). Works on virtuozzo versions <= 2.6.x. We really aren’t using this anymore now that we have netflow.<br />
<br />
== linuxtrafficwatch2.pl ==<br />
linuxtrafficwatch2.pl<br />
checks traffic usage and emails support@johncomapnies.com when a ve reaches the warning level (35G) and<br />
the limit (40G). Works on virtuozzo version 2.6.x. We really aren’t using this anymore now that we have netflow.<br />
<br />
== load.pl ==<br />
load.pl<br />
feeds info to load mrtg - executed by inetd.<br />
<br />
== mb (linux) ==<br />
mb <mount|umount><br />
(nfs) mounts and umounts dirs to backup2. Shortcuts are mbm and mbu to mount and unmount. <br />
<br />
== migrate ==<br />
migrate <ip of node migrating to> <veid> <target_veid> [target dir: vz | vz1 | vz2]<br />
this is basically a wrapper for <tt>vzmigrate</tt> – vzmigrate is a util to seamlessly move a ve from one host to another. This wrapper was written cause vz virtuozzo version 2.6 had a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables. This script mitigates that. Since it makes multiple ssh connections to the target host, it’s a good idea to put the pub key for the src system in the authorized_keys file on the target host. In addition, it emails ve owners when their migration starts and stops (if they place email addresses in a file on their system: /migrate_notify). To move everyone off a system, you’d do:<br />
for f in `vl`; do migrate <ip> $f; done<br />
<br />
== migrateonline ==<br />
migrateonline <ip of node migrating to> <veid> <target_veid> [target dir: vz | vz1 | vz2]<br />
this is the same as migrate but will migrate a ve in <tt>–online</tt> mode which means it won’t be shut down at the end of the migration. This only works when migrating ve’s between 2 machines running a 2.6 kernel (currently tempvirt1-2. virt16-19, virt12). If you get an error that the machine you’re trying to migrate to has a different CPU or features, etc, then you have to edit the file and add the –f switch to the vzmigrate line- you can basically ignore this kind of warning (but never ignore a warning about missing templates on the destination node). NOTE: This edit (if made to migrateonline) will be overwritten by the base script during each night’s backup.<br />
<br />
== netstatbackup ==<br />
DEPRECATED<br />
netstatbackup <br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.5.x. <br />
<br />
== netstatbackup2 ==<br />
DEPRECATED<br />
netstatbackup2<br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.6.x. <br />
<br />
== netstatreset ==<br />
DEPRECATED<br />
netstatreset<br />
writes traffic count data to a logfile and resets counters to 0. Works on virtuozzo versions 2.5.x <br />
<br />
== netstatreset2 ==<br />
DEPRECATED<br />
netstatreset2<br />
writes traffic count data to a logfile. Works on virtuozzo versions 2.6.x. <br />
<br />
== orphanedbackupwatchlinux ==<br />
orphanedbackupwatchlinux <br />
looks for directories on backup2 which aren’t configured in backup.config and offers to <br />
delete them<br />
<br />
== rsync.backup (linux) ==<br />
rsync.backup<br />
does customer backups (relies on backup.config)<br />
<br />
== startvirt.pl ==<br />
startvirt.pl<br />
forks off start ve commands – keeps 6 running at a time. This is not to be used on systems where fastboot is enabled as it circumvents the benefit of the fastboot. The script will occasionally not exit gracefully and will continue to use up CPU, so it should be watched. Also, don’t exit from the script till you’re sure all ve’s are started – if you do you need to start them manually and may have to free up locks. On some systems, the startvirt script doesn’t exit cleanly and you have to ^C out of it. Be careful though- doing so can leave some VE’s in an odd bootup state and you may need to ‘vr’ them manually. You should check to see which ve’s aren’t running and/or confirm all have started when ^C’ing out of startvirt.<br />
<br />
== taskdone (linux) ==<br />
taskdone<br />
when called will email support@johncompanies.com with the hostname of the machine from which it was <br />
executed as the subject<br />
<br />
== vb (linux) ==<br />
vb<br />
the equivalent of: <tt>vi /usr/local/sbin/backup.config</tt><br />
<br />
== vemakeXX ==<br />
DEPRECATED<br />
vemakerh9 <br />
ve create script for RH9 (see vemake)<br />
<br />
vemakedebian30 <br />
ve create script for debian 3.0 (Woody) (see vemake)<br />
<br />
vemakedebian31 <br />
ve create script for debian 3.1 (Sarge) (see vemake)<br />
<br />
vemakedebian40 <br />
ve create script for debian 4.0 (Etch) (see vemake)<br />
<br />
vemakefedora, vemakefedora2, vemakefedora4, vemakefedora5, vemakefedora6, vemakefedora7<br />
ve create script for fedora core 1, 2, 4, 5, 6, 7 respectively (see vemake)<br />
<br />
vemakecentos3, vemakecentos4<br />
ve create script for centos 3, 4 respectively (see vemake)<br />
<br />
vemakesuse, vemakesuse93, vemakesuse100<br />
ve create script for suse 9.2, 9.3, 10.0 respectively (see vemake)<br />
<br />
vemakeubuntu5, vemakeubuntu606, vemakeubuntu606 vemakeubuntu704<br />
ve create script for ubuntu 5.10, 6.06, 6.10, 7.04 respectively (see vemake)<br />
<br />
== vemove ==<br />
DEPRECATED<br />
vemove <veid> <target_ip> </vz/private/123><br />
this script simplifies the old way of moving ve’s from one system to another - in short moving a ve to or from a virt running virtuozzo < 2.6.x<br />
It’s the equivalent of:<br />
<tt>tar cfpP - <veid> --ignore-failed-read | (ssh -2 -c arcfour <target_ip> "split - -b 1024m </vz/private/123>.tar" )</tt>This should only be used if migrate/vzmigrate can’t be used. <br />
<br />
== vim.watchdog ==<br />
vim.watchdog <br />
looks for and kills procs matching “vi|vim|nano|pine|elm” that are running for a long time & consuming lots of cpu. Works on virtuozzo versions 2.5.x<br />
<br />
== vim.watchdog2 ==<br />
vim.watchdog2<br />
looks for and kills procs matching “vi|vim|nano|pine|elm” that are running for a long time & consuming lots of cpu.<br />
Works on virtuozzo versions 2.6.x.<br />
<br />
== vzmigrate ==<br />
vzmigrate <target_ip> -r no <veid>:[dst veid]:[dst /vzX/private/veid]:[dst /vzX/root/veid]<br />
(this is the raw command “wrapped” by migrate/migrateonline) this will seamlessly move a ve from one host to another. The ve will run for the duration of the migration till the very end when it’s shut down, ip moved and started up on the target system. The filesystem on the src will remain. This should be watched – occasionally the move will timeout and leave the system shut down. If target private and root aren’t specified it just puts it in /vz. Only works when both systems are running virtuozzo 2.6.x<br />
<br />
== vztrafdump.sh ==<br />
DEPRECATED<br />
vztrafdump.sh<br />
writes traffic usage info by ve to a file called jc_traffic_dump in each ve’s / dir. Works on virtuozzo versions <= 2.5.x. <br />
<br />
== vztrafdump2.sh ==<br />
DEPRECATED<br />
vztrafdump2.sh<br />
writes traffic usage info by ve to a file called jc_traffic_dump in each ve’s / dir. Works on virtuozzo versions 2.6.x. <br />
<br />
== addtun ==<br />
addtun <veid><br />
Add’s tun device to ve.<br />
<br />
== bwcap ==<br />
bwcap <veid> <kbps><br />
Ex: <tt>bwcap 1234 512</tt><br />
Caps a VE’s bandwidth to the amount given<br />
<br />
== setdisk ==<br />
setdisk <veid> <diskspace in GB><br />
Ex: <tt>setdisk 1234 5</tt><br />
Gives a VE’s a given amount of disk space<br />
<br />
== vdf ==<br />
vdf <veid> <br />
the equivalent of: <tt>vzctl exec <veid> df –h</tt><br />
<br />
== vdff ==<br />
vdff<br />
runs a (condensed) vdf for all ve’s in your pwd (must be run from /vz/privateN)<br />
<br />
== mvbackups ==<br />
mvbackups <veid> <target_machine> (virt1) <target_dir> (vz1)<br />
moves backups from one location to another on the backup server, and provides you with option to remove entries from current backup.config, and simple paste command to add the config to backup.config on the target server<br />
<br />
== checkquota ==<br />
checkquota<br />
for all the ve’s in the cwd (run from /vz/private, /vz1/private, etc) reports what vz quota says they’re using and what the actual usage is (as reported by du)<br />
<br />
== clearquota ==<br />
clearquota <veid><br />
Recalculates a ve’s quota, prints out the usage before and after. The equivalent of:<br />
<tt>vdf <veid>; v stop <veid>; vzquota drop <veid>; v start <veid>; vdf <veid></tt><br />
<br />
== dprocs ==<br />
dprocs<br />
Sometimes the server’s have a large number of processes get stuck in the D state- this script shows (every 3 secs) which VE’s have D procs, which procs<br />
are stuck and a running average of the top “offenders”<br />
<br />
== vzstat ==<br />
vstat<br />
sort of like top for VZ. sort VEs by CPU usage by pressing 'o' and then 'c' keys</div>Support