VPS Management

Common Problems

Login to any machine without a password

This is possible via the use of ssh keys. The process is thus:

1. place the public key for your user (root@mail) in the /root/.ssh/authorized_keys file on the server you wish to login to

cat /root/.ssh/id_dsa.pub

(paste that into authorized_keys on the target server). If the file doesn't exist, create it.

2. enable root login (usually only applies to FreeBSD). Edit the /etc/ssh/sshd_config on the target server and change: #PermitRootLogin no to PermitRootLogin yes

3. Restart the sshd on the target machine. First, find the sshd process:

jailps <hostname> | grep sshd 

or

vp <VEID> | grep sshd

Look for the process resembling:

root     17296  0.0  0.0  5280 1036 ?        Ss    2011   4:27 /usr/sbin/sshd 

(this is the sshd)

Not:

root      6270  0.5  0.0  6808 2536 ?        Ss   14:33   0:00 sshd: root [priv]

(this is an sshd child- someone already ssh'd in as root)

Restart the sshd:

kill -1 <PID>

Ex:

kill -1 17296

You may now ssh in.

Once you're done, IF you enabled root login, you should repeat steps 2 and 3 to disable root logins.

Letting someone in who has locked themselves out (killed sshd, lost pwd)

There are two ways people frequently lock themselves out - either they forget a password, or they kill off sshd somehow.

These are actually both fairly easy to solve. First, let's say someone kills off their sshd, or somehow mangles /etc/ssh/sshd_config such that it no longer lets them in.

Their email may be very short, or it may have all sorts of details about how you should fix sshd_config to let them in ... just ignore all of this. They can fix their own mangled sshd. Fixing this is very simple. First, edit the /etc/inetd.conf on their system and uncomment the telnet line:

telnet stream  tcp     nowait  root    /usr/libexec/telnetd    telnetd
#telnet stream  tcp6    nowait  root    /usr/libexec/telnetd    telnetd

(just leave the tcp6 version of telnet commented)

Then, use jailps to list the processes on their system, and find their inetd process. Then simply:

kill -HUP (pid)

where (pid) is the PID of their inetd process. Now they have telnet running on their system and they can log in and do whatever they need to do.

The only complications that could occur are:

a) their firewall config on our firewall has port 23 blocked, in which case you will need to open that - will be covered in a different lesson.

b) they are not running inetd, so you can't HUP it. If this happens, edit their /etc/rc.conf, add the inetd_enable="YES" line, and then kill their jail with /tmp/jailkill.pl - then restart their jail with the jail line from their quad/safe file. Easy.

If they have forgotten a password,

On 6.x+ you can reset their password with:

jexec <jailID from jls> passwd root

Note: the default password for 6.x jails is 8ico2987, for 4.x it is p455agfa

On 4.x, you need to cd to their etc directory ... for instance:

cd /mnt/data2/198.78.65.136-col00261-DIR/etc

and run:

vipw -d .

Then paste in these two lines (theres a paste with these):

root:$1$krszPxhk$xkCepSnz3mIikT3vCtJCt0:0:0::0:0:Charlie &:/root:/bin/csh
user:$1$Mx9p5Npk$QdMU6c8YQqp2FW2M3irEh/:1001:1001::0:0:User &:/home/user:/bin/sh

overwriting the lines they already have for "user" and "root" - then just tell them that both user and root have been reset to the default password of p455agfa.

For linux, just passwd inside shell or

vzctl set <veid> --userpasswd root:p455agfa –save

Starting in 2009 we began giving out randomized passwords for FreeBSD and Linux as the default password. That is stored with each system in Mgmt. You should look for and reset the password to that password in the event of a reset and refer the customer to use their original password from their welcome email- this way we don’t have to send the password again via email (in clear text).

sendmail can’t be contacted from ext ip (only locally)

By default redhat puts this line in sendmail.mc:

DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')

which makes it only answer on localhost. Comment it out like:

dnl DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')

and then rebuild sendmail.cf with:

m4 /etc/mail/sendmail.mc > /etc/sendmail.cf

virt doesn’t properly let go of ve’s ip(s) when moved to another system

On virtuozzo 2.6 systems, it's been observed that when moving ips from one virt to another that sometimes the routing table will not get updated to reflect the removal of the ip addresses.

A recent example was a customer that was moving to a new ve on a new virt and the ip addresses were traded between the two ve's. After the trade the two systems were not able to talk to each other. When looking at the routing table for the old system all the ip addresses were still in the routing table as being local, like so:

netstat -rn | grep 69.55.225.149
69.55.225.149   0.0.0.0         255.255.255.255 UH       40 0          0 venet0

This was preventing traffic to the other system from being routed properly. The solution is to manually delete the route:

route delete 69.55.225.149 gw 0.0.0.0

Supposedly, this was fixed in 2.6.1

sshd on FreeBSD 6.2 segfaults

First try to reinstall ssh

cd /usr/src/secure
cd lib/libssh
make depend && make all install
cd ../../usr.sbin/sshd
make depend && make all install
cd ../../usr.bin/ssh
make depend && make all install

Failing that, find the library that’s messed up:

ldd /usr/sbin/sshd
         libssh.so.3 => /usr/lib/libssh.so.3 (0x280a3000) 
         libutil.so.5 => /lib/libutil.so.5 (0x280d8000) 
         libz.so.3 => /lib/libz.so.3 (0x280e4000) 
         libwrap.so.4 => /usr/lib/libwrap.so.4 (0x280f5000) 
         libpam.so.3 => /usr/lib/libpam.so.3 (0x280fc000) 
         libbsm.so.1 => /usr/lib/libbsm.so.1 (0x28103000) 
         libgssapi.so.8 => /usr/lib/libgssapi.so.8 (0x28112000) 
         libkrb5.so.8 => /usr/lib/libkrb5.so.8 (0x28120000) 
         libasn1.so.8 => /usr/lib/libasn1.so.8 (0x28154000) 
         libcom_err.so.3 => /usr/lib/libcom_err.so.3 (0x28175000) 
         libroken.so.8 => /usr/lib/libroken.so.8 (0x28177000) 
         libcrypto.so.4 => /lib/libcrypto.so.4 (0x28183000) 
         libcrypt.so.3 => /lib/libcrypt.so.3 (0x28276000) 
         libc.so.6 => /lib/libc.so.6 (0x2828e000) 
         libmd.so.3 => /lib/libmd.so.3 (0x28373000)

md5 them and compare to other jail hosts or jails running on host

for libcrypto reinstall:

/usr/src/crypto
make depend && make all install

Moving a server to a different account in the management system

This is a tough one. You gotta touch a bunch of tables in the mysql database.

  systems
  subs
  ipmap

basically, get the sysid and go off that:

update systems set cid='newcid' where sysid=xxxxx
update subs set cid='newcid' where sysid=xxxxx
update ipmap set cid='newcid' where sysid=xxxxx

You then need to rename the server disk space to change the old ID to the new one.

For a jail -

stopjail <hostname>
mv 69.55.2XX.XXX-col0<oldid>     69.55.2XX.XXX-col0<newid>
mv 69.55.2XX.XXX-col0<oldid>-DIR 69.55.2XX.XXX-col0<newid>-DIR
vi /usr/local/jail/bin/quad1       [Update the ID in the quad file]
startjail <hostname>

For a virt - You will have to

• stop the virt
• rename the directories
• modify the config file /vzconf/<veid>.conf
• rename the virt conf file
• restart the new virt.

that ought to do it..

Re-enable a customer account in the management system

For example to re-instate col03438, In mysql:

show tables ;
select * from customers where cid='col03438' ;
update customers set status=1 where cid='col03438' ;
select * from  systems where cid='col03438' ;
update systems set stop_date=NULL where cid='col03438' ;
update systems set cancel_date=NULL where cid='col03438' ;
update systems set status=1 where cid='col03438' ;
select * from subs where cid='col03438' ;
update subs set stop_date=NULL where cid='col03438' ;
update subs set status=1 where cid='col03438' 

Ganeti VPS

This is how to create a VPS on the Lamphost ganeti cluster. I have created scripts for creating a FreeBSD VPS and one for creating a Linux VPS.

Ganeti Install a Linux VPS

To create a linux VPS use

  /root/new <name>  <disk size> <memory> <IP Addr> [<OS>]

( for example /root/new vps.example.com 30 2 69.55.238.10 centos-8 )

You need to customize the install by adding a "jci" account and a customer account (col0XXXX) as well. Make sure the customer acct (col0XXXX) is in a group to allow sudo.

Ganeti Install a FreeBSD VPS

To create a FreeBSD VPS, we have to do a full install. This will create a new VPS. on gn1, do

  /root/newfreebsd12 <name>  <disk size> <memory> <IP Addr> [<OS>]

You will then have to do the install via VNC.

gnt-instance info <name> | grep -i connection

Use your favorite VNC client (I use Tiger VNC). to connect

You need to customize the install by adding a "jci" account and a customer account (col0XXXX) as well.
Make sure the customer acct (col0XXXX) is in a group to allow sudo (wheel).

You will need to install sudo.

 
pkg install sudo    # install sudo package
visudo              # to add wheel account to sudoers
vi /etc/group       # add jci and user acct to wheel

You will also have to set up a console in /etc/ttys.

vi /etc/ttys
  (change onifconsole to on for ttyu0).

Ganeti restart a VPS

To shutdown a ganeti VPS

gnt-instance reboot <VPSname>

Ganeti shutdown a VPS

To shutdown a ganeti VPS

gnt-instance shutdown <VPSname>

To permanently delete a ganeti VPS

gnt-instance remove <VPSname>

Ganeti Re-Intsall a VPS

Re-installing the OS will destroy all data on the VPS. Login to the Ganeti master node (currently gn1).

To reinstall the OS on a VPS the VPS must be shutdown.

gnt-instance shutdown <VPSname>

Do the reinstall.

gnt-instance reinstall <VPSname>

Restart the VPS.

gnt-instance startup <VPSname>

Now you need to do the customization for the VPS. This is basically the same as the Dedicated server customization.

https://69.55.229.8/index.php/New_Signups#Dedicated_Server_Install_Checklist

Ganeti login to a VPS

To login to a ganeti VPS

gnt-instance console <VPSname>
<carriage return>
<carriage return>
<carriage return>

Use jci and appropriate Ganeti jci password.

or you can ssh to the VPS name or IP address.

FreeBSD 10.1 VPS (jail5 and jail6)

This is information on how to manage the FreeBSD 10.1 VPS servers on Jail5.

Making new customer VPS

 virtmake IP[,IP] <CID> hostname ipfw# email {FSP|FB|FS|FM|FP} [size in GB]

Note: For now, CID must be unique in the last 2 digits of the CID.

Must do a "vm start <CID>" after creating a VPS.

Starting VPS

To start all vms on the system

vm start

To start a single VPS

vm start <CID>

List running VPS

vm status

Attach to a user VPS console

vm attach <CID>

To detach from a console, type tilde control-d (~ ctl-d).

Stop a VPS

vm stop <CID>

Restart a VPS

vm restart <CID>

Modify a VPS

To modify a VPS config, stop the VPS

vm stop <CID>

Mount the VPS data file

vm mount <CID>

cd /data/vm/private/<CID>/mount

Modify the user files necessary.

Restart the VPS.

vm umount <CID>

vm start <CID>

Notes on a new FBSD 10.x VPS (jail6)

#! /bin/sh
#
#  new  <colo#>  <disk>


memory=1024

x="$1"

disk="$2"
if [ "y" == "y$disk" ] ; then
    disk=60
fi

disk2=`expr $disk - 4`


vm image provision 04dcdf96-d843-11e6-b131-001e67fcc64f col0$x
# vm configure col0$x
sed -e "s/memory=256M/memory=${memory}M/" /vms/col01114/col01114.conf >/vms/col0$x/col0$x.conf
zfs set quota=${disk}G   zroot/vms/col0$x
zfs set volsize=${disk}G zroot/vms/col0$x/disk0
# vm start col0$x


exit
 
vm console col0$x

#
# to increase the customer disk
#
zfs set quota=${disk}G   zroot/vms/col0$x
zfs set volsize=${disk}G zroot/vms/col0$x/disk0

##  on the customer VPS

gpart recover vtbd0
gpart show vtbd0
swapoff /dev/vtbd0p3
gpart delete -i 3 vtbd0
gpart show vtbd0
sysctl kern.geom.debugflags=16
gpart resize -i 2 -a 4k -s $disk2\G vtbd0

growfs /dev/vtbd0p2

gpart show vtbd0
gpart add -t freebsd-swap -a 4k vtbd0
gpart show vtbd0
swapon /dev/vtbd0p3

df -h
reboot

vi /etc/rc.conf
  hostname
  IP Addr
passwd root
passwd user
clear history
  cp /dev/null /var/log/utx.log
  cp /dev/null ~root/.history
  history -c

FreeBSD Jails

Starting jails: Quad/Safe Files

FreeBSD customer systems do not start up automatically at boot time. When one of our freebsd machines boots up, it boots up, and does nothing else. To start jails, we put the commands to start each jail into a shell script(s) and run the script(s). Jail startup is something that needs to be actively monitored, which is why we don’t just run the script automatically. More on monitoring later.

NOTE: >=7.x we have moved to 1 quad file: quad1. Startups are not done by running each quad, but rather startalljails which relies on the contents of quad1. The specifics of this are lower in this article. What follows here applies for pre 7.x systems.

There are eight files in /usr/local/jail/rc.d:

jail3# ls /usr/local/jail/rc.d/
quad1   quad2   quad3   quad4   safe1   safe2   safe3   safe4
jail3#

four quad files and four safe files.

Each file contains an even number of system startup blocks (total number of jails divided by 4)

The reason for this is, if we make one large script to startup all the systems at boot time, it will take too long - the first system in the script will start up right after system boot, which is great, but the last system may not start for another 20 minutes.

Since there is no way to parralelize this during the startup procedure, we simply open four terminals (in screen window 9) and run each script, one in each terminal. This way they all run simultaneously, and the very last system in each startup script gets started in 1/4th the time it would if there was one large file

The files are generally organized so that quad/safe 1&2 have only jails from disk 1, and quad/safe 3&4 have jails from disk 2. This helps ensure that only 2 fscks on any disk are going on at once. Further, they are balanced so that all quad/safe’s finish executing around the same time. We do this by making sure each quad/safe has a similar number of jails and represents a similar number of inodes (see js).

The other, very important reason we do it this way, and this is the reason there are quad files and safe files, is that in the event of a system crash, every single vn-backed filesystem that was mounted at the time of system crash needs to be fsck'd. However, fsck'ing takes time, so if we shut the system down gracefully, we don't want to fsck.

Therefore, we have two sets of scripts - the four quad scripts are identical to the four safe scripts except for the fact that the quad scripts contain fsck commands for each filesystem.

So, if you shut a system down gracefully, start four terminals and run safe1 in window one, and safe2 in window 2, and so on.

If you crash, start four terminals (or go to screen window 9) and run quad1 in window one, and quad2 in window 2, and so on.

Here is a snip of (a 4.x version) quad2 from jail17:

vnconfig /dev/vn16 /mnt/data2/69.55.228.7-col00820
fsck -y /dev/vn16
mount /dev/vn16c /mnt/data2/69.55.228.7-col00820-DIR
chmod 0666 /mnt/data2/69.55.228.7-col00820-DIR/dev/null
jail /mnt/data2/69.55.228.7-col00820-DIR mail1.phimail.com 69.55.228.7 /bin/sh /etc/rc

# moved to data2 col00368
#vnconfig /dev/vn28 /mnt/data2/69.55.236.132-col00368
#fsck -y /dev/vn28
#mount /dev/vn28c /mnt/data2/69.55.236.132-col00368-DIR
#chmod 0666 /mnt/data2/69.55.236.132-col00368-DIR/dev/null
#jail /mnt/data2/69.55.236.132-col00368-DIR limehouse.org 69.55.236.132 /bin/sh /etc/rc

echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’
vnconfig /dev/vn22 /mnt/data2/69.55.228.13-col01063
fsck -y /dev/vn22
mount /dev/vn22c /mnt/data2/69.55.228.13-col01063-DIR
chmod 0666 /mnt/data2/69.55.228.13-col01063-DIR/dev/null
jail /mnt/data2/69.55.228.13-col01063-DIR www.widestream.com.au 69.55.228.13 /bin/sh /etc/rc

# cancelled col00106
#vnconfig /dev/vn15 /mnt/data2/69.55.238.5-col00106
#fsck -y /dev/vn15
#mount /dev/vn15c /mnt/data2/69.55.238.5-col00106-DIR
#chmod 0666 /mnt/data2/69.55.238.5-col00106-DIR/dev/null
#jail /mnt/data2/69.55.238.5-col00106-DIR mail.azebu.net 69.55.238.5 /bin/sh /etc/rc

As you can see, two of the systems specified are commented out - presumably those customers cancelled, or were moved to new servers.

As you can see, the vnconfig line is the simpler command line, not the longer one that was used when it was first configured. As you can see, all that is done is, vnconfig the filesystem, then fsck it, then mount it. The fourth command is the `jail` command used to start the system – but that will be covered later.

Here is the safe2 file from jail17:

vnconfig /dev/vn16 /mnt/data2/69.55.228.7-col00820
mount /dev/vn16c /mnt/data2/69.55.228.7-col00820-DIR
chmod 0666 /mnt/data2/69.55.228.7-col00820-DIR/dev/null
jail /mnt/data2/69.55.228.7-col00820-DIR mail1.phimail.com 69.55.228.7 /bin/sh /etc/rc

# moved to data2 col00368
#vnconfig /dev/vn28 /mnt/data2/69.55.236.132-col00368
#mount /dev/vn28c /mnt/data2/69.55.236.132-col00368-DIR
#chmod 0666 /mnt/data2/69.55.236.132-col00368-DIR/dev/null
#jail /mnt/data2/69.55.236.132-col00368-DIR limehouse.org 69.55.236.132 /bin/sh /etc/rc

echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’
vnconfig /dev/vn22 /mnt/data2/69.55.228.13-col01063
mount /dev/vn22c /mnt/data2/69.55.228.13-col01063-DIR
chmod 0666 /mnt/data2/69.55.228.13-col01063-DIR/dev/null
jail /mnt/data2/69.55.228.13-col01063-DIR www.widestream.com.au 69.55.228.13 /bin/sh /etc/rc

# cancelled col00106
#vnconfig /dev/vn15 /mnt/data2/69.55.238.5-col00106
#mount /dev/vn15c /mnt/data2/69.55.238.5-col00106-DIR
#chmod 0666 /mnt/data2/69.55.238.5-col00106-DIR/dev/null
#jail /mnt/data2/69.55.238.5-col00106-DIR mail.azebu.net 69.55.238.5 /bin/sh /etc/rc

As you can see, it is exactly the same, but it does not have the fsck lines.

Take a look at the last entry - note that the file is named:

/mnt/data2/69.55.238.5-col00106

and the mount point is named:

/mnt/data2/69.55.238.5-col00106-DIR

This is the general format on all the FreeBSD systems. The file is always named:

IP-custnumber

and the directory is named:

IP-custnumber-DIR

If you run safe when you need a fsck, the mount will fail and jail will fail:

# mount /dev/vn1c /mnt/data2/jails/65.248.2.131-ns1.kozubik.com-DIR
mount: /dev/vn1c: Operation not permitted

No reboot needed, just run the quad script

Starting with 6.x jails, we added block delimiters to the quad/safe files, the block looks like:

echo '## begin ##: nuie.solaris.mu'
fsck -y /dev/concat/v30v31a
mount /dev/concat/v30v31a /mnt/data1/69.55.228.218-col01441-DIR
mount_devfs devfs /mnt/data1/69.55.228.218-col01441-DIR/dev
devfs -m /mnt/data1/69.55.228.218-col01441-DIR/dev rule -s 3 applyset
jail /mnt/data1/69.55.228.218-col01441-DIR nuie.solaris.mu 69.55.228.218 /bin/sh /etc/rc
echo '## end ##: nuie.solaris.mu'

These are more than just informative when running quad/safe’s, the echo lines MUST be present for certain tools to work properly. So it’s important that any updates to the hostname also be updated on the 2 echo lines. For example, if you try to startjail a jail with a hostname which is on the jail line but not the echo lines, the command will return with host not found.

FreeBSD 7.x+ notes

Starting with the release of FreeBSD 7.x, we are doing jail startups in a slightly different way. First, thereis only 1 file: /usr/local/jail/rc.d/quad1 There are no other quads or corresponding safe files. The reason for this is twofold, 1. We can pass –C to fsck which will tell is to skip the fsck if the fs is clean (no more need for safe files), 2. We have a new startup script which can be launched multiple times, running in parallel to start jails, where quad1 is the master jail file. Quad1 could still be run as a shell script, but it would take a very long time for it to run completely so it’s not advisable; or you should break it down into smaller chunks (like quad1, quad2, quad3, etc)

Here is a snip of (a 7.x version) quad1 from jail2:

echo '## begin ##: projects.tw.com'
mdconfig -a -t vnode -f /mnt/data1/69.55.230.46-col01213 -u 50
fsck -Cy /dev/md50c
mount /dev/md50c /mnt/data1/69.55.230.46-col01213-DIR
mount -t devfs devfs /mnt/data1/69.55.230.46-col01213-DIR/dev
devfs -m /mnt/data1/69.55.230.46-col01213-DIR/dev rule -s 3 applyset
jail /mnt/data1/69.55.230.46-col01213-DIR projects.tw.com 69.55.230.46 /bin/sh /etc/rc
echo '## end ##: projects.tw.com'

Cancelled jails are no longer commented out and stored in quad1, rather they’re moved to /usr/local/jail/rc.d/deprecated

To start these jails, start the 4 ssh sessions as you would for a normal crash and then instead of running quad1-4, instead run startalljails in each window. IMPORTANT- before running startalljails you should make sure you ran preboot once as it will clear out all the lockfiles and enable startalljails to work properly.

Problems with the quad/safe files

When you run the quad/safe files, there are two problems that can occur - either a particular system will hang during initialization, OR a system will spit out output to the screen, impeding your ability to do anything. Or both.

First off, when you start a jail, you see output like this:

Skipping disk checks ...
adjkerntz[25285]: sysctl(put_wallclock): Operation not permitted
Doing initial network setup:.
ifconfig: ioctl (SIOCDIFADDR): permission denied
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384
Additional routing options: TCP keepalive=YESsysctl:
net.inet.tcp.always_keepalive: Operation not permitted.
Routing daemons:.
Additional daemons: syslogd.
Doing additional network setup:.
Starting final network daemons:.
ELF ldconfig path: /usr/lib /usr/lib/compat /usr/X11R6/lib /usr/local/lib
a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout /usr/X11R6/lib/aout
Starting standard daemons: inetd cron sshd sendmail sendmail-clientmqueue.
Initial rc.i386 initialization:.
Configuring syscons: blanktime.
Additional ABI support:.
Local package initialization:.
Additional TCP options:.

Now, let's look at this line, near the end:

Local package initialization:.

This is where a list of daemons that are set to start at boot time willshow up. You might see something like:

Local package initialization: mysqld apache sendmail sendmail-clientmqueue

Or something like this:

Local package initialization: postgres postfix apache

The problem is that many systems (about 4-5 per machine) will hang on that line. Basically it will get to some of the way through the total daemons to be started:

Local package initialization: mysqld apache

and will just sit there. Forever.

Fortunately, pressing ctrl-c will break out of it. Not only will it break out of it, but it will also continue on that same line and start the other daemons:

Local package initialization: mysqld apache ^c sendmail-clientmqueue

and then continue on to finish the startup, and then move to the next system to be started.

So what does this mean? It means that if a machine crashes, and you start four screen-windows to run four quads or four safes, you need to periodically cycle between them and see if any systems are stuck at that point, causing their quad/safe file to hang. A good rule of thumb is, if you see a system at that point in the startup, give it another 100 seconds - if it is still at the exact same spot, hit ctrl-c. Its also a good idea to go back into the quad file (just before the first command in the jail startup block) and note that this jail tends to need a control-c or more time as follows:

echo '### NOTE ### slow sendmail'
echo '### NOTE ###: ^C @ Starting sendmail.'

NEVER hit ctrl-c repeatedly if you don't get an immediate response - that will cause the following jail’s startup commands to be aborted.

A second problem that can occur is that a jail - maybe the first one in that particular quad/safe, maybe the last one, or maybe one in the middle, will start spitting out status or error messages from one of its init scripts. This is not a problem - basically, hit enter a few times and see if you get a prompt - if you do get a prompt, that means that the quad/safe script has already completed. Therefore it is safe to log out (and log out of the user that you su'd from) and then log back in (if necessary).

The tricky thing is, if a system in the middle starts flooding with messages, and you hit enter a few times and don't get a prompt. Are you not getting a prompt because some subsequent system is hanging at the initialization, as we discussede above ? Or are you not getting a prompt because that quad file is currently running an fsck ? Usually you can tell by scrolling back in screen’s history to see what it was doing before you started getting the messages.

If you don’t get clues from history, you have to use your judgement - instead of giving it 100 seconds to respond, perhaps give it 2-3 mins ... if you still get no response (no prompt) when you hit enter, hit ctrl-c. However, be aware that you might still be hitting ctrl-c in the middle of an fsck. This means you will get an error like "filesystem still marked dirty" and then the vnconfig for it will fail and so will the jail command, and the next system in the quad file will then start starting up.

If this happens, just wait until the end of all the quad files have finished, and start that system manually.

If things really get weird, like a screen flooded with errors, and you can't get a prompt, and ctrl-c does nothing, then you need to just eventually (give it ten mins or so) just kill that window with ctrl-p, then k, and then log in again and manually check which systems are now running and which aren't, and manually start up any that are not.

Don't EVER risk running a particular quad/safe file a second time. If the quad/safe script gets executed twice, reboot the machine immediately.

So, for all the above reasons, anytime a machine crashes and you run all the quads or all the safes, always check every jail afterwards to make sure it is running - even if you have no hangs or complications at all. Run this command:

jailpsall

Note: postboot also populates ipfw counts, so it should not be run multiple times, use jailpsall for subsequent extensive ps’ing

And make sure they all show as running. If one does not show as running, check its /etc/rc.conf file first to see if maybe it is using a different hostname first before starting it manually.

One thing we have implemented to alleviate these startup hangs and noisy jails, is to put jail start blocks that are slow or hangy at the bottom of the safe/quad file. Further, for each bad jail we note in each quad/safe just before the start block something like:

echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’

That way we’ll be prepared to ^C when we see that message appear during the quad/safe startup process. If you observe a new, undocumented hang, after the quad/safe has finished, place a line similar to the above in the quad file, move the jail start block to the end of the file, then run buildsafe

Making new customer jail 4.x

1. run js to figure out which partition, IP to put it on, find an unused vn, and choose which quad it should go in

2. use col00xxx for both hostnames if they don’t give you a hostname

3. copy over dir and ip to pending customer screen

Usage: jailmake IP filepath vnX hostname shorthost quad/safe# ipfw# email [gb disk, default 4]

jail14# /tmp/jailmake 69.55.226.152 /mnt/data2/69.55.226.152-col00182 vn23 negev.cerastes.org negev 4 182 cerastes@cerastes.org

Making new customer jail 6.x

1. run js to figure out which gvinum vols are free (and which mnt (disk) each gvinum goes with/to), IP to put it on and choose which quad it should go in

2. use col00xxx for both hostnames if they don’t give you a hostname

3. copy over dir and ip to pending customer screen

Usage: jailmake IP filepath vN[,vN] hostname shorthost quad/safe# ipfw# email

jail19# jailmake 69.55.236.17 /mnt/data1/69.55.236.17-col01574 v10 uncle-leo.commadev.com uncle-leo 1 1574 lee@commadev.com,lee@gmail.com

Changing an ip for freebsd VPS

• stopjail (hostname)
• on systems using vnfile or mdfile: rename vnfile/mdfile, using new ip
• rename dir using new ip
• Edit quad (make sure to update all lines: directory, vn/mdfile)
• buildsafe (not necessary on systems having only quad1)
• if you're adding an ip not already on the host machine: ipadd x.x.x.x
• startjail (hostname)

if backups: mvbackups ...

• Edit dir/ip in mgmt
• if any rules, update any firewall rules to use new ip (careful not to make sure there arent multiple rules per ipfw#- search by colo#)
• (if customer asks and has domains) update their domains on ns1c
• optional: update ipfw counters on the host server

Rename a gconcat vol

• stop jail, unmount everything
• gconcat stop vol
• gconcat clear /dev/gvinum/vN
• gconcat clear /dev/gvinum/vN+1
• gconcat label -v newname /dev/gvinum/vN /dev/gvinum/vN+1
• bsdlabel /dev/concat/newname

(make sure a partition is not fstype unused, if so change to: 4.2BSD)

Remaking a VPS (on same jail)

1. take him down (stopjail)
2. make note of host, vn/gvin/md, ip
3. remove from quad/safe
4. move vnfile (rename)
5. remove dir (jailmake will make a new one)
6. remove ipfw counts
7. jailmake
8. update backup script
9. update db with new dir/ip (use the “move” link, mark old as stopped today)
10. update firewall if changed ip

Recovering from a crash (FreeBSD)

Diagnose whether you have a crash

The most important thing is to get the machine and all jails back up as soon as possible. Note the time, you’ll need to create a crash log entry (Mgmt. -> Reference -> CrashLog). The first thing to do is head over to the serial console screen and see if there’s any kernel error messages output. Try to copy any messages (or just a sample of repeating messages) you see into the notes section of the crash log. If there are no messages, the machine may just be really busy- wait a bit (5-10min) to see if it comes back. If it's still pinging, odds are its very busy. Note, if you see messages about swap space exhausted, the server is obviously out of memory, however it may recover briefly enough for you to get a jtop in to see who's lauched a ton of procs (most likely) and then issue a quick jailkill to get it back under control.

If it doesn't come back, or the messages indicate a fatal error, you will need to proceed with a power cycle (ctrl+alt+del will not work).

Power cycle the server

If this machine is not a Dell 2950 with a DRAC card (i.e. if you can’t ssh into the DRAC card (as root, using the standard root pass) and issue

racadm serveraction hardreset

then you will need someone at the data center power the macine off, wait 30 sec, then turn it back on. Make sure to re-attach via console:

tip jailX

immediately after power down.

(Re)attach to the console

Stay on the console the entire time during boot. As the BIOS posts- look out for the RAID card output- does everything look healthy? The output may be scrambled, look for "DEGRADED" or "FAILED". Once the OS starts booting you will be disconnected (dropped back to the shell on the console server) a couple times during the boot up. The reason you want to quickly re-attach is two-fold: 1. If you don’t reattach quickly then you won’t get any console output, 2. you want to be attached before the server potentially starts (an extensive) fsck. If you attach after the fsck begins, you’ll have seen no indication it started an fsck and the server will appear frozen during startup- no output, no response.

IMPORTANT NOTE: on some older FreeBSD systems, there will be no output to the video (KVM) console as it boots up. The console output is redirected to the serial port ... so if a jail crashes, and you attach a kvm, the output during the bootup procedure will not be shown on the screen. However, when the bootup is done, you will get a login prompt on the screen and will be able to log in as normal. /boot/loader.conf is where serial console redirect output lives, so comment that if you want to catch output on kvm. On newer systems it sends most output to both locations.

Assess the heath of the server

Once the server boots up fully, you should be able to ssh in. Look around- make sure all the mounts are there and reporting the correct size/usage (i.e. /mnt/data1 /mnt/data2 /mnt/data3 - look in /etc/fstab to determine which mount points should be there), check to see if RAID mirrors are healthy. See megacli, aaccheck

Before you start the jails, you need to run preboot. This will do some assurance checks to make sure things are prepped to start the jails. Any issues that come out of preboot need to be addressed before starting jails.

Start jails

More on starting jails Customer jails (the VPSs) do not start up automatically at boot time. When a FreeBSD machines boots up, it boots up, and does nothing else. To start jails, we put the commands to start each jail into a shell script(s) and run the script(s). Jail startup is something that needs to be actively monitored, which is why we don’t just run the script automatically.

In order to start jails, we run the quad files: quad1 quad2 quad3 and quad4 (on new systems there is only quad1). If the machine was cleanly rebooted- which wouldn't be the case if this was a crash, you may run the safe files (safe1 safe2 safe3 safe4) in lieu of quads.

Open up 4 logins to the server (use the windows in a9) In each of the 4 windows you will:

If there is a startalljails script (and only quad1), run that command in each of the 4 windows. It will parse through the quad1 file and start each jail. Follow the instructions here for monitoring startup. Note that you can be a little more lenient with jails that take awhile to start- startalljails will work around the slow jails and start the rest. As long as there aren't 4 jails which are "hung" during startup, the rest will get started eventually. -or- If there is no startalljails script, there will be multiple quad files. In each of the 4 windows, start each of the quads. i.e. start quad1 in window1, quad2 in window2 and so on. DO NOT start any quad twice. It will crash the server. If you accidentally do this, just jailkill all the jails which are in the quad and run the quad again. Follow the instructions here for monitoring quad startup.

Note the time the last jail boots- this is what you will enter in the crash log.

Save the crash log.

Check to make sure all jails have started

There's a simple script which will make sure all jails have started, and enter the ipfw counter rules: postboot Run postboot, which will do a jailps on each jail it finds (excluding commented out jails) in the quad file(s). We're looking for 2 things:

1. systems spawning out of control or too many procs
2. jails which haven't started

On 7.x and newer systems it will print out the problems (which jails haven't started) at the conclusion of postboot. On older systems you will need to watch closely to see if/when there's a problem, namely:

[hostname] doesnt exist on this server

When you get this message, it means one of 2 things: 1. the jail really didn't start: When a jail doesn't start it usually boils down to a problem in the quad file. Perhaps the path name is wrong (data1 vs data2) or the name of the vn/mdfile is wrong. Once this is corrected, you will need to run the commands from the quad file manually, or you may use startjail <hostname>

2. the customer has changed their hostname (and not told us) so their jail is running, just under a different hostname: On systems with jls, this is easy to rectify. First, get the customer info: g <hostname> Then look for the customer in jls: jls | grep <col0XXXX> From there you will see their new hostname- you should update that hostname in the quad file: don't forget to edit it on the ## begin ## and ## end ## lines, and in mgmt. On older systems without jls, this will be harder, you will need to look further to see their hostname- perhaps its in their /etc/rc.conf

Once all jails are started, do some spot checks- try to ssh or browse to some customers, just to make sure things are really ok.

Adding disk to a 7.x/8.x jail

or

Moving customer to a different drive (md)

NOTE: this doesn’t apply to mx2 which uses gvinum. Use same procedure as 6.x NOTE: if you unmount before mdconfig, re-mdconfig (attach) then unmount then mdconfig -u again

(parts to change/customize are red)

If someone wants more disk space, there’s a paste for it, send it to them (explains about downtime, etc).

1. Figure out the space avail from js. Ideally, you want to put the customers new space on a different partition (and create the new md on the new partition).

2. make a mental note of how much space they're currently using

3. jailkill <hostname>

4. Umount it (including their devfs) but leave the md config’d (so if you use stopjail, you will have to re-mdconfig it)

5. g <customerID> to get the info (IP/cust#) needed to feed the new mdfile and mount name, and to see the current md device.

6a. When there's enough room to place new system on an alternate, or the same drive: USE CAUTION not to overwrite (touch, mdconfig) existing md!!
touch /mnt/data3/69.55.234.66-col01334
mdconfig -a -t vnode -s 10g -f /mnt/data3/69.55.234.66-col01334 -u 97
newfs /dev/md97

Optional- if new space is on a different drive, move the mount point directory AND use that directory in the mount and cd commands below:
mv /mnt/data1/69.55.234.66-col01334-DIR /mnt/data3/69.55.234.66-col01334-DIR

mount /dev/md97 /mnt/data3/69.55.234.66-col01334-DIR
cd /mnt/data3/69.55.234.66-col01334-DIR

confirm you are mounted to /dev/md97 and space is correct:
df .

do the dump and pipe directly to restore:
dump -0a -f - /dev/md1 | restore -r -f -
rm restoresymtable

when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure

md-unconfig old system:
mdconfig -d -u 1

archive old mdfile.
mv /mnt/data1/69.55.237.26-col00241 /mnt/data1/old-col00241-mdfile-noarchive-20091211

edit quad (vq1) to point to new (/mnt/data3) location AND new md number (md97)

restart the jail:
startjail <hostname>

6b. When there's not enough room on an alternate partition, or on the same drive...but there is enough room if you were to remove the existing customer's space:

mount backup nfs mounts:
mbm
(run df to confirm backup mounts are mounted)

dump the customer to backup2 or backup1
dump -0a -f /backup4/col00241.20120329.noarchive.dump /dev/md1
(when complete WITHOUT errors, du the dump file to confirm it matches size, roughly, with usage)

unconfigure and remove old mdfile
mdconfig -d -u 1
rm /mnt/data1/69.55.237.26-col00241
(there should now be enough space to recreate your bigger system. If not, run sync a couple times)

create the new system (ok to reuse old mdfile and md#):
touch /mnt/data1/69.55.234.66-col01334
mdconfig -a -t vnode -s 10g -f /mnt/data1/69.55.234.66-col01334 -u 1
newfs /dev/md1
mount /dev/md1 /mnt/data1/69.55.234.66-col01334-DIR
cd /mnt/data1/69.55.234.66-col01334-DIR

confirm you are mounted to /dev/md1 and space is correct:
df .

do the restore from the dumpfile on the backup server:
restore -r -f /backup4/col00241.20120329.noarchive.dump .
rm restoresymtable

when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure

umount nfs:
mbu

If md# changed (or mount point), edit quad (vq1) to point to new (/mnt/data3) location AND new md number (md1)

restart the jail:
startjail <hostname>

7. update disk (and dir if applicable) in mgmt screen

8. update backup list AND move backups, if applicatble

Ex: mvbackups 69.55.237.26-col00241 jail9 data3

9. Optional: archive old mdfile

mbm
gzip -c old-col01588-mdfile-noarchive-20120329 > /deprecated/old-col01588-mdfile-noarchive-20120329.gz
mbu
rm old-col01588-mdfile-noarchive-20120329

Adding disk to a 6.x jail (gvinum/gconcat)

or

Moving customer to a different drive (gvinum/gconcat)

(parts to change are highlighted)

If someone wants more disk space, there’s a paste for it, send it to them (explains about downtime, etc).

1. Figure out the space avail from js. Ideally, you want to put the customers new space on a different partition (and create the new md on the new partition).

2. make a mental note of how much space they're currently using

3. stopjail <hostname>

4. g <customerID> to get the info (IP/cust#) needed to feed the new mount name and existing volume/device.

5a. When there's enough room to place new system on an alternate, or the same drive (using only UNUSED - including if it's in use by the system in question - gvinum volumes):

Configure the new device:
A. for a 2G system (single gvinum volume):
bsdlabel -r -w /dev/gvinum/v123
newfs /dev/gvinum/v123a

-or-

B. for a >2G system (create a gconcat volume):
try to grab a contiguous block of gvinum volumes. gconcat volumes MAY NOT span drives (i.e. you cannot use a gvinum volume from data3 and a volume from data2 in the same gconcat volume).

gconcat label v82-v84 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84
bsdlabel -r -w /dev/concat/v82-v84
newfs /dev/concat/v82-v84a

Other valid gconcat examples:
gconcat label v82-v84v109v112 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84 /dev/gvinum/v109 /dev/gvinum/v112
gconcat label v82v83 /dev/gvinum/v82 /dev/gvinum/v83
Note, long names will truncate: v144v145v148-v115 will truncate to v144v145v148-v1 (so you will refer to it as v144v145v148-v1 thereafter)

Optional- if new volume is on a different drive, move the mount point directory (get the drive from js output) AND use that directory in the mount and cd commands below:
mv /mnt/data1/69.55.234.66-col01334-DIR /mnt/data3/69.55.234.66-col01334-DIR

confirm you are mounted to the device (/dev/gvinum/v123a OR /dev/concat/v82-v84) and space is correct:
A. mount /dev/gvinum/v123a /mnt/data3/69.55.234.66-col01334-DIR
-or-
B. mount /dev/concat/v82-v84a /mnt/data3/69.55.234.66-col01334-DIR

cd /mnt/data3/69.55.234.66-col01334-DIR
df .

do the dump and pipe directly to restore:
dump -0a -f - /dev/gvinum/v1 | restore -r -f -
rm restoresymtable

when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure

edit quad (vq1) to point to new (/mnt/data3) location AND new volume (/dev/gvinum/v123a or /dev/concat/v82-v84a) , run buildsafe

restart the jail:
startjail <hostname>

5b. When there's not enough room on an alternate partition, or on the same drive...but there is enough room if you were to remove the existing customer's space (i.e. if you want/need to reuse the existing gvinum volumes and add on more):

mount backup nfs mounts:
mbm
(run df to confirm backup mounts are mounted)

dump the customer to backup2 or backup1
dump -0a -f /backup4/col00241.20120329.noarchive.dump /dev/gconcat/v106-v107
(when complete WITHOUT errors, du the dump file to confirm it matches size, roughly, with usage)

unconfigure the old gconcat volume
list member gvinum volumes:

gconcat list v106v107

Output will resemble:

Geom name: v106v107
State: UP
Status: Total=2, Online=2
Type: AUTOMATIC
ID: 3530663882
Providers:
1. Name: concat/v106v107
   Mediasize: 4294966272 (4.0G)
   Sectorsize: 512
   Mode: r1w1e2
Consumers:
1. Name: gvinum/sd/v106.p0.s0
   Mediasize: 2147483648 (2.0G)
   Sectorsize: 512
   Mode: r1w1e3
   Start: 0
   End: 2147483136
2. Name: gvinum/sd/v107.p0.s0
   Mediasize: 2147483648 (2.0G)
   Sectorsize: 512
   Mode: r1w1e3
   Start: 2147483136
   End: 4294966272

stop volume and clear members
gconcat stop v106v107
gconcat clear gvinum/sd/v106.p0.s0 gvinum/sd/v107.p0.s0

create new device- and its ok to reuse old/former members
try to grab a contiguous block of gvinum volumes. gconcat volumes MAY NOT span drives (i.e. you cannot use a gvinum volume from data3 and a volume from data2 in the same gconcat volume).
gconcat label v82-v84v106v107 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84 /dev/gvinum/v106 /dev/gvinum/v107
bsdlabel -r -w /dev/concat/v82-v84v106v107
newfs /dev/concat/v82-v84v106v107a

Optional- if new volume is on a different drive, move the mount point directory (get the drive from js output) AND use that directory in the mount and cd commands below:
mv /mnt/data1/69.55.234.66-col01334-DIR /mnt/data3/69.55.234.66-col01334-DIR

confirm you are mounted to the device (/dev/concat/v82-v84v106v107) and space is correct:
mount /dev/concat/v82-v84v106v107a /mnt/data3/69.55.234.66-col01334-DIR

cd /mnt/data3/69.55.234.66-col01334-DIR
df .

do the restore from the dumpfile on the backup server:
restore -r -f /backup4/col00241.20120329.noarchive.dump .
rm restoresymtable

when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure

edit quad (vq1) to point to new (/mnt/data3) location AND new volume (/dev/concat/v82-v84v106v107a), run buildsafe

restart the jail:
startjail <hostname>

TODO: clean up/clear old gvin/gconcat vol

6. update disk (and dir if applicable) in mgmt screen

7. update backup list AND move backups, if applicatble

Ex: mvbackups 69.55.237.26-col00241 jail9 data3

DEPRECATED - steps to tack on a new gvin to existing gconcat- leads to corrupted fs bsdlabel -e /dev/concat/v82-v84

To figure out new size of the c partition, multiply 4194304 by the # of 2G gvinum volumes and subtract the # of 2G volumes: 10G: 4194304 * 5 – 5 = 20971515 8G: 4194304 * 4 – 4 = 16777212 6G: 4194304 * 3 – 3 = 12582909 4G: 4194304 * 2 – 2 = 8388606

To figure out the new size of the a partition, subtract 16 from the c partition: 10G: 20971515 – 16 = 20971499 8G: 16777212 – 16 = 16777196 6G: 12582909 – 16 = 12582893 4G: 8388606 – 16 = 8388590

Orig: 8 partitions:

1. size offset fstype [fsize bsize bps/cpg]

 a:  8388590       16    4.2BSD     2048 16384 28552
 c:  8388606        0    unused        0     0         # "raw" part, don't edit

New: 8 partitions:

1. size offset fstype [fsize bsize bps/cpg]

 a: 12582893       16    4.2BSD     2048 16384 28552
 c: 12582909        0    unused        0     0         # "raw" part, don't edit

sync; sync

growfs /dev/concat/v82-v84a

fsck –fy /dev/concat/v82-v84a

sync

fsck –fy /dev/concat/v82-v84a

(keep running fsck’s till NO errors)

Adding disk to a 4.x jail

NOTE: if you unmount before vnconfig, re-vnconfig then unmount then vnunconfig

If someone wants more disk space, there’s a paste for it, send it to them.

1. Figure out the space avail from js. Ideally, you want to dump to a different partition and create the new vn on the current partition. If no space to make make dump, then move old vn’s to backup or worse, dump to backup.

2. g <customerID> to get the info

3. make vnfile, newfs and mount. USE CAUTION not to overwrite existing vn.
Ex: vnconfig –T -S 9g -s labels -c /dev/vn22 /mnt/data2/65.214.160.117-col00496
disklabel -r -w vn30 auto
newfs /dev/vn30c
mkdir /mnt/data2/65.214.160.117-col00496-DIR
mount /dev/vn30c /mnt/data2/69.55.237.26-col00241-DIR
cd /mnt/data2/65.214.160.117-col00496-DIR

4. jailkill <hostname>

5. dump -0a -f - /dev/vn1 | restore -r -f -

6. rm restoresymtable

7. unmount and unvnconfig old system:
umount /dev/vnNNc
vnconfig –u /dev/vnNN

8. move vnfile. Ex: mv /mnt/data1/69.55.237.26-col00241 /mnt/data1/old-col00241-vnfile-20110312-noarchive

9. remove old dir: rmdir /mnt/data1/69.55.237.26-col00241-DIR

10. edit quad/safe to point to new location, run buildsafe

11. start jail: startjail <hostname>

12. update disk (and dir if applicable) in mgmt screen

13. mv backups if necessary

WARNING: if you restore to unmounted vn then you’re actually restoring to /mnt/data1 you can tar and do above to recover

NOTE: you can’t move a customer to a system that has a base sys that is diff from the one they came from (BSD 4.8 to 4.5 = won’t work)

NOTE: we don’t mount procfs's to anyone’s system by default.

NOTE: mount_nulls: also goofy mount_nulls seen in df -k on jail2
also needs to be done in safe and quad
if he wants another, use safe/quad as example and add it to file. also exec command at prompt to add mount (no reboot, remount or rejail necessary)

Moving customer to another jail machine

Systems should only be moved between similarly-versioned jails. If you cannot get a match, you should go to freebsd.org and see what the differences were between the version you're moving from and the new version. Many of the updates will be to drivers and kernels - these don't affect the customer. Perhaps they updated traceroute or a library...you should copy the new/changed files into the VPS/jail on the new host. An alternate method for moving to a different version would be to buildworld to bring their full distribution up to date. Optional steps for this kind of upgrade are included below.

1. g <customerID>

2. jailkill <hostname>

3. create new device on target system
4.x:

vnconfig –T –S 4g –s labels –c /dev/vn1 /mnt/data1/69.55.22x.x-col00XXX
disklabel –r –w vn1 auto
newfs /dev/vn1c

6.x:

bsdlabel -r -w /dev/gvinum/v1
newfs /dev/gvinum/v1a
 or 
gconcat label v1-v3 /dev/gvinum/v1 /dev/gvinum/v2 /dev/gvinum/v3
bsdlabel -r -w /dev/concat/v1-v3
newfs /dev/concat/v1-v3a

7.x+:
Run jailmakeempty then skip step 4 & 7 below

4. make and mount the dir on new system.

Ex:

mkdir /mnt/data2/69.55.230.3-col00123-DIR
chmod 755 /mnt/data2/69.55.230.3-col00123-DIR
mount [device] /mnt/data2/69.55.230.3-col00123-DIR

5. stopjail <hostname> 1

6. dump fs to new system. ex: dumpremoterestore /dev/vn51 10.1.4.118 /mnt/data2/69.55.239.45-col00688-DIR (make sure you can ssh as root on remote machine)

6a. OPTIONAL BUILDWORLD:

cd /usr/src
make world DESTDIR=/mnt/data2/69.55.xxx.xx-col0xxxx-DIR

cd etc
make distribution DESTDIR=/mnt/data2/69.55.xxx.xx-col0xxxx-DIR

(you may have to rm an openssh file and re-make dist)

rm -rf /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc/periodic/daily/400.status-disks
vi /etc/periodic/security/100.chksetuid

replace: MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`
with: MP='/' (use single quotes)

Ask user if they want ports overwritten with current, if yes:

cp -r /usr/ports /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/usr

on source:

cd /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc; vipw –d .

(copy in all info)

on target:

cd /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc; vipw –d . 

(paste all info)

on source:

cat /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc/group

(copy in all info)

on target:

cat > /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc/group 

(paste all info)

7. edit quad on source system copy over entries for the jail to the target system take care that the vn/gvinum/gconcat/md devices on the target system are’t in use and that the /mnt/dataN path doesn’t need changing

8. run buildsafe (if this is <=6.x) on target system (copies changes made to quad into safe file)

9. remove ip from source system. Ex: ipdel 69.55.230.3

10. add ip to target system. Ex: ipadd 69.55.230.3

11. start new system, with startjail or manually, pasting entries found from running g <customerID> on the new system

12. run canceljail col0xxxx on source system. This should prompt you to remove backups if any existed. Do not let it do this, rather:

13. if backups existed move them to the new host via the mvbackups script

14. edit quad on source system, edit comment to reflect a move rather than a cancel, ex: # moved to jail2 col00241

15. edit mgmt to reflect new host and dir for new system

16. optional add ipfw rules on new system. Ex: ipfw add 01231 count ip from 69.55.230.2 to any ipfw add 01232 count ip from any to 69.55.230.2

Increasing inodes for a VPS

when doing the newfs:

newfs -i  4096 ...

Jail services unable to contact services within same jail

This is due to a messed up routing table (only seen it on jail2 before).

route delete 69.55.228.65/32 route add -net 69.55.228.65/32 -iface fxp0 -nostatic -cloning route add 69.55.228.65 -iface lo0

/dev/null permission resets

Applies to 4.x jails only

For some reason that I do not understand at all, the /dev/null node in customer jails, after the system is restarted, often reverts permissions to 0600 ... which is bad, because non-root processes cannot redirect to /dev/null anymore. In fact, a fair number of server daemons will complain and/or fail if this is the case.

So, after a restart, after all systems have restarted, run postboot, which does something like:

for f in `df -k | grep /dev/vn | awk '{print $6}'` ; do chmod 0666 $f/dev/null ; done

Later, after the system is up, if anyone ever complains about their /dev/null permissions for any reason ... really if anyone mentions /dev/null in any way, tell them to run:

chmod 0666 /dev/null

We have ultimately fixed this by adding chmods to the quad/safe's

mknod in a jail

For a customer who wants to run a chroot name server (named) in a jail.

cd /mnt/data1/<user directory>/var/named
mknod -c 1 1 null
mknod -c 2 3 random

Postfix Problems

This hasn't happened in a long time, probably no longer valid with modern OS's

Postfix is an alternate MTA - people replace sendmail with it in much the same way that many people also replace sendmail with qmail. The problem is that if you install postfix inside of a jail, by default it will not work properly - the user will see error messages like this:

incoming mail generates errors like this:

mail_queue_enter: create file incoming/298266.55902: File too large

and further, sending mail generates errors like this:

postdrop: warning: mail_queue_enter: create file maildrop/627930.56676: File too large

This is very easy to solve. In fact, the FreeBSD welcome email that is sent out by `jailmake` now contains this block of text:

- if you plan on installing postfix, email us first and ask for a necessary patch.

So, anytime you see a support email that says anything about postfix not working, paste that line from the welcome email into your response to them, and right away, forward the postfix patch to them in an email.

Here are the instructions:

Simply copy the patch into /usr/ports/mail/postfix/files, rename the file to ‘patch-file_limit.c’ and then cd to /usr/ports/mail/postfix and run `make install`.

The postfix patch is the very first email in the support email box, and we never delete or save it because we always want it there to forward to people. The instructions in the email are very clear, and it solves their problem 100% of the time.

Problems with `find`

There was a bug in the FreeBSD vn-filesystem code. The bug causes the system to crash or hang when someone runs the `find` command inside their vn-backed filesystem (and even though the bug may have been fixed, due to the i/o overhead, we still don't like find).

Now, this does not always happen - however, any jail machine with more than 20 systems on it will crash every night if every system on it runs the daily periodic script out of their crontab, and updates their locate database with the find command.

(the locate database is a small, FreeBSD specific database that is populated nightly from a cron job, and is then used to provide fast answers to the `locate` command)

In normal systems (32-40 jails on them), all that has to be done is make sure nobody runs the daily periodic. This means that the file /etc/crontab, instead of looking like this:

# do daily/weekly/monthly maintenance
1      3       *       *       *       root    periodic daily
15     4       *       *       6       root    periodic weekly
30     5       1       *       *       root    periodic monthly

Needs to look like this:

# DO NOT UNCOMMENT THESE - contact support@johncompanies.com for details
#1      3       *       *       *       root    periodic daily
#15     4       *       *       6       root    periodic weekly
#30     5       1       *       *       root    periodic monthly
## DO NOT UNCOMMENT THESE ^^^^^^^^

and the problem will generally not occur at all. New filesystem images are always altered in this way, so all systems are set like this.

However, if you have enough systems running, even casual uses of find, outside of the periodic scripts, can cause the system to crash. We are seeing this on jail13, which crashes about once per week.

There is not yet any resolution to this problem.

So, if you have a freebsd system crash, see if anyone has uncommented their daily periodic line - you can see them for everyone by running:

/bin/sh

and then running:

for f in `df -k | grep vn | awk '{print $6}'` ; do echo `cat $f/etc/crontab | grep "periodic daily"` $f ; done | more

(all one line, one command)

And then edit those /etc/crontabs and comment them out again.

Problems un-mounting - and with mount_null’s

If you cannot unmount a filesystem, beacuse it says the filesystem is busy, it is because of three things:

a) the jail is still running

b) you are actually in that directory, even though the jail is stopped

c) there are still dev, null_mount or linprocfs mount points mounted inside that directory.

d) when trying to umount null_mounts that are really long and you get an error like “No such file or directory”, it’s an OS bug where the dir is truncated. No known fix

e) there are still files open somewhere inside the dir. Use fstat | grep <cid> to find the process that has files open

f) Starting with 6.x, the jail mechanism does a poor job of keeping track of processes running in a jail and if it thinks there are still procs running, it will refuse to umount the disk. If this is happening you should see a low number in the #REF column when you run jls. In this case you can safely umount –f the mount.

Please note -if you forcibly unmount a (4.x) filesystem that has null_mounts still mounted in it, the system will crash within 10-15 mins.

Misc jail Items

We are overselling hard drive space on jail2, jail8, jail9, a couple jails on jail17, jail4, jail12 and jail18. Even though the vn file shows 4G size, it doesn’t actually occupy that amount of space on the disk. So be careful not to fill up drives where we’re overselling – use oversellcheck to confirm you’re not oversold by more than 10G. There are other truncated jails, they are generally noted in a the file on the root system: /root/truncated

The act of moving a truncated vn to another system un-does the truncating- the truncated vn is filled with 0’s and it occupies physical disk space for which it’s configured. So, you should use dumpremote to preserve the truncation.

• if you are getting disk full messages for a BSD customer, it's fairly safe to clear out their /usr/ports/distfiles dir
• 4.x: ps and top an only be run by root in these jails. Done on purpose:

As for `ps` and `top` - non-root users can run them, just not successfully because we have locked the permissions on /dev/mem and /dev/kmem to be root-readable only. That is why non-root users cannot successfully run `ps` and `top`.

• user quotas do not work on freebsd jails - you cannot set up quotas at all, and that's that.
• You cannot inject a process into a 4.x jail, only HUP running processes. 6.x and onward you can with jexec
• jails see base machine’s uptime/load when running top/w
• if someone is unable to get in- cant ping, etc- see if they were blocked by castle (DoS), see if their ip is on the system (post reboot it was lost cause wasn’t in the rc.conf). preboot should catch that.
• in FreeBSD you cant su to root unless you belong to wheel group – so if you remove your acct and setup a new one, we have to add it to wheel group (add to /etc/group)
• Dmesg from underlying sys is seen in customers dmesg on jail
• Popper process going crazy @ 40% for 10min = someone who leaves mail on server
• Don’t force umounts on 4.x jails – it crashes the machine, generally ok on newer machines (running md)
• Good book for admin http://search.barnesandnoble.com/booksearch/isbnInquiry.asp?userid=t824VyRAYz&isbn=0596005164&itm=2
• Self-sign ssl cert http://httpd.apache.org/docs/2.0/ssl/ssl_faq.html#selfcert
• conversation with Glenn about semaphores and pgsql

SDBoody: hey, these are valid amounts/figures/increments right:
 kern.ipc.semmni=1280
 kern.ipc.semmns=1280
 
gr8feen: probably... I always forget exactly what those are, so I usually have to look them up
SDBoody: semaphores
SDBoody: i took the current 1024 and added 256 to them
SDBoody: need more for pgsql
gr8feen: I meant the mni and mns parts...
gr8feen: some of those are not ones you just want to add to...   hang on a sec and I'll look them up..
gr8feen: what's semmsl set to?
SDBoody: kern.ipc.semmsl: 1024
 
SDBoody: kern.ipc.msgseg: 2048 
 kern.ipc.msgssz: 8 
 kern.ipc.msgtql: 40 
 kern.ipc.msgmnb: 2048 
 kern.ipc.msgmni: 40 
 kern.ipc.msgmax: 16384 
 kern.ipc.semaem: 16384 
 kern.ipc.semvmx: 65534 
 kern.ipc.semusz: 152 
 kern.ipc.semume: 10 
 kern.ipc.semopm: 100 
 kern.ipc.semmsl: 1024 
 kern.ipc.semmnu: 512 
 kern.ipc.semmns: 1024 
 kern.ipc.semmni: 1024 
 kern.ipc.semmap: 768 
 kern.ipc.shm_allow_removed: 0 
 kern.ipc.shm_use_phys: 1 
 kern.ipc.shmall: 262144 
 kern.ipc.shmseg: 256 
 kern.ipc.shmmni: 784 
 kern.ipc.shmmin: 1 
 kern.ipc.shmmax: 536870912 
 kern.ipc.maxsockets: 25600 
 
gr8feen: ok...msl is max per id, mni is max ids, mns is max number of semaphores... so you probably want something like mns = mni * msl
gr8feen: which one did you run out of?
SDBoody: not sure how to tell- ipcs shows the sems in use add up to 1024
SDBoody: there are 59 entries
gr8feen: I'm assuming you tried to start postgres and it failed?
SDBoody: yes
gr8feen: it should have logged why, somewhere..
gr8feen: if I recall, it'll tell you which one it ran out of
SDBoody: > DETAIL:  Failed system call was semget(1, 17, 03600).
 
gr8feen: so it wanted an id with 17 semaphores...I'd start by making mns = 17*mni and leave mni and mnl set to what they are now and see what it does
SDBoody: i think mni is plenty high
SDBoody: ok, more reasonable 17408
gr8feen: yeah...just change that one and see how it goes..
SDBoody: and leave mni alone at 1024?
gr8feen: yeah...mni id the max number of ids...but if it's trying to get something like 17 per id, your going to hit mns before you hit anything else
SDBoody: right, but doesn't hurt to have it that high (assuming)

gr8feen: not really...  I think those get allocated out of ram that you cant page out, but it's still such a small amount that it really doesn't matter
SDBoody: looks like that worked, thx!
gr8feen: cool

see semaphores: ipcs -a -s

FreeBSD VPS Management Tools

These files are located in /usr/local/jail/rc.d and /usr/local/jail/bin

jailmake

Applies to 7.x+ On older systems syntax differs, run jailmake once to see.

Note: this procedure differs on mx2 which is 7.x but still uses gvinum

1. run js to figure out which md’s are in use, which disk has enough space, IP to put it on
2. use col00xxx for both hostnames if they don’t give you a hostname
3. copy over dir, ip and password to pending customer screen

Usage: jailmake IP[,IP] CID disk[1|2|3] md# hostname shorthost ipfw# email [size in GB]

Ex:

Jail2# jailmake 69.55.234.66 col01334 3 97 vps.bsd.it vps 1334 fb@bsd.it

jailps

jailps [hostname]

DEPRECATED FOR jps: displays processes belonging to/running inside a jail. The command takes one (optional) argument – the hostname of the jail you wish to query. If you don’t supply an argument, all processes on the machine are listed and grouped by jail.

jps

jps [hostname]

displays processes belonging to/running inside a jail. The command takes one (optional) argument – the hostname or ID of the jail you wish to query.

jailkill

jailkill <hostname>

stops all process running in a jail.

You can also run:

jailkill <JID>

problems

Occasionally you will hit an issue where jail will not kill off:

jail9# jailkill www.domain.com
www.domain.com .. killed: none
jail9#

Because no processes are running under that hostname. You cannot use jailps.pl either:

jail9# jailps www.domain.com
www.domain.com doesn’t exist on this server
jail9#

The reasons for this are usually:

• the jail is no longer running

• the jail's hostname has changed

In this case,

>=6.x: run a jls|grep <jail's IP> to find the correct hostname, then update the quad file, then kill the jail.

<6.x: the first step is to cat their /etc/rc.conf file to see if you can tell what they set the new hostname to. This very often works. For example:

cat /mnt/data2/198.78.65.136-col00261-DIR/etc/rc.conf

But maybe they set the hostname with the hostname command, and the original hostname is still in /etc/rc.conf.

The welcome email clearly states that they should tell us if they change their hostname, so there is no problem in just emailing them and asking them what they set the new hostname to.

Once you know the new hostname OR if a customer simply emails to inform you that they have set the hostname to something different, you need to edit the quad and safe files that their system is in to input the new hostname.

However, if push comes to shove and you cannot find out the hostname from them or from their system, then you need to start doing some detective work.

The easiest thing to do is run jailps looking for a hostname similar to their original hostname. Or you could get into the /bin/sh shell by running:

/bin/sh

and then looking at every hostname of every process:

for f in `ls /proc` ; do cat /proc/$f/status ; done

and scanning for a hostname that is either similar to their original hostname, or that you don't see in any of the quad safe files.

This is very brute force though, and it is possible that catting every file in /proc is dangerous - I don't recommend it. A better thing would be to identify any processes that you know belong to this system – perhaps the reason you are trying to find this system is because they are running something bad - and just catting the status from only that PID.

Somewhere there’s a jail where there may be 2 systems named www. Look at /etc/rc.conf and make sure they’re both really www. If they are, jailkill www, jailps www to make sure not running. Then immediately restart the other one, as the fqdn (as found from a rev nslookup)

• on >=6.x the hostname may not yet be hashed:

jail9 /# jls
 JID Hostname                    Path                                  IP Address(es)
   1 bitnet.dgate.org            /mnt/data1/69.55.232.50-col02094-DIR  69.55.232.50
   2 ns3.hctc.net                /mnt/data1/69.55.234.52-col01925-DIR  69.55.234.52
   3 bsd1                        /mnt/data1/69.55.232.44-col00155-DIR  69.55.232.44
   4 let2.bbag.org               /mnt/data1/69.55.230.92-col00202-DIR  69.55.230.92
   5 post.org                    /mnt/data2/69.55.232.51-col02095-DIR  69.55.232.51 ...
   6 ns2                         /mnt/data1/69.55.232.47-col01506-DIR  69.55.232.47 ...
   7 arlen.server.net            /mnt/data1/69.55.232.52-col01171-DIR  69.55.232.52
   8 deskfood.com                /mnt/data1/69.55.232.71-col00419-DIR  69.55.232.71
   9 mirage.confluentforms.com   /mnt/data1/69.55.232.54-col02105-DIR  69.55.232.54 ...
  10 beachmember.com             /mnt/data1/69.55.232.59-col02107-DIR  69.55.232.59
  11 www.agottem.com             /mnt/data1/69.55.232.60-col02109-DIR  69.55.232.60
  12 sdhobbit.myglance.org       /mnt/data1/69.55.236.82-col01708-DIR  69.55.236.82
  13 ns1.jnielsen.net            /mnt/data1/69.55.234.48-col00204-DIR  69.55.234.48 ...
  14 ymt.rollingegg.net          /mnt/data2/69.55.236.71-col01678-DIR  69.55.236.71
  15 verse.unixlore.net          /mnt/data1/69.55.232.58-col02131-DIR  69.55.232.58
  16 smcc-mail.org               /mnt/data2/69.55.232.68-col02144-DIR  69.55.232.68
  17 kasoutsuki.w4jdh.net        /mnt/data2/69.55.232.46-col02147-DIR  69.55.232.46
  18 dili.thium.net              /mnt/data2/69.55.232.80-col01901-DIR  69.55.232.80
  20 www.tekmarsis.com           /mnt/data2/69.55.232.66-col02155-DIR  69.55.232.66
  21 vps.yoxel.net               /mnt/data2/69.55.236.67-col01673-DIR  69.55.236.67
  22 smitty.twitalertz.com       /mnt/data2/69.55.232.84-col02153-DIR  69.55.232.84
  23 deliver4.klatha.com         /mnt/data2/69.55.232.67-col02160-DIR  69.55.232.67
  24 nideffer.com                /mnt/data2/69.55.232.65-col00412-DIR  69.55.232.65
  25 usa.hanyuan.com             /mnt/data2/69.55.232.57-col02163-DIR  69.55.232.57
  26 daifuku.ppbh.com            /mnt/data2/69.55.236.91-col01720-DIR  69.55.236.91
  27 collins.greencape.net       /mnt/data2/69.55.232.83-col01294-DIR  69.55.232.83
  28 ragebox.com                 /mnt/data2/69.55.230.104-col01278-DIR 69.55.230.104
  29 outside.mt.net              /mnt/data2/69.55.232.72-col02166-DIR  69.55.232.72
  30 vps.payneful.ca             /mnt/data2/69.55.234.98-col01999-DIR  69.55.234.98
  31 higgins                     /mnt/data2/69.55.232.87-col02165-DIR  69.55.232.87 ...
  32 ozymandius                  /mnt/data2/69.55.228.96-col01233-DIR  69.55.228.96
  33 trusted.realtors.org        /mnt/data2/69.55.238.72-col02170-DIR  69.55.238.72
  34 jc1.flanderous.com          /mnt/data2/69.55.239.22-col01504-DIR  69.55.239.22
  36 guppylog.com                /mnt/data2/69.55.238.73-col00036-DIR  69.55.238.73
  40 haliohost.com               /mnt/data2/69.55.234.41-col01916-DIR  69.55.234.41 ...
  41 satyr.jorge.cc              /mnt/data1/69.55.232.70-col01963-DIR  69.55.232.70
jail9 /# jailkill satyr.jorge.cc
ERROR: jail_: jail "satyr,jorge,cc" not found

Note how it's saying satyr,jorge,cc is not found, and not satyr.jorge.cc.

The jail subsystem tracks things using comma-delimited hostnames. That is created every few hours:

jail9 /# crontab -l
0 0,6,12,18 * * * /usr/local/jail/bin/sync_jail_names

So if we run this manually:

jail9 /# /usr/local/jail/bin/sync_jail_names

Then kill the jail:

jail9 /# jailkill satyr.jorge.cc

successfully killed: satyr,jorge,cc

It worked.

If you ever see this when trying to kill a jail:

# jailkill e-scribe.com
killing JID: 6 hostname: e-scribe.com
3 procs running
3 procs running
3 procs running
3 procs running
...

jailkill probably got lost trying to kill off the jail. Just ctrl-c the jailkill process, then run a jailps on the hostname, and kill -9 any process which is still running. Keep running jailps and kill -9 till all processes are gone.

jailpsall

jailpsall

will run a jailps on all jails configured in the quad files (this is different from jailps with no arguments as it won’t help you find a “hidden” system)

jailpsw

jailpsw

will run a jailps with an extra -w to provide wider output

jt (>=7.x)

jt

displays the top 20 processes on the server (the top 20 processes from top) and which jail owns them. This is very helpful for determining who is doing what when the server is very busy.

jtop (>=7.x)

jtop

a wrapper for top displaying processes on the server and which jail owns them. Constantly updates, like top.

jtop (<7.x)

jtop

displays the top 20 processes on the server (the top 20 processes from top) and which jail owns them. This is very helpful for determining who is doing what when the server is very busy.

stopjail

stopjail <hostname> [1]

this will jailkill, umount and vnconfig –u a jail. If passed an optional 2nd argument, it will not exit before umounting and un-vnconfig’ing in the event jailkill returns no processes killed. This is useful if you just want to umount and vnconfig –u a jail you’ve already killed. It is intelligent in that it won’t try to umount or vnconfig –u if it’s not necessary.

startjail

startjail <hostname>

this will start vnconfig, mount (including linprocfs and null-mounts), and start a jail. Essentially, it reads the jail’s relevant block from the right quad file and executes it. It is intelligent in that it won’t try to mount or vnconfig if it’s not necessary.

jpid

jpid <pid>

displays information about a process – including which jail owns it. It’s the equivalent of running cat /proc/<pid>/status

canceljail

canceljail <hostname> [1]

this will stop a jail (the equivalent of stopjail), check for backups (offer to remove them from the backup server and the backup.config), rename the vnfile, remove the dir, and edit quad/safe. If passed an optional 2nd argument, it will not exit upon failing to kill and processes owned by the jail. This is useful if you just want to cancel a jail which is already stopped.

jls

jls [-v]

Lists all jails running:

JID #REF IP Address      Hostname                     Path
 101  135 69.55.224.148   mail.pc9.org                 /mnt/data2/69.55.224.148-col01034-DIR

1. REF is the number of references or procs(?) running

Running with -v will give you all IPs assigned to each jail (7.2 up)

JID #REF Hostname                     Path                                  IP Address(es)
 101  139 mail.pc9.org                 /mnt/data2/69.55.224.148-col01034-DIR 69.55.224.14869.55.234.85

startalljails

startalljails

7.2+ only. This will parse through quad1 and start all jails. It utilizes lockfiles so it won’t try to start a jail more than once- therefore multiple instances can be running in parallel without fear of starting a jail twice. If a jail startup gets stuck, you can ^C without fear of killing the script. IMPORTANT- before running startalljails you should make sure you ran preboot once as it will clear out all the lockfiles and enable startalljails to work properly.

aaccheck.sh

aaccheck.sh

displayes the output of container list and task list from aaccli

backup

backup

backup script called nightly to update jail scripts and do customer backups

buildsafe

buildsafe

creates safe files based on quads (automatically removing the fsck’s). This will destructively overwrite safe files

checkload.pl

checkload.pl

this was intended to be setup as a cronjob to watch processes on a jail when the load rises above a certain level. Not currently in use.

checkprio.pl

checkprio.pl

will look for any process (other than the current shell’s csh, sh, sshd procs) with a non-normal priority and normalize it

diskusagemon

diskusagemon <mount point> <1k blocks>

watches a mount point’s disk use, when it reaches the level specified in the 2nd argument, it exits. This is useful when doing a restore and you want to be paged as it’s nearing completion. Best used as: diskusagemon /asd/asd 1234; pagexxx

dumprestore

dumprestore <dumpfile>

this is a perl expect script which automatically enters ‘1’ and ‘y’. It seems to cause restore to fail to set owner permissions on large restores.

g

g <search>

greps the quad/safe files for the given search parameter

gather.pl

gather.pl

gathers up data about jails configured and writes to a file. Used for audits against the db

gb

gb <search>

greps backup.config for the given search parameter

gbg

gbg <search>

greps backup.config for the given search parameter and presents just the directories (for clean pasting)

ipfwbackup

ipfwbackup

writes ipfw traffic count data to a logfile

ipfwreset

ipfwreset

writes ipfw traffic count data to a logfile and resets counters to 0

js

js

output varies by OS version, but generally provides information about the base jail: - which vn’s are in use - disk usage - info about the contents of quads - the # of inodes represented by the jails contained in the group (133.2 in the example below), and how many jails per data mount, as well as subtotals - ips bound to the base machine but not in use by a jail - free gvinum volumes, or unused vn’s or used md’s

/usr/local/jail/rc.d/quad1:
        /mnt/data1 133.2 (1)
        /mnt/data2 1040.5 (7)
        total 1173.7 (8)
/usr/local/jail/rc.d/quad2:
        /mnt/data1 983.4 (6)
        total 983.4 (6)
/usr/local/jail/rc.d/quad3:
        /mnt/data1 693.4 (4)
        /mnt/data2 371.6 (3)
        total 1065 (7)
/usr/local/jail/rc.d/quad4:
        /mnt/data1 466.6 (3)
        /mnt/data2 882.2 (5)
        total 1348.8 (8)
/mnt/data1: 2276.6 (14)
/mnt/data2: 2294.3 (15)

Available IPs:
69.55.230.11 69.55.230.13 69.55.228.200

Available volumes:
v78 /mnt/data2 2G
v79 /mnt/data2 2G
v80 /mnt/data2 2G

load.pl

load.pl

feeds info to load mrtg - executed by inetd.

makevirginjail

makevirginjail

Only on some systems, makes an empty jail (doesn't do restore step)

mb

mb <mount|umount>

(nfs) mounts and umounts dirs to backup2. Shortcuts are mbm and mbu to mount and unmount.

notify.sh

notify.sh

emails reboot@johncompanies.com – intended to be called at boot time to alert us to a machine which panics and reboots and isn’t caught by bb or castle.

orphanedbackupwatch

orphanedbackupwatch

looks for directories on backup2 which aren’t configured in backup.config and offers to delete them

postboot

postboot

to be run after a machine reboot and quad/safe’s are done executing. It will:

• do chmod 666 on each jail’s /dev/null
• add ipfw counts
• run jailpsall (so you can see if a configured jail isn’t running)

preboot

preboot

to be run before running quad/safe – checks for misconfigurations:

• a jail configured in a quad but not a safe
• a jail is listed more than once in a quad
• the ip assigned to a jail isn’t configured on the machine
• alias numbering skips in the rc.conf (resulting in the above)
• orphaned vnfile's that aren't mentioned in a quad/safe
• ip mismatches between dir/vnfile name and the jail’s ip
• dir/vnfiles's in quad/safe that don’t exist

quadanalyze.pl

quadanalyze.pl

called by js, produces the info (seen above with js explanation) about the contents of quad (inode count, # of jails, etc.)

rsync.backup

rsync.backup

does customer backups (relies on backup.config)

taskdone

taskdone

when called will email support@johncompanies.com with the hostname of the machine from which it was executed as the subject

topten

topten

summarizes the top 10 traffic users (called by ipfwreset)

trafficgather.pl

trafficgather.pl [yy-mm]

sends a traffic usage summary by jail to support@johncomapnies.com and payments@johncompanies.com. Optional arguments are year and month (must be in the past). If not passed, assumes last month. Relies on traffic logs created by ipfwreset and ipfwbackup

trafficwatch.pl

trafficwatch.pl

checks traffic usage and emails support@johncomapnies.com when a jail reaches the warning level (35G) and the limit (40G). We really aren’t using this anymore now that we have netflow.

trafstats

trafstats

writes ipfw traffic usage info by jail to a file called jc_traffic_dump in each jail’s / dir

truncate_jailmake

truncate_jailmake

a version of jailmake which creates truncated vnfiles.

vb

vb

the equivalent of: vi /usr/local/jail/bin/backup.config

vs (freebsd)

vs<n>

the equivalent of: vi /usr/local/jail/rc.d/safe<n>

vq<n> the equivalent of: vi /usr/local/jail/rc.d/quad<n>

dumpremote

dumpremote <user@machine> </remote/location/file-dump> <vnX>

ex: dumpremote user@10.1.4.117 /mnt/data3/remote.echoditto.com-dump 7 this will dump a vn filesystem to a remote machine and location

oversellcheck

oversellcheck

displays how much a disk is oversold or undersold taking into account truncated vn files. Only for use on 4.x systems

mvbackups (freebsd)

mvbackups <dir> (1.1.1.1-col00001-DIR) <target_machine> (jail1) <target_dir> (data1)

moves backups from one location to another on the backup server, and provides you with option to remove entries from current backup.config, and simple paste command to add the config to backup.config on the target server

jailnice

jailnice <hostname>

applies renice 19 [PID] and rtprio 31 –[PID] to each process in the given jail

dumpremoterestore

dumpremoterestore <device> <ip of target machine> <dir on target machine>

ex: dumpremoterestore /dev/vn51 10.1.4.118 /mnt/data2/69.55.239.45-col00688-DIR dumps a device and restores it to a directory on a remote machine. Requires that you enable root ssh on the remote machine.

psj

psj

shows just the procs running on the base system – a ps auxw but without jail’d procs present

perc5iraidchk

perc5iraidchk

checks for degraded arrays on Dell 2950 systems with Perc5/6 controllers

perc4eraidchk

perc4eraidchk

checks for degraded arrays on Dell 2850 systems with Perc4e/Di controllers

Virtuozzo VPS

Making new customer VE (vm)

This applies only to new virts >= 4.x

grab ip from ipmap (if opened from the pending cust screen it should take you to the right block). You can also run vzlist -a to see what block is in use, generally. Try to find an IP that's in the same block of class C IP's already on the box.

1. confirm ip you want to use isn’t in use via Mgmt. -> IP Map in management screens

 also, ping the IP address.

2. put CT on whichever partition has more space

 df -h

3. vm CID IP hostname /vz[1|2] email[,email] template ( <LB|LBP|LS|LM|LMP|LP> [disk] | <gb_disk/mb_ram/gb_burst> )

vm col00009 69.55.230.238 centos.testdave.com /vz1 dsmith@johncompanies.com centos-6-x86_64 LM

4. copy veid, dir, ip and password to pending customer screen. activate customer

Making new customer VE (vemakexxx)

This applies to older virts with old templates. This should probably not be used at all anymore.

1. look thru hist for ip

2. confirm ip you want to use isn’t in use via Mgmt. -> IP Map in management screens

3. put ve on whichever partition has more space

vemakerh9 <veid> <ip> <hostname> <mount> <email> [gb disk]; <256|384|512> <veid>
vemakerh9 866 69.55.226.109 ngentu.com /vz1 ayo@ngantu.com,asd@asd.com 5; 256 866

4. copy (veid), dir, and ip to pending customer screen (pass set to p455agfa)

Note: We use VEID (Virtual Environment ID) and CTID (Container ID) interchangably. Similarly, VE and CT. They mean the same thing. VZPP = VirtuoZzo Power Panel (the control panel for each CT)

All linux systems exist in /vz, /vz1 or /vz2 - since each linux machine holds roughly 60-90 customers, there will be roughly 30-45 in each partition.

The actual filesystem of the system in question is in:

/vz(1-2)/private/(VEID)

Where VEID is the identifier for that system - an all-numeric string larger than 100.

The actual mounted and running systems are in the corresponding:

/vz(1-2)/root/(VEID)

But we rarely interact with any system from this mount point.

You should never need to touch the root portion of their system – however you can traverse their filesystem by going to /vz(1-2)/private/(VEID)/root (/vz(1-2)/private/(VEID)/fs/root on 4.x systems) the root of their filesystem is in that directory, and their entire system is underneath that.

Every VE has a startup script in /etc/sysconfig/vz-scripts (which is symlinked as /vzconf on all systems) - the VE startup script is simply named (VEID).conf - it contains all the system parameters for that VE:

# Configuration file generated by vzsplit for 60 VE
# on HN with total amount of physical mem 2011 Mb

VERSION="2"
CLASSID="2"

ONBOOT="yes"

KMEMSIZE="8100000:8200000"
LOCKEDPAGES="322:322"
PRIVVMPAGES="610000:615000"
SHMPAGES="33000:34500"
NUMPROC="410:415"
PHYSPAGES="0:2147483647"
VMGUARPAGES="13019:2147483647"
OOMGUARPAGES="13019:2147483647"
NUMTCPSOCK="1210:1215"
NUMFLOCK="107:117"
NUMPTY="19:19"
NUMSIGINFO="274:274"
TCPSNDBUF="1800000:1900000"
TCPRCVBUF="1800000:1900000"
OTHERSOCKBUF="900000:950000"
DGRAMRCVBUF="200000:200000"
NUMOTHERSOCK="650:660"
DCACHE="786432:818029"
NUMFILE="7500:7600"
AVNUMPROC="51:51"
IPTENTRIES="155:155"
DISKSPACE="4194304:4613734"
DISKINODES="400000:420000"
CPUUNITS="1412"
QUOTAUGIDLIMIT="2000"
VE_ROOT="/vz1/root/636"
VE_PRIVATE="/vz1/private/636"
NAMESERVER="69.55.225.225 69.55.230.3"
OSTEMPLATE="vzredhat-7.3/20030305"
VE_TYPE="regular"
IP_ADDRESS="69.55.225.229"
HOSTNAME="textengine.net"

As you can see, the hostname is set here, the disk space is set here, the number of inodes, the number of files that can be open, the number of tcp sockets, etc. - all are set here.

In fact, everything that can be set on this customer system is set in this conf file.

All interaction with the customer system is done with the VEID. You start the system by running:

vzctl start 999

You stop it by running:

vzctl stop 999

You execute commands in it by running:

vzctl exec 999 df -k

You enter into it, via a root-shell backdoor with:

vzctl enter 999

and you set parameters for the system, while it is still running, with:

vzctl set 999 --diskspace 6100000:6200000 --save

vzctl is the most commonly used command - we have aliased v to vzctl since we use it so often. We’ll continue to use vzctl in our examples, but feel free to use just v.

Let's say the user wants more diskspace. You can cat their conf file and see:

DISKSPACE="4194304:4613734"

So right now they have 4gigs of space. You can then change it to 6 with:

vzctl set 999 --diskspace 6100000:6200000 --save

IMPORTANT: all issuances of the vzctl set command need to end with –save - if they don't, the setting will be set, but it will not be saved to the conf file, and they will not have those settings next time they boot.

All of the tunables in the conf file can be set with the vzctl set command. Note that in the conf file, and on the vzctl set command line, we always issue two numbers seperated by a colon - that is because we are setting the hard and soft limits. Always set the hard limit slightly above the soft limit, as you see it is in the conf file for all those settings.

There are also things you can set with `vzctl set` that are not in the conf file as settings, per se. For instance, you can add IPs:

vzctl set 999 --ipadd 10.10.10.10 --save

or multiple IPs:

vzctl set 999 --ipadd 10.10.10.10 --ipadd 10.10.20.30 --save

or change the hostname:

vzctl set 999 --hostname www.example.com --save

You can even set the nameservers:

vzctl set 999 --nameserver 198.78.66.4 --nameserver 198.78.70.180 --save

Although you probably will never do that.

You can disable a VPS from being started (by VZPP or reboot) (4.x):

vzctl set 999 --disabled yes --save 

You can disable a VPS from being started (by VZPP or reboot) (<=3.x):

vzctl set 999 --onboot=no --save 

You can disable a VPS from using his control panel:

vzctl set 999 --offline_management=no --save 

You can suspend a VPS, so it can be resumed in the same state it was in when it was stopped (4.x):

vzctl suspend 999

and to resume it:

vzctl resume 999

to see who owns process:

vzpid <PID>

to mount up an unmounted ve:

vzctl mount 827

To see network stats for CT's:

# vznetstat
VEID    Net.Class  Output(bytes)   Input(bytes)
-----------------------------------------------
24218     1            484M             39M
24245     1            463M            143M
2451      1           2224M            265M
2454      1           2616M            385M
4149      1            125M             68M
418       1           1560M             34M
472       1           1219M            315M
726       1            628M            317M
763       1            223M             82M
771       1           4234M            437M
-----------------------------------------------
Total:               13780M           2090M

One thing that sometimes comes up on older systems that we created with smaller defaults is that the system would run out of inodes. The user will email and say they cannot create any more files or grow any files larger, but they will also say that they are not out of diskspace ... they are running:

df -k

and seeing how much space is free - and they are not out of space. They are most likely out of inodes - which they would see by running:

df -i

So, the first thing you should do is enter their system with:

vzctl enter 999

and run: df -i

to confirm your theory. Then exit their system. Then simply cat their conf file and see what their inodes are set to (probably 200000:200000, since that was the old default on the older systems) and run:

vzctl set 999 --diskinodes 400000:400000 --save

If they are not out of inodes, then a good possibility is that they have maxed out their numfile configuration variable, which controls how many files they can have in their system. The current default is 7500 (which nobody has ever hit), but the old default was as low as 2000, so you would run something like:

vzctl set 999 --numfile 7500:7500 --save

---

You cannot start or stop a VE if your pwd is its private (/vz/private/999) or root (/vz/root/999) directories, or anywhere below them.

Recovering from a crash (linux)

Diagnose whether you have a crash

The most important thing is to get the machine and all ve’s back up as soon as possible. Note the time, you’ll need to create a crash log entry (Mgmt. -> Reference -> CrashLog). The first thing to do is head over to the serial console screen and see if there’s any kernel error messages output. Try to copy any messages (or just a sample of repeating messages) you see into the notes section of the crash log – these will also likely need to be sent to virtuozzo for interpretation. If the messages are spewing too fast, hit ^O + H to start a screen log dump which you can ob1182.pts-38.bb serve after the machine is rebooted. Additionally, if the machine is responsive, you can get a trace to send to virtuozzo by hooking up a kvm and entering these 3 sequences:

alt+print screen+m
alt+print screen+p
alt+print screen+t

If there are no messages, the machine may just be really busy- wait a bit (5-10min) to see if it comes back. If it's still pinging, odds are its very busy. If it doesn't come back, or the messages indicate a fatal error, you will need to proceed with a power cycle (ctrl+alt+del will not work).

Power cycle the server

If this machine is not a Dell 2950 with a DRAC card (i.e. if you can’t ssh into the DRAC card and issue racadm serveraction hardreset, then you will need someone at the data center to power the macine off, wait 30 sec, then turn it back on. Make sure to re-attach via console (tip virtxx) immediately after power down.

(Re)attach to the console

Stay on the console the entire time during boot. As the BIOS posts- look out for the RAID card output- does everything look healthy? The output may be scrambled, look for "DEGRADED" or "FAILED". Once the OS starts booting you will be disconnected (dropped back to the shell on the console server) a couple times during the boot up. The reason you want to quickly re-attach is two-fold: 1. If you don’t reattach quickly then you won’t get any console output, 2. you want to be attached before the server potentially starts (an extensive) fsck. If you attach after the fsck begins, you’ll have seen no indication it started an fsck and the server will appear frozen during startup- no output, no response.

Start containers/VE's/VPSs

When the machine begins to start VE’s, it’s safe to leave the console and login via ssh. All virts should be set to auto start all the VEs after a crash. Further, most (newer) virts are set to “fastboot” it’s VE’s (to find out, do:

grep -i fast /etc/sysconfig/vz 

and look for VZFASTBOOT=yes). If this was set prior to the machine’s crash (setting it after the machine boots will not have any effect until the vz service is restarted) it will start each ve as fast as possible, in serial, then go thru each VE (serially), shutting it down running a vzquota (disk usage) check, then bringing it back up. The benefit is that all VE’s are brought up quickly (within 15min or so depending on the #), the downside is a customer watching closely will notice 2 outages – 1st the machine crash, 2nd their quota check (which will be a much shorter downtime- on the order of a few minutes).

Where “fastboot” is not set to yes (i.e on quar1), vz will start them consecutively, checking the quotas one at a time, and the 60th VE may not start until an hour or two later - this is not acceptable.

The good news is, if you run vzctl start for a VE that is already started, you will simply get an error: VE is already started. Further, if you attempt to vzctl start a VE that is in the process of being started, you will simply get an error: unable to lock VE. So, there is no danger in simply running scripts to start smaller sets of VEs. If the system is not autostarting, then there is no issue, and even if it does, when it conflicts, one process (yours or the autostart) will lose, and just move on to the next one.

A script has been written to assist with ve starts: startvirt.pl which will start 6 ve’s at once until there are no more left. If startvirt.pl is used on a system where “fastboot” was on, it will circumvent the fastboot for ve’s started by startvirt.pl – they will go through the complete quota check before starting- therefore this is not advisable when a system has crashed. When a system is booted cleanly, and there's no need for vzquota checks, then startvirt.pl is safe and advisable to run.

Make sure all containers are running

You can quickly get a feel for how many ve’s are started by running:

[root@virt4 log]# vs
VEID 16066 exist mounted running
VEID 16067 exist mounted running
VEID 4102 exist mounted running
VEID 4112 exist mounted running
VEID 4116 exist mounted running
VEID 4122 exist mounted running
VEID 4123 exist mounted running
VEID 4124 exist mounted running
VEID 4132 exist mounted running
VEID 4148 exist mounted running
VEID 4151 exist mounted running
VEID 4155 exist mounted running
VEID 42 exist mounted running
VEID 432 exist mounted running
VEID 434 exist mounted running
VEID 442 exist mounted running
VEID 450 exist mounted running
VEID 452 exist mounted running
VEID 453 exist mounted running
VEID 454 exist mounted running
VEID 462 exist mounted running
VEID 463 exist mounted running
VEID 464 exist mounted running
VEID 465 exist mounted running
VEID 477 exist mounted running
VEID 484 exist mounted running
VEID 486 exist mounted running
VEID 490 exist mounted running

So to see how many ve’s have started:

[root@virt11 root]# vs | grep running | wc -l
     39

And to see how many haven’t:

[root@virt11 root]# vs | grep down | wc -l
     0

And how many we should have running:

[root@virt11 root]# vs | wc -l
     39

Another tool you can use to see which ve’s have started, among other things is vzstat. It will give you CPU, memory, and other stats on each ve and the overall system. It’s a good thing to watch as ve’s are starting (note the VENum parameter, it will tell you how many have started):

4:37pm, up 3 days,  5:31,  1 user, load average: 1.57, 1.68, 1.79
VENum 40, procs 1705: running 2, sleeping 1694, unint 0, zombie 9, stopped 0
CPU [ OK ]: VEs  57%, VE0   0%, user   8%, sys   7%, idle  85%, lat(ms) 412/2
Mem [ OK ]: total 6057MB, free 9MB/54MB (low/high), lat(ms) 0/0
Swap [ OK ]: tot 6142MB, free 4953MB, in 0.000MB/s, out 0.000MB/s
Net [ OK ]: tot: in  0.043MB/s  402pkt/s, out  0.382MB/s 4116pkt/s
Disks [ OK ]: in 0.002MB/s, out 0.000MB/s

  VEID ST    %VM     %KM         PROC    CPU     SOCK FCNT MLAT IP
     1 OK 1.0/17  0.0/0.4    0/32/256 0.0/0.5 39/1256    0    9 69.55.227.152
    21 OK 1.3/39  0.1/0.2    0/46/410 0.2/2.8 23/1860    0    6 69.55.239.60
   133 OK 3.1/39  0.1/0.3    1/34/410 6.3/2.8 98/1860    0    0 69.55.227.147
   263 OK 2.3/39  0.1/0.2    0/56/410 0.3/2.8 34/1860    0    1 69.55.237.74
   456 OK  17/39  0.1/0.2   0/100/410 0.1/2.8 48/1860    0   11 69.55.236.65
   476 OK 0.6/39  0.0/0.2    0/33/410 0.1/2.8 96/1860    0   10 69.55.227.151
   524 OK 1.8/39  0.1/0.2    0/33/410 0.0/2.8 28/1860    0    0 69.55.227.153
   594 OK 3.1/39  0.1/0.2    0/45/410 0.0/2.8 87/1860    0    1 69.55.239.40
   670 OK 7.7/39  0.2/0.3    0/98/410 0.0/2.8 64/1860    0  216 69.55.225.136
   691 OK 2.0/39  0.1/0.2    0/31/410 0.0/0.7 25/1860    0    1 69.55.234.96
   744 OK 0.1/17  0.0/0.5    0/10/410 0.0/0.7  7/1860    0    6 69.55.224.253
   755 OK 1.1/39  0.0/0.2    0/27/410 0.0/2.8 33/1860    0    0 192.168.1.4
   835 OK 1.1/39  0.0/0.2    0/19/410 0.0/2.8  5/1860    0    0 69.55.227.134
   856 OK 0.3/39  0.0/0.2    0/13/410 0.0/2.8 16/1860    0    0 69.55.227.137
   936 OK 3.2/52  0.2/0.4    0/75/410 0.2/0.7 69/1910    0    8 69.55.224.181
  1020 OK 3.9/39  0.1/0.2    0/60/410 0.1/0.7 55/1860    0    8 69.55.227.52
  1027 OK 0.3/39  0.0/0.2    0/14/410 0.0/2.8 17/1860    0    0 69.55.227.83
  1029 OK 1.9/39  0.1/0.2    0/48/410 0.2/2.8 25/1860    0    5 69.55.227.85
  1032 OK  12/39  0.1/0.4    0/80/410 0.0/2.8 41/1860    0    8 69.55.227.90

When you are all done, you will want to make sure that all the VEs really did get started, run vs one more time.

Note the time all ve’s are back up and enter that into and save the crash log entry.

Occasionally, a ve will not start automatically. The most common reason for a ve not to come up normally is the ve was at it’s disk limit before the crash, and will not start since they’re over the limit. To overcome this, set the disk space to current usage level (the system will give this to you when it fails to start), start the ve, then re-set the disk space back to the prior level. Lastly, contact the customer to let them know they’re out of disk (or allocate more disk if they're entitled to more).

Hitting performance barriers and fixing them

There are multiple modes virtuozzo offers to allocate resources to a ve. We utilize 2: SLM and UBC parameters On our 4.x systems, we use all SLM – it’s simpler to manage and understand. There are a few systems on virt19/18 that may also use SLM. Everything else uses UBC. You can tell a SLM ve by:

SLMMODE="all"

in their conf file.

TODO: detail SLM modes and parameters.

If someone is in SLM mode and they hit memory resource limits, they simply need to upgrade to more memory.

The following applies to everyone else (UBC).

Customers will often email and say that they are getting out of memory errors - a common one is "cannot fork" ... basically, anytime you see something odd like this, it means they are hitting one of their limits that is in place in their conf file.

The conf file, however, simply shows their limits - how do we know what they are currently at ?

The answer is a file called v - this file contains the current status (and peaks) of their performance settings, and also counts how many times they have hit the barrier. The output of the file looks like this:

764: kmemsize         384113     898185    8100000    8200000          0
     lockedpages           0          0        322        322          0
     privvmpages        1292       7108     610000     615000          0
     shmpages            270        528      33000      34500          0
     dummy                 0          0          0          0          0
     numproc               8         23        410        415          0
     physpages            48       5624          0 2147483647          0
     vmguarpages           0          0      13019 2147483647          0
     oomguarpages        641       6389      13019 2147483647          0
     numtcpsock            3         21       1210       1215          0
     numflock              1          3        107        117          0
     numpty                0          2         19         19          0
     numsiginfo            0          4        274        274          0
     tcpsndbuf             0      80928    1800000    1900000          0 
     tcprcvbuf             0     108976    1800000    1900000          0
     othersockbuf       2224      37568     900000     950000          0
     dgramrcvbuf           0       4272     200000     200000          0
     numothersock          3          9        650        660          0
     dcachesize        53922     100320     786432     818029          0
     numfile             161        382       7500       7600          0
     dummy                 0          0          0          0          0
     dummy                 0          0          0          0          0
     dummy                 0          0          0          0          0
     numiptent             4          4        155        155          0

The first column is the name of the counter in question - the same names we saw in the systems conf file. The second column is the _current_ value of that counter, the third column is the max that that counter has ever risen to, the fourth column is the soft limit, and the fifth column is the hard limit (which is the same as the numbers in that systems conf file).

The sixth number is the failcount - how many times the current usage has risen to hit the barrier. It will increase as soon as the current usage hits the soft limit.

The problem with /proc/user_beancounters is that it actually contains that set of data for every running VE - so you can't just cat /proc/user_beancounters - it is too long and you get info for every other running system.

You can vzctl enter the system and run:

vzctl enter 9999
cat /proc/user_beancounters

inside their system, and you will just see the stats for their particular system, but entering their system every time you want to see it is combersome.

So, I wrote a simple script called "vzs" which simply greps for the VEID, and spits out the next 20 or so lines (however many lines there are in the output, I forget) after it. For instance:

# vzs 765:
765: kmemsize        2007936    2562780    8100000    8200000          0
     lockedpages           0          8        322        322          0
     privvmpages       26925      71126     610000     615000          0
     shmpages          16654      16750      33000      34500          0
     dummy                 0          0          0          0          0
     numproc              41         57        410        415          0
     physpages          1794      49160          0 2147483647          0
     vmguarpages           0          0      13019 2147483647          0
     oomguarpages       4780      51270      13019 2147483647          0
     numtcpsock           23         37       1210       1215          0
     numflock             17         39        107        117          0
     numpty                1          3         19         19          0
     numsiginfo            0          6        274        274          0
     tcpsndbuf         22240     333600    1800000    1900000          0
     tcprcvbuf             0     222656    1800000    1900000          0
     othersockbuf     104528     414944     900000     950000          0
     dgramrcvbuf           0       4448     200000     200000          0
     numothersock         73        105        650        660          0
     dcachesize       247038     309111     786432     818029          0
     numfile             904       1231       7500       7600          0
     dummy                 0          0          0          0          0
     dummy                 0          0          0          0          0
     dummy                 0          0          0          0          0
     numiptent             4          4        155        155          0

That showed us just the portion of /proc/user_beancounters for system 765.

When you run the vzs command, always add a : after the VEID.

So, if a customer complains about some out of memory errors, or no more files, or no more ptys, or just has an unspecific complain about processes dying, etc., the very first thing you need to do is check their beancounters with vzs. Usually you will spot an item that has a high failcount and needs to be upped.

At that point you could simply up the counter with `vzctl set`. Generally pick a number 10-20% higher than the old one, and make the hard limit slightly larger than the the soft limit. However our systems now come in several levels and those levels have more/different memory allocations. If someone is complaining about something other than a memory limit (pty, numiptent, numflock), it’s generally safe to increase it, at least to the same level as what’s in the /vzconf/4unlimited file on the newest virt. If someone is hitting a memory limit, first make sure they are given what they deserve:

(refer to mgmt -> payments -> packages)

To set those levels, you use the setmem command.

The alternate (DEPRECATED) method would be to use one of 3 commands: 256 <veid> 300 <veid> 384 <veid> 512 <veid>

If the levels were not right (you’d run vzs <veid> before and after to see the effect) tell the customer they’ve been adjusted and be done with it. If the levels were right, tell the customer they must upgrade to a higher package, tell them how to see level (control panel) and that they can reboot their system to escape this lockup contidion.

Customers can also complain that their site is totally unreachable, or complain that it is down ... if the underlying machine is up, and all seems well, you may notice in the beancounters that network-specific counters are failing - such as numtcpsock, tcpsndbuf or tcprcvbuf. This will keep them from talking on the network and make it seem like their system is down. Again, just up the limits and things should be fine.

On virts 1-4, you should first look at the default settings for that item on a later virt, such as virt 8 - we have increased the defaults a lot since the early machines. So, if you are going to up a counter on virt2, instead of upping it by 10-20%, instead up it to the new default that you see on virt8.

Moving a VE to another virt (migrate/migrateonline)

This will take a while to complete - and it is best to do this at night when the load is light on both machines.

There are different methods for this, depending on which version of virtuozzo is installed on the src. and dst. virt. To check which version is running:

[root@virt12 private]# cat /etc/virtuozzo-release
Virtuozzo release 2.6.0

Ok, let's say that the VE is 1212, and vital stats are:

[root@virt12 sbin]# vc 1212
VE_ROOT="/vz1/root/1212"
VE_PRIVATE="/vz1/private/1212"
OSTEMPLATE="fedora-core-2/20040903"
IP_ADDRESS="69.55.229.84"
TEMPLATES="devel-fc2/20040903 php-fc2/20040813 mysql-fc2/20040812 postgresql-fc2/20040813 mod_perl-fc2/20040812 mod_ssl-fc2/20040811 jre-fc2/20040823 jdk-fc2/20040823 mailman-fc2/20040823 analog-fc2/20040824 proftpd-fc2/20040818 tomcat-fc2/20040823 usermin-fc2/20040909 webmin-fc2/20040909 uw-imap-fc2/20040830 phpBB-fc2/20040831 spamassassin-fc2/20040910 PostNuke-fc2/20040824 sl-webalizer-fc2/20040
818"

[root@virt12 sbin]# vzctl exec 1212 df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/vzfs             4.0G  405M  3.7G  10% /

From this you can see that he’s using (and will minimally need free on the dst server) ~400MB, and he’s running on a Fedora 2 template, version 20040903. He’s also got a bunch of other templates installed. It’s is vital that all these templates exist on the dst system. To confirm that, on the dst system run:

For < 3.0:

[root@virt14 private]# vzpkgls | grep fc2
devel-fc2 20040903
PostNuke-fc2 20040824
analog-fc2 20040824
awstats-fc2 20040824
bbClone-fc2 20040824
jdk-fc2 20040823
jre-fc2 20040823
mailman-fc2 20040823
mod_frontpage-fc2 20040816
mod_perl-fc2 20040812
mod_ssl-fc2 20040811
mysql-fc2 20040812
openwebmail-fc2 20040817
php-fc2 20040813
phpBB-fc2 20040831
postgresql-fc2 20040813
proftpd-fc2 20040818
sl-webalizer-fc2 20040818
spamassassin-fc2 20040910
tomcat-fc2 20040823
usermin-fc2 20040909
uw-imap-fc2 20040830
webmin-fc2 20040909
[root@virt14 private]# vzpkgls | grep fedora
fedora-core-1 20040121 20040818
fedora-core-devel-1 20040121 20040818
fedora-core-2 20040903
[root@virt14 private]#

For these older systems, you can simply match up the date on the template.

For >= 3.0:

[root@virt19 /vz2/private]# vzpkg list
centos-5-x86                    2008-01-07 22:05:57
centos-5-x86    devel
centos-5-x86    jre
centos-5-x86    jsdk
centos-5-x86    mod_perl
centos-5-x86    mod_ssl
centos-5-x86    mysql
centos-5-x86    php
centos-5-x86    plesk9
centos-5-x86    plesk9-antivirus
centos-5-x86    plesk9-api
centos-5-x86    plesk9-atmail
centos-5-x86    plesk9-backup
centos-5-x86    plesk9-horde
centos-5-x86    plesk9-mailman
centos-5-x86    plesk9-mod-bw
centos-5-x86    plesk9-postfix
centos-5-x86    plesk9-ppwse
centos-5-x86    plesk9-psa-firewall
centos-5-x86    plesk9-psa-vpn
centos-5-x86    plesk9-psa-fileserver
centos-5-x86    plesk9-qmail
centos-5-x86    plesk9-sb-publish
centos-5-x86    plesk9-vault
centos-5-x86    plesk9-vault-most-popular
centos-5-x86    plesk9-watchdog

On these newer systems, it's difficult to tell whether the template on the dst matches exactly the src. Just cause a centos-5-x86 is listed on both servers doesn't mean all the same packages are there on the dst. To truly know, you must perform a sample rsync:

rsync -avn /vz/template/centos/5/x86/ root@10.1.4.61:/vz/template/centos/5/x86/

if you see a ton of output from the dry run command, then clearly there are some differences. You may opt to let the rsync complete (without running in dry run mode) the only downside is you've now used up more space on the dst and also the centos template will be a mess with old and new data- it will be difficult if not impossible to undo (if someday we wanted to reclaim the space).

If you choose to merge templates, you should closely inspect the dry run output. You should also take care to exclude anything in the /config directory. For example:

rsync -av -e ssh --stats --exclude=x86/config  /vz/template/ubuntu/10.04/ root@10.1.4.62:/vz/template/ubuntu/10.04/

Which will avoid this directory and contents:

[root@virt11 /vz2/private]# ls /vz/template/ubuntu/10.04/x86/config*
app  os

This is important to avoid since the config may differ on the destination and we are really only interested in making sure the pacakges are there, not overwriting a newer config with an older one.

If the dst system was missing a template, you have 2 choices:

1. put the missing template on the dst system. 2 choices here:
   1. Install the template from rpm (found under backup2: /mnt/data4/vzrpms/distro/) or
   2. rsync over the template (found under /vz/template) - see above
2. put the ve on a system which has all the proper templates

pre-seeding a migration

When migrating a customer (or when doing many) depending on how much data you have to transfer, it can take some time. Further, it can be difficult to gauge when a migration will complete or how long it will take. To help speed up the process and get a better idea about how long it will take you can pre-transfer a customer's data to the destination server. If done correctly, vzmigrate will see the pre-transferred data and pick up where you left off, having much less to transfer (just changed/new files).

We believe vzmigrate uses rsync to do it's transfer. Therefore not only can you use rsync to do a pre-seed, you can also run rsync to see what is causing a repeatedly-failing vzmigrate to fail.

There's no magic to a pre-seed, you just need to make sure it's named correctly.

Given:

source: /vz1/private/1234

and you want to migrate to /vz2 on the target system, your rsync would look like:

rsync -av /vz1/private/1234/ root@x.x.x.x:/vz2/private/1234.migrated/

After running that successful rsync, the ensuing migrateonline (or migrate) will take much less time to complete- depending on the # of files to be analyzed and the # of changed files. In any case, it'll be much much faster than had you just started the migration from scratch.

Further, as we discuss elsewhere in this topic, a failed migration can be moved from /vz/private/1234 to /vz/private/1234.migrated on the destination if you want to restart a failed migration. This should only be done if the migration failed and the CT is not running on the destination HN.

migrateonline intructions: src >=3.x -> dst>=3.x

A script called migrateonline was written to handle this kind of move. It is basically a wrapper for vzmigrate – vzmigrate is a util to seamlessly- as no no reboot of the ve necessary- move a ve from one host to another. This wrapper was initially written cause vz virtuozzo version 2.6.0 has a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables, causing problems when the ve was started up on the dst system. migrate mitigates that. Since it makes multiple ssh connections to the dst virt, it’s a good idea to put the pub key for the src virt in the authorized_keys file on the dst virt. In addition, migrateonline emails ve owners when their migration starts and stops. For this to happen they need to put email addresses (on a single line, space delimited) in a file on their system: /migrate_notify. If the optional target dir is not specified, the ve will be moved to the same private/root location as it was on the src virt. Note: migrate is equivalent to migrateonline, but will migrate a ve AND restart it in the process.

[root@virt12 sbin]# migrateonline
usage: /usr/local/sbin/migrateonline <ip of node migrating to> <veid> [target dir: vz | vz1 | vz2]
[root@virt12 sbin]# migrateonline 10.1.4.64 1212 vz
starting to migrate 1212 at Sat Mar 26 22:40:38 PST 2005
Turning off offline_management
Saved parameters for VE 1212
migrating with no start on 10.1.4.64
Connection to destination HN (10.1.4.64) is successfully established
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...
Syncing private area '/vz1/private/1212'
- 100% |*************************************************|
done
Successfully completed
clearing the arp cache
now going to 10.1.4.64 and clear cache and starting
starting it
Delete port redirection
Adding port redirection to VE(1): 4643 8443
Adding IP address(es) to pool: 69.55.229.84
Saved parameters for VE 1212
Starting VE ...
VE is mounted
Adding port redirection to VE(1): 4643 8443
Adding IP address(es): 69.55.229.84
Hostname for VE set: fourmajor.com
File resolv.conf was modified
VE start in progress...
finished migrating 1212 at Sat Mar 26 22 22:52:01 PST 2005
[root@virt12 sbin]#

Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine.

If they had backups, use the mvbackups command to move their backups to the new server:

mvbackups 1212 virt14 vz

Rename the ve

[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/migrated-1212

[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/old-1212-migrated-20120404-noarchive

Update the customer’s systems in mgmt to reflect the new path and server.

IF migrateonline does not work, you can try again using simply migrate- this will result in a brief reboot for the ve. Before you try again, make sure of a few things:

Depending on where in the migration died, there may be partial data on the dst system in 1 of 2 places: (given the example above)

/vz/private/1212

or

/vz/private/1212.migrated

before you run migrate again, you'll want to rename so that all data is in 1212.migrated:

mv /vz/private/1212 /vz/private/1212.migrated

this way, it will pick up where it left off and transfer only new files.

Likewise, if you want to speed up a migration, you can pre-seed the dst as follows:

[root@virt12 sbin]# rsync -avSH /vz/private/1212/ root@10.1.4.64:/vz/private/1212.migrated/

then when you run migrate or migrateonline, it will only need to move the changed files- the migration will complete quickly

migrateonline/migrate failures (migrate manually)

Lets say for whatever reason the migration fails. If it fails with migrateonline, you should try migrate (which will reboot the customer, so notify them ahead of time).

You may want to run a pre-seed rsync to see if you can find the problem. On older virts, we've seen this problem due to a large logfile (which you can find and encourage the customer to remove/compress):

for f in `find / -size +1048576k`; do ls -lh $f; done

You may also see migration failing due to quota issues.

You can try to resolve by copying any quota file into the file you need:

cp /var/vzquota/quota.1 /var/vzquota/quota.xxx

If it complains about quota running you should then be able to stop it

vzquota off xxxx

If all else fails, migrate to a new VEID i.e. 1234 becomes 12341

If the rsync or migrate fails, you can always move someone manually:

1. stop ve:

v stop 1234

2. copy over data

rsync -avSH /vz/private/1234/ root@1.1.1.1:/vzX/private/1234/

NOTE: if you've previously seeded the data (run rsync while the VE was up/running), and this is a subsequent rsync, make sure the last rsync you do (while the VE is not running, has the --delete option in the rsync)

3. copy over conf

scp /vzconf/1234.conf root@1.1.1.1:/vzconf

4. on dst, edit the conf to reflect the right vzX dir

vi /vzconf/1234.conf

5. on src remove the IPs

ipdel 1234 2.2.2.2 3.3.3.3

6. on dst add IPs

ipadd 1234 2.2.2.2 3.3.3.3

7. on dst, start ve:

v start 1324

8. cancel, then archive ve on src per above instrs.

migrate src=2.6.0 -> dst>=2.6.0, or mass-migration with customer notify

A script called migrate was written to handle this kind of move. It is basically a wrapper for vzmigrate – vzmigrate is a util to seamlessly move a ve from one host to another. This wrapper was initially written cause vz virtuozzo version 2.6.0 has a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables, causing problems when the ve was started up on the dst system. migrate mitigates that. Since it makes multiple ssh connections to the dst virt, it’s a good idea to put the pub key for the src virt in the authorized_keys file on the dst virt. In addition, migrate emails ve owners when their migration starts and stops. For this to happen they need to put email addresses (on a single line, space delimited) in a file on their system: /migrate_notify. If the optional target dir is not specified, the ve will be moved to the same private/root location as it was on the src virt. Note: migrateonline is equivalent to migrate, but will migrate a ve from one 2.6 kernel machine to another 2.6 kernel machine without restarting the ve.

[root@virt12 sbin]# migrate
usage: /usr/local/sbin/migrate <ip of node migrating to> <veid> [target dir: vz | vz1 | vz2]
[root@virt12 sbin]# migrate 10.1.4.64 1212 vz
starting to migrate 1212 at Sat Mar 26 22:40:38 PST 2005
Turning off offline_management
Saved parameters for VE 1212
migrating with no start on 10.1.4.64
Connection to destination HN (10.1.4.64) is successfully established
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...
Syncing private area '/vz1/private/1212'
- 100% |*************************************************|
done
Successfully completed
clearing the arp cache
now going to 10.1.4.64 and clear cache and starting
starting it
Delete port redirection
Adding port redirection to VE(1): 4643 8443
Adding IP address(es) to pool: 69.55.229.84
Saved parameters for VE 1212
Starting VE ...
VE is mounted
Adding port redirection to VE(1): 4643 8443
Adding IP address(es): 69.55.229.84
Hostname for VE set: fourmajor.com
File resolv.conf was modified
VE start in progress...
finished migrating 1212 at Sat Mar 26 22 22:52:01 PST 2005
[root@virt12 sbin]#

Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine (backup2 or mail). Cancel the ve (first we have to rename things which migrate changed so cancelve will find them):

[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/1212.conf

On 2.6.1 you’ll also have to move the private area:

[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/1212

[root@virt12 sbin]# cancelve 1212
v stop 1212
v set 1212 --offline_management=no --save
Delete port redirection
Deleting IP address(es) from pool: 69.55.229.84
Saved parameters for VE 1212
mv /vzconf/1212.conf /vzconf/deprecated-1212
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414
don't forget to remove firewall rules and domains!
[root@virt12 sbin]#

Note: if the system had backups, cancelve would offer to remove them. You want to say no to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt. Then go to backup2 and move the dirs. So you’d do something like this:

mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/

We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. If moving to the same drive, you can safely preserve hardlinks and move all files with:

sh
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212  /mnt/data1/virt14/$f/vz/private/; done

To move everyone off a system, you’d do:

for f in `vl`; do migrate <ip> $f; done

Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.

vzmigrate: src=2.6.1 -> dst>=2.6.0

This version of vzmigrate works properly with regard to handling ips. It will not notify ve owners of moves as in the above example. Other than that it’s essentially the same.

[root@virt12 sbin]#  vzmigrate 10.1.4.64 -r no 1212:1212:/vz/private/1212:/vz/root/1212
migrating on 10.1.4.64
Connection to destination HN (10.1.4.64) is successfully established
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...
Syncing private area '/vz1/private/1212'
- 100% |*************************************************|
done
Successfully completed
Adding port redirection to VE(1): 4643 8443
Adding IP address(es) to pool: 69.55.229.84
Saved parameters for VE 1212
Starting VE ...
VE is mounted
Adding port redirection to VE(1): 4643 8443
Adding IP address(es): 69.55.229.84
Hostname for VE set: fourmajor.com
File resolv.conf was modified
VE start in progress...
[root@virt12 sbin]#

Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine (backup2 or mail). Cancel the ve (first we have to rename things which vzmigrate changed so cancelve will find them):

[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/1212.conf
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/1212

[root@virt12 sbin]# cancelve 1212
v stop 1212
v set 1212 --offline_management=no --save
Delete port redirection
Deleting IP address(es) from pool: 69.55.229.84
Saved parameters for VE 1212
mv /vzconf/1212.conf /vzconf/deprecated-1212
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414
don't forget to remove firewall rules and domains!
[root@virt12 sbin]#

Note: if the system had backups, cancelve would offer to remove them. You want to say no to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt. Then go to backup2 and move the dirs. So you’d do something like this:

mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/

We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. If moving to the same drive, you can safely preserve hardlinks and move all files with:

sh
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212  /mnt/data1/virt14/$f/vz/private/; done

Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.

src=2.5.x

First, go to the private dir:

cd /vz1/private/

Stop the VE - make sure it stops totally cleanly.

vzctl stop 1212

Then you’d use vemove - a script written to copy over the config, create tarballs of the ve’s data on the destination virt, and cancel the ve on the source system (in this example we’re going to put a ve that was in /vz1/private on the src virt, in /vz/private on the dst virt):

[root@virt12 sbin]# vemove
ERROR: Usage: vemove veid target_ip target_path_dir
[root@virt12 sbin]# vemove 1212 10.1.4.64 /vz/private/1212
tar cfpP - 1212 --ignore-failed-read | (ssh -2 -c arcfour 10.1.4.64 "split - -b 1024m /vz/private/1212.tar" )
scp /vzconf/1212.conf 10.1.4.64:/vzconf
cancelve 1212
v stop 1212
v set 1212 --offline_management=no --save
Delete port redirection
Deleting IP address(es) from pool: 69.55.229.84
Saved parameters for VE 1212
mv /vzconf/1212.conf /vzconf/deprecated-1212
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414
don't forget to remove firewall rules and domains!
[root@virt12 sbin]#

Note: if the system had backups, cancelve would offer to remove them. You want to say no to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt. Then go to backup2 and move the dirs. So you’d do something like this:

mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/

We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. If moving to the same drive, you can safely preserve hardlinks and move all files with:

sh
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212  /mnt/data1/virt14/$f/vz/private/; done

When you are done, go to /vz/private on the dst virt you will have files like this:

1212.taraa
1212.tarab
1212.tarac

Each one 1024m (or less, for the last one) in size.

on the dst server and run:

cat 1212.tar?? | tar xpPBf -

and after 20 mins or so it will be totally untarred. Now since the conf file is already there, you can go ahead and start the system.

vzctl start 1212

Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.

NOTE: you MUST tar the system up using the virtuozzo version of tar that is on all the virt systems, and further you MUST untar the tarball with the virtuozzo tar, using these options: `tar xpPBf -`

If you tar up an entire VE and move it to a non-virtuozzo machine, that is ok, and you can untar it there with normal tar commands, but do not untar it and then repack it with a normal tar and expect it to work - you need to use virtuozzo tar commands on virtuozzo tarballs to make it work.

The backups are sort of an exception, since we are just (usually) restoring user data that was created after we gave them the system, and therefore has nothing to do with magic symlinks or vz-rpms, etc.

Moving a VE on the same virt

Easy way:
Scenario 1: ve 123 is to be renamed 1231 and moved from vz1 to vz

vzmlocal 123:1231:/vz/private/1231:/vz/root/1231

Scenario 2: ve 123 is to be moved vz1 to vz

vzmlocal 123:123:/vz/private/123:/vz/root/123

vzmlocal will reboot the ve at the end of the move

Note: don't forget to fix /usr/local/sbin/backup.config for new disk location

Manual/old way:

1) vzctl stop 123
2) mv /vz1/private/123 /vz/private/.
(or cp -a if you want to copy) 3) in /etc/sysconfig/vz-scripts/123.conf change value
of 'VE_PRIVATE' variable to point to a new private area location 4) vzctl start 123
5) update backups if needed: mvbackups 123 virtX virt1 vz
6) update management scerens

Notes: a) absolute path to private area is stored in quota file /var/vzquota/quota.123 - so during first startup quota will be recalculated.
b) if you're going to write some script to do a job, you MUST be sure that $VEID won't be expanded to in ve config file - ie. you need to escape '$'. Otherwise you might have:

VE_PRIVATE="/vz/private/"

in config, and 'vzctl destroy' for this VE ID will remove everything under /vz/private/ directory.

Adding a veth device to a VE

Not totally sure what this is, but a customer asked for it and here's what we did (as instructed by vz support):

v set 99 --netif_add eth99  --save
ipdel 99 69.55.230.58
v set 99 --ifname eth99 --ipadd 69.55.230.58 --save
v set 99 --ifname eth99 --gateway 69.55.230.1 --save

vznetcfg net new veth_net

# vznetcfg net list
Network ID        Status      Master Interface  Slave Interfaces
net99             active      eth0              veth77.77,veth99.99
veth_net          active

# vznetcfg if list
Name             Type       Network ID       Addresses
br0              bridge     veth_net
br99             bridge     net99
veth99.99        veth       net99
eth1             nic                         10.1.4.62/24
eth0             nic        net99            69.55.227.70/24

vznetcfg br attach br0 eth0

(will remove 99 from orig net and move to veth_net)
vznetcfg net addif veth_net veth99.99

# vznetcfg net list
Network ID        Status      Master Interface  Slave Interfaces
net99             active
veth_net          active      eth0              veth77.77,veth99.99

(delete the old crap)
vznetcfg net del net99


Then, to add another device in

v set 77 --netif_add eth77  --save
ipdel 77 69.55.230.78
v set 77 --ifname eth77 --ipadd 69.55.230.78 --save
v set 77 --ifname eth77 --gateway 69.55.230.1 --save
v set 77 --save --ifname eth77 --network veth_net

# vznetcfg if list
Name             Type       Network ID       Addresses
veth77.77        veth
br0              bridge     veth_net
veth99.99        veth       veth_net
eth1             nic                         10.1.4.62/24
eth0             nic        veth_net         69.55.227.70/24

vznetcfg net addif veth_net veth77.77

# vznetcfg if list
Name             Type       Network ID       Addresses
veth77.77        veth       veth_net
br0              bridge     veth_net
veth99.99        veth       veth_net
eth1             nic                         10.1.4.62/24
eth0             nic        veth_net         69.55.227.70/24

# vznetcfg net list
Network ID        Status      Master Interface  Slave Interfaces
veth_net          active      eth0              veth77.77,veth99.99

another example

v set 1182 --netif_add eth1182  --save
ipdel 1182 69.55.236.217
v set 1182 --ifname eth1182 --ipadd 69.55.236.217 --save
v set 1182 --ifname eth1182 --gateway 69.55.236.1 --save
vznetcfg net addif veth_net veth1182.1182
v set 1182 --save --ifname eth1182 --network veth_net


unused/not working commands:
ifconfig veth99.0 0
vznetcfg net list
vznetcfg br new br99 net99
vznetcfg br attach br99 eth0
vznetcfg br show
vznetcfg if list
vznetcfg net addif net99 veth99.99
vznetcfg net addif eth0 net99

---------

vznetcfg br new br1182 net1182

vznetcfg br attach br99 eth0
vznetcfg if list
vznetcfg net addif net99 veth99.99
vznetcfg net addif eth0 net99



vznetcfg net addif eth0 net1182


# vznetcfg net addif veth99.0 net99
--- 8< ---

vznetcfg net new net
# vznetcfg net addif eth0 net99


# vzctl set 99 --save --netif_add eth0 (at this stage veth99.0 interface have to appear
on node)
# vzctl set 99 --save --ifname eth0 --ipadd 69.55.230.58 (and probably few more arguments
here - see 'man vzctl')
# vznetcfg net addif veth99.0 net99

Assigning/remove ip from a VE

1. Add or remove ips:

ipdel 1234 1.1.1.1 2.2.2.2
ipadd 1234 1.1.1.1 2.2.2.2

2. update Mgmt screens

3. offer to update any DNS we do for them

4. check to see if we had rules for old IP in firwall

Enabling tun device for a ve

Note, there’s a command for this: addtun

For posterity: Make sure the tun.o module is already loaded before Virtuozzo is started:

lsmod 

Allow the VPS to use the TUN/TAP device:

vzctl set 101 --devices c:10:200:rw --save 

Create the corresponding device inside the VPS and set the proper permissions:

vzctl exec 101 mkdir -p /dev/net 
vzctl exec 101 mknod /dev/net/tun c 10 200 
vzctl exec 101 chmod 600 /dev/net/tun

Remaking a system (on same virt)

1. cancelve (or v destroy x - ONLY if you're POSITIVE no data needs to be saved)

2. vemake using same veid

3. mvbackups or vb (if new mount point)

4. update mgmt with new dir/ip

5. update firewall if changed ip

Re-initialize quota for a VE

There’s a commamd for this now: clearquota

For posterity:

vzctl stop 1 vzquota drop 1 vzctl start 1

Traffic accounting on linux

DEPRECATED - all tracking is done via bwdb now. This is how we used to track traffic.

TODO: update for diff versions of vz

Unlike FreeBSD, where we have to add firewall count rules to the system to count the traffic, on virtuozzo counts the traffic for us. You an see the current traffic stats by running `vznetstat`:

# vznetstat
VEID    Net.Class  Output(bytes)   Input(bytes)
-----------------------------------------------
24218     1            484M             39M
24245     1            463M            143M
2451      1           2224M            265M
2454      1           2616M            385M
4149      1            125M             68M
418       1           1560M             34M
472       1           1219M            315M
726       1            628M            317M
763       1            223M             82M
771       1           4234M            437M
-----------------------------------------------
Total:               13780M           2090M

As you can see the VEID is on a line with the in and out bytes. So, we simply run a cron job:

4,9,14,19,24,29,34,39,44,49,55,59 * * * * /root/vztrafdump.sh

Just like we do on FreeBSD - this one goes through all the VEs in /vz/private and greps the line from vznetstat that matches them and dumps it in /jc_traffic_dump on their system. Then it does it again for all the VEs in /vz1/private. It is important to note that vznetstat runs only once, and the grepping is done from a temporary file that contains that output - we do this because running vznetstat once for each VE that we read out of /vz/private and /vz1/private would take way too long and be too intensive.

You do not need to do anything to facilitate this other than make sure that that cron job is running - the vznetstat counters are always running, and any new VEs that are added to the system will be accounted for automatically.

Traffic resetting no longer works with vz 2.6, so we disable the vztrafdump.sh on those virts.

Watchdog script

On some of the older virts, we have a watchdog running that kills procs that are deemed bad per the following:

/root/watchdog from quar1

if echo $line | awk '{print $(NF-3)}' | grep [5-9]...
  then
# 50-90%
      if echo $line | awk '{print $(NF-1)}' | grep "...:.."
      then
# running for > 99min
        echo $line >> /root/wd.log
        /usr/sbin/vzpid $pid | tail -1 >> /root/wd.log
        kill -9 $pid

      fi
      if echo $line | awk '{print $(NF-1)}' | grep "....m"
      then
# running for > 1000min
        echo $line >> /root/wd.log
        /usr/sbin/vzpid $pid | tail -1 >> /root/wd.log
        kill -9 $pid

      fi
  fi

  if echo $line | awk '{print $(NF-3)}' | grep [1-9]...
  then
# running for 10-90 percent
    if echo $line | awk '{print $NF}' | egrep 'cfusion|counter|vchkpw'
    then

      if echo $line | awk '{print $(NF-1)}' | grep "[2-9]:.."
      then
# between 2-9min
        echo $line >> /root/wd.log
        /usr/sbin/vzpid $pid | tail -1 >> /root/wd.log
        kill -9 $pid

      elif echo $line | awk '{print $(NF-1)}' | grep "[0-9][0-9]:.."
      then
# up to 99min
        echo $line >> /root/wd.log
        /usr/sbin/vzpid $pid | tail -1 >> /root/wd.log
        kill -9 $pid

      fi
    fi
  fi

Misc Linux Items

We are overselling hard drive space ... when you configure a linux system with a certain amount of disk space (the default is 4gigs) you do not actually use up 4gigs of space on the system. The diskspace setting for a user is simply a cap, and they only use up as much space on the actual disk drive as they are actually using.

When you create a new linux system, even though there are some 300 RPMs or so installed, if you run `df -k` you will see that the entire 4gig partition is empty - no space is being used. This is because the files in their system are "magic symlinks" to the template for their OS that is in /vz/template - however, any changes to any of those files will "disconnect" them and they will immediately begin using space in their system. Further, any new files uploaded (even if those new files overwrite existing files) will take up space on the partition.

Cant change file /etc/sysconfig/network

if you see this:

[root@virt8 root]# vzctl stop 160 ; vzctl start 160
VE is not running
Starting VE ...
VE is unmounted
VE is mounted
Adding IP address(es): 69.55.226.83 69.55.226.84
bash ERROR: Can't change file /etc/sysconfig/network
Deleting IP address(es): 69.55.226.83 69.55.226.84
VE is unmounted
[root@virt8 root]#

it probably means they no longer have /bin/bash - copy one in for them

ALSO: another possibility is that they have removed the `ed` RPM from their system - it needs to be reinstalled into their system. But since their system is down, this is tricky ...

VE startup scripts used by 'vzctl' want package 'ed' to be available inside VE. So if package 'ed' will be enabled in OS template config and OS template itself VE #827 is based on - this error should be fixed.

yes, it is possible to add RPM to VE while it not running. Try to do following:

# cd /vz/template/<OS_template_with_ed_package>/
# vzctl mount 827
# rpm -Uvh --root /vz/root/827 --veid 827 ed-0.2-25.i386.vz.rpm

Usually theres an error, but its ok

Note: replace 'ed-0.2-25.i386.vz.rpm' in last command with actual version of 'ed' package you have.

---

What template is user using

So how do I know what template the user has ? cat their conf file and it is listed in there. For example, if the conf file has:

[root@virt12 sbin]# vc 1103
…snip…
OSTEMPLATE="debian-3.0/20030822"
TEMPLATES="mod_perl-deb30/20030707 mod_ssl-deb30/20030703 mysql-deb30/20030707 proftpd-deb30/20030703 webmin-deb30/20030823 "
…

then they are on debian 3.0, all of their system RPMs are in /vz/template/debian-3.0, and they are using version 20030822 of that debian 3.0 template. Also, they’ve also got additional packages installed (mod_perl, mod_ssl, etc). Those are also found under /vz/template

---

Edits needed to run java

Edits needed to run java:

When we first created the VEs, the default setting for privvmpages was 93000:94000 ... which was high enough that most people never had problems ... however, you can;t run java or jdk or tomcat or anything java related with that setting. We have found that by setting privvmpages to 610000:615000 that java runs just fine. That is now the default setting. It is exceedingly rare that anyone needs it higher than that, although we have seen it once or twice.

Any problems with java at all - the first thing you need to do is see if the failcnt has raised for privvmpages.

# vzctl start 160
Starting VE ...
vzquota : (error) Quota on syscall for 160: Device or resource busy
Running vzquota on failed for VE 160 [3]

This is because my pwd is _in_ their private directory - you can't start it until you move out

People seem to have trouble with php if they are clueless newbies. Here are two common problems/solutions:

no... but i figured it out myself. problem was the php.ini file that came vanilla with the account was not configured to work with apache (the ENGINE directive was set to off).

everything else seems fine now.

---

the problem was in the php.ini file. I noticed that is wasnt showing the code when it was in an html file so I looked at the php.ini file and had to change it so it recognized <? tags aswell as <?php tags.

Also, make sure added to httpd.conf

   AddType application/x-httpd-php .php

---

Set time zone

You can set the time zone:

You can change the timezone by doing this:

ln -sf /usr/share/zoneinfo/<zone> /etc/localtime

where <zone> is the zone you want in the /usr/share/zoneinfo/ directory.

---

Failing shm_open calls

Failing shm_open calls:

first, please check if /dev/shm is mounted inside VE. 'cat /proc/mounts' command should show something like this:

tmpfs /dev/shm tmpfs rw 0 0

If /dev/shm is not mounted you have 2 ways to solve issue: 1. execute following command inside VE (doesn't require VE reboot):

mount -t tmpfs none /dev/shm

2. add following string to /etc/fstab inside VE and reboot it:

tmpfs         /dev/shm        tmpfs           defaults        0 0

---

You can have a mounted but not running ve Just:

vzctl mount <veid>

---

Debian can't get on the network

When a debian sys can’t get on the network, and you try:

vzctl set 1046 --ipadd 69.55.227.117
Adding IP address(es): 69.55.227.117
Failed to bring up lo.
Failed to bring up venet0.

They probably removed iproute package, which must be the one from swsoft. To restore:

# dpkg -i --veid=1046 --admindir=/vz1/private/1046/root/var/lib/dpkg --instdir=/vz1/private/1046/root/ /vz/template/debian-3.0/iproute_20010824-8_i386.vz.deb
(Reading database ... 16007 files and directories currently installed.)
Preparing to replace iproute 20010824-8 (using .../iproute_20010824-8_i386.vz.deb) ...
Unpacking replacement iproute ...
Setting up iproute (20010824-8) ...

Then restart their ve

CentOS can't get on the network

When CentOS system was running, but can't get on the network, they probably updated iproute via yum.

Copy an older version of the iproute rpm to their VPS.

cp -p /root/iproute/iproute-2.6.32-23.el6.<i686>.rpm /vz2/private/<1527>/fs/root

Enter their ve, install rpm, restart network, and prevent yum from updating iproute.

ve <1527>
rpm -i --force iproute-2.6.32-23.el6.<i686>.rpm
rpm -q iproute
rpm -e <new iproute rpm>
service network restart
echo "exclude=iproute" >>/etc/yum/ip-exclude
echo "include=/etc/yum/ip-exclude" >>/etc/yum.conf

---

in a ve i do:

cd /
du -h .

and get: 483M .

i do:

bash-2.05a# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/vzfs             4.0G  2.3G  1.7G  56% /

how can this be?

Is it possible that quota file was corrupted somehow? Please try to:

vzctl stop <VEID>
vzquota drop <VEID>
vzquota init <VEID>
vzctl start <VEID>

---

How to stop vz from starting after reboot:

VIRTUOZZO=no 

in

/etc/sysconfig/vz

To start:

service vz start

(after setting VIRTUOZZO=yes in /etc/sysconfig/vz)

service vz restart will do some kind of 'soft reboot' -- restart all VPSes and reload modules without rebooting the node

if you need to shut down all VPSes really really fast, run killall -9 init

---

Postfix tip:

You may want to tweak settings: default_process_limit=10

---

• uptime/load that vz users see isn’t what we see- it's their own load
• Vz overwrites resolve.conf on every boot to what’s in ve config file. Also overwrites hostname
• Quotas in vz: set Gid_limit in vz config file and restart ve. Errors leading up to this will be out of files but no inode or space issues. set to 2000 in conf, and restart vz

Virtuozzo VPS Management Tools

vm

To create a new Linux VPS

• get an IP address from mgmt screen
• run df -h to figure out which disk has enough space
• run vm with no arguments to get syntax and exact template name
• use col0xxxx for hostname if they don’t give you a hostname
• copy over dir, veid, ip and password to pending customer screen

vm CID IP hostname /vz[1|2] email[,email] template ( <LB|LBP|LS|LM|LMP|LP> [disk] | <gb_disk/mb_ram/gb_burst> )

disable

To disable a customers Virtuozzo VPS server.

• stop the server

 vzctl stop <veid>

• prevent restart on server reboot and user restart via Control Panel

 vzctl set <veid> --disabled=yes --offline_management=no --save

cancelve

When a customer cancels all service.

cancelve <veid>

this will:

• stop a ve
• check for backups (offer to remove them from the backup server

and the backup.config)

• rename the private dir
• check for PTR, provide the commands to reset to default
• and rename the ve’s config
• remind you to remove firewall rules
• remind you to remove DNS entries

ipadd

ipadd  <veid> <ip> [ip] [ip]

add’s ip(s) to a ve

ipdel

ipdel <veid> <ip> [ip] [ip]

removes ip(s) from a ve

vc

vc <veid>

display the contents of the vz config file for the <veid>

the equivalent of: cat /vzconf/<veid>.conf

vl

vl

List all virtual environments running on this machine.

will displays a list of ve #’s, 1 per line. (ostensibly to use in a for loop)

vp

vp <veid>

show processes running under this <veid>.

the equivalent of: vzps auxww –E <veid>

vpe

DEPRECATED

vpe <veid> 

this will allow you to do a vp when a ve is running out of control, the equivalent of (deprecated since vp operates outside the VPS):

vzctl set <veid> --kmemsize 2100000:2200000
vzctl exec <veid> ps auxw
vzctl set <veid> --kmemsize (ve’s orig lvalue):(ve’s orig hvalue)

vt

vt <veid>

the equivalent of: vztop –E <veid>

vr

vr <veid>

Restart a <veid>.

the equivalent of: vzctl stop <veid>; vzctl start <veid> You can run this even if the ve is down - the stop command will just fail

vs

vs [veid]

displays status (output of vzctl status <veid>) on each ve configured on the system (as determined by /vzconf/*.conf) If passed an argument, gives the status for just that ve. A running system looks like: VEID 16066 exist mounted running

A ve which is not running (but does exist) looks like: VEID 9990 exist unmounted down

A ve which is not running and doesn’t exist looks like: VEID 421 deleted unmounted down

vs2

vs2 [veid]

this is similar to vs in that it displays status (output of vzctl status <veid>) on each ve, but the difference is it’s list comes from doing an ls on the data dirs. This was meant to catch the rare case where a ve configured but exists.

vw

vw [veid]

displays the output of ‘w’ (the equivalent of vzctl exec <veid> w) for each configured ve (as determined by /vzconf/*.conf). Useful for determing which ve is contributing to a heavily-loaded system. If passed an argument, gives ‘w‘ output for just that ve. Ex:

[root@virt2 etc]# vw
134
 10:52pm  up 79 days,  6:14,  0 users,  load average: 0.02, 0.02, 0.00
USER     TTY      FROM              LOGIN@   IDLE   JCPU   PCPU  WHAT
16027
  2:52pm  up 7 days, 19:54,  0 users,  load average: 0.00, 0.00, 0.00
USER     TTY      FROM              LOGIN@   IDLE   JCPU   PCPU  WHAT
16055
  2:52pm  up 79 days,  6:38,  0 users,  load average: 0.00, 0.04, 0.07
USER     TTY      FROM              LOGIN@   IDLE   JCPU   PCPU  WHAT

vwe

vwe [constraint]

just like vw, but takes a constraint as an argument, only show’s ve’s with loads >= the constraint provided. If no constraint is provided, 1 is used by default

vzs

vzs [veid]

displays the beancounter status for all ve’s, or a particular ve if an argument is passed

ve

ve <veid>

enter a <veid> as root.

the equivalent of: vzctl enter <veid>

vx

vx <veid> <command>

execute a single command in the <veid> environment.

the equivalent of: /usr/sbin/vzctl exec <veid> <command>

dprocs

dprocs [count]

a script which outputs a continuous report (or a certain number of reports if an option is passed) of processes stuck in the D state and which VPS’s those procs belong to.

setmem

setmem VEID <256|512|768|1024|1280|1536|2048> [burst GB]

adjusts the memory resources for the VE. If this is on VZ ver >4, you will also be prompted to provide a burst ram amount. Sample usage:

# setmem 21671 1024 4
/usr/sbin/vzctl set 21671 --ram 1024m --swap 3072m --save

afacheck.sh

afacheck.sh

displays the health/status of containers and mirrors on an adaptec card (currently quar1, tempvirt1-2, virt9, virt10)- all other are LSI

backup

backup

backup script called nightly to update virt scripts and do customer backups

checkload.pl

checkload.pl

this was intended to be setup as a cronjob to watch processes on a virt when the load rises above a certain level. Not currently in use.

findbackuppigs.pl

findbackuppigs.pl

looks for files larger than 50MB which customers have asked us to backup. Emails matches to linux@johncompanies.com

gatherlinux.pl

gatherlinux.pl

gathers up data about ve’s configured and writes to a file. Used for audits against the db

gb

gb <search>

greps backup.config for the given search parameter

gbg

gbg <search>

greps backup.config for the given search parameter and presents just the directories (for clean pasting)

linuxtrafficgather.pl

linuxtrafficgather.pl [yy-mm]

sends a traffic usage summary by ve to support@johncomapnies.com and payments@johncompanies.com. Optional arguments are year and month (must be in the past). If not passed, assumes last month. Relies on traffic logs created by netstatreset and netstatbackup

linuxtrafficwatch.pl

linuxtrafficwatch.pl

checks traffic usage and emails support@johncomapnies.com when a ve reaches the warning level (35G) and the limit (40G). Works on virtuozzo versions <= 2.6.x. We really aren’t using this anymore now that we have netflow.

linuxtrafficwatch2.pl

linuxtrafficwatch2.pl

checks traffic usage and emails support@johncomapnies.com when a ve reaches the warning level (35G) and the limit (40G). Works on virtuozzo version 2.6.x. We really aren’t using this anymore now that we have netflow.

load.pl

load.pl

feeds info to load mrtg - executed by inetd.

mb (linux)

mb <mount|umount>

(nfs) mounts and umounts dirs to backup2. Shortcuts are mbm and mbu to mount and unmount.

migrate

migrate <ip of node migrating to> <veid> <target_veid> [target dir: vz | vz1 | vz2]

this is basically a wrapper for vzmigrate – vzmigrate is a util to seamlessly move a ve from one host to another. This wrapper was written cause vz virtuozzo version 2.6 had a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables. This script mitigates that. Since it makes multiple ssh connections to the target host, it’s a good idea to put the pub key for the src system in the authorized_keys file on the target host. In addition, it emails ve owners when their migration starts and stops (if they place email addresses in a file on their system: /migrate_notify). To move everyone off a system, you’d do:

for f in `vl`; do migrate <ip> $f; done

migrateonline

migrateonline <ip of node migrating to> <veid> <target_veid> [target dir: vz | vz1 | vz2]

this is the same as migrate but will migrate a ve in –online mode which means it won’t be shut down at the end of the migration. This only works when migrating ve’s between 2 machines running a 2.6 kernel (currently tempvirt1-2. virt16-19, virt12). If you get an error that the machine you’re trying to migrate to has a different CPU or features, etc, then you have to edit the file and add the –f switch to the vzmigrate line- you can basically ignore this kind of warning (but never ignore a warning about missing templates on the destination node). NOTE: This edit (if made to migrateonline) will be overwritten by the base script during each night’s backup.

netstatbackup

DEPRECATED

netstatbackup 

writes traffic count data to a logfile. Works on virtuozzo versions 2.5.x.

netstatbackup2

DEPRECATED

netstatbackup2

writes traffic count data to a logfile. Works on virtuozzo versions 2.6.x.

netstatreset

DEPRECATED

netstatreset

writes traffic count data to a logfile and resets counters to 0. Works on virtuozzo versions 2.5.x

netstatreset2

DEPRECATED

netstatreset2

writes traffic count data to a logfile. Works on virtuozzo versions 2.6.x.

orphanedbackupwatchlinux

orphanedbackupwatchlinux 

looks for directories on backup2 which aren’t configured in backup.config and offers to delete them

rsync.backup (linux)

rsync.backup

does customer backups (relies on backup.config)

startvirt.pl

startvirt.pl

forks off start ve commands – keeps 6 running at a time. This is not to be used on systems where fastboot is enabled as it circumvents the benefit of the fastboot. The script will occasionally not exit gracefully and will continue to use up CPU, so it should be watched. Also, don’t exit from the script till you’re sure all ve’s are started – if you do you need to start them manually and may have to free up locks. On some systems, the startvirt script doesn’t exit cleanly and you have to ^C out of it. Be careful though- doing so can leave some VE’s in an odd bootup state and you may need to ‘vr’ them manually. You should check to see which ve’s aren’t running and/or confirm all have started when ^C’ing out of startvirt.

taskdone (linux)

taskdone

when called will email support@johncompanies.com with the hostname of the machine from which it was executed as the subject

vb (linux)

vb

the equivalent of: vi /usr/local/sbin/backup.config

vemakeXX

DEPRECATED

vemakerh9 

ve create script for RH9 (see vemake)

vemakedebian30 

ve create script for debian 3.0 (Woody) (see vemake)

vemakedebian31 

ve create script for debian 3.1 (Sarge) (see vemake)

vemakedebian40 

ve create script for debian 4.0 (Etch) (see vemake)

vemakefedora, vemakefedora2, vemakefedora4, vemakefedora5, vemakefedora6, vemakefedora7

ve create script for fedora core 1, 2, 4, 5, 6, 7 respectively (see vemake)

vemakecentos3, vemakecentos4

ve create script for centos 3, 4 respectively (see vemake)

vemakesuse, vemakesuse93, vemakesuse100

ve create script for suse 9.2, 9.3, 10.0 respectively (see vemake)

vemakeubuntu5, vemakeubuntu606, vemakeubuntu606 vemakeubuntu704

ve create script for ubuntu 5.10, 6.06, 6.10, 7.04 respectively (see vemake)

vemove

DEPRECATED

vemove <veid> <target_ip> </vz/private/123>

this script simplifies the old way of moving ve’s from one system to another - in short moving a ve to or from a virt running virtuozzo < 2.6.x It’s the equivalent of: tar cfpP - <veid> --ignore-failed-read | (ssh -2 -c arcfour <target_ip> "split - -b 1024m </vz/private/123>.tar" )This should only be used if migrate/vzmigrate can’t be used.

vim.watchdog

vim.watchdog 

looks for and kills procs matching “vi|vim|nano|pine|elm” that are running for a long time & consuming lots of cpu. Works on virtuozzo versions 2.5.x

vim.watchdog2

vim.watchdog2

looks for and kills procs matching “vi|vim|nano|pine|elm” that are running for a long time & consuming lots of cpu. Works on virtuozzo versions 2.6.x.

vzmigrate

vzmigrate <target_ip> -r no <veid>:[dst veid]:[dst /vzX/private/veid]:[dst /vzX/root/veid]

(this is the raw command “wrapped” by migrate/migrateonline) this will seamlessly move a ve from one host to another. The ve will run for the duration of the migration till the very end when it’s shut down, ip moved and started up on the target system. The filesystem on the src will remain. This should be watched – occasionally the move will timeout and leave the system shut down. If target private and root aren’t specified it just puts it in /vz. Only works when both systems are running virtuozzo 2.6.x

vztrafdump.sh

DEPRECATED

vztrafdump.sh

writes traffic usage info by ve to a file called jc_traffic_dump in each ve’s / dir. Works on virtuozzo versions <= 2.5.x.

vztrafdump2.sh

DEPRECATED

vztrafdump2.sh

writes traffic usage info by ve to a file called jc_traffic_dump in each ve’s / dir. Works on virtuozzo versions 2.6.x.

addtun

addtun <veid>

Add’s tun device to ve.

bwcap

bwcap <veid> <kbps>

Ex: bwcap 1234 512 Caps a VE’s bandwidth to the amount given

setdisk

setdisk <veid> <diskspace in GB>

Ex: setdisk 1234 5 Gives a VE’s a given amount of disk space

vdf

vdf <veid> 

the equivalent of: vzctl exec <veid> df –h

vdff

vdff

runs a (condensed) vdf for all ve’s in your pwd (must be run from /vz/privateN)

mvbackups

mvbackups <veid> <target_machine> (virt1) <target_dir> (vz1)

moves backups from one location to another on the backup server, and provides you with option to remove entries from current backup.config, and simple paste command to add the config to backup.config on the target server

checkquota

checkquota

for all the ve’s in the cwd (run from /vz/private, /vz1/private, etc) reports what vz quota says they’re using and what the actual usage is (as reported by du)

clearquota

clearquota <veid>

Recalculates a ve’s quota, prints out the usage before and after. The equivalent of: vdf <veid>; v stop <veid>; vzquota drop <veid>; v start <veid>; vdf <veid>

dprocs

dprocs

Sometimes the server’s have a large number of processes get stuck in the D state- this script shows (every 3 secs) which VE’s have D procs, which procs are stuck and a running average of the top “offenders”

vzstat

vstat

sort of like top for VZ. sort VEs by CPU usage by pressing 'o' and then 'c' keys

stopvirt

stopvirt

will stop VEs as fast as it can, 6 at a time. May not exit when complete so you should watch vzstat in another window.