Editing VPS Management (section)

= Virtuozzo VPS =

== Making new customer VE (vm) ==

This applies only to new virts >= 4.x

grab ip from ipmap (if opened from the pending cust screen it should take you to the right block). You can also run vzlist -a to see what block is in use, generally. Try to find an IP that's in the same block of class C IP's already on the box.

1. confirm ip you want to use isn’t in use via Mgmt. -> IP Map in management screens
  also, ping the IP address.

2. put CT on whichever partition has more space
  df -h

3.  vm CID IP hostname /vz[1|2] email[,email] template ( <LB|LBP|LS|LM|LMP|LP> [disk] | <gb_disk/mb_ram/gb_burst> ) 
 vm col00009 69.55.230.238 centos.testdave.com /vz1 dsmith@johncompanies.com centos-6-x86_64 LM

4. copy veid, dir, ip and password to pending customer screen. activate customer

== Making new customer VE (vemakexxx) ==

This applies to older virts with old templates. This should probably not be used at all anymore.


1. look thru hist for ip

2. confirm ip you want to use isn’t in use via Mgmt. -> IP Map in management screens

3. put ve on whichever partition has more space
 vemakerh9 <veid> <ip> <hostname> <mount> <email> [gb disk]; <256|384|512> <veid>
 vemakerh9 866 69.55.226.109 ngentu.com /vz1 ayo@ngantu.com,asd@asd.com 5; 256 866

4. copy (veid), dir, and ip to pending customer screen (pass set to p455agfa)



Note: We use VEID (Virtual Environment ID) and CTID (Container ID) interchangably. Similarly, VE and CT. They mean the same thing.
VZPP = VirtuoZzo Power Panel (the control panel for each CT)

All linux systems exist in /vz, /vz1 or /vz2 - since each linux machine holds roughly 60-90 customers, there will be roughly 30-45 in each partition.

The actual filesystem of the system in question is in:

 /vz(1-2)/private/(VEID)

Where VEID is the identifier for that system - an all-numeric string larger than 100.

The actual mounted and running systems are in the corresponding:

 /vz(1-2)/root/(VEID)

But we rarely interact with any system from this mount point.

You should never need to touch the root portion of their system – however you can traverse their filesystem by going to <tt>/vz(1-2)/private/(VEID)/root</tt> (<tt>/vz(1-2)/private/(VEID)/fs/root</tt> on 4.x systems) the root of their filesystem is in that directory, and their entire system is underneath that.

Every VE has a startup script in <tt>/etc/sysconfig/vz-scripts</tt>  (which is symlinked as <tt>/vzconf</tt> on all systems) - the VE startup script is simply named <tt>(VEID).conf</tt> - it contains all the system parameters for that VE:

<pre># Configuration file generated by vzsplit for 60 VE
# on HN with total amount of physical mem 2011 Mb

VERSION="2"
CLASSID="2"

ONBOOT="yes"

KMEMSIZE="8100000:8200000"
LOCKEDPAGES="322:322"
PRIVVMPAGES="610000:615000"
SHMPAGES="33000:34500"
NUMPROC="410:415"
PHYSPAGES="0:2147483647"
VMGUARPAGES="13019:2147483647"
OOMGUARPAGES="13019:2147483647"
NUMTCPSOCK="1210:1215"
NUMFLOCK="107:117"
NUMPTY="19:19"
NUMSIGINFO="274:274"
TCPSNDBUF="1800000:1900000"
TCPRCVBUF="1800000:1900000"
OTHERSOCKBUF="900000:950000"
DGRAMRCVBUF="200000:200000"
NUMOTHERSOCK="650:660"
DCACHE="786432:818029"
NUMFILE="7500:7600"
AVNUMPROC="51:51"
IPTENTRIES="155:155"
DISKSPACE="4194304:4613734"
DISKINODES="400000:420000"
CPUUNITS="1412"
QUOTAUGIDLIMIT="2000"
VE_ROOT="/vz1/root/636"
VE_PRIVATE="/vz1/private/636"
NAMESERVER="69.55.225.225 69.55.230.3"
OSTEMPLATE="vzredhat-7.3/20030305"
VE_TYPE="regular"
IP_ADDRESS="69.55.225.229"
HOSTNAME="textengine.net"</pre>


As you can see, the hostname is set here, the disk space is set here, the number of inodes, the number of files that can be open, the number of tcp sockets, etc. - all are set here.

In fact, everything that can be set on this customer system is set in this conf file.


All interaction with the customer system is done with the VEID.  You start the system by running:

 vzctl start 999

You stop it by running:

 vzctl stop 999

You execute commands in it by running:

 vzctl exec 999 df -k

You enter into it, via a root-shell backdoor with:

 vzctl enter 999

and you set parameters for the system, while it is still running, with:

 vzctl set 999 --diskspace 6100000:6200000 --save

<tt>vzctl</tt> is the most commonly used command - we have aliased <tt>v</tt> to <tt>vzctl</tt> since we use it so often. We’ll continue to use <tt>vzctl</tt> in our examples, but feel free to use just <tt>v</tt>.

Let's say the user wants more diskspace.  You can cat their conf file and see:

 DISKSPACE="4194304:4613734"

So right now they have 4gigs of space.  You can then change it to 6 with:

 vzctl set 999 --diskspace 6100000:6200000 --save

IMPORTANT:  all issuances of the vzctl set command need to end with <tt>–save</tt> - if they don't, the setting will be set, but it will not be saved to the conf file, and they will not have those settings next time they boot.

All of the tunables in the conf file can be set with the vzctl set command.  Note that in the conf file, and on the vzctl set command line, we always issue two numbers seperated by a colon - that is because we are setting the hard and soft limits.  Always set the hard limit slightly above the soft limit, as you see it is in the conf file for all those settings.

There are also things you can set with `<tt>vzctl set</tt>` that are not in the conf file as settings, per se.  For instance, you can add IPs:

 vzctl set 999 --ipadd 10.10.10.10 --save

or multiple IPs:

 vzctl set 999 --ipadd 10.10.10.10 --ipadd 10.10.20.30 --save

or change the hostname:

 vzctl set 999 --hostname www.example.com --save

You can even set the nameservers:

 vzctl set 999 --nameserver 198.78.66.4 --nameserver 198.78.70.180 --save

Although you probably will never do that.

You can disable a VPS from being started (by VZPP or reboot) (4.x):

 vzctl set 999 --disabled yes --save 

You can disable a VPS from being started (by VZPP or reboot) (<=3.x):

 vzctl set 999 --onboot=no --save 

You can disable a VPS from using his control panel:

 vzctl set 999 --offline_management=no --save 

You can suspend a VPS, so it can be resumed in the same state it was in when it was stopped (4.x):

 vzctl suspend 999

and to resume it:

 vzctl resume 999

to see who owns process:
 vzpid <PID>

to mount up an unmounted ve:
 vzctl mount 827


To see network stats for CT's:
<pre># vznetstat
VEID    Net.Class  Output(bytes)   Input(bytes)
-----------------------------------------------
24218     1            484M             39M
24245     1            463M            143M
2451      1           2224M            265M
2454      1           2616M            385M
4149      1            125M             68M
418       1           1560M             34M
472       1           1219M            315M
726       1            628M            317M
763       1            223M             82M
771       1           4234M            437M
-----------------------------------------------
Total:               13780M           2090M</pre>


One thing that sometimes comes up on older systems that we created with smaller defaults is that the system would run out of inodes.  The user will email and say they cannot create any more files or grow any files larger, but they will also say that they are not out of diskspace ... they are running:

 df -k

and seeing how much space is free - and they are not out of space.  They are most likely out of inodes - which they would see by running:

 df -i

So, the first thing you should do is enter their system with:

 vzctl enter 999

and run:  <tt>df -i</tt>

to confirm your theory.  Then exit their system.  Then simply cat their conf file and see what their inodes are set to (probably 200000:200000, since that was the old default on the older systems) and run:

 vzctl set 999 --diskinodes 400000:400000 --save

If they are not out of inodes, then a good possibility is that they have maxed out their numfile configuration variable, which controls how many files they can have in their system.  The current default is 7500 (which nobody has ever hit), but the old default was as low as 2000, so you would run something like:

 vzctl set 999 --numfile 7500:7500 --save

----

You cannot start or stop a VE if your pwd is its private (/vz/private/999) or root (/vz/root/999) directories, or anywhere below them.

== Recovering from a crash (linux) ==

=== Diagnose whether you have a crash ===
The most important thing is to get the machine and all ve’s back up as soon as possible. Note the time, you’ll need to create a crash log entry (Mgmt. -> Reference -> CrashLog). The first thing to do is head over to the [[Screen#Screen_Organization|serial console screen]] and see if there’s any kernel error messages output. Try to copy any messages (or just a sample of repeating messages) you see into the notes section of the crash log – these will also likely need to be sent to virtuozzo for interpretation. If the messages are spewing too fast, hit ^O + H to start a screen log dump which you can ob1182.pts-38.bb serve after the machine is rebooted. Additionally, if the  machine is responsive, you can get a trace to send to virtuozzo by hooking up a kvm and entering these 3 sequences:
<pre>alt+print screen+m
alt+print screen+p
alt+print screen+t</pre>

If there are no messages, the machine may just be really busy- wait a bit (5-10min) to see if it comes back. If it's still pinging, odds are its very busy. If it doesn't come back, or the messages indicate a fatal error, you will need to proceed with a power cycle (ctrl+alt+del will not work).

=== Power cycle the server ===
If this machine is not a Dell 2950 with a [[DRAC/RMM#DRAC|DRAC card]] (i.e. if you can’t ssh into the DRAC card and issue racadm serveraction hardreset, then you will need someone at the data center to power the macine off, wait 30 sec, then turn it back on.  Make sure to re-attach via console (<tt>tip virtxx</tt>) immediately after power down. 

=== (Re)attach to the console ===
Stay on the console the entire time during boot. As the BIOS posts- look out for the RAID card output- does everything look healthy? The output may be scrambled, look for "DEGRADED" or "FAILED". Once the OS starts booting you will be disconnected (dropped back to the shell on the console server) a couple times during the boot up. The reason you want to quickly re-attach is two-fold: 1. If you don’t reattach quickly then you won’t get any console output, 2. you want to be attached before the server ''potentially'' starts (an extensive) fsck. If you attach after the fsck begins, you’ll have seen no indication it started an fsck and the server will appear frozen during startup- no output, no response. 

=== Start containers/VE's/VPSs ===
When the machine begins to start VE’s, it’s safe to leave the console and login via ssh. All virts should be set to auto start all the VEs after a crash. Further, most (newer) virts are set to “fastboot” it’s VE’s (to find out, do:
 grep -i fast /etc/sysconfig/vz 
and look for <tt>VZFASTBOOT=yes</tt>). If this was set prior to the machine’s crash (setting it after the machine boots will not have any effect until the vz service is restarted) it will start each ve as fast as possible, in serial, then go thru each VE (serially), shutting it down running a vzquota (disk usage) check, then bringing it back up. The benefit is that all VE’s are brought up quickly (within 15min or so depending on the #), the downside is a customer watching closely will notice 2 outages – 1st the machine crash, 2nd their quota check (which will be a much shorter downtime- on the order of a few minutes). 

Where “fastboot” is not set to yes (i.e on quar1), vz will start them consecutively, checking the quotas one at a time, and the 60th VE may not start until an hour or two later - this is not acceptable.

The good news is, if you run vzctl start for a VE that is already started, you will simply get an error: <tt>VE is already started</tt>.  Further, if you attempt to vzctl start a VE that is in the process of being started, you will simply get an error: unable to lock VE.  So, there is no danger in simply running scripts to start smaller sets of VEs.  If the system is not autostarting, then there is no issue, and even if it does, when it conflicts, one process (yours or the autostart) will lose, and just move on to the next one.

A script has been written to assist with ve starts: [[#startvirt.pl|startvirt.pl]] which will start 6 ve’s at once until there are no more left.  If startvirt.pl  is used on a system where “fastboot” was on,  it will circumvent the fastboot for ve’s started by startvirt.pl – they will go through the complete quota check before starting- therefore this is not advisable when a system has crashed. When a system is booted cleanly, and there's no need for vzquota checks, then startvirt.pl is safe and advisable to run.

=== Make sure all containers are running ===
You can quickly get a feel for how many ve’s are started by running:

<pre>[root@virt4 log]# vs
VEID 16066 exist mounted running
VEID 16067 exist mounted running
VEID 4102 exist mounted running
VEID 4112 exist mounted running
VEID 4116 exist mounted running
VEID 4122 exist mounted running
VEID 4123 exist mounted running
VEID 4124 exist mounted running
VEID 4132 exist mounted running
VEID 4148 exist mounted running
VEID 4151 exist mounted running
VEID 4155 exist mounted running
VEID 42 exist mounted running
VEID 432 exist mounted running
VEID 434 exist mounted running
VEID 442 exist mounted running
VEID 450 exist mounted running
VEID 452 exist mounted running
VEID 453 exist mounted running
VEID 454 exist mounted running
VEID 462 exist mounted running
VEID 463 exist mounted running
VEID 464 exist mounted running
VEID 465 exist mounted running
VEID 477 exist mounted running
VEID 484 exist mounted running
VEID 486 exist mounted running
VEID 490 exist mounted running</pre>

So to see how many ve’s have started:
<pre>[root@virt11 root]# vs | grep running | wc -l
     39</pre>

And to see how many haven’t:
<pre>[root@virt11 root]# vs | grep down | wc -l
     0</pre>

And how many we should have running:
<pre>[root@virt11 root]# vs | wc -l
     39</pre>

Another tool you can use to see which ve’s have started, among other things is [[#vzstat|vzstat]]. It will give you CPU, memory, and other  stats on each ve and the overall system. It’s a good thing to watch as ve’s are starting (note the VENum parameter, it will tell you how many have started):

  <pre>4:37pm, up 3 days,  5:31,  1 user, load average: 1.57, 1.68, 1.79
VENum 40, procs 1705: running 2, sleeping 1694, unint 0, zombie 9, stopped 0
CPU [ OK ]: VEs  57%, VE0   0%, user   8%, sys   7%, idle  85%, lat(ms) 412/2
Mem [ OK ]: total 6057MB, free 9MB/54MB (low/high), lat(ms) 0/0
Swap [ OK ]: tot 6142MB, free 4953MB, in 0.000MB/s, out 0.000MB/s
Net [ OK ]: tot: in  0.043MB/s  402pkt/s, out  0.382MB/s 4116pkt/s
Disks [ OK ]: in 0.002MB/s, out 0.000MB/s

  VEID ST    %VM     %KM         PROC    CPU     SOCK FCNT MLAT IP
     1 OK 1.0/17  0.0/0.4    0/32/256 0.0/0.5 39/1256    0    9 69.55.227.152
    21 OK 1.3/39  0.1/0.2    0/46/410 0.2/2.8 23/1860    0    6 69.55.239.60
   133 OK 3.1/39  0.1/0.3    1/34/410 6.3/2.8 98/1860    0    0 69.55.227.147
   263 OK 2.3/39  0.1/0.2    0/56/410 0.3/2.8 34/1860    0    1 69.55.237.74
   456 OK  17/39  0.1/0.2   0/100/410 0.1/2.8 48/1860    0   11 69.55.236.65
   476 OK 0.6/39  0.0/0.2    0/33/410 0.1/2.8 96/1860    0   10 69.55.227.151
   524 OK 1.8/39  0.1/0.2    0/33/410 0.0/2.8 28/1860    0    0 69.55.227.153
   594 OK 3.1/39  0.1/0.2    0/45/410 0.0/2.8 87/1860    0    1 69.55.239.40
   670 OK 7.7/39  0.2/0.3    0/98/410 0.0/2.8 64/1860    0  216 69.55.225.136
   691 OK 2.0/39  0.1/0.2    0/31/410 0.0/0.7 25/1860    0    1 69.55.234.96
   744 OK 0.1/17  0.0/0.5    0/10/410 0.0/0.7  7/1860    0    6 69.55.224.253
   755 OK 1.1/39  0.0/0.2    0/27/410 0.0/2.8 33/1860    0    0 192.168.1.4
   835 OK 1.1/39  0.0/0.2    0/19/410 0.0/2.8  5/1860    0    0 69.55.227.134
   856 OK 0.3/39  0.0/0.2    0/13/410 0.0/2.8 16/1860    0    0 69.55.227.137
   936 OK 3.2/52  0.2/0.4    0/75/410 0.2/0.7 69/1910    0    8 69.55.224.181
  1020 OK 3.9/39  0.1/0.2    0/60/410 0.1/0.7 55/1860    0    8 69.55.227.52
  1027 OK 0.3/39  0.0/0.2    0/14/410 0.0/2.8 17/1860    0    0 69.55.227.83
  1029 OK 1.9/39  0.1/0.2    0/48/410 0.2/2.8 25/1860    0    5 69.55.227.85
  1032 OK  12/39  0.1/0.4    0/80/410 0.0/2.8 41/1860    0    8 69.55.227.90</pre>

When you are all done, you will want to make sure that all the VEs really did get started, run vs one more time.

Note the time all ve’s are back up and enter that into and save the crash log entry.

Occasionally, a ve will not start automatically. The most common reason for a ve not to come up normally is the ve was at it’s disk limit before the crash, and will not start since they’re over the limit. To overcome this, set the disk space to current usage level (the system will give this to you when it fails to start), start the ve, then re-set the disk space back to the prior level. Lastly, contact the customer to let them know they’re out of disk (or allocate more disk if they're entitled to more).

== Hitting performance barriers and fixing them ==

There are multiple modes virtuozzo offers to allocate resources to a ve. We utilize 2: SLM and UBC parameters
On our 4.x systems, we use all SLM – it’s simpler to manage and understand. There are a few systems on virt19/18 that may also use SLM. Everything else uses UBC. 
You can tell a SLM ve by:

 SLMMODE="all"

in their conf file. 

TODO: detail SLM modes and parameters.

If someone is in SLM mode and they hit memory resource limits, they simply need to upgrade to more memory.

The following applies to everyone else (UBC).

Customers will often email and say that they are getting out of memory errors - a common one is "cannot fork" ... basically, anytime you see something odd like this, it means they are hitting one of their limits that is in place in their conf file.

The conf file, however, simply shows their limits - how do we know what they are currently at ?

The answer is a file called v - this file contains the current status (and peaks) of their  performance settings, and also counts how many times they have hit the barrier.  The output of the file looks like this:

<pre>764: kmemsize         384113     898185    8100000    8200000          0
     lockedpages           0          0        322        322          0
     privvmpages        1292       7108     610000     615000          0
     shmpages            270        528      33000      34500          0
     dummy                 0          0          0          0          0
     numproc               8         23        410        415          0
     physpages            48       5624          0 2147483647          0
     vmguarpages           0          0      13019 2147483647          0
     oomguarpages        641       6389      13019 2147483647          0
     numtcpsock            3         21       1210       1215          0
     numflock              1          3        107        117          0
     numpty                0          2         19         19          0
     numsiginfo            0          4        274        274          0
     tcpsndbuf             0      80928    1800000    1900000          0 
     tcprcvbuf             0     108976    1800000    1900000          0
     othersockbuf       2224      37568     900000     950000          0
     dgramrcvbuf           0       4272     200000     200000          0
     numothersock          3          9        650        660          0
     dcachesize        53922     100320     786432     818029          0
     numfile             161        382       7500       7600          0
     dummy                 0          0          0          0          0
     dummy                 0          0          0          0          0
     dummy                 0          0          0          0          0
     numiptent             4          4        155        155          0</pre>

The first column is the name of the counter in question - the same names we saw in the systems conf file.  The second column is the _current_ value of that counter, the third column is the max that that counter has ever risen to, the fourth column is the soft limit, and the fifth column is the hard limit (which is the same as the numbers in that systems conf file).

The sixth number is the failcount - how many times the current usage has risen to hit the barrier.  It will increase as soon as the current usage hits the soft limit.

The problem with /proc/user_beancounters is that it actually contains that set of data for every running VE - so you can't just cat /proc/user_beancounters - it is too long and you get info for every other running system.

You can vzctl enter the system and run:

 vzctl enter 9999
 cat /proc/user_beancounters

inside their system, and you will just see the stats for their particular system, but entering their system every time you want to see it is combersome.

So, I wrote a simple script called "vzs" which simply greps for the VEID, and spits out the next 20 or so lines (however many lines there are in the output, I forget) after it.  For instance:

<pre># vzs 765:
765: kmemsize        2007936    2562780    8100000    8200000          0
     lockedpages           0          8        322        322          0
     privvmpages       26925      71126     610000     615000          0
     shmpages          16654      16750      33000      34500          0
     dummy                 0          0          0          0          0
     numproc              41         57        410        415          0
     physpages          1794      49160          0 2147483647          0
     vmguarpages           0          0      13019 2147483647          0
     oomguarpages       4780      51270      13019 2147483647          0
     numtcpsock           23         37       1210       1215          0
     numflock             17         39        107        117          0
     numpty                1          3         19         19          0
     numsiginfo            0          6        274        274          0
     tcpsndbuf         22240     333600    1800000    1900000          0
     tcprcvbuf             0     222656    1800000    1900000          0
     othersockbuf     104528     414944     900000     950000          0
     dgramrcvbuf           0       4448     200000     200000          0
     numothersock         73        105        650        660          0
     dcachesize       247038     309111     786432     818029          0
     numfile             904       1231       7500       7600          0
     dummy                 0          0          0          0          0
     dummy                 0          0          0          0          0
     dummy                 0          0          0          0          0
     numiptent             4          4        155        155          0</pre>

That showed us just the portion of /proc/user_beancounters for system 765.

When you run the vzs command, always add a : after the VEID.

So, if a customer complains about some out of memory errors, or no more files, or no more ptys, or just has an unspecific complain about processes dying, etc., the very first thing you need to do is check their beancounters with vzs.  Usually you will spot an item that has a high failcount and needs to be upped.

At that point you could simply up the counter with `vzctl set`.  Generally pick a number 10-20% higher than the old one, and make the hard limit slightly larger than the the soft limit. However our systems now come in several levels and those levels have more/different memory allocations. If someone is complaining about something other than a memory limit (pty, numiptent, numflock), it’s generally safe to increase it, at least to the same level as what’s in the /vzconf/4unlimited file on the newest virt. If someone is hitting a memory limit, first make sure they are given what they deserve:

(refer to mgmt -> payments -> packages)

To set those levels, you use the [[#setmem|setmem]] command. 

The alternate (DEPRECATED) method would be to use one of 3 commands:
256 <veid>
300 <veid>
384 <veid>
512 <veid>

If the levels were not right (you’d run vzs <veid> before and after to see the effect) tell the customer they’ve been adjusted and be done with it. If the levels were right, tell the customer they must upgrade to a higher package, tell them how to see level (control panel) and that they can reboot their system to escape this lockup contidion.

Customers can also complain that their site is totally unreachable, or complain that it is down ... if the underlying machine is up, and all seems well, you may notice in the beancounters that network-specific counters are failing - such as numtcpsock, tcpsndbuf or tcprcvbuf.  This will keep them from talking on the network and make it seem like their system is down.  Again, just up the limits and things should be fine.

On virts 1-4, you should first look at the default settings for that item on a later virt, such as virt 8 - we have increased the defaults a lot since the early machines.  So, if you are going to up a counter on virt2, instead of upping it by 10-20%, instead up it to the new default that you see on virt8.

== Moving a VE to another virt (migrate/migrateonline) ==

This will take a while to complete - and it is best to do this at night when the load is light on both machines.

There are different methods for this, depending on which version of virtuozzo is installed on the src. and dst. virt. 
To check which version is running: 
 [root@virt12 private]# cat /etc/virtuozzo-release
 Virtuozzo release 2.6.0

Ok, let's say that the VE is 1212, and vital stats are:
<pre>[root@virt12 sbin]# vc 1212
VE_ROOT="/vz1/root/1212"
VE_PRIVATE="/vz1/private/1212"
OSTEMPLATE="fedora-core-2/20040903"
IP_ADDRESS="69.55.229.84"
TEMPLATES="devel-fc2/20040903 php-fc2/20040813 mysql-fc2/20040812 postgresql-fc2/20040813 mod_perl-fc2/20040812 mod_ssl-fc2/20040811 jre-fc2/20040823 jdk-fc2/20040823 mailman-fc2/20040823 analog-fc2/20040824 proftpd-fc2/20040818 tomcat-fc2/20040823 usermin-fc2/20040909 webmin-fc2/20040909 uw-imap-fc2/20040830 phpBB-fc2/20040831 spamassassin-fc2/20040910 PostNuke-fc2/20040824 sl-webalizer-fc2/20040
818"

[root@virt12 sbin]# vzctl exec 1212 df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/vzfs             4.0G  405M  3.7G  10% /</pre>

From this you can see that he’s using (and will minimally need free on the dst server) ~400MB, and he’s running on a Fedora 2 template, version 20040903. He’s also got a bunch of other templates installed. It’s is '''vital''' that '''all''' these templates exist on the dst system. To confirm that, on the dst system run:

For < 3.0:

<pre>[root@virt14 private]# vzpkgls | grep fc2
devel-fc2 20040903
PostNuke-fc2 20040824
analog-fc2 20040824
awstats-fc2 20040824
bbClone-fc2 20040824
jdk-fc2 20040823
jre-fc2 20040823
mailman-fc2 20040823
mod_frontpage-fc2 20040816
mod_perl-fc2 20040812
mod_ssl-fc2 20040811
mysql-fc2 20040812
openwebmail-fc2 20040817
php-fc2 20040813
phpBB-fc2 20040831
postgresql-fc2 20040813
proftpd-fc2 20040818
sl-webalizer-fc2 20040818
spamassassin-fc2 20040910
tomcat-fc2 20040823
usermin-fc2 20040909
uw-imap-fc2 20040830
webmin-fc2 20040909
[root@virt14 private]# vzpkgls | grep fedora
fedora-core-1 20040121 20040818
fedora-core-devel-1 20040121 20040818
fedora-core-2 20040903
[root@virt14 private]#</pre>

For these older systems, you can simply match up the date on the template. 

For >= 3.0:

<pre>[root@virt19 /vz2/private]# vzpkg list
centos-5-x86                    2008-01-07 22:05:57
centos-5-x86    devel
centos-5-x86    jre
centos-5-x86    jsdk
centos-5-x86    mod_perl
centos-5-x86    mod_ssl
centos-5-x86    mysql
centos-5-x86    php
centos-5-x86    plesk9
centos-5-x86    plesk9-antivirus
centos-5-x86    plesk9-api
centos-5-x86    plesk9-atmail
centos-5-x86    plesk9-backup
centos-5-x86    plesk9-horde
centos-5-x86    plesk9-mailman
centos-5-x86    plesk9-mod-bw
centos-5-x86    plesk9-postfix
centos-5-x86    plesk9-ppwse
centos-5-x86    plesk9-psa-firewall
centos-5-x86    plesk9-psa-vpn
centos-5-x86    plesk9-psa-fileserver
centos-5-x86    plesk9-qmail
centos-5-x86    plesk9-sb-publish
centos-5-x86    plesk9-vault
centos-5-x86    plesk9-vault-most-popular
centos-5-x86    plesk9-watchdog</pre>

On these newer systems, it's difficult to tell whether the template on the dst matches exactly the src. Just cause a centos-5-x86 is listed on both servers doesn't mean all the same packages are there on the dst. To truly know, you must perform a sample rsync:

 rsync -avn /vz/template/centos/5/x86/ root@10.1.4.61:/vz/template/centos/5/x86/

if you see a ton of output from the dry run command, then clearly there are some differences. You may opt to let the rsync complete (without running in dry run mode) the only downside is you've now used up more space on the dst and also the centos template will be a mess with old and new data- it will be difficult if not impossible to undo (if someday we wanted to reclaim the space).

If you choose to merge templates, you should closely inspect the dry run output. You should also take care to exclude anything in the /config directory. For example:

 rsync -av -e ssh --stats --exclude=x86/config  /vz/template/ubuntu/10.04/ root@10.1.4.62:/vz/template/ubuntu/10.04/

Which will avoid this directory and contents:
<pre>[root@virt11 /vz2/private]# ls /vz/template/ubuntu/10.04/x86/config*
app  os</pre>

This is important to avoid since the config may differ on the destination and we are really only interested in making sure the pacakges are there, not overwriting a newer config with an older one.

If the dst system was missing a template, you have 2 choices: 
# put the missing template on the dst system. 2 choices here: 
## Install the template from rpm (found under backup2: /mnt/data4/vzrpms/distro/) or 
## rsync over the template (found under /vz/template) - see above
# put the ve on a system which has all the proper templates

=== pre-seeding a migration ===

When migrating a customer (or when doing many) depending on how much data you have to transfer, it can take some time. Further, it can be difficult to gauge when a migration will complete or how long it will take. To help speed up the process and get a better idea about how long it will take you can pre-transfer a customer's data to the destination server. If done correctly, vzmigrate will see the pre-transferred data and pick up where you left off, having much less to transfer (just changed/new files). 

We believe vzmigrate uses rsync to do it's transfer. Therefore not only can you use rsync to do a pre-seed, you can also run rsync to see what is causing a repeatedly-failing vzmigrate to fail. 

There's no magic to a pre-seed, you just need to make sure it's named correctly.

Given:

source: /vz1/private/1234

and you want to migrate to /vz2 on the target system, your rsync would look like:

 rsync -av /vz1/private/1234/ root@x.x.x.x:/vz2/private/1234.migrated/

After running that successful rsync, the ensuing migrateonline (or migrate) will take much less time to complete- depending on the # of files to be analyzed and the # of changed files. In any case, it'll be much much faster than had you just started the migration from scratch.

Further, as we discuss elsewhere in this topic, a failed migration can be moved from <tt>/vz/private/1234</tt> to <tt>/vz/private/1234.migrated</tt> on the destination if you want to restart a failed migration. This should '''only''' be done if the migration failed and the CT is not running on the destination HN.

=== migrateonline intructions: src >=3.x -> dst>=3.x ===

A script called [[#migrateonline|migrateonline]] was written to handle this kind of move. It is basically a wrapper for <tt>vzmigrate</tt> – vzmigrate is a util to seamlessly- as no no reboot of the ve necessary- move a ve from one host to another. This wrapper was initially written cause vz virtuozzo version 2.6.0 has a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables, causing problems when the ve was started up on the dst system. [[#migrate|migrate]] mitigates that. Since it makes multiple ssh connections to the dst virt, it’s a good idea to put the pub key for the src virt in the authorized_keys file on the dst virt. In addition, migrateonline emails ve owners when their migration starts and stops. For this to happen they need to put email addresses (on a single line, space delimited) in a file on their system: /migrate_notify. If the optional target dir is not specified, the ve will be moved to the same private/root location as it was on the src virt. Note: <tt>migrate</tt> is equivalent to <tt>migrateonline</tt>, but will <tt>migrate</tt> a ve AND restart it in the process.

<pre>[root@virt12 sbin]# migrateonline
usage: /usr/local/sbin/migrateonline <ip of node migrating to> <veid> [target dir: vz | vz1 | vz2]
[root@virt12 sbin]# migrateonline 10.1.4.64 1212 vz
starting to migrate 1212 at Sat Mar 26 22:40:38 PST 2005
Turning off offline_management
Saved parameters for VE 1212
migrating with no start on 10.1.4.64
Connection to destination HN (10.1.4.64) is successfully established
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...
Syncing private area '/vz1/private/1212'
- 100% |*************************************************|
done
Successfully completed
clearing the arp cache
now going to 10.1.4.64 and clear cache and starting
starting it
Delete port redirection
Adding port redirection to VE(1): 4643 8443
Adding IP address(es) to pool: 69.55.229.84
Saved parameters for VE 1212
Starting VE ...
VE is mounted
Adding port redirection to VE(1): 4643 8443
Adding IP address(es): 69.55.229.84
Hostname for VE set: fourmajor.com
File resolv.conf was modified
VE start in progress...
finished migrating 1212 at Sat Mar 26 22 22:52:01 PST 2005
[root@virt12 sbin]#</pre>

Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine.

If they had backups, use the mvbackups command to move their backups to the new server:

 mvbackups 1212 virt14 vz

Rename the ve

 [root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/migrated-1212

 [root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/old-1212-migrated-20120404-noarchive

Update the customer’s systems in mgmt to reflect the new path and server.


IF migrateonline does not work, you can try again using simply migrate- this will result in a brief reboot for the ve.
Before you try again, make sure of a few things:

Depending on where in the migration died, there may be partial data on the dst system in 1 of 2 places:
(given the example above)

 /vz/private/1212

or 

 /vz/private/1212.migrated

before you run migrate again, you'll want to rename so that all data is in 
1212.migrated:

 mv /vz/private/1212 /vz/private/1212.migrated

this way, it will pick up where it left off and transfer only new files.

Likewise, if you want to speed up a migration, you can pre-seed the dst as follows:

 [root@virt12 sbin]# rsync -avSH /vz/private/1212/ root@10.1.4.64:/vz/private/1212.migrated/

then when you run migrate or migrateonline, it will only need to move the changed files- the migration will complete quickly

=== migrateonline/migrate failures (migrate manually) ===

Lets say for whatever reason the migration fails. If it fails with [[#migrateonline|migrateonline]], you should try [[#migrate|migrate]] (which will reboot the customer, so notify them ahead of time).

You may want to run a [[#pre-seeding_a_migration|pre-seed]] rsync to see if you can find the problem. On older virts, we've seen this problem due to a large logfile (which you can find and encourage the customer to remove/compress):
 for f in `find / -size +1048576k`; do ls -lh $f; done

You may also see migration failing due to quota issues.

You can try to resolve by copying any quota file into the file you need:

 cp /var/vzquota/quota.1 /var/vzquota/quota.xxx

If it complains about quota running you should then be able to stop it

 vzquota off xxxx

If all else fails, migrate to a new VEID
i.e. 1234 becomes 12341


If the rsync or [[#migrate|migrate]] fails, you can always move someone manually:

1. stop ve: <br>
 v stop 1234

2. copy over data<br>
 rsync -avSH /vz/private/1234/ root@1.1.1.1:/vzX/private/1234/

NOTE: if you've previously seeded the data (run rsync while the VE was up/running), and this is a subsequent rsync, make sure the last rsync you do (while the VE is not running, has the --delete option in the rsync)

3. copy over conf<br>
 scp /vzconf/1234.conf root@1.1.1.1:/vzconf

4. on dst, edit the conf to reflect the right vzX dir<br>
 vi /vzconf/1234.conf

5. on src remove the IPs<br>
 ipdel 1234 2.2.2.2 3.3.3.3

6. on dst add IPs <br>
 ipadd 1234 2.2.2.2 3.3.3.3

7. on dst, start ve: <br>
 v start 1324

8. cancel, then archive ve on src per above instrs.

=== migrate src=2.6.0 -> dst>=2.6.0, or mass-migration with customer notify ===

A script called <tt>migrate</tt> was written to handle this kind of move. It is basically a wrapper for vzmigrate – vzmigrate is a util to seamlessly move a ve from one host to another. This wrapper was initially written cause vz virtuozzo version 2.6.0 has a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables, causing problems when the ve was started up on the dst system. migrate mitigates that. Since it makes multiple ssh connections to the dst virt, it’s a good idea to put the pub key for the src virt in the authorized_keys file on the dst virt. In addition, migrate emails ve owners when their migration starts and stops. For this to happen they need to put email addresses (on a single line, space delimited) in a file on their system: /migrate_notify. If the optional target dir is not specified, the ve will be moved to the same private/root location as it was on the src virt. Note: migrateonline is equivalent to migrate, but will migrate a ve from one 2.6 '''kernel''' machine to another 2.6 kernel machine without restarting the ve.

<pre>[root@virt12 sbin]# migrate
usage: /usr/local/sbin/migrate <ip of node migrating to> <veid> [target dir: vz | vz1 | vz2]
[root@virt12 sbin]# migrate 10.1.4.64 1212 vz
starting to migrate 1212 at Sat Mar 26 22:40:38 PST 2005
Turning off offline_management
Saved parameters for VE 1212
migrating with no start on 10.1.4.64
Connection to destination HN (10.1.4.64) is successfully established
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...
Syncing private area '/vz1/private/1212'
- 100% |*************************************************|
done
Successfully completed
clearing the arp cache
now going to 10.1.4.64 and clear cache and starting
starting it
Delete port redirection
Adding port redirection to VE(1): 4643 8443
Adding IP address(es) to pool: 69.55.229.84
Saved parameters for VE 1212
Starting VE ...
VE is mounted
Adding port redirection to VE(1): 4643 8443
Adding IP address(es): 69.55.229.84
Hostname for VE set: fourmajor.com
File resolv.conf was modified
VE start in progress...
finished migrating 1212 at Sat Mar 26 22 22:52:01 PST 2005
[root@virt12 sbin]#</pre>

Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine (backup2 or mail).
Cancel the ve (first we have to rename things which migrate changed so cancelve will find them):

 [root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/1212.conf

On 2.6.1 you’ll also have to move the private area:
 [root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/1212

<pre>[root@virt12 sbin]# cancelve 1212
v stop 1212
v set 1212 --offline_management=no --save
Delete port redirection
Deleting IP address(es) from pool: 69.55.229.84
Saved parameters for VE 1212
mv /vzconf/1212.conf /vzconf/deprecated-1212
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414
don't forget to remove firewall rules and domains!
[root@virt12 sbin]#</pre>

Note: if the system had backups, [[#cancelve|cancelve]] would offer to remove them. You want to say '''no''' to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.
Then go to backup2 and move the dirs. So you’d do something like this:
 mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/

We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. 
If moving to the same drive, you can safely preserve hardlinks and move all files with:
 sh
 for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212  /mnt/data1/virt14/$f/vz/private/; done

To move everyone off a system, you’d do:
 for f in `vl`; do migrate <ip> $f; done

Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.

=== vzmigrate: src=2.6.1 -> dst>=2.6.0 ===

This version of vzmigrate works properly with regard to handling ips. It will not notify ve owners of moves as in the above example. Other than that it’s essentially the same.

<pre>[root@virt12 sbin]#  vzmigrate 10.1.4.64 -r no 1212:1212:/vz/private/1212:/vz/root/1212
migrating on 10.1.4.64
Connection to destination HN (10.1.4.64) is successfully established
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...
Syncing private area '/vz1/private/1212'
- 100% |*************************************************|
done
Successfully completed
Adding port redirection to VE(1): 4643 8443
Adding IP address(es) to pool: 69.55.229.84
Saved parameters for VE 1212
Starting VE ...
VE is mounted
Adding port redirection to VE(1): 4643 8443
Adding IP address(es): 69.55.229.84
Hostname for VE set: fourmajor.com
File resolv.conf was modified
VE start in progress...
[root@virt12 sbin]#</pre>

Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine (backup2 or mail).
Cancel the ve (first we have to rename things which vzmigrate changed so cancelve will find them):

<pre>[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/1212.conf
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/1212

[root@virt12 sbin]# cancelve 1212
v stop 1212
v set 1212 --offline_management=no --save
Delete port redirection
Deleting IP address(es) from pool: 69.55.229.84
Saved parameters for VE 1212
mv /vzconf/1212.conf /vzconf/deprecated-1212
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414
don't forget to remove firewall rules and domains!
[root@virt12 sbin]#</pre>

Note: if the system had backups, <tt>cancelve</tt> would offer to remove them. You want to say no to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.
Then go to backup2 and move the dirs. So you’d do something like this:
 mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/

We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. 
If moving to the same drive, you can safely preserve hardlinks and move all files with:
 sh
 for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212  /mnt/data1/virt14/$f/vz/private/; done

Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.

=== src=2.5.x ===

First, go to the private dir:

 cd /vz1/private/

Stop the VE - make sure it stops totally cleanly.
 
 vzctl stop 1212

Then you’d use vemove - a script written to copy over the config, create tarballs of the ve’s data on the destination virt, and cancel the ve on the source system (in this example we’re going to put a ve that was in /vz1/private on the src virt, in /vz/private on the dst virt):

<pre>[root@virt12 sbin]# vemove
ERROR: Usage: vemove veid target_ip target_path_dir
[root@virt12 sbin]# vemove 1212 10.1.4.64 /vz/private/1212
tar cfpP - 1212 --ignore-failed-read | (ssh -2 -c arcfour 10.1.4.64 "split - -b 1024m /vz/private/1212.tar" )
scp /vzconf/1212.conf 10.1.4.64:/vzconf
cancelve 1212
v stop 1212
v set 1212 --offline_management=no --save
Delete port redirection
Deleting IP address(es) from pool: 69.55.229.84
Saved parameters for VE 1212
mv /vzconf/1212.conf /vzconf/deprecated-1212
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414
don't forget to remove firewall rules and domains!
[root@virt12 sbin]#</pre>

Note: if the system had backups, cancelve would offer to remove them. You want to say no to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.
Then go to backup2 and move the dirs. So you’d do something like this:
 mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/

We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive. 
If moving to the same drive, you can safely preserve hardlinks and move all files with:
 sh
 for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212  /mnt/data1/virt14/$f/vz/private/; done

When you are done, go to /vz/private on the dst virt you will have files like this:

<pre>1212.taraa
1212.tarab
1212.tarac</pre>

Each one 1024m (or less, for the last one) in size.

on the dst server and run:

 cat 1212.tar?? | tar xpPBf -

and after 20 mins or so it will be totally untarred.  Now since the conf
file is already there, you can go ahead and start the system.

 vzctl start 1212

Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.

NOTE: you MUST tar the system up using the virtuozzo version of tar that
is on all the virt systems, and further you MUST untar the tarball with
the virtuozzo tar, using these options:  `<tt>tar xpPBf -</tt>`

If you tar up an entire VE and move it to a non-virtuozzo machine, that is
ok, and you can untar it there with normal tar commands, but do not untar
it and then repack it with a normal tar and expect it to work - you need
to use virtuozzo tar commands on virtuozzo tarballs to make it work.

The backups are sort of an exception, since we are just (usually)
restoring user data that was created after we gave them the system, and
therefore has nothing to do with magic symlinks or vz-rpms, etc.

== Moving a VE on the same virt ==

Easy way:<br>
Scenario 1: ve 123 is to be renamed 1231 and moved from vz1 to vz

 vzmlocal 123:1231:/vz/private/1231:/vz/root/1231

Scenario 2: ve 123 is to be moved vz1 to vz

 vzmlocal 123:123:/vz/private/123:/vz/root/123

vzmlocal will reboot the ve at the end of the move

'''Note:''' don't forget to fix /usr/local/sbin/backup.config for new disk location

Manual/old way:

1) <tt>vzctl stop 123</tt><br>
2) <tt>mv /vz1/private/123 /vz/private/.</tt><br>
(or cp -a if you want to copy)
3) in <tt>/etc/sysconfig/vz-scripts/123.conf</tt> change value<br>
of '<tt>VE_PRIVATE</tt>' variable to point to a new private area location
4) <tt>vzctl start 123</tt><br>
5) update backups if needed: <tt>mvbackups 123 virtX virt1 vz</tt><br>
6) update management scerens<br>

Notes: a) absolute path to private area is stored in quota file <tt>/var/vzquota/quota.123</tt> - so during first startup quota will be recalculated.<br>
b) if you're going to write some script to do a job, you MUST be sure that $VEID won't be expanded to '' in ve config file - ie. you need to escape '$'. Otherwise you might have:

 VE_PRIVATE="/vz/private/"

in config, and 'vzctl destroy' for this VE ID '''will remove everything under /vz/private/ directory'''.

== Adding a veth device to a VE ==

Not totally sure what this is, but a customer asked for it and here's what we did (as instructed by vz support):

<pre>v set 99 --netif_add eth99  --save
ipdel 99 69.55.230.58
v set 99 --ifname eth99 --ipadd 69.55.230.58 --save
v set 99 --ifname eth99 --gateway 69.55.230.1 --save

vznetcfg net new veth_net

# vznetcfg net list
Network ID        Status      Master Interface  Slave Interfaces
net99             active      eth0              veth77.77,veth99.99
veth_net          active

# vznetcfg if list
Name             Type       Network ID       Addresses
br0              bridge     veth_net
br99             bridge     net99
veth99.99        veth       net99
eth1             nic                         10.1.4.62/24
eth0             nic        net99            69.55.227.70/24

vznetcfg br attach br0 eth0

(will remove 99 from orig net and move to veth_net)
vznetcfg net addif veth_net veth99.99

# vznetcfg net list
Network ID        Status      Master Interface  Slave Interfaces
net99             active
veth_net          active      eth0              veth77.77,veth99.99

(delete the old crap)
vznetcfg net del net99


Then, to add another device in

v set 77 --netif_add eth77  --save
ipdel 77 69.55.230.78
v set 77 --ifname eth77 --ipadd 69.55.230.78 --save
v set 77 --ifname eth77 --gateway 69.55.230.1 --save
v set 77 --save --ifname eth77 --network veth_net

# vznetcfg if list
Name             Type       Network ID       Addresses
veth77.77        veth
br0              bridge     veth_net
veth99.99        veth       veth_net
eth1             nic                         10.1.4.62/24
eth0             nic        veth_net         69.55.227.70/24

vznetcfg net addif veth_net veth77.77

# vznetcfg if list
Name             Type       Network ID       Addresses
veth77.77        veth       veth_net
br0              bridge     veth_net
veth99.99        veth       veth_net
eth1             nic                         10.1.4.62/24
eth0             nic        veth_net         69.55.227.70/24

# vznetcfg net list
Network ID        Status      Master Interface  Slave Interfaces
veth_net          active      eth0              veth77.77,veth99.99

another example

v set 1182 --netif_add eth1182  --save
ipdel 1182 69.55.236.217
v set 1182 --ifname eth1182 --ipadd 69.55.236.217 --save
v set 1182 --ifname eth1182 --gateway 69.55.236.1 --save
vznetcfg net addif veth_net veth1182.1182
v set 1182 --save --ifname eth1182 --network veth_net


unused/not working commands:
ifconfig veth99.0 0
vznetcfg net list
vznetcfg br new br99 net99
vznetcfg br attach br99 eth0
vznetcfg br show
vznetcfg if list
vznetcfg net addif net99 veth99.99
vznetcfg net addif eth0 net99

---------

vznetcfg br new br1182 net1182

vznetcfg br attach br99 eth0
vznetcfg if list
vznetcfg net addif net99 veth99.99
vznetcfg net addif eth0 net99



vznetcfg net addif eth0 net1182


# vznetcfg net addif veth99.0 net99
--- 8< ---

vznetcfg net new net
# vznetcfg net addif eth0 net99


# vzctl set 99 --save --netif_add eth0 (at this stage veth99.0 interface have to appear
on node)
# vzctl set 99 --save --ifname eth0 --ipadd 69.55.230.58 (and probably few more arguments
here - see 'man vzctl')
# vznetcfg net addif veth99.0 net99</pre>


== Assigning/remove ip from a VE ==

1. Add or remove ips:
 ipdel 1234 1.1.1.1 2.2.2.2
 ipadd 1234 1.1.1.1 2.2.2.2

2. update Mgmt screens

3. offer to update any DNS we do for them

4. check to see if we had rules for old IP in firwall

== Enabling tun device for a ve ==
Note, there’s a command for this: [[#addtun|addtun]]

For posterity:
Make sure the tun.o module is already loaded before Virtuozzo is started: 
 lsmod 
Allow the VPS to use the TUN/TAP device: 
 vzctl set 101 --devices c:10:200:rw --save 
Create the corresponding device inside the VPS and set the proper permissions: 
 vzctl exec 101 mkdir -p /dev/net 
 vzctl exec 101 mknod /dev/net/tun c 10 200 
 vzctl exec 101 chmod 600 /dev/net/tun

== Remaking a system (on same virt) ==

1. [[#cancelve|cancelve]] (or v destroy x - ONLY if you're POSITIVE no data needs to be saved)

2. [[#vemake|vemake]] using same veid

3. [[#mvbackups|mvbackups]] or [[#vb|vb]] (if new mount point)

4. update mgmt with new dir/ip 

5. update firewall if changed ip

== Re-initialize quota for a VE ==

There’s a commamd for this now: [[#clearquota|clearquota]]

For posterity:

vzctl stop 1
vzquota drop 1
vzctl start 1


== Traffic accounting on linux ==

DEPRECATED - all tracking is done via bwdb now. This is how we used to track traffic.

TODO: update for diff versions of vz

Unlike FreeBSD, where we have to add firewall count rules to the system to count the traffic, on virtuozzo counts the traffic for us.  You an see the current traffic stats by running `vznetstat`:

<pre># vznetstat
VEID    Net.Class  Output(bytes)   Input(bytes)
-----------------------------------------------
24218     1            484M             39M
24245     1            463M            143M
2451      1           2224M            265M
2454      1           2616M            385M
4149      1            125M             68M
418       1           1560M             34M
472       1           1219M            315M
726       1            628M            317M
763       1            223M             82M
771       1           4234M            437M
-----------------------------------------------
Total:               13780M           2090M</pre>

As you can see the VEID is on a line with the in and out bytes.  So, we simply run a cron job:

 4,9,14,19,24,29,34,39,44,49,55,59 * * * * /root/vztrafdump.sh

Just like we do on FreeBSD - this one goes through all the VEs in /vz/private and greps the line from vznetstat that matches them and dumps it in /jc_traffic_dump on their system.  Then it does it again for all the VEs in /vz1/private.  It is important to note that vznetstat runs only once, and the grepping is done from a temporary file that contains that output - we do this because running vznetstat once for each VE that we read out of /vz/private and /vz1/private would take way too long and be too intensive.

You do not need to do anything to facilitate this other than make sure that that cron job is running - the vznetstat counters are always running, and any new VEs that are added to the system will be accounted for automatically.

Traffic resetting no longer works with vz 2.6, so we disable the vztrafdump.sh on those virts.

== Watchdog script ==

On some of the older virts, we have a watchdog running that kills procs that are deemed bad per the following:

/root/watchdog from quar1

  <pre>if echo $line | awk '{print $(NF-3)}' | grep [5-9]...
  then
# 50-90%
      if echo $line | awk '{print $(NF-1)}' | grep "...:.."
      then
# running for > 99min
        echo $line >> /root/wd.log
        /usr/sbin/vzpid $pid | tail -1 >> /root/wd.log
        kill -9 $pid

      fi
      if echo $line | awk '{print $(NF-1)}' | grep "....m"
      then
# running for > 1000min
        echo $line >> /root/wd.log
        /usr/sbin/vzpid $pid | tail -1 >> /root/wd.log
        kill -9 $pid

      fi
  fi

  if echo $line | awk '{print $(NF-3)}' | grep [1-9]...
  then
# running for 10-90 percent
    if echo $line | awk '{print $NF}' | egrep 'cfusion|counter|vchkpw'
    then

      if echo $line | awk '{print $(NF-1)}' | grep "[2-9]:.."
      then
# between 2-9min
        echo $line >> /root/wd.log
        /usr/sbin/vzpid $pid | tail -1 >> /root/wd.log
        kill -9 $pid

      elif echo $line | awk '{print $(NF-1)}' | grep "[0-9][0-9]:.."
      then
# up to 99min
        echo $line >> /root/wd.log
        /usr/sbin/vzpid $pid | tail -1 >> /root/wd.log
        kill -9 $pid

      fi
    fi
  fi</pre>


== Misc Linux Items ==

We are overselling hard drive space ... when you configure a linux system with a certain amount of disk space (the default is 4gigs) you do not actually use up 4gigs of space on the system.  The diskspace setting for a user is simply a cap, and they only use up as much space on the actual disk drive as they are actually using.

When you create a new linux system, even though there are some 300 RPMs or so installed, if you run `df -k` you will see that the entire 4gig partition is empty - no space is being used.  This is because the files in their system are "magic symlinks" to the template for their OS that is in /vz/template - however, any changes to any of those files will "disconnect" them and they will immediately begin using space in their system.  Further, any new files uploaded (even if those new files overwrite existing files) will take up space on the partition.

=== Cant change file /etc/sysconfig/network ===
if you see this:

<pre>[root@virt8 root]# vzctl stop 160 ; vzctl start 160
VE is not running
Starting VE ...
VE is unmounted
VE is mounted
Adding IP address(es): 69.55.226.83 69.55.226.84
bash ERROR: Can't change file /etc/sysconfig/network
Deleting IP address(es): 69.55.226.83 69.55.226.84
VE is unmounted
[root@virt8 root]#</pre>

it probably means they no longer have /bin/bash - copy one in for them
 
ALSO: another possibility is that they have removed the `ed` RPM from their system - it needs to be reinstalled into their system.  But since their system is down, this is tricky ...

VE startup scripts used by 'vzctl' want package 'ed' to be available inside VE. So if package 'ed' will be enabled in OS template config and OS template itself VE #827 is based on - this error should be fixed.

yes, it is possible to add RPM to VE while it not running.
Try to do following:

<pre># cd /vz/template/<OS_template_with_ed_package>/
# vzctl mount 827
# rpm -Uvh --root /vz/root/827 --veid 827 ed-0.2-25.i386.vz.rpm</pre>

Usually theres an error, but its ok

Note: replace 'ed-0.2-25.i386.vz.rpm' in last command with actual
version of 'ed' package you have.

----
=== What template is user using ===
So how do I know what template the user has ?  cat their conf file and it is listed in there.  For example, if the conf file has:

<pre>[root@virt12 sbin]# vc 1103
…snip…
OSTEMPLATE="debian-3.0/20030822"
TEMPLATES="mod_perl-deb30/20030707 mod_ssl-deb30/20030703 mysql-deb30/20030707 proftpd-deb30/20030703 webmin-deb30/20030823 "
…</pre>

then they are on debian 3.0, all of their system RPMs are in /vz/template/debian-3.0, and they are using version 20030822 of that debian 3.0 template. Also, they’ve also got additional packages installed (mod_perl, mod_ssl, etc).  Those are also found under /vz/template

----
=== Edits needed to run java ===

Edits needed to run java:

When we first created the VEs, the default setting for privvmpages was 93000:94000 ... which was high enough that most people never had problems ... however, you can;t run java or jdk or tomcat or anything java related with that setting.  We have found that by setting privvmpages to 610000:615000 that java runs just fine.  That is now the default setting. It is exceedingly rare that anyone needs it higher than that, although we have seen it once or twice.

Any problems with java at all - the first thing you need to do is see if the failcnt has raised for privvmpages.

<pre># vzctl start 160
Starting VE ...
vzquota : (error) Quota on syscall for 160: Device or resource busy
Running vzquota on failed for VE 160 [3]</pre>

This is because my pwd is _in_ their private directory - you can't start it until you move out

People seem to have trouble with php if they are clueless newbies.  Here are two common problems/solutions:

no... but i figured it out myself. problem was the php.ini file that came
vanilla with the account was not configured to work with apache (the
ENGINE directive was set to off).

everything else seems fine now.

----

the problem was in the php.ini file.  I noticed that is wasnt showing
the code when it was in an html file so I looked at the php.ini file
and had to change it so it recognized <? tags aswell as <?php tags.

Also, make sure added to httpd.conf
    AddType application/x-httpd-php .php

----
=== Set time zone ===

You can set the time zone:

You can change the timezone by doing this:

 ln -sf /usr/share/zoneinfo/<zone> /etc/localtime

where <zone> is the zone you want in the /usr/share/zoneinfo/ directory.

----
=== Failing shm_open calls ===

Failing shm_open calls:

first, please check if /dev/shm is mounted inside VE.
'cat /proc/mounts' command should show something like this:
 tmpfs /dev/shm tmpfs rw 0 0

If /dev/shm is not mounted you have 2 ways to solve issue:
1. execute following command inside VE (doesn't require VE reboot):
 mount -t tmpfs none /dev/shm
2. add following string to /etc/fstab inside VE and reboot it:
 tmpfs         /dev/shm        tmpfs           defaults        0 0

----

You can have a mounted but not running ve
Just:
 vzctl mount <veid>

----
=== Debian can't get on the network ===
When a debian sys can’t get on the network, and you try:

<pre>vzctl set 1046 --ipadd 69.55.227.117
Adding IP address(es): 69.55.227.117
Failed to bring up lo.
Failed to bring up venet0.</pre>

They probably removed iproute package, which must be the one from swsoft. To restore:
<pre># dpkg -i --veid=1046 --admindir=/vz1/private/1046/root/var/lib/dpkg --instdir=/vz1/private/1046/root/ /vz/template/debian-3.0/iproute_20010824-8_i386.vz.deb
(Reading database ... 16007 files and directories currently installed.)
Preparing to replace iproute 20010824-8 (using .../iproute_20010824-8_i386.vz.deb) ...
Unpacking replacement iproute ...
Setting up iproute (20010824-8) ...</pre>

Then restart their ve

=== CentOS can't get on the network ===

When CentOS system was running, but can't get on the network, they probably updated iproute via yum.

Copy an older version of the iproute rpm to their VPS.

<pre>cp -p /root/iproute/iproute-2.6.32-23.el6.<i686>.rpm /vz2/private/<1527>/fs/root</pre>

Enter their ve, install rpm, restart network, and prevent yum from updating iproute.

<pre>
ve <1527>
rpm -i --force iproute-2.6.32-23.el6.<i686>.rpm
rpm -q iproute
rpm -e <new iproute rpm>
service network restart
echo "exclude=iproute" >>/etc/yum/ip-exclude
echo "include=/etc/yum/ip-exclude" >>/etc/yum.conf
</pre>




----

in a ve i do:

<pre>cd /
du -h .</pre>

and get: 483M    .

i do:

<pre>bash-2.05a# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/vzfs             4.0G  2.3G  1.7G  56% /</pre>

how can this be?

Is it possible that quota file was corrupted somehow? Please try to:   
<pre>vzctl stop <VEID>
vzquota drop <VEID>
vzquota init <VEID>
vzctl start <VEID></pre>

----

How to stop vz from starting after reboot:

 VIRTUOZZO=no 
in 
 /etc/sysconfig/vz

To start: 
 service vz start
(after setting VIRTUOZZO=yes in /etc/sysconfig/vz)

service vz restart will do some kind of 'soft reboot' -- restart all
VPSes and reload modules without rebooting the node

if you need to shut down all VPSes really really fast, run killall -9 init

----

Postfix tip:

You may want to tweak settings: default_process_limit=10

----

* uptime/load that vz users see isn’t what we see- it's their own load
* Vz overwrites resolve.conf on every boot to what’s in ve config file. Also overwrites hostname
* Quotas in vz: set Gid_limit in vz config file and restart ve. Errors leading up to this will be out of files but no inode or space issues. set to 2000 in conf, and restart vz