Editing VPS Management (section)

== Hitting performance barriers and fixing them ==

There are multiple modes virtuozzo offers to allocate resources to a ve. We utilize 2: SLM and UBC parameters
On our 4.x systems, we use all SLM – it’s simpler to manage and understand. There are a few systems on virt19/18 that may also use SLM. Everything else uses UBC. 
You can tell a SLM ve by:

 SLMMODE="all"

in their conf file. 

TODO: detail SLM modes and parameters.

If someone is in SLM mode and they hit memory resource limits, they simply need to upgrade to more memory.

The following applies to everyone else (UBC).

Customers will often email and say that they are getting out of memory errors - a common one is "cannot fork" ... basically, anytime you see something odd like this, it means they are hitting one of their limits that is in place in their conf file.

The conf file, however, simply shows their limits - how do we know what they are currently at ?

The answer is a file called v - this file contains the current status (and peaks) of their  performance settings, and also counts how many times they have hit the barrier.  The output of the file looks like this:

<pre>764: kmemsize         384113     898185    8100000    8200000          0
     lockedpages           0          0        322        322          0
     privvmpages        1292       7108     610000     615000          0
     shmpages            270        528      33000      34500          0
     dummy                 0          0          0          0          0
     numproc               8         23        410        415          0
     physpages            48       5624          0 2147483647          0
     vmguarpages           0          0      13019 2147483647          0
     oomguarpages        641       6389      13019 2147483647          0
     numtcpsock            3         21       1210       1215          0
     numflock              1          3        107        117          0
     numpty                0          2         19         19          0
     numsiginfo            0          4        274        274          0
     tcpsndbuf             0      80928    1800000    1900000          0 
     tcprcvbuf             0     108976    1800000    1900000          0
     othersockbuf       2224      37568     900000     950000          0
     dgramrcvbuf           0       4272     200000     200000          0
     numothersock          3          9        650        660          0
     dcachesize        53922     100320     786432     818029          0
     numfile             161        382       7500       7600          0
     dummy                 0          0          0          0          0
     dummy                 0          0          0          0          0
     dummy                 0          0          0          0          0
     numiptent             4          4        155        155          0</pre>

The first column is the name of the counter in question - the same names we saw in the systems conf file.  The second column is the _current_ value of that counter, the third column is the max that that counter has ever risen to, the fourth column is the soft limit, and the fifth column is the hard limit (which is the same as the numbers in that systems conf file).

The sixth number is the failcount - how many times the current usage has risen to hit the barrier.  It will increase as soon as the current usage hits the soft limit.

The problem with /proc/user_beancounters is that it actually contains that set of data for every running VE - so you can't just cat /proc/user_beancounters - it is too long and you get info for every other running system.

You can vzctl enter the system and run:

 vzctl enter 9999
 cat /proc/user_beancounters

inside their system, and you will just see the stats for their particular system, but entering their system every time you want to see it is combersome.

So, I wrote a simple script called "vzs" which simply greps for the VEID, and spits out the next 20 or so lines (however many lines there are in the output, I forget) after it.  For instance:

<pre># vzs 765:
765: kmemsize        2007936    2562780    8100000    8200000          0
     lockedpages           0          8        322        322          0
     privvmpages       26925      71126     610000     615000          0
     shmpages          16654      16750      33000      34500          0
     dummy                 0          0          0          0          0
     numproc              41         57        410        415          0
     physpages          1794      49160          0 2147483647          0
     vmguarpages           0          0      13019 2147483647          0
     oomguarpages       4780      51270      13019 2147483647          0
     numtcpsock           23         37       1210       1215          0
     numflock             17         39        107        117          0
     numpty                1          3         19         19          0
     numsiginfo            0          6        274        274          0
     tcpsndbuf         22240     333600    1800000    1900000          0
     tcprcvbuf             0     222656    1800000    1900000          0
     othersockbuf     104528     414944     900000     950000          0
     dgramrcvbuf           0       4448     200000     200000          0
     numothersock         73        105        650        660          0
     dcachesize       247038     309111     786432     818029          0
     numfile             904       1231       7500       7600          0
     dummy                 0          0          0          0          0
     dummy                 0          0          0          0          0
     dummy                 0          0          0          0          0
     numiptent             4          4        155        155          0</pre>

That showed us just the portion of /proc/user_beancounters for system 765.

When you run the vzs command, always add a : after the VEID.

So, if a customer complains about some out of memory errors, or no more files, or no more ptys, or just has an unspecific complain about processes dying, etc., the very first thing you need to do is check their beancounters with vzs.  Usually you will spot an item that has a high failcount and needs to be upped.

At that point you could simply up the counter with `vzctl set`.  Generally pick a number 10-20% higher than the old one, and make the hard limit slightly larger than the the soft limit. However our systems now come in several levels and those levels have more/different memory allocations. If someone is complaining about something other than a memory limit (pty, numiptent, numflock), it’s generally safe to increase it, at least to the same level as what’s in the /vzconf/4unlimited file on the newest virt. If someone is hitting a memory limit, first make sure they are given what they deserve:

(refer to mgmt -> payments -> packages)

To set those levels, you use the [[#setmem|setmem]] command. 

The alternate (DEPRECATED) method would be to use one of 3 commands:
256 <veid>
300 <veid>
384 <veid>
512 <veid>

If the levels were not right (you’d run vzs <veid> before and after to see the effect) tell the customer they’ve been adjusted and be done with it. If the levels were right, tell the customer they must upgrade to a higher package, tell them how to see level (control panel) and that they can reboot their system to escape this lockup contidion.

Customers can also complain that their site is totally unreachable, or complain that it is down ... if the underlying machine is up, and all seems well, you may notice in the beancounters that network-specific counters are failing - such as numtcpsock, tcpsndbuf or tcprcvbuf.  This will keep them from talking on the network and make it seem like their system is down.  Again, just up the limits and things should be fine.

On virts 1-4, you should first look at the default settings for that item on a later virt, such as virt 8 - we have increased the defaults a lot since the early machines.  So, if you are going to up a counter on virt2, instead of upping it by 10-20%, instead up it to the new default that you see on virt8.