JCWiki - User contributions [en]

IPKVM

2014-07-14T18:19:53Z

72.207.101.92:

We have 3 Belkin F1DE101H IPKVMs:

https://kvm1.johncompanies.com (@ i2b) 
https://kvm2.johncompanies.com (@ castle) 
https://kvm3.johncompanies.com (@ i2b) 

Current Locations:

kvm1: i2b Cab 4 col01224 6/14/14 obc-c.sea-incorporated.com 
kvm2: castle Cab 6-16 col01372 web3.bookoo 7/8/14 
kvm3: i2b Cab 8 ? ? 

Manual: http://cache-www.belkin.com/support/dl/p75075_f1de101h_man1.pdf

Each kvm is accessed via a web interface. With the KVM you have keyboard/mouse/video virtually in a java applet. You can also mount up ISO images that then appear as a usb CD-ROM drive to the server/os.

The KVMs have 4 cable/plugs: 1 video, 1 USB, and 1 PS/2 keyboard, 1 PS/2 mouse

There are also "monitor" ports on the KVM, allowing someone at the data center to piggy-back on the KVM, seeing what the KVM sees (since the KVM will be occupying all PS2/video ports on the server).

We use these KVMs internally when we need to load/rescue a server, and we also provide them to customers so they may load, rescue, troubleshoot their dedicated servers. We NEVER allow VPS customers to use the KVMs to access the host machine on which their VPS runs.

The location and server to which each KVM is attached is located in a file on mail ~user/kvm
This is usually in screen p4 in the mailboxes screen. When we know a KVM is no longer in use, we place a "DONE" after its entry.

There are a handful of options in the KVM we will modify on a regular basis:

KVM Console: this will launch the viewer window- where you may interact with the server as though you were standing in front of it with a monitor/keyboard/mouse

CD-ROM Image: on this screen we setup an ISO image to appear as a local USB drive to the server/OS. The options we provide are:
Share host: 69.55.230.10 (castle) or 69.55.229.4 (i2b) 
Share name: data 
Path to image: the file name of the ISO 
User, Password: (leave blank) 
Any ISO which you place on backup2:/data2/iso (castle) or backup3:/data/iso (i2b) will be available to the KVM for mounting.

We periodically take this list and publish as http://johncompanies.com/knowledge/isos.html so customers know what is available. However, not all images are available at each location, as noted.

If you or the customer wants to use their own ISO, we can either download that to the backup server or they can make their local CD-ROM drive appear on the server by using Belkin drive redirection software:
http://johncompanies.com/knowledge/Drive Redirection 1.0.5 Belkin.exe

Users: each time we allocate the KVM to a new customer, we reset the password for the guest user. You may use anything, but usually we just use the customer's CID: col0xxxx. To change, select 'guest' from the 'Existing users', enter password (and confirm) and click 'Modify'. Note, the guest user has a lower level of access, for instance they cannot change KVM settings, like Keyboard/Mouse

Keyboard/Mouse: on this screen we can adjust which emulation the KVM should make available to the server- USB or PS/2 or Auto. To change the setting, select Auto, USB or PS/2 from the 'Host Interface' dropdown. Click 'Apply' at the bottom of the screen to save. DON'T CLICK 'Update' - it won't save the setting. The customer, using the guest account, cannot modify this.

Unit Reset: this screen gives options to reset parts of the IPKVM or the entire IPKVM. Usually we are resetting PS/2 or USB to get the keyboard to respond- rarely does the entire unit need to be reset.

Notes regarding assignment

This is a floating KVM - we move it around on an as-needed basis. With 2 KVM's at i2b and 1 at castle, usually we don't have conflicts where multiple customers need to use it at exactly the same time. Castle does have their own IPKVM which we can also use.

The steps for assigning our KVM to a new customer are:

1. determine which KVM to use. Which KVM's are free/done/available? Note, we don't always mark a KVM "D(one)" (in the ~user/kvm file) since customers don't always say when they're done. So, you have to use a bit of judgement to know who was using it last and what for. If it was for a reload, they may still be using it. If it was to get back in after a crash or network config mistake, chances are they're done and you can take it. If in doubt, login to the KVM and see if it's actively being used.

2. reset the guest password. This locks out the current (soon to be previous) user- we don't want them to be able to login to the KVM and control someone elses server.

3. contact the data center asking them to move the kvm. Usually this will take the form of: "please move kvmN from cabinet A to col0XXX in cabinet B". We usually email castle and use http://portal.i2bnetworks.com to notify i2b, but if you or the customer is on a rush, call the data center- i2b can be especially slow to handle tickets. castle/i2b phone numbers are in the ~user/kvm file on mail.

4. confirm the KVM is up- usually we ping it and watch for it to start pinging again, it takes about 1min to boot up after being plugged in. Login, make sure the keyboard is working- if not adjust input and/or reset as necessary (Keyboard/Mouse, Unit Reset).

5. contact the customer and send them the kvm email from pastes- EDITS: kvm, password, IP of share host: 69.55.230.10 (castle) or 69.55.229.4 (i2b)

Notes regarding usage

Occasionally, even after plugging in the KVM and it appears it has power, it will not come up- the nic lights don't light up. If this happens ask the noc staff to pull power and/or use a paper clip to reset the unit.

Some servers do not have legacy ports, therefore if the setting in Keyboard/Mouse was PS/2, the keyboard would not work. Conversely, some OSs don't recognize USB keyboards, you'd want to force PS/2. Usually this takes effect right away, but occasionally you'll want to reset USB and/or PS/2 via Unit Reset.

FreeBSD in particular will not recognize a PS/2 keyboard which wasn't present at OS boot. So you will need to either try USB first, or reboot to get PS/2 recognized.

If ever things aren't working (no vid or kbd not working), ask the tech to confirm the cables are not plugged into the monitor port (the monitor ports are labeled "console", and are left of the "CPU/KVM" ports- which go to the server).

To get BIOS to recognize the USB CD-ROM, you may need to tweak several settings in the BIOS screens.
Options vary per hardware, some options to check:
Boot optical: enable 
Boot removable: enable 
Boot network: disable 
USB boot: enable 
Boot USB first: enable 

In the boot menu, make “Belkin Virtual” (or whatever it's recognized as) first in boot order. In order for the BIOS to see the CD-ROM, it must be configured via the IPKVM before you reboot the server

Usually a frozen server will not respond to a ctrl+alt+del (both from kbd sequence and also from the button in the KVM applet), just like it wouldn't if a real kbd was attached.

If you logout or timeout you'll need to remount the image. Some OS/MB combinations don't like the USB drive method. In these cases we suggest using the ftp/netinstall setup option to avoid having to access the CD/USB drive. OS's known to have this issue: Debian, FreeBSD

If you need to communicate with the customer, 2 people can use the KVM applet at the same time and there's a chat window available from the options drop down menu.

If ever you find that you are locked out of keyboard access after attaching an ISO image (this sometimes happens on modern motherboards which do not have any ps/2 connectors) you can usually get keyboard back by doing one or more of the following:

* reset the usb (in the reset menu)
* "reactivate" the ISO image
* reset the entire kvm
* sometimes attaching a physical keyboard to the system (then rebooting with that keyboard attached) solves the problem
* use 2 IPKVMs: 1 providing the ISO and the other acting as video/mouse

= 3wfirmware.iso =

This is a special custom-made iso that we place various drivers and flashing programs on so we can easily flash bios's without needing to go to the DC with a USB or CD drive. To place new software on the iso:

Copy the iso file over to a server that can mount it (linux-based box, i.e. virt16);
scp backup2:/d2/iso/3wfirmware.iso /tmp/3wfirmware.iso

mount it up:
mount -t vfat -o loop /tmp/3wfirmware.iso /mnt/usb/

copy files over to /mnt/usb

umount it
umount /mnt/usb

copy back to backup server:

scp /tmp/3wfirmware.iso backup2:/d2/iso/3wfirmware.iso

VPS Management

2014-06-13T18:08:52Z

72.207.101.92: /* mknod in a jail */

= Common Problems =
== Login to any machine without a password ==

This is possible via the use of ssh keys. The process is thus:

1. place the public key for your user (root@mail) in the /root/.ssh/authorized_keys file on the server you wish to login to
cat /root/.ssh/id_dsa.pub
(paste that into authorized_keys on the target server). If the file doesn't exist, create it.

2. enable root login (usually only applies to FreeBSD). Edit the /etc/ssh/sshd_config on the target server and change:
<tt>#PermitRootLogin no</tt>
to
<tt>PermitRootLogin yes</tt>

3. Restart the sshd on the target machine. First, find the sshd process:
jailps <hostname> | grep sshd
or
vp <VEID> | grep sshd

Look for the process resembling:
root 17296 0.0 0.0 5280 1036 ? Ss 2011 4:27 /usr/sbin/sshd
(this is the sshd)

Not:
root 6270 0.5 0.0 6808 2536 ? Ss 14:33 0:00 sshd: root [priv]
(this is an sshd child- someone already ssh'd in as root)

Restart the sshd:
kill -1 <PID>

Ex:
kill -1 17296

You may now ssh in.

Once you're done, IF you enabled root login, you should repeat steps 2 and 3 to disable root logins.

== Letting someone in who has locked themselves out (killed sshd, lost pwd) ==

There are two ways people frequently lock themselves out - either they forget a password, or they kill off sshd somehow.

These are actually both fairly easy to solve. First, let's say someone kills off their sshd, or somehow mangles /etc/ssh/sshd_config such that it no longer lets them in.

Their email may be very short, or it may have all sorts of details about how you should fix sshd_config to let them in ... just ignore all of this. They can fix their own mangled sshd. Fixing this is very simple. First, edit the /etc/inetd.conf on their system and uncomment the telnet line:

telnet stream tcp nowait root /usr/libexec/telnetd telnetd
#telnet stream tcp6 nowait root /usr/libexec/telnetd telnetd

(just leave the tcp6 version of telnet commented)

Then, use jailps to list the processes on their system, and find their inetd process. Then simply:

kill -HUP (pid)

where (pid) is the PID of their inetd process. Now they have telnet running on their system and they can log in and do whatever they need to do.

The only complications that could occur are:

a) their firewall config on our firewall has port 23 blocked, in which case you will need to open that - will be covered in a different lesson.

b) they are not running inetd, so you can't HUP it. If this happens, edit their /etc/rc.conf, add the inetd_enable="YES" line, and then kill
their jail with /tmp/jailkill.pl - then restart their jail with the jail line from their quad/safe file. Easy.

If they have forgotten a password,

On 6.x+ you can reset their password with:
jexec <jailID from jls> passwd root

Note: the default password for 6.x jails is 8ico2987, for 4.x it is p455agfa

On 4.x, you need to cd to their etc directory
... for instance:

cd /mnt/data2/198.78.65.136-col00261-DIR/etc

and run:

vipw -d .

Then paste in these two lines (theres a paste with these):

root:$1$krszPxhk$xkCepSnz3mIikT3vCtJCt0:0:0::0:0:Charlie &:/root:/bin/csh
user:$1$Mx9p5Npk$QdMU6c8YQqp2FW2M3irEh/:1001:1001::0:0:User &:/home/user:/bin/sh

overwriting the lines they already have for "user" and "root" - then just tell them that both user and root have been reset to the default password of p455agfa.

For linux, just passwd inside shell or
vzctl set <veid> --userpasswd root:p455agfa –save

Starting in 2009 we began giving out randomized passwords for FreeBSD and Linux as the default password. That is stored with each system in Mgmt. You should look for and reset the password to that password in the event of a reset and refer the customer to use their original password from their welcome email- this way we don’t have to send the password again via email (in clear text).

== sendmail can’t be contacted from ext ip (only locally) ==

By default redhat puts this line in sendmail.mc:

DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')

which makes it only answer on localhost. Comment it out like:

dnl DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')

and then rebuild sendmail.cf with:

m4 /etc/mail/sendmail.mc > /etc/sendmail.cf

== virt doesn’t properly let go of ve’s ip(s) when moved to another system ==

On virtuozzo 2.6 systems, it's been observed that when moving ips from one virt to another that sometimes the routing table will not get updated to reflect the removal of the ip addresses.

A recent example was a customer that was moving to a new ve on a new virt and the ip addresses were traded between the two ve's. After the trade the two systems were not able to talk to each other. When looking at the routing table for the old system all the ip addresses were still in the routing table as being local, like so:

<pre>netstat -rn | grep 69.55.225.149
69.55.225.149 0.0.0.0 255.255.255.255 UH 40 0 0 venet0</pre>

This was preventing traffic to the other system from being routed properly.
The solution is to manually delete the route:

route delete 69.55.225.149 gw 0.0.0.0

Supposedly, this was fixed in 2.6.1

== sshd on FreeBSD 6.2 segfaults ==

First try to reinstall ssh

<pre>cd /usr/src/secure
cd lib/libssh
make depend && make all install
cd ../../usr.sbin/sshd
make depend && make all install
cd ../../usr.bin/ssh
make depend && make all install</pre>

Failing that, find the library that’s messed up:

<pre>ldd /usr/sbin/sshd
libssh.so.3 => /usr/lib/libssh.so.3 (0x280a3000)
libutil.so.5 => /lib/libutil.so.5 (0x280d8000)
libz.so.3 => /lib/libz.so.3 (0x280e4000)
libwrap.so.4 => /usr/lib/libwrap.so.4 (0x280f5000)
libpam.so.3 => /usr/lib/libpam.so.3 (0x280fc000)
libbsm.so.1 => /usr/lib/libbsm.so.1 (0x28103000)
libgssapi.so.8 => /usr/lib/libgssapi.so.8 (0x28112000)
libkrb5.so.8 => /usr/lib/libkrb5.so.8 (0x28120000)
libasn1.so.8 => /usr/lib/libasn1.so.8 (0x28154000)
libcom_err.so.3 => /usr/lib/libcom_err.so.3 (0x28175000)
libroken.so.8 => /usr/lib/libroken.so.8 (0x28177000)
libcrypto.so.4 => /lib/libcrypto.so.4 (0x28183000)
libcrypt.so.3 => /lib/libcrypt.so.3 (0x28276000)
libc.so.6 => /lib/libc.so.6 (0x2828e000)
libmd.so.3 => /lib/libmd.so.3 (0x28373000)</pre>

md5 them and compare to other jail hosts or jails running on host

for libcrypto reinstall:
<pre>/usr/src/crypto
make depend && make all install</pre>

== Moving a server to a different account in the management system ==

This is a tough one. You gotta touch a bunch of tables in the mysql database.
<pre>
systems
subs
ipmap
</pre>

basically, get the sysid and go off that:

<pre>
update systems set cid='newcid' where sysid=xxxxx
update subs set cid='newcid' where sysid=xxxxx
update ipmap set cid='newcid' where sysid=xxxxx
</pre>

You then need to rename the server disk space to change the old ID to the new one.

For a jail -
<pre>
stopjail <hostname>
mv 69.55.2XX.XXX-col0<oldid> 69.55.2XX.XXX-col0<newid>
mv 69.55.2XX.XXX-col0<oldid>-DIR 69.55.2XX.XXX-col0<newid>-DIR
vi /usr/local/jail/bin/quad1 [Update the ID in the quad file]
startjail <hostname>
</pre>

For a virt -
You will have to
* stop the virt
* rename the directories
* modify the config file /vzconf/<veid>.conf
* rename the virt conf file
* restart the new virt.

that ought to do it..

= FreeBSD VPS =

== Starting jails: Quad/Safe Files ==

FreeBSD customer systems do not start up automatically at boot time. When one of our freebsd machines boots up, it boots up, and does nothing else. To start jails, we put the commands to start each jail into a shell script(s) and run the script(s). Jail startup is something that needs to be actively monitored, which is why we don’t just run the script automatically. More on monitoring later.

NOTE: >=7.x we have moved to 1 quad file: <tt>quad1</tt>. Startups are not done by running each quad, but rather [[#startalljails|startalljails]] which relies on the contents of <tt>quad1</tt>. The specifics of this are lower in this article. What follows here applies for pre 7.x systems.

There are eight files in <tt>/usr/local/jail/rc.d</tt>:

<pre>jail3# ls /usr/local/jail/rc.d/
quad1 quad2 quad3 quad4 safe1 safe2 safe3 safe4
jail3#</pre>

four quad files and four safe files.

Each file contains an even number of system startup blocks (total number of jails divided by 4)

The reason for this is, if we make one large script to startup all the systems at boot time, it will take too long - the first system in the script will start up right after system boot, which is great, but the last system may not start for another 20 minutes.

Since there is no way to parralelize this during the startup procedure, we simply open four terminals (in screen window 9) and run each script, one in each terminal. This way they all run simultaneously, and the very last system in each startup script gets started in 1/4th the time it would if there was one large file

The files are generally organized so that quad/safe 1&2 have only jails from disk 1, and quad/safe 3&4 have jails from disk 2. This helps ensure that only 2 fscks on any disk are going on at once. Further, they are balanced so that all quad/safe’s finish executing around the same time. We do this by making sure each quad/safe has a similar number of jails and represents a similar number of inodes (see js).

The other, very important reason we do it this way, and this is the reason there are quad files and safe files, is that in the event of a system crash, every single vn-backed filesystem that was mounted at the time of system crash needs to be fsck'd. However, fsck'ing takes time, so if we shut the system down gracefully, we don't want to fsck.

Therefore, we have two sets of scripts - the four quad scripts are identical to the four safe scripts except for the fact that the quad scripts contain fsck commands for each filesystem.

So, if you shut a system down gracefully, start four terminals and run safe1 in window one, and safe2 in window 2, and so on.

If you crash, start four terminals (or go to screen window 9) and run quad1 in window one, and quad2 in window 2, and so on.

Here is a snip of (a 4.x version) quad2 from jail17:

<pre>vnconfig /dev/vn16 /mnt/data2/69.55.228.7-col00820
fsck -y /dev/vn16
mount /dev/vn16c /mnt/data2/69.55.228.7-col00820-DIR
chmod 0666 /mnt/data2/69.55.228.7-col00820-DIR/dev/null
jail /mnt/data2/69.55.228.7-col00820-DIR mail1.phimail.com 69.55.228.7 /bin/sh /etc/rc

# moved to data2 col00368
#vnconfig /dev/vn28 /mnt/data2/69.55.236.132-col00368
#fsck -y /dev/vn28
#mount /dev/vn28c /mnt/data2/69.55.236.132-col00368-DIR
#chmod 0666 /mnt/data2/69.55.236.132-col00368-DIR/dev/null
#jail /mnt/data2/69.55.236.132-col00368-DIR limehouse.org 69.55.236.132 /bin/sh /etc/rc

echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’
vnconfig /dev/vn22 /mnt/data2/69.55.228.13-col01063
fsck -y /dev/vn22
mount /dev/vn22c /mnt/data2/69.55.228.13-col01063-DIR
chmod 0666 /mnt/data2/69.55.228.13-col01063-DIR/dev/null
jail /mnt/data2/69.55.228.13-col01063-DIR www.widestream.com.au 69.55.228.13 /bin/sh /etc/rc

# cancelled col00106
#vnconfig /dev/vn15 /mnt/data2/69.55.238.5-col00106
#fsck -y /dev/vn15
#mount /dev/vn15c /mnt/data2/69.55.238.5-col00106-DIR
#chmod 0666 /mnt/data2/69.55.238.5-col00106-DIR/dev/null
#jail /mnt/data2/69.55.238.5-col00106-DIR mail.azebu.net 69.55.238.5 /bin/sh /etc/rc</pre>

As you can see, two of the systems specified are commented out - presumably those customers cancelled, or were moved to new servers.

As you can see, the vnconfig line is the simpler command line, not the longer one that was used when it was first configured. As you can see, all that is done is, vnconfig the filesystem, then fsck it, then mount it. The fourth command is the `jail` command used to start the system – but that will be covered later.

Here is the safe2 file from jail17:

<pre>vnconfig /dev/vn16 /mnt/data2/69.55.228.7-col00820
mount /dev/vn16c /mnt/data2/69.55.228.7-col00820-DIR
chmod 0666 /mnt/data2/69.55.228.7-col00820-DIR/dev/null
jail /mnt/data2/69.55.228.7-col00820-DIR mail1.phimail.com 69.55.228.7 /bin/sh /etc/rc

# moved to data2 col00368
#vnconfig /dev/vn28 /mnt/data2/69.55.236.132-col00368
#mount /dev/vn28c /mnt/data2/69.55.236.132-col00368-DIR
#chmod 0666 /mnt/data2/69.55.236.132-col00368-DIR/dev/null
#jail /mnt/data2/69.55.236.132-col00368-DIR limehouse.org 69.55.236.132 /bin/sh /etc/rc

echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’
vnconfig /dev/vn22 /mnt/data2/69.55.228.13-col01063
mount /dev/vn22c /mnt/data2/69.55.228.13-col01063-DIR
chmod 0666 /mnt/data2/69.55.228.13-col01063-DIR/dev/null
jail /mnt/data2/69.55.228.13-col01063-DIR www.widestream.com.au 69.55.228.13 /bin/sh /etc/rc

# cancelled col00106
#vnconfig /dev/vn15 /mnt/data2/69.55.238.5-col00106
#mount /dev/vn15c /mnt/data2/69.55.238.5-col00106-DIR
#chmod 0666 /mnt/data2/69.55.238.5-col00106-DIR/dev/null
#jail /mnt/data2/69.55.238.5-col00106-DIR mail.azebu.net 69.55.238.5 /bin/sh /etc/rc</pre>

As you can see, it is exactly the same, but it does not have the fsck lines.

Take a look at the last entry - note that the file is named:

/mnt/data2/69.55.238.5-col00106

and the mount point is named:

/mnt/data2/69.55.238.5-col00106-DIR

This is the general format on all the FreeBSD systems. The file is always named:

IP-custnumber

and the directory is named:

IP-custnumber-DIR

If you run safe when you need a fsck, the mount will fail and jail will fail:

# mount /dev/vn1c /mnt/data2/jails/65.248.2.131-ns1.kozubik.com-DIR
mount: /dev/vn1c: Operation not permitted

No reboot needed, just run the quad script

Starting with 6.x jails, we added block delimiters to the quad/safe files, the block looks like:

<pre>echo '## begin ##: nuie.solaris.mu'
fsck -y /dev/concat/v30v31a
mount /dev/concat/v30v31a /mnt/data1/69.55.228.218-col01441-DIR
mount_devfs devfs /mnt/data1/69.55.228.218-col01441-DIR/dev
devfs -m /mnt/data1/69.55.228.218-col01441-DIR/dev rule -s 3 applyset
jail /mnt/data1/69.55.228.218-col01441-DIR nuie.solaris.mu 69.55.228.218 /bin/sh /etc/rc
echo '## end ##: nuie.solaris.mu'</pre>

These are more than just informative when running quad/safe’s, the echo lines MUST be present for certain tools to work properly. So it’s important that any updates to the hostname also be updated on the 2 echo lines. For example, if you try to startjail a jail with a hostname which is on the jail line but not the echo lines, the command will return with host not found.

=== FreeBSD 7.x+ notes ===

Starting with the release of FreeBSD 7.x, we are doing jail startups in a slightly different way. First, thereis only 1 file: <tt>/usr/local/jail/rc.d/quad1</tt>
There are no other quads or corresponding safe files. The reason for this is twofold, 1. We can pass –C to fsck which will tell is to skip the fsck if the fs is clean (no more need for safe files), 2. We have a new startup script which can be launched multiple times, running in parallel to start jails, where quad1 is the master jail file.
Quad1 could still be run as a shell script, but it would take a very long time for it to run completely so it’s not advisable; or you should break it down into smaller chunks (like quad1, quad2, quad3, etc)

Here is a snip of (a 7.x version) quad1 from jail2:

<pre>echo '## begin ##: projects.tw.com'
mdconfig -a -t vnode -f /mnt/data1/69.55.230.46-col01213 -u 50
fsck -Cy /dev/md50c
mount /dev/md50c /mnt/data1/69.55.230.46-col01213-DIR
mount -t devfs devfs /mnt/data1/69.55.230.46-col01213-DIR/dev
devfs -m /mnt/data1/69.55.230.46-col01213-DIR/dev rule -s 3 applyset
jail /mnt/data1/69.55.230.46-col01213-DIR projects.tw.com 69.55.230.46 /bin/sh /etc/rc
echo '## end ##: projects.tw.com'</pre>

Cancelled jails are no longer commented out and stored in quad1, rather they’re moved to <tt>/usr/local/jail/rc.d/deprecated</tt>

To start these jails, start the 4 ssh sessions as you would for a normal crash and then instead of running quad1-4, instead run startalljails in each window. IMPORTANT- before running startalljails you should make sure you ran preboot once as it will clear out all the lockfiles and enable startalljails to work properly.

== Problems with the quad/safe files ==

When you run the quad/safe files, there are two problems that can occur - either a particular system will hang during initialization, OR a system will spit out output to the screen, impeding your ability to do anything. Or both.

First off, when you start a jail, you see output like this:

<pre>Skipping disk checks ...
adjkerntz[25285]: sysctl(put_wallclock): Operation not permitted
Doing initial network setup:.
ifconfig: ioctl (SIOCDIFADDR): permission denied
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384
Additional routing options: TCP keepalive=YESsysctl:
net.inet.tcp.always_keepalive: Operation not permitted.
Routing daemons:.
Additional daemons: syslogd.
Doing additional network setup:.
Starting final network daemons:.
ELF ldconfig path: /usr/lib /usr/lib/compat /usr/X11R6/lib /usr/local/lib
a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout /usr/X11R6/lib/aout
Starting standard daemons: inetd cron sshd sendmail sendmail-clientmqueue.
Initial rc.i386 initialization:.
Configuring syscons: blanktime.
Additional ABI support:.
Local package initialization:.
Additional TCP options:.</pre>

Now, let's look at this line, near the end:

Local package initialization:.

This is where a list of daemons that are set to start at boot time willshow up. You might see something like:

Local package initialization: mysqld apache sendmail sendmail-clientmqueue

Or something like this:

Local package initialization: postgres postfix apache

The problem is that many systems (about 4-5 per machine) will hang on that line. Basically it will get to some of the way through the total daemons to be started:

Local package initialization: mysqld apache

and will just sit there. Forever.

Fortunately, pressing ctrl-c will break out of it. Not only will it break out of it, but it will also continue on that same line and start the other daemons:

Local package initialization: mysqld apache ^c sendmail-clientmqueue

and then continue on to finish the startup, and then move to the next system to be started.

So what does this mean? It means that if a machine crashes, and you start four screen-windows to run four quads or four safes, you need to periodically cycle between them and see if any systems are stuck at that point, causing their quad/safe file to hang. A good rule of thumb is, if you see a system at that point in the startup, give it another 100 seconds - if it is still at the exact same spot, hit ctrl-c. Its also a good idea to go back into the quad file (just before the first command in the jail startup block) and note that this jail tends to need a control-c or more time as follows:
<pre>echo '### NOTE ### slow sendmail'
echo '### NOTE ###: ^C @ Starting sendmail.'</pre>

'''NEVER''' hit ctrl-c repeatedly if you don't get an immediate response - that will cause the following jail’s startup commands to be aborted.

A second problem that can occur is that a jail - maybe the first one in that particular quad/safe, maybe the last one, or maybe one in the middle, will start spitting out status or error messages from one of its init scripts. This is not a problem - basically, hit enter a few times and see if you get a prompt - if you do get a prompt, that means that the quad/safe script has already completed. Therefore it is safe to log out (and log out of the user that you su'd from) and then log back in (if necessary).

The tricky thing is, if a system in the middle starts flooding with messages, and you hit enter a few times and don't get a prompt. Are you not getting a prompt because some subsequent system is hanging at the initialization, as we discussede above ? Or are you not getting a prompt because that quad file is currently running an fsck ? Usually you can tell by scrolling back in screen’s history to see what it was doing before you started getting the messages.

If you don’t get clues from history, you have to use your judgement - instead of giving it 100 seconds to respond, perhaps give it 2-3 mins ... if you still get no response (no prompt) when you hit enter, hit ctrl-c. However, be aware that you might still be hitting ctrl-c in the middle of an fsck. This means you will get an error like "filesystem still marked dirty" and then the vnconfig for it will fail and so will the jail command, and the next system in the quad file will then start starting up.

If this happens, just wait until the end of all the quad files have finished, and start that system manually.

If things really get weird, like a screen flooded with errors, and you can't get a prompt, and ctrl-c does nothing, then you need to just eventually (give it ten mins or so) just kill that window with ctrl-p, then k, and then log in again and manually check which systems are now running and which aren't, and manually start up any that are not.

Don't EVER risk running a particular quad/safe file a second time.
If the quad/safe script gets executed twice, reboot the machine immediately.

So, for all the above reasons, anytime a machine crashes and you run all the quads or all the safes, '''always''' check every jail afterwards to make sure it is running - even if you have no hangs or complications at all.
Run this command:

<tt>[[#jailpsall|jailpsall]]</tt>

Note: [[#postboot|postboot]] also populates ipfw counts, so it '''should not be run multiple times''', use <tt>jailpsall</tt> for subsequent extensive ps’ing

And make sure they all show as running. If one does not show as running, check its /etc/rc.conf file first to see if maybe it is using a different hostname first before starting it manually.

One thing we have implemented to alleviate these startup hangs and noisy jails, is to put jail start blocks that are slow or hangy at the bottom of the safe/quad file. Further, for each bad jail we note in each quad/safe just before the start block something like:

echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’

That way we’ll be prepared to ^C when we see that message appear during the quad/safe startup process. If you observe a new, undocumented hang, '''after''' the quad/safe has finished, place a line similar to the above in the quad file, move the jail start block to the end of the file, then run [[#buildsafe|buildsafe]]

== Making new customer jail 4.x ==

1. run [[#js|js]] to figure out which partition, IP to put it on, find an unused vn, and choose which quad it should go in

2. use col00xxx for both hostnames if they don’t give you a hostname

3. copy over dir and ip to pending customer screen

Usage: <tt>jailmake IP filepath vnX hostname shorthost quad/safe# ipfw# email [gb disk, default 4]</tt>

jail14# /tmp/jailmake 69.55.226.152 /mnt/data2/69.55.226.152-col00182 vn23 negev.cerastes.org negev 4 182 cerastes@cerastes.org

== Making new customer jail 6.x ==

1. run [[#js|js]] to figure out which gvinum vols are free (and which mnt (disk) each gvinum goes with/to), IP to put it on and choose which quad it should go in

2. use col00xxx for both hostnames if they don’t give you a hostname

3. copy over dir and ip to pending customer screen

Usage: <tt>jailmake IP filepath vN[,vN] hostname shorthost quad/safe# ipfw# email</tt>

jail19# jailmake 69.55.236.17 /mnt/data1/69.55.236.17-col01574 v10 uncle-leo.commadev.com uncle-leo 1 1574 lee@commadev.com,lee@gmail.com

== Changing an ip for freebsd VPS ==

*stopjail (hostname)
*on systems using vnfile or mdfile: rename vnfile/mdfile, using new ip
*rename dir using new ip
*Edit quad (make sure to update all lines: directory, vn/mdfile)
*buildsafe (not necessary on systems having only quad1)
*if you're adding an ip not already on the host machine: <tt>ipadd x.x.x.x</tt>
*startjail (hostname)
if backups: <tt>mvbackups</tt> ...
*Edit dir/ip in mgmt
*if any rules, update any firewall rules to use new ip (careful not to make sure there arent multiple rules per ipfw#- search by colo#)
*(if customer asks and has domains) update their domains on ns1c
*optional: update ipfw counters on the host server

== Rename a gconcat vol ==

*stop jail, unmount everything
*gconcat stop vol
*gconcat clear /dev/gvinum/vN
*gconcat clear /dev/gvinum/vN+1
*gconcat label -v newname /dev/gvinum/vN /dev/gvinum/vN+1
*bsdlabel /dev/concat/newname
(make sure a partition is not fstype unused, if so change to: 4.2BSD)

== Remaking a VPS (on same jail) ==

# take him down (stopjail)
# make note of host, vn/gvin/md, ip
# remove from quad/safe
# move vnfile (rename)
# remove dir (jailmake will make a new one)
# remove ipfw counts
# jailmake
# update backup script
# update db with new dir/ip (use the “move” link, mark old as stopped today)
# update firewall if changed ip

== Recovering from a crash (FreeBSD) ==

=== Diagnose whether you have a crash ===
The most important thing is to get the machine and all jails back up as soon as possible. Note the time, you’ll need to create a crash log entry (Mgmt. -> Reference -> CrashLog). The first thing to do is head over to the [[Screen#Screen_Organization|serial console screen]] and see if there’s any kernel error messages output. Try to copy any messages (or just a sample of repeating messages) you see into the notes section of the crash log. If there are no messages, the machine may just be really busy- wait a bit (5-10min) to see if it comes back. If it's still pinging, odds are its very busy. Note, if you see messages about swap space exhausted, the server is obviously out of memory, however it may recover briefly enough for you to get a jtop in to see who's lauched a ton of procs (most likely) and then issue a quick jailkill to get it back under control.

If it doesn't come back, or the messages indicate a fatal error, you will need to proceed with a power cycle (ctrl+alt+del will not work).

=== Power cycle the server ===
If this machine is not a Dell 2950 with a [[DRAC/RMM#DRAC|DRAC card]] (i.e. if you can’t ssh into the DRAC card (as root, using the standard root pass) and issue
racadm serveraction hardreset
then you will need someone at the data center power the macine off, wait 30 sec, then turn it back on. Make sure to re-attach via console:
tip jailX
immediately after power down.

=== (Re)attach to the console ===
Stay on the console the entire time during boot. As the BIOS posts- look out for the RAID card output- does everything look healthy? The output may be scrambled, look for "DEGRADED" or "FAILED". Once the OS starts booting you will be disconnected (dropped back to the shell on the console server) a couple times during the boot up. The reason you want to quickly re-attach is two-fold: 1. If you don’t reattach quickly then you won’t get any console output, 2. you want to be attached before the server ''potentially'' starts (an extensive) fsck. If you attach after the fsck begins, you’ll have seen no indication it started an fsck and the server will appear frozen during startup- no output, no response.

IMPORTANT NOTE: on some older FreeBSD systems, there will be no output to the video (KVM) console as it boots up. The console output is redirected to the serial port ... so if a jail crashes, and you attach a kvm, the output during the bootup procedure will not be shown on the screen. However, when the bootup is done, you will get a login prompt on the screen and will be able to log in as normal. <tt>/boot/loader.conf</tt> is where serial console redirect output lives, so comment that if you want to catch output on kvm.
On newer systems it sends most output to both locations.

=== Assess the heath of the server ===
Once the server boots up fully, you should be able to ssh in. Look around- make sure all the mounts are there and reporting the correct size/usage (i.e. /mnt/data1 /mnt/data2 /mnt/data3 - look in /etc/fstab to determine which mount points should be there), check to see if RAID mirrors are healthy. See [[RAID_Cards#Common_CLI_commands_.28megacli.29|megacli]], [[#aaccheck|aaccheck]]

Before you start the jails, you need to run [[#preboot|preboot]]. This will do some assurance checks to make sure things are prepped to start the jails. Any issues that come out of preboot need to be addressed before starting jails.

=== Start jails ===
[[#Starting_jails:_Quad.2FSafe_Files|More on starting jails]]
Customer jails (the VPSs) do not start up automatically at boot time. When a FreeBSD machines boots up, it boots up, and does nothing else. To start jails, we put the commands to start each jail into a shell script(s) and run the script(s). Jail startup is something that needs to be actively monitored, which is why we don’t just run the script automatically.

In order to start jails, we run the quad files: quad1 quad2 quad3 and quad4 (on new systems there is only quad1). If the machine was cleanly rebooted- which wouldn't be the case if this was a crash, you may run the safe files (safe1 safe2 safe3 safe4) in lieu of quads.

Open up 4 logins to the server (use the windows in [[Screen#Screen_Organization|a9]])
In each of the 4 windows you will:

If there is a [[#startalljails|startalljails]] script (and only quad1), run that command in each of the 4 windows. It will parse through the quad1 file and start each jail. Follow the instructions [[#Problems_with_the_quad.2Fsafe_files|here]] for monitoring startup. Note that you can be a little more lenient with jails that take awhile to start- startalljails will work around the slow jails and start the rest. As long as there aren't 4 jails which are "hung" during startup, the rest will get started eventually.
-or-
If there is no startalljails script, there will be multiple quad files. In each of the 4 windows, start each of the quads. i.e. start quad1 in window1, quad2 in window2 and so on. DO NOT start any quad twice. It will crash the server. If you accidentally do this, just jailkill all the jails which are in the quad and run the quad again. Follow the instructions here for monitoring quad startup.

Note the time the last jail boots- this is what you will enter in the crash log.

Save the crash log.

=== Check to make sure all jails have started ===
There's a simple script which will make sure all jails have started, and enter the ipfw counter rules: [[#postboot|postboot]]
Run postboot, which will do a jailps on each jail it finds (excluding commented out jails) in the quad file(s). We're looking for 2 things:
# systems spawning out of control or too many procs
# jails which haven't started
On 7.x and newer systems it will print out the problems (which jails haven't started) at the conclusion of postboot.
On older systems you will need to watch closely to see if/when there's a problem, namely:

[hostname] doesnt exist on this server

When you get this message, it means one of 2 things:
1. the jail really didn't start:
When a jail doesn't start it usually boils down to a problem in the quad file. Perhaps the path name is wrong (data1 vs data2) or the name of the vn/mdfile is wrong. Once this is corrected, you will need to run the commands from the quad file manually, or you may use <tt>startjail <hostname></tt>

2. the customer has changed their hostname (and not told us) so their jail ''is'' running, just under a different hostname:
On systems with jls, this is easy to rectify. First, get the customer info: <tt>g <hostname></tt>
Then look for the customer in jls: <tt>jls | grep <col0XXXX></tt>
From there you will see their new hostname- you should update that hostname in the quad file: don't forget to edit it on the <tt>## begin ##</tt> and <tt>## end ##</tt> lines, and in mgmt.
On older systems without jls, this will be harder, you will need to look further to see their hostname- perhaps its in their /etc/rc.conf

Once all jails are started, do some spot checks- try to ssh or browse to some customers, just to make sure things are really ok.

== Adding disk to a 7.x/8.x jail ==
or
== Moving customer to a different drive (md) ==

NOTE: this doesn’t apply to mx2 which uses gvinum. Use same procedure as 6.x
NOTE: if you unmount before mdconfig, re-mdconfig (attach) then unmount then mdconfig -u again

(parts to change/customize are <tt>red</tt>)

If someone wants more disk space, there’s a paste for it, send it to them (explains about downtime, etc).

1. Figure out the space avail from <tt>js</tt>. Ideally, you want to put the customers new space on a different partition (and create the new md on the new partition).

2. make a mental note of how much space they're currently using

3. <tt>jailkill <hostname></tt>

4. Umount it (including their devfs) but leave the md config’d (so if you use stopjail, you will have to re-mdconfig it)

5. <tt>g <customerID></tt> to get the info (IP/cust#) needed to feed the new mdfile and mount name, and to see the current md device.

6a. When there's enough room to place new system on an alternate, or the same drive:
USE CAUTION not to overwrite (touch, mdconfig) existing md!! 
<tt>touch /mnt/data3/69.55.234.66-col01334 
mdconfig -a -t vnode -s 10g -f /mnt/data3/69.55.234.66-col01334 -u 97 
newfs /dev/md97</tt>

Optional- if new space is on a different drive, move the mount point directory AND use that directory in the mount and cd commands below: 
<tt>mv /mnt/data1/69.55.234.66-col01334-DIR /mnt/data3/69.55.234.66-col01334-DIR</tt>

<tt>mount /dev/md97 /mnt/data3/69.55.234.66-col01334-DIR 
cd /mnt/data3/69.55.234.66-col01334-DIR</tt>

confirm you are mounted to /dev/md97 and space is correct: 
<tt>df .</tt>

do the dump and pipe directly to restore: 
<tt>dump -0a -f - /dev/md1 | restore -r -f - 
rm restoresymtable</tt>

when dump/restore completes successfully, use <tt>df</tt> to confirm the restored data size matches the original usage figure

md-unconfig old system: 
<tt>mdconfig -d -u 1</tt>

archive old mdfile. 
<tt>mv /mnt/data1/69.55.237.26-col00241 /mnt/data1/old-col00241-mdfile-noarchive-20091211</tt>

edit quad (vq1) to point to new (/mnt/data3) location AND new md number (md97)

restart the jail: 
<tt>startjail <hostname></tt>

6b. When there's not enough room on an alternate partition, or on the same drive...but there is enough room if you were to remove the existing customer's space:

mount backup nfs mounts: 
<tt>mbm</tt> 
(run <tt>df</tt> to confirm backup mounts are mounted)

dump the customer to backup2 or backup1 
<tt>dump -0a -f /backup4/col00241.20120329.noarchive.dump /dev/md1</tt> 
(when complete WITHOUT errors, <tt>du</tt> the dump file to confirm it matches size, roughly, with usage)

unconfigure and remove old mdfile 
<tt>mdconfig -d -u 1 
rm /mnt/data1/69.55.237.26-col00241</tt> 
(there should now be enough space to recreate your bigger system. If not, run sync a couple times)

create the new system (ok to reuse old mdfile and md#): 
<tt>touch /mnt/data1/69.55.234.66-col01334 
mdconfig -a -t vnode -s 10g -f /mnt/data1/69.55.234.66-col01334 -u 1 
newfs /dev/md1 
mount /dev/md1 /mnt/data1/69.55.234.66-col01334-DIR 
cd /mnt/data1/69.55.234.66-col01334-DIR</tt>

confirm you are mounted to /dev/md1 and space is correct: 
<tt>df .</tt>

do the restore from the dumpfile on the backup server: 
<tt>restore -r -f /backup4/col00241.20120329.noarchive.dump . 
rm restoresymtable</tt>

when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure

umount nfs: 
<tt>mbu</tt>

If md# changed (or mount point), edit quad (<tt>vq1</tt>) to point to new (/mnt/data3) location AND new md number (md1)

restart the jail: 
<tt>startjail <hostname></tt>

7. update disk (and dir if applicable) in mgmt screen

8. update backup list AND move backups, if applicatble

Ex: <tt>mvbackups 69.55.237.26-col00241 jail9 data3</tt>

9. Optional: archive old mdfile

<tt>mbm 
gzip -c old-col01588-mdfile-noarchive-20120329 > /deprecated/old-col01588-mdfile-noarchive-20120329.gz 
mbu 
rm old-col01588-mdfile-noarchive-20120329</tt>

== Adding disk to a 6.x jail (gvinum/gconcat) ==
or
== Moving customer to a different drive (gvinum/gconcat) ==

(parts to change are highlighted)

If someone wants more disk space, there’s a paste for it, send it to them (explains about downtime, etc).

1. Figure out the space avail from [[#js|js]]. Ideally, you want to put the customers new space on a different partition (and create the new md on the new partition).

2. make a mental note of how much space they're currently using

3. <tt>[[#stopjail|stopjail]] <hostname></tt>

4. <tt>[[#g|g]] <customerID></tt> to get the info (IP/cust#) needed to feed the new mount name and existing volume/device.

5a. When there's enough room to place new system on an alternate, or the same drive (using only UNUSED - including if it's in use by the system in question - gvinum volumes):

Configure the new device: 
A. for a 2G system (single gvinum volume): 
<tt>bsdlabel -r -w /dev/gvinum/v123 
newfs /dev/gvinum/v123a</tt> 

-or-

B. for a >2G system (create a gconcat volume): 
try to grab a contiguous block of gvinum volumes. gconcat volumes MAY NOT span drives (i.e. you cannot use a gvinum volume from data3 and a volume from data2 in the same gconcat volume).

<tt>gconcat label v82-v84 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84 
bsdlabel -r -w /dev/concat/v82-v84 
newfs /dev/concat/v82-v84a</tt>

Other valid gconcat examples: 
<tt>gconcat label v82-v84v109v112 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84 /dev/gvinum/v109 /dev/gvinum/v112 
gconcat label v82v83 /dev/gvinum/v82 /dev/gvinum/v83</tt> 
Note, long names will truncate: v144v145v148-v115 will truncate to v144v145v148-v1 (so you will refer to it as v144v145v148-v1 thereafter)

Optional- if new volume is on a different drive, move the mount point directory (get the drive from js output) AND use that directory in the mount and cd commands below: 
<tt>mv /mnt/data1/69.55.234.66-col01334-DIR /mnt/data3/69.55.234.66-col01334-DIR</tt>

confirm you are mounted to the device (<tt>/dev/gvinum/v123a</tt> OR <tt>/dev/concat/v82-v84</tt>) and space is correct: 
A. <tt>mount /dev/gvinum/v123a /mnt/data3/69.55.234.66-col01334-DIR</tt> 
-or- 
B. <tt>mount /dev/concat/v82-v84a /mnt/data3/69.55.234.66-col01334-DIR</tt>

<tt>cd /mnt/data3/69.55.234.66-col01334-DIR 
df .</tt>

do the dump and pipe directly to restore: 
<tt>dump -0a -f - /dev/gvinum/v1 | restore -r -f - 
rm restoresymtable</tt>

when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure

edit quad (<tt>vq1</tt>) to point to new (/mnt/data3) location AND new volume (<tt>/dev/gvinum/v123a</tt> or <tt>/dev/concat/v82-v84a</tt>) , run <tt>buildsafe</tt>

restart the jail: 
<tt>startjail <hostname></tt>

5b. When there's not enough room on an alternate partition, or on the same drive...but there is enough room if you were to remove the existing customer's space (i.e. if you want/need to reuse the existing gvinum volumes and add on more):

mount backup nfs mounts: 
<tt>mbm</tt> 
(run df to confirm backup mounts are mounted)

dump the customer to backup2 or backup1 
<tt>dump -0a -f /backup4/col00241.20120329.noarchive.dump /dev/gconcat/v106-v107</tt> 
(when complete WITHOUT errors, du the dump file to confirm it matches size, roughly, with usage)

unconfigure the old gconcat volume 
list member gvinum volumes:

<tt>gconcat list v106v107</tt>

Output will resemble: 
<pre>Geom name: v106v107
State: UP
Status: Total=2, Online=2
Type: AUTOMATIC
ID: 3530663882
Providers:
1. Name: concat/v106v107
Mediasize: 4294966272 (4.0G)
Sectorsize: 512
Mode: r1w1e2
Consumers:
1. Name: gvinum/sd/v106.p0.s0
Mediasize: 2147483648 (2.0G)
Sectorsize: 512
Mode: r1w1e3
Start: 0
End: 2147483136
2. Name: gvinum/sd/v107.p0.s0
Mediasize: 2147483648 (2.0G)
Sectorsize: 512
Mode: r1w1e3
Start: 2147483136
End: 4294966272</pre>

stop volume and clear members 
<tt>gconcat stop v106v107 
gconcat clear gvinum/sd/v106.p0.s0 gvinum/sd/v107.p0.s0</tt>

create new device- and its ok to reuse old/former members 
try to grab a contiguous block of gvinum volumes. gconcat volumes MAY NOT span drives (i.e. you cannot use a gvinum volume from data3 and a volume from data2 in the same gconcat volume). 
<tt>gconcat label v82-v84v106v107 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84 /dev/gvinum/v106 /dev/gvinum/v107 
bsdlabel -r -w /dev/concat/v82-v84v106v107 
newfs /dev/concat/v82-v84v106v107a</tt>

Optional- if new volume is on a different drive, move the mount point directory (get the drive from js output) AND use that directory in the mount and cd commands below: 
<tt>mv /mnt/data1/69.55.234.66-col01334-DIR /mnt/data3/69.55.234.66-col01334-DIR</tt>

confirm you are mounted to the device (/dev/concat/v82-v84v106v107) and space is correct: 
<tt>mount /dev/concat/v82-v84v106v107a /mnt/data3/69.55.234.66-col01334-DIR</tt>

<tt>cd /mnt/data3/69.55.234.66-col01334-DIR 
df .</tt>

do the restore from the dumpfile on the backup server: 
<tt>restore -r -f /backup4/col00241.20120329.noarchive.dump . 
rm restoresymtable</tt>

when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure

edit quad (<tt>vq1</tt>) to point to new (/mnt/data3) location AND new volume (<tt>/dev/concat/v82-v84v106v107a</tt>), run buildsafe

restart the jail: 
<tt>startjail <hostname></tt>

TODO: clean up/clear old gvin/gconcat vol

6. update disk (and dir if applicable) in mgmt screen

7. update backup list AND move backups, if applicatble

Ex: <tt>mvbackups 69.55.237.26-col00241 jail9 data3</tt>

DEPRECATED - steps to tack on a new gvin to existing gconcat- leads to corrupted fs
bsdlabel -e /dev/concat/v82-v84

To figure out new size of the c partition, multiply 4194304 by the # of 2G gvinum volumes and subtract the # of 2G volumes:
10G: 4194304 * 5 – 5 = 20971515
8G: 4194304 * 4 – 4 = 16777212
6G: 4194304 * 3 – 3 = 12582909
4G: 4194304 * 2 – 2 = 8388606

To figure out the new size of the a partition, subtract 16 from the c partition:
10G: 20971515 – 16 = 20971499
8G: 16777212 – 16 = 16777196
6G: 12582909 – 16 = 12582893
4G: 8388606 – 16 = 8388590

Orig:
8 partitions:
# size offset fstype [fsize bsize bps/cpg]
a: 8388590 16 4.2BSD 2048 16384 28552
c: 8388606 0 unused 0 0 # "raw" part, don't edit

New:
8 partitions:
# size offset fstype [fsize bsize bps/cpg]
a: 12582893 16 4.2BSD 2048 16384 28552
c: 12582909 0 unused 0 0 # "raw" part, don't edit

sync; sync

growfs /dev/concat/v82-v84a

fsck –fy /dev/concat/v82-v84a

sync

fsck –fy /dev/concat/v82-v84a

(keep running fsck’s till NO errors)

== Adding disk to a 4.x jail ==

NOTE: if you unmount before vnconfig, re-vnconfig then unmount then vnunconfig

If someone wants more disk space, there’s a paste for it, send it to them.

1. Figure out the space avail from [[#js|js]]. Ideally, you want to dump to a different partition and create the new vn on the current partition. If no space to make make dump, then move old vn’s to backup or worse, dump to backup.

2. <tt>g <customerID></tt> to get the info

3. make vnfile, newfs and mount. USE CAUTION not to overwrite existing vn. 
Ex: <tt>vnconfig –T -S 9g -s labels -c /dev/vn22 /mnt/data2/65.214.160.117-col00496 
disklabel -r -w vn30 auto 
newfs /dev/vn30c 
mkdir /mnt/data2/65.214.160.117-col00496-DIR 
mount /dev/vn30c /mnt/data2/69.55.237.26-col00241-DIR 
cd /mnt/data2/65.214.160.117-col00496-DIR</tt>

4. <tt>jailkill <hostname></tt>

5. <tt>dump -0a -f - /dev/vn1 | restore -r -f -</tt>

6. <tt>rm restoresymtable</tt>

7. unmount and unvnconfig old system: 
<tt>umount /dev/vnNNc 
vnconfig –u /dev/vnNN</tt>

8. move vnfile. Ex: <tt>mv /mnt/data1/69.55.237.26-col00241 /mnt/data1/old-col00241-vnfile-20110312-noarchive</tt>

9. remove old dir: <tt>rmdir /mnt/data1/69.55.237.26-col00241-DIR</tt>

10. edit quad/safe to point to new location, run <tt>[[#buildsafe|buildsafe]]</tt>

11. start jail: <tt>[[#startjail|startjail]] <hostname></tt>

12. update disk (and dir if applicable) in mgmt screen

13. mv backups if necessary

WARNING: if you restore to unmounted vn then you’re actually restoring to /mnt/data1
you can tar and do above to recover

NOTE: you can’t move a customer to a system that has a base sys that is diff from the one they came from (BSD 4.8 to 4.5 = won’t work)

NOTE: we don’t mount procfs's to anyone’s system by default.

NOTE: mount_nulls:
also goofy mount_nulls seen in df -k on jail2 
also needs to be done in safe and quad 
if he wants another, use safe/quad as example and add it to file. also exec command at prompt to add mount (no reboot, remount or rejail necessary)

== Moving customer to another jail machine ==

Systems should only be moved between similarly-versioned jails. If you cannot get a match, you should go to freebsd.org and see what the differences were between the version you're moving from and the new version. Many of the updates will be to drivers and kernels - these don't affect the customer. Perhaps they updated traceroute or a library...you should copy the new/changed files into the VPS/jail on the new host. An alternate method for moving to a different version would be to buildworld to bring their full distribution up to date. Optional steps for this kind of upgrade are included below.

1. <tt>g <customerID></tt>

2. <tt>[[#jailkill|jailkill]] <hostname></tt>

3. create new device on target system 
4.x: 
<pre>vnconfig –T –S 4g –s labels –c /dev/vn1 /mnt/data1/69.55.22x.x-col00XXX
disklabel –r –w vn1 auto
newfs /dev/vn1c</pre>

6.x: 
<pre>bsdlabel -r -w /dev/gvinum/v1
newfs /dev/gvinum/v1a
or
gconcat label v1-v3 /dev/gvinum/v1 /dev/gvinum/v2 /dev/gvinum/v3
bsdlabel -r -w /dev/concat/v1-v3
newfs /dev/concat/v1-v3a</pre>

7.x+: 
Run jailmakeempty then skip step 4 & 7 below

4. make and mount the dir on new system. 
Ex: <pre>mkdir /mnt/data2/69.55.230.3-col00123-DIR
chmod 755 /mnt/data2/69.55.230.3-col00123-DIR
mount [device] /mnt/data2/69.55.230.3-col00123-DIR</pre>

5. [[#stopjail|stopjail]] <hostname> 1

6. dump fs to new system. ex: <tt>[[#dumpremoterestore|dumpremoterestore]] /dev/vn51 10.1.4.118 /mnt/data2/69.55.239.45-col00688-DIR</tt> (make sure you can ssh as root on remote machine)

6a. OPTIONAL BUILDWORLD: 
cd /usr/src
make world DESTDIR=/mnt/data2/69.55.xxx.xx-col0xxxx-DIR

cd etc
make distribution DESTDIR=/mnt/data2/69.55.xxx.xx-col0xxxx-DIR

(you may have to rm an openssh file and re-make dist)

rm -rf /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc/periodic/daily/400.status-disks
vi /etc/periodic/security/100.chksetuid
replace: <tt>MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`</tt> 
with: <tt>MP='/' (use single quotes)</tt>

Ask user if they want ports overwritten with current, if yes:
cp -r /usr/ports /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/usr

on source:
cd /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc; vipw –d .
(copy in all info)

on target:
cd /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc; vipw –d .
(paste all info)

on source:
cat /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc/group
(copy in all info)

on target:
cat > /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc/group
(paste all info)

7. edit quad on source system copy over entries for the jail to the target system
take care that the vn/gvinum/gconcat/md devices on the target system are’t in use and that the /mnt/dataN path doesn’t need changing

8. run [[#buildsafe|buildsafe]] (if this is <=6.x) on target system (copies changes made to quad into safe file)

9. remove ip from source system. Ex: <tt>ipdel 69.55.230.3</tt>

10. add ip to target system. Ex: <tt>ipadd 69.55.230.3</tt>

11. start new system, with [[#startjail|startjail]] or manually, pasting entries found from running <tt>g <customerID></tt> on the new system

12. run <tt>[[#canceljail|canceljail]] col0xxxx</tt> on source system. This should prompt you to remove backups if any existed. Do not let it do this, rather:

13. if backups existed move them to the new host via the <tt>[[#mvbackups_.28freebsd.29|mvbackups]] script</tt>

14. edit quad on source system, edit comment to reflect a move rather than a cancel, ex: <tt># moved to jail2 col00241</tt>

15. edit mgmt to reflect new host and dir for new system

16. optional add ipfw rules on new system. Ex:
<tt>ipfw add 01231 count ip from 69.55.230.2 to any
ipfw add 01232 count ip from any to 69.55.230.2</tt>

== Increasing inodes for a VPS ==

when doing the newfs:
newfs -i 4096 ...

== Jail services unable to contact services within same jail ==

This is due to a messed up routing table (only seen it on jail2 before).

route delete 69.55.228.65/32 route add -net 69.55.228.65/32 -iface fxp0 -nostatic -cloning route add 69.55.228.65 -iface lo0

== /dev/null permission resets ==

Applies to 4.x jails only

For some reason that I do not understand at all, the /dev/null node in customer jails, after the system is restarted, often reverts permissions to 0600 ... which is bad, because non-root processes cannot redirect to /dev/null anymore. In fact, a fair number of server daemons will complain and/or fail if this is the case.

So, after a restart, after all systems have restarted, run postboot, which does something like:

for f in `df -k | grep /dev/vn | awk '{print $6}'` ; do chmod 0666 $f/dev/null ; done

Later, after the system is up, if anyone ever complains about their /dev/null permissions for any reason ... really if anyone mentions
/dev/null in any way, tell them to run:

chmod 0666 /dev/null

We have ultimately fixed this by adding chmods to the quad/safe's

== mknod in a jail ==

For a customer who wants to run a chroot name server (named) in a jail.

<pre>
cd /mnt/data1/<user directory>/var/named
mknod -c 1 1 null
mknod -c 2 3 random
</pre>

== Postfix Problems ==

This hasn't happened in a long time, probably no longer valid with modern OS's

Postfix is an alternate MTA - people replace sendmail with it in much the same way that many people also replace sendmail with qmail. The problem is that if you install postfix inside of a jail, by default it will not work properly - the user will see error messages like this:

incoming mail generates errors like this:

mail_queue_enter: create file incoming/298266.55902: File too large

and further, sending mail generates errors like this:

postdrop: warning: mail_queue_enter: create file maildrop/627930.56676: File too large

This is very easy to solve. In fact, the FreeBSD welcome email that is sent out by `jailmake` now contains this block of text:

- if you plan on installing postfix, email us first and ask for a necessary patch.

So, anytime you see a support email that says anything about postfix not working, paste that line from the welcome email into your response to them, and right away, forward the postfix patch to them in an email.

Here are the instructions:

Simply copy the patch into /usr/ports/mail/postfix/files, rename the file to ‘patch-file_limit.c’ and then cd to /usr/ports/mail/postfix and run `make install`.

The postfix patch is the very first email in the support email box, and we never delete or save it because we always want it there to forward to people. The instructions in the email are very clear, and it solves their problem 100% of the time.

== Problems with `find` ==

There was a bug in the FreeBSD vn-filesystem code. The bug causes the system to crash or hang when someone runs the `find` command inside their vn-backed filesystem (and even though the bug may have been fixed, due to the i/o overhead, we still don't like find).

Now, this does not always happen - however, any jail machine with more than 20 systems on it will crash every night if every system on it runs the daily periodic script out of their crontab, and updates their locate database with the find command.

(the locate database is a small, FreeBSD specific database that is populated nightly from a cron job, and is then used to provide fast answers to the `locate` command)

In normal systems (32-40 jails on them), all that has to be done is make sure nobody runs the daily periodic. This means that the file /etc/crontab, instead of looking like this:

<pre># do daily/weekly/monthly maintenance
1 3 * * * root periodic daily
15 4 * * 6 root periodic weekly
30 5 1 * * root periodic monthly</pre>

Needs to look like this:

<pre># DO NOT UNCOMMENT THESE - contact support@johncompanies.com for details
#1 3 * * * root periodic daily
#15 4 * * 6 root periodic weekly
#30 5 1 * * root periodic monthly
## DO NOT UNCOMMENT THESE ^^^^^^^^</pre>

and the problem will generally not occur at all. New filesystem images are always altered in this way, so all systems are set like this.

However, if you have enough systems running, even casual uses of find, outside of the periodic scripts, can cause the system to crash. We are seeing this on jail13, which crashes about once per week.

There is not yet any resolution to this problem.

So, if you have a freebsd system crash, see if anyone has uncommented their daily periodic line - you can see them for everyone by running:

/bin/sh

and then running:

<pre>for f in `df -k | grep vn | awk '{print $6}'` ; do echo `cat $f/etc/crontab | grep "periodic daily"` $f ; done | more</pre>

(all one line, one command)

And then edit those /etc/crontabs and comment them out again.

== Problems un-mounting - and with mount_null’s ==

If you cannot unmount a filesystem, beacuse it says the filesystem is busy, it is because of three things:

a) the jail is still running

b) you are actually in that directory, even though the jail is stopped

c) there are still dev, null_mount or linprocfs mount points mounted inside that directory.

d) when trying to umount null_mounts that are really long and you get an error like “No such file or directory”, it’s an OS bug where the dir is truncated. No known fix

e) there are still files open somewhere inside the dir. Use <tt>fstat | grep <cid></tt> to find the process that has files open

f) Starting with 6.x, the jail mechanism does a poor job of keeping track of processes running in a jail and if it thinks there are still procs running, it will refuse to umount the disk. If this is happening you should see a low number in the #REF column when you run jls. In this case you ''can'' safely <tt>umount –f</tt> the mount.

Please note -if you forcibly unmount a (4.x) filesystem that has null_mounts
still mounted in it, the system '''will crash''' within 10-15 mins.

== Misc jail Items ==

We are overselling hard drive space on jail2, jail8, jail9, a couple jails on jail17, jail4, jail12 and jail18.
Even though the vn file shows 4G size, it doesn’t actually occupy that amount of space on the disk. So be careful not to fill up drives where we’re overselling – use oversellcheck to confirm you’re not oversold by more than 10G.
There are other truncated jails, they are generally noted in a the file on the root system: /root/truncated

The act of moving a truncated vn to another system un-does the truncating- the truncated vn is filled with 0’s and it occupies physical disk space for which it’s configured. So, you should use dumpremote to preserve the truncation.

* if you are getting disk full messages for a BSD customer, it's fairly safe to clear out their /usr/ports/distfiles dir
* 4.x: ps and top an only be run by root in these jails. Done on purpose:
As for `ps` and `top` - non-root users can run them, just not successfully because we have locked the permissions on /dev/mem and /dev/kmem to be root-readable only. That is why non-root users cannot successfully run `ps` and `top`.
* user quotas do not work on freebsd jails - you cannot set up quotas at all, and that's that.
* You cannot inject a process into a 4.x jail, only HUP running processes. 6.x and onward you can with jexec
* jails see base machine’s uptime/load when running top/w
* if someone is unable to get in- cant ping, etc- see if they were blocked by castle (DoS), see if their ip is on the system (post reboot it was lost cause wasn’t in the rc.conf). preboot should catch that.
* in FreeBSD you cant su to root unless you belong to wheel group – so if you remove your acct and setup a new one, we have to add it to wheel group (add to /etc/group)
* Dmesg from underlying sys is seen in customers dmesg on jail
* Popper process going crazy @ 40% for 10min = someone who leaves mail on server
* Don’t force umounts on 4.x jails – it crashes the machine, generally ok on newer machines (running md)
* Good book for admin http://search.barnesandnoble.com/booksearch/isbnInquiry.asp?userid=t824VyRAYz&isbn=0596005164&itm=2
* Self-sign ssl cert http://httpd.apache.org/docs/2.0/ssl/ssl_faq.html#selfcert
* conversation with Glenn about semaphores and pgsql
<pre>SDBoody: hey, these are valid amounts/figures/increments right:
kern.ipc.semmni=1280
kern.ipc.semmns=1280

gr8feen: probably... I always forget exactly what those are, so I usually have to look them up
SDBoody: semaphores
SDBoody: i took the current 1024 and added 256 to them
SDBoody: need more for pgsql
gr8feen: I meant the mni and mns parts...
gr8feen: some of those are not ones you just want to add to... hang on a sec and I'll look them up..
gr8feen: what's semmsl set to?
SDBoody: kern.ipc.semmsl: 1024

SDBoody: kern.ipc.msgseg: 2048
kern.ipc.msgssz: 8
kern.ipc.msgtql: 40
kern.ipc.msgmnb: 2048
kern.ipc.msgmni: 40
kern.ipc.msgmax: 16384
kern.ipc.semaem: 16384
kern.ipc.semvmx: 65534
kern.ipc.semusz: 152
kern.ipc.semume: 10
kern.ipc.semopm: 100
kern.ipc.semmsl: 1024
kern.ipc.semmnu: 512
kern.ipc.semmns: 1024
kern.ipc.semmni: 1024
kern.ipc.semmap: 768
kern.ipc.shm_allow_removed: 0
kern.ipc.shm_use_phys: 1
kern.ipc.shmall: 262144
kern.ipc.shmseg: 256
kern.ipc.shmmni: 784
kern.ipc.shmmin: 1
kern.ipc.shmmax: 536870912
kern.ipc.maxsockets: 25600

gr8feen: ok...msl is max per id, mni is max ids, mns is max number of semaphores... so you probably want something like mns = mni * msl
gr8feen: which one did you run out of?
SDBoody: not sure how to tell- ipcs shows the sems in use add up to 1024
SDBoody: there are 59 entries
gr8feen: I'm assuming you tried to start postgres and it failed?
SDBoody: yes
gr8feen: it should have logged why, somewhere..
gr8feen: if I recall, it'll tell you which one it ran out of
SDBoody: > DETAIL: Failed system call was semget(1, 17, 03600).

gr8feen: so it wanted an id with 17 semaphores...I'd start by making mns = 17*mni and leave mni and mnl set to what they are now and see what it does
SDBoody: i think mni is plenty high
SDBoody: ok, more reasonable 17408
gr8feen: yeah...just change that one and see how it goes..
SDBoody: and leave mni alone at 1024?
gr8feen: yeah...mni id the max number of ids...but if it's trying to get something like 17 per id, your going to hit mns before you hit anything else
SDBoody: right, but doesn't hurt to have it that high (assuming)

gr8feen: not really... I think those get allocated out of ram that you cant page out, but it's still such a small amount that it really doesn't matter
SDBoody: looks like that worked, thx!
gr8feen: cool

see semaphores: ipcs -a -s</pre>

= FreeBSD VPS Management Tools =

These files are located in /usr/local/jail/rc.d and /usr/local/jail/bin

== jailmake ==

Applies to 7.x+
On older systems syntax differs, run jailmake once to see.

Note: this procedure differs on mx2 which is 7.x but still uses gvinum

# run js to figure out which md’s are in use, which disk has enough space, IP to put it on
# use col00xxx for both hostnames if they don’t give you a hostname
# copy over dir, ip and password to pending customer screen

<tt>Usage: jailmake IP[,IP] CID disk[1|2|3] md# hostname shorthost ipfw# email [size in GB]</tt>

Ex:

Jail2# jailmake 69.55.234.66 col01334 3 97 vps.bsd.it vps 1334 fb@bsd.it

== jailps ==
jailps [hostname]
DEPRECATED FOR jps: displays processes belonging to/running inside a jail. The command
takes one (optional) argument – the hostname of the jail you wish to query. If you don’t
supply an argument, all processes on the machine are listed and grouped by jail.

== jps ==
jps [hostname]
displays processes belonging to/running inside a jail. The command
takes one (optional) argument – the hostname or ID of the jail you wish to query.

== jailkill ==
jailkill <hostname>
stops all process running in a jail.

You can also run:
jailkill <JID>

=== problems ===
Occasionally you will hit an issue where jail will not kill off:

<pre>jail9# jailkill www.domain.com
www.domain.com .. killed: none
jail9#</pre>
Because no processes are running under that hostname. You cannot use jailps.pl either:

<pre>jail9# jailps www.domain.com
www.domain.com doesn’t exist on this server
jail9#</pre>

The reasons for this are usually:
* the jail is no longer running

* the jail's hostname has changed
In this case,

>=6.x: run a <tt>jls|grep <jail's IP></tt> to find the correct hostname, then update the quad file, then kill the jail.

<6.x: the first step is to cat their /etc/rc.conf file to see if you can tell what they set the new hostname to. This very often works. For example:

cat /mnt/data2/198.78.65.136-col00261-DIR/etc/rc.conf

But maybe they set the hostname with the hostname command, and the original hostname is still in /etc/rc.conf.

The welcome email clearly states that they should tell us if they change their hostname, so there is no problem in just emailing them and asking them what they set the new hostname to.

Once you know the new hostname OR if a customer simply emails to inform you that they have set the hostname to something different, you need to edit the quad and safe files that their system is in to input the new hostname.

However, if push comes to shove and you cannot find out the hostname from them or from their system, then you need to start doing some detective work.

The easiest thing to do is run jailps looking for a hostname similar to their original hostname. Or you could get into the /bin/sh shell by running:

/bin/sh

and then looking at every hostname of every process:

for f in `ls /proc` ; do cat /proc/$f/status ; done

and scanning for a hostname that is either similar to their original hostname, or that you don't see in any of the quad safe files.

This is very brute force though, and it is possible that catting every file in /proc is dangerous - I don't recommend it. A better thing would be to identify any processes that you know belong to this system – perhaps the reason you are trying to find this system is because they are running something bad - and just catting the status from only that PID.

Somewhere there’s a jail where there may be 2 systems named www. Look at /etc/rc.conf and make sure they’re both really www. If they are, jailkill www, jailps www to make sure not running. Then immediately restart the other one, as the fqdn (as found from a rev nslookup)

* on >=6.x the hostname may not yet be hashed:
<pre>jail9 /# jls
JID Hostname Path IP Address(es)
1 bitnet.dgate.org /mnt/data1/69.55.232.50-col02094-DIR 69.55.232.50
2 ns3.hctc.net /mnt/data1/69.55.234.52-col01925-DIR 69.55.234.52
3 bsd1 /mnt/data1/69.55.232.44-col00155-DIR 69.55.232.44
4 let2.bbag.org /mnt/data1/69.55.230.92-col00202-DIR 69.55.230.92
5 post.org /mnt/data2/69.55.232.51-col02095-DIR 69.55.232.51 ...
6 ns2 /mnt/data1/69.55.232.47-col01506-DIR 69.55.232.47 ...
7 arlen.server.net /mnt/data1/69.55.232.52-col01171-DIR 69.55.232.52
8 deskfood.com /mnt/data1/69.55.232.71-col00419-DIR 69.55.232.71
9 mirage.confluentforms.com /mnt/data1/69.55.232.54-col02105-DIR 69.55.232.54 ...
10 beachmember.com /mnt/data1/69.55.232.59-col02107-DIR 69.55.232.59
11 www.agottem.com /mnt/data1/69.55.232.60-col02109-DIR 69.55.232.60
12 sdhobbit.myglance.org /mnt/data1/69.55.236.82-col01708-DIR 69.55.236.82
13 ns1.jnielsen.net /mnt/data1/69.55.234.48-col00204-DIR 69.55.234.48 ...
14 ymt.rollingegg.net /mnt/data2/69.55.236.71-col01678-DIR 69.55.236.71
15 verse.unixlore.net /mnt/data1/69.55.232.58-col02131-DIR 69.55.232.58
16 smcc-mail.org /mnt/data2/69.55.232.68-col02144-DIR 69.55.232.68
17 kasoutsuki.w4jdh.net /mnt/data2/69.55.232.46-col02147-DIR 69.55.232.46
18 dili.thium.net /mnt/data2/69.55.232.80-col01901-DIR 69.55.232.80
20 www.tekmarsis.com /mnt/data2/69.55.232.66-col02155-DIR 69.55.232.66
21 vps.yoxel.net /mnt/data2/69.55.236.67-col01673-DIR 69.55.236.67
22 smitty.twitalertz.com /mnt/data2/69.55.232.84-col02153-DIR 69.55.232.84
23 deliver4.klatha.com /mnt/data2/69.55.232.67-col02160-DIR 69.55.232.67
24 nideffer.com /mnt/data2/69.55.232.65-col00412-DIR 69.55.232.65
25 usa.hanyuan.com /mnt/data2/69.55.232.57-col02163-DIR 69.55.232.57
26 daifuku.ppbh.com /mnt/data2/69.55.236.91-col01720-DIR 69.55.236.91
27 collins.greencape.net /mnt/data2/69.55.232.83-col01294-DIR 69.55.232.83
28 ragebox.com /mnt/data2/69.55.230.104-col01278-DIR 69.55.230.104
29 outside.mt.net /mnt/data2/69.55.232.72-col02166-DIR 69.55.232.72
30 vps.payneful.ca /mnt/data2/69.55.234.98-col01999-DIR 69.55.234.98
31 higgins /mnt/data2/69.55.232.87-col02165-DIR 69.55.232.87 ...
32 ozymandius /mnt/data2/69.55.228.96-col01233-DIR 69.55.228.96
33 trusted.realtors.org /mnt/data2/69.55.238.72-col02170-DIR 69.55.238.72
34 jc1.flanderous.com /mnt/data2/69.55.239.22-col01504-DIR 69.55.239.22
36 guppylog.com /mnt/data2/69.55.238.73-col00036-DIR 69.55.238.73
40 haliohost.com /mnt/data2/69.55.234.41-col01916-DIR 69.55.234.41 ...
41 satyr.jorge.cc /mnt/data1/69.55.232.70-col01963-DIR 69.55.232.70
jail9 /# jailkill satyr.jorge.cc
ERROR: jail_: jail "satyr,jorge,cc" not found
</pre>

Note how it's saying <tt>satyr,jorge,cc</tt> is not found, and not <tt>satyr.jorge.cc</tt>.

The jail subsystem tracks things using comma-delimited hostnames. That is created every few hours:

jail9 /# crontab -l
0 0,6,12,18 * * * /usr/local/jail/bin/sync_jail_names

So if we run this manually:
jail9 /# /usr/local/jail/bin/sync_jail_names

Then kill the jail:
jail9 /# jailkill satyr.jorge.cc
successfully killed: satyr,jorge,cc</pre>

It worked.

If you ever see this when trying to kill a jail:

<pre># jailkill e-scribe.com
killing JID: 6 hostname: e-scribe.com
3 procs running
3 procs running
3 procs running
3 procs running
...</pre>

<tt>[[#jailkill|jailkill]]</tt> probably got lost trying to kill off the jail. Just ctrl-c the jailkill process, then run a jailps on the hostname, and <tt>kill -9</tt> any process which is still running. Keep running jailps and <tt>kill -9</tt> till all processes are gone.

== jailpsall ==
jailpsall
will run a jailps on all jails configured in the quad files (this is different from
jailps with no arguments as it won’t help you find a “hidden” system)

== jailpsw ==
jailpsw
will run a jailps with an extra -w to provide wider output

== jt (>=7.x) ==
jt
displays the top 20 processes on the server (the top 20 processes from top) and
which jail owns them. This is very helpful for determining who is doing what when
the server is very busy.

== jtop (>=7.x) ==
jtop
a wrapper for top displaying processes on the server and which jail owns them. Constantly updates, like top.

== jtop (<7.x) ==
jtop
displays the top 20 processes on the server (the top 20 processes from top) and
which jail owns them. This is very helpful for determining who is doing what when
the server is very busy.

== stopjail ==
stopjail <hostname> [1]
this will jailkill, umount and vnconfig –u a jail. If passed an optional 2nd
argument, it will not exit before umounting and un-vnconfig’ing in the event
jailkill returns no processes killed. This is useful if you just want to umount
and vnconfig –u a jail you’ve already killed. It is intelligent in that it won’t
try to umount or vnconfig –u if it’s not necessary.

== startjail ==
startjail <hostname>
this will start vnconfig, mount (including linprocfs and null-mounts), and start a jail.
Essentially, it reads the jail’s relevant block from the right quad file and executes it.
It is intelligent in that it won’t try to mount or vnconfig if it’s not necessary.

== jpid ==
jpid <pid>
displays information about a process – including which jail owns it.
It’s the equivalent of running cat /proc/<pid>/status

== canceljail ==
canceljail <hostname> [1]
this will stop a jail (the equivalent of stopjail), check for backups (offer to remove them
from the backup server and the backup.config), rename the vnfile, remove the dir, and
edit quad/safe. If passed an optional 2nd argument, it will not exit upon failing to kill
and processes owned by the jail. This is useful if you just want to cancel a jail which
is already stopped.

== jls ==
jls [-v]
Lists all jails running:
<pre>JID #REF IP Address Hostname Path
101 135 69.55.224.148 mail.pc9.org /mnt/data2/69.55.224.148-col01034-DIR</pre>

#REF is the number of references or procs(?) running

Running with -v will give you all IPs assigned to each jail (7.2 up)
<pre>JID #REF Hostname Path IP Address(es)
101 139 mail.pc9.org /mnt/data2/69.55.224.148-col01034-DIR 69.55.224.14869.55.234.85</pre>

== startalljails ==
startalljails
7.2+ only. This will parse through quad1 and start all jails. It utilizes lockfiles so it won’t try to start a jail more than once- therefore multiple instances can be running in parallel without fear of starting a jail twice. If a jail startup gets stuck, you can ^C without fear of killing the script. IMPORTANT- before running startalljails you should make sure you ran preboot once as it will clear out all the lockfiles and enable startalljails to work properly.

== aaccheck.sh ==
aaccheck.sh
displayes the output of container list and task list from aaccli

== backup ==
backup
backup script called nightly to update jail scripts and do customer backups

== buildsafe ==
buildsafe
creates safe files based on quads (automatically removing the fsck’s). This will destructively overwrite safe files

== checkload.pl ==
checkload.pl
this was intended to be setup as a cronjob to watch processes on a jail when the load
rises above a certain level. Not currently in use.

== checkprio.pl ==
checkprio.pl
will look for any process (other than the current shell’s csh, sh, sshd procs) with a non-normal priority and normalize it

== diskusagemon ==
diskusagemon <mount point> <1k blocks>
watches a mount point’s disk use, when it reaches the level specified in the 2nd argument,
it exits. This is useful when doing a restore and you want to be paged as it’s nearing completion.
Best used as: <tt>diskusagemon /asd/asd 1234; pagexxx</tt>

== dumprestore ==
dumprestore <dumpfile>
this is a perl expect script which automatically enters ‘1’ and ‘y’. It seems to cause restore to fail
to set owner permissions on large restores.

== g ==
g <search>
greps the quad/safe files for the given search parameter

== gather.pl ==
gather.pl
gathers up data about jails configured and writes to a file. Used for audits against the db

== gb ==
gb <search>
greps backup.config for the given search parameter

== gbg ==
gbg <search>
greps backup.config for the given search parameter and presents just the directories (for clean pasting)

== ipfwbackup ==
ipfwbackup
writes ipfw traffic count data to a logfile

== ipfwreset ==
ipfwreset
writes ipfw traffic count data to a logfile and resets counters to 0

== js ==
js
output varies by OS version, but generally provides information about the base jail:
- which vn’s are in use
- disk usage
- info about the contents of quads
- the # of inodes represented by the jails contained in the group (133.2 in the example below), and how many jails per data mount, as well as subtotals
- ips bound to the base machine but not in use by a jail
- free gvinum volumes, or unused vn’s or used md’s

<pre>/usr/local/jail/rc.d/quad1:
/mnt/data1 133.2 (1)
/mnt/data2 1040.5 (7)
total 1173.7 (8)
/usr/local/jail/rc.d/quad2:
/mnt/data1 983.4 (6)
total 983.4 (6)
/usr/local/jail/rc.d/quad3:
/mnt/data1 693.4 (4)
/mnt/data2 371.6 (3)
total 1065 (7)
/usr/local/jail/rc.d/quad4:
/mnt/data1 466.6 (3)
/mnt/data2 882.2 (5)
total 1348.8 (8)
/mnt/data1: 2276.6 (14)
/mnt/data2: 2294.3 (15)

Available IPs:
69.55.230.11 69.55.230.13 69.55.228.200

Available volumes:
v78 /mnt/data2 2G
v79 /mnt/data2 2G
v80 /mnt/data2 2G</pre>

== load.pl ==
load.pl
feeds info to load mrtg - executed by inetd.

== makevirginjail ==
makevirginjail
Only on some systems, makes an empty jail (doesn't do restore step)

== mb ==
mb <mount|umount>
(nfs) mounts and umounts dirs to backup2. Shortcuts are mbm and mbu to mount and unmount.

== notify.sh ==
notify.sh
emails reboot@johncompanies.com – intended to be called at boot time to alert us to a machine which panics and reboots and isn’t caught by bb or castle.

== orphanedbackupwatch ==
orphanedbackupwatch
looks for directories on backup2 which aren’t configured in backup.config and offers to delete them

== postboot ==
postboot
to be run after a machine reboot and quad/safe’s are done executing. It will:
* do chmod 666 on each jail’s /dev/null
* add ipfw counts
* run jailpsall (so you can see if a configured jail isn’t running)

== preboot ==
preboot
to be run before running quad/safe – checks for misconfigurations:
* a jail configured in a quad but not a safe
* a jail is listed more than once in a quad
* the ip assigned to a jail isn’t configured on the machine
* alias numbering skips in the rc.conf (resulting in the above)
* orphaned vnfile's that aren't mentioned in a quad/safe
* ip mismatches between dir/vnfile name and the jail’s ip
* dir/vnfiles's in quad/safe that don’t exist

== quadanalyze.pl ==
quadanalyze.pl
called by js, produces the info (seen above with js explanation) about the contents of quad (inode count, # of jails, etc.)

== rsync.backup ==
rsync.backup
does customer backups (relies on backup.config)

== taskdone ==
taskdone
when called will email support@johncompanies.com with the hostname of the machine from which it was executed as the subject

== topten ==
topten
summarizes the top 10 traffic users (called by ipfwreset)

== trafficgather.pl ==
trafficgather.pl [yy-mm]
sends a traffic usage summary by jail to support@johncomapnies.com and payments@johncompanies.com. Optional arguments are year and month (must be in the past). If not passed, assumes last month. Relies on traffic logs created by ipfwreset and ipfwbackup

== trafficwatch.pl ==
trafficwatch.pl
checks traffic usage and emails support@johncomapnies.com when a jail reaches the warning level (35G) and the limit (40G). We really aren’t using this anymore now that we have netflow.

== trafstats ==
trafstats
writes ipfw traffic usage info by jail to a file called jc_traffic_dump in each jail’s / dir

== truncate_jailmake ==
truncate_jailmake
a version of jailmake which creates truncated vnfiles.

== vb ==
vb
the equivalent of: <tt>vi /usr/local/jail/bin/backup.config</tt>

== vs (freebsd) ==
vs<n>
the equivalent of: <tt>vi /usr/local/jail/rc.d/safe<n></tt>

vq<n>
the equivalent of: vi /usr/local/jail/rc.d/quad<n>

== dumpremote ==
dumpremote <user@machine> </remote/location/file-dump> <vnX>
ex: dumpremote user@10.1.4.117 /mnt/data3/remote.echoditto.com-dump 7
this will dump a vn filesystem to a remote machine and location

== oversellcheck ==
oversellcheck
displays how much a disk is oversold or undersold taking into account truncated vn files. Only for use on 4.x systems

== mvbackups (freebsd) ==
mvbackups <dir> (1.1.1.1-col00001-DIR) <target_machine> (jail1) <target_dir> (data1)
moves backups from one location to another on the backup server, and provides you with option to remove entries from current backup.config, and simple paste command to add the config to backup.config on the target server

== jailnice ==
jailnice <hostname>
applies <tt>renice 19 [PID]</tt> and <tt>rtprio 31 –[PID]</tt> to each process in the given jail

== dumpremoterestore ==
dumpremoterestore <device> <ip of target machine> <dir on target machine>
ex: dumpremoterestore /dev/vn51 10.1.4.118 /mnt/data2/69.55.239.45-col00688-DIR
dumps a device and restores it to a directory on a remote machine. Requires that you enable root ssh on the
remote machine.

== psj ==
psj
shows just the procs running on the base system – a ps auxw but without jail’d procs present

== perc5iraidchk ==
perc5iraidchk
checks for degraded arrays on Dell 2950 systems with Perc5/6 controllers

== perc4eraidchk ==
perc4eraidchk
checks for degraded arrays on Dell 2850 systems with Perc4e/Di controllers

= Virtuozzo VPS =

== Making new customer VE (vm) ==

This applies only to new virts >= 4.x

grab ip from ipmap (if opened from the pending cust screen it should take you to the right block). You can also run vzlist -a to see what block is in use, generally. Try to find an IP that's in the same block of class C IP's already on the box.

1. confirm ip you want to use isn’t in use via Mgmt. -> IP Map in management screens
also, ping the IP address.

2. put CT on whichever partition has more space
df -h

3. vm CID IP hostname /vz[1|2] email[,email] template ( <LB|LBP|LS|LM|LMP|LP> [disk] | <gb_disk/mb_ram/gb_burst> )
vm col00009 69.55.230.238 centos.testdave.com /vz1 dsmith@johncompanies.com centos-6-x86_64 LM

4. copy veid, dir, ip and password to pending customer screen. activate customer

== Making new customer VE (vemakexxx) ==

This applies to older virts with old templates. This should probably not be used at all anymore.

1. look thru hist for ip

2. confirm ip you want to use isn’t in use via Mgmt. -> IP Map in management screens

3. put ve on whichever partition has more space
vemakerh9 <veid> <ip> <hostname> <mount> <email> [gb disk]; <256|384|512> <veid>
vemakerh9 866 69.55.226.109 ngentu.com /vz1 ayo@ngantu.com,asd@asd.com 5; 256 866

4. copy (veid), dir, and ip to pending customer screen (pass set to p455agfa)

Note: We use VEID (Virtual Environment ID) and CTID (Container ID) interchangably. Similarly, VE and CT. They mean the same thing.
VZPP = VirtuoZzo Power Panel (the control panel for each CT)

All linux systems exist in /vz, /vz1 or /vz2 - since each linux machine holds roughly 60-90 customers, there will be roughly 30-45 in each partition.

The actual filesystem of the system in question is in:

/vz(1-2)/private/(VEID)

Where VEID is the identifier for that system - an all-numeric string larger than 100.

The actual mounted and running systems are in the corresponding:

/vz(1-2)/root/(VEID)

But we rarely interact with any system from this mount point.

You should never need to touch the root portion of their system – however you can traverse their filesystem by going to <tt>/vz(1-2)/private/(VEID)/root</tt> (<tt>/vz(1-2)/private/(VEID)/fs/root</tt> on 4.x systems) the root of their filesystem is in that directory, and their entire system is underneath that.

Every VE has a startup script in <tt>/etc/sysconfig/vz-scripts</tt> (which is symlinked as <tt>/vzconf</tt> on all systems) - the VE startup script is simply named <tt>(VEID).conf</tt> - it contains all the system parameters for that VE:

<pre># Configuration file generated by vzsplit for 60 VE
# on HN with total amount of physical mem 2011 Mb

VERSION="2"
CLASSID="2"

ONBOOT="yes"

KMEMSIZE="8100000:8200000"
LOCKEDPAGES="322:322"
PRIVVMPAGES="610000:615000"
SHMPAGES="33000:34500"
NUMPROC="410:415"
PHYSPAGES="0:2147483647"
VMGUARPAGES="13019:2147483647"
OOMGUARPAGES="13019:2147483647"
NUMTCPSOCK="1210:1215"
NUMFLOCK="107:117"
NUMPTY="19:19"
NUMSIGINFO="274:274"
TCPSNDBUF="1800000:1900000"
TCPRCVBUF="1800000:1900000"
OTHERSOCKBUF="900000:950000"
DGRAMRCVBUF="200000:200000"
NUMOTHERSOCK="650:660"
DCACHE="786432:818029"
NUMFILE="7500:7600"
AVNUMPROC="51:51"
IPTENTRIES="155:155"
DISKSPACE="4194304:4613734"
DISKINODES="400000:420000"
CPUUNITS="1412"
QUOTAUGIDLIMIT="2000"
VE_ROOT="/vz1/root/636"
VE_PRIVATE="/vz1/private/636"
NAMESERVER="69.55.225.225 69.55.230.3"
OSTEMPLATE="vzredhat-7.3/20030305"
VE_TYPE="regular"
IP_ADDRESS="69.55.225.229"
HOSTNAME="textengine.net"</pre>

As you can see, the hostname is set here, the disk space is set here, the number of inodes, the number of files that can be open, the number of tcp sockets, etc. - all are set here.

In fact, everything that can be set on this customer system is set in this conf file.

All interaction with the customer system is done with the VEID. You start the system by running:

vzctl start 999

You stop it by running:

vzctl stop 999

You execute commands in it by running:

vzctl exec 999 df -k

You enter into it, via a root-shell backdoor with:

vzctl enter 999

and you set parameters for the system, while it is still running, with:

vzctl set 999 --diskspace 6100000:6200000 --save

<tt>vzctl</tt> is the most commonly used command - we have aliased <tt>v</tt> to <tt>vzctl</tt> since we use it so often. We’ll continue to use <tt>vzctl</tt> in our examples, but feel free to use just <tt>v</tt>.

Let's say the user wants more diskspace. You can cat their conf file and see:

DISKSPACE="4194304:4613734"

So right now they have 4gigs of space. You can then change it to 6 with:

vzctl set 999 --diskspace 6100000:6200000 --save

IMPORTANT: all issuances of the vzctl set command need to end with <tt>–save</tt> - if they don't, the setting will be set, but it will not be saved to the conf file, and they will not have those settings next time they boot.

All of the tunables in the conf file can be set with the vzctl set command. Note that in the conf file, and on the vzctl set command line, we always issue two numbers seperated by a colon - that is because we are setting the hard and soft limits. Always set the hard limit slightly above the soft limit, as you see it is in the conf file for all those settings.

There are also things you can set with `<tt>vzctl set</tt>` that are not in the conf file as settings, per se. For instance, you can add IPs:

vzctl set 999 --ipadd 10.10.10.10 --save

or multiple IPs:

vzctl set 999 --ipadd 10.10.10.10 --ipadd 10.10.20.30 --save

or change the hostname:

vzctl set 999 --hostname www.example.com --save

You can even set the nameservers:

vzctl set 999 --nameserver 198.78.66.4 --nameserver 198.78.70.180 --save

Although you probably will never do that.

You can disable a VPS from being started (by VZPP or reboot) (4.x):

vzctl set 999 --disabled yes --save

You can disable a VPS from being started (by VZPP or reboot) (<=3.x):

vzctl set 999 --onboot=no --save

You can disable a VPS from using his control panel:

vzctl set 999 --offline_management=no --save

You can suspend a VPS, so it can be resumed in the same state it was in when it was stopped (4.x):

vzctl suspend 999

and to resume it:

vzctl resume 999

to see who owns process:
vzpid <PID>

to mount up an unmounted ve:
vzctl mount 827

To see network stats for CT's:
<pre># vznetstat
VEID Net.Class Output(bytes) Input(bytes)
-----------------------------------------------
24218 1 484M 39M
24245 1 463M 143M
2451 1 2224M 265M
2454 1 2616M 385M
4149 1 125M 68M
418 1 1560M 34M
472 1 1219M 315M
726 1 628M 317M
763 1 223M 82M
771 1 4234M 437M
-----------------------------------------------
Total: 13780M 2090M</pre>

One thing that sometimes comes up on older systems that we created with smaller defaults is that the system would run out of inodes. The user will email and say they cannot create any more files or grow any files larger, but they will also say that they are not out of diskspace ... they are running:

df -k

and seeing how much space is free - and they are not out of space. They are most likely out of inodes - which they would see by running:

df -i

So, the first thing you should do is enter their system with:

vzctl enter 999

and run: <tt>df -i</tt>

to confirm your theory. Then exit their system. Then simply cat their conf file and see what their inodes are set to (probably 200000:200000, since that was the old default on the older systems) and run:

vzctl set 999 --diskinodes 400000:400000 --save

If they are not out of inodes, then a good possibility is that they have maxed out their numfile configuration variable, which controls how many files they can have in their system. The current default is 7500 (which nobody has ever hit), but the old default was as low as 2000, so you would run something like:

vzctl set 999 --numfile 7500:7500 --save

----

You cannot start or stop a VE if your pwd is its private (/vz/private/999) or root (/vz/root/999) directories, or anywhere below them.

== Recovering from a crash (linux) ==

=== Diagnose whether you have a crash ===
The most important thing is to get the machine and all ve’s back up as soon as possible. Note the time, you’ll need to create a crash log entry (Mgmt. -> Reference -> CrashLog). The first thing to do is head over to the [[Screen#Screen_Organization|serial console screen]] and see if there’s any kernel error messages output. Try to copy any messages (or just a sample of repeating messages) you see into the notes section of the crash log – these will also likely need to be sent to virtuozzo for interpretation. If the messages are spewing too fast, hit ^O + H to start a screen log dump which you can ob1182.pts-38.bb serve after the machine is rebooted. Additionally, if the machine is responsive, you can get a trace to send to virtuozzo by hooking up a kvm and entering these 3 sequences:
<pre>alt+print screen+m
alt+print screen+p
alt+print screen+t</pre>

If there are no messages, the machine may just be really busy- wait a bit (5-10min) to see if it comes back. If it's still pinging, odds are its very busy. If it doesn't come back, or the messages indicate a fatal error, you will need to proceed with a power cycle (ctrl+alt+del will not work).

=== Power cycle the server ===
If this machine is not a Dell 2950 with a [[DRAC/RMM#DRAC|DRAC card]] (i.e. if you can’t ssh into the DRAC card and issue racadm serveraction hardreset, then you will need someone at the data center to power the macine off, wait 30 sec, then turn it back on. Make sure to re-attach via console (<tt>tip virtxx</tt>) immediately after power down.

=== (Re)attach to the console ===
Stay on the console the entire time during boot. As the BIOS posts- look out for the RAID card output- does everything look healthy? The output may be scrambled, look for "DEGRADED" or "FAILED". Once the OS starts booting you will be disconnected (dropped back to the shell on the console server) a couple times during the boot up. The reason you want to quickly re-attach is two-fold: 1. If you don’t reattach quickly then you won’t get any console output, 2. you want to be attached before the server ''potentially'' starts (an extensive) fsck. If you attach after the fsck begins, you’ll have seen no indication it started an fsck and the server will appear frozen during startup- no output, no response.

=== Start containers/VE's/VPSs ===
When the machine begins to start VE’s, it’s safe to leave the console and login via ssh. All virts should be set to auto start all the VEs after a crash. Further, most (newer) virts are set to “fastboot” it’s VE’s (to find out, do:
grep -i fast /etc/sysconfig/vz
and look for <tt>VZFASTBOOT=yes</tt>). If this was set prior to the machine’s crash (setting it after the machine boots will not have any effect until the vz service is restarted) it will start each ve as fast as possible, in serial, then go thru each VE (serially), shutting it down running a vzquota (disk usage) check, then bringing it back up. The benefit is that all VE’s are brought up quickly (within 15min or so depending on the #), the downside is a customer watching closely will notice 2 outages – 1st the machine crash, 2nd their quota check (which will be a much shorter downtime- on the order of a few minutes).

Where “fastboot” is not set to yes (i.e on quar1), vz will start them consecutively, checking the quotas one at a time, and the 60th VE may not start until an hour or two later - this is not acceptable.

The good news is, if you run vzctl start for a VE that is already started, you will simply get an error: <tt>VE is already started</tt>. Further, if you attempt to vzctl start a VE that is in the process of being started, you will simply get an error: unable to lock VE. So, there is no danger in simply running scripts to start smaller sets of VEs. If the system is not autostarting, then there is no issue, and even if it does, when it conflicts, one process (yours or the autostart) will lose, and just move on to the next one.

A script has been written to assist with ve starts: [[#startvirt.pl|startvirt.pl]] which will start 6 ve’s at once until there are no more left. If startvirt.pl is used on a system where “fastboot” was on, it will circumvent the fastboot for ve’s started by startvirt.pl – they will go through the complete quota check before starting- therefore this is not advisable when a system has crashed. When a system is booted cleanly, and there's no need for vzquota checks, then startvirt.pl is safe and advisable to run.

=== Make sure all containers are running ===
You can quickly get a feel for how many ve’s are started by running:

<pre>[root@virt4 log]# vs
VEID 16066 exist mounted running
VEID 16067 exist mounted running
VEID 4102 exist mounted running
VEID 4112 exist mounted running
VEID 4116 exist mounted running
VEID 4122 exist mounted running
VEID 4123 exist mounted running
VEID 4124 exist mounted running
VEID 4132 exist mounted running
VEID 4148 exist mounted running
VEID 4151 exist mounted running
VEID 4155 exist mounted running
VEID 42 exist mounted running
VEID 432 exist mounted running
VEID 434 exist mounted running
VEID 442 exist mounted running
VEID 450 exist mounted running
VEID 452 exist mounted running
VEID 453 exist mounted running
VEID 454 exist mounted running
VEID 462 exist mounted running
VEID 463 exist mounted running
VEID 464 exist mounted running
VEID 465 exist mounted running
VEID 477 exist mounted running
VEID 484 exist mounted running
VEID 486 exist mounted running
VEID 490 exist mounted running</pre>

So to see how many ve’s have started:
<pre>[root@virt11 root]# vs | grep running | wc -l
39</pre>

And to see how many haven’t:
<pre>[root@virt11 root]# vs | grep down | wc -l
0</pre>

And how many we should have running:
<pre>[root@virt11 root]# vs | wc -l
39</pre>

Another tool you can use to see which ve’s have started, among other things is [[#vzstat|vzstat]]. It will give you CPU, memory, and other stats on each ve and the overall system. It’s a good thing to watch as ve’s are starting (note the VENum parameter, it will tell you how many have started):

<pre>4:37pm, up 3 days, 5:31, 1 user, load average: 1.57, 1.68, 1.79
VENum 40, procs 1705: running 2, sleeping 1694, unint 0, zombie 9, stopped 0
CPU [ OK ]: VEs 57%, VE0 0%, user 8%, sys 7%, idle 85%, lat(ms) 412/2
Mem [ OK ]: total 6057MB, free 9MB/54MB (low/high), lat(ms) 0/0
Swap [ OK ]: tot 6142MB, free 4953MB, in 0.000MB/s, out 0.000MB/s
Net [ OK ]: tot: in 0.043MB/s 402pkt/s, out 0.382MB/s 4116pkt/s
Disks [ OK ]: in 0.002MB/s, out 0.000MB/s

VEID ST %VM %KM PROC CPU SOCK FCNT MLAT IP
1 OK 1.0/17 0.0/0.4 0/32/256 0.0/0.5 39/1256 0 9 69.55.227.152
21 OK 1.3/39 0.1/0.2 0/46/410 0.2/2.8 23/1860 0 6 69.55.239.60
133 OK 3.1/39 0.1/0.3 1/34/410 6.3/2.8 98/1860 0 0 69.55.227.147
263 OK 2.3/39 0.1/0.2 0/56/410 0.3/2.8 34/1860 0 1 69.55.237.74
456 OK 17/39 0.1/0.2 0/100/410 0.1/2.8 48/1860 0 11 69.55.236.65
476 OK 0.6/39 0.0/0.2 0/33/410 0.1/2.8 96/1860 0 10 69.55.227.151
524 OK 1.8/39 0.1/0.2 0/33/410 0.0/2.8 28/1860 0 0 69.55.227.153
594 OK 3.1/39 0.1/0.2 0/45/410 0.0/2.8 87/1860 0 1 69.55.239.40
670 OK 7.7/39 0.2/0.3 0/98/410 0.0/2.8 64/1860 0 216 69.55.225.136
691 OK 2.0/39 0.1/0.2 0/31/410 0.0/0.7 25/1860 0 1 69.55.234.96
744 OK 0.1/17 0.0/0.5 0/10/410 0.0/0.7 7/1860 0 6 69.55.224.253
755 OK 1.1/39 0.0/0.2 0/27/410 0.0/2.8 33/1860 0 0 192.168.1.4
835 OK 1.1/39 0.0/0.2 0/19/410 0.0/2.8 5/1860 0 0 69.55.227.134
856 OK 0.3/39 0.0/0.2 0/13/410 0.0/2.8 16/1860 0 0 69.55.227.137
936 OK 3.2/52 0.2/0.4 0/75/410 0.2/0.7 69/1910 0 8 69.55.224.181
1020 OK 3.9/39 0.1/0.2 0/60/410 0.1/0.7 55/1860 0 8 69.55.227.52
1027 OK 0.3/39 0.0/0.2 0/14/410 0.0/2.8 17/1860 0 0 69.55.227.83
1029 OK 1.9/39 0.1/0.2 0/48/410 0.2/2.8 25/1860 0 5 69.55.227.85
1032 OK 12/39 0.1/0.4 0/80/410 0.0/2.8 41/1860 0 8 69.55.227.90</pre>

When you are all done, you will want to make sure that all the VEs really did get started, run vs one more time.

Note the time all ve’s are back up and enter that into and save the crash log entry.

Occasionally, a ve will not start automatically. The most common reason for a ve not to come up normally is the ve was at it’s disk limit before the crash, and will not start since they’re over the limit. To overcome this, set the disk space to current usage level (the system will give this to you when it fails to start), start the ve, then re-set the disk space back to the prior level. Lastly, contact the customer to let them know they’re out of disk (or allocate more disk if they're entitled to more).

== Hitting performance barriers and fixing them ==

There are multiple modes virtuozzo offers to allocate resources to a ve. We utilize 2: SLM and UBC parameters
On our 4.x systems, we use all SLM – it’s simpler to manage and understand. There are a few systems on virt19/18 that may also use SLM. Everything else uses UBC.
You can tell a SLM ve by:

SLMMODE="all"

in their conf file.

TODO: detail SLM modes and parameters.

If someone is in SLM mode and they hit memory resource limits, they simply need to upgrade to more memory.

The following applies to everyone else (UBC).

Customers will often email and say that they are getting out of memory errors - a common one is "cannot fork" ... basically, anytime you see something odd like this, it means they are hitting one of their limits that is in place in their conf file.

The conf file, however, simply shows their limits - how do we know what they are currently at ?

The answer is a file called v - this file contains the current status (and peaks) of their performance settings, and also counts how many times they have hit the barrier. The output of the file looks like this:

<pre>764: kmemsize 384113 898185 8100000 8200000 0
lockedpages 0 0 322 322 0
privvmpages 1292 7108 610000 615000 0
shmpages 270 528 33000 34500 0
dummy 0 0 0 0 0
numproc 8 23 410 415 0
physpages 48 5624 0 2147483647 0
vmguarpages 0 0 13019 2147483647 0
oomguarpages 641 6389 13019 2147483647 0
numtcpsock 3 21 1210 1215 0
numflock 1 3 107 117 0
numpty 0 2 19 19 0
numsiginfo 0 4 274 274 0
tcpsndbuf 0 80928 1800000 1900000 0
tcprcvbuf 0 108976 1800000 1900000 0
othersockbuf 2224 37568 900000 950000 0
dgramrcvbuf 0 4272 200000 200000 0
numothersock 3 9 650 660 0
dcachesize 53922 100320 786432 818029 0
numfile 161 382 7500 7600 0
dummy 0 0 0 0 0
dummy 0 0 0 0 0
dummy 0 0 0 0 0
numiptent 4 4 155 155 0</pre>

The first column is the name of the counter in question - the same names we saw in the systems conf file. The second column is the _current_ value of that counter, the third column is the max that that counter has ever risen to, the fourth column is the soft limit, and the fifth column is the hard limit (which is the same as the numbers in that systems conf file).

The sixth number is the failcount - how many times the current usage has risen to hit the barrier. It will increase as soon as the current usage hits the soft limit.

The problem with /proc/user_beancounters is that it actually contains that set of data for every running VE - so you can't just cat /proc/user_beancounters - it is too long and you get info for every other running system.

You can vzctl enter the system and run:

vzctl enter 9999
cat /proc/user_beancounters

inside their system, and you will just see the stats for their particular system, but entering their system every time you want to see it is combersome.

So, I wrote a simple script called "vzs" which simply greps for the VEID, and spits out the next 20 or so lines (however many lines there are in the output, I forget) after it. For instance:

<pre># vzs 765:
765: kmemsize 2007936 2562780 8100000 8200000 0
lockedpages 0 8 322 322 0
privvmpages 26925 71126 610000 615000 0
shmpages 16654 16750 33000 34500 0
dummy 0 0 0 0 0
numproc 41 57 410 415 0
physpages 1794 49160 0 2147483647 0
vmguarpages 0 0 13019 2147483647 0
oomguarpages 4780 51270 13019 2147483647 0
numtcpsock 23 37 1210 1215 0
numflock 17 39 107 117 0
numpty 1 3 19 19 0
numsiginfo 0 6 274 274 0
tcpsndbuf 22240 333600 1800000 1900000 0
tcprcvbuf 0 222656 1800000 1900000 0
othersockbuf 104528 414944 900000 950000 0
dgramrcvbuf 0 4448 200000 200000 0
numothersock 73 105 650 660 0
dcachesize 247038 309111 786432 818029 0
numfile 904 1231 7500 7600 0
dummy 0 0 0 0 0
dummy 0 0 0 0 0
dummy 0 0 0 0 0
numiptent 4 4 155 155 0</pre>

That showed us just the portion of /proc/user_beancounters for system 765.

When you run the vzs command, always add a : after the VEID.

So, if a customer complains about some out of memory errors, or no more files, or no more ptys, or just has an unspecific complain about processes dying, etc., the very first thing you need to do is check their beancounters with vzs. Usually you will spot an item that has a high failcount and needs to be upped.

At that point you could simply up the counter with `vzctl set`. Generally pick a number 10-20% higher than the old one, and make the hard limit slightly larger than the the soft limit. However our systems now come in several levels and those levels have more/different memory allocations. If someone is complaining about something other than a memory limit (pty, numiptent, numflock), it’s generally safe to increase it, at least to the same level as what’s in the /vzconf/4unlimited file on the newest virt. If someone is hitting a memory limit, first make sure they are given what they deserve:

(refer to mgmt -> payments -> packages)

To set those levels, you use the [[#setmem|setmem]] command.

The alternate (DEPRECATED) method would be to use one of 3 commands:
256 <veid>
300 <veid>
384 <veid>
512 <veid>

If the levels were not right (you’d run vzs <veid> before and after to see the effect) tell the customer they’ve been adjusted and be done with it. If the levels were right, tell the customer they must upgrade to a higher package, tell them how to see level (control panel) and that they can reboot their system to escape this lockup contidion.

Customers can also complain that their site is totally unreachable, or complain that it is down ... if the underlying machine is up, and all seems well, you may notice in the beancounters that network-specific counters are failing - such as numtcpsock, tcpsndbuf or tcprcvbuf. This will keep them from talking on the network and make it seem like their system is down. Again, just up the limits and things should be fine.

On virts 1-4, you should first look at the default settings for that item on a later virt, such as virt 8 - we have increased the defaults a lot since the early machines. So, if you are going to up a counter on virt2, instead of upping it by 10-20%, instead up it to the new default that you see on virt8.

== Moving a VE to another virt (migrate/migrateonline) ==

This will take a while to complete - and it is best to do this at night when the load is light on both machines.

There are different methods for this, depending on which version of virtuozzo is installed on the src. and dst. virt.
To check which version is running:
[root@virt12 private]# cat /etc/virtuozzo-release
Virtuozzo release 2.6.0

Ok, let's say that the VE is 1212, and vital stats are:
<pre>[root@virt12 sbin]# vc 1212
VE_ROOT="/vz1/root/1212"
VE_PRIVATE="/vz1/private/1212"
OSTEMPLATE="fedora-core-2/20040903"
IP_ADDRESS="69.55.229.84"
TEMPLATES="devel-fc2/20040903 php-fc2/20040813 mysql-fc2/20040812 postgresql-fc2/20040813 mod_perl-fc2/20040812 mod_ssl-fc2/20040811 jre-fc2/20040823 jdk-fc2/20040823 mailman-fc2/20040823 analog-fc2/20040824 proftpd-fc2/20040818 tomcat-fc2/20040823 usermin-fc2/20040909 webmin-fc2/20040909 uw-imap-fc2/20040830 phpBB-fc2/20040831 spamassassin-fc2/20040910 PostNuke-fc2/20040824 sl-webalizer-fc2/20040
818"

[root@virt12 sbin]# vzctl exec 1212 df -h
Filesystem Size Used Avail Use% Mounted on
/dev/vzfs 4.0G 405M 3.7G 10% /</pre>

From this you can see that he’s using (and will minimally need free on the dst server) ~400MB, and he’s running on a Fedora 2 template, version 20040903. He’s also got a bunch of other templates installed. It’s is '''vital''' that '''all''' these templates exist on the dst system. To confirm that, on the dst system run:

For < 3.0:

<pre>[root@virt14 private]# vzpkgls | grep fc2
devel-fc2 20040903
PostNuke-fc2 20040824
analog-fc2 20040824
awstats-fc2 20040824
bbClone-fc2 20040824
jdk-fc2 20040823
jre-fc2 20040823
mailman-fc2 20040823
mod_frontpage-fc2 20040816
mod_perl-fc2 20040812
mod_ssl-fc2 20040811
mysql-fc2 20040812
openwebmail-fc2 20040817
php-fc2 20040813
phpBB-fc2 20040831
postgresql-fc2 20040813
proftpd-fc2 20040818
sl-webalizer-fc2 20040818
spamassassin-fc2 20040910
tomcat-fc2 20040823
usermin-fc2 20040909
uw-imap-fc2 20040830
webmin-fc2 20040909
[root@virt14 private]# vzpkgls | grep fedora
fedora-core-1 20040121 20040818
fedora-core-devel-1 20040121 20040818
fedora-core-2 20040903
[root@virt14 private]#</pre>

For these older systems, you can simply match up the date on the template.

For >= 3.0:

<pre>[root@virt19 /vz2/private]# vzpkg list
centos-5-x86 2008-01-07 22:05:57
centos-5-x86 devel
centos-5-x86 jre
centos-5-x86 jsdk
centos-5-x86 mod_perl
centos-5-x86 mod_ssl
centos-5-x86 mysql
centos-5-x86 php
centos-5-x86 plesk9
centos-5-x86 plesk9-antivirus
centos-5-x86 plesk9-api
centos-5-x86 plesk9-atmail
centos-5-x86 plesk9-backup
centos-5-x86 plesk9-horde
centos-5-x86 plesk9-mailman
centos-5-x86 plesk9-mod-bw
centos-5-x86 plesk9-postfix
centos-5-x86 plesk9-ppwse
centos-5-x86 plesk9-psa-firewall
centos-5-x86 plesk9-psa-vpn
centos-5-x86 plesk9-psa-fileserver
centos-5-x86 plesk9-qmail
centos-5-x86 plesk9-sb-publish
centos-5-x86 plesk9-vault
centos-5-x86 plesk9-vault-most-popular
centos-5-x86 plesk9-watchdog</pre>

On these newer systems, it's difficult to tell whether the template on the dst matches exactly the src. Just cause a centos-5-x86 is listed on both servers doesn't mean all the same packages are there on the dst. To truly know, you must perform a sample rsync:

rsync -avn /vz/template/centos/5/x86/ root@10.1.4.61:/vz/template/centos/5/x86/

if you see a ton of output from the dry run command, then clearly there are some differences. You may opt to let the rsync complete (without running in dry run mode) the only downside is you've now used up more space on the dst and also the centos template will be a mess with old and new data- it will be difficult if not impossible to undo (if someday we wanted to reclaim the space).

If you choose to merge templates, you should closely inspect the dry run output. You should also take care to exclude anything in the /config directory. For example:

rsync -av -e ssh --stats --exclude=x86/config /vz/template/ubuntu/10.04/ root@10.1.4.62:/vz/template/ubuntu/10.04/

Which will avoid this directory and contents:
<pre>[root@virt11 /vz2/private]# ls /vz/template/ubuntu/10.04/x86/config*
app os</pre>

This is important to avoid since the config may differ on the destination and we are really only interested in making sure the pacakges are there, not overwriting a newer config with an older one.

If the dst system was missing a template, you have 2 choices:
# put the missing template on the dst system. 2 choices here:
## Install the template from rpm (found under backup2: /mnt/data4/vzrpms/distro/) or
## rsync over the template (found under /vz/template) - see above
# put the ve on a system which has all the proper templates

=== pre-seeding a migration ===

When migrating a customer (or when doing many) depending on how much data you have to transfer, it can take some time. Further, it can be difficult to gauge when a migration will complete or how long it will take. To help speed up the process and get a better idea about how long it will take you can pre-transfer a customer's data to the destination server. If done correctly, vzmigrate will see the pre-transferred data and pick up where you left off, having much less to transfer (just changed/new files).

We believe vzmigrate uses rsync to do it's transfer. Therefore not only can you use rsync to do a pre-seed, you can also run rsync to see what is causing a repeatedly-failing vzmigrate to fail.

There's no magic to a pre-seed, you just need to make sure it's named correctly.

Given:

source: /vz1/private/1234

and you want to migrate to /vz2 on the target system, your rsync would look like:

rsync -av /vz1/private/1234/ root@x.x.x.x:/vz2/private/1234.migrated/

After running that successful rsync, the ensuing migrateonline (or migrate) will take much less time to complete- depending on the # of files to be analyzed and the # of changed files. In any case, it'll be much much faster than had you just started the migration from scratch.

Further, as we discuss elsewhere in this topic, a failed migration can be moved from <tt>/vz/private/1234</tt> to <tt>/vz/private/1234.migrated</tt> on the destination if you want to restart a failed migration. This should '''only''' be done if the migration failed and the CT is not running on the destination HN.

=== migrateonline intructions: src >=3.x -> dst>=3.x ===

A script called [[#migrateonline|migrateonline]] was written to handle this kind of move. It is basically a wrapper for <tt>vzmigrate</tt> – vzmigrate is a util to seamlessly- as no no reboot of the ve necessary- move a ve from one host to another. This wrapper was initially written cause vz virtuozzo version 2.6.0 has a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables, causing problems when the ve was started up on the dst system. [[#migrate|migrate]] mitigates that. Since it makes multiple ssh connections to the dst virt, it’s a good idea to put the pub key for the src virt in the authorized_keys file on the dst virt. In addition, migrateonline emails ve owners when their migration starts and stops. For this to happen they need to put email addresses (on a single line, space delimited) in a file on their system: /migrate_notify. If the optional target dir is not specified, the ve will be moved to the same private/root location as it was on the src virt. Note: <tt>migrate</tt> is equivalent to <tt>migrateonline</tt>, but will <tt>migrate</tt> a ve AND restart it in the process.

<pre>[root@virt12 sbin]# migrateonline
usage: /usr/local/sbin/migrateonline <ip of node migrating to> <veid> [target dir: vz | vz1 | vz2]
[root@virt12 sbin]# migrateonline 10.1.4.64 1212 vz
starting to migrate 1212 at Sat Mar 26 22:40:38 PST 2005
Turning off offline_management
Saved parameters for VE 1212
migrating with no start on 10.1.4.64
Connection to destination HN (10.1.4.64) is successfully established
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...
Syncing private area '/vz1/private/1212'
- 100% |*************************************************|
done
Successfully completed
clearing the arp cache
now going to 10.1.4.64 and clear cache and starting
starting it
Delete port redirection
Adding port redirection to VE(1): 4643 8443
Adding IP address(es) to pool: 69.55.229.84
Saved parameters for VE 1212
Starting VE ...
VE is mounted
Adding port redirection to VE(1): 4643 8443
Adding IP address(es): 69.55.229.84
Hostname for VE set: fourmajor.com
File resolv.conf was modified
VE start in progress...
finished migrating 1212 at Sat Mar 26 22 22:52:01 PST 2005
[root@virt12 sbin]#</pre>

Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine.

If they had backups, use the mvbackups command to move their backups to the new server:

mvbackups 1212 virt14 vz

Rename the ve

[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/migrated-1212

[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/old-1212-migrated-20120404-noarchive

Update the customer’s systems in mgmt to reflect the new path and server.

IF migrateonline does not work, you can try again using simply migrate- this will result in a brief reboot for the ve.
Before you try again, make sure of a few things:

Depending on where in the migration died, there may be partial data on the dst system in 1 of 2 places:
(given the example above)

/vz/private/1212

or

/vz/private/1212.migrated

before you run migrate again, you'll want to rename so that all data is in
1212.migrated:

mv /vz/private/1212 /vz/private/1212.migrated

this way, it will pick up where it left off and transfer only new files.

Likewise, if you want to speed up a migration, you can pre-seed the dst as follows:

[root@virt12 sbin]# rsync -avSH /vz/private/1212/ root@10.1.4.64:/vz/private/1212.migrated/

then when you run migrate or migrateonline, it will only need to move the changed files- the migration will complete quickly

=== migrateonline/migrate failures (migrate manually) ===

Lets say for whatever reason the migration fails. If it fails with [[#migrateonline|migrateonline]], you should try [[#migrate|migrate]] (which will reboot the customer, so notify them ahead of time).

You may want to run a [[#pre-seeding_a_migration|pre-seed]] rsync to see if you can find the problem. On older virts, we've seen this problem due to a large logfile (which you can find and encourage the customer to remove/compress):
for f in `find / -size +1048576k`; do ls -lh $f; done

You may also see migration failing due to quota issues.

You can try to resolve by copying any quota file into the file you need:

cp /var/vzquota/quota.1 /var/vzquota/quota.xxx

If it complains about quota running you should then be able to stop it

vzquota off xxxx

If all else fails, migrate to a new VEID
i.e. 1234 becomes 12341

If the rsync or [[#migrate|migrate]] fails, you can always move someone manually:

1. stop ve: 
v stop 1234

2. copy over data 
rsync -avSH /vz/private/1234/ root@1.1.1.1:/vzX/private/1234/

NOTE: if you've previously seeded the data (run rsync while the VE was up/running), and this is a subsequent rsync, make sure the last rsync you do (while the VE is not running, has the --delete option in the rsync)

3. copy over conf 
scp /vzconf/1234.conf root@1.1.1.1:/vzconf

4. on dst, edit the conf to reflect the right vzX dir 
vi /vzconf/1234.conf

5. on src remove the IPs 
ipdel 1234 2.2.2.2 3.3.3.3

6. on dst add IPs 
ipadd 1234 2.2.2.2 3.3.3.3

7. on dst, start ve: 
v start 1324

8. cancel, then archive ve on src per above instrs.

=== migrate src=2.6.0 -> dst>=2.6.0, or mass-migration with customer notify ===

A script called <tt>migrate</tt> was written to handle this kind of move. It is basically a wrapper for vzmigrate – vzmigrate is a util to seamlessly move a ve from one host to another. This wrapper was initially written cause vz virtuozzo version 2.6.0 has a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables, causing problems when the ve was started up on the dst system. migrate mitigates that. Since it makes multiple ssh connections to the dst virt, it’s a good idea to put the pub key for the src virt in the authorized_keys file on the dst virt. In addition, migrate emails ve owners when their migration starts and stops. For this to happen they need to put email addresses (on a single line, space delimited) in a file on their system: /migrate_notify. If the optional target dir is not specified, the ve will be moved to the same private/root location as it was on the src virt. Note: migrateonline is equivalent to migrate, but will migrate a ve from one 2.6 '''kernel''' machine to another 2.6 kernel machine without restarting the ve.

<pre>[root@virt12 sbin]# migrate
usage: /usr/local/sbin/migrate <ip of node migrating to> <veid> [target dir: vz | vz1 | vz2]
[root@virt12 sbin]# migrate 10.1.4.64 1212 vz
starting to migrate 1212 at Sat Mar 26 22:40:38 PST 2005
Turning off offline_management
Saved parameters for VE 1212
migrating with no start on 10.1.4.64
Connection to destination HN (10.1.4.64) is successfully established
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...
Syncing private area '/vz1/private/1212'
- 100% |*************************************************|
done
Successfully completed
clearing the arp cache
now going to 10.1.4.64 and clear cache and starting
starting it
Delete port redirection
Adding port redirection to VE(1): 4643 8443
Adding IP address(es) to pool: 69.55.229.84
Saved parameters for VE 1212
Starting VE ...
VE is mounted
Adding port redirection to VE(1): 4643 8443
Adding IP address(es): 69.55.229.84
Hostname for VE set: fourmajor.com
File resolv.conf was modified
VE start in progress...
finished migrating 1212 at Sat Mar 26 22 22:52:01 PST 2005
[root@virt12 sbin]#</pre>

Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine (backup2 or mail).
Cancel the ve (first we have to rename things which migrate changed so cancelve will find them):

[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/1212.conf

On 2.6.1 you’ll also have to move the private area:
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/1212

<pre>[root@virt12 sbin]# cancelve 1212
v stop 1212
v set 1212 --offline_management=no --save
Delete port redirection
Deleting IP address(es) from pool: 69.55.229.84
Saved parameters for VE 1212
mv /vzconf/1212.conf /vzconf/deprecated-1212
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414
don't forget to remove firewall rules and domains!
[root@virt12 sbin]#</pre>

Note: if the system had backups, [[#cancelve|cancelve]] would offer to remove them. You want to say '''no''' to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.
Then go to backup2 and move the dirs. So you’d do something like this:
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/

We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive.
If moving to the same drive, you can safely preserve hardlinks and move all files with:
sh
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done

To move everyone off a system, you’d do:
for f in `vl`; do migrate <ip> $f; done

Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.

=== vzmigrate: src=2.6.1 -> dst>=2.6.0 ===

This version of vzmigrate works properly with regard to handling ips. It will not notify ve owners of moves as in the above example. Other than that it’s essentially the same.

<pre>[root@virt12 sbin]# vzmigrate 10.1.4.64 -r no 1212:1212:/vz/private/1212:/vz/root/1212
migrating on 10.1.4.64
Connection to destination HN (10.1.4.64) is successfully established
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...
Syncing private area '/vz1/private/1212'
- 100% |*************************************************|
done
Successfully completed
Adding port redirection to VE(1): 4643 8443
Adding IP address(es) to pool: 69.55.229.84
Saved parameters for VE 1212
Starting VE ...
VE is mounted
Adding port redirection to VE(1): 4643 8443
Adding IP address(es): 69.55.229.84
Hostname for VE set: fourmajor.com
File resolv.conf was modified
VE start in progress...
[root@virt12 sbin]#</pre>

Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine (backup2 or mail).
Cancel the ve (first we have to rename things which vzmigrate changed so cancelve will find them):

<pre>[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/1212.conf
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/1212

[root@virt12 sbin]# cancelve 1212
v stop 1212
v set 1212 --offline_management=no --save
Delete port redirection
Deleting IP address(es) from pool: 69.55.229.84
Saved parameters for VE 1212
mv /vzconf/1212.conf /vzconf/deprecated-1212
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414
don't forget to remove firewall rules and domains!
[root@virt12 sbin]#</pre>

Note: if the system had backups, <tt>cancelve</tt> would offer to remove them. You want to say no to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.
Then go to backup2 and move the dirs. So you’d do something like this:
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/

We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive.
If moving to the same drive, you can safely preserve hardlinks and move all files with:
sh
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done

Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.

=== src=2.5.x ===

First, go to the private dir:

cd /vz1/private/

Stop the VE - make sure it stops totally cleanly.

vzctl stop 1212

Then you’d use vemove - a script written to copy over the config, create tarballs of the ve’s data on the destination virt, and cancel the ve on the source system (in this example we’re going to put a ve that was in /vz1/private on the src virt, in /vz/private on the dst virt):

<pre>[root@virt12 sbin]# vemove
ERROR: Usage: vemove veid target_ip target_path_dir
[root@virt12 sbin]# vemove 1212 10.1.4.64 /vz/private/1212
tar cfpP - 1212 --ignore-failed-read | (ssh -2 -c arcfour 10.1.4.64 "split - -b 1024m /vz/private/1212.tar" )
scp /vzconf/1212.conf 10.1.4.64:/vzconf
cancelve 1212
v stop 1212
v set 1212 --offline_management=no --save
Delete port redirection
Deleting IP address(es) from pool: 69.55.229.84
Saved parameters for VE 1212
mv /vzconf/1212.conf /vzconf/deprecated-1212
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414
don't forget to remove firewall rules and domains!
[root@virt12 sbin]#</pre>

Note: if the system had backups, cancelve would offer to remove them. You want to say no to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.
Then go to backup2 and move the dirs. So you’d do something like this:
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/

We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive.
If moving to the same drive, you can safely preserve hardlinks and move all files with:
sh
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done

When you are done, go to /vz/private on the dst virt you will have files like this:

<pre>1212.taraa
1212.tarab
1212.tarac</pre>

Each one 1024m (or less, for the last one) in size.

on the dst server and run:

cat 1212.tar?? | tar xpPBf -

and after 20 mins or so it will be totally untarred. Now since the conf
file is already there, you can go ahead and start the system.

vzctl start 1212

Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.

NOTE: you MUST tar the system up using the virtuozzo version of tar that
is on all the virt systems, and further you MUST untar the tarball with
the virtuozzo tar, using these options: `<tt>tar xpPBf -</tt>`

If you tar up an entire VE and move it to a non-virtuozzo machine, that is
ok, and you can untar it there with normal tar commands, but do not untar
it and then repack it with a normal tar and expect it to work - you need
to use virtuozzo tar commands on virtuozzo tarballs to make it work.

The backups are sort of an exception, since we are just (usually)
restoring user data that was created after we gave them the system, and
therefore has nothing to do with magic symlinks or vz-rpms, etc.

== Moving a VE on the same virt ==

Easy way: 
Scenario 1: ve 123 is to be renamed 1231 and moved from vz1 to vz

vzmlocal 123:1231:/vz/private/1231:/vz/root/1231

Scenario 2: ve 123 is to be moved vz1 to vz

vzmlocal 123:123:/vz/private/123:/vz/root/123

vzmlocal will reboot the ve at the end of the move

Manual/old way:

1) <tt>vzctl stop 123</tt> 
2) <tt>mv /vz1/private/123 /vz/private/.</tt> 
(or cp -a if you want to copy)
3) in <tt>/etc/sysconfig/vz-scripts/123.conf</tt> change value 
of '<tt>VE_PRIVATE</tt>' variable to point to a new private area location
4) <tt>vzctl start 123</tt> 
5) update backups if needed: <tt>mvbackups 123 virtX virt1 vz</tt> 
6) update management scerens 

Notes: a) absolute path to private area is stored in quota file <tt>/var/vzquota/quota.123</tt> - so during first startup quota will be recalculated. 
b) if you're going to write some script to do a job, you MUST be sure that $VEID won't be expanded to '' in ve config file - ie. you need to escape '$'. Otherwise you might have:

VE_PRIVATE="/vz/private/"

in config, and 'vzctl destroy' for this VE ID '''will remove everything under /vz/private/ directory'''.

== Adding a veth device to a VE ==

Not totally sure what this is, but a customer asked for it and here's what we did (as instructed by vz support):

<pre>v set 99 --netif_add eth99 --save
ipdel 99 69.55.230.58
v set 99 --ifname eth99 --ipadd 69.55.230.58 --save
v set 99 --ifname eth99 --gateway 69.55.230.1 --save

vznetcfg net new veth_net

# vznetcfg net list
Network ID Status Master Interface Slave Interfaces
net99 active eth0 veth77.77,veth99.99
veth_net active

# vznetcfg if list
Name Type Network ID Addresses
br0 bridge veth_net
br99 bridge net99
veth99.99 veth net99
eth1 nic 10.1.4.62/24
eth0 nic net99 69.55.227.70/24

vznetcfg br attach br0 eth0

(will remove 99 from orig net and move to veth_net)
vznetcfg net addif veth_net veth99.99

# vznetcfg net list
Network ID Status Master Interface Slave Interfaces
net99 active
veth_net active eth0 veth77.77,veth99.99

(delete the old crap)
vznetcfg net del net99

Then, to add another device in

v set 77 --netif_add eth77 --save
ipdel 77 69.55.230.78
v set 77 --ifname eth77 --ipadd 69.55.230.78 --save
v set 77 --ifname eth77 --gateway 69.55.230.1 --save
v set 77 --save --ifname eth77 --network veth_net

# vznetcfg if list
Name Type Network ID Addresses
veth77.77 veth
br0 bridge veth_net
veth99.99 veth veth_net
eth1 nic 10.1.4.62/24
eth0 nic veth_net 69.55.227.70/24

vznetcfg net addif veth_net veth77.77

# vznetcfg if list
Name Type Network ID Addresses
veth77.77 veth veth_net
br0 bridge veth_net
veth99.99 veth veth_net
eth1 nic 10.1.4.62/24
eth0 nic veth_net 69.55.227.70/24

# vznetcfg net list
Network ID Status Master Interface Slave Interfaces
veth_net active eth0 veth77.77,veth99.99

another example

v set 1182 --netif_add eth1182 --save
ipdel 1182 69.55.236.217
v set 1182 --ifname eth1182 --ipadd 69.55.236.217 --save
v set 1182 --ifname eth1182 --gateway 69.55.236.1 --save
vznetcfg net addif veth_net veth1182.1182
v set 1182 --save --ifname eth1182 --network veth_net

unused/not working commands:
ifconfig veth99.0 0
vznetcfg net list
vznetcfg br new br99 net99
vznetcfg br attach br99 eth0
vznetcfg br show
vznetcfg if list
vznetcfg net addif net99 veth99.99
vznetcfg net addif eth0 net99

---------

vznetcfg br new br1182 net1182

vznetcfg br attach br99 eth0
vznetcfg if list
vznetcfg net addif net99 veth99.99
vznetcfg net addif eth0 net99

vznetcfg net addif eth0 net1182

# vznetcfg net addif veth99.0 net99
--- 8< ---

vznetcfg net new net
# vznetcfg net addif eth0 net99

# vzctl set 99 --save --netif_add eth0 (at this stage veth99.0 interface have to appear
on node)
# vzctl set 99 --save --ifname eth0 --ipadd 69.55.230.58 (and probably few more arguments
here - see 'man vzctl')
# vznetcfg net addif veth99.0 net99</pre>

== Assigning/remove ip from a VE ==

1. Add or remove ips:
ipdel 1234 1.1.1.1 2.2.2.2
ipadd 1234 1.1.1.1 2.2.2.2

2. update Mgmt screens

3. offer to update any DNS we do for them

4. check to see if we had rules for old IP in firwall

== Enabling tun device for a ve ==
Note, there’s a command for this: [[#addtun|addtun]]

For posterity:
Make sure the tun.o module is already loaded before Virtuozzo is started:
lsmod
Allow the VPS to use the TUN/TAP device:
vzctl set 101 --devices c:10:200:rw --save
Create the corresponding device inside the VPS and set the proper permissions:
vzctl exec 101 mkdir -p /dev/net
vzctl exec 101 mknod /dev/net/tun c 10 200
vzctl exec 101 chmod 600 /dev/net/tun

== Remaking a system (on same virt) ==

1. [[#cancelve|cancelve]] (or v destroy x - ONLY if you're POSITIVE no data needs to be saved)

2. [[#vemake|vemake]] using same veid

3. [[#mvbackups|mvbackups]] or [[#vb|vb]] (if new mount point)

4. update mgmt with new dir/ip

5. update firewall if changed ip

== Re-initialize quota for a VE ==

There’s a commamd for this now: [[#clearquota|clearquota]]

For posterity:

vzctl stop 1
vzquota drop 1
vzctl start 1

== Traffic accounting on linux ==

DEPRECATED - all tracking is done via bwdb now. This is how we used to track traffic.

TODO: update for diff versions of vz

Unlike FreeBSD, where we have to add firewall count rules to the system to count the traffic, on virtuozzo counts the traffic for us. You an see the current traffic stats by running `vznetstat`:

<pre># vznetstat
VEID Net.Class Output(bytes) Input(bytes)
-----------------------------------------------
24218 1 484M 39M
24245 1 463M 143M
2451 1 2224M 265M
2454 1 2616M 385M
4149 1 125M 68M
418 1 1560M 34M
472 1 1219M 315M
726 1 628M 317M
763 1 223M 82M
771 1 4234M 437M
-----------------------------------------------
Total: 13780M 2090M</pre>

As you can see the VEID is on a line with the in and out bytes. So, we simply run a cron job:

4,9,14,19,24,29,34,39,44,49,55,59 * * * * /root/vztrafdump.sh

Just like we do on FreeBSD - this one goes through all the VEs in /vz/private and greps the line from vznetstat that matches them and dumps it in /jc_traffic_dump on their system. Then it does it again for all the VEs in /vz1/private. It is important to note that vznetstat runs only once, and the grepping is done from a temporary file that contains that output - we do this because running vznetstat once for each VE that we read out of /vz/private and /vz1/private would take way too long and be too intensive.

You do not need to do anything to facilitate this other than make sure that that cron job is running - the vznetstat counters are always running, and any new VEs that are added to the system will be accounted for automatically.

Traffic resetting no longer works with vz 2.6, so we disable the vztrafdump.sh on those virts.

== Watchdog script ==

On some of the older virts, we have a watchdog running that kills procs that are deemed bad per the following:

/root/watchdog from quar1

<pre>if echo $line | awk '{print $(NF-3)}' | grep [5-9]...
then
# 50-90%
if echo $line | awk '{print $(NF-1)}' | grep "...:.."
then
# running for > 99min
echo $line >> /root/wd.log
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log
kill -9 $pid

fi
if echo $line | awk '{print $(NF-1)}' | grep "....m"
then
# running for > 1000min
echo $line >> /root/wd.log
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log
kill -9 $pid

fi
fi

if echo $line | awk '{print $(NF-3)}' | grep [1-9]...
then
# running for 10-90 percent
if echo $line | awk '{print $NF}' | egrep 'cfusion|counter|vchkpw'
then

if echo $line | awk '{print $(NF-1)}' | grep "[2-9]:.."
then
# between 2-9min
echo $line >> /root/wd.log
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log
kill -9 $pid

elif echo $line | awk '{print $(NF-1)}' | grep "[0-9][0-9]:.."
then
# up to 99min
echo $line >> /root/wd.log
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log
kill -9 $pid

fi
fi
fi</pre>

== Misc Linux Items ==

We are overselling hard drive space ... when you configure a linux system with a certain amount of disk space (the default is 4gigs) you do not actually use up 4gigs of space on the system. The diskspace setting for a user is simply a cap, and they only use up as much space on the actual disk drive as they are actually using.

When you create a new linux system, even though there are some 300 RPMs or so installed, if you run `df -k` you will see that the entire 4gig partition is empty - no space is being used. This is because the files in their system are "magic symlinks" to the template for their OS that is in /vz/template - however, any changes to any of those files will "disconnect" them and they will immediately begin using space in their system. Further, any new files uploaded (even if those new files overwrite existing files) will take up space on the partition.

if you see this:

<pre>[root@virt8 root]# vzctl stop 160 ; vzctl start 160
VE is not running
Starting VE ...
VE is unmounted
VE is mounted
Adding IP address(es): 69.55.226.83 69.55.226.84
bash ERROR: Can't change file /etc/sysconfig/network
Deleting IP address(es): 69.55.226.83 69.55.226.84
VE is unmounted
[root@virt8 root]#</pre>

it probably means they no longer have /bin/bash - copy one in for them

ALSO: another possibility is that they have removed the `ed` RPM from their system - it needs to be reinstalled into their system. But since their system is down, this is tricky ...

VE startup scripts used by 'vzctl' want package 'ed' to be available inside VE. So if package 'ed' will be enabled in OS template config and OS template itself VE #827 is based on - this error should be fixed.

yes, it is possible to add RPM to VE while it not running.
Try to do following:

<pre># cd /vz/template/<OS_template_with_ed_package>/
# vzctl mount 827
# rpm -Uvh --root /vz/root/827 --veid 827 ed-0.2-25.i386.vz.rpm</pre>

Usually theres an error, but its ok

Note: replace 'ed-0.2-25.i386.vz.rpm' in last command with actual
version of 'ed' package you have.

----

So how do I know what template the user has ? cat their conf file and it is listed in there. For example, if the conf file has:

<pre>[root@virt12 sbin]# vc 1103
…snip…
OSTEMPLATE="debian-3.0/20030822"
TEMPLATES="mod_perl-deb30/20030707 mod_ssl-deb30/20030703 mysql-deb30/20030707 proftpd-deb30/20030703 webmin-deb30/20030823 "
…</pre>

then they are on debian 3.0, all of their system RPMs are in /vz/template/debian-3.0, and they are using version 20030822 of that debian 3.0 template. Also, they’ve also got additional packages installed (mod_perl, mod_ssl, etc). Those are also found under /vz/template

----

Edits needed to run java:

When we first created the VEs, the default setting for privvmpages was 93000:94000 ... which was high enough that most people never had problems ... however, you can;t run java or jdk or tomcat or anything java related with that setting. We have found that by setting privvmpages to 610000:615000 that java runs just fine. That is now the default setting. It is exceedingly rare that anyone needs it higher than that, although we have seen it once or twice.

Any problems with java at all - the first thing you need to do is see if the failcnt has raised for privvmpages.

<pre># vzctl start 160
Starting VE ...
vzquota : (error) Quota on syscall for 160: Device or resource busy
Running vzquota on failed for VE 160 [3]</pre>

This is because my pwd is _in_ their private directory - you can't start it until you move out

People seem to have trouble with php if they are clueless newbies. Here are two common problems/solutions:

no... but i figured it out myself. problem was the php.ini file that came
vanilla with the account was not configured to work with apache (the
ENGINE directive was set to off).

everything else seems fine now.

----

the problem was in the php.ini file. I noticed that is wasnt showing
the code when it was in an html file so I looked at the php.ini file
and had to change it so it recognized <? tags aswell as <?php tags.

Also, make sure added to httpd.conf
AddType application/x-httpd-php .php

----

You can set the time zone:

You can change the timezone by doing this:

ln -sf /usr/share/zoneinfo/<zone> /etc/localtime

where <zone> is the zone you want in the /usr/share/zoneinfo/ directory.

----

Failing shm_open calls:

first, please check if /dev/shm is mounted inside VE.
'cat /proc/mounts' command should show something like this:
tmpfs /dev/shm tmpfs rw 0 0

If /dev/shm is not mounted you have 2 ways to solve issue:
1. execute following command inside VE (doesn't require VE reboot):
mount -t tmpfs none /dev/shm
2. add following string to /etc/fstab inside VE and reboot it:
tmpfs /dev/shm tmpfs defaults 0 0

----

You can have a mounted but not running ve
Just:
vzctl mount <veid>

----

When a debian sys can’t get on the network, and you try:

<pre>vzctl set 1046 --ipadd 69.55.227.117
Adding IP address(es): 69.55.227.117
Failed to bring up lo.
Failed to bring up venet0.</pre>

They probably removed iproute package, which must be the one from swsoft. To restore:
<pre># dpkg -i --veid=1046 --admindir=/vz1/private/1046/root/var/lib/dpkg --instdir=/vz1/private/1046/root/ /vz/template/debian-3.0/iproute_20010824-8_i386.vz.deb
(Reading database ... 16007 files and directories currently installed.)
Preparing to replace iproute 20010824-8 (using .../iproute_20010824-8_i386.vz.deb) ...
Unpacking replacement iproute ...
Setting up iproute (20010824-8) ...</pre>

Then restart their ve

----

in a ve i do:

<pre>cd /
du -h .</pre>

and get: 483M .

i do:

<pre>bash-2.05a# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/vzfs 4.0G 2.3G 1.7G 56% /</pre>

how can this be?

Is it possible that quota file was corrupted somehow? Please try to:
<pre>vzctl stop <VEID>
vzquota drop <VEID>
vzquota init <VEID>
vzctl start <VEID></pre>

----

How to stop vz from starting after reboot:

VIRTUOZZO=no
in
/etc/sysconfig/vz

To start:
service vz start
(after setting VIRTUOZZO=yes in /etc/sysconfig/vz)

service vz restart will do some kind of 'soft reboot' -- restart all
VPSes and reload modules without rebooting the node

if you need to shut down all VPSes really really fast, run killall -9 init

----

Postfix tip:

You may want to tweak settings: default_process_limit=10

----

* uptime/load that vz users see isn’t what we see- it's their own load
* Vz overwrites resolve.conf on every boot to what’s in ve config file. Also overwrites hostname
* Quotas in vz: set Gid_limit in vz config file and restart ve. Errors leading up to this will be out of files but no inode or space issues. set to 2000 in conf, and restart vz

= Virtuozzo VPS Management Tools =

== vm ==

To create a new Linux VPS

* get an IP address from mgmt screen
* run df -h to figure out which disk has enough space
* run vm with no arguments to get syntax and exact template name
* use col0xxxx for hostname if they don’t give you a hostname
* copy over dir, veid, ip and password to pending customer screen

vm CID IP hostname /vz[1|2] email[,email] template ( <LB|LBP|LS|LM|LMP|LP> [disk] | <gb_disk/mb_ram/gb_burst> )

== disable ==
To disable a customers Virtuozzo VPS server.

* stop the server
vzctl stop <veid>
* prevent restart on server reboot and user restart via Control Panel
vzctl set <veid> --disabled=yes --offline_management=no --save

== cancelve ==
When a customer cancels all service.

cancelve <veid>
this will:
* stop a ve
* check for backups (offer to remove them from the backup server
and the backup.config)
* rename the private dir
* check for PTR, provide the commands to reset to default
* and rename the ve’s config
* remind you to remove firewall rules
* remind you to remove DNS entries

== ipadd ==
ipadd <veid> <ip> [ip] [ip]
add’s ip(s) to a ve

== ipdel ==
ipdel <veid> <ip> [ip] [ip]
removes ip(s) from a ve

== vc ==
vc <veid>

display the contents of the vz config file for the <veid>

the equivalent of: <tt>cat /vzconf/<veid>.conf</tt>

== vl ==
vl

List all virtual environments running on this machine.

will displays a list of ve #’s, 1 per line. (ostensibly to use in a for loop)

== vp ==
vp <veid>

show processes running under this <veid>.

the equivalent of: <tt>vzps auxww –E <veid></tt>

== vpe ==
DEPRECATED
vpe <veid>
this will allow you to do a vp when a ve is running out of control, the equivalent of (deprecated since vp operates outside the VPS):
<pre>vzctl set <veid> --kmemsize 2100000:2200000
vzctl exec <veid> ps auxw
vzctl set <veid> --kmemsize (ve’s orig lvalue):(ve’s orig hvalue)</pre>

== vt ==
vt <veid>
the equivalent of: <tt>vztop –E <veid></tt>

== vr ==
vr <veid>

Restart a <veid>.

the equivalent of: <tt>vzctl stop <veid>; vzctl start <veid></tt>
You can run this even if the ve is down - the stop command will just fail

== vs ==
vs [veid]
displays status (output of <tt>vzctl status <veid></tt>) on each ve configured on the system (as determined by <tt>/vzconf/*.conf</tt>)
If passed an argument, gives the status for just that ve.
A running system looks like:
<tt>VEID 16066 exist mounted running</tt>

A ve which is not running (but does exist) looks like:
<tt>VEID 9990 exist unmounted down</tt>

A ve which is not running and doesn’t exist looks like:
<tt>VEID 421 deleted unmounted down</tt>

== vs2 ==
vs2 [veid]
this is similar to vs in that it displays status (output of <tt>vzctl status <veid></tt>) on each ve,
but the difference is it’s list comes from doing an ls on the data dirs. This was meant to catch
the rare case where a ve configured but exists.

== vw ==
vw [veid]
displays the output of ‘<tt>w</tt>’ (the equivalent of <tt>vzctl exec <veid> w</tt>) for each configured ve (as determined by <tt>/vzconf/*.conf</tt>). Useful for determing which ve is contributing to a heavily-loaded system.
If passed an argument, gives ‘<tt>w</tt>‘ output for just that ve.
Ex:
<pre>[root@virt2 etc]# vw
134
10:52pm up 79 days, 6:14, 0 users, load average: 0.02, 0.02, 0.00
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
16027
2:52pm up 7 days, 19:54, 0 users, load average: 0.00, 0.00, 0.00
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
16055
2:52pm up 79 days, 6:38, 0 users, load average: 0.00, 0.04, 0.07
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT</pre>

== vwe ==
vwe [constraint]
just like <tt>vw</tt>, but takes a constraint as an argument, only show’s ve’s with loads >= the constraint provided. If no constraint is provided, 1 is used by default

== vzs ==
vzs [veid]
displays the beancounter status for all ve’s, or a particular ve if an argument is passed

== ve ==
ve <veid>

enter a <veid> as root.

the equivalent of: <tt>vzctl enter <veid></tt>

== vx ==
vx <veid> <command>

execute a single command in the <veid> environment.

the equivalent of: <tt>/usr/sbin/vzctl exec <veid> <command></tt>

== dprocs ==
dprocs [count]
a script which outputs a continuous report (or a certain number of reports if an option is passed) of processes stuck in the D state and which VPS’s those procs belong to.

== setmem ==
setmem VEID <256|512|768|1024|1280|1536|2048> [burst GB]
adjusts the memory resources for the VE. If this is on VZ ver >4, you will also be prompted to provide a burst ram amount. Sample usage:

<pre># setmem 21671 1024 4
/usr/sbin/vzctl set 21671 --ram 1024m --swap 3072m --save
</pre>

== afacheck.sh ==
afacheck.sh
displays the health/status of containers and mirrors on an adaptec card (currently quar1, tempvirt1-2, virt9, virt10)- all other are LSI

== backup ==
backup
backup script called nightly to update virt scripts and do customer backups

== checkload.pl ==
checkload.pl
this was intended to be setup as a cronjob to watch processes on a virt when the load
rises above a certain level. Not currently in use.

== findbackuppigs.pl ==
findbackuppigs.pl
looks for files larger than 50MB which customers have asked us to backup. Emails matches
to linux@johncompanies.com

== gatherlinux.pl ==
gatherlinux.pl
gathers up data about ve’s configured and writes to a file. Used for audits against the db

== gb ==
gb <search>
greps backup.config for the given search parameter

== gbg ==
gbg <search>
greps backup.config for the given search parameter and presents just the directories (for clean pasting)

== linuxtrafficgather.pl ==
linuxtrafficgather.pl [yy-mm]
sends a traffic usage summary by ve to support@johncomapnies.com and payments@johncompanies.com.
Optional arguments are year and month (must be in the past). If not passed, assumes last month. Relies on
traffic logs created by netstatreset and netstatbackup

== linuxtrafficwatch.pl ==
linuxtrafficwatch.pl
checks traffic usage and emails support@johncomapnies.com when a ve reaches the warning level (35G) and
the limit (40G). Works on virtuozzo versions <= 2.6.x. We really aren’t using this anymore now that we have netflow.

== linuxtrafficwatch2.pl ==
linuxtrafficwatch2.pl
checks traffic usage and emails support@johncomapnies.com when a ve reaches the warning level (35G) and
the limit (40G). Works on virtuozzo version 2.6.x. We really aren’t using this anymore now that we have netflow.

== load.pl ==
load.pl
feeds info to load mrtg - executed by inetd.

== mb (linux) ==
mb <mount|umount>
(nfs) mounts and umounts dirs to backup2. Shortcuts are mbm and mbu to mount and unmount.

== migrate ==
migrate <ip of node migrating to> <veid> <target_veid> [target dir: vz | vz1 | vz2]
this is basically a wrapper for <tt>vzmigrate</tt> – vzmigrate is a util to seamlessly move a ve from one host to another. This wrapper was written cause vz virtuozzo version 2.6 had a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables. This script mitigates that. Since it makes multiple ssh connections to the target host, it’s a good idea to put the pub key for the src system in the authorized_keys file on the target host. In addition, it emails ve owners when their migration starts and stops (if they place email addresses in a file on their system: /migrate_notify). To move everyone off a system, you’d do:
for f in `vl`; do migrate <ip> $f; done

== migrateonline ==
migrateonline <ip of node migrating to> <veid> <target_veid> [target dir: vz | vz1 | vz2]
this is the same as migrate but will migrate a ve in <tt>–online</tt> mode which means it won’t be shut down at the end of the migration. This only works when migrating ve’s between 2 machines running a 2.6 kernel (currently tempvirt1-2. virt16-19, virt12). If you get an error that the machine you’re trying to migrate to has a different CPU or features, etc, then you have to edit the file and add the –f switch to the vzmigrate line- you can basically ignore this kind of warning (but never ignore a warning about missing templates on the destination node). NOTE: This edit (if made to migrateonline) will be overwritten by the base script during each night’s backup.

== netstatbackup ==
DEPRECATED
netstatbackup
writes traffic count data to a logfile. Works on virtuozzo versions 2.5.x.

== netstatbackup2 ==
DEPRECATED
netstatbackup2
writes traffic count data to a logfile. Works on virtuozzo versions 2.6.x.

== netstatreset ==
DEPRECATED
netstatreset
writes traffic count data to a logfile and resets counters to 0. Works on virtuozzo versions 2.5.x

== netstatreset2 ==
DEPRECATED
netstatreset2
writes traffic count data to a logfile. Works on virtuozzo versions 2.6.x.

== orphanedbackupwatchlinux ==
orphanedbackupwatchlinux
looks for directories on backup2 which aren’t configured in backup.config and offers to
delete them

== rsync.backup (linux) ==
rsync.backup
does customer backups (relies on backup.config)

== startvirt.pl ==
startvirt.pl
forks off start ve commands – keeps 6 running at a time. This is not to be used on systems where fastboot is enabled as it circumvents the benefit of the fastboot. The script will occasionally not exit gracefully and will continue to use up CPU, so it should be watched. Also, don’t exit from the script till you’re sure all ve’s are started – if you do you need to start them manually and may have to free up locks. On some systems, the startvirt script doesn’t exit cleanly and you have to ^C out of it. Be careful though- doing so can leave some VE’s in an odd bootup state and you may need to ‘vr’ them manually. You should check to see which ve’s aren’t running and/or confirm all have started when ^C’ing out of startvirt.

== taskdone (linux) ==
taskdone
when called will email support@johncompanies.com with the hostname of the machine from which it was
executed as the subject

== vb (linux) ==
vb
the equivalent of: <tt>vi /usr/local/sbin/backup.config</tt>

== vemakeXX ==
DEPRECATED
vemakerh9
ve create script for RH9 (see vemake)

vemakedebian30
ve create script for debian 3.0 (Woody) (see vemake)

vemakedebian31
ve create script for debian 3.1 (Sarge) (see vemake)

vemakedebian40
ve create script for debian 4.0 (Etch) (see vemake)

vemakefedora, vemakefedora2, vemakefedora4, vemakefedora5, vemakefedora6, vemakefedora7
ve create script for fedora core 1, 2, 4, 5, 6, 7 respectively (see vemake)

vemakecentos3, vemakecentos4
ve create script for centos 3, 4 respectively (see vemake)

vemakesuse, vemakesuse93, vemakesuse100
ve create script for suse 9.2, 9.3, 10.0 respectively (see vemake)

vemakeubuntu5, vemakeubuntu606, vemakeubuntu606 vemakeubuntu704
ve create script for ubuntu 5.10, 6.06, 6.10, 7.04 respectively (see vemake)

== vemove ==
DEPRECATED
vemove <veid> <target_ip> </vz/private/123>
this script simplifies the old way of moving ve’s from one system to another - in short moving a ve to or from a virt running virtuozzo < 2.6.x
It’s the equivalent of:
<tt>tar cfpP - <veid> --ignore-failed-read | (ssh -2 -c arcfour <target_ip> "split - -b 1024m </vz/private/123>.tar" )</tt>This should only be used if migrate/vzmigrate can’t be used.

== vim.watchdog ==
vim.watchdog
looks for and kills procs matching “vi|vim|nano|pine|elm” that are running for a long time & consuming lots of cpu. Works on virtuozzo versions 2.5.x

== vim.watchdog2 ==
vim.watchdog2
looks for and kills procs matching “vi|vim|nano|pine|elm” that are running for a long time & consuming lots of cpu.
Works on virtuozzo versions 2.6.x.

== vzmigrate ==
vzmigrate <target_ip> -r no <veid>:[dst veid]:[dst /vzX/private/veid]:[dst /vzX/root/veid]
(this is the raw command “wrapped” by migrate/migrateonline) this will seamlessly move a ve from one host to another. The ve will run for the duration of the migration till the very end when it’s shut down, ip moved and started up on the target system. The filesystem on the src will remain. This should be watched – occasionally the move will timeout and leave the system shut down. If target private and root aren’t specified it just puts it in /vz. Only works when both systems are running virtuozzo 2.6.x

== vztrafdump.sh ==
DEPRECATED
vztrafdump.sh
writes traffic usage info by ve to a file called jc_traffic_dump in each ve’s / dir. Works on virtuozzo versions <= 2.5.x.

== vztrafdump2.sh ==
DEPRECATED
vztrafdump2.sh
writes traffic usage info by ve to a file called jc_traffic_dump in each ve’s / dir. Works on virtuozzo versions 2.6.x.

== addtun ==
addtun <veid>
Add’s tun device to ve.

== bwcap ==
bwcap <veid> <kbps>
Ex: <tt>bwcap 1234 512</tt>
Caps a VE’s bandwidth to the amount given

== setdisk ==
setdisk <veid> <diskspace in GB>
Ex: <tt>setdisk 1234 5</tt>
Gives a VE’s a given amount of disk space

== vdf ==
vdf <veid>
the equivalent of: <tt>vzctl exec <veid> df –h</tt>

== vdff ==
vdff
runs a (condensed) vdf for all ve’s in your pwd (must be run from /vz/privateN)

== mvbackups ==
mvbackups <veid> <target_machine> (virt1) <target_dir> (vz1)
moves backups from one location to another on the backup server, and provides you with option to remove entries from current backup.config, and simple paste command to add the config to backup.config on the target server

== checkquota ==
checkquota
for all the ve’s in the cwd (run from /vz/private, /vz1/private, etc) reports what vz quota says they’re using and what the actual usage is (as reported by du)

== clearquota ==
clearquota <veid>
Recalculates a ve’s quota, prints out the usage before and after. The equivalent of:
<tt>vdf <veid>; v stop <veid>; vzquota drop <veid>; v start <veid>; vdf <veid></tt>

== dprocs ==
dprocs
Sometimes the server’s have a large number of processes get stuck in the D state- this script shows (every 3 secs) which VE’s have D procs, which procs
are stuck and a running average of the top “offenders”

== vzstat ==
vstat
sort of like top for VZ. sort VEs by CPU usage by pressing 'o' and then 'c' keys

== stopvirt ==
stopvirt
will stop VEs as fast as it can, 6 at a time. May not exit when complete so you should watch [[#vzstat|vzstat]] in another window.

VPS Management

2014-06-13T18:07:02Z

72.207.101.92: /* mknod in a jail */

= Common Problems =
== Login to any machine without a password ==

This is possible via the use of ssh keys. The process is thus:

1. place the public key for your user (root@mail) in the /root/.ssh/authorized_keys file on the server you wish to login to
cat /root/.ssh/id_dsa.pub
(paste that into authorized_keys on the target server). If the file doesn't exist, create it.

2. enable root login (usually only applies to FreeBSD). Edit the /etc/ssh/sshd_config on the target server and change:
<tt>#PermitRootLogin no</tt>
to
<tt>PermitRootLogin yes</tt>

3. Restart the sshd on the target machine. First, find the sshd process:
jailps <hostname> | grep sshd
or
vp <VEID> | grep sshd

Look for the process resembling:
root 17296 0.0 0.0 5280 1036 ? Ss 2011 4:27 /usr/sbin/sshd
(this is the sshd)

Not:
root 6270 0.5 0.0 6808 2536 ? Ss 14:33 0:00 sshd: root [priv]
(this is an sshd child- someone already ssh'd in as root)

Restart the sshd:
kill -1 <PID>

Ex:
kill -1 17296

You may now ssh in.

Once you're done, IF you enabled root login, you should repeat steps 2 and 3 to disable root logins.

== Letting someone in who has locked themselves out (killed sshd, lost pwd) ==

There are two ways people frequently lock themselves out - either they forget a password, or they kill off sshd somehow.

These are actually both fairly easy to solve. First, let's say someone kills off their sshd, or somehow mangles /etc/ssh/sshd_config such that it no longer lets them in.

Their email may be very short, or it may have all sorts of details about how you should fix sshd_config to let them in ... just ignore all of this. They can fix their own mangled sshd. Fixing this is very simple. First, edit the /etc/inetd.conf on their system and uncomment the telnet line:

telnet stream tcp nowait root /usr/libexec/telnetd telnetd
#telnet stream tcp6 nowait root /usr/libexec/telnetd telnetd

(just leave the tcp6 version of telnet commented)

Then, use jailps to list the processes on their system, and find their inetd process. Then simply:

kill -HUP (pid)

where (pid) is the PID of their inetd process. Now they have telnet running on their system and they can log in and do whatever they need to do.

The only complications that could occur are:

a) their firewall config on our firewall has port 23 blocked, in which case you will need to open that - will be covered in a different lesson.

b) they are not running inetd, so you can't HUP it. If this happens, edit their /etc/rc.conf, add the inetd_enable="YES" line, and then kill
their jail with /tmp/jailkill.pl - then restart their jail with the jail line from their quad/safe file. Easy.

If they have forgotten a password,

On 6.x+ you can reset their password with:
jexec <jailID from jls> passwd root

Note: the default password for 6.x jails is 8ico2987, for 4.x it is p455agfa

On 4.x, you need to cd to their etc directory
... for instance:

cd /mnt/data2/198.78.65.136-col00261-DIR/etc

and run:

vipw -d .

Then paste in these two lines (theres a paste with these):

root:$1$krszPxhk$xkCepSnz3mIikT3vCtJCt0:0:0::0:0:Charlie &:/root:/bin/csh
user:$1$Mx9p5Npk$QdMU6c8YQqp2FW2M3irEh/:1001:1001::0:0:User &:/home/user:/bin/sh

overwriting the lines they already have for "user" and "root" - then just tell them that both user and root have been reset to the default password of p455agfa.

For linux, just passwd inside shell or
vzctl set <veid> --userpasswd root:p455agfa –save

Starting in 2009 we began giving out randomized passwords for FreeBSD and Linux as the default password. That is stored with each system in Mgmt. You should look for and reset the password to that password in the event of a reset and refer the customer to use their original password from their welcome email- this way we don’t have to send the password again via email (in clear text).

== sendmail can’t be contacted from ext ip (only locally) ==

By default redhat puts this line in sendmail.mc:

DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')

which makes it only answer on localhost. Comment it out like:

dnl DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')

and then rebuild sendmail.cf with:

m4 /etc/mail/sendmail.mc > /etc/sendmail.cf

== virt doesn’t properly let go of ve’s ip(s) when moved to another system ==

On virtuozzo 2.6 systems, it's been observed that when moving ips from one virt to another that sometimes the routing table will not get updated to reflect the removal of the ip addresses.

A recent example was a customer that was moving to a new ve on a new virt and the ip addresses were traded between the two ve's. After the trade the two systems were not able to talk to each other. When looking at the routing table for the old system all the ip addresses were still in the routing table as being local, like so:

<pre>netstat -rn | grep 69.55.225.149
69.55.225.149 0.0.0.0 255.255.255.255 UH 40 0 0 venet0</pre>

This was preventing traffic to the other system from being routed properly.
The solution is to manually delete the route:

route delete 69.55.225.149 gw 0.0.0.0

Supposedly, this was fixed in 2.6.1

== sshd on FreeBSD 6.2 segfaults ==

First try to reinstall ssh

<pre>cd /usr/src/secure
cd lib/libssh
make depend && make all install
cd ../../usr.sbin/sshd
make depend && make all install
cd ../../usr.bin/ssh
make depend && make all install</pre>

Failing that, find the library that’s messed up:

<pre>ldd /usr/sbin/sshd
libssh.so.3 => /usr/lib/libssh.so.3 (0x280a3000)
libutil.so.5 => /lib/libutil.so.5 (0x280d8000)
libz.so.3 => /lib/libz.so.3 (0x280e4000)
libwrap.so.4 => /usr/lib/libwrap.so.4 (0x280f5000)
libpam.so.3 => /usr/lib/libpam.so.3 (0x280fc000)
libbsm.so.1 => /usr/lib/libbsm.so.1 (0x28103000)
libgssapi.so.8 => /usr/lib/libgssapi.so.8 (0x28112000)
libkrb5.so.8 => /usr/lib/libkrb5.so.8 (0x28120000)
libasn1.so.8 => /usr/lib/libasn1.so.8 (0x28154000)
libcom_err.so.3 => /usr/lib/libcom_err.so.3 (0x28175000)
libroken.so.8 => /usr/lib/libroken.so.8 (0x28177000)
libcrypto.so.4 => /lib/libcrypto.so.4 (0x28183000)
libcrypt.so.3 => /lib/libcrypt.so.3 (0x28276000)
libc.so.6 => /lib/libc.so.6 (0x2828e000)
libmd.so.3 => /lib/libmd.so.3 (0x28373000)</pre>

md5 them and compare to other jail hosts or jails running on host

for libcrypto reinstall:
<pre>/usr/src/crypto
make depend && make all install</pre>

== Moving a server to a different account in the management system ==

This is a tough one. You gotta touch a bunch of tables in the mysql database.
<pre>
systems
subs
ipmap
</pre>

basically, get the sysid and go off that:

<pre>
update systems set cid='newcid' where sysid=xxxxx
update subs set cid='newcid' where sysid=xxxxx
update ipmap set cid='newcid' where sysid=xxxxx
</pre>

You then need to rename the server disk space to change the old ID to the new one.

For a jail -
<pre>
stopjail <hostname>
mv 69.55.2XX.XXX-col0<oldid> 69.55.2XX.XXX-col0<newid>
mv 69.55.2XX.XXX-col0<oldid>-DIR 69.55.2XX.XXX-col0<newid>-DIR
vi /usr/local/jail/bin/quad1 [Update the ID in the quad file]
startjail <hostname>
</pre>

For a virt -
You will have to
* stop the virt
* rename the directories
* modify the config file /vzconf/<veid>.conf
* rename the virt conf file
* restart the new virt.

that ought to do it..

= FreeBSD VPS =

== Starting jails: Quad/Safe Files ==

FreeBSD customer systems do not start up automatically at boot time. When one of our freebsd machines boots up, it boots up, and does nothing else. To start jails, we put the commands to start each jail into a shell script(s) and run the script(s). Jail startup is something that needs to be actively monitored, which is why we don’t just run the script automatically. More on monitoring later.

NOTE: >=7.x we have moved to 1 quad file: <tt>quad1</tt>. Startups are not done by running each quad, but rather [[#startalljails|startalljails]] which relies on the contents of <tt>quad1</tt>. The specifics of this are lower in this article. What follows here applies for pre 7.x systems.

There are eight files in <tt>/usr/local/jail/rc.d</tt>:

<pre>jail3# ls /usr/local/jail/rc.d/
quad1 quad2 quad3 quad4 safe1 safe2 safe3 safe4
jail3#</pre>

four quad files and four safe files.

Each file contains an even number of system startup blocks (total number of jails divided by 4)

The reason for this is, if we make one large script to startup all the systems at boot time, it will take too long - the first system in the script will start up right after system boot, which is great, but the last system may not start for another 20 minutes.

Since there is no way to parralelize this during the startup procedure, we simply open four terminals (in screen window 9) and run each script, one in each terminal. This way they all run simultaneously, and the very last system in each startup script gets started in 1/4th the time it would if there was one large file

The files are generally organized so that quad/safe 1&2 have only jails from disk 1, and quad/safe 3&4 have jails from disk 2. This helps ensure that only 2 fscks on any disk are going on at once. Further, they are balanced so that all quad/safe’s finish executing around the same time. We do this by making sure each quad/safe has a similar number of jails and represents a similar number of inodes (see js).

The other, very important reason we do it this way, and this is the reason there are quad files and safe files, is that in the event of a system crash, every single vn-backed filesystem that was mounted at the time of system crash needs to be fsck'd. However, fsck'ing takes time, so if we shut the system down gracefully, we don't want to fsck.

Therefore, we have two sets of scripts - the four quad scripts are identical to the four safe scripts except for the fact that the quad scripts contain fsck commands for each filesystem.

So, if you shut a system down gracefully, start four terminals and run safe1 in window one, and safe2 in window 2, and so on.

If you crash, start four terminals (or go to screen window 9) and run quad1 in window one, and quad2 in window 2, and so on.

Here is a snip of (a 4.x version) quad2 from jail17:

<pre>vnconfig /dev/vn16 /mnt/data2/69.55.228.7-col00820
fsck -y /dev/vn16
mount /dev/vn16c /mnt/data2/69.55.228.7-col00820-DIR
chmod 0666 /mnt/data2/69.55.228.7-col00820-DIR/dev/null
jail /mnt/data2/69.55.228.7-col00820-DIR mail1.phimail.com 69.55.228.7 /bin/sh /etc/rc

# moved to data2 col00368
#vnconfig /dev/vn28 /mnt/data2/69.55.236.132-col00368
#fsck -y /dev/vn28
#mount /dev/vn28c /mnt/data2/69.55.236.132-col00368-DIR
#chmod 0666 /mnt/data2/69.55.236.132-col00368-DIR/dev/null
#jail /mnt/data2/69.55.236.132-col00368-DIR limehouse.org 69.55.236.132 /bin/sh /etc/rc

echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’
vnconfig /dev/vn22 /mnt/data2/69.55.228.13-col01063
fsck -y /dev/vn22
mount /dev/vn22c /mnt/data2/69.55.228.13-col01063-DIR
chmod 0666 /mnt/data2/69.55.228.13-col01063-DIR/dev/null
jail /mnt/data2/69.55.228.13-col01063-DIR www.widestream.com.au 69.55.228.13 /bin/sh /etc/rc

# cancelled col00106
#vnconfig /dev/vn15 /mnt/data2/69.55.238.5-col00106
#fsck -y /dev/vn15
#mount /dev/vn15c /mnt/data2/69.55.238.5-col00106-DIR
#chmod 0666 /mnt/data2/69.55.238.5-col00106-DIR/dev/null
#jail /mnt/data2/69.55.238.5-col00106-DIR mail.azebu.net 69.55.238.5 /bin/sh /etc/rc</pre>

As you can see, two of the systems specified are commented out - presumably those customers cancelled, or were moved to new servers.

As you can see, the vnconfig line is the simpler command line, not the longer one that was used when it was first configured. As you can see, all that is done is, vnconfig the filesystem, then fsck it, then mount it. The fourth command is the `jail` command used to start the system – but that will be covered later.

Here is the safe2 file from jail17:

<pre>vnconfig /dev/vn16 /mnt/data2/69.55.228.7-col00820
mount /dev/vn16c /mnt/data2/69.55.228.7-col00820-DIR
chmod 0666 /mnt/data2/69.55.228.7-col00820-DIR/dev/null
jail /mnt/data2/69.55.228.7-col00820-DIR mail1.phimail.com 69.55.228.7 /bin/sh /etc/rc

# moved to data2 col00368
#vnconfig /dev/vn28 /mnt/data2/69.55.236.132-col00368
#mount /dev/vn28c /mnt/data2/69.55.236.132-col00368-DIR
#chmod 0666 /mnt/data2/69.55.236.132-col00368-DIR/dev/null
#jail /mnt/data2/69.55.236.132-col00368-DIR limehouse.org 69.55.236.132 /bin/sh /etc/rc

echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’
vnconfig /dev/vn22 /mnt/data2/69.55.228.13-col01063
mount /dev/vn22c /mnt/data2/69.55.228.13-col01063-DIR
chmod 0666 /mnt/data2/69.55.228.13-col01063-DIR/dev/null
jail /mnt/data2/69.55.228.13-col01063-DIR www.widestream.com.au 69.55.228.13 /bin/sh /etc/rc

# cancelled col00106
#vnconfig /dev/vn15 /mnt/data2/69.55.238.5-col00106
#mount /dev/vn15c /mnt/data2/69.55.238.5-col00106-DIR
#chmod 0666 /mnt/data2/69.55.238.5-col00106-DIR/dev/null
#jail /mnt/data2/69.55.238.5-col00106-DIR mail.azebu.net 69.55.238.5 /bin/sh /etc/rc</pre>

As you can see, it is exactly the same, but it does not have the fsck lines.

Take a look at the last entry - note that the file is named:

/mnt/data2/69.55.238.5-col00106

and the mount point is named:

/mnt/data2/69.55.238.5-col00106-DIR

This is the general format on all the FreeBSD systems. The file is always named:

IP-custnumber

and the directory is named:

IP-custnumber-DIR

If you run safe when you need a fsck, the mount will fail and jail will fail:

# mount /dev/vn1c /mnt/data2/jails/65.248.2.131-ns1.kozubik.com-DIR
mount: /dev/vn1c: Operation not permitted

No reboot needed, just run the quad script

Starting with 6.x jails, we added block delimiters to the quad/safe files, the block looks like:

<pre>echo '## begin ##: nuie.solaris.mu'
fsck -y /dev/concat/v30v31a
mount /dev/concat/v30v31a /mnt/data1/69.55.228.218-col01441-DIR
mount_devfs devfs /mnt/data1/69.55.228.218-col01441-DIR/dev
devfs -m /mnt/data1/69.55.228.218-col01441-DIR/dev rule -s 3 applyset
jail /mnt/data1/69.55.228.218-col01441-DIR nuie.solaris.mu 69.55.228.218 /bin/sh /etc/rc
echo '## end ##: nuie.solaris.mu'</pre>

These are more than just informative when running quad/safe’s, the echo lines MUST be present for certain tools to work properly. So it’s important that any updates to the hostname also be updated on the 2 echo lines. For example, if you try to startjail a jail with a hostname which is on the jail line but not the echo lines, the command will return with host not found.

=== FreeBSD 7.x+ notes ===

Starting with the release of FreeBSD 7.x, we are doing jail startups in a slightly different way. First, thereis only 1 file: <tt>/usr/local/jail/rc.d/quad1</tt>
There are no other quads or corresponding safe files. The reason for this is twofold, 1. We can pass –C to fsck which will tell is to skip the fsck if the fs is clean (no more need for safe files), 2. We have a new startup script which can be launched multiple times, running in parallel to start jails, where quad1 is the master jail file.
Quad1 could still be run as a shell script, but it would take a very long time for it to run completely so it’s not advisable; or you should break it down into smaller chunks (like quad1, quad2, quad3, etc)

Here is a snip of (a 7.x version) quad1 from jail2:

<pre>echo '## begin ##: projects.tw.com'
mdconfig -a -t vnode -f /mnt/data1/69.55.230.46-col01213 -u 50
fsck -Cy /dev/md50c
mount /dev/md50c /mnt/data1/69.55.230.46-col01213-DIR
mount -t devfs devfs /mnt/data1/69.55.230.46-col01213-DIR/dev
devfs -m /mnt/data1/69.55.230.46-col01213-DIR/dev rule -s 3 applyset
jail /mnt/data1/69.55.230.46-col01213-DIR projects.tw.com 69.55.230.46 /bin/sh /etc/rc
echo '## end ##: projects.tw.com'</pre>

Cancelled jails are no longer commented out and stored in quad1, rather they’re moved to <tt>/usr/local/jail/rc.d/deprecated</tt>

To start these jails, start the 4 ssh sessions as you would for a normal crash and then instead of running quad1-4, instead run startalljails in each window. IMPORTANT- before running startalljails you should make sure you ran preboot once as it will clear out all the lockfiles and enable startalljails to work properly.

== Problems with the quad/safe files ==

When you run the quad/safe files, there are two problems that can occur - either a particular system will hang during initialization, OR a system will spit out output to the screen, impeding your ability to do anything. Or both.

First off, when you start a jail, you see output like this:

<pre>Skipping disk checks ...
adjkerntz[25285]: sysctl(put_wallclock): Operation not permitted
Doing initial network setup:.
ifconfig: ioctl (SIOCDIFADDR): permission denied
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384
Additional routing options: TCP keepalive=YESsysctl:
net.inet.tcp.always_keepalive: Operation not permitted.
Routing daemons:.
Additional daemons: syslogd.
Doing additional network setup:.
Starting final network daemons:.
ELF ldconfig path: /usr/lib /usr/lib/compat /usr/X11R6/lib /usr/local/lib
a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout /usr/X11R6/lib/aout
Starting standard daemons: inetd cron sshd sendmail sendmail-clientmqueue.
Initial rc.i386 initialization:.
Configuring syscons: blanktime.
Additional ABI support:.
Local package initialization:.
Additional TCP options:.</pre>

Now, let's look at this line, near the end:

Local package initialization:.

This is where a list of daemons that are set to start at boot time willshow up. You might see something like:

Local package initialization: mysqld apache sendmail sendmail-clientmqueue

Or something like this:

Local package initialization: postgres postfix apache

The problem is that many systems (about 4-5 per machine) will hang on that line. Basically it will get to some of the way through the total daemons to be started:

Local package initialization: mysqld apache

and will just sit there. Forever.

Fortunately, pressing ctrl-c will break out of it. Not only will it break out of it, but it will also continue on that same line and start the other daemons:

Local package initialization: mysqld apache ^c sendmail-clientmqueue

and then continue on to finish the startup, and then move to the next system to be started.

So what does this mean? It means that if a machine crashes, and you start four screen-windows to run four quads or four safes, you need to periodically cycle between them and see if any systems are stuck at that point, causing their quad/safe file to hang. A good rule of thumb is, if you see a system at that point in the startup, give it another 100 seconds - if it is still at the exact same spot, hit ctrl-c. Its also a good idea to go back into the quad file (just before the first command in the jail startup block) and note that this jail tends to need a control-c or more time as follows:
<pre>echo '### NOTE ### slow sendmail'
echo '### NOTE ###: ^C @ Starting sendmail.'</pre>

'''NEVER''' hit ctrl-c repeatedly if you don't get an immediate response - that will cause the following jail’s startup commands to be aborted.

A second problem that can occur is that a jail - maybe the first one in that particular quad/safe, maybe the last one, or maybe one in the middle, will start spitting out status or error messages from one of its init scripts. This is not a problem - basically, hit enter a few times and see if you get a prompt - if you do get a prompt, that means that the quad/safe script has already completed. Therefore it is safe to log out (and log out of the user that you su'd from) and then log back in (if necessary).

The tricky thing is, if a system in the middle starts flooding with messages, and you hit enter a few times and don't get a prompt. Are you not getting a prompt because some subsequent system is hanging at the initialization, as we discussede above ? Or are you not getting a prompt because that quad file is currently running an fsck ? Usually you can tell by scrolling back in screen’s history to see what it was doing before you started getting the messages.

If you don’t get clues from history, you have to use your judgement - instead of giving it 100 seconds to respond, perhaps give it 2-3 mins ... if you still get no response (no prompt) when you hit enter, hit ctrl-c. However, be aware that you might still be hitting ctrl-c in the middle of an fsck. This means you will get an error like "filesystem still marked dirty" and then the vnconfig for it will fail and so will the jail command, and the next system in the quad file will then start starting up.

If this happens, just wait until the end of all the quad files have finished, and start that system manually.

If things really get weird, like a screen flooded with errors, and you can't get a prompt, and ctrl-c does nothing, then you need to just eventually (give it ten mins or so) just kill that window with ctrl-p, then k, and then log in again and manually check which systems are now running and which aren't, and manually start up any that are not.

Don't EVER risk running a particular quad/safe file a second time.
If the quad/safe script gets executed twice, reboot the machine immediately.

So, for all the above reasons, anytime a machine crashes and you run all the quads or all the safes, '''always''' check every jail afterwards to make sure it is running - even if you have no hangs or complications at all.
Run this command:

<tt>[[#jailpsall|jailpsall]]</tt>

Note: [[#postboot|postboot]] also populates ipfw counts, so it '''should not be run multiple times''', use <tt>jailpsall</tt> for subsequent extensive ps’ing

And make sure they all show as running. If one does not show as running, check its /etc/rc.conf file first to see if maybe it is using a different hostname first before starting it manually.

One thing we have implemented to alleviate these startup hangs and noisy jails, is to put jail start blocks that are slow or hangy at the bottom of the safe/quad file. Further, for each bad jail we note in each quad/safe just before the start block something like:

echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’

That way we’ll be prepared to ^C when we see that message appear during the quad/safe startup process. If you observe a new, undocumented hang, '''after''' the quad/safe has finished, place a line similar to the above in the quad file, move the jail start block to the end of the file, then run [[#buildsafe|buildsafe]]

== Making new customer jail 4.x ==

1. run [[#js|js]] to figure out which partition, IP to put it on, find an unused vn, and choose which quad it should go in

2. use col00xxx for both hostnames if they don’t give you a hostname

3. copy over dir and ip to pending customer screen

Usage: <tt>jailmake IP filepath vnX hostname shorthost quad/safe# ipfw# email [gb disk, default 4]</tt>

jail14# /tmp/jailmake 69.55.226.152 /mnt/data2/69.55.226.152-col00182 vn23 negev.cerastes.org negev 4 182 cerastes@cerastes.org

== Making new customer jail 6.x ==

1. run [[#js|js]] to figure out which gvinum vols are free (and which mnt (disk) each gvinum goes with/to), IP to put it on and choose which quad it should go in

2. use col00xxx for both hostnames if they don’t give you a hostname

3. copy over dir and ip to pending customer screen

Usage: <tt>jailmake IP filepath vN[,vN] hostname shorthost quad/safe# ipfw# email</tt>

jail19# jailmake 69.55.236.17 /mnt/data1/69.55.236.17-col01574 v10 uncle-leo.commadev.com uncle-leo 1 1574 lee@commadev.com,lee@gmail.com

== Changing an ip for freebsd VPS ==

*stopjail (hostname)
*on systems using vnfile or mdfile: rename vnfile/mdfile, using new ip
*rename dir using new ip
*Edit quad (make sure to update all lines: directory, vn/mdfile)
*buildsafe (not necessary on systems having only quad1)
*if you're adding an ip not already on the host machine: <tt>ipadd x.x.x.x</tt>
*startjail (hostname)
if backups: <tt>mvbackups</tt> ...
*Edit dir/ip in mgmt
*if any rules, update any firewall rules to use new ip (careful not to make sure there arent multiple rules per ipfw#- search by colo#)
*(if customer asks and has domains) update their domains on ns1c
*optional: update ipfw counters on the host server

== Rename a gconcat vol ==

*stop jail, unmount everything
*gconcat stop vol
*gconcat clear /dev/gvinum/vN
*gconcat clear /dev/gvinum/vN+1
*gconcat label -v newname /dev/gvinum/vN /dev/gvinum/vN+1
*bsdlabel /dev/concat/newname
(make sure a partition is not fstype unused, if so change to: 4.2BSD)

== Remaking a VPS (on same jail) ==

# take him down (stopjail)
# make note of host, vn/gvin/md, ip
# remove from quad/safe
# move vnfile (rename)
# remove dir (jailmake will make a new one)
# remove ipfw counts
# jailmake
# update backup script
# update db with new dir/ip (use the “move” link, mark old as stopped today)
# update firewall if changed ip

== Recovering from a crash (FreeBSD) ==

=== Diagnose whether you have a crash ===
The most important thing is to get the machine and all jails back up as soon as possible. Note the time, you’ll need to create a crash log entry (Mgmt. -> Reference -> CrashLog). The first thing to do is head over to the [[Screen#Screen_Organization|serial console screen]] and see if there’s any kernel error messages output. Try to copy any messages (or just a sample of repeating messages) you see into the notes section of the crash log. If there are no messages, the machine may just be really busy- wait a bit (5-10min) to see if it comes back. If it's still pinging, odds are its very busy. Note, if you see messages about swap space exhausted, the server is obviously out of memory, however it may recover briefly enough for you to get a jtop in to see who's lauched a ton of procs (most likely) and then issue a quick jailkill to get it back under control.

If it doesn't come back, or the messages indicate a fatal error, you will need to proceed with a power cycle (ctrl+alt+del will not work).

=== Power cycle the server ===
If this machine is not a Dell 2950 with a [[DRAC/RMM#DRAC|DRAC card]] (i.e. if you can’t ssh into the DRAC card (as root, using the standard root pass) and issue
racadm serveraction hardreset
then you will need someone at the data center power the macine off, wait 30 sec, then turn it back on. Make sure to re-attach via console:
tip jailX
immediately after power down.

=== (Re)attach to the console ===
Stay on the console the entire time during boot. As the BIOS posts- look out for the RAID card output- does everything look healthy? The output may be scrambled, look for "DEGRADED" or "FAILED". Once the OS starts booting you will be disconnected (dropped back to the shell on the console server) a couple times during the boot up. The reason you want to quickly re-attach is two-fold: 1. If you don’t reattach quickly then you won’t get any console output, 2. you want to be attached before the server ''potentially'' starts (an extensive) fsck. If you attach after the fsck begins, you’ll have seen no indication it started an fsck and the server will appear frozen during startup- no output, no response.

IMPORTANT NOTE: on some older FreeBSD systems, there will be no output to the video (KVM) console as it boots up. The console output is redirected to the serial port ... so if a jail crashes, and you attach a kvm, the output during the bootup procedure will not be shown on the screen. However, when the bootup is done, you will get a login prompt on the screen and will be able to log in as normal. <tt>/boot/loader.conf</tt> is where serial console redirect output lives, so comment that if you want to catch output on kvm.
On newer systems it sends most output to both locations.

=== Assess the heath of the server ===
Once the server boots up fully, you should be able to ssh in. Look around- make sure all the mounts are there and reporting the correct size/usage (i.e. /mnt/data1 /mnt/data2 /mnt/data3 - look in /etc/fstab to determine which mount points should be there), check to see if RAID mirrors are healthy. See [[RAID_Cards#Common_CLI_commands_.28megacli.29|megacli]], [[#aaccheck|aaccheck]]

Before you start the jails, you need to run [[#preboot|preboot]]. This will do some assurance checks to make sure things are prepped to start the jails. Any issues that come out of preboot need to be addressed before starting jails.

=== Start jails ===
[[#Starting_jails:_Quad.2FSafe_Files|More on starting jails]]
Customer jails (the VPSs) do not start up automatically at boot time. When a FreeBSD machines boots up, it boots up, and does nothing else. To start jails, we put the commands to start each jail into a shell script(s) and run the script(s). Jail startup is something that needs to be actively monitored, which is why we don’t just run the script automatically.

In order to start jails, we run the quad files: quad1 quad2 quad3 and quad4 (on new systems there is only quad1). If the machine was cleanly rebooted- which wouldn't be the case if this was a crash, you may run the safe files (safe1 safe2 safe3 safe4) in lieu of quads.

Open up 4 logins to the server (use the windows in [[Screen#Screen_Organization|a9]])
In each of the 4 windows you will:

If there is a [[#startalljails|startalljails]] script (and only quad1), run that command in each of the 4 windows. It will parse through the quad1 file and start each jail. Follow the instructions [[#Problems_with_the_quad.2Fsafe_files|here]] for monitoring startup. Note that you can be a little more lenient with jails that take awhile to start- startalljails will work around the slow jails and start the rest. As long as there aren't 4 jails which are "hung" during startup, the rest will get started eventually.
-or-
If there is no startalljails script, there will be multiple quad files. In each of the 4 windows, start each of the quads. i.e. start quad1 in window1, quad2 in window2 and so on. DO NOT start any quad twice. It will crash the server. If you accidentally do this, just jailkill all the jails which are in the quad and run the quad again. Follow the instructions here for monitoring quad startup.

Note the time the last jail boots- this is what you will enter in the crash log.

Save the crash log.

=== Check to make sure all jails have started ===
There's a simple script which will make sure all jails have started, and enter the ipfw counter rules: [[#postboot|postboot]]
Run postboot, which will do a jailps on each jail it finds (excluding commented out jails) in the quad file(s). We're looking for 2 things:
# systems spawning out of control or too many procs
# jails which haven't started
On 7.x and newer systems it will print out the problems (which jails haven't started) at the conclusion of postboot.
On older systems you will need to watch closely to see if/when there's a problem, namely:

[hostname] doesnt exist on this server

When you get this message, it means one of 2 things:
1. the jail really didn't start:
When a jail doesn't start it usually boils down to a problem in the quad file. Perhaps the path name is wrong (data1 vs data2) or the name of the vn/mdfile is wrong. Once this is corrected, you will need to run the commands from the quad file manually, or you may use <tt>startjail <hostname></tt>

2. the customer has changed their hostname (and not told us) so their jail ''is'' running, just under a different hostname:
On systems with jls, this is easy to rectify. First, get the customer info: <tt>g <hostname></tt>
Then look for the customer in jls: <tt>jls | grep <col0XXXX></tt>
From there you will see their new hostname- you should update that hostname in the quad file: don't forget to edit it on the <tt>## begin ##</tt> and <tt>## end ##</tt> lines, and in mgmt.
On older systems without jls, this will be harder, you will need to look further to see their hostname- perhaps its in their /etc/rc.conf

Once all jails are started, do some spot checks- try to ssh or browse to some customers, just to make sure things are really ok.

== Adding disk to a 7.x/8.x jail ==
or
== Moving customer to a different drive (md) ==

NOTE: this doesn’t apply to mx2 which uses gvinum. Use same procedure as 6.x
NOTE: if you unmount before mdconfig, re-mdconfig (attach) then unmount then mdconfig -u again

(parts to change/customize are <tt>red</tt>)

If someone wants more disk space, there’s a paste for it, send it to them (explains about downtime, etc).

1. Figure out the space avail from <tt>js</tt>. Ideally, you want to put the customers new space on a different partition (and create the new md on the new partition).

2. make a mental note of how much space they're currently using

3. <tt>jailkill <hostname></tt>

4. Umount it (including their devfs) but leave the md config’d (so if you use stopjail, you will have to re-mdconfig it)

5. <tt>g <customerID></tt> to get the info (IP/cust#) needed to feed the new mdfile and mount name, and to see the current md device.

6a. When there's enough room to place new system on an alternate, or the same drive:
USE CAUTION not to overwrite (touch, mdconfig) existing md!! 
<tt>touch /mnt/data3/69.55.234.66-col01334 
mdconfig -a -t vnode -s 10g -f /mnt/data3/69.55.234.66-col01334 -u 97 
newfs /dev/md97</tt>

Optional- if new space is on a different drive, move the mount point directory AND use that directory in the mount and cd commands below: 
<tt>mv /mnt/data1/69.55.234.66-col01334-DIR /mnt/data3/69.55.234.66-col01334-DIR</tt>

<tt>mount /dev/md97 /mnt/data3/69.55.234.66-col01334-DIR 
cd /mnt/data3/69.55.234.66-col01334-DIR</tt>

confirm you are mounted to /dev/md97 and space is correct: 
<tt>df .</tt>

do the dump and pipe directly to restore: 
<tt>dump -0a -f - /dev/md1 | restore -r -f - 
rm restoresymtable</tt>

when dump/restore completes successfully, use <tt>df</tt> to confirm the restored data size matches the original usage figure

md-unconfig old system: 
<tt>mdconfig -d -u 1</tt>

archive old mdfile. 
<tt>mv /mnt/data1/69.55.237.26-col00241 /mnt/data1/old-col00241-mdfile-noarchive-20091211</tt>

edit quad (vq1) to point to new (/mnt/data3) location AND new md number (md97)

restart the jail: 
<tt>startjail <hostname></tt>

6b. When there's not enough room on an alternate partition, or on the same drive...but there is enough room if you were to remove the existing customer's space:

mount backup nfs mounts: 
<tt>mbm</tt> 
(run <tt>df</tt> to confirm backup mounts are mounted)

dump the customer to backup2 or backup1 
<tt>dump -0a -f /backup4/col00241.20120329.noarchive.dump /dev/md1</tt> 
(when complete WITHOUT errors, <tt>du</tt> the dump file to confirm it matches size, roughly, with usage)

unconfigure and remove old mdfile 
<tt>mdconfig -d -u 1 
rm /mnt/data1/69.55.237.26-col00241</tt> 
(there should now be enough space to recreate your bigger system. If not, run sync a couple times)

create the new system (ok to reuse old mdfile and md#): 
<tt>touch /mnt/data1/69.55.234.66-col01334 
mdconfig -a -t vnode -s 10g -f /mnt/data1/69.55.234.66-col01334 -u 1 
newfs /dev/md1 
mount /dev/md1 /mnt/data1/69.55.234.66-col01334-DIR 
cd /mnt/data1/69.55.234.66-col01334-DIR</tt>

confirm you are mounted to /dev/md1 and space is correct: 
<tt>df .</tt>

do the restore from the dumpfile on the backup server: 
<tt>restore -r -f /backup4/col00241.20120329.noarchive.dump . 
rm restoresymtable</tt>

when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure

umount nfs: 
<tt>mbu</tt>

If md# changed (or mount point), edit quad (<tt>vq1</tt>) to point to new (/mnt/data3) location AND new md number (md1)

restart the jail: 
<tt>startjail <hostname></tt>

7. update disk (and dir if applicable) in mgmt screen

8. update backup list AND move backups, if applicatble

Ex: <tt>mvbackups 69.55.237.26-col00241 jail9 data3</tt>

9. Optional: archive old mdfile

<tt>mbm 
gzip -c old-col01588-mdfile-noarchive-20120329 > /deprecated/old-col01588-mdfile-noarchive-20120329.gz 
mbu 
rm old-col01588-mdfile-noarchive-20120329</tt>

== Adding disk to a 6.x jail (gvinum/gconcat) ==
or
== Moving customer to a different drive (gvinum/gconcat) ==

(parts to change are highlighted)

If someone wants more disk space, there’s a paste for it, send it to them (explains about downtime, etc).

1. Figure out the space avail from [[#js|js]]. Ideally, you want to put the customers new space on a different partition (and create the new md on the new partition).

2. make a mental note of how much space they're currently using

3. <tt>[[#stopjail|stopjail]] <hostname></tt>

4. <tt>[[#g|g]] <customerID></tt> to get the info (IP/cust#) needed to feed the new mount name and existing volume/device.

5a. When there's enough room to place new system on an alternate, or the same drive (using only UNUSED - including if it's in use by the system in question - gvinum volumes):

Configure the new device: 
A. for a 2G system (single gvinum volume): 
<tt>bsdlabel -r -w /dev/gvinum/v123 
newfs /dev/gvinum/v123a</tt> 

-or-

B. for a >2G system (create a gconcat volume): 
try to grab a contiguous block of gvinum volumes. gconcat volumes MAY NOT span drives (i.e. you cannot use a gvinum volume from data3 and a volume from data2 in the same gconcat volume).

<tt>gconcat label v82-v84 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84 
bsdlabel -r -w /dev/concat/v82-v84 
newfs /dev/concat/v82-v84a</tt>

Other valid gconcat examples: 
<tt>gconcat label v82-v84v109v112 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84 /dev/gvinum/v109 /dev/gvinum/v112 
gconcat label v82v83 /dev/gvinum/v82 /dev/gvinum/v83</tt> 
Note, long names will truncate: v144v145v148-v115 will truncate to v144v145v148-v1 (so you will refer to it as v144v145v148-v1 thereafter)

Optional- if new volume is on a different drive, move the mount point directory (get the drive from js output) AND use that directory in the mount and cd commands below: 
<tt>mv /mnt/data1/69.55.234.66-col01334-DIR /mnt/data3/69.55.234.66-col01334-DIR</tt>

confirm you are mounted to the device (<tt>/dev/gvinum/v123a</tt> OR <tt>/dev/concat/v82-v84</tt>) and space is correct: 
A. <tt>mount /dev/gvinum/v123a /mnt/data3/69.55.234.66-col01334-DIR</tt> 
-or- 
B. <tt>mount /dev/concat/v82-v84a /mnt/data3/69.55.234.66-col01334-DIR</tt>

<tt>cd /mnt/data3/69.55.234.66-col01334-DIR 
df .</tt>

do the dump and pipe directly to restore: 
<tt>dump -0a -f - /dev/gvinum/v1 | restore -r -f - 
rm restoresymtable</tt>

when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure

edit quad (<tt>vq1</tt>) to point to new (/mnt/data3) location AND new volume (<tt>/dev/gvinum/v123a</tt> or <tt>/dev/concat/v82-v84a</tt>) , run <tt>buildsafe</tt>

restart the jail: 
<tt>startjail <hostname></tt>

5b. When there's not enough room on an alternate partition, or on the same drive...but there is enough room if you were to remove the existing customer's space (i.e. if you want/need to reuse the existing gvinum volumes and add on more):

mount backup nfs mounts: 
<tt>mbm</tt> 
(run df to confirm backup mounts are mounted)

dump the customer to backup2 or backup1 
<tt>dump -0a -f /backup4/col00241.20120329.noarchive.dump /dev/gconcat/v106-v107</tt> 
(when complete WITHOUT errors, du the dump file to confirm it matches size, roughly, with usage)

unconfigure the old gconcat volume 
list member gvinum volumes:

<tt>gconcat list v106v107</tt>

Output will resemble: 
<pre>Geom name: v106v107
State: UP
Status: Total=2, Online=2
Type: AUTOMATIC
ID: 3530663882
Providers:
1. Name: concat/v106v107
Mediasize: 4294966272 (4.0G)
Sectorsize: 512
Mode: r1w1e2
Consumers:
1. Name: gvinum/sd/v106.p0.s0
Mediasize: 2147483648 (2.0G)
Sectorsize: 512
Mode: r1w1e3
Start: 0
End: 2147483136
2. Name: gvinum/sd/v107.p0.s0
Mediasize: 2147483648 (2.0G)
Sectorsize: 512
Mode: r1w1e3
Start: 2147483136
End: 4294966272</pre>

stop volume and clear members 
<tt>gconcat stop v106v107 
gconcat clear gvinum/sd/v106.p0.s0 gvinum/sd/v107.p0.s0</tt>

create new device- and its ok to reuse old/former members 
try to grab a contiguous block of gvinum volumes. gconcat volumes MAY NOT span drives (i.e. you cannot use a gvinum volume from data3 and a volume from data2 in the same gconcat volume). 
<tt>gconcat label v82-v84v106v107 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84 /dev/gvinum/v106 /dev/gvinum/v107 
bsdlabel -r -w /dev/concat/v82-v84v106v107 
newfs /dev/concat/v82-v84v106v107a</tt>

Optional- if new volume is on a different drive, move the mount point directory (get the drive from js output) AND use that directory in the mount and cd commands below: 
<tt>mv /mnt/data1/69.55.234.66-col01334-DIR /mnt/data3/69.55.234.66-col01334-DIR</tt>

confirm you are mounted to the device (/dev/concat/v82-v84v106v107) and space is correct: 
<tt>mount /dev/concat/v82-v84v106v107a /mnt/data3/69.55.234.66-col01334-DIR</tt>

<tt>cd /mnt/data3/69.55.234.66-col01334-DIR 
df .</tt>

do the restore from the dumpfile on the backup server: 
<tt>restore -r -f /backup4/col00241.20120329.noarchive.dump . 
rm restoresymtable</tt>

when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure

edit quad (<tt>vq1</tt>) to point to new (/mnt/data3) location AND new volume (<tt>/dev/concat/v82-v84v106v107a</tt>), run buildsafe

restart the jail: 
<tt>startjail <hostname></tt>

TODO: clean up/clear old gvin/gconcat vol

6. update disk (and dir if applicable) in mgmt screen

7. update backup list AND move backups, if applicatble

Ex: <tt>mvbackups 69.55.237.26-col00241 jail9 data3</tt>

DEPRECATED - steps to tack on a new gvin to existing gconcat- leads to corrupted fs
bsdlabel -e /dev/concat/v82-v84

To figure out new size of the c partition, multiply 4194304 by the # of 2G gvinum volumes and subtract the # of 2G volumes:
10G: 4194304 * 5 – 5 = 20971515
8G: 4194304 * 4 – 4 = 16777212
6G: 4194304 * 3 – 3 = 12582909
4G: 4194304 * 2 – 2 = 8388606

To figure out the new size of the a partition, subtract 16 from the c partition:
10G: 20971515 – 16 = 20971499
8G: 16777212 – 16 = 16777196
6G: 12582909 – 16 = 12582893
4G: 8388606 – 16 = 8388590

Orig:
8 partitions:
# size offset fstype [fsize bsize bps/cpg]
a: 8388590 16 4.2BSD 2048 16384 28552
c: 8388606 0 unused 0 0 # "raw" part, don't edit

New:
8 partitions:
# size offset fstype [fsize bsize bps/cpg]
a: 12582893 16 4.2BSD 2048 16384 28552
c: 12582909 0 unused 0 0 # "raw" part, don't edit

sync; sync

growfs /dev/concat/v82-v84a

fsck –fy /dev/concat/v82-v84a

sync

fsck –fy /dev/concat/v82-v84a

(keep running fsck’s till NO errors)

== Adding disk to a 4.x jail ==

NOTE: if you unmount before vnconfig, re-vnconfig then unmount then vnunconfig

If someone wants more disk space, there’s a paste for it, send it to them.

1. Figure out the space avail from [[#js|js]]. Ideally, you want to dump to a different partition and create the new vn on the current partition. If no space to make make dump, then move old vn’s to backup or worse, dump to backup.

2. <tt>g <customerID></tt> to get the info

3. make vnfile, newfs and mount. USE CAUTION not to overwrite existing vn. 
Ex: <tt>vnconfig –T -S 9g -s labels -c /dev/vn22 /mnt/data2/65.214.160.117-col00496 
disklabel -r -w vn30 auto 
newfs /dev/vn30c 
mkdir /mnt/data2/65.214.160.117-col00496-DIR 
mount /dev/vn30c /mnt/data2/69.55.237.26-col00241-DIR 
cd /mnt/data2/65.214.160.117-col00496-DIR</tt>

4. <tt>jailkill <hostname></tt>

5. <tt>dump -0a -f - /dev/vn1 | restore -r -f -</tt>

6. <tt>rm restoresymtable</tt>

7. unmount and unvnconfig old system: 
<tt>umount /dev/vnNNc 
vnconfig –u /dev/vnNN</tt>

8. move vnfile. Ex: <tt>mv /mnt/data1/69.55.237.26-col00241 /mnt/data1/old-col00241-vnfile-20110312-noarchive</tt>

9. remove old dir: <tt>rmdir /mnt/data1/69.55.237.26-col00241-DIR</tt>

10. edit quad/safe to point to new location, run <tt>[[#buildsafe|buildsafe]]</tt>

11. start jail: <tt>[[#startjail|startjail]] <hostname></tt>

12. update disk (and dir if applicable) in mgmt screen

13. mv backups if necessary

WARNING: if you restore to unmounted vn then you’re actually restoring to /mnt/data1
you can tar and do above to recover

NOTE: you can’t move a customer to a system that has a base sys that is diff from the one they came from (BSD 4.8 to 4.5 = won’t work)

NOTE: we don’t mount procfs's to anyone’s system by default.

NOTE: mount_nulls:
also goofy mount_nulls seen in df -k on jail2 
also needs to be done in safe and quad 
if he wants another, use safe/quad as example and add it to file. also exec command at prompt to add mount (no reboot, remount or rejail necessary)

== Moving customer to another jail machine ==

Systems should only be moved between similarly-versioned jails. If you cannot get a match, you should go to freebsd.org and see what the differences were between the version you're moving from and the new version. Many of the updates will be to drivers and kernels - these don't affect the customer. Perhaps they updated traceroute or a library...you should copy the new/changed files into the VPS/jail on the new host. An alternate method for moving to a different version would be to buildworld to bring their full distribution up to date. Optional steps for this kind of upgrade are included below.

1. <tt>g <customerID></tt>

2. <tt>[[#jailkill|jailkill]] <hostname></tt>

3. create new device on target system 
4.x: 
<pre>vnconfig –T –S 4g –s labels –c /dev/vn1 /mnt/data1/69.55.22x.x-col00XXX
disklabel –r –w vn1 auto
newfs /dev/vn1c</pre>

6.x: 
<pre>bsdlabel -r -w /dev/gvinum/v1
newfs /dev/gvinum/v1a
or
gconcat label v1-v3 /dev/gvinum/v1 /dev/gvinum/v2 /dev/gvinum/v3
bsdlabel -r -w /dev/concat/v1-v3
newfs /dev/concat/v1-v3a</pre>

7.x+: 
Run jailmakeempty then skip step 4 & 7 below

4. make and mount the dir on new system. 
Ex: <pre>mkdir /mnt/data2/69.55.230.3-col00123-DIR
chmod 755 /mnt/data2/69.55.230.3-col00123-DIR
mount [device] /mnt/data2/69.55.230.3-col00123-DIR</pre>

5. [[#stopjail|stopjail]] <hostname> 1

6. dump fs to new system. ex: <tt>[[#dumpremoterestore|dumpremoterestore]] /dev/vn51 10.1.4.118 /mnt/data2/69.55.239.45-col00688-DIR</tt> (make sure you can ssh as root on remote machine)

6a. OPTIONAL BUILDWORLD: 
cd /usr/src
make world DESTDIR=/mnt/data2/69.55.xxx.xx-col0xxxx-DIR

cd etc
make distribution DESTDIR=/mnt/data2/69.55.xxx.xx-col0xxxx-DIR

(you may have to rm an openssh file and re-make dist)

rm -rf /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc/periodic/daily/400.status-disks
vi /etc/periodic/security/100.chksetuid
replace: <tt>MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`</tt> 
with: <tt>MP='/' (use single quotes)</tt>

Ask user if they want ports overwritten with current, if yes:
cp -r /usr/ports /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/usr

on source:
cd /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc; vipw –d .
(copy in all info)

on target:
cd /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc; vipw –d .
(paste all info)

on source:
cat /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc/group
(copy in all info)

on target:
cat > /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc/group
(paste all info)

7. edit quad on source system copy over entries for the jail to the target system
take care that the vn/gvinum/gconcat/md devices on the target system are’t in use and that the /mnt/dataN path doesn’t need changing

8. run [[#buildsafe|buildsafe]] (if this is <=6.x) on target system (copies changes made to quad into safe file)

9. remove ip from source system. Ex: <tt>ipdel 69.55.230.3</tt>

10. add ip to target system. Ex: <tt>ipadd 69.55.230.3</tt>

11. start new system, with [[#startjail|startjail]] or manually, pasting entries found from running <tt>g <customerID></tt> on the new system

12. run <tt>[[#canceljail|canceljail]] col0xxxx</tt> on source system. This should prompt you to remove backups if any existed. Do not let it do this, rather:

13. if backups existed move them to the new host via the <tt>[[#mvbackups_.28freebsd.29|mvbackups]] script</tt>

14. edit quad on source system, edit comment to reflect a move rather than a cancel, ex: <tt># moved to jail2 col00241</tt>

15. edit mgmt to reflect new host and dir for new system

16. optional add ipfw rules on new system. Ex:
<tt>ipfw add 01231 count ip from 69.55.230.2 to any
ipfw add 01232 count ip from any to 69.55.230.2</tt>

== Increasing inodes for a VPS ==

when doing the newfs:
newfs -i 4096 ...

== Jail services unable to contact services within same jail ==

This is due to a messed up routing table (only seen it on jail2 before).

route delete 69.55.228.65/32 route add -net 69.55.228.65/32 -iface fxp0 -nostatic -cloning route add 69.55.228.65 -iface lo0

== /dev/null permission resets ==

Applies to 4.x jails only

For some reason that I do not understand at all, the /dev/null node in customer jails, after the system is restarted, often reverts permissions to 0600 ... which is bad, because non-root processes cannot redirect to /dev/null anymore. In fact, a fair number of server daemons will complain and/or fail if this is the case.

So, after a restart, after all systems have restarted, run postboot, which does something like:

for f in `df -k | grep /dev/vn | awk '{print $6}'` ; do chmod 0666 $f/dev/null ; done

Later, after the system is up, if anyone ever complains about their /dev/null permissions for any reason ... really if anyone mentions
/dev/null in any way, tell them to run:

chmod 0666 /dev/null

We have ultimately fixed this by adding chmods to the quad/safe's

== mknod in a jail ==

<pre>
cd /mnt/data1/<user directory>/var/named
mknod -c 1 1 null
mknod -c 2 3 random
</pre>

== Postfix Problems ==

This hasn't happened in a long time, probably no longer valid with modern OS's

Postfix is an alternate MTA - people replace sendmail with it in much the same way that many people also replace sendmail with qmail. The problem is that if you install postfix inside of a jail, by default it will not work properly - the user will see error messages like this:

incoming mail generates errors like this:

mail_queue_enter: create file incoming/298266.55902: File too large

and further, sending mail generates errors like this:

postdrop: warning: mail_queue_enter: create file maildrop/627930.56676: File too large

This is very easy to solve. In fact, the FreeBSD welcome email that is sent out by `jailmake` now contains this block of text:

- if you plan on installing postfix, email us first and ask for a necessary patch.

So, anytime you see a support email that says anything about postfix not working, paste that line from the welcome email into your response to them, and right away, forward the postfix patch to them in an email.

Here are the instructions:

Simply copy the patch into /usr/ports/mail/postfix/files, rename the file to ‘patch-file_limit.c’ and then cd to /usr/ports/mail/postfix and run `make install`.

The postfix patch is the very first email in the support email box, and we never delete or save it because we always want it there to forward to people. The instructions in the email are very clear, and it solves their problem 100% of the time.

== Problems with `find` ==

There was a bug in the FreeBSD vn-filesystem code. The bug causes the system to crash or hang when someone runs the `find` command inside their vn-backed filesystem (and even though the bug may have been fixed, due to the i/o overhead, we still don't like find).

Now, this does not always happen - however, any jail machine with more than 20 systems on it will crash every night if every system on it runs the daily periodic script out of their crontab, and updates their locate database with the find command.

(the locate database is a small, FreeBSD specific database that is populated nightly from a cron job, and is then used to provide fast answers to the `locate` command)

In normal systems (32-40 jails on them), all that has to be done is make sure nobody runs the daily periodic. This means that the file /etc/crontab, instead of looking like this:

<pre># do daily/weekly/monthly maintenance
1 3 * * * root periodic daily
15 4 * * 6 root periodic weekly
30 5 1 * * root periodic monthly</pre>

Needs to look like this:

<pre># DO NOT UNCOMMENT THESE - contact support@johncompanies.com for details
#1 3 * * * root periodic daily
#15 4 * * 6 root periodic weekly
#30 5 1 * * root periodic monthly
## DO NOT UNCOMMENT THESE ^^^^^^^^</pre>

and the problem will generally not occur at all. New filesystem images are always altered in this way, so all systems are set like this.

However, if you have enough systems running, even casual uses of find, outside of the periodic scripts, can cause the system to crash. We are seeing this on jail13, which crashes about once per week.

There is not yet any resolution to this problem.

So, if you have a freebsd system crash, see if anyone has uncommented their daily periodic line - you can see them for everyone by running:

/bin/sh

and then running:

<pre>for f in `df -k | grep vn | awk '{print $6}'` ; do echo `cat $f/etc/crontab | grep "periodic daily"` $f ; done | more</pre>

(all one line, one command)

And then edit those /etc/crontabs and comment them out again.

== Problems un-mounting - and with mount_null’s ==

If you cannot unmount a filesystem, beacuse it says the filesystem is busy, it is because of three things:

a) the jail is still running

b) you are actually in that directory, even though the jail is stopped

c) there are still dev, null_mount or linprocfs mount points mounted inside that directory.

d) when trying to umount null_mounts that are really long and you get an error like “No such file or directory”, it’s an OS bug where the dir is truncated. No known fix

e) there are still files open somewhere inside the dir. Use <tt>fstat | grep <cid></tt> to find the process that has files open

f) Starting with 6.x, the jail mechanism does a poor job of keeping track of processes running in a jail and if it thinks there are still procs running, it will refuse to umount the disk. If this is happening you should see a low number in the #REF column when you run jls. In this case you ''can'' safely <tt>umount –f</tt> the mount.

Please note -if you forcibly unmount a (4.x) filesystem that has null_mounts
still mounted in it, the system '''will crash''' within 10-15 mins.

== Misc jail Items ==

We are overselling hard drive space on jail2, jail8, jail9, a couple jails on jail17, jail4, jail12 and jail18.
Even though the vn file shows 4G size, it doesn’t actually occupy that amount of space on the disk. So be careful not to fill up drives where we’re overselling – use oversellcheck to confirm you’re not oversold by more than 10G.
There are other truncated jails, they are generally noted in a the file on the root system: /root/truncated

The act of moving a truncated vn to another system un-does the truncating- the truncated vn is filled with 0’s and it occupies physical disk space for which it’s configured. So, you should use dumpremote to preserve the truncation.

* if you are getting disk full messages for a BSD customer, it's fairly safe to clear out their /usr/ports/distfiles dir
* 4.x: ps and top an only be run by root in these jails. Done on purpose:
As for `ps` and `top` - non-root users can run them, just not successfully because we have locked the permissions on /dev/mem and /dev/kmem to be root-readable only. That is why non-root users cannot successfully run `ps` and `top`.
* user quotas do not work on freebsd jails - you cannot set up quotas at all, and that's that.
* You cannot inject a process into a 4.x jail, only HUP running processes. 6.x and onward you can with jexec
* jails see base machine’s uptime/load when running top/w
* if someone is unable to get in- cant ping, etc- see if they were blocked by castle (DoS), see if their ip is on the system (post reboot it was lost cause wasn’t in the rc.conf). preboot should catch that.
* in FreeBSD you cant su to root unless you belong to wheel group – so if you remove your acct and setup a new one, we have to add it to wheel group (add to /etc/group)
* Dmesg from underlying sys is seen in customers dmesg on jail
* Popper process going crazy @ 40% for 10min = someone who leaves mail on server
* Don’t force umounts on 4.x jails – it crashes the machine, generally ok on newer machines (running md)
* Good book for admin http://search.barnesandnoble.com/booksearch/isbnInquiry.asp?userid=t824VyRAYz&isbn=0596005164&itm=2
* Self-sign ssl cert http://httpd.apache.org/docs/2.0/ssl/ssl_faq.html#selfcert
* conversation with Glenn about semaphores and pgsql
<pre>SDBoody: hey, these are valid amounts/figures/increments right:
kern.ipc.semmni=1280
kern.ipc.semmns=1280

gr8feen: probably... I always forget exactly what those are, so I usually have to look them up
SDBoody: semaphores
SDBoody: i took the current 1024 and added 256 to them
SDBoody: need more for pgsql
gr8feen: I meant the mni and mns parts...
gr8feen: some of those are not ones you just want to add to... hang on a sec and I'll look them up..
gr8feen: what's semmsl set to?
SDBoody: kern.ipc.semmsl: 1024

SDBoody: kern.ipc.msgseg: 2048
kern.ipc.msgssz: 8
kern.ipc.msgtql: 40
kern.ipc.msgmnb: 2048
kern.ipc.msgmni: 40
kern.ipc.msgmax: 16384
kern.ipc.semaem: 16384
kern.ipc.semvmx: 65534
kern.ipc.semusz: 152
kern.ipc.semume: 10
kern.ipc.semopm: 100
kern.ipc.semmsl: 1024
kern.ipc.semmnu: 512
kern.ipc.semmns: 1024
kern.ipc.semmni: 1024
kern.ipc.semmap: 768
kern.ipc.shm_allow_removed: 0
kern.ipc.shm_use_phys: 1
kern.ipc.shmall: 262144
kern.ipc.shmseg: 256
kern.ipc.shmmni: 784
kern.ipc.shmmin: 1
kern.ipc.shmmax: 536870912
kern.ipc.maxsockets: 25600

gr8feen: ok...msl is max per id, mni is max ids, mns is max number of semaphores... so you probably want something like mns = mni * msl
gr8feen: which one did you run out of?
SDBoody: not sure how to tell- ipcs shows the sems in use add up to 1024
SDBoody: there are 59 entries
gr8feen: I'm assuming you tried to start postgres and it failed?
SDBoody: yes
gr8feen: it should have logged why, somewhere..
gr8feen: if I recall, it'll tell you which one it ran out of
SDBoody: > DETAIL: Failed system call was semget(1, 17, 03600).

gr8feen: so it wanted an id with 17 semaphores...I'd start by making mns = 17*mni and leave mni and mnl set to what they are now and see what it does
SDBoody: i think mni is plenty high
SDBoody: ok, more reasonable 17408
gr8feen: yeah...just change that one and see how it goes..
SDBoody: and leave mni alone at 1024?
gr8feen: yeah...mni id the max number of ids...but if it's trying to get something like 17 per id, your going to hit mns before you hit anything else
SDBoody: right, but doesn't hurt to have it that high (assuming)

gr8feen: not really... I think those get allocated out of ram that you cant page out, but it's still such a small amount that it really doesn't matter
SDBoody: looks like that worked, thx!
gr8feen: cool

see semaphores: ipcs -a -s</pre>

= FreeBSD VPS Management Tools =

These files are located in /usr/local/jail/rc.d and /usr/local/jail/bin

== jailmake ==

Applies to 7.x+
On older systems syntax differs, run jailmake once to see.

Note: this procedure differs on mx2 which is 7.x but still uses gvinum

# run js to figure out which md’s are in use, which disk has enough space, IP to put it on
# use col00xxx for both hostnames if they don’t give you a hostname
# copy over dir, ip and password to pending customer screen

<tt>Usage: jailmake IP[,IP] CID disk[1|2|3] md# hostname shorthost ipfw# email [size in GB]</tt>

Ex:

Jail2# jailmake 69.55.234.66 col01334 3 97 vps.bsd.it vps 1334 fb@bsd.it

== jailps ==
jailps [hostname]
DEPRECATED FOR jps: displays processes belonging to/running inside a jail. The command
takes one (optional) argument – the hostname of the jail you wish to query. If you don’t
supply an argument, all processes on the machine are listed and grouped by jail.

== jps ==
jps [hostname]
displays processes belonging to/running inside a jail. The command
takes one (optional) argument – the hostname or ID of the jail you wish to query.

== jailkill ==
jailkill <hostname>
stops all process running in a jail.

You can also run:
jailkill <JID>

=== problems ===
Occasionally you will hit an issue where jail will not kill off:

<pre>jail9# jailkill www.domain.com
www.domain.com .. killed: none
jail9#</pre>
Because no processes are running under that hostname. You cannot use jailps.pl either:

<pre>jail9# jailps www.domain.com
www.domain.com doesn’t exist on this server
jail9#</pre>

The reasons for this are usually:
* the jail is no longer running

* the jail's hostname has changed
In this case,

>=6.x: run a <tt>jls|grep <jail's IP></tt> to find the correct hostname, then update the quad file, then kill the jail.

<6.x: the first step is to cat their /etc/rc.conf file to see if you can tell what they set the new hostname to. This very often works. For example:

cat /mnt/data2/198.78.65.136-col00261-DIR/etc/rc.conf

But maybe they set the hostname with the hostname command, and the original hostname is still in /etc/rc.conf.

The welcome email clearly states that they should tell us if they change their hostname, so there is no problem in just emailing them and asking them what they set the new hostname to.

Once you know the new hostname OR if a customer simply emails to inform you that they have set the hostname to something different, you need to edit the quad and safe files that their system is in to input the new hostname.

However, if push comes to shove and you cannot find out the hostname from them or from their system, then you need to start doing some detective work.

The easiest thing to do is run jailps looking for a hostname similar to their original hostname. Or you could get into the /bin/sh shell by running:

/bin/sh

and then looking at every hostname of every process:

for f in `ls /proc` ; do cat /proc/$f/status ; done

and scanning for a hostname that is either similar to their original hostname, or that you don't see in any of the quad safe files.

This is very brute force though, and it is possible that catting every file in /proc is dangerous - I don't recommend it. A better thing would be to identify any processes that you know belong to this system – perhaps the reason you are trying to find this system is because they are running something bad - and just catting the status from only that PID.

Somewhere there’s a jail where there may be 2 systems named www. Look at /etc/rc.conf and make sure they’re both really www. If they are, jailkill www, jailps www to make sure not running. Then immediately restart the other one, as the fqdn (as found from a rev nslookup)

* on >=6.x the hostname may not yet be hashed:
<pre>jail9 /# jls
JID Hostname Path IP Address(es)
1 bitnet.dgate.org /mnt/data1/69.55.232.50-col02094-DIR 69.55.232.50
2 ns3.hctc.net /mnt/data1/69.55.234.52-col01925-DIR 69.55.234.52
3 bsd1 /mnt/data1/69.55.232.44-col00155-DIR 69.55.232.44
4 let2.bbag.org /mnt/data1/69.55.230.92-col00202-DIR 69.55.230.92
5 post.org /mnt/data2/69.55.232.51-col02095-DIR 69.55.232.51 ...
6 ns2 /mnt/data1/69.55.232.47-col01506-DIR 69.55.232.47 ...
7 arlen.server.net /mnt/data1/69.55.232.52-col01171-DIR 69.55.232.52
8 deskfood.com /mnt/data1/69.55.232.71-col00419-DIR 69.55.232.71
9 mirage.confluentforms.com /mnt/data1/69.55.232.54-col02105-DIR 69.55.232.54 ...
10 beachmember.com /mnt/data1/69.55.232.59-col02107-DIR 69.55.232.59
11 www.agottem.com /mnt/data1/69.55.232.60-col02109-DIR 69.55.232.60
12 sdhobbit.myglance.org /mnt/data1/69.55.236.82-col01708-DIR 69.55.236.82
13 ns1.jnielsen.net /mnt/data1/69.55.234.48-col00204-DIR 69.55.234.48 ...
14 ymt.rollingegg.net /mnt/data2/69.55.236.71-col01678-DIR 69.55.236.71
15 verse.unixlore.net /mnt/data1/69.55.232.58-col02131-DIR 69.55.232.58
16 smcc-mail.org /mnt/data2/69.55.232.68-col02144-DIR 69.55.232.68
17 kasoutsuki.w4jdh.net /mnt/data2/69.55.232.46-col02147-DIR 69.55.232.46
18 dili.thium.net /mnt/data2/69.55.232.80-col01901-DIR 69.55.232.80
20 www.tekmarsis.com /mnt/data2/69.55.232.66-col02155-DIR 69.55.232.66
21 vps.yoxel.net /mnt/data2/69.55.236.67-col01673-DIR 69.55.236.67
22 smitty.twitalertz.com /mnt/data2/69.55.232.84-col02153-DIR 69.55.232.84
23 deliver4.klatha.com /mnt/data2/69.55.232.67-col02160-DIR 69.55.232.67
24 nideffer.com /mnt/data2/69.55.232.65-col00412-DIR 69.55.232.65
25 usa.hanyuan.com /mnt/data2/69.55.232.57-col02163-DIR 69.55.232.57
26 daifuku.ppbh.com /mnt/data2/69.55.236.91-col01720-DIR 69.55.236.91
27 collins.greencape.net /mnt/data2/69.55.232.83-col01294-DIR 69.55.232.83
28 ragebox.com /mnt/data2/69.55.230.104-col01278-DIR 69.55.230.104
29 outside.mt.net /mnt/data2/69.55.232.72-col02166-DIR 69.55.232.72
30 vps.payneful.ca /mnt/data2/69.55.234.98-col01999-DIR 69.55.234.98
31 higgins /mnt/data2/69.55.232.87-col02165-DIR 69.55.232.87 ...
32 ozymandius /mnt/data2/69.55.228.96-col01233-DIR 69.55.228.96
33 trusted.realtors.org /mnt/data2/69.55.238.72-col02170-DIR 69.55.238.72
34 jc1.flanderous.com /mnt/data2/69.55.239.22-col01504-DIR 69.55.239.22
36 guppylog.com /mnt/data2/69.55.238.73-col00036-DIR 69.55.238.73
40 haliohost.com /mnt/data2/69.55.234.41-col01916-DIR 69.55.234.41 ...
41 satyr.jorge.cc /mnt/data1/69.55.232.70-col01963-DIR 69.55.232.70
jail9 /# jailkill satyr.jorge.cc
ERROR: jail_: jail "satyr,jorge,cc" not found
</pre>

Note how it's saying <tt>satyr,jorge,cc</tt> is not found, and not <tt>satyr.jorge.cc</tt>.

The jail subsystem tracks things using comma-delimited hostnames. That is created every few hours:

jail9 /# crontab -l
0 0,6,12,18 * * * /usr/local/jail/bin/sync_jail_names

So if we run this manually:
jail9 /# /usr/local/jail/bin/sync_jail_names

Then kill the jail:
jail9 /# jailkill satyr.jorge.cc
successfully killed: satyr,jorge,cc</pre>

It worked.

If you ever see this when trying to kill a jail:

<pre># jailkill e-scribe.com
killing JID: 6 hostname: e-scribe.com
3 procs running
3 procs running
3 procs running
3 procs running
...</pre>

<tt>[[#jailkill|jailkill]]</tt> probably got lost trying to kill off the jail. Just ctrl-c the jailkill process, then run a jailps on the hostname, and <tt>kill -9</tt> any process which is still running. Keep running jailps and <tt>kill -9</tt> till all processes are gone.

== jailpsall ==
jailpsall
will run a jailps on all jails configured in the quad files (this is different from
jailps with no arguments as it won’t help you find a “hidden” system)

== jailpsw ==
jailpsw
will run a jailps with an extra -w to provide wider output

== jt (>=7.x) ==
jt
displays the top 20 processes on the server (the top 20 processes from top) and
which jail owns them. This is very helpful for determining who is doing what when
the server is very busy.

== jtop (>=7.x) ==
jtop
a wrapper for top displaying processes on the server and which jail owns them. Constantly updates, like top.

== jtop (<7.x) ==
jtop
displays the top 20 processes on the server (the top 20 processes from top) and
which jail owns them. This is very helpful for determining who is doing what when
the server is very busy.

== stopjail ==
stopjail <hostname> [1]
this will jailkill, umount and vnconfig –u a jail. If passed an optional 2nd
argument, it will not exit before umounting and un-vnconfig’ing in the event
jailkill returns no processes killed. This is useful if you just want to umount
and vnconfig –u a jail you’ve already killed. It is intelligent in that it won’t
try to umount or vnconfig –u if it’s not necessary.

== startjail ==
startjail <hostname>
this will start vnconfig, mount (including linprocfs and null-mounts), and start a jail.
Essentially, it reads the jail’s relevant block from the right quad file and executes it.
It is intelligent in that it won’t try to mount or vnconfig if it’s not necessary.

== jpid ==
jpid <pid>
displays information about a process – including which jail owns it.
It’s the equivalent of running cat /proc/<pid>/status

== canceljail ==
canceljail <hostname> [1]
this will stop a jail (the equivalent of stopjail), check for backups (offer to remove them
from the backup server and the backup.config), rename the vnfile, remove the dir, and
edit quad/safe. If passed an optional 2nd argument, it will not exit upon failing to kill
and processes owned by the jail. This is useful if you just want to cancel a jail which
is already stopped.

== jls ==
jls [-v]
Lists all jails running:
<pre>JID #REF IP Address Hostname Path
101 135 69.55.224.148 mail.pc9.org /mnt/data2/69.55.224.148-col01034-DIR</pre>

#REF is the number of references or procs(?) running

Running with -v will give you all IPs assigned to each jail (7.2 up)
<pre>JID #REF Hostname Path IP Address(es)
101 139 mail.pc9.org /mnt/data2/69.55.224.148-col01034-DIR 69.55.224.14869.55.234.85</pre>

== startalljails ==
startalljails
7.2+ only. This will parse through quad1 and start all jails. It utilizes lockfiles so it won’t try to start a jail more than once- therefore multiple instances can be running in parallel without fear of starting a jail twice. If a jail startup gets stuck, you can ^C without fear of killing the script. IMPORTANT- before running startalljails you should make sure you ran preboot once as it will clear out all the lockfiles and enable startalljails to work properly.

== aaccheck.sh ==
aaccheck.sh
displayes the output of container list and task list from aaccli

== backup ==
backup
backup script called nightly to update jail scripts and do customer backups

== buildsafe ==
buildsafe
creates safe files based on quads (automatically removing the fsck’s). This will destructively overwrite safe files

== checkload.pl ==
checkload.pl
this was intended to be setup as a cronjob to watch processes on a jail when the load
rises above a certain level. Not currently in use.

== checkprio.pl ==
checkprio.pl
will look for any process (other than the current shell’s csh, sh, sshd procs) with a non-normal priority and normalize it

== diskusagemon ==
diskusagemon <mount point> <1k blocks>
watches a mount point’s disk use, when it reaches the level specified in the 2nd argument,
it exits. This is useful when doing a restore and you want to be paged as it’s nearing completion.
Best used as: <tt>diskusagemon /asd/asd 1234; pagexxx</tt>

== dumprestore ==
dumprestore <dumpfile>
this is a perl expect script which automatically enters ‘1’ and ‘y’. It seems to cause restore to fail
to set owner permissions on large restores.

== g ==
g <search>
greps the quad/safe files for the given search parameter

== gather.pl ==
gather.pl
gathers up data about jails configured and writes to a file. Used for audits against the db

== gb ==
gb <search>
greps backup.config for the given search parameter

== gbg ==
gbg <search>
greps backup.config for the given search parameter and presents just the directories (for clean pasting)

== ipfwbackup ==
ipfwbackup
writes ipfw traffic count data to a logfile

== ipfwreset ==
ipfwreset
writes ipfw traffic count data to a logfile and resets counters to 0

== js ==
js
output varies by OS version, but generally provides information about the base jail:
- which vn’s are in use
- disk usage
- info about the contents of quads
- the # of inodes represented by the jails contained in the group (133.2 in the example below), and how many jails per data mount, as well as subtotals
- ips bound to the base machine but not in use by a jail
- free gvinum volumes, or unused vn’s or used md’s

<pre>/usr/local/jail/rc.d/quad1:
/mnt/data1 133.2 (1)
/mnt/data2 1040.5 (7)
total 1173.7 (8)
/usr/local/jail/rc.d/quad2:
/mnt/data1 983.4 (6)
total 983.4 (6)
/usr/local/jail/rc.d/quad3:
/mnt/data1 693.4 (4)
/mnt/data2 371.6 (3)
total 1065 (7)
/usr/local/jail/rc.d/quad4:
/mnt/data1 466.6 (3)
/mnt/data2 882.2 (5)
total 1348.8 (8)
/mnt/data1: 2276.6 (14)
/mnt/data2: 2294.3 (15)

Available IPs:
69.55.230.11 69.55.230.13 69.55.228.200

Available volumes:
v78 /mnt/data2 2G
v79 /mnt/data2 2G
v80 /mnt/data2 2G</pre>

== load.pl ==
load.pl
feeds info to load mrtg - executed by inetd.

== makevirginjail ==
makevirginjail
Only on some systems, makes an empty jail (doesn't do restore step)

== mb ==
mb <mount|umount>
(nfs) mounts and umounts dirs to backup2. Shortcuts are mbm and mbu to mount and unmount.

== notify.sh ==
notify.sh
emails reboot@johncompanies.com – intended to be called at boot time to alert us to a machine which panics and reboots and isn’t caught by bb or castle.

== orphanedbackupwatch ==
orphanedbackupwatch
looks for directories on backup2 which aren’t configured in backup.config and offers to delete them

== postboot ==
postboot
to be run after a machine reboot and quad/safe’s are done executing. It will:
* do chmod 666 on each jail’s /dev/null
* add ipfw counts
* run jailpsall (so you can see if a configured jail isn’t running)

== preboot ==
preboot
to be run before running quad/safe – checks for misconfigurations:
* a jail configured in a quad but not a safe
* a jail is listed more than once in a quad
* the ip assigned to a jail isn’t configured on the machine
* alias numbering skips in the rc.conf (resulting in the above)
* orphaned vnfile's that aren't mentioned in a quad/safe
* ip mismatches between dir/vnfile name and the jail’s ip
* dir/vnfiles's in quad/safe that don’t exist

== quadanalyze.pl ==
quadanalyze.pl
called by js, produces the info (seen above with js explanation) about the contents of quad (inode count, # of jails, etc.)

== rsync.backup ==
rsync.backup
does customer backups (relies on backup.config)

== taskdone ==
taskdone
when called will email support@johncompanies.com with the hostname of the machine from which it was executed as the subject

== topten ==
topten
summarizes the top 10 traffic users (called by ipfwreset)

== trafficgather.pl ==
trafficgather.pl [yy-mm]
sends a traffic usage summary by jail to support@johncomapnies.com and payments@johncompanies.com. Optional arguments are year and month (must be in the past). If not passed, assumes last month. Relies on traffic logs created by ipfwreset and ipfwbackup

== trafficwatch.pl ==
trafficwatch.pl
checks traffic usage and emails support@johncomapnies.com when a jail reaches the warning level (35G) and the limit (40G). We really aren’t using this anymore now that we have netflow.

== trafstats ==
trafstats
writes ipfw traffic usage info by jail to a file called jc_traffic_dump in each jail’s / dir

== truncate_jailmake ==
truncate_jailmake
a version of jailmake which creates truncated vnfiles.

== vb ==
vb
the equivalent of: <tt>vi /usr/local/jail/bin/backup.config</tt>

== vs (freebsd) ==
vs<n>
the equivalent of: <tt>vi /usr/local/jail/rc.d/safe<n></tt>

vq<n>
the equivalent of: vi /usr/local/jail/rc.d/quad<n>

== dumpremote ==
dumpremote <user@machine> </remote/location/file-dump> <vnX>
ex: dumpremote user@10.1.4.117 /mnt/data3/remote.echoditto.com-dump 7
this will dump a vn filesystem to a remote machine and location

== oversellcheck ==
oversellcheck
displays how much a disk is oversold or undersold taking into account truncated vn files. Only for use on 4.x systems

== mvbackups (freebsd) ==
mvbackups <dir> (1.1.1.1-col00001-DIR) <target_machine> (jail1) <target_dir> (data1)
moves backups from one location to another on the backup server, and provides you with option to remove entries from current backup.config, and simple paste command to add the config to backup.config on the target server

== jailnice ==
jailnice <hostname>
applies <tt>renice 19 [PID]</tt> and <tt>rtprio 31 –[PID]</tt> to each process in the given jail

== dumpremoterestore ==
dumpremoterestore <device> <ip of target machine> <dir on target machine>
ex: dumpremoterestore /dev/vn51 10.1.4.118 /mnt/data2/69.55.239.45-col00688-DIR
dumps a device and restores it to a directory on a remote machine. Requires that you enable root ssh on the
remote machine.

== psj ==
psj
shows just the procs running on the base system – a ps auxw but without jail’d procs present

== perc5iraidchk ==
perc5iraidchk
checks for degraded arrays on Dell 2950 systems with Perc5/6 controllers

== perc4eraidchk ==
perc4eraidchk
checks for degraded arrays on Dell 2850 systems with Perc4e/Di controllers

= Virtuozzo VPS =

== Making new customer VE (vm) ==

This applies only to new virts >= 4.x

grab ip from ipmap (if opened from the pending cust screen it should take you to the right block). You can also run vzlist -a to see what block is in use, generally. Try to find an IP that's in the same block of class C IP's already on the box.

1. confirm ip you want to use isn’t in use via Mgmt. -> IP Map in management screens
also, ping the IP address.

2. put CT on whichever partition has more space
df -h

3. vm CID IP hostname /vz[1|2] email[,email] template ( <LB|LBP|LS|LM|LMP|LP> [disk] | <gb_disk/mb_ram/gb_burst> )
vm col00009 69.55.230.238 centos.testdave.com /vz1 dsmith@johncompanies.com centos-6-x86_64 LM

4. copy veid, dir, ip and password to pending customer screen. activate customer

== Making new customer VE (vemakexxx) ==

This applies to older virts with old templates. This should probably not be used at all anymore.

1. look thru hist for ip

2. confirm ip you want to use isn’t in use via Mgmt. -> IP Map in management screens

3. put ve on whichever partition has more space
vemakerh9 <veid> <ip> <hostname> <mount> <email> [gb disk]; <256|384|512> <veid>
vemakerh9 866 69.55.226.109 ngentu.com /vz1 ayo@ngantu.com,asd@asd.com 5; 256 866

4. copy (veid), dir, and ip to pending customer screen (pass set to p455agfa)

Note: We use VEID (Virtual Environment ID) and CTID (Container ID) interchangably. Similarly, VE and CT. They mean the same thing.
VZPP = VirtuoZzo Power Panel (the control panel for each CT)

All linux systems exist in /vz, /vz1 or /vz2 - since each linux machine holds roughly 60-90 customers, there will be roughly 30-45 in each partition.

The actual filesystem of the system in question is in:

/vz(1-2)/private/(VEID)

Where VEID is the identifier for that system - an all-numeric string larger than 100.

The actual mounted and running systems are in the corresponding:

/vz(1-2)/root/(VEID)

But we rarely interact with any system from this mount point.

You should never need to touch the root portion of their system – however you can traverse their filesystem by going to <tt>/vz(1-2)/private/(VEID)/root</tt> (<tt>/vz(1-2)/private/(VEID)/fs/root</tt> on 4.x systems) the root of their filesystem is in that directory, and their entire system is underneath that.

Every VE has a startup script in <tt>/etc/sysconfig/vz-scripts</tt> (which is symlinked as <tt>/vzconf</tt> on all systems) - the VE startup script is simply named <tt>(VEID).conf</tt> - it contains all the system parameters for that VE:

<pre># Configuration file generated by vzsplit for 60 VE
# on HN with total amount of physical mem 2011 Mb

VERSION="2"
CLASSID="2"

ONBOOT="yes"

KMEMSIZE="8100000:8200000"
LOCKEDPAGES="322:322"
PRIVVMPAGES="610000:615000"
SHMPAGES="33000:34500"
NUMPROC="410:415"
PHYSPAGES="0:2147483647"
VMGUARPAGES="13019:2147483647"
OOMGUARPAGES="13019:2147483647"
NUMTCPSOCK="1210:1215"
NUMFLOCK="107:117"
NUMPTY="19:19"
NUMSIGINFO="274:274"
TCPSNDBUF="1800000:1900000"
TCPRCVBUF="1800000:1900000"
OTHERSOCKBUF="900000:950000"
DGRAMRCVBUF="200000:200000"
NUMOTHERSOCK="650:660"
DCACHE="786432:818029"
NUMFILE="7500:7600"
AVNUMPROC="51:51"
IPTENTRIES="155:155"
DISKSPACE="4194304:4613734"
DISKINODES="400000:420000"
CPUUNITS="1412"
QUOTAUGIDLIMIT="2000"
VE_ROOT="/vz1/root/636"
VE_PRIVATE="/vz1/private/636"
NAMESERVER="69.55.225.225 69.55.230.3"
OSTEMPLATE="vzredhat-7.3/20030305"
VE_TYPE="regular"
IP_ADDRESS="69.55.225.229"
HOSTNAME="textengine.net"</pre>

As you can see, the hostname is set here, the disk space is set here, the number of inodes, the number of files that can be open, the number of tcp sockets, etc. - all are set here.

In fact, everything that can be set on this customer system is set in this conf file.

All interaction with the customer system is done with the VEID. You start the system by running:

vzctl start 999

You stop it by running:

vzctl stop 999

You execute commands in it by running:

vzctl exec 999 df -k

You enter into it, via a root-shell backdoor with:

vzctl enter 999

and you set parameters for the system, while it is still running, with:

vzctl set 999 --diskspace 6100000:6200000 --save

<tt>vzctl</tt> is the most commonly used command - we have aliased <tt>v</tt> to <tt>vzctl</tt> since we use it so often. We’ll continue to use <tt>vzctl</tt> in our examples, but feel free to use just <tt>v</tt>.

Let's say the user wants more diskspace. You can cat their conf file and see:

DISKSPACE="4194304:4613734"

So right now they have 4gigs of space. You can then change it to 6 with:

vzctl set 999 --diskspace 6100000:6200000 --save

IMPORTANT: all issuances of the vzctl set command need to end with <tt>–save</tt> - if they don't, the setting will be set, but it will not be saved to the conf file, and they will not have those settings next time they boot.

All of the tunables in the conf file can be set with the vzctl set command. Note that in the conf file, and on the vzctl set command line, we always issue two numbers seperated by a colon - that is because we are setting the hard and soft limits. Always set the hard limit slightly above the soft limit, as you see it is in the conf file for all those settings.

There are also things you can set with `<tt>vzctl set</tt>` that are not in the conf file as settings, per se. For instance, you can add IPs:

vzctl set 999 --ipadd 10.10.10.10 --save

or multiple IPs:

vzctl set 999 --ipadd 10.10.10.10 --ipadd 10.10.20.30 --save

or change the hostname:

vzctl set 999 --hostname www.example.com --save

You can even set the nameservers:

vzctl set 999 --nameserver 198.78.66.4 --nameserver 198.78.70.180 --save

Although you probably will never do that.

You can disable a VPS from being started (by VZPP or reboot) (4.x):

vzctl set 999 --disabled yes --save

You can disable a VPS from being started (by VZPP or reboot) (<=3.x):

vzctl set 999 --onboot=no --save

You can disable a VPS from using his control panel:

vzctl set 999 --offline_management=no --save

You can suspend a VPS, so it can be resumed in the same state it was in when it was stopped (4.x):

vzctl suspend 999

and to resume it:

vzctl resume 999

to see who owns process:
vzpid <PID>

to mount up an unmounted ve:
vzctl mount 827

To see network stats for CT's:
<pre># vznetstat
VEID Net.Class Output(bytes) Input(bytes)
-----------------------------------------------
24218 1 484M 39M
24245 1 463M 143M
2451 1 2224M 265M
2454 1 2616M 385M
4149 1 125M 68M
418 1 1560M 34M
472 1 1219M 315M
726 1 628M 317M
763 1 223M 82M
771 1 4234M 437M
-----------------------------------------------
Total: 13780M 2090M</pre>

One thing that sometimes comes up on older systems that we created with smaller defaults is that the system would run out of inodes. The user will email and say they cannot create any more files or grow any files larger, but they will also say that they are not out of diskspace ... they are running:

df -k

and seeing how much space is free - and they are not out of space. They are most likely out of inodes - which they would see by running:

df -i

So, the first thing you should do is enter their system with:

vzctl enter 999

and run: <tt>df -i</tt>

to confirm your theory. Then exit their system. Then simply cat their conf file and see what their inodes are set to (probably 200000:200000, since that was the old default on the older systems) and run:

vzctl set 999 --diskinodes 400000:400000 --save

If they are not out of inodes, then a good possibility is that they have maxed out their numfile configuration variable, which controls how many files they can have in their system. The current default is 7500 (which nobody has ever hit), but the old default was as low as 2000, so you would run something like:

vzctl set 999 --numfile 7500:7500 --save

----

You cannot start or stop a VE if your pwd is its private (/vz/private/999) or root (/vz/root/999) directories, or anywhere below them.

== Recovering from a crash (linux) ==

=== Diagnose whether you have a crash ===
The most important thing is to get the machine and all ve’s back up as soon as possible. Note the time, you’ll need to create a crash log entry (Mgmt. -> Reference -> CrashLog). The first thing to do is head over to the [[Screen#Screen_Organization|serial console screen]] and see if there’s any kernel error messages output. Try to copy any messages (or just a sample of repeating messages) you see into the notes section of the crash log – these will also likely need to be sent to virtuozzo for interpretation. If the messages are spewing too fast, hit ^O + H to start a screen log dump which you can ob1182.pts-38.bb serve after the machine is rebooted. Additionally, if the machine is responsive, you can get a trace to send to virtuozzo by hooking up a kvm and entering these 3 sequences:
<pre>alt+print screen+m
alt+print screen+p
alt+print screen+t</pre>

If there are no messages, the machine may just be really busy- wait a bit (5-10min) to see if it comes back. If it's still pinging, odds are its very busy. If it doesn't come back, or the messages indicate a fatal error, you will need to proceed with a power cycle (ctrl+alt+del will not work).

=== Power cycle the server ===
If this machine is not a Dell 2950 with a [[DRAC/RMM#DRAC|DRAC card]] (i.e. if you can’t ssh into the DRAC card and issue racadm serveraction hardreset, then you will need someone at the data center to power the macine off, wait 30 sec, then turn it back on. Make sure to re-attach via console (<tt>tip virtxx</tt>) immediately after power down.

=== (Re)attach to the console ===
Stay on the console the entire time during boot. As the BIOS posts- look out for the RAID card output- does everything look healthy? The output may be scrambled, look for "DEGRADED" or "FAILED". Once the OS starts booting you will be disconnected (dropped back to the shell on the console server) a couple times during the boot up. The reason you want to quickly re-attach is two-fold: 1. If you don’t reattach quickly then you won’t get any console output, 2. you want to be attached before the server ''potentially'' starts (an extensive) fsck. If you attach after the fsck begins, you’ll have seen no indication it started an fsck and the server will appear frozen during startup- no output, no response.

=== Start containers/VE's/VPSs ===
When the machine begins to start VE’s, it’s safe to leave the console and login via ssh. All virts should be set to auto start all the VEs after a crash. Further, most (newer) virts are set to “fastboot” it’s VE’s (to find out, do:
grep -i fast /etc/sysconfig/vz
and look for <tt>VZFASTBOOT=yes</tt>). If this was set prior to the machine’s crash (setting it after the machine boots will not have any effect until the vz service is restarted) it will start each ve as fast as possible, in serial, then go thru each VE (serially), shutting it down running a vzquota (disk usage) check, then bringing it back up. The benefit is that all VE’s are brought up quickly (within 15min or so depending on the #), the downside is a customer watching closely will notice 2 outages – 1st the machine crash, 2nd their quota check (which will be a much shorter downtime- on the order of a few minutes).

Where “fastboot” is not set to yes (i.e on quar1), vz will start them consecutively, checking the quotas one at a time, and the 60th VE may not start until an hour or two later - this is not acceptable.

The good news is, if you run vzctl start for a VE that is already started, you will simply get an error: <tt>VE is already started</tt>. Further, if you attempt to vzctl start a VE that is in the process of being started, you will simply get an error: unable to lock VE. So, there is no danger in simply running scripts to start smaller sets of VEs. If the system is not autostarting, then there is no issue, and even if it does, when it conflicts, one process (yours or the autostart) will lose, and just move on to the next one.

A script has been written to assist with ve starts: [[#startvirt.pl|startvirt.pl]] which will start 6 ve’s at once until there are no more left. If startvirt.pl is used on a system where “fastboot” was on, it will circumvent the fastboot for ve’s started by startvirt.pl – they will go through the complete quota check before starting- therefore this is not advisable when a system has crashed. When a system is booted cleanly, and there's no need for vzquota checks, then startvirt.pl is safe and advisable to run.

=== Make sure all containers are running ===
You can quickly get a feel for how many ve’s are started by running:

<pre>[root@virt4 log]# vs
VEID 16066 exist mounted running
VEID 16067 exist mounted running
VEID 4102 exist mounted running
VEID 4112 exist mounted running
VEID 4116 exist mounted running
VEID 4122 exist mounted running
VEID 4123 exist mounted running
VEID 4124 exist mounted running
VEID 4132 exist mounted running
VEID 4148 exist mounted running
VEID 4151 exist mounted running
VEID 4155 exist mounted running
VEID 42 exist mounted running
VEID 432 exist mounted running
VEID 434 exist mounted running
VEID 442 exist mounted running
VEID 450 exist mounted running
VEID 452 exist mounted running
VEID 453 exist mounted running
VEID 454 exist mounted running
VEID 462 exist mounted running
VEID 463 exist mounted running
VEID 464 exist mounted running
VEID 465 exist mounted running
VEID 477 exist mounted running
VEID 484 exist mounted running
VEID 486 exist mounted running
VEID 490 exist mounted running</pre>

So to see how many ve’s have started:
<pre>[root@virt11 root]# vs | grep running | wc -l
39</pre>

And to see how many haven’t:
<pre>[root@virt11 root]# vs | grep down | wc -l
0</pre>

And how many we should have running:
<pre>[root@virt11 root]# vs | wc -l
39</pre>

Another tool you can use to see which ve’s have started, among other things is [[#vzstat|vzstat]]. It will give you CPU, memory, and other stats on each ve and the overall system. It’s a good thing to watch as ve’s are starting (note the VENum parameter, it will tell you how many have started):

<pre>4:37pm, up 3 days, 5:31, 1 user, load average: 1.57, 1.68, 1.79
VENum 40, procs 1705: running 2, sleeping 1694, unint 0, zombie 9, stopped 0
CPU [ OK ]: VEs 57%, VE0 0%, user 8%, sys 7%, idle 85%, lat(ms) 412/2
Mem [ OK ]: total 6057MB, free 9MB/54MB (low/high), lat(ms) 0/0
Swap [ OK ]: tot 6142MB, free 4953MB, in 0.000MB/s, out 0.000MB/s
Net [ OK ]: tot: in 0.043MB/s 402pkt/s, out 0.382MB/s 4116pkt/s
Disks [ OK ]: in 0.002MB/s, out 0.000MB/s

VEID ST %VM %KM PROC CPU SOCK FCNT MLAT IP
1 OK 1.0/17 0.0/0.4 0/32/256 0.0/0.5 39/1256 0 9 69.55.227.152
21 OK 1.3/39 0.1/0.2 0/46/410 0.2/2.8 23/1860 0 6 69.55.239.60
133 OK 3.1/39 0.1/0.3 1/34/410 6.3/2.8 98/1860 0 0 69.55.227.147
263 OK 2.3/39 0.1/0.2 0/56/410 0.3/2.8 34/1860 0 1 69.55.237.74
456 OK 17/39 0.1/0.2 0/100/410 0.1/2.8 48/1860 0 11 69.55.236.65
476 OK 0.6/39 0.0/0.2 0/33/410 0.1/2.8 96/1860 0 10 69.55.227.151
524 OK 1.8/39 0.1/0.2 0/33/410 0.0/2.8 28/1860 0 0 69.55.227.153
594 OK 3.1/39 0.1/0.2 0/45/410 0.0/2.8 87/1860 0 1 69.55.239.40
670 OK 7.7/39 0.2/0.3 0/98/410 0.0/2.8 64/1860 0 216 69.55.225.136
691 OK 2.0/39 0.1/0.2 0/31/410 0.0/0.7 25/1860 0 1 69.55.234.96
744 OK 0.1/17 0.0/0.5 0/10/410 0.0/0.7 7/1860 0 6 69.55.224.253
755 OK 1.1/39 0.0/0.2 0/27/410 0.0/2.8 33/1860 0 0 192.168.1.4
835 OK 1.1/39 0.0/0.2 0/19/410 0.0/2.8 5/1860 0 0 69.55.227.134
856 OK 0.3/39 0.0/0.2 0/13/410 0.0/2.8 16/1860 0 0 69.55.227.137
936 OK 3.2/52 0.2/0.4 0/75/410 0.2/0.7 69/1910 0 8 69.55.224.181
1020 OK 3.9/39 0.1/0.2 0/60/410 0.1/0.7 55/1860 0 8 69.55.227.52
1027 OK 0.3/39 0.0/0.2 0/14/410 0.0/2.8 17/1860 0 0 69.55.227.83
1029 OK 1.9/39 0.1/0.2 0/48/410 0.2/2.8 25/1860 0 5 69.55.227.85
1032 OK 12/39 0.1/0.4 0/80/410 0.0/2.8 41/1860 0 8 69.55.227.90</pre>

When you are all done, you will want to make sure that all the VEs really did get started, run vs one more time.

Note the time all ve’s are back up and enter that into and save the crash log entry.

Occasionally, a ve will not start automatically. The most common reason for a ve not to come up normally is the ve was at it’s disk limit before the crash, and will not start since they’re over the limit. To overcome this, set the disk space to current usage level (the system will give this to you when it fails to start), start the ve, then re-set the disk space back to the prior level. Lastly, contact the customer to let them know they’re out of disk (or allocate more disk if they're entitled to more).

== Hitting performance barriers and fixing them ==

There are multiple modes virtuozzo offers to allocate resources to a ve. We utilize 2: SLM and UBC parameters
On our 4.x systems, we use all SLM – it’s simpler to manage and understand. There are a few systems on virt19/18 that may also use SLM. Everything else uses UBC.
You can tell a SLM ve by:

SLMMODE="all"

in their conf file.

TODO: detail SLM modes and parameters.

If someone is in SLM mode and they hit memory resource limits, they simply need to upgrade to more memory.

The following applies to everyone else (UBC).

Customers will often email and say that they are getting out of memory errors - a common one is "cannot fork" ... basically, anytime you see something odd like this, it means they are hitting one of their limits that is in place in their conf file.

The conf file, however, simply shows their limits - how do we know what they are currently at ?

The answer is a file called v - this file contains the current status (and peaks) of their performance settings, and also counts how many times they have hit the barrier. The output of the file looks like this:

<pre>764: kmemsize 384113 898185 8100000 8200000 0
lockedpages 0 0 322 322 0
privvmpages 1292 7108 610000 615000 0
shmpages 270 528 33000 34500 0
dummy 0 0 0 0 0
numproc 8 23 410 415 0
physpages 48 5624 0 2147483647 0
vmguarpages 0 0 13019 2147483647 0
oomguarpages 641 6389 13019 2147483647 0
numtcpsock 3 21 1210 1215 0
numflock 1 3 107 117 0
numpty 0 2 19 19 0
numsiginfo 0 4 274 274 0
tcpsndbuf 0 80928 1800000 1900000 0
tcprcvbuf 0 108976 1800000 1900000 0
othersockbuf 2224 37568 900000 950000 0
dgramrcvbuf 0 4272 200000 200000 0
numothersock 3 9 650 660 0
dcachesize 53922 100320 786432 818029 0
numfile 161 382 7500 7600 0
dummy 0 0 0 0 0
dummy 0 0 0 0 0
dummy 0 0 0 0 0
numiptent 4 4 155 155 0</pre>

The first column is the name of the counter in question - the same names we saw in the systems conf file. The second column is the _current_ value of that counter, the third column is the max that that counter has ever risen to, the fourth column is the soft limit, and the fifth column is the hard limit (which is the same as the numbers in that systems conf file).

The sixth number is the failcount - how many times the current usage has risen to hit the barrier. It will increase as soon as the current usage hits the soft limit.

The problem with /proc/user_beancounters is that it actually contains that set of data for every running VE - so you can't just cat /proc/user_beancounters - it is too long and you get info for every other running system.

You can vzctl enter the system and run:

vzctl enter 9999
cat /proc/user_beancounters

inside their system, and you will just see the stats for their particular system, but entering their system every time you want to see it is combersome.

So, I wrote a simple script called "vzs" which simply greps for the VEID, and spits out the next 20 or so lines (however many lines there are in the output, I forget) after it. For instance:

<pre># vzs 765:
765: kmemsize 2007936 2562780 8100000 8200000 0
lockedpages 0 8 322 322 0
privvmpages 26925 71126 610000 615000 0
shmpages 16654 16750 33000 34500 0
dummy 0 0 0 0 0
numproc 41 57 410 415 0
physpages 1794 49160 0 2147483647 0
vmguarpages 0 0 13019 2147483647 0
oomguarpages 4780 51270 13019 2147483647 0
numtcpsock 23 37 1210 1215 0
numflock 17 39 107 117 0
numpty 1 3 19 19 0
numsiginfo 0 6 274 274 0
tcpsndbuf 22240 333600 1800000 1900000 0
tcprcvbuf 0 222656 1800000 1900000 0
othersockbuf 104528 414944 900000 950000 0
dgramrcvbuf 0 4448 200000 200000 0
numothersock 73 105 650 660 0
dcachesize 247038 309111 786432 818029 0
numfile 904 1231 7500 7600 0
dummy 0 0 0 0 0
dummy 0 0 0 0 0
dummy 0 0 0 0 0
numiptent 4 4 155 155 0</pre>

That showed us just the portion of /proc/user_beancounters for system 765.

When you run the vzs command, always add a : after the VEID.

So, if a customer complains about some out of memory errors, or no more files, or no more ptys, or just has an unspecific complain about processes dying, etc., the very first thing you need to do is check their beancounters with vzs. Usually you will spot an item that has a high failcount and needs to be upped.

At that point you could simply up the counter with `vzctl set`. Generally pick a number 10-20% higher than the old one, and make the hard limit slightly larger than the the soft limit. However our systems now come in several levels and those levels have more/different memory allocations. If someone is complaining about something other than a memory limit (pty, numiptent, numflock), it’s generally safe to increase it, at least to the same level as what’s in the /vzconf/4unlimited file on the newest virt. If someone is hitting a memory limit, first make sure they are given what they deserve:

(refer to mgmt -> payments -> packages)

To set those levels, you use the [[#setmem|setmem]] command.

The alternate (DEPRECATED) method would be to use one of 3 commands:
256 <veid>
300 <veid>
384 <veid>
512 <veid>

If the levels were not right (you’d run vzs <veid> before and after to see the effect) tell the customer they’ve been adjusted and be done with it. If the levels were right, tell the customer they must upgrade to a higher package, tell them how to see level (control panel) and that they can reboot their system to escape this lockup contidion.

Customers can also complain that their site is totally unreachable, or complain that it is down ... if the underlying machine is up, and all seems well, you may notice in the beancounters that network-specific counters are failing - such as numtcpsock, tcpsndbuf or tcprcvbuf. This will keep them from talking on the network and make it seem like their system is down. Again, just up the limits and things should be fine.

On virts 1-4, you should first look at the default settings for that item on a later virt, such as virt 8 - we have increased the defaults a lot since the early machines. So, if you are going to up a counter on virt2, instead of upping it by 10-20%, instead up it to the new default that you see on virt8.

== Moving a VE to another virt (migrate/migrateonline) ==

This will take a while to complete - and it is best to do this at night when the load is light on both machines.

There are different methods for this, depending on which version of virtuozzo is installed on the src. and dst. virt.
To check which version is running:
[root@virt12 private]# cat /etc/virtuozzo-release
Virtuozzo release 2.6.0

Ok, let's say that the VE is 1212, and vital stats are:
<pre>[root@virt12 sbin]# vc 1212
VE_ROOT="/vz1/root/1212"
VE_PRIVATE="/vz1/private/1212"
OSTEMPLATE="fedora-core-2/20040903"
IP_ADDRESS="69.55.229.84"
TEMPLATES="devel-fc2/20040903 php-fc2/20040813 mysql-fc2/20040812 postgresql-fc2/20040813 mod_perl-fc2/20040812 mod_ssl-fc2/20040811 jre-fc2/20040823 jdk-fc2/20040823 mailman-fc2/20040823 analog-fc2/20040824 proftpd-fc2/20040818 tomcat-fc2/20040823 usermin-fc2/20040909 webmin-fc2/20040909 uw-imap-fc2/20040830 phpBB-fc2/20040831 spamassassin-fc2/20040910 PostNuke-fc2/20040824 sl-webalizer-fc2/20040
818"

[root@virt12 sbin]# vzctl exec 1212 df -h
Filesystem Size Used Avail Use% Mounted on
/dev/vzfs 4.0G 405M 3.7G 10% /</pre>

From this you can see that he’s using (and will minimally need free on the dst server) ~400MB, and he’s running on a Fedora 2 template, version 20040903. He’s also got a bunch of other templates installed. It’s is '''vital''' that '''all''' these templates exist on the dst system. To confirm that, on the dst system run:

For < 3.0:

<pre>[root@virt14 private]# vzpkgls | grep fc2
devel-fc2 20040903
PostNuke-fc2 20040824
analog-fc2 20040824
awstats-fc2 20040824
bbClone-fc2 20040824
jdk-fc2 20040823
jre-fc2 20040823
mailman-fc2 20040823
mod_frontpage-fc2 20040816
mod_perl-fc2 20040812
mod_ssl-fc2 20040811
mysql-fc2 20040812
openwebmail-fc2 20040817
php-fc2 20040813
phpBB-fc2 20040831
postgresql-fc2 20040813
proftpd-fc2 20040818
sl-webalizer-fc2 20040818
spamassassin-fc2 20040910
tomcat-fc2 20040823
usermin-fc2 20040909
uw-imap-fc2 20040830
webmin-fc2 20040909
[root@virt14 private]# vzpkgls | grep fedora
fedora-core-1 20040121 20040818
fedora-core-devel-1 20040121 20040818
fedora-core-2 20040903
[root@virt14 private]#</pre>

For these older systems, you can simply match up the date on the template.

For >= 3.0:

<pre>[root@virt19 /vz2/private]# vzpkg list
centos-5-x86 2008-01-07 22:05:57
centos-5-x86 devel
centos-5-x86 jre
centos-5-x86 jsdk
centos-5-x86 mod_perl
centos-5-x86 mod_ssl
centos-5-x86 mysql
centos-5-x86 php
centos-5-x86 plesk9
centos-5-x86 plesk9-antivirus
centos-5-x86 plesk9-api
centos-5-x86 plesk9-atmail
centos-5-x86 plesk9-backup
centos-5-x86 plesk9-horde
centos-5-x86 plesk9-mailman
centos-5-x86 plesk9-mod-bw
centos-5-x86 plesk9-postfix
centos-5-x86 plesk9-ppwse
centos-5-x86 plesk9-psa-firewall
centos-5-x86 plesk9-psa-vpn
centos-5-x86 plesk9-psa-fileserver
centos-5-x86 plesk9-qmail
centos-5-x86 plesk9-sb-publish
centos-5-x86 plesk9-vault
centos-5-x86 plesk9-vault-most-popular
centos-5-x86 plesk9-watchdog</pre>

On these newer systems, it's difficult to tell whether the template on the dst matches exactly the src. Just cause a centos-5-x86 is listed on both servers doesn't mean all the same packages are there on the dst. To truly know, you must perform a sample rsync:

rsync -avn /vz/template/centos/5/x86/ root@10.1.4.61:/vz/template/centos/5/x86/

if you see a ton of output from the dry run command, then clearly there are some differences. You may opt to let the rsync complete (without running in dry run mode) the only downside is you've now used up more space on the dst and also the centos template will be a mess with old and new data- it will be difficult if not impossible to undo (if someday we wanted to reclaim the space).

If you choose to merge templates, you should closely inspect the dry run output. You should also take care to exclude anything in the /config directory. For example:

rsync -av -e ssh --stats --exclude=x86/config /vz/template/ubuntu/10.04/ root@10.1.4.62:/vz/template/ubuntu/10.04/

Which will avoid this directory and contents:
<pre>[root@virt11 /vz2/private]# ls /vz/template/ubuntu/10.04/x86/config*
app os</pre>

This is important to avoid since the config may differ on the destination and we are really only interested in making sure the pacakges are there, not overwriting a newer config with an older one.

If the dst system was missing a template, you have 2 choices:
# put the missing template on the dst system. 2 choices here:
## Install the template from rpm (found under backup2: /mnt/data4/vzrpms/distro/) or
## rsync over the template (found under /vz/template) - see above
# put the ve on a system which has all the proper templates

=== pre-seeding a migration ===

When migrating a customer (or when doing many) depending on how much data you have to transfer, it can take some time. Further, it can be difficult to gauge when a migration will complete or how long it will take. To help speed up the process and get a better idea about how long it will take you can pre-transfer a customer's data to the destination server. If done correctly, vzmigrate will see the pre-transferred data and pick up where you left off, having much less to transfer (just changed/new files).

We believe vzmigrate uses rsync to do it's transfer. Therefore not only can you use rsync to do a pre-seed, you can also run rsync to see what is causing a repeatedly-failing vzmigrate to fail.

There's no magic to a pre-seed, you just need to make sure it's named correctly.

Given:

source: /vz1/private/1234

and you want to migrate to /vz2 on the target system, your rsync would look like:

rsync -av /vz1/private/1234/ root@x.x.x.x:/vz2/private/1234.migrated/

After running that successful rsync, the ensuing migrateonline (or migrate) will take much less time to complete- depending on the # of files to be analyzed and the # of changed files. In any case, it'll be much much faster than had you just started the migration from scratch.

Further, as we discuss elsewhere in this topic, a failed migration can be moved from <tt>/vz/private/1234</tt> to <tt>/vz/private/1234.migrated</tt> on the destination if you want to restart a failed migration. This should '''only''' be done if the migration failed and the CT is not running on the destination HN.

=== migrateonline intructions: src >=3.x -> dst>=3.x ===

A script called [[#migrateonline|migrateonline]] was written to handle this kind of move. It is basically a wrapper for <tt>vzmigrate</tt> – vzmigrate is a util to seamlessly- as no no reboot of the ve necessary- move a ve from one host to another. This wrapper was initially written cause vz virtuozzo version 2.6.0 has a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables, causing problems when the ve was started up on the dst system. [[#migrate|migrate]] mitigates that. Since it makes multiple ssh connections to the dst virt, it’s a good idea to put the pub key for the src virt in the authorized_keys file on the dst virt. In addition, migrateonline emails ve owners when their migration starts and stops. For this to happen they need to put email addresses (on a single line, space delimited) in a file on their system: /migrate_notify. If the optional target dir is not specified, the ve will be moved to the same private/root location as it was on the src virt. Note: <tt>migrate</tt> is equivalent to <tt>migrateonline</tt>, but will <tt>migrate</tt> a ve AND restart it in the process.

<pre>[root@virt12 sbin]# migrateonline
usage: /usr/local/sbin/migrateonline <ip of node migrating to> <veid> [target dir: vz | vz1 | vz2]
[root@virt12 sbin]# migrateonline 10.1.4.64 1212 vz
starting to migrate 1212 at Sat Mar 26 22:40:38 PST 2005
Turning off offline_management
Saved parameters for VE 1212
migrating with no start on 10.1.4.64
Connection to destination HN (10.1.4.64) is successfully established
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...
Syncing private area '/vz1/private/1212'
- 100% |*************************************************|
done
Successfully completed
clearing the arp cache
now going to 10.1.4.64 and clear cache and starting
starting it
Delete port redirection
Adding port redirection to VE(1): 4643 8443
Adding IP address(es) to pool: 69.55.229.84
Saved parameters for VE 1212
Starting VE ...
VE is mounted
Adding port redirection to VE(1): 4643 8443
Adding IP address(es): 69.55.229.84
Hostname for VE set: fourmajor.com
File resolv.conf was modified
VE start in progress...
finished migrating 1212 at Sat Mar 26 22 22:52:01 PST 2005
[root@virt12 sbin]#</pre>

Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine.

If they had backups, use the mvbackups command to move their backups to the new server:

mvbackups 1212 virt14 vz

Rename the ve

[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/migrated-1212

[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/old-1212-migrated-20120404-noarchive

Update the customer’s systems in mgmt to reflect the new path and server.

IF migrateonline does not work, you can try again using simply migrate- this will result in a brief reboot for the ve.
Before you try again, make sure of a few things:

Depending on where in the migration died, there may be partial data on the dst system in 1 of 2 places:
(given the example above)

/vz/private/1212

or

/vz/private/1212.migrated

before you run migrate again, you'll want to rename so that all data is in
1212.migrated:

mv /vz/private/1212 /vz/private/1212.migrated

this way, it will pick up where it left off and transfer only new files.

Likewise, if you want to speed up a migration, you can pre-seed the dst as follows:

[root@virt12 sbin]# rsync -avSH /vz/private/1212/ root@10.1.4.64:/vz/private/1212.migrated/

then when you run migrate or migrateonline, it will only need to move the changed files- the migration will complete quickly

=== migrateonline/migrate failures (migrate manually) ===

Lets say for whatever reason the migration fails. If it fails with [[#migrateonline|migrateonline]], you should try [[#migrate|migrate]] (which will reboot the customer, so notify them ahead of time).

You may want to run a [[#pre-seeding_a_migration|pre-seed]] rsync to see if you can find the problem. On older virts, we've seen this problem due to a large logfile (which you can find and encourage the customer to remove/compress):
for f in `find / -size +1048576k`; do ls -lh $f; done

You may also see migration failing due to quota issues.

You can try to resolve by copying any quota file into the file you need:

cp /var/vzquota/quota.1 /var/vzquota/quota.xxx

If it complains about quota running you should then be able to stop it

vzquota off xxxx

If all else fails, migrate to a new VEID
i.e. 1234 becomes 12341

If the rsync or [[#migrate|migrate]] fails, you can always move someone manually:

1. stop ve: 
v stop 1234

2. copy over data 
rsync -avSH /vz/private/1234/ root@1.1.1.1:/vzX/private/1234/

NOTE: if you've previously seeded the data (run rsync while the VE was up/running), and this is a subsequent rsync, make sure the last rsync you do (while the VE is not running, has the --delete option in the rsync)

3. copy over conf 
scp /vzconf/1234.conf root@1.1.1.1:/vzconf

4. on dst, edit the conf to reflect the right vzX dir 
vi /vzconf/1234.conf

5. on src remove the IPs 
ipdel 1234 2.2.2.2 3.3.3.3

6. on dst add IPs 
ipadd 1234 2.2.2.2 3.3.3.3

7. on dst, start ve: 
v start 1324

8. cancel, then archive ve on src per above instrs.

=== migrate src=2.6.0 -> dst>=2.6.0, or mass-migration with customer notify ===

A script called <tt>migrate</tt> was written to handle this kind of move. It is basically a wrapper for vzmigrate – vzmigrate is a util to seamlessly move a ve from one host to another. This wrapper was initially written cause vz virtuozzo version 2.6.0 has a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables, causing problems when the ve was started up on the dst system. migrate mitigates that. Since it makes multiple ssh connections to the dst virt, it’s a good idea to put the pub key for the src virt in the authorized_keys file on the dst virt. In addition, migrate emails ve owners when their migration starts and stops. For this to happen they need to put email addresses (on a single line, space delimited) in a file on their system: /migrate_notify. If the optional target dir is not specified, the ve will be moved to the same private/root location as it was on the src virt. Note: migrateonline is equivalent to migrate, but will migrate a ve from one 2.6 '''kernel''' machine to another 2.6 kernel machine without restarting the ve.

<pre>[root@virt12 sbin]# migrate
usage: /usr/local/sbin/migrate <ip of node migrating to> <veid> [target dir: vz | vz1 | vz2]
[root@virt12 sbin]# migrate 10.1.4.64 1212 vz
starting to migrate 1212 at Sat Mar 26 22:40:38 PST 2005
Turning off offline_management
Saved parameters for VE 1212
migrating with no start on 10.1.4.64
Connection to destination HN (10.1.4.64) is successfully established
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...
Syncing private area '/vz1/private/1212'
- 100% |*************************************************|
done
Successfully completed
clearing the arp cache
now going to 10.1.4.64 and clear cache and starting
starting it
Delete port redirection
Adding port redirection to VE(1): 4643 8443
Adding IP address(es) to pool: 69.55.229.84
Saved parameters for VE 1212
Starting VE ...
VE is mounted
Adding port redirection to VE(1): 4643 8443
Adding IP address(es): 69.55.229.84
Hostname for VE set: fourmajor.com
File resolv.conf was modified
VE start in progress...
finished migrating 1212 at Sat Mar 26 22 22:52:01 PST 2005
[root@virt12 sbin]#</pre>

Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine (backup2 or mail).
Cancel the ve (first we have to rename things which migrate changed so cancelve will find them):

[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/1212.conf

On 2.6.1 you’ll also have to move the private area:
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/1212

<pre>[root@virt12 sbin]# cancelve 1212
v stop 1212
v set 1212 --offline_management=no --save
Delete port redirection
Deleting IP address(es) from pool: 69.55.229.84
Saved parameters for VE 1212
mv /vzconf/1212.conf /vzconf/deprecated-1212
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414
don't forget to remove firewall rules and domains!
[root@virt12 sbin]#</pre>

Note: if the system had backups, [[#cancelve|cancelve]] would offer to remove them. You want to say '''no''' to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.
Then go to backup2 and move the dirs. So you’d do something like this:
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/

We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive.
If moving to the same drive, you can safely preserve hardlinks and move all files with:
sh
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done

To move everyone off a system, you’d do:
for f in `vl`; do migrate <ip> $f; done

Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.

=== vzmigrate: src=2.6.1 -> dst>=2.6.0 ===

This version of vzmigrate works properly with regard to handling ips. It will not notify ve owners of moves as in the above example. Other than that it’s essentially the same.

<pre>[root@virt12 sbin]# vzmigrate 10.1.4.64 -r no 1212:1212:/vz/private/1212:/vz/root/1212
migrating on 10.1.4.64
Connection to destination HN (10.1.4.64) is successfully established
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...
Syncing private area '/vz1/private/1212'
- 100% |*************************************************|
done
Successfully completed
Adding port redirection to VE(1): 4643 8443
Adding IP address(es) to pool: 69.55.229.84
Saved parameters for VE 1212
Starting VE ...
VE is mounted
Adding port redirection to VE(1): 4643 8443
Adding IP address(es): 69.55.229.84
Hostname for VE set: fourmajor.com
File resolv.conf was modified
VE start in progress...
[root@virt12 sbin]#</pre>

Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine (backup2 or mail).
Cancel the ve (first we have to rename things which vzmigrate changed so cancelve will find them):

<pre>[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/1212.conf
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/1212

[root@virt12 sbin]# cancelve 1212
v stop 1212
v set 1212 --offline_management=no --save
Delete port redirection
Deleting IP address(es) from pool: 69.55.229.84
Saved parameters for VE 1212
mv /vzconf/1212.conf /vzconf/deprecated-1212
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414
don't forget to remove firewall rules and domains!
[root@virt12 sbin]#</pre>

Note: if the system had backups, <tt>cancelve</tt> would offer to remove them. You want to say no to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.
Then go to backup2 and move the dirs. So you’d do something like this:
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/

We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive.
If moving to the same drive, you can safely preserve hardlinks and move all files with:
sh
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done

Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.

=== src=2.5.x ===

First, go to the private dir:

cd /vz1/private/

Stop the VE - make sure it stops totally cleanly.

vzctl stop 1212

Then you’d use vemove - a script written to copy over the config, create tarballs of the ve’s data on the destination virt, and cancel the ve on the source system (in this example we’re going to put a ve that was in /vz1/private on the src virt, in /vz/private on the dst virt):

<pre>[root@virt12 sbin]# vemove
ERROR: Usage: vemove veid target_ip target_path_dir
[root@virt12 sbin]# vemove 1212 10.1.4.64 /vz/private/1212
tar cfpP - 1212 --ignore-failed-read | (ssh -2 -c arcfour 10.1.4.64 "split - -b 1024m /vz/private/1212.tar" )
scp /vzconf/1212.conf 10.1.4.64:/vzconf
cancelve 1212
v stop 1212
v set 1212 --offline_management=no --save
Delete port redirection
Deleting IP address(es) from pool: 69.55.229.84
Saved parameters for VE 1212
mv /vzconf/1212.conf /vzconf/deprecated-1212
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414
don't forget to remove firewall rules and domains!
[root@virt12 sbin]#</pre>

Note: if the system had backups, cancelve would offer to remove them. You want to say no to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.
Then go to backup2 and move the dirs. So you’d do something like this:
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/

We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive.
If moving to the same drive, you can safely preserve hardlinks and move all files with:
sh
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done

When you are done, go to /vz/private on the dst virt you will have files like this:

<pre>1212.taraa
1212.tarab
1212.tarac</pre>

Each one 1024m (or less, for the last one) in size.

on the dst server and run:

cat 1212.tar?? | tar xpPBf -

and after 20 mins or so it will be totally untarred. Now since the conf
file is already there, you can go ahead and start the system.

vzctl start 1212

Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.

NOTE: you MUST tar the system up using the virtuozzo version of tar that
is on all the virt systems, and further you MUST untar the tarball with
the virtuozzo tar, using these options: `<tt>tar xpPBf -</tt>`

If you tar up an entire VE and move it to a non-virtuozzo machine, that is
ok, and you can untar it there with normal tar commands, but do not untar
it and then repack it with a normal tar and expect it to work - you need
to use virtuozzo tar commands on virtuozzo tarballs to make it work.

The backups are sort of an exception, since we are just (usually)
restoring user data that was created after we gave them the system, and
therefore has nothing to do with magic symlinks or vz-rpms, etc.

== Moving a VE on the same virt ==

Easy way: 
Scenario 1: ve 123 is to be renamed 1231 and moved from vz1 to vz

vzmlocal 123:1231:/vz/private/1231:/vz/root/1231

Scenario 2: ve 123 is to be moved vz1 to vz

vzmlocal 123:123:/vz/private/123:/vz/root/123

vzmlocal will reboot the ve at the end of the move

Manual/old way:

1) <tt>vzctl stop 123</tt> 
2) <tt>mv /vz1/private/123 /vz/private/.</tt> 
(or cp -a if you want to copy)
3) in <tt>/etc/sysconfig/vz-scripts/123.conf</tt> change value 
of '<tt>VE_PRIVATE</tt>' variable to point to a new private area location
4) <tt>vzctl start 123</tt> 
5) update backups if needed: <tt>mvbackups 123 virtX virt1 vz</tt> 
6) update management scerens 

Notes: a) absolute path to private area is stored in quota file <tt>/var/vzquota/quota.123</tt> - so during first startup quota will be recalculated. 
b) if you're going to write some script to do a job, you MUST be sure that $VEID won't be expanded to '' in ve config file - ie. you need to escape '$'. Otherwise you might have:

VE_PRIVATE="/vz/private/"

in config, and 'vzctl destroy' for this VE ID '''will remove everything under /vz/private/ directory'''.

== Adding a veth device to a VE ==

Not totally sure what this is, but a customer asked for it and here's what we did (as instructed by vz support):

<pre>v set 99 --netif_add eth99 --save
ipdel 99 69.55.230.58
v set 99 --ifname eth99 --ipadd 69.55.230.58 --save
v set 99 --ifname eth99 --gateway 69.55.230.1 --save

vznetcfg net new veth_net

# vznetcfg net list
Network ID Status Master Interface Slave Interfaces
net99 active eth0 veth77.77,veth99.99
veth_net active

# vznetcfg if list
Name Type Network ID Addresses
br0 bridge veth_net
br99 bridge net99
veth99.99 veth net99
eth1 nic 10.1.4.62/24
eth0 nic net99 69.55.227.70/24

vznetcfg br attach br0 eth0

(will remove 99 from orig net and move to veth_net)
vznetcfg net addif veth_net veth99.99

# vznetcfg net list
Network ID Status Master Interface Slave Interfaces
net99 active
veth_net active eth0 veth77.77,veth99.99

(delete the old crap)
vznetcfg net del net99

Then, to add another device in

v set 77 --netif_add eth77 --save
ipdel 77 69.55.230.78
v set 77 --ifname eth77 --ipadd 69.55.230.78 --save
v set 77 --ifname eth77 --gateway 69.55.230.1 --save
v set 77 --save --ifname eth77 --network veth_net

# vznetcfg if list
Name Type Network ID Addresses
veth77.77 veth
br0 bridge veth_net
veth99.99 veth veth_net
eth1 nic 10.1.4.62/24
eth0 nic veth_net 69.55.227.70/24

vznetcfg net addif veth_net veth77.77

# vznetcfg if list
Name Type Network ID Addresses
veth77.77 veth veth_net
br0 bridge veth_net
veth99.99 veth veth_net
eth1 nic 10.1.4.62/24
eth0 nic veth_net 69.55.227.70/24

# vznetcfg net list
Network ID Status Master Interface Slave Interfaces
veth_net active eth0 veth77.77,veth99.99

another example

v set 1182 --netif_add eth1182 --save
ipdel 1182 69.55.236.217
v set 1182 --ifname eth1182 --ipadd 69.55.236.217 --save
v set 1182 --ifname eth1182 --gateway 69.55.236.1 --save
vznetcfg net addif veth_net veth1182.1182
v set 1182 --save --ifname eth1182 --network veth_net

unused/not working commands:
ifconfig veth99.0 0
vznetcfg net list
vznetcfg br new br99 net99
vznetcfg br attach br99 eth0
vznetcfg br show
vznetcfg if list
vznetcfg net addif net99 veth99.99
vznetcfg net addif eth0 net99

---------

vznetcfg br new br1182 net1182

vznetcfg br attach br99 eth0
vznetcfg if list
vznetcfg net addif net99 veth99.99
vznetcfg net addif eth0 net99

vznetcfg net addif eth0 net1182

# vznetcfg net addif veth99.0 net99
--- 8< ---

vznetcfg net new net
# vznetcfg net addif eth0 net99

# vzctl set 99 --save --netif_add eth0 (at this stage veth99.0 interface have to appear
on node)
# vzctl set 99 --save --ifname eth0 --ipadd 69.55.230.58 (and probably few more arguments
here - see 'man vzctl')
# vznetcfg net addif veth99.0 net99</pre>

== Assigning/remove ip from a VE ==

1. Add or remove ips:
ipdel 1234 1.1.1.1 2.2.2.2
ipadd 1234 1.1.1.1 2.2.2.2

2. update Mgmt screens

3. offer to update any DNS we do for them

4. check to see if we had rules for old IP in firwall

== Enabling tun device for a ve ==
Note, there’s a command for this: [[#addtun|addtun]]

For posterity:
Make sure the tun.o module is already loaded before Virtuozzo is started:
lsmod
Allow the VPS to use the TUN/TAP device:
vzctl set 101 --devices c:10:200:rw --save
Create the corresponding device inside the VPS and set the proper permissions:
vzctl exec 101 mkdir -p /dev/net
vzctl exec 101 mknod /dev/net/tun c 10 200
vzctl exec 101 chmod 600 /dev/net/tun

== Remaking a system (on same virt) ==

1. [[#cancelve|cancelve]] (or v destroy x - ONLY if you're POSITIVE no data needs to be saved)

2. [[#vemake|vemake]] using same veid

3. [[#mvbackups|mvbackups]] or [[#vb|vb]] (if new mount point)

4. update mgmt with new dir/ip

5. update firewall if changed ip

== Re-initialize quota for a VE ==

There’s a commamd for this now: [[#clearquota|clearquota]]

For posterity:

vzctl stop 1
vzquota drop 1
vzctl start 1

== Traffic accounting on linux ==

DEPRECATED - all tracking is done via bwdb now. This is how we used to track traffic.

TODO: update for diff versions of vz

Unlike FreeBSD, where we have to add firewall count rules to the system to count the traffic, on virtuozzo counts the traffic for us. You an see the current traffic stats by running `vznetstat`:

<pre># vznetstat
VEID Net.Class Output(bytes) Input(bytes)
-----------------------------------------------
24218 1 484M 39M
24245 1 463M 143M
2451 1 2224M 265M
2454 1 2616M 385M
4149 1 125M 68M
418 1 1560M 34M
472 1 1219M 315M
726 1 628M 317M
763 1 223M 82M
771 1 4234M 437M
-----------------------------------------------
Total: 13780M 2090M</pre>

As you can see the VEID is on a line with the in and out bytes. So, we simply run a cron job:

4,9,14,19,24,29,34,39,44,49,55,59 * * * * /root/vztrafdump.sh

Just like we do on FreeBSD - this one goes through all the VEs in /vz/private and greps the line from vznetstat that matches them and dumps it in /jc_traffic_dump on their system. Then it does it again for all the VEs in /vz1/private. It is important to note that vznetstat runs only once, and the grepping is done from a temporary file that contains that output - we do this because running vznetstat once for each VE that we read out of /vz/private and /vz1/private would take way too long and be too intensive.

You do not need to do anything to facilitate this other than make sure that that cron job is running - the vznetstat counters are always running, and any new VEs that are added to the system will be accounted for automatically.

Traffic resetting no longer works with vz 2.6, so we disable the vztrafdump.sh on those virts.

== Watchdog script ==

On some of the older virts, we have a watchdog running that kills procs that are deemed bad per the following:

/root/watchdog from quar1

<pre>if echo $line | awk '{print $(NF-3)}' | grep [5-9]...
then
# 50-90%
if echo $line | awk '{print $(NF-1)}' | grep "...:.."
then
# running for > 99min
echo $line >> /root/wd.log
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log
kill -9 $pid

fi
if echo $line | awk '{print $(NF-1)}' | grep "....m"
then
# running for > 1000min
echo $line >> /root/wd.log
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log
kill -9 $pid

fi
fi

if echo $line | awk '{print $(NF-3)}' | grep [1-9]...
then
# running for 10-90 percent
if echo $line | awk '{print $NF}' | egrep 'cfusion|counter|vchkpw'
then

if echo $line | awk '{print $(NF-1)}' | grep "[2-9]:.."
then
# between 2-9min
echo $line >> /root/wd.log
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log
kill -9 $pid

elif echo $line | awk '{print $(NF-1)}' | grep "[0-9][0-9]:.."
then
# up to 99min
echo $line >> /root/wd.log
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log
kill -9 $pid

fi
fi
fi</pre>

== Misc Linux Items ==

We are overselling hard drive space ... when you configure a linux system with a certain amount of disk space (the default is 4gigs) you do not actually use up 4gigs of space on the system. The diskspace setting for a user is simply a cap, and they only use up as much space on the actual disk drive as they are actually using.

When you create a new linux system, even though there are some 300 RPMs or so installed, if you run `df -k` you will see that the entire 4gig partition is empty - no space is being used. This is because the files in their system are "magic symlinks" to the template for their OS that is in /vz/template - however, any changes to any of those files will "disconnect" them and they will immediately begin using space in their system. Further, any new files uploaded (even if those new files overwrite existing files) will take up space on the partition.

if you see this:

<pre>[root@virt8 root]# vzctl stop 160 ; vzctl start 160
VE is not running
Starting VE ...
VE is unmounted
VE is mounted
Adding IP address(es): 69.55.226.83 69.55.226.84
bash ERROR: Can't change file /etc/sysconfig/network
Deleting IP address(es): 69.55.226.83 69.55.226.84
VE is unmounted
[root@virt8 root]#</pre>

it probably means they no longer have /bin/bash - copy one in for them

ALSO: another possibility is that they have removed the `ed` RPM from their system - it needs to be reinstalled into their system. But since their system is down, this is tricky ...

VE startup scripts used by 'vzctl' want package 'ed' to be available inside VE. So if package 'ed' will be enabled in OS template config and OS template itself VE #827 is based on - this error should be fixed.

yes, it is possible to add RPM to VE while it not running.
Try to do following:

<pre># cd /vz/template/<OS_template_with_ed_package>/
# vzctl mount 827
# rpm -Uvh --root /vz/root/827 --veid 827 ed-0.2-25.i386.vz.rpm</pre>

Usually theres an error, but its ok

Note: replace 'ed-0.2-25.i386.vz.rpm' in last command with actual
version of 'ed' package you have.

----

So how do I know what template the user has ? cat their conf file and it is listed in there. For example, if the conf file has:

<pre>[root@virt12 sbin]# vc 1103
…snip…
OSTEMPLATE="debian-3.0/20030822"
TEMPLATES="mod_perl-deb30/20030707 mod_ssl-deb30/20030703 mysql-deb30/20030707 proftpd-deb30/20030703 webmin-deb30/20030823 "
…</pre>

then they are on debian 3.0, all of their system RPMs are in /vz/template/debian-3.0, and they are using version 20030822 of that debian 3.0 template. Also, they’ve also got additional packages installed (mod_perl, mod_ssl, etc). Those are also found under /vz/template

----

Edits needed to run java:

When we first created the VEs, the default setting for privvmpages was 93000:94000 ... which was high enough that most people never had problems ... however, you can;t run java or jdk or tomcat or anything java related with that setting. We have found that by setting privvmpages to 610000:615000 that java runs just fine. That is now the default setting. It is exceedingly rare that anyone needs it higher than that, although we have seen it once or twice.

Any problems with java at all - the first thing you need to do is see if the failcnt has raised for privvmpages.

<pre># vzctl start 160
Starting VE ...
vzquota : (error) Quota on syscall for 160: Device or resource busy
Running vzquota on failed for VE 160 [3]</pre>

This is because my pwd is _in_ their private directory - you can't start it until you move out

People seem to have trouble with php if they are clueless newbies. Here are two common problems/solutions:

no... but i figured it out myself. problem was the php.ini file that came
vanilla with the account was not configured to work with apache (the
ENGINE directive was set to off).

everything else seems fine now.

----

the problem was in the php.ini file. I noticed that is wasnt showing
the code when it was in an html file so I looked at the php.ini file
and had to change it so it recognized <? tags aswell as <?php tags.

Also, make sure added to httpd.conf
AddType application/x-httpd-php .php

----

You can set the time zone:

You can change the timezone by doing this:

ln -sf /usr/share/zoneinfo/<zone> /etc/localtime

where <zone> is the zone you want in the /usr/share/zoneinfo/ directory.

----

Failing shm_open calls:

first, please check if /dev/shm is mounted inside VE.
'cat /proc/mounts' command should show something like this:
tmpfs /dev/shm tmpfs rw 0 0

If /dev/shm is not mounted you have 2 ways to solve issue:
1. execute following command inside VE (doesn't require VE reboot):
mount -t tmpfs none /dev/shm
2. add following string to /etc/fstab inside VE and reboot it:
tmpfs /dev/shm tmpfs defaults 0 0

----

You can have a mounted but not running ve
Just:
vzctl mount <veid>

----

When a debian sys can’t get on the network, and you try:

<pre>vzctl set 1046 --ipadd 69.55.227.117
Adding IP address(es): 69.55.227.117
Failed to bring up lo.
Failed to bring up venet0.</pre>

They probably removed iproute package, which must be the one from swsoft. To restore:
<pre># dpkg -i --veid=1046 --admindir=/vz1/private/1046/root/var/lib/dpkg --instdir=/vz1/private/1046/root/ /vz/template/debian-3.0/iproute_20010824-8_i386.vz.deb
(Reading database ... 16007 files and directories currently installed.)
Preparing to replace iproute 20010824-8 (using .../iproute_20010824-8_i386.vz.deb) ...
Unpacking replacement iproute ...
Setting up iproute (20010824-8) ...</pre>

Then restart their ve

----

in a ve i do:

<pre>cd /
du -h .</pre>

and get: 483M .

i do:

<pre>bash-2.05a# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/vzfs 4.0G 2.3G 1.7G 56% /</pre>

how can this be?

Is it possible that quota file was corrupted somehow? Please try to:
<pre>vzctl stop <VEID>
vzquota drop <VEID>
vzquota init <VEID>
vzctl start <VEID></pre>

----

How to stop vz from starting after reboot:

VIRTUOZZO=no
in
/etc/sysconfig/vz

To start:
service vz start
(after setting VIRTUOZZO=yes in /etc/sysconfig/vz)

service vz restart will do some kind of 'soft reboot' -- restart all
VPSes and reload modules without rebooting the node

if you need to shut down all VPSes really really fast, run killall -9 init

----

Postfix tip:

You may want to tweak settings: default_process_limit=10

----

* uptime/load that vz users see isn’t what we see- it's their own load
* Vz overwrites resolve.conf on every boot to what’s in ve config file. Also overwrites hostname
* Quotas in vz: set Gid_limit in vz config file and restart ve. Errors leading up to this will be out of files but no inode or space issues. set to 2000 in conf, and restart vz

= Virtuozzo VPS Management Tools =

== vm ==

To create a new Linux VPS

* get an IP address from mgmt screen
* run df -h to figure out which disk has enough space
* run vm with no arguments to get syntax and exact template name
* use col0xxxx for hostname if they don’t give you a hostname
* copy over dir, veid, ip and password to pending customer screen

vm CID IP hostname /vz[1|2] email[,email] template ( <LB|LBP|LS|LM|LMP|LP> [disk] | <gb_disk/mb_ram/gb_burst> )

== disable ==
To disable a customers Virtuozzo VPS server.

* stop the server
vzctl stop <veid>
* prevent restart on server reboot and user restart via Control Panel
vzctl set <veid> --disabled=yes --offline_management=no --save

== cancelve ==
When a customer cancels all service.

cancelve <veid>
this will:
* stop a ve
* check for backups (offer to remove them from the backup server
and the backup.config)
* rename the private dir
* check for PTR, provide the commands to reset to default
* and rename the ve’s config
* remind you to remove firewall rules
* remind you to remove DNS entries

== ipadd ==
ipadd <veid> <ip> [ip] [ip]
add’s ip(s) to a ve

== ipdel ==
ipdel <veid> <ip> [ip] [ip]
removes ip(s) from a ve

== vc ==
vc <veid>

display the contents of the vz config file for the <veid>

the equivalent of: <tt>cat /vzconf/<veid>.conf</tt>

== vl ==
vl

List all virtual environments running on this machine.

will displays a list of ve #’s, 1 per line. (ostensibly to use in a for loop)

== vp ==
vp <veid>

show processes running under this <veid>.

the equivalent of: <tt>vzps auxww –E <veid></tt>

== vpe ==
DEPRECATED
vpe <veid>
this will allow you to do a vp when a ve is running out of control, the equivalent of (deprecated since vp operates outside the VPS):
<pre>vzctl set <veid> --kmemsize 2100000:2200000
vzctl exec <veid> ps auxw
vzctl set <veid> --kmemsize (ve’s orig lvalue):(ve’s orig hvalue)</pre>

== vt ==
vt <veid>
the equivalent of: <tt>vztop –E <veid></tt>

== vr ==
vr <veid>

Restart a <veid>.

the equivalent of: <tt>vzctl stop <veid>; vzctl start <veid></tt>
You can run this even if the ve is down - the stop command will just fail

== vs ==
vs [veid]
displays status (output of <tt>vzctl status <veid></tt>) on each ve configured on the system (as determined by <tt>/vzconf/*.conf</tt>)
If passed an argument, gives the status for just that ve.
A running system looks like:
<tt>VEID 16066 exist mounted running</tt>

A ve which is not running (but does exist) looks like:
<tt>VEID 9990 exist unmounted down</tt>

A ve which is not running and doesn’t exist looks like:
<tt>VEID 421 deleted unmounted down</tt>

== vs2 ==
vs2 [veid]
this is similar to vs in that it displays status (output of <tt>vzctl status <veid></tt>) on each ve,
but the difference is it’s list comes from doing an ls on the data dirs. This was meant to catch
the rare case where a ve configured but exists.

== vw ==
vw [veid]
displays the output of ‘<tt>w</tt>’ (the equivalent of <tt>vzctl exec <veid> w</tt>) for each configured ve (as determined by <tt>/vzconf/*.conf</tt>). Useful for determing which ve is contributing to a heavily-loaded system.
If passed an argument, gives ‘<tt>w</tt>‘ output for just that ve.
Ex:
<pre>[root@virt2 etc]# vw
134
10:52pm up 79 days, 6:14, 0 users, load average: 0.02, 0.02, 0.00
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
16027
2:52pm up 7 days, 19:54, 0 users, load average: 0.00, 0.00, 0.00
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
16055
2:52pm up 79 days, 6:38, 0 users, load average: 0.00, 0.04, 0.07
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT</pre>

== vwe ==
vwe [constraint]
just like <tt>vw</tt>, but takes a constraint as an argument, only show’s ve’s with loads >= the constraint provided. If no constraint is provided, 1 is used by default

== vzs ==
vzs [veid]
displays the beancounter status for all ve’s, or a particular ve if an argument is passed

== ve ==
ve <veid>

enter a <veid> as root.

the equivalent of: <tt>vzctl enter <veid></tt>

== vx ==
vx <veid> <command>

execute a single command in the <veid> environment.

the equivalent of: <tt>/usr/sbin/vzctl exec <veid> <command></tt>

== dprocs ==
dprocs [count]
a script which outputs a continuous report (or a certain number of reports if an option is passed) of processes stuck in the D state and which VPS’s those procs belong to.

== setmem ==
setmem VEID <256|512|768|1024|1280|1536|2048> [burst GB]
adjusts the memory resources for the VE. If this is on VZ ver >4, you will also be prompted to provide a burst ram amount. Sample usage:

<pre># setmem 21671 1024 4
/usr/sbin/vzctl set 21671 --ram 1024m --swap 3072m --save
</pre>

== afacheck.sh ==
afacheck.sh
displays the health/status of containers and mirrors on an adaptec card (currently quar1, tempvirt1-2, virt9, virt10)- all other are LSI

== backup ==
backup
backup script called nightly to update virt scripts and do customer backups

== checkload.pl ==
checkload.pl
this was intended to be setup as a cronjob to watch processes on a virt when the load
rises above a certain level. Not currently in use.

== findbackuppigs.pl ==
findbackuppigs.pl
looks for files larger than 50MB which customers have asked us to backup. Emails matches
to linux@johncompanies.com

== gatherlinux.pl ==
gatherlinux.pl
gathers up data about ve’s configured and writes to a file. Used for audits against the db

== gb ==
gb <search>
greps backup.config for the given search parameter

== gbg ==
gbg <search>
greps backup.config for the given search parameter and presents just the directories (for clean pasting)

== linuxtrafficgather.pl ==
linuxtrafficgather.pl [yy-mm]
sends a traffic usage summary by ve to support@johncomapnies.com and payments@johncompanies.com.
Optional arguments are year and month (must be in the past). If not passed, assumes last month. Relies on
traffic logs created by netstatreset and netstatbackup

== linuxtrafficwatch.pl ==
linuxtrafficwatch.pl
checks traffic usage and emails support@johncomapnies.com when a ve reaches the warning level (35G) and
the limit (40G). Works on virtuozzo versions <= 2.6.x. We really aren’t using this anymore now that we have netflow.

== linuxtrafficwatch2.pl ==
linuxtrafficwatch2.pl
checks traffic usage and emails support@johncomapnies.com when a ve reaches the warning level (35G) and
the limit (40G). Works on virtuozzo version 2.6.x. We really aren’t using this anymore now that we have netflow.

== load.pl ==
load.pl
feeds info to load mrtg - executed by inetd.

== mb (linux) ==
mb <mount|umount>
(nfs) mounts and umounts dirs to backup2. Shortcuts are mbm and mbu to mount and unmount.

== migrate ==
migrate <ip of node migrating to> <veid> <target_veid> [target dir: vz | vz1 | vz2]
this is basically a wrapper for <tt>vzmigrate</tt> – vzmigrate is a util to seamlessly move a ve from one host to another. This wrapper was written cause vz virtuozzo version 2.6 had a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables. This script mitigates that. Since it makes multiple ssh connections to the target host, it’s a good idea to put the pub key for the src system in the authorized_keys file on the target host. In addition, it emails ve owners when their migration starts and stops (if they place email addresses in a file on their system: /migrate_notify). To move everyone off a system, you’d do:
for f in `vl`; do migrate <ip> $f; done

== migrateonline ==
migrateonline <ip of node migrating to> <veid> <target_veid> [target dir: vz | vz1 | vz2]
this is the same as migrate but will migrate a ve in <tt>–online</tt> mode which means it won’t be shut down at the end of the migration. This only works when migrating ve’s between 2 machines running a 2.6 kernel (currently tempvirt1-2. virt16-19, virt12). If you get an error that the machine you’re trying to migrate to has a different CPU or features, etc, then you have to edit the file and add the –f switch to the vzmigrate line- you can basically ignore this kind of warning (but never ignore a warning about missing templates on the destination node). NOTE: This edit (if made to migrateonline) will be overwritten by the base script during each night’s backup.

== netstatbackup ==
DEPRECATED
netstatbackup
writes traffic count data to a logfile. Works on virtuozzo versions 2.5.x.

== netstatbackup2 ==
DEPRECATED
netstatbackup2
writes traffic count data to a logfile. Works on virtuozzo versions 2.6.x.

== netstatreset ==
DEPRECATED
netstatreset
writes traffic count data to a logfile and resets counters to 0. Works on virtuozzo versions 2.5.x

== netstatreset2 ==
DEPRECATED
netstatreset2
writes traffic count data to a logfile. Works on virtuozzo versions 2.6.x.

== orphanedbackupwatchlinux ==
orphanedbackupwatchlinux
looks for directories on backup2 which aren’t configured in backup.config and offers to
delete them

== rsync.backup (linux) ==
rsync.backup
does customer backups (relies on backup.config)

== startvirt.pl ==
startvirt.pl
forks off start ve commands – keeps 6 running at a time. This is not to be used on systems where fastboot is enabled as it circumvents the benefit of the fastboot. The script will occasionally not exit gracefully and will continue to use up CPU, so it should be watched. Also, don’t exit from the script till you’re sure all ve’s are started – if you do you need to start them manually and may have to free up locks. On some systems, the startvirt script doesn’t exit cleanly and you have to ^C out of it. Be careful though- doing so can leave some VE’s in an odd bootup state and you may need to ‘vr’ them manually. You should check to see which ve’s aren’t running and/or confirm all have started when ^C’ing out of startvirt.

== taskdone (linux) ==
taskdone
when called will email support@johncompanies.com with the hostname of the machine from which it was
executed as the subject

== vb (linux) ==
vb
the equivalent of: <tt>vi /usr/local/sbin/backup.config</tt>

== vemakeXX ==
DEPRECATED
vemakerh9
ve create script for RH9 (see vemake)

vemakedebian30
ve create script for debian 3.0 (Woody) (see vemake)

vemakedebian31
ve create script for debian 3.1 (Sarge) (see vemake)

vemakedebian40
ve create script for debian 4.0 (Etch) (see vemake)

vemakefedora, vemakefedora2, vemakefedora4, vemakefedora5, vemakefedora6, vemakefedora7
ve create script for fedora core 1, 2, 4, 5, 6, 7 respectively (see vemake)

vemakecentos3, vemakecentos4
ve create script for centos 3, 4 respectively (see vemake)

vemakesuse, vemakesuse93, vemakesuse100
ve create script for suse 9.2, 9.3, 10.0 respectively (see vemake)

vemakeubuntu5, vemakeubuntu606, vemakeubuntu606 vemakeubuntu704
ve create script for ubuntu 5.10, 6.06, 6.10, 7.04 respectively (see vemake)

== vemove ==
DEPRECATED
vemove <veid> <target_ip> </vz/private/123>
this script simplifies the old way of moving ve’s from one system to another - in short moving a ve to or from a virt running virtuozzo < 2.6.x
It’s the equivalent of:
<tt>tar cfpP - <veid> --ignore-failed-read | (ssh -2 -c arcfour <target_ip> "split - -b 1024m </vz/private/123>.tar" )</tt>This should only be used if migrate/vzmigrate can’t be used.

== vim.watchdog ==
vim.watchdog
looks for and kills procs matching “vi|vim|nano|pine|elm” that are running for a long time & consuming lots of cpu. Works on virtuozzo versions 2.5.x

== vim.watchdog2 ==
vim.watchdog2
looks for and kills procs matching “vi|vim|nano|pine|elm” that are running for a long time & consuming lots of cpu.
Works on virtuozzo versions 2.6.x.

== vzmigrate ==
vzmigrate <target_ip> -r no <veid>:[dst veid]:[dst /vzX/private/veid]:[dst /vzX/root/veid]
(this is the raw command “wrapped” by migrate/migrateonline) this will seamlessly move a ve from one host to another. The ve will run for the duration of the migration till the very end when it’s shut down, ip moved and started up on the target system. The filesystem on the src will remain. This should be watched – occasionally the move will timeout and leave the system shut down. If target private and root aren’t specified it just puts it in /vz. Only works when both systems are running virtuozzo 2.6.x

== vztrafdump.sh ==
DEPRECATED
vztrafdump.sh
writes traffic usage info by ve to a file called jc_traffic_dump in each ve’s / dir. Works on virtuozzo versions <= 2.5.x.

== vztrafdump2.sh ==
DEPRECATED
vztrafdump2.sh
writes traffic usage info by ve to a file called jc_traffic_dump in each ve’s / dir. Works on virtuozzo versions 2.6.x.

== addtun ==
addtun <veid>
Add’s tun device to ve.

== bwcap ==
bwcap <veid> <kbps>
Ex: <tt>bwcap 1234 512</tt>
Caps a VE’s bandwidth to the amount given

== setdisk ==
setdisk <veid> <diskspace in GB>
Ex: <tt>setdisk 1234 5</tt>
Gives a VE’s a given amount of disk space

== vdf ==
vdf <veid>
the equivalent of: <tt>vzctl exec <veid> df –h</tt>

== vdff ==
vdff
runs a (condensed) vdf for all ve’s in your pwd (must be run from /vz/privateN)

== mvbackups ==
mvbackups <veid> <target_machine> (virt1) <target_dir> (vz1)
moves backups from one location to another on the backup server, and provides you with option to remove entries from current backup.config, and simple paste command to add the config to backup.config on the target server

== checkquota ==
checkquota
for all the ve’s in the cwd (run from /vz/private, /vz1/private, etc) reports what vz quota says they’re using and what the actual usage is (as reported by du)

== clearquota ==
clearquota <veid>
Recalculates a ve’s quota, prints out the usage before and after. The equivalent of:
<tt>vdf <veid>; v stop <veid>; vzquota drop <veid>; v start <veid>; vdf <veid></tt>

== dprocs ==
dprocs
Sometimes the server’s have a large number of processes get stuck in the D state- this script shows (every 3 secs) which VE’s have D procs, which procs
are stuck and a running average of the top “offenders”

== vzstat ==
vstat
sort of like top for VZ. sort VEs by CPU usage by pressing 'o' and then 'c' keys

== stopvirt ==
stopvirt
will stop VEs as fast as it can, 6 at a time. May not exit when complete so you should watch [[#vzstat|vzstat]] in another window.

VPS Management

2014-06-13T18:06:25Z

72.207.101.92: /* /dev/null permission resets */

= Common Problems =
== Login to any machine without a password ==

This is possible via the use of ssh keys. The process is thus:

1. place the public key for your user (root@mail) in the /root/.ssh/authorized_keys file on the server you wish to login to
cat /root/.ssh/id_dsa.pub
(paste that into authorized_keys on the target server). If the file doesn't exist, create it.

2. enable root login (usually only applies to FreeBSD). Edit the /etc/ssh/sshd_config on the target server and change:
<tt>#PermitRootLogin no</tt>
to
<tt>PermitRootLogin yes</tt>

3. Restart the sshd on the target machine. First, find the sshd process:
jailps <hostname> | grep sshd
or
vp <VEID> | grep sshd

Look for the process resembling:
root 17296 0.0 0.0 5280 1036 ? Ss 2011 4:27 /usr/sbin/sshd
(this is the sshd)

Not:
root 6270 0.5 0.0 6808 2536 ? Ss 14:33 0:00 sshd: root [priv]
(this is an sshd child- someone already ssh'd in as root)

Restart the sshd:
kill -1 <PID>

Ex:
kill -1 17296

You may now ssh in.

Once you're done, IF you enabled root login, you should repeat steps 2 and 3 to disable root logins.

== Letting someone in who has locked themselves out (killed sshd, lost pwd) ==

There are two ways people frequently lock themselves out - either they forget a password, or they kill off sshd somehow.

These are actually both fairly easy to solve. First, let's say someone kills off their sshd, or somehow mangles /etc/ssh/sshd_config such that it no longer lets them in.

Their email may be very short, or it may have all sorts of details about how you should fix sshd_config to let them in ... just ignore all of this. They can fix their own mangled sshd. Fixing this is very simple. First, edit the /etc/inetd.conf on their system and uncomment the telnet line:

telnet stream tcp nowait root /usr/libexec/telnetd telnetd
#telnet stream tcp6 nowait root /usr/libexec/telnetd telnetd

(just leave the tcp6 version of telnet commented)

Then, use jailps to list the processes on their system, and find their inetd process. Then simply:

kill -HUP (pid)

where (pid) is the PID of their inetd process. Now they have telnet running on their system and they can log in and do whatever they need to do.

The only complications that could occur are:

a) their firewall config on our firewall has port 23 blocked, in which case you will need to open that - will be covered in a different lesson.

b) they are not running inetd, so you can't HUP it. If this happens, edit their /etc/rc.conf, add the inetd_enable="YES" line, and then kill
their jail with /tmp/jailkill.pl - then restart their jail with the jail line from their quad/safe file. Easy.

If they have forgotten a password,

On 6.x+ you can reset their password with:
jexec <jailID from jls> passwd root

Note: the default password for 6.x jails is 8ico2987, for 4.x it is p455agfa

On 4.x, you need to cd to their etc directory
... for instance:

cd /mnt/data2/198.78.65.136-col00261-DIR/etc

and run:

vipw -d .

Then paste in these two lines (theres a paste with these):

root:$1$krszPxhk$xkCepSnz3mIikT3vCtJCt0:0:0::0:0:Charlie &:/root:/bin/csh
user:$1$Mx9p5Npk$QdMU6c8YQqp2FW2M3irEh/:1001:1001::0:0:User &:/home/user:/bin/sh

overwriting the lines they already have for "user" and "root" - then just tell them that both user and root have been reset to the default password of p455agfa.

For linux, just passwd inside shell or
vzctl set <veid> --userpasswd root:p455agfa –save

Starting in 2009 we began giving out randomized passwords for FreeBSD and Linux as the default password. That is stored with each system in Mgmt. You should look for and reset the password to that password in the event of a reset and refer the customer to use their original password from their welcome email- this way we don’t have to send the password again via email (in clear text).

== sendmail can’t be contacted from ext ip (only locally) ==

By default redhat puts this line in sendmail.mc:

DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')

which makes it only answer on localhost. Comment it out like:

dnl DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')

and then rebuild sendmail.cf with:

m4 /etc/mail/sendmail.mc > /etc/sendmail.cf

== virt doesn’t properly let go of ve’s ip(s) when moved to another system ==

On virtuozzo 2.6 systems, it's been observed that when moving ips from one virt to another that sometimes the routing table will not get updated to reflect the removal of the ip addresses.

A recent example was a customer that was moving to a new ve on a new virt and the ip addresses were traded between the two ve's. After the trade the two systems were not able to talk to each other. When looking at the routing table for the old system all the ip addresses were still in the routing table as being local, like so:

<pre>netstat -rn | grep 69.55.225.149
69.55.225.149 0.0.0.0 255.255.255.255 UH 40 0 0 venet0</pre>

This was preventing traffic to the other system from being routed properly.
The solution is to manually delete the route:

route delete 69.55.225.149 gw 0.0.0.0

Supposedly, this was fixed in 2.6.1

== sshd on FreeBSD 6.2 segfaults ==

First try to reinstall ssh

<pre>cd /usr/src/secure
cd lib/libssh
make depend && make all install
cd ../../usr.sbin/sshd
make depend && make all install
cd ../../usr.bin/ssh
make depend && make all install</pre>

Failing that, find the library that’s messed up:

<pre>ldd /usr/sbin/sshd
libssh.so.3 => /usr/lib/libssh.so.3 (0x280a3000)
libutil.so.5 => /lib/libutil.so.5 (0x280d8000)
libz.so.3 => /lib/libz.so.3 (0x280e4000)
libwrap.so.4 => /usr/lib/libwrap.so.4 (0x280f5000)
libpam.so.3 => /usr/lib/libpam.so.3 (0x280fc000)
libbsm.so.1 => /usr/lib/libbsm.so.1 (0x28103000)
libgssapi.so.8 => /usr/lib/libgssapi.so.8 (0x28112000)
libkrb5.so.8 => /usr/lib/libkrb5.so.8 (0x28120000)
libasn1.so.8 => /usr/lib/libasn1.so.8 (0x28154000)
libcom_err.so.3 => /usr/lib/libcom_err.so.3 (0x28175000)
libroken.so.8 => /usr/lib/libroken.so.8 (0x28177000)
libcrypto.so.4 => /lib/libcrypto.so.4 (0x28183000)
libcrypt.so.3 => /lib/libcrypt.so.3 (0x28276000)
libc.so.6 => /lib/libc.so.6 (0x2828e000)
libmd.so.3 => /lib/libmd.so.3 (0x28373000)</pre>

md5 them and compare to other jail hosts or jails running on host

for libcrypto reinstall:
<pre>/usr/src/crypto
make depend && make all install</pre>

== Moving a server to a different account in the management system ==

This is a tough one. You gotta touch a bunch of tables in the mysql database.
<pre>
systems
subs
ipmap
</pre>

basically, get the sysid and go off that:

<pre>
update systems set cid='newcid' where sysid=xxxxx
update subs set cid='newcid' where sysid=xxxxx
update ipmap set cid='newcid' where sysid=xxxxx
</pre>

You then need to rename the server disk space to change the old ID to the new one.

For a jail -
<pre>
stopjail <hostname>
mv 69.55.2XX.XXX-col0<oldid> 69.55.2XX.XXX-col0<newid>
mv 69.55.2XX.XXX-col0<oldid>-DIR 69.55.2XX.XXX-col0<newid>-DIR
vi /usr/local/jail/bin/quad1 [Update the ID in the quad file]
startjail <hostname>
</pre>

For a virt -
You will have to
* stop the virt
* rename the directories
* modify the config file /vzconf/<veid>.conf
* rename the virt conf file
* restart the new virt.

that ought to do it..

= FreeBSD VPS =

== Starting jails: Quad/Safe Files ==

FreeBSD customer systems do not start up automatically at boot time. When one of our freebsd machines boots up, it boots up, and does nothing else. To start jails, we put the commands to start each jail into a shell script(s) and run the script(s). Jail startup is something that needs to be actively monitored, which is why we don’t just run the script automatically. More on monitoring later.

NOTE: >=7.x we have moved to 1 quad file: <tt>quad1</tt>. Startups are not done by running each quad, but rather [[#startalljails|startalljails]] which relies on the contents of <tt>quad1</tt>. The specifics of this are lower in this article. What follows here applies for pre 7.x systems.

There are eight files in <tt>/usr/local/jail/rc.d</tt>:

<pre>jail3# ls /usr/local/jail/rc.d/
quad1 quad2 quad3 quad4 safe1 safe2 safe3 safe4
jail3#</pre>

four quad files and four safe files.

Each file contains an even number of system startup blocks (total number of jails divided by 4)

The reason for this is, if we make one large script to startup all the systems at boot time, it will take too long - the first system in the script will start up right after system boot, which is great, but the last system may not start for another 20 minutes.

Since there is no way to parralelize this during the startup procedure, we simply open four terminals (in screen window 9) and run each script, one in each terminal. This way they all run simultaneously, and the very last system in each startup script gets started in 1/4th the time it would if there was one large file

The files are generally organized so that quad/safe 1&2 have only jails from disk 1, and quad/safe 3&4 have jails from disk 2. This helps ensure that only 2 fscks on any disk are going on at once. Further, they are balanced so that all quad/safe’s finish executing around the same time. We do this by making sure each quad/safe has a similar number of jails and represents a similar number of inodes (see js).

The other, very important reason we do it this way, and this is the reason there are quad files and safe files, is that in the event of a system crash, every single vn-backed filesystem that was mounted at the time of system crash needs to be fsck'd. However, fsck'ing takes time, so if we shut the system down gracefully, we don't want to fsck.

Therefore, we have two sets of scripts - the four quad scripts are identical to the four safe scripts except for the fact that the quad scripts contain fsck commands for each filesystem.

So, if you shut a system down gracefully, start four terminals and run safe1 in window one, and safe2 in window 2, and so on.

If you crash, start four terminals (or go to screen window 9) and run quad1 in window one, and quad2 in window 2, and so on.

Here is a snip of (a 4.x version) quad2 from jail17:

<pre>vnconfig /dev/vn16 /mnt/data2/69.55.228.7-col00820
fsck -y /dev/vn16
mount /dev/vn16c /mnt/data2/69.55.228.7-col00820-DIR
chmod 0666 /mnt/data2/69.55.228.7-col00820-DIR/dev/null
jail /mnt/data2/69.55.228.7-col00820-DIR mail1.phimail.com 69.55.228.7 /bin/sh /etc/rc

# moved to data2 col00368
#vnconfig /dev/vn28 /mnt/data2/69.55.236.132-col00368
#fsck -y /dev/vn28
#mount /dev/vn28c /mnt/data2/69.55.236.132-col00368-DIR
#chmod 0666 /mnt/data2/69.55.236.132-col00368-DIR/dev/null
#jail /mnt/data2/69.55.236.132-col00368-DIR limehouse.org 69.55.236.132 /bin/sh /etc/rc

echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’
vnconfig /dev/vn22 /mnt/data2/69.55.228.13-col01063
fsck -y /dev/vn22
mount /dev/vn22c /mnt/data2/69.55.228.13-col01063-DIR
chmod 0666 /mnt/data2/69.55.228.13-col01063-DIR/dev/null
jail /mnt/data2/69.55.228.13-col01063-DIR www.widestream.com.au 69.55.228.13 /bin/sh /etc/rc

# cancelled col00106
#vnconfig /dev/vn15 /mnt/data2/69.55.238.5-col00106
#fsck -y /dev/vn15
#mount /dev/vn15c /mnt/data2/69.55.238.5-col00106-DIR
#chmod 0666 /mnt/data2/69.55.238.5-col00106-DIR/dev/null
#jail /mnt/data2/69.55.238.5-col00106-DIR mail.azebu.net 69.55.238.5 /bin/sh /etc/rc</pre>

As you can see, two of the systems specified are commented out - presumably those customers cancelled, or were moved to new servers.

As you can see, the vnconfig line is the simpler command line, not the longer one that was used when it was first configured. As you can see, all that is done is, vnconfig the filesystem, then fsck it, then mount it. The fourth command is the `jail` command used to start the system – but that will be covered later.

Here is the safe2 file from jail17:

<pre>vnconfig /dev/vn16 /mnt/data2/69.55.228.7-col00820
mount /dev/vn16c /mnt/data2/69.55.228.7-col00820-DIR
chmod 0666 /mnt/data2/69.55.228.7-col00820-DIR/dev/null
jail /mnt/data2/69.55.228.7-col00820-DIR mail1.phimail.com 69.55.228.7 /bin/sh /etc/rc

# moved to data2 col00368
#vnconfig /dev/vn28 /mnt/data2/69.55.236.132-col00368
#mount /dev/vn28c /mnt/data2/69.55.236.132-col00368-DIR
#chmod 0666 /mnt/data2/69.55.236.132-col00368-DIR/dev/null
#jail /mnt/data2/69.55.236.132-col00368-DIR limehouse.org 69.55.236.132 /bin/sh /etc/rc

echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’
vnconfig /dev/vn22 /mnt/data2/69.55.228.13-col01063
mount /dev/vn22c /mnt/data2/69.55.228.13-col01063-DIR
chmod 0666 /mnt/data2/69.55.228.13-col01063-DIR/dev/null
jail /mnt/data2/69.55.228.13-col01063-DIR www.widestream.com.au 69.55.228.13 /bin/sh /etc/rc

# cancelled col00106
#vnconfig /dev/vn15 /mnt/data2/69.55.238.5-col00106
#mount /dev/vn15c /mnt/data2/69.55.238.5-col00106-DIR
#chmod 0666 /mnt/data2/69.55.238.5-col00106-DIR/dev/null
#jail /mnt/data2/69.55.238.5-col00106-DIR mail.azebu.net 69.55.238.5 /bin/sh /etc/rc</pre>

As you can see, it is exactly the same, but it does not have the fsck lines.

Take a look at the last entry - note that the file is named:

/mnt/data2/69.55.238.5-col00106

and the mount point is named:

/mnt/data2/69.55.238.5-col00106-DIR

This is the general format on all the FreeBSD systems. The file is always named:

IP-custnumber

and the directory is named:

IP-custnumber-DIR

If you run safe when you need a fsck, the mount will fail and jail will fail:

# mount /dev/vn1c /mnt/data2/jails/65.248.2.131-ns1.kozubik.com-DIR
mount: /dev/vn1c: Operation not permitted

No reboot needed, just run the quad script

Starting with 6.x jails, we added block delimiters to the quad/safe files, the block looks like:

<pre>echo '## begin ##: nuie.solaris.mu'
fsck -y /dev/concat/v30v31a
mount /dev/concat/v30v31a /mnt/data1/69.55.228.218-col01441-DIR
mount_devfs devfs /mnt/data1/69.55.228.218-col01441-DIR/dev
devfs -m /mnt/data1/69.55.228.218-col01441-DIR/dev rule -s 3 applyset
jail /mnt/data1/69.55.228.218-col01441-DIR nuie.solaris.mu 69.55.228.218 /bin/sh /etc/rc
echo '## end ##: nuie.solaris.mu'</pre>

These are more than just informative when running quad/safe’s, the echo lines MUST be present for certain tools to work properly. So it’s important that any updates to the hostname also be updated on the 2 echo lines. For example, if you try to startjail a jail with a hostname which is on the jail line but not the echo lines, the command will return with host not found.

=== FreeBSD 7.x+ notes ===

Starting with the release of FreeBSD 7.x, we are doing jail startups in a slightly different way. First, thereis only 1 file: <tt>/usr/local/jail/rc.d/quad1</tt>
There are no other quads or corresponding safe files. The reason for this is twofold, 1. We can pass –C to fsck which will tell is to skip the fsck if the fs is clean (no more need for safe files), 2. We have a new startup script which can be launched multiple times, running in parallel to start jails, where quad1 is the master jail file.
Quad1 could still be run as a shell script, but it would take a very long time for it to run completely so it’s not advisable; or you should break it down into smaller chunks (like quad1, quad2, quad3, etc)

Here is a snip of (a 7.x version) quad1 from jail2:

<pre>echo '## begin ##: projects.tw.com'
mdconfig -a -t vnode -f /mnt/data1/69.55.230.46-col01213 -u 50
fsck -Cy /dev/md50c
mount /dev/md50c /mnt/data1/69.55.230.46-col01213-DIR
mount -t devfs devfs /mnt/data1/69.55.230.46-col01213-DIR/dev
devfs -m /mnt/data1/69.55.230.46-col01213-DIR/dev rule -s 3 applyset
jail /mnt/data1/69.55.230.46-col01213-DIR projects.tw.com 69.55.230.46 /bin/sh /etc/rc
echo '## end ##: projects.tw.com'</pre>

Cancelled jails are no longer commented out and stored in quad1, rather they’re moved to <tt>/usr/local/jail/rc.d/deprecated</tt>

To start these jails, start the 4 ssh sessions as you would for a normal crash and then instead of running quad1-4, instead run startalljails in each window. IMPORTANT- before running startalljails you should make sure you ran preboot once as it will clear out all the lockfiles and enable startalljails to work properly.

== Problems with the quad/safe files ==

When you run the quad/safe files, there are two problems that can occur - either a particular system will hang during initialization, OR a system will spit out output to the screen, impeding your ability to do anything. Or both.

First off, when you start a jail, you see output like this:

<pre>Skipping disk checks ...
adjkerntz[25285]: sysctl(put_wallclock): Operation not permitted
Doing initial network setup:.
ifconfig: ioctl (SIOCDIFADDR): permission denied
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384
Additional routing options: TCP keepalive=YESsysctl:
net.inet.tcp.always_keepalive: Operation not permitted.
Routing daemons:.
Additional daemons: syslogd.
Doing additional network setup:.
Starting final network daemons:.
ELF ldconfig path: /usr/lib /usr/lib/compat /usr/X11R6/lib /usr/local/lib
a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout /usr/X11R6/lib/aout
Starting standard daemons: inetd cron sshd sendmail sendmail-clientmqueue.
Initial rc.i386 initialization:.
Configuring syscons: blanktime.
Additional ABI support:.
Local package initialization:.
Additional TCP options:.</pre>

Now, let's look at this line, near the end:

Local package initialization:.

This is where a list of daemons that are set to start at boot time willshow up. You might see something like:

Local package initialization: mysqld apache sendmail sendmail-clientmqueue

Or something like this:

Local package initialization: postgres postfix apache

The problem is that many systems (about 4-5 per machine) will hang on that line. Basically it will get to some of the way through the total daemons to be started:

Local package initialization: mysqld apache

and will just sit there. Forever.

Fortunately, pressing ctrl-c will break out of it. Not only will it break out of it, but it will also continue on that same line and start the other daemons:

Local package initialization: mysqld apache ^c sendmail-clientmqueue

and then continue on to finish the startup, and then move to the next system to be started.

So what does this mean? It means that if a machine crashes, and you start four screen-windows to run four quads or four safes, you need to periodically cycle between them and see if any systems are stuck at that point, causing their quad/safe file to hang. A good rule of thumb is, if you see a system at that point in the startup, give it another 100 seconds - if it is still at the exact same spot, hit ctrl-c. Its also a good idea to go back into the quad file (just before the first command in the jail startup block) and note that this jail tends to need a control-c or more time as follows:
<pre>echo '### NOTE ### slow sendmail'
echo '### NOTE ###: ^C @ Starting sendmail.'</pre>

'''NEVER''' hit ctrl-c repeatedly if you don't get an immediate response - that will cause the following jail’s startup commands to be aborted.

A second problem that can occur is that a jail - maybe the first one in that particular quad/safe, maybe the last one, or maybe one in the middle, will start spitting out status or error messages from one of its init scripts. This is not a problem - basically, hit enter a few times and see if you get a prompt - if you do get a prompt, that means that the quad/safe script has already completed. Therefore it is safe to log out (and log out of the user that you su'd from) and then log back in (if necessary).

The tricky thing is, if a system in the middle starts flooding with messages, and you hit enter a few times and don't get a prompt. Are you not getting a prompt because some subsequent system is hanging at the initialization, as we discussede above ? Or are you not getting a prompt because that quad file is currently running an fsck ? Usually you can tell by scrolling back in screen’s history to see what it was doing before you started getting the messages.

If you don’t get clues from history, you have to use your judgement - instead of giving it 100 seconds to respond, perhaps give it 2-3 mins ... if you still get no response (no prompt) when you hit enter, hit ctrl-c. However, be aware that you might still be hitting ctrl-c in the middle of an fsck. This means you will get an error like "filesystem still marked dirty" and then the vnconfig for it will fail and so will the jail command, and the next system in the quad file will then start starting up.

If this happens, just wait until the end of all the quad files have finished, and start that system manually.

If things really get weird, like a screen flooded with errors, and you can't get a prompt, and ctrl-c does nothing, then you need to just eventually (give it ten mins or so) just kill that window with ctrl-p, then k, and then log in again and manually check which systems are now running and which aren't, and manually start up any that are not.

Don't EVER risk running a particular quad/safe file a second time.
If the quad/safe script gets executed twice, reboot the machine immediately.

So, for all the above reasons, anytime a machine crashes and you run all the quads or all the safes, '''always''' check every jail afterwards to make sure it is running - even if you have no hangs or complications at all.
Run this command:

<tt>[[#jailpsall|jailpsall]]</tt>

Note: [[#postboot|postboot]] also populates ipfw counts, so it '''should not be run multiple times''', use <tt>jailpsall</tt> for subsequent extensive ps’ing

And make sure they all show as running. If one does not show as running, check its /etc/rc.conf file first to see if maybe it is using a different hostname first before starting it manually.

One thing we have implemented to alleviate these startup hangs and noisy jails, is to put jail start blocks that are slow or hangy at the bottom of the safe/quad file. Further, for each bad jail we note in each quad/safe just before the start block something like:

echo ‘### NOTE ### ^C @ Local package initialization: pgsqlmesg: /dev/ttyp1: Operation not permitted’

That way we’ll be prepared to ^C when we see that message appear during the quad/safe startup process. If you observe a new, undocumented hang, '''after''' the quad/safe has finished, place a line similar to the above in the quad file, move the jail start block to the end of the file, then run [[#buildsafe|buildsafe]]

== Making new customer jail 4.x ==

1. run [[#js|js]] to figure out which partition, IP to put it on, find an unused vn, and choose which quad it should go in

2. use col00xxx for both hostnames if they don’t give you a hostname

3. copy over dir and ip to pending customer screen

Usage: <tt>jailmake IP filepath vnX hostname shorthost quad/safe# ipfw# email [gb disk, default 4]</tt>

jail14# /tmp/jailmake 69.55.226.152 /mnt/data2/69.55.226.152-col00182 vn23 negev.cerastes.org negev 4 182 cerastes@cerastes.org

== Making new customer jail 6.x ==

1. run [[#js|js]] to figure out which gvinum vols are free (and which mnt (disk) each gvinum goes with/to), IP to put it on and choose which quad it should go in

2. use col00xxx for both hostnames if they don’t give you a hostname

3. copy over dir and ip to pending customer screen

Usage: <tt>jailmake IP filepath vN[,vN] hostname shorthost quad/safe# ipfw# email</tt>

jail19# jailmake 69.55.236.17 /mnt/data1/69.55.236.17-col01574 v10 uncle-leo.commadev.com uncle-leo 1 1574 lee@commadev.com,lee@gmail.com

== Changing an ip for freebsd VPS ==

*stopjail (hostname)
*on systems using vnfile or mdfile: rename vnfile/mdfile, using new ip
*rename dir using new ip
*Edit quad (make sure to update all lines: directory, vn/mdfile)
*buildsafe (not necessary on systems having only quad1)
*if you're adding an ip not already on the host machine: <tt>ipadd x.x.x.x</tt>
*startjail (hostname)
if backups: <tt>mvbackups</tt> ...
*Edit dir/ip in mgmt
*if any rules, update any firewall rules to use new ip (careful not to make sure there arent multiple rules per ipfw#- search by colo#)
*(if customer asks and has domains) update their domains on ns1c
*optional: update ipfw counters on the host server

== Rename a gconcat vol ==

*stop jail, unmount everything
*gconcat stop vol
*gconcat clear /dev/gvinum/vN
*gconcat clear /dev/gvinum/vN+1
*gconcat label -v newname /dev/gvinum/vN /dev/gvinum/vN+1
*bsdlabel /dev/concat/newname
(make sure a partition is not fstype unused, if so change to: 4.2BSD)

== Remaking a VPS (on same jail) ==

# take him down (stopjail)
# make note of host, vn/gvin/md, ip
# remove from quad/safe
# move vnfile (rename)
# remove dir (jailmake will make a new one)
# remove ipfw counts
# jailmake
# update backup script
# update db with new dir/ip (use the “move” link, mark old as stopped today)
# update firewall if changed ip

== Recovering from a crash (FreeBSD) ==

=== Diagnose whether you have a crash ===
The most important thing is to get the machine and all jails back up as soon as possible. Note the time, you’ll need to create a crash log entry (Mgmt. -> Reference -> CrashLog). The first thing to do is head over to the [[Screen#Screen_Organization|serial console screen]] and see if there’s any kernel error messages output. Try to copy any messages (or just a sample of repeating messages) you see into the notes section of the crash log. If there are no messages, the machine may just be really busy- wait a bit (5-10min) to see if it comes back. If it's still pinging, odds are its very busy. Note, if you see messages about swap space exhausted, the server is obviously out of memory, however it may recover briefly enough for you to get a jtop in to see who's lauched a ton of procs (most likely) and then issue a quick jailkill to get it back under control.

If it doesn't come back, or the messages indicate a fatal error, you will need to proceed with a power cycle (ctrl+alt+del will not work).

=== Power cycle the server ===
If this machine is not a Dell 2950 with a [[DRAC/RMM#DRAC|DRAC card]] (i.e. if you can’t ssh into the DRAC card (as root, using the standard root pass) and issue
racadm serveraction hardreset
then you will need someone at the data center power the macine off, wait 30 sec, then turn it back on. Make sure to re-attach via console:
tip jailX
immediately after power down.

=== (Re)attach to the console ===
Stay on the console the entire time during boot. As the BIOS posts- look out for the RAID card output- does everything look healthy? The output may be scrambled, look for "DEGRADED" or "FAILED". Once the OS starts booting you will be disconnected (dropped back to the shell on the console server) a couple times during the boot up. The reason you want to quickly re-attach is two-fold: 1. If you don’t reattach quickly then you won’t get any console output, 2. you want to be attached before the server ''potentially'' starts (an extensive) fsck. If you attach after the fsck begins, you’ll have seen no indication it started an fsck and the server will appear frozen during startup- no output, no response.

IMPORTANT NOTE: on some older FreeBSD systems, there will be no output to the video (KVM) console as it boots up. The console output is redirected to the serial port ... so if a jail crashes, and you attach a kvm, the output during the bootup procedure will not be shown on the screen. However, when the bootup is done, you will get a login prompt on the screen and will be able to log in as normal. <tt>/boot/loader.conf</tt> is where serial console redirect output lives, so comment that if you want to catch output on kvm.
On newer systems it sends most output to both locations.

=== Assess the heath of the server ===
Once the server boots up fully, you should be able to ssh in. Look around- make sure all the mounts are there and reporting the correct size/usage (i.e. /mnt/data1 /mnt/data2 /mnt/data3 - look in /etc/fstab to determine which mount points should be there), check to see if RAID mirrors are healthy. See [[RAID_Cards#Common_CLI_commands_.28megacli.29|megacli]], [[#aaccheck|aaccheck]]

Before you start the jails, you need to run [[#preboot|preboot]]. This will do some assurance checks to make sure things are prepped to start the jails. Any issues that come out of preboot need to be addressed before starting jails.

=== Start jails ===
[[#Starting_jails:_Quad.2FSafe_Files|More on starting jails]]
Customer jails (the VPSs) do not start up automatically at boot time. When a FreeBSD machines boots up, it boots up, and does nothing else. To start jails, we put the commands to start each jail into a shell script(s) and run the script(s). Jail startup is something that needs to be actively monitored, which is why we don’t just run the script automatically.

In order to start jails, we run the quad files: quad1 quad2 quad3 and quad4 (on new systems there is only quad1). If the machine was cleanly rebooted- which wouldn't be the case if this was a crash, you may run the safe files (safe1 safe2 safe3 safe4) in lieu of quads.

Open up 4 logins to the server (use the windows in [[Screen#Screen_Organization|a9]])
In each of the 4 windows you will:

If there is a [[#startalljails|startalljails]] script (and only quad1), run that command in each of the 4 windows. It will parse through the quad1 file and start each jail. Follow the instructions [[#Problems_with_the_quad.2Fsafe_files|here]] for monitoring startup. Note that you can be a little more lenient with jails that take awhile to start- startalljails will work around the slow jails and start the rest. As long as there aren't 4 jails which are "hung" during startup, the rest will get started eventually.
-or-
If there is no startalljails script, there will be multiple quad files. In each of the 4 windows, start each of the quads. i.e. start quad1 in window1, quad2 in window2 and so on. DO NOT start any quad twice. It will crash the server. If you accidentally do this, just jailkill all the jails which are in the quad and run the quad again. Follow the instructions here for monitoring quad startup.

Note the time the last jail boots- this is what you will enter in the crash log.

Save the crash log.

=== Check to make sure all jails have started ===
There's a simple script which will make sure all jails have started, and enter the ipfw counter rules: [[#postboot|postboot]]
Run postboot, which will do a jailps on each jail it finds (excluding commented out jails) in the quad file(s). We're looking for 2 things:
# systems spawning out of control or too many procs
# jails which haven't started
On 7.x and newer systems it will print out the problems (which jails haven't started) at the conclusion of postboot.
On older systems you will need to watch closely to see if/when there's a problem, namely:

[hostname] doesnt exist on this server

When you get this message, it means one of 2 things:
1. the jail really didn't start:
When a jail doesn't start it usually boils down to a problem in the quad file. Perhaps the path name is wrong (data1 vs data2) or the name of the vn/mdfile is wrong. Once this is corrected, you will need to run the commands from the quad file manually, or you may use <tt>startjail <hostname></tt>

2. the customer has changed their hostname (and not told us) so their jail ''is'' running, just under a different hostname:
On systems with jls, this is easy to rectify. First, get the customer info: <tt>g <hostname></tt>
Then look for the customer in jls: <tt>jls | grep <col0XXXX></tt>
From there you will see their new hostname- you should update that hostname in the quad file: don't forget to edit it on the <tt>## begin ##</tt> and <tt>## end ##</tt> lines, and in mgmt.
On older systems without jls, this will be harder, you will need to look further to see their hostname- perhaps its in their /etc/rc.conf

Once all jails are started, do some spot checks- try to ssh or browse to some customers, just to make sure things are really ok.

== Adding disk to a 7.x/8.x jail ==
or
== Moving customer to a different drive (md) ==

NOTE: this doesn’t apply to mx2 which uses gvinum. Use same procedure as 6.x
NOTE: if you unmount before mdconfig, re-mdconfig (attach) then unmount then mdconfig -u again

(parts to change/customize are <tt>red</tt>)

If someone wants more disk space, there’s a paste for it, send it to them (explains about downtime, etc).

1. Figure out the space avail from <tt>js</tt>. Ideally, you want to put the customers new space on a different partition (and create the new md on the new partition).

2. make a mental note of how much space they're currently using

3. <tt>jailkill <hostname></tt>

4. Umount it (including their devfs) but leave the md config’d (so if you use stopjail, you will have to re-mdconfig it)

5. <tt>g <customerID></tt> to get the info (IP/cust#) needed to feed the new mdfile and mount name, and to see the current md device.

6a. When there's enough room to place new system on an alternate, or the same drive:
USE CAUTION not to overwrite (touch, mdconfig) existing md!! 
<tt>touch /mnt/data3/69.55.234.66-col01334 
mdconfig -a -t vnode -s 10g -f /mnt/data3/69.55.234.66-col01334 -u 97 
newfs /dev/md97</tt>

Optional- if new space is on a different drive, move the mount point directory AND use that directory in the mount and cd commands below: 
<tt>mv /mnt/data1/69.55.234.66-col01334-DIR /mnt/data3/69.55.234.66-col01334-DIR</tt>

<tt>mount /dev/md97 /mnt/data3/69.55.234.66-col01334-DIR 
cd /mnt/data3/69.55.234.66-col01334-DIR</tt>

confirm you are mounted to /dev/md97 and space is correct: 
<tt>df .</tt>

do the dump and pipe directly to restore: 
<tt>dump -0a -f - /dev/md1 | restore -r -f - 
rm restoresymtable</tt>

when dump/restore completes successfully, use <tt>df</tt> to confirm the restored data size matches the original usage figure

md-unconfig old system: 
<tt>mdconfig -d -u 1</tt>

archive old mdfile. 
<tt>mv /mnt/data1/69.55.237.26-col00241 /mnt/data1/old-col00241-mdfile-noarchive-20091211</tt>

edit quad (vq1) to point to new (/mnt/data3) location AND new md number (md97)

restart the jail: 
<tt>startjail <hostname></tt>

6b. When there's not enough room on an alternate partition, or on the same drive...but there is enough room if you were to remove the existing customer's space:

mount backup nfs mounts: 
<tt>mbm</tt> 
(run <tt>df</tt> to confirm backup mounts are mounted)

dump the customer to backup2 or backup1 
<tt>dump -0a -f /backup4/col00241.20120329.noarchive.dump /dev/md1</tt> 
(when complete WITHOUT errors, <tt>du</tt> the dump file to confirm it matches size, roughly, with usage)

unconfigure and remove old mdfile 
<tt>mdconfig -d -u 1 
rm /mnt/data1/69.55.237.26-col00241</tt> 
(there should now be enough space to recreate your bigger system. If not, run sync a couple times)

create the new system (ok to reuse old mdfile and md#): 
<tt>touch /mnt/data1/69.55.234.66-col01334 
mdconfig -a -t vnode -s 10g -f /mnt/data1/69.55.234.66-col01334 -u 1 
newfs /dev/md1 
mount /dev/md1 /mnt/data1/69.55.234.66-col01334-DIR 
cd /mnt/data1/69.55.234.66-col01334-DIR</tt>

confirm you are mounted to /dev/md1 and space is correct: 
<tt>df .</tt>

do the restore from the dumpfile on the backup server: 
<tt>restore -r -f /backup4/col00241.20120329.noarchive.dump . 
rm restoresymtable</tt>

when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure

umount nfs: 
<tt>mbu</tt>

If md# changed (or mount point), edit quad (<tt>vq1</tt>) to point to new (/mnt/data3) location AND new md number (md1)

restart the jail: 
<tt>startjail <hostname></tt>

7. update disk (and dir if applicable) in mgmt screen

8. update backup list AND move backups, if applicatble

Ex: <tt>mvbackups 69.55.237.26-col00241 jail9 data3</tt>

9. Optional: archive old mdfile

<tt>mbm 
gzip -c old-col01588-mdfile-noarchive-20120329 > /deprecated/old-col01588-mdfile-noarchive-20120329.gz 
mbu 
rm old-col01588-mdfile-noarchive-20120329</tt>

== Adding disk to a 6.x jail (gvinum/gconcat) ==
or
== Moving customer to a different drive (gvinum/gconcat) ==

(parts to change are highlighted)

If someone wants more disk space, there’s a paste for it, send it to them (explains about downtime, etc).

1. Figure out the space avail from [[#js|js]]. Ideally, you want to put the customers new space on a different partition (and create the new md on the new partition).

2. make a mental note of how much space they're currently using

3. <tt>[[#stopjail|stopjail]] <hostname></tt>

4. <tt>[[#g|g]] <customerID></tt> to get the info (IP/cust#) needed to feed the new mount name and existing volume/device.

5a. When there's enough room to place new system on an alternate, or the same drive (using only UNUSED - including if it's in use by the system in question - gvinum volumes):

Configure the new device: 
A. for a 2G system (single gvinum volume): 
<tt>bsdlabel -r -w /dev/gvinum/v123 
newfs /dev/gvinum/v123a</tt> 

-or-

B. for a >2G system (create a gconcat volume): 
try to grab a contiguous block of gvinum volumes. gconcat volumes MAY NOT span drives (i.e. you cannot use a gvinum volume from data3 and a volume from data2 in the same gconcat volume).

<tt>gconcat label v82-v84 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84 
bsdlabel -r -w /dev/concat/v82-v84 
newfs /dev/concat/v82-v84a</tt>

Other valid gconcat examples: 
<tt>gconcat label v82-v84v109v112 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84 /dev/gvinum/v109 /dev/gvinum/v112 
gconcat label v82v83 /dev/gvinum/v82 /dev/gvinum/v83</tt> 
Note, long names will truncate: v144v145v148-v115 will truncate to v144v145v148-v1 (so you will refer to it as v144v145v148-v1 thereafter)

Optional- if new volume is on a different drive, move the mount point directory (get the drive from js output) AND use that directory in the mount and cd commands below: 
<tt>mv /mnt/data1/69.55.234.66-col01334-DIR /mnt/data3/69.55.234.66-col01334-DIR</tt>

confirm you are mounted to the device (<tt>/dev/gvinum/v123a</tt> OR <tt>/dev/concat/v82-v84</tt>) and space is correct: 
A. <tt>mount /dev/gvinum/v123a /mnt/data3/69.55.234.66-col01334-DIR</tt> 
-or- 
B. <tt>mount /dev/concat/v82-v84a /mnt/data3/69.55.234.66-col01334-DIR</tt>

<tt>cd /mnt/data3/69.55.234.66-col01334-DIR 
df .</tt>

do the dump and pipe directly to restore: 
<tt>dump -0a -f - /dev/gvinum/v1 | restore -r -f - 
rm restoresymtable</tt>

when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure

edit quad (<tt>vq1</tt>) to point to new (/mnt/data3) location AND new volume (<tt>/dev/gvinum/v123a</tt> or <tt>/dev/concat/v82-v84a</tt>) , run <tt>buildsafe</tt>

restart the jail: 
<tt>startjail <hostname></tt>

5b. When there's not enough room on an alternate partition, or on the same drive...but there is enough room if you were to remove the existing customer's space (i.e. if you want/need to reuse the existing gvinum volumes and add on more):

mount backup nfs mounts: 
<tt>mbm</tt> 
(run df to confirm backup mounts are mounted)

dump the customer to backup2 or backup1 
<tt>dump -0a -f /backup4/col00241.20120329.noarchive.dump /dev/gconcat/v106-v107</tt> 
(when complete WITHOUT errors, du the dump file to confirm it matches size, roughly, with usage)

unconfigure the old gconcat volume 
list member gvinum volumes:

<tt>gconcat list v106v107</tt>

Output will resemble: 
<pre>Geom name: v106v107
State: UP
Status: Total=2, Online=2
Type: AUTOMATIC
ID: 3530663882
Providers:
1. Name: concat/v106v107
Mediasize: 4294966272 (4.0G)
Sectorsize: 512
Mode: r1w1e2
Consumers:
1. Name: gvinum/sd/v106.p0.s0
Mediasize: 2147483648 (2.0G)
Sectorsize: 512
Mode: r1w1e3
Start: 0
End: 2147483136
2. Name: gvinum/sd/v107.p0.s0
Mediasize: 2147483648 (2.0G)
Sectorsize: 512
Mode: r1w1e3
Start: 2147483136
End: 4294966272</pre>

stop volume and clear members 
<tt>gconcat stop v106v107 
gconcat clear gvinum/sd/v106.p0.s0 gvinum/sd/v107.p0.s0</tt>

create new device- and its ok to reuse old/former members 
try to grab a contiguous block of gvinum volumes. gconcat volumes MAY NOT span drives (i.e. you cannot use a gvinum volume from data3 and a volume from data2 in the same gconcat volume). 
<tt>gconcat label v82-v84v106v107 /dev/gvinum/v82 /dev/gvinum/v83 /dev/gvinum/v84 /dev/gvinum/v106 /dev/gvinum/v107 
bsdlabel -r -w /dev/concat/v82-v84v106v107 
newfs /dev/concat/v82-v84v106v107a</tt>

Optional- if new volume is on a different drive, move the mount point directory (get the drive from js output) AND use that directory in the mount and cd commands below: 
<tt>mv /mnt/data1/69.55.234.66-col01334-DIR /mnt/data3/69.55.234.66-col01334-DIR</tt>

confirm you are mounted to the device (/dev/concat/v82-v84v106v107) and space is correct: 
<tt>mount /dev/concat/v82-v84v106v107a /mnt/data3/69.55.234.66-col01334-DIR</tt>

<tt>cd /mnt/data3/69.55.234.66-col01334-DIR 
df .</tt>

do the restore from the dumpfile on the backup server: 
<tt>restore -r -f /backup4/col00241.20120329.noarchive.dump . 
rm restoresymtable</tt>

when dump/restore completes successfully, use df to confirm the restored data size matches the original usage figure

edit quad (<tt>vq1</tt>) to point to new (/mnt/data3) location AND new volume (<tt>/dev/concat/v82-v84v106v107a</tt>), run buildsafe

restart the jail: 
<tt>startjail <hostname></tt>

TODO: clean up/clear old gvin/gconcat vol

6. update disk (and dir if applicable) in mgmt screen

7. update backup list AND move backups, if applicatble

Ex: <tt>mvbackups 69.55.237.26-col00241 jail9 data3</tt>

DEPRECATED - steps to tack on a new gvin to existing gconcat- leads to corrupted fs
bsdlabel -e /dev/concat/v82-v84

To figure out new size of the c partition, multiply 4194304 by the # of 2G gvinum volumes and subtract the # of 2G volumes:
10G: 4194304 * 5 – 5 = 20971515
8G: 4194304 * 4 – 4 = 16777212
6G: 4194304 * 3 – 3 = 12582909
4G: 4194304 * 2 – 2 = 8388606

To figure out the new size of the a partition, subtract 16 from the c partition:
10G: 20971515 – 16 = 20971499
8G: 16777212 – 16 = 16777196
6G: 12582909 – 16 = 12582893
4G: 8388606 – 16 = 8388590

Orig:
8 partitions:
# size offset fstype [fsize bsize bps/cpg]
a: 8388590 16 4.2BSD 2048 16384 28552
c: 8388606 0 unused 0 0 # "raw" part, don't edit

New:
8 partitions:
# size offset fstype [fsize bsize bps/cpg]
a: 12582893 16 4.2BSD 2048 16384 28552
c: 12582909 0 unused 0 0 # "raw" part, don't edit

sync; sync

growfs /dev/concat/v82-v84a

fsck –fy /dev/concat/v82-v84a

sync

fsck –fy /dev/concat/v82-v84a

(keep running fsck’s till NO errors)

== Adding disk to a 4.x jail ==

NOTE: if you unmount before vnconfig, re-vnconfig then unmount then vnunconfig

If someone wants more disk space, there’s a paste for it, send it to them.

1. Figure out the space avail from [[#js|js]]. Ideally, you want to dump to a different partition and create the new vn on the current partition. If no space to make make dump, then move old vn’s to backup or worse, dump to backup.

2. <tt>g <customerID></tt> to get the info

3. make vnfile, newfs and mount. USE CAUTION not to overwrite existing vn. 
Ex: <tt>vnconfig –T -S 9g -s labels -c /dev/vn22 /mnt/data2/65.214.160.117-col00496 
disklabel -r -w vn30 auto 
newfs /dev/vn30c 
mkdir /mnt/data2/65.214.160.117-col00496-DIR 
mount /dev/vn30c /mnt/data2/69.55.237.26-col00241-DIR 
cd /mnt/data2/65.214.160.117-col00496-DIR</tt>

4. <tt>jailkill <hostname></tt>

5. <tt>dump -0a -f - /dev/vn1 | restore -r -f -</tt>

6. <tt>rm restoresymtable</tt>

7. unmount and unvnconfig old system: 
<tt>umount /dev/vnNNc 
vnconfig –u /dev/vnNN</tt>

8. move vnfile. Ex: <tt>mv /mnt/data1/69.55.237.26-col00241 /mnt/data1/old-col00241-vnfile-20110312-noarchive</tt>

9. remove old dir: <tt>rmdir /mnt/data1/69.55.237.26-col00241-DIR</tt>

10. edit quad/safe to point to new location, run <tt>[[#buildsafe|buildsafe]]</tt>

11. start jail: <tt>[[#startjail|startjail]] <hostname></tt>

12. update disk (and dir if applicable) in mgmt screen

13. mv backups if necessary

WARNING: if you restore to unmounted vn then you’re actually restoring to /mnt/data1
you can tar and do above to recover

NOTE: you can’t move a customer to a system that has a base sys that is diff from the one they came from (BSD 4.8 to 4.5 = won’t work)

NOTE: we don’t mount procfs's to anyone’s system by default.

NOTE: mount_nulls:
also goofy mount_nulls seen in df -k on jail2 
also needs to be done in safe and quad 
if he wants another, use safe/quad as example and add it to file. also exec command at prompt to add mount (no reboot, remount or rejail necessary)

== Moving customer to another jail machine ==

Systems should only be moved between similarly-versioned jails. If you cannot get a match, you should go to freebsd.org and see what the differences were between the version you're moving from and the new version. Many of the updates will be to drivers and kernels - these don't affect the customer. Perhaps they updated traceroute or a library...you should copy the new/changed files into the VPS/jail on the new host. An alternate method for moving to a different version would be to buildworld to bring their full distribution up to date. Optional steps for this kind of upgrade are included below.

1. <tt>g <customerID></tt>

2. <tt>[[#jailkill|jailkill]] <hostname></tt>

3. create new device on target system 
4.x: 
<pre>vnconfig –T –S 4g –s labels –c /dev/vn1 /mnt/data1/69.55.22x.x-col00XXX
disklabel –r –w vn1 auto
newfs /dev/vn1c</pre>

6.x: 
<pre>bsdlabel -r -w /dev/gvinum/v1
newfs /dev/gvinum/v1a
or
gconcat label v1-v3 /dev/gvinum/v1 /dev/gvinum/v2 /dev/gvinum/v3
bsdlabel -r -w /dev/concat/v1-v3
newfs /dev/concat/v1-v3a</pre>

7.x+: 
Run jailmakeempty then skip step 4 & 7 below

4. make and mount the dir on new system. 
Ex: <pre>mkdir /mnt/data2/69.55.230.3-col00123-DIR
chmod 755 /mnt/data2/69.55.230.3-col00123-DIR
mount [device] /mnt/data2/69.55.230.3-col00123-DIR</pre>

5. [[#stopjail|stopjail]] <hostname> 1

6. dump fs to new system. ex: <tt>[[#dumpremoterestore|dumpremoterestore]] /dev/vn51 10.1.4.118 /mnt/data2/69.55.239.45-col00688-DIR</tt> (make sure you can ssh as root on remote machine)

6a. OPTIONAL BUILDWORLD: 
cd /usr/src
make world DESTDIR=/mnt/data2/69.55.xxx.xx-col0xxxx-DIR

cd etc
make distribution DESTDIR=/mnt/data2/69.55.xxx.xx-col0xxxx-DIR

(you may have to rm an openssh file and re-make dist)

rm -rf /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc/periodic/daily/400.status-disks
vi /etc/periodic/security/100.chksetuid
replace: <tt>MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`</tt> 
with: <tt>MP='/' (use single quotes)</tt>

Ask user if they want ports overwritten with current, if yes:
cp -r /usr/ports /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/usr

on source:
cd /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc; vipw –d .
(copy in all info)

on target:
cd /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc; vipw –d .
(paste all info)

on source:
cat /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc/group
(copy in all info)

on target:
cat > /mnt/data2/69.55.xxx.xx-col0xxxx-DIR/etc/group
(paste all info)

7. edit quad on source system copy over entries for the jail to the target system
take care that the vn/gvinum/gconcat/md devices on the target system are’t in use and that the /mnt/dataN path doesn’t need changing

8. run [[#buildsafe|buildsafe]] (if this is <=6.x) on target system (copies changes made to quad into safe file)

9. remove ip from source system. Ex: <tt>ipdel 69.55.230.3</tt>

10. add ip to target system. Ex: <tt>ipadd 69.55.230.3</tt>

11. start new system, with [[#startjail|startjail]] or manually, pasting entries found from running <tt>g <customerID></tt> on the new system

12. run <tt>[[#canceljail|canceljail]] col0xxxx</tt> on source system. This should prompt you to remove backups if any existed. Do not let it do this, rather:

13. if backups existed move them to the new host via the <tt>[[#mvbackups_.28freebsd.29|mvbackups]] script</tt>

14. edit quad on source system, edit comment to reflect a move rather than a cancel, ex: <tt># moved to jail2 col00241</tt>

15. edit mgmt to reflect new host and dir for new system

16. optional add ipfw rules on new system. Ex:
<tt>ipfw add 01231 count ip from 69.55.230.2 to any
ipfw add 01232 count ip from any to 69.55.230.2</tt>

== Increasing inodes for a VPS ==

when doing the newfs:
newfs -i 4096 ...

== Jail services unable to contact services within same jail ==

This is due to a messed up routing table (only seen it on jail2 before).

route delete 69.55.228.65/32 route add -net 69.55.228.65/32 -iface fxp0 -nostatic -cloning route add 69.55.228.65 -iface lo0

== /dev/null permission resets ==

Applies to 4.x jails only

For some reason that I do not understand at all, the /dev/null node in customer jails, after the system is restarted, often reverts permissions to 0600 ... which is bad, because non-root processes cannot redirect to /dev/null anymore. In fact, a fair number of server daemons will complain and/or fail if this is the case.

So, after a restart, after all systems have restarted, run postboot, which does something like:

for f in `df -k | grep /dev/vn | awk '{print $6}'` ; do chmod 0666 $f/dev/null ; done

Later, after the system is up, if anyone ever complains about their /dev/null permissions for any reason ... really if anyone mentions
/dev/null in any way, tell them to run:

chmod 0666 /dev/null

We have ultimately fixed this by adding chmods to the quad/safe's

== mknod in a jail ==

cd /mnt/data1/<user directory>/var/named
mknod -c 1 1 null
mknod -c 2 3 random

== Postfix Problems ==

This hasn't happened in a long time, probably no longer valid with modern OS's

Postfix is an alternate MTA - people replace sendmail with it in much the same way that many people also replace sendmail with qmail. The problem is that if you install postfix inside of a jail, by default it will not work properly - the user will see error messages like this:

incoming mail generates errors like this:

mail_queue_enter: create file incoming/298266.55902: File too large

and further, sending mail generates errors like this:

postdrop: warning: mail_queue_enter: create file maildrop/627930.56676: File too large

This is very easy to solve. In fact, the FreeBSD welcome email that is sent out by `jailmake` now contains this block of text:

- if you plan on installing postfix, email us first and ask for a necessary patch.

So, anytime you see a support email that says anything about postfix not working, paste that line from the welcome email into your response to them, and right away, forward the postfix patch to them in an email.

Here are the instructions:

Simply copy the patch into /usr/ports/mail/postfix/files, rename the file to ‘patch-file_limit.c’ and then cd to /usr/ports/mail/postfix and run `make install`.

The postfix patch is the very first email in the support email box, and we never delete or save it because we always want it there to forward to people. The instructions in the email are very clear, and it solves their problem 100% of the time.

== Problems with `find` ==

There was a bug in the FreeBSD vn-filesystem code. The bug causes the system to crash or hang when someone runs the `find` command inside their vn-backed filesystem (and even though the bug may have been fixed, due to the i/o overhead, we still don't like find).

Now, this does not always happen - however, any jail machine with more than 20 systems on it will crash every night if every system on it runs the daily periodic script out of their crontab, and updates their locate database with the find command.

(the locate database is a small, FreeBSD specific database that is populated nightly from a cron job, and is then used to provide fast answers to the `locate` command)

In normal systems (32-40 jails on them), all that has to be done is make sure nobody runs the daily periodic. This means that the file /etc/crontab, instead of looking like this:

<pre># do daily/weekly/monthly maintenance
1 3 * * * root periodic daily
15 4 * * 6 root periodic weekly
30 5 1 * * root periodic monthly</pre>

Needs to look like this:

<pre># DO NOT UNCOMMENT THESE - contact support@johncompanies.com for details
#1 3 * * * root periodic daily
#15 4 * * 6 root periodic weekly
#30 5 1 * * root periodic monthly
## DO NOT UNCOMMENT THESE ^^^^^^^^</pre>

and the problem will generally not occur at all. New filesystem images are always altered in this way, so all systems are set like this.

However, if you have enough systems running, even casual uses of find, outside of the periodic scripts, can cause the system to crash. We are seeing this on jail13, which crashes about once per week.

There is not yet any resolution to this problem.

So, if you have a freebsd system crash, see if anyone has uncommented their daily periodic line - you can see them for everyone by running:

/bin/sh

and then running:

<pre>for f in `df -k | grep vn | awk '{print $6}'` ; do echo `cat $f/etc/crontab | grep "periodic daily"` $f ; done | more</pre>

(all one line, one command)

And then edit those /etc/crontabs and comment them out again.

== Problems un-mounting - and with mount_null’s ==

If you cannot unmount a filesystem, beacuse it says the filesystem is busy, it is because of three things:

a) the jail is still running

b) you are actually in that directory, even though the jail is stopped

c) there are still dev, null_mount or linprocfs mount points mounted inside that directory.

d) when trying to umount null_mounts that are really long and you get an error like “No such file or directory”, it’s an OS bug where the dir is truncated. No known fix

e) there are still files open somewhere inside the dir. Use <tt>fstat | grep <cid></tt> to find the process that has files open

f) Starting with 6.x, the jail mechanism does a poor job of keeping track of processes running in a jail and if it thinks there are still procs running, it will refuse to umount the disk. If this is happening you should see a low number in the #REF column when you run jls. In this case you ''can'' safely <tt>umount –f</tt> the mount.

Please note -if you forcibly unmount a (4.x) filesystem that has null_mounts
still mounted in it, the system '''will crash''' within 10-15 mins.

== Misc jail Items ==

We are overselling hard drive space on jail2, jail8, jail9, a couple jails on jail17, jail4, jail12 and jail18.
Even though the vn file shows 4G size, it doesn’t actually occupy that amount of space on the disk. So be careful not to fill up drives where we’re overselling – use oversellcheck to confirm you’re not oversold by more than 10G.
There are other truncated jails, they are generally noted in a the file on the root system: /root/truncated

The act of moving a truncated vn to another system un-does the truncating- the truncated vn is filled with 0’s and it occupies physical disk space for which it’s configured. So, you should use dumpremote to preserve the truncation.

* if you are getting disk full messages for a BSD customer, it's fairly safe to clear out their /usr/ports/distfiles dir
* 4.x: ps and top an only be run by root in these jails. Done on purpose:
As for `ps` and `top` - non-root users can run them, just not successfully because we have locked the permissions on /dev/mem and /dev/kmem to be root-readable only. That is why non-root users cannot successfully run `ps` and `top`.
* user quotas do not work on freebsd jails - you cannot set up quotas at all, and that's that.
* You cannot inject a process into a 4.x jail, only HUP running processes. 6.x and onward you can with jexec
* jails see base machine’s uptime/load when running top/w
* if someone is unable to get in- cant ping, etc- see if they were blocked by castle (DoS), see if their ip is on the system (post reboot it was lost cause wasn’t in the rc.conf). preboot should catch that.
* in FreeBSD you cant su to root unless you belong to wheel group – so if you remove your acct and setup a new one, we have to add it to wheel group (add to /etc/group)
* Dmesg from underlying sys is seen in customers dmesg on jail
* Popper process going crazy @ 40% for 10min = someone who leaves mail on server
* Don’t force umounts on 4.x jails – it crashes the machine, generally ok on newer machines (running md)
* Good book for admin http://search.barnesandnoble.com/booksearch/isbnInquiry.asp?userid=t824VyRAYz&isbn=0596005164&itm=2
* Self-sign ssl cert http://httpd.apache.org/docs/2.0/ssl/ssl_faq.html#selfcert
* conversation with Glenn about semaphores and pgsql
<pre>SDBoody: hey, these are valid amounts/figures/increments right:
kern.ipc.semmni=1280
kern.ipc.semmns=1280

gr8feen: probably... I always forget exactly what those are, so I usually have to look them up
SDBoody: semaphores
SDBoody: i took the current 1024 and added 256 to them
SDBoody: need more for pgsql
gr8feen: I meant the mni and mns parts...
gr8feen: some of those are not ones you just want to add to... hang on a sec and I'll look them up..
gr8feen: what's semmsl set to?
SDBoody: kern.ipc.semmsl: 1024

SDBoody: kern.ipc.msgseg: 2048
kern.ipc.msgssz: 8
kern.ipc.msgtql: 40
kern.ipc.msgmnb: 2048
kern.ipc.msgmni: 40
kern.ipc.msgmax: 16384
kern.ipc.semaem: 16384
kern.ipc.semvmx: 65534
kern.ipc.semusz: 152
kern.ipc.semume: 10
kern.ipc.semopm: 100
kern.ipc.semmsl: 1024
kern.ipc.semmnu: 512
kern.ipc.semmns: 1024
kern.ipc.semmni: 1024
kern.ipc.semmap: 768
kern.ipc.shm_allow_removed: 0
kern.ipc.shm_use_phys: 1
kern.ipc.shmall: 262144
kern.ipc.shmseg: 256
kern.ipc.shmmni: 784
kern.ipc.shmmin: 1
kern.ipc.shmmax: 536870912
kern.ipc.maxsockets: 25600

gr8feen: ok...msl is max per id, mni is max ids, mns is max number of semaphores... so you probably want something like mns = mni * msl
gr8feen: which one did you run out of?
SDBoody: not sure how to tell- ipcs shows the sems in use add up to 1024
SDBoody: there are 59 entries
gr8feen: I'm assuming you tried to start postgres and it failed?
SDBoody: yes
gr8feen: it should have logged why, somewhere..
gr8feen: if I recall, it'll tell you which one it ran out of
SDBoody: > DETAIL: Failed system call was semget(1, 17, 03600).

gr8feen: so it wanted an id with 17 semaphores...I'd start by making mns = 17*mni and leave mni and mnl set to what they are now and see what it does
SDBoody: i think mni is plenty high
SDBoody: ok, more reasonable 17408
gr8feen: yeah...just change that one and see how it goes..
SDBoody: and leave mni alone at 1024?
gr8feen: yeah...mni id the max number of ids...but if it's trying to get something like 17 per id, your going to hit mns before you hit anything else
SDBoody: right, but doesn't hurt to have it that high (assuming)

gr8feen: not really... I think those get allocated out of ram that you cant page out, but it's still such a small amount that it really doesn't matter
SDBoody: looks like that worked, thx!
gr8feen: cool

see semaphores: ipcs -a -s</pre>

= FreeBSD VPS Management Tools =

These files are located in /usr/local/jail/rc.d and /usr/local/jail/bin

== jailmake ==

Applies to 7.x+
On older systems syntax differs, run jailmake once to see.

Note: this procedure differs on mx2 which is 7.x but still uses gvinum

# run js to figure out which md’s are in use, which disk has enough space, IP to put it on
# use col00xxx for both hostnames if they don’t give you a hostname
# copy over dir, ip and password to pending customer screen

<tt>Usage: jailmake IP[,IP] CID disk[1|2|3] md# hostname shorthost ipfw# email [size in GB]</tt>

Ex:

Jail2# jailmake 69.55.234.66 col01334 3 97 vps.bsd.it vps 1334 fb@bsd.it

== jailps ==
jailps [hostname]
DEPRECATED FOR jps: displays processes belonging to/running inside a jail. The command
takes one (optional) argument – the hostname of the jail you wish to query. If you don’t
supply an argument, all processes on the machine are listed and grouped by jail.

== jps ==
jps [hostname]
displays processes belonging to/running inside a jail. The command
takes one (optional) argument – the hostname or ID of the jail you wish to query.

== jailkill ==
jailkill <hostname>
stops all process running in a jail.

You can also run:
jailkill <JID>

=== problems ===
Occasionally you will hit an issue where jail will not kill off:

<pre>jail9# jailkill www.domain.com
www.domain.com .. killed: none
jail9#</pre>
Because no processes are running under that hostname. You cannot use jailps.pl either:

<pre>jail9# jailps www.domain.com
www.domain.com doesn’t exist on this server
jail9#</pre>

The reasons for this are usually:
* the jail is no longer running

* the jail's hostname has changed
In this case,

>=6.x: run a <tt>jls|grep <jail's IP></tt> to find the correct hostname, then update the quad file, then kill the jail.

<6.x: the first step is to cat their /etc/rc.conf file to see if you can tell what they set the new hostname to. This very often works. For example:

cat /mnt/data2/198.78.65.136-col00261-DIR/etc/rc.conf

But maybe they set the hostname with the hostname command, and the original hostname is still in /etc/rc.conf.

The welcome email clearly states that they should tell us if they change their hostname, so there is no problem in just emailing them and asking them what they set the new hostname to.

Once you know the new hostname OR if a customer simply emails to inform you that they have set the hostname to something different, you need to edit the quad and safe files that their system is in to input the new hostname.

However, if push comes to shove and you cannot find out the hostname from them or from their system, then you need to start doing some detective work.

The easiest thing to do is run jailps looking for a hostname similar to their original hostname. Or you could get into the /bin/sh shell by running:

/bin/sh

and then looking at every hostname of every process:

for f in `ls /proc` ; do cat /proc/$f/status ; done

and scanning for a hostname that is either similar to their original hostname, or that you don't see in any of the quad safe files.

This is very brute force though, and it is possible that catting every file in /proc is dangerous - I don't recommend it. A better thing would be to identify any processes that you know belong to this system – perhaps the reason you are trying to find this system is because they are running something bad - and just catting the status from only that PID.

Somewhere there’s a jail where there may be 2 systems named www. Look at /etc/rc.conf and make sure they’re both really www. If they are, jailkill www, jailps www to make sure not running. Then immediately restart the other one, as the fqdn (as found from a rev nslookup)

* on >=6.x the hostname may not yet be hashed:
<pre>jail9 /# jls
JID Hostname Path IP Address(es)
1 bitnet.dgate.org /mnt/data1/69.55.232.50-col02094-DIR 69.55.232.50
2 ns3.hctc.net /mnt/data1/69.55.234.52-col01925-DIR 69.55.234.52
3 bsd1 /mnt/data1/69.55.232.44-col00155-DIR 69.55.232.44
4 let2.bbag.org /mnt/data1/69.55.230.92-col00202-DIR 69.55.230.92
5 post.org /mnt/data2/69.55.232.51-col02095-DIR 69.55.232.51 ...
6 ns2 /mnt/data1/69.55.232.47-col01506-DIR 69.55.232.47 ...
7 arlen.server.net /mnt/data1/69.55.232.52-col01171-DIR 69.55.232.52
8 deskfood.com /mnt/data1/69.55.232.71-col00419-DIR 69.55.232.71
9 mirage.confluentforms.com /mnt/data1/69.55.232.54-col02105-DIR 69.55.232.54 ...
10 beachmember.com /mnt/data1/69.55.232.59-col02107-DIR 69.55.232.59
11 www.agottem.com /mnt/data1/69.55.232.60-col02109-DIR 69.55.232.60
12 sdhobbit.myglance.org /mnt/data1/69.55.236.82-col01708-DIR 69.55.236.82
13 ns1.jnielsen.net /mnt/data1/69.55.234.48-col00204-DIR 69.55.234.48 ...
14 ymt.rollingegg.net /mnt/data2/69.55.236.71-col01678-DIR 69.55.236.71
15 verse.unixlore.net /mnt/data1/69.55.232.58-col02131-DIR 69.55.232.58
16 smcc-mail.org /mnt/data2/69.55.232.68-col02144-DIR 69.55.232.68
17 kasoutsuki.w4jdh.net /mnt/data2/69.55.232.46-col02147-DIR 69.55.232.46
18 dili.thium.net /mnt/data2/69.55.232.80-col01901-DIR 69.55.232.80
20 www.tekmarsis.com /mnt/data2/69.55.232.66-col02155-DIR 69.55.232.66
21 vps.yoxel.net /mnt/data2/69.55.236.67-col01673-DIR 69.55.236.67
22 smitty.twitalertz.com /mnt/data2/69.55.232.84-col02153-DIR 69.55.232.84
23 deliver4.klatha.com /mnt/data2/69.55.232.67-col02160-DIR 69.55.232.67
24 nideffer.com /mnt/data2/69.55.232.65-col00412-DIR 69.55.232.65
25 usa.hanyuan.com /mnt/data2/69.55.232.57-col02163-DIR 69.55.232.57
26 daifuku.ppbh.com /mnt/data2/69.55.236.91-col01720-DIR 69.55.236.91
27 collins.greencape.net /mnt/data2/69.55.232.83-col01294-DIR 69.55.232.83
28 ragebox.com /mnt/data2/69.55.230.104-col01278-DIR 69.55.230.104
29 outside.mt.net /mnt/data2/69.55.232.72-col02166-DIR 69.55.232.72
30 vps.payneful.ca /mnt/data2/69.55.234.98-col01999-DIR 69.55.234.98
31 higgins /mnt/data2/69.55.232.87-col02165-DIR 69.55.232.87 ...
32 ozymandius /mnt/data2/69.55.228.96-col01233-DIR 69.55.228.96
33 trusted.realtors.org /mnt/data2/69.55.238.72-col02170-DIR 69.55.238.72
34 jc1.flanderous.com /mnt/data2/69.55.239.22-col01504-DIR 69.55.239.22
36 guppylog.com /mnt/data2/69.55.238.73-col00036-DIR 69.55.238.73
40 haliohost.com /mnt/data2/69.55.234.41-col01916-DIR 69.55.234.41 ...
41 satyr.jorge.cc /mnt/data1/69.55.232.70-col01963-DIR 69.55.232.70
jail9 /# jailkill satyr.jorge.cc
ERROR: jail_: jail "satyr,jorge,cc" not found
</pre>

Note how it's saying <tt>satyr,jorge,cc</tt> is not found, and not <tt>satyr.jorge.cc</tt>.

The jail subsystem tracks things using comma-delimited hostnames. That is created every few hours:

jail9 /# crontab -l
0 0,6,12,18 * * * /usr/local/jail/bin/sync_jail_names

So if we run this manually:
jail9 /# /usr/local/jail/bin/sync_jail_names

Then kill the jail:
jail9 /# jailkill satyr.jorge.cc
successfully killed: satyr,jorge,cc</pre>

It worked.

If you ever see this when trying to kill a jail:

<pre># jailkill e-scribe.com
killing JID: 6 hostname: e-scribe.com
3 procs running
3 procs running
3 procs running
3 procs running
...</pre>

<tt>[[#jailkill|jailkill]]</tt> probably got lost trying to kill off the jail. Just ctrl-c the jailkill process, then run a jailps on the hostname, and <tt>kill -9</tt> any process which is still running. Keep running jailps and <tt>kill -9</tt> till all processes are gone.

== jailpsall ==
jailpsall
will run a jailps on all jails configured in the quad files (this is different from
jailps with no arguments as it won’t help you find a “hidden” system)

== jailpsw ==
jailpsw
will run a jailps with an extra -w to provide wider output

== jt (>=7.x) ==
jt
displays the top 20 processes on the server (the top 20 processes from top) and
which jail owns them. This is very helpful for determining who is doing what when
the server is very busy.

== jtop (>=7.x) ==
jtop
a wrapper for top displaying processes on the server and which jail owns them. Constantly updates, like top.

== jtop (<7.x) ==
jtop
displays the top 20 processes on the server (the top 20 processes from top) and
which jail owns them. This is very helpful for determining who is doing what when
the server is very busy.

== stopjail ==
stopjail <hostname> [1]
this will jailkill, umount and vnconfig –u a jail. If passed an optional 2nd
argument, it will not exit before umounting and un-vnconfig’ing in the event
jailkill returns no processes killed. This is useful if you just want to umount
and vnconfig –u a jail you’ve already killed. It is intelligent in that it won’t
try to umount or vnconfig –u if it’s not necessary.

== startjail ==
startjail <hostname>
this will start vnconfig, mount (including linprocfs and null-mounts), and start a jail.
Essentially, it reads the jail’s relevant block from the right quad file and executes it.
It is intelligent in that it won’t try to mount or vnconfig if it’s not necessary.

== jpid ==
jpid <pid>
displays information about a process – including which jail owns it.
It’s the equivalent of running cat /proc/<pid>/status

== canceljail ==
canceljail <hostname> [1]
this will stop a jail (the equivalent of stopjail), check for backups (offer to remove them
from the backup server and the backup.config), rename the vnfile, remove the dir, and
edit quad/safe. If passed an optional 2nd argument, it will not exit upon failing to kill
and processes owned by the jail. This is useful if you just want to cancel a jail which
is already stopped.

== jls ==
jls [-v]
Lists all jails running:
<pre>JID #REF IP Address Hostname Path
101 135 69.55.224.148 mail.pc9.org /mnt/data2/69.55.224.148-col01034-DIR</pre>

#REF is the number of references or procs(?) running

Running with -v will give you all IPs assigned to each jail (7.2 up)
<pre>JID #REF Hostname Path IP Address(es)
101 139 mail.pc9.org /mnt/data2/69.55.224.148-col01034-DIR 69.55.224.14869.55.234.85</pre>

== startalljails ==
startalljails
7.2+ only. This will parse through quad1 and start all jails. It utilizes lockfiles so it won’t try to start a jail more than once- therefore multiple instances can be running in parallel without fear of starting a jail twice. If a jail startup gets stuck, you can ^C without fear of killing the script. IMPORTANT- before running startalljails you should make sure you ran preboot once as it will clear out all the lockfiles and enable startalljails to work properly.

== aaccheck.sh ==
aaccheck.sh
displayes the output of container list and task list from aaccli

== backup ==
backup
backup script called nightly to update jail scripts and do customer backups

== buildsafe ==
buildsafe
creates safe files based on quads (automatically removing the fsck’s). This will destructively overwrite safe files

== checkload.pl ==
checkload.pl
this was intended to be setup as a cronjob to watch processes on a jail when the load
rises above a certain level. Not currently in use.

== checkprio.pl ==
checkprio.pl
will look for any process (other than the current shell’s csh, sh, sshd procs) with a non-normal priority and normalize it

== diskusagemon ==
diskusagemon <mount point> <1k blocks>
watches a mount point’s disk use, when it reaches the level specified in the 2nd argument,
it exits. This is useful when doing a restore and you want to be paged as it’s nearing completion.
Best used as: <tt>diskusagemon /asd/asd 1234; pagexxx</tt>

== dumprestore ==
dumprestore <dumpfile>
this is a perl expect script which automatically enters ‘1’ and ‘y’. It seems to cause restore to fail
to set owner permissions on large restores.

== g ==
g <search>
greps the quad/safe files for the given search parameter

== gather.pl ==
gather.pl
gathers up data about jails configured and writes to a file. Used for audits against the db

== gb ==
gb <search>
greps backup.config for the given search parameter

== gbg ==
gbg <search>
greps backup.config for the given search parameter and presents just the directories (for clean pasting)

== ipfwbackup ==
ipfwbackup
writes ipfw traffic count data to a logfile

== ipfwreset ==
ipfwreset
writes ipfw traffic count data to a logfile and resets counters to 0

== js ==
js
output varies by OS version, but generally provides information about the base jail:
- which vn’s are in use
- disk usage
- info about the contents of quads
- the # of inodes represented by the jails contained in the group (133.2 in the example below), and how many jails per data mount, as well as subtotals
- ips bound to the base machine but not in use by a jail
- free gvinum volumes, or unused vn’s or used md’s

<pre>/usr/local/jail/rc.d/quad1:
/mnt/data1 133.2 (1)
/mnt/data2 1040.5 (7)
total 1173.7 (8)
/usr/local/jail/rc.d/quad2:
/mnt/data1 983.4 (6)
total 983.4 (6)
/usr/local/jail/rc.d/quad3:
/mnt/data1 693.4 (4)
/mnt/data2 371.6 (3)
total 1065 (7)
/usr/local/jail/rc.d/quad4:
/mnt/data1 466.6 (3)
/mnt/data2 882.2 (5)
total 1348.8 (8)
/mnt/data1: 2276.6 (14)
/mnt/data2: 2294.3 (15)

Available IPs:
69.55.230.11 69.55.230.13 69.55.228.200

Available volumes:
v78 /mnt/data2 2G
v79 /mnt/data2 2G
v80 /mnt/data2 2G</pre>

== load.pl ==
load.pl
feeds info to load mrtg - executed by inetd.

== makevirginjail ==
makevirginjail
Only on some systems, makes an empty jail (doesn't do restore step)

== mb ==
mb <mount|umount>
(nfs) mounts and umounts dirs to backup2. Shortcuts are mbm and mbu to mount and unmount.

== notify.sh ==
notify.sh
emails reboot@johncompanies.com – intended to be called at boot time to alert us to a machine which panics and reboots and isn’t caught by bb or castle.

== orphanedbackupwatch ==
orphanedbackupwatch
looks for directories on backup2 which aren’t configured in backup.config and offers to delete them

== postboot ==
postboot
to be run after a machine reboot and quad/safe’s are done executing. It will:
* do chmod 666 on each jail’s /dev/null
* add ipfw counts
* run jailpsall (so you can see if a configured jail isn’t running)

== preboot ==
preboot
to be run before running quad/safe – checks for misconfigurations:
* a jail configured in a quad but not a safe
* a jail is listed more than once in a quad
* the ip assigned to a jail isn’t configured on the machine
* alias numbering skips in the rc.conf (resulting in the above)
* orphaned vnfile's that aren't mentioned in a quad/safe
* ip mismatches between dir/vnfile name and the jail’s ip
* dir/vnfiles's in quad/safe that don’t exist

== quadanalyze.pl ==
quadanalyze.pl
called by js, produces the info (seen above with js explanation) about the contents of quad (inode count, # of jails, etc.)

== rsync.backup ==
rsync.backup
does customer backups (relies on backup.config)

== taskdone ==
taskdone
when called will email support@johncompanies.com with the hostname of the machine from which it was executed as the subject

== topten ==
topten
summarizes the top 10 traffic users (called by ipfwreset)

== trafficgather.pl ==
trafficgather.pl [yy-mm]
sends a traffic usage summary by jail to support@johncomapnies.com and payments@johncompanies.com. Optional arguments are year and month (must be in the past). If not passed, assumes last month. Relies on traffic logs created by ipfwreset and ipfwbackup

== trafficwatch.pl ==
trafficwatch.pl
checks traffic usage and emails support@johncomapnies.com when a jail reaches the warning level (35G) and the limit (40G). We really aren’t using this anymore now that we have netflow.

== trafstats ==
trafstats
writes ipfw traffic usage info by jail to a file called jc_traffic_dump in each jail’s / dir

== truncate_jailmake ==
truncate_jailmake
a version of jailmake which creates truncated vnfiles.

== vb ==
vb
the equivalent of: <tt>vi /usr/local/jail/bin/backup.config</tt>

== vs (freebsd) ==
vs<n>
the equivalent of: <tt>vi /usr/local/jail/rc.d/safe<n></tt>

vq<n>
the equivalent of: vi /usr/local/jail/rc.d/quad<n>

== dumpremote ==
dumpremote <user@machine> </remote/location/file-dump> <vnX>
ex: dumpremote user@10.1.4.117 /mnt/data3/remote.echoditto.com-dump 7
this will dump a vn filesystem to a remote machine and location

== oversellcheck ==
oversellcheck
displays how much a disk is oversold or undersold taking into account truncated vn files. Only for use on 4.x systems

== mvbackups (freebsd) ==
mvbackups <dir> (1.1.1.1-col00001-DIR) <target_machine> (jail1) <target_dir> (data1)
moves backups from one location to another on the backup server, and provides you with option to remove entries from current backup.config, and simple paste command to add the config to backup.config on the target server

== jailnice ==
jailnice <hostname>
applies <tt>renice 19 [PID]</tt> and <tt>rtprio 31 –[PID]</tt> to each process in the given jail

== dumpremoterestore ==
dumpremoterestore <device> <ip of target machine> <dir on target machine>
ex: dumpremoterestore /dev/vn51 10.1.4.118 /mnt/data2/69.55.239.45-col00688-DIR
dumps a device and restores it to a directory on a remote machine. Requires that you enable root ssh on the
remote machine.

== psj ==
psj
shows just the procs running on the base system – a ps auxw but without jail’d procs present

== perc5iraidchk ==
perc5iraidchk
checks for degraded arrays on Dell 2950 systems with Perc5/6 controllers

== perc4eraidchk ==
perc4eraidchk
checks for degraded arrays on Dell 2850 systems with Perc4e/Di controllers

= Virtuozzo VPS =

== Making new customer VE (vm) ==

This applies only to new virts >= 4.x

grab ip from ipmap (if opened from the pending cust screen it should take you to the right block). You can also run vzlist -a to see what block is in use, generally. Try to find an IP that's in the same block of class C IP's already on the box.

1. confirm ip you want to use isn’t in use via Mgmt. -> IP Map in management screens
also, ping the IP address.

2. put CT on whichever partition has more space
df -h

3. vm CID IP hostname /vz[1|2] email[,email] template ( <LB|LBP|LS|LM|LMP|LP> [disk] | <gb_disk/mb_ram/gb_burst> )
vm col00009 69.55.230.238 centos.testdave.com /vz1 dsmith@johncompanies.com centos-6-x86_64 LM

4. copy veid, dir, ip and password to pending customer screen. activate customer

== Making new customer VE (vemakexxx) ==

This applies to older virts with old templates. This should probably not be used at all anymore.

1. look thru hist for ip

2. confirm ip you want to use isn’t in use via Mgmt. -> IP Map in management screens

3. put ve on whichever partition has more space
vemakerh9 <veid> <ip> <hostname> <mount> <email> [gb disk]; <256|384|512> <veid>
vemakerh9 866 69.55.226.109 ngentu.com /vz1 ayo@ngantu.com,asd@asd.com 5; 256 866

4. copy (veid), dir, and ip to pending customer screen (pass set to p455agfa)

Note: We use VEID (Virtual Environment ID) and CTID (Container ID) interchangably. Similarly, VE and CT. They mean the same thing.
VZPP = VirtuoZzo Power Panel (the control panel for each CT)

All linux systems exist in /vz, /vz1 or /vz2 - since each linux machine holds roughly 60-90 customers, there will be roughly 30-45 in each partition.

The actual filesystem of the system in question is in:

/vz(1-2)/private/(VEID)

Where VEID is the identifier for that system - an all-numeric string larger than 100.

The actual mounted and running systems are in the corresponding:

/vz(1-2)/root/(VEID)

But we rarely interact with any system from this mount point.

You should never need to touch the root portion of their system – however you can traverse their filesystem by going to <tt>/vz(1-2)/private/(VEID)/root</tt> (<tt>/vz(1-2)/private/(VEID)/fs/root</tt> on 4.x systems) the root of their filesystem is in that directory, and their entire system is underneath that.

Every VE has a startup script in <tt>/etc/sysconfig/vz-scripts</tt> (which is symlinked as <tt>/vzconf</tt> on all systems) - the VE startup script is simply named <tt>(VEID).conf</tt> - it contains all the system parameters for that VE:

<pre># Configuration file generated by vzsplit for 60 VE
# on HN with total amount of physical mem 2011 Mb

VERSION="2"
CLASSID="2"

ONBOOT="yes"

KMEMSIZE="8100000:8200000"
LOCKEDPAGES="322:322"
PRIVVMPAGES="610000:615000"
SHMPAGES="33000:34500"
NUMPROC="410:415"
PHYSPAGES="0:2147483647"
VMGUARPAGES="13019:2147483647"
OOMGUARPAGES="13019:2147483647"
NUMTCPSOCK="1210:1215"
NUMFLOCK="107:117"
NUMPTY="19:19"
NUMSIGINFO="274:274"
TCPSNDBUF="1800000:1900000"
TCPRCVBUF="1800000:1900000"
OTHERSOCKBUF="900000:950000"
DGRAMRCVBUF="200000:200000"
NUMOTHERSOCK="650:660"
DCACHE="786432:818029"
NUMFILE="7500:7600"
AVNUMPROC="51:51"
IPTENTRIES="155:155"
DISKSPACE="4194304:4613734"
DISKINODES="400000:420000"
CPUUNITS="1412"
QUOTAUGIDLIMIT="2000"
VE_ROOT="/vz1/root/636"
VE_PRIVATE="/vz1/private/636"
NAMESERVER="69.55.225.225 69.55.230.3"
OSTEMPLATE="vzredhat-7.3/20030305"
VE_TYPE="regular"
IP_ADDRESS="69.55.225.229"
HOSTNAME="textengine.net"</pre>

As you can see, the hostname is set here, the disk space is set here, the number of inodes, the number of files that can be open, the number of tcp sockets, etc. - all are set here.

In fact, everything that can be set on this customer system is set in this conf file.

All interaction with the customer system is done with the VEID. You start the system by running:

vzctl start 999

You stop it by running:

vzctl stop 999

You execute commands in it by running:

vzctl exec 999 df -k

You enter into it, via a root-shell backdoor with:

vzctl enter 999

and you set parameters for the system, while it is still running, with:

vzctl set 999 --diskspace 6100000:6200000 --save

<tt>vzctl</tt> is the most commonly used command - we have aliased <tt>v</tt> to <tt>vzctl</tt> since we use it so often. We’ll continue to use <tt>vzctl</tt> in our examples, but feel free to use just <tt>v</tt>.

Let's say the user wants more diskspace. You can cat their conf file and see:

DISKSPACE="4194304:4613734"

So right now they have 4gigs of space. You can then change it to 6 with:

vzctl set 999 --diskspace 6100000:6200000 --save

IMPORTANT: all issuances of the vzctl set command need to end with <tt>–save</tt> - if they don't, the setting will be set, but it will not be saved to the conf file, and they will not have those settings next time they boot.

All of the tunables in the conf file can be set with the vzctl set command. Note that in the conf file, and on the vzctl set command line, we always issue two numbers seperated by a colon - that is because we are setting the hard and soft limits. Always set the hard limit slightly above the soft limit, as you see it is in the conf file for all those settings.

There are also things you can set with `<tt>vzctl set</tt>` that are not in the conf file as settings, per se. For instance, you can add IPs:

vzctl set 999 --ipadd 10.10.10.10 --save

or multiple IPs:

vzctl set 999 --ipadd 10.10.10.10 --ipadd 10.10.20.30 --save

or change the hostname:

vzctl set 999 --hostname www.example.com --save

You can even set the nameservers:

vzctl set 999 --nameserver 198.78.66.4 --nameserver 198.78.70.180 --save

Although you probably will never do that.

You can disable a VPS from being started (by VZPP or reboot) (4.x):

vzctl set 999 --disabled yes --save

You can disable a VPS from being started (by VZPP or reboot) (<=3.x):

vzctl set 999 --onboot=no --save

You can disable a VPS from using his control panel:

vzctl set 999 --offline_management=no --save

You can suspend a VPS, so it can be resumed in the same state it was in when it was stopped (4.x):

vzctl suspend 999

and to resume it:

vzctl resume 999

to see who owns process:
vzpid <PID>

to mount up an unmounted ve:
vzctl mount 827

To see network stats for CT's:
<pre># vznetstat
VEID Net.Class Output(bytes) Input(bytes)
-----------------------------------------------
24218 1 484M 39M
24245 1 463M 143M
2451 1 2224M 265M
2454 1 2616M 385M
4149 1 125M 68M
418 1 1560M 34M
472 1 1219M 315M
726 1 628M 317M
763 1 223M 82M
771 1 4234M 437M
-----------------------------------------------
Total: 13780M 2090M</pre>

One thing that sometimes comes up on older systems that we created with smaller defaults is that the system would run out of inodes. The user will email and say they cannot create any more files or grow any files larger, but they will also say that they are not out of diskspace ... they are running:

df -k

and seeing how much space is free - and they are not out of space. They are most likely out of inodes - which they would see by running:

df -i

So, the first thing you should do is enter their system with:

vzctl enter 999

and run: <tt>df -i</tt>

to confirm your theory. Then exit their system. Then simply cat their conf file and see what their inodes are set to (probably 200000:200000, since that was the old default on the older systems) and run:

vzctl set 999 --diskinodes 400000:400000 --save

If they are not out of inodes, then a good possibility is that they have maxed out their numfile configuration variable, which controls how many files they can have in their system. The current default is 7500 (which nobody has ever hit), but the old default was as low as 2000, so you would run something like:

vzctl set 999 --numfile 7500:7500 --save

----

You cannot start or stop a VE if your pwd is its private (/vz/private/999) or root (/vz/root/999) directories, or anywhere below them.

== Recovering from a crash (linux) ==

=== Diagnose whether you have a crash ===
The most important thing is to get the machine and all ve’s back up as soon as possible. Note the time, you’ll need to create a crash log entry (Mgmt. -> Reference -> CrashLog). The first thing to do is head over to the [[Screen#Screen_Organization|serial console screen]] and see if there’s any kernel error messages output. Try to copy any messages (or just a sample of repeating messages) you see into the notes section of the crash log – these will also likely need to be sent to virtuozzo for interpretation. If the messages are spewing too fast, hit ^O + H to start a screen log dump which you can ob1182.pts-38.bb serve after the machine is rebooted. Additionally, if the machine is responsive, you can get a trace to send to virtuozzo by hooking up a kvm and entering these 3 sequences:
<pre>alt+print screen+m
alt+print screen+p
alt+print screen+t</pre>

If there are no messages, the machine may just be really busy- wait a bit (5-10min) to see if it comes back. If it's still pinging, odds are its very busy. If it doesn't come back, or the messages indicate a fatal error, you will need to proceed with a power cycle (ctrl+alt+del will not work).

=== Power cycle the server ===
If this machine is not a Dell 2950 with a [[DRAC/RMM#DRAC|DRAC card]] (i.e. if you can’t ssh into the DRAC card and issue racadm serveraction hardreset, then you will need someone at the data center to power the macine off, wait 30 sec, then turn it back on. Make sure to re-attach via console (<tt>tip virtxx</tt>) immediately after power down.

=== (Re)attach to the console ===
Stay on the console the entire time during boot. As the BIOS posts- look out for the RAID card output- does everything look healthy? The output may be scrambled, look for "DEGRADED" or "FAILED". Once the OS starts booting you will be disconnected (dropped back to the shell on the console server) a couple times during the boot up. The reason you want to quickly re-attach is two-fold: 1. If you don’t reattach quickly then you won’t get any console output, 2. you want to be attached before the server ''potentially'' starts (an extensive) fsck. If you attach after the fsck begins, you’ll have seen no indication it started an fsck and the server will appear frozen during startup- no output, no response.

=== Start containers/VE's/VPSs ===
When the machine begins to start VE’s, it’s safe to leave the console and login via ssh. All virts should be set to auto start all the VEs after a crash. Further, most (newer) virts are set to “fastboot” it’s VE’s (to find out, do:
grep -i fast /etc/sysconfig/vz
and look for <tt>VZFASTBOOT=yes</tt>). If this was set prior to the machine’s crash (setting it after the machine boots will not have any effect until the vz service is restarted) it will start each ve as fast as possible, in serial, then go thru each VE (serially), shutting it down running a vzquota (disk usage) check, then bringing it back up. The benefit is that all VE’s are brought up quickly (within 15min or so depending on the #), the downside is a customer watching closely will notice 2 outages – 1st the machine crash, 2nd their quota check (which will be a much shorter downtime- on the order of a few minutes).

Where “fastboot” is not set to yes (i.e on quar1), vz will start them consecutively, checking the quotas one at a time, and the 60th VE may not start until an hour or two later - this is not acceptable.

The good news is, if you run vzctl start for a VE that is already started, you will simply get an error: <tt>VE is already started</tt>. Further, if you attempt to vzctl start a VE that is in the process of being started, you will simply get an error: unable to lock VE. So, there is no danger in simply running scripts to start smaller sets of VEs. If the system is not autostarting, then there is no issue, and even if it does, when it conflicts, one process (yours or the autostart) will lose, and just move on to the next one.

A script has been written to assist with ve starts: [[#startvirt.pl|startvirt.pl]] which will start 6 ve’s at once until there are no more left. If startvirt.pl is used on a system where “fastboot” was on, it will circumvent the fastboot for ve’s started by startvirt.pl – they will go through the complete quota check before starting- therefore this is not advisable when a system has crashed. When a system is booted cleanly, and there's no need for vzquota checks, then startvirt.pl is safe and advisable to run.

=== Make sure all containers are running ===
You can quickly get a feel for how many ve’s are started by running:

<pre>[root@virt4 log]# vs
VEID 16066 exist mounted running
VEID 16067 exist mounted running
VEID 4102 exist mounted running
VEID 4112 exist mounted running
VEID 4116 exist mounted running
VEID 4122 exist mounted running
VEID 4123 exist mounted running
VEID 4124 exist mounted running
VEID 4132 exist mounted running
VEID 4148 exist mounted running
VEID 4151 exist mounted running
VEID 4155 exist mounted running
VEID 42 exist mounted running
VEID 432 exist mounted running
VEID 434 exist mounted running
VEID 442 exist mounted running
VEID 450 exist mounted running
VEID 452 exist mounted running
VEID 453 exist mounted running
VEID 454 exist mounted running
VEID 462 exist mounted running
VEID 463 exist mounted running
VEID 464 exist mounted running
VEID 465 exist mounted running
VEID 477 exist mounted running
VEID 484 exist mounted running
VEID 486 exist mounted running
VEID 490 exist mounted running</pre>

So to see how many ve’s have started:
<pre>[root@virt11 root]# vs | grep running | wc -l
39</pre>

And to see how many haven’t:
<pre>[root@virt11 root]# vs | grep down | wc -l
0</pre>

And how many we should have running:
<pre>[root@virt11 root]# vs | wc -l
39</pre>

Another tool you can use to see which ve’s have started, among other things is [[#vzstat|vzstat]]. It will give you CPU, memory, and other stats on each ve and the overall system. It’s a good thing to watch as ve’s are starting (note the VENum parameter, it will tell you how many have started):

<pre>4:37pm, up 3 days, 5:31, 1 user, load average: 1.57, 1.68, 1.79
VENum 40, procs 1705: running 2, sleeping 1694, unint 0, zombie 9, stopped 0
CPU [ OK ]: VEs 57%, VE0 0%, user 8%, sys 7%, idle 85%, lat(ms) 412/2
Mem [ OK ]: total 6057MB, free 9MB/54MB (low/high), lat(ms) 0/0
Swap [ OK ]: tot 6142MB, free 4953MB, in 0.000MB/s, out 0.000MB/s
Net [ OK ]: tot: in 0.043MB/s 402pkt/s, out 0.382MB/s 4116pkt/s
Disks [ OK ]: in 0.002MB/s, out 0.000MB/s

VEID ST %VM %KM PROC CPU SOCK FCNT MLAT IP
1 OK 1.0/17 0.0/0.4 0/32/256 0.0/0.5 39/1256 0 9 69.55.227.152
21 OK 1.3/39 0.1/0.2 0/46/410 0.2/2.8 23/1860 0 6 69.55.239.60
133 OK 3.1/39 0.1/0.3 1/34/410 6.3/2.8 98/1860 0 0 69.55.227.147
263 OK 2.3/39 0.1/0.2 0/56/410 0.3/2.8 34/1860 0 1 69.55.237.74
456 OK 17/39 0.1/0.2 0/100/410 0.1/2.8 48/1860 0 11 69.55.236.65
476 OK 0.6/39 0.0/0.2 0/33/410 0.1/2.8 96/1860 0 10 69.55.227.151
524 OK 1.8/39 0.1/0.2 0/33/410 0.0/2.8 28/1860 0 0 69.55.227.153
594 OK 3.1/39 0.1/0.2 0/45/410 0.0/2.8 87/1860 0 1 69.55.239.40
670 OK 7.7/39 0.2/0.3 0/98/410 0.0/2.8 64/1860 0 216 69.55.225.136
691 OK 2.0/39 0.1/0.2 0/31/410 0.0/0.7 25/1860 0 1 69.55.234.96
744 OK 0.1/17 0.0/0.5 0/10/410 0.0/0.7 7/1860 0 6 69.55.224.253
755 OK 1.1/39 0.0/0.2 0/27/410 0.0/2.8 33/1860 0 0 192.168.1.4
835 OK 1.1/39 0.0/0.2 0/19/410 0.0/2.8 5/1860 0 0 69.55.227.134
856 OK 0.3/39 0.0/0.2 0/13/410 0.0/2.8 16/1860 0 0 69.55.227.137
936 OK 3.2/52 0.2/0.4 0/75/410 0.2/0.7 69/1910 0 8 69.55.224.181
1020 OK 3.9/39 0.1/0.2 0/60/410 0.1/0.7 55/1860 0 8 69.55.227.52
1027 OK 0.3/39 0.0/0.2 0/14/410 0.0/2.8 17/1860 0 0 69.55.227.83
1029 OK 1.9/39 0.1/0.2 0/48/410 0.2/2.8 25/1860 0 5 69.55.227.85
1032 OK 12/39 0.1/0.4 0/80/410 0.0/2.8 41/1860 0 8 69.55.227.90</pre>

When you are all done, you will want to make sure that all the VEs really did get started, run vs one more time.

Note the time all ve’s are back up and enter that into and save the crash log entry.

Occasionally, a ve will not start automatically. The most common reason for a ve not to come up normally is the ve was at it’s disk limit before the crash, and will not start since they’re over the limit. To overcome this, set the disk space to current usage level (the system will give this to you when it fails to start), start the ve, then re-set the disk space back to the prior level. Lastly, contact the customer to let them know they’re out of disk (or allocate more disk if they're entitled to more).

== Hitting performance barriers and fixing them ==

There are multiple modes virtuozzo offers to allocate resources to a ve. We utilize 2: SLM and UBC parameters
On our 4.x systems, we use all SLM – it’s simpler to manage and understand. There are a few systems on virt19/18 that may also use SLM. Everything else uses UBC.
You can tell a SLM ve by:

SLMMODE="all"

in their conf file.

TODO: detail SLM modes and parameters.

If someone is in SLM mode and they hit memory resource limits, they simply need to upgrade to more memory.

The following applies to everyone else (UBC).

Customers will often email and say that they are getting out of memory errors - a common one is "cannot fork" ... basically, anytime you see something odd like this, it means they are hitting one of their limits that is in place in their conf file.

The conf file, however, simply shows their limits - how do we know what they are currently at ?

The answer is a file called v - this file contains the current status (and peaks) of their performance settings, and also counts how many times they have hit the barrier. The output of the file looks like this:

<pre>764: kmemsize 384113 898185 8100000 8200000 0
lockedpages 0 0 322 322 0
privvmpages 1292 7108 610000 615000 0
shmpages 270 528 33000 34500 0
dummy 0 0 0 0 0
numproc 8 23 410 415 0
physpages 48 5624 0 2147483647 0
vmguarpages 0 0 13019 2147483647 0
oomguarpages 641 6389 13019 2147483647 0
numtcpsock 3 21 1210 1215 0
numflock 1 3 107 117 0
numpty 0 2 19 19 0
numsiginfo 0 4 274 274 0
tcpsndbuf 0 80928 1800000 1900000 0
tcprcvbuf 0 108976 1800000 1900000 0
othersockbuf 2224 37568 900000 950000 0
dgramrcvbuf 0 4272 200000 200000 0
numothersock 3 9 650 660 0
dcachesize 53922 100320 786432 818029 0
numfile 161 382 7500 7600 0
dummy 0 0 0 0 0
dummy 0 0 0 0 0
dummy 0 0 0 0 0
numiptent 4 4 155 155 0</pre>

The first column is the name of the counter in question - the same names we saw in the systems conf file. The second column is the _current_ value of that counter, the third column is the max that that counter has ever risen to, the fourth column is the soft limit, and the fifth column is the hard limit (which is the same as the numbers in that systems conf file).

The sixth number is the failcount - how many times the current usage has risen to hit the barrier. It will increase as soon as the current usage hits the soft limit.

The problem with /proc/user_beancounters is that it actually contains that set of data for every running VE - so you can't just cat /proc/user_beancounters - it is too long and you get info for every other running system.

You can vzctl enter the system and run:

vzctl enter 9999
cat /proc/user_beancounters

inside their system, and you will just see the stats for their particular system, but entering their system every time you want to see it is combersome.

So, I wrote a simple script called "vzs" which simply greps for the VEID, and spits out the next 20 or so lines (however many lines there are in the output, I forget) after it. For instance:

<pre># vzs 765:
765: kmemsize 2007936 2562780 8100000 8200000 0
lockedpages 0 8 322 322 0
privvmpages 26925 71126 610000 615000 0
shmpages 16654 16750 33000 34500 0
dummy 0 0 0 0 0
numproc 41 57 410 415 0
physpages 1794 49160 0 2147483647 0
vmguarpages 0 0 13019 2147483647 0
oomguarpages 4780 51270 13019 2147483647 0
numtcpsock 23 37 1210 1215 0
numflock 17 39 107 117 0
numpty 1 3 19 19 0
numsiginfo 0 6 274 274 0
tcpsndbuf 22240 333600 1800000 1900000 0
tcprcvbuf 0 222656 1800000 1900000 0
othersockbuf 104528 414944 900000 950000 0
dgramrcvbuf 0 4448 200000 200000 0
numothersock 73 105 650 660 0
dcachesize 247038 309111 786432 818029 0
numfile 904 1231 7500 7600 0
dummy 0 0 0 0 0
dummy 0 0 0 0 0
dummy 0 0 0 0 0
numiptent 4 4 155 155 0</pre>

That showed us just the portion of /proc/user_beancounters for system 765.

When you run the vzs command, always add a : after the VEID.

So, if a customer complains about some out of memory errors, or no more files, or no more ptys, or just has an unspecific complain about processes dying, etc., the very first thing you need to do is check their beancounters with vzs. Usually you will spot an item that has a high failcount and needs to be upped.

At that point you could simply up the counter with `vzctl set`. Generally pick a number 10-20% higher than the old one, and make the hard limit slightly larger than the the soft limit. However our systems now come in several levels and those levels have more/different memory allocations. If someone is complaining about something other than a memory limit (pty, numiptent, numflock), it’s generally safe to increase it, at least to the same level as what’s in the /vzconf/4unlimited file on the newest virt. If someone is hitting a memory limit, first make sure they are given what they deserve:

(refer to mgmt -> payments -> packages)

To set those levels, you use the [[#setmem|setmem]] command.

The alternate (DEPRECATED) method would be to use one of 3 commands:
256 <veid>
300 <veid>
384 <veid>
512 <veid>

If the levels were not right (you’d run vzs <veid> before and after to see the effect) tell the customer they’ve been adjusted and be done with it. If the levels were right, tell the customer they must upgrade to a higher package, tell them how to see level (control panel) and that they can reboot their system to escape this lockup contidion.

Customers can also complain that their site is totally unreachable, or complain that it is down ... if the underlying machine is up, and all seems well, you may notice in the beancounters that network-specific counters are failing - such as numtcpsock, tcpsndbuf or tcprcvbuf. This will keep them from talking on the network and make it seem like their system is down. Again, just up the limits and things should be fine.

On virts 1-4, you should first look at the default settings for that item on a later virt, such as virt 8 - we have increased the defaults a lot since the early machines. So, if you are going to up a counter on virt2, instead of upping it by 10-20%, instead up it to the new default that you see on virt8.

== Moving a VE to another virt (migrate/migrateonline) ==

This will take a while to complete - and it is best to do this at night when the load is light on both machines.

There are different methods for this, depending on which version of virtuozzo is installed on the src. and dst. virt.
To check which version is running:
[root@virt12 private]# cat /etc/virtuozzo-release
Virtuozzo release 2.6.0

Ok, let's say that the VE is 1212, and vital stats are:
<pre>[root@virt12 sbin]# vc 1212
VE_ROOT="/vz1/root/1212"
VE_PRIVATE="/vz1/private/1212"
OSTEMPLATE="fedora-core-2/20040903"
IP_ADDRESS="69.55.229.84"
TEMPLATES="devel-fc2/20040903 php-fc2/20040813 mysql-fc2/20040812 postgresql-fc2/20040813 mod_perl-fc2/20040812 mod_ssl-fc2/20040811 jre-fc2/20040823 jdk-fc2/20040823 mailman-fc2/20040823 analog-fc2/20040824 proftpd-fc2/20040818 tomcat-fc2/20040823 usermin-fc2/20040909 webmin-fc2/20040909 uw-imap-fc2/20040830 phpBB-fc2/20040831 spamassassin-fc2/20040910 PostNuke-fc2/20040824 sl-webalizer-fc2/20040
818"

[root@virt12 sbin]# vzctl exec 1212 df -h
Filesystem Size Used Avail Use% Mounted on
/dev/vzfs 4.0G 405M 3.7G 10% /</pre>

From this you can see that he’s using (and will minimally need free on the dst server) ~400MB, and he’s running on a Fedora 2 template, version 20040903. He’s also got a bunch of other templates installed. It’s is '''vital''' that '''all''' these templates exist on the dst system. To confirm that, on the dst system run:

For < 3.0:

<pre>[root@virt14 private]# vzpkgls | grep fc2
devel-fc2 20040903
PostNuke-fc2 20040824
analog-fc2 20040824
awstats-fc2 20040824
bbClone-fc2 20040824
jdk-fc2 20040823
jre-fc2 20040823
mailman-fc2 20040823
mod_frontpage-fc2 20040816
mod_perl-fc2 20040812
mod_ssl-fc2 20040811
mysql-fc2 20040812
openwebmail-fc2 20040817
php-fc2 20040813
phpBB-fc2 20040831
postgresql-fc2 20040813
proftpd-fc2 20040818
sl-webalizer-fc2 20040818
spamassassin-fc2 20040910
tomcat-fc2 20040823
usermin-fc2 20040909
uw-imap-fc2 20040830
webmin-fc2 20040909
[root@virt14 private]# vzpkgls | grep fedora
fedora-core-1 20040121 20040818
fedora-core-devel-1 20040121 20040818
fedora-core-2 20040903
[root@virt14 private]#</pre>

For these older systems, you can simply match up the date on the template.

For >= 3.0:

<pre>[root@virt19 /vz2/private]# vzpkg list
centos-5-x86 2008-01-07 22:05:57
centos-5-x86 devel
centos-5-x86 jre
centos-5-x86 jsdk
centos-5-x86 mod_perl
centos-5-x86 mod_ssl
centos-5-x86 mysql
centos-5-x86 php
centos-5-x86 plesk9
centos-5-x86 plesk9-antivirus
centos-5-x86 plesk9-api
centos-5-x86 plesk9-atmail
centos-5-x86 plesk9-backup
centos-5-x86 plesk9-horde
centos-5-x86 plesk9-mailman
centos-5-x86 plesk9-mod-bw
centos-5-x86 plesk9-postfix
centos-5-x86 plesk9-ppwse
centos-5-x86 plesk9-psa-firewall
centos-5-x86 plesk9-psa-vpn
centos-5-x86 plesk9-psa-fileserver
centos-5-x86 plesk9-qmail
centos-5-x86 plesk9-sb-publish
centos-5-x86 plesk9-vault
centos-5-x86 plesk9-vault-most-popular
centos-5-x86 plesk9-watchdog</pre>

On these newer systems, it's difficult to tell whether the template on the dst matches exactly the src. Just cause a centos-5-x86 is listed on both servers doesn't mean all the same packages are there on the dst. To truly know, you must perform a sample rsync:

rsync -avn /vz/template/centos/5/x86/ root@10.1.4.61:/vz/template/centos/5/x86/

if you see a ton of output from the dry run command, then clearly there are some differences. You may opt to let the rsync complete (without running in dry run mode) the only downside is you've now used up more space on the dst and also the centos template will be a mess with old and new data- it will be difficult if not impossible to undo (if someday we wanted to reclaim the space).

If you choose to merge templates, you should closely inspect the dry run output. You should also take care to exclude anything in the /config directory. For example:

rsync -av -e ssh --stats --exclude=x86/config /vz/template/ubuntu/10.04/ root@10.1.4.62:/vz/template/ubuntu/10.04/

Which will avoid this directory and contents:
<pre>[root@virt11 /vz2/private]# ls /vz/template/ubuntu/10.04/x86/config*
app os</pre>

This is important to avoid since the config may differ on the destination and we are really only interested in making sure the pacakges are there, not overwriting a newer config with an older one.

If the dst system was missing a template, you have 2 choices:
# put the missing template on the dst system. 2 choices here:
## Install the template from rpm (found under backup2: /mnt/data4/vzrpms/distro/) or
## rsync over the template (found under /vz/template) - see above
# put the ve on a system which has all the proper templates

=== pre-seeding a migration ===

When migrating a customer (or when doing many) depending on how much data you have to transfer, it can take some time. Further, it can be difficult to gauge when a migration will complete or how long it will take. To help speed up the process and get a better idea about how long it will take you can pre-transfer a customer's data to the destination server. If done correctly, vzmigrate will see the pre-transferred data and pick up where you left off, having much less to transfer (just changed/new files).

We believe vzmigrate uses rsync to do it's transfer. Therefore not only can you use rsync to do a pre-seed, you can also run rsync to see what is causing a repeatedly-failing vzmigrate to fail.

There's no magic to a pre-seed, you just need to make sure it's named correctly.

Given:

source: /vz1/private/1234

and you want to migrate to /vz2 on the target system, your rsync would look like:

rsync -av /vz1/private/1234/ root@x.x.x.x:/vz2/private/1234.migrated/

After running that successful rsync, the ensuing migrateonline (or migrate) will take much less time to complete- depending on the # of files to be analyzed and the # of changed files. In any case, it'll be much much faster than had you just started the migration from scratch.

Further, as we discuss elsewhere in this topic, a failed migration can be moved from <tt>/vz/private/1234</tt> to <tt>/vz/private/1234.migrated</tt> on the destination if you want to restart a failed migration. This should '''only''' be done if the migration failed and the CT is not running on the destination HN.

=== migrateonline intructions: src >=3.x -> dst>=3.x ===

A script called [[#migrateonline|migrateonline]] was written to handle this kind of move. It is basically a wrapper for <tt>vzmigrate</tt> – vzmigrate is a util to seamlessly- as no no reboot of the ve necessary- move a ve from one host to another. This wrapper was initially written cause vz virtuozzo version 2.6.0 has a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables, causing problems when the ve was started up on the dst system. [[#migrate|migrate]] mitigates that. Since it makes multiple ssh connections to the dst virt, it’s a good idea to put the pub key for the src virt in the authorized_keys file on the dst virt. In addition, migrateonline emails ve owners when their migration starts and stops. For this to happen they need to put email addresses (on a single line, space delimited) in a file on their system: /migrate_notify. If the optional target dir is not specified, the ve will be moved to the same private/root location as it was on the src virt. Note: <tt>migrate</tt> is equivalent to <tt>migrateonline</tt>, but will <tt>migrate</tt> a ve AND restart it in the process.

<pre>[root@virt12 sbin]# migrateonline
usage: /usr/local/sbin/migrateonline <ip of node migrating to> <veid> [target dir: vz | vz1 | vz2]
[root@virt12 sbin]# migrateonline 10.1.4.64 1212 vz
starting to migrate 1212 at Sat Mar 26 22:40:38 PST 2005
Turning off offline_management
Saved parameters for VE 1212
migrating with no start on 10.1.4.64
Connection to destination HN (10.1.4.64) is successfully established
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...
Syncing private area '/vz1/private/1212'
- 100% |*************************************************|
done
Successfully completed
clearing the arp cache
now going to 10.1.4.64 and clear cache and starting
starting it
Delete port redirection
Adding port redirection to VE(1): 4643 8443
Adding IP address(es) to pool: 69.55.229.84
Saved parameters for VE 1212
Starting VE ...
VE is mounted
Adding port redirection to VE(1): 4643 8443
Adding IP address(es): 69.55.229.84
Hostname for VE set: fourmajor.com
File resolv.conf was modified
VE start in progress...
finished migrating 1212 at Sat Mar 26 22 22:52:01 PST 2005
[root@virt12 sbin]#</pre>

Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine.

If they had backups, use the mvbackups command to move their backups to the new server:

mvbackups 1212 virt14 vz

Rename the ve

[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/migrated-1212

[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/old-1212-migrated-20120404-noarchive

Update the customer’s systems in mgmt to reflect the new path and server.

IF migrateonline does not work, you can try again using simply migrate- this will result in a brief reboot for the ve.
Before you try again, make sure of a few things:

Depending on where in the migration died, there may be partial data on the dst system in 1 of 2 places:
(given the example above)

/vz/private/1212

or

/vz/private/1212.migrated

before you run migrate again, you'll want to rename so that all data is in
1212.migrated:

mv /vz/private/1212 /vz/private/1212.migrated

this way, it will pick up where it left off and transfer only new files.

Likewise, if you want to speed up a migration, you can pre-seed the dst as follows:

[root@virt12 sbin]# rsync -avSH /vz/private/1212/ root@10.1.4.64:/vz/private/1212.migrated/

then when you run migrate or migrateonline, it will only need to move the changed files- the migration will complete quickly

=== migrateonline/migrate failures (migrate manually) ===

Lets say for whatever reason the migration fails. If it fails with [[#migrateonline|migrateonline]], you should try [[#migrate|migrate]] (which will reboot the customer, so notify them ahead of time).

You may want to run a [[#pre-seeding_a_migration|pre-seed]] rsync to see if you can find the problem. On older virts, we've seen this problem due to a large logfile (which you can find and encourage the customer to remove/compress):
for f in `find / -size +1048576k`; do ls -lh $f; done

You may also see migration failing due to quota issues.

You can try to resolve by copying any quota file into the file you need:

cp /var/vzquota/quota.1 /var/vzquota/quota.xxx

If it complains about quota running you should then be able to stop it

vzquota off xxxx

If all else fails, migrate to a new VEID
i.e. 1234 becomes 12341

If the rsync or [[#migrate|migrate]] fails, you can always move someone manually:

1. stop ve: 
v stop 1234

2. copy over data 
rsync -avSH /vz/private/1234/ root@1.1.1.1:/vzX/private/1234/

NOTE: if you've previously seeded the data (run rsync while the VE was up/running), and this is a subsequent rsync, make sure the last rsync you do (while the VE is not running, has the --delete option in the rsync)

3. copy over conf 
scp /vzconf/1234.conf root@1.1.1.1:/vzconf

4. on dst, edit the conf to reflect the right vzX dir 
vi /vzconf/1234.conf

5. on src remove the IPs 
ipdel 1234 2.2.2.2 3.3.3.3

6. on dst add IPs 
ipadd 1234 2.2.2.2 3.3.3.3

7. on dst, start ve: 
v start 1324

8. cancel, then archive ve on src per above instrs.

=== migrate src=2.6.0 -> dst>=2.6.0, or mass-migration with customer notify ===

A script called <tt>migrate</tt> was written to handle this kind of move. It is basically a wrapper for vzmigrate – vzmigrate is a util to seamlessly move a ve from one host to another. This wrapper was initially written cause vz virtuozzo version 2.6.0 has a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables, causing problems when the ve was started up on the dst system. migrate mitigates that. Since it makes multiple ssh connections to the dst virt, it’s a good idea to put the pub key for the src virt in the authorized_keys file on the dst virt. In addition, migrate emails ve owners when their migration starts and stops. For this to happen they need to put email addresses (on a single line, space delimited) in a file on their system: /migrate_notify. If the optional target dir is not specified, the ve will be moved to the same private/root location as it was on the src virt. Note: migrateonline is equivalent to migrate, but will migrate a ve from one 2.6 '''kernel''' machine to another 2.6 kernel machine without restarting the ve.

<pre>[root@virt12 sbin]# migrate
usage: /usr/local/sbin/migrate <ip of node migrating to> <veid> [target dir: vz | vz1 | vz2]
[root@virt12 sbin]# migrate 10.1.4.64 1212 vz
starting to migrate 1212 at Sat Mar 26 22:40:38 PST 2005
Turning off offline_management
Saved parameters for VE 1212
migrating with no start on 10.1.4.64
Connection to destination HN (10.1.4.64) is successfully established
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...
Syncing private area '/vz1/private/1212'
- 100% |*************************************************|
done
Successfully completed
clearing the arp cache
now going to 10.1.4.64 and clear cache and starting
starting it
Delete port redirection
Adding port redirection to VE(1): 4643 8443
Adding IP address(es) to pool: 69.55.229.84
Saved parameters for VE 1212
Starting VE ...
VE is mounted
Adding port redirection to VE(1): 4643 8443
Adding IP address(es): 69.55.229.84
Hostname for VE set: fourmajor.com
File resolv.conf was modified
VE start in progress...
finished migrating 1212 at Sat Mar 26 22 22:52:01 PST 2005
[root@virt12 sbin]#</pre>

Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine (backup2 or mail).
Cancel the ve (first we have to rename things which migrate changed so cancelve will find them):

[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/1212.conf

On 2.6.1 you’ll also have to move the private area:
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/1212

<pre>[root@virt12 sbin]# cancelve 1212
v stop 1212
v set 1212 --offline_management=no --save
Delete port redirection
Deleting IP address(es) from pool: 69.55.229.84
Saved parameters for VE 1212
mv /vzconf/1212.conf /vzconf/deprecated-1212
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414
don't forget to remove firewall rules and domains!
[root@virt12 sbin]#</pre>

Note: if the system had backups, [[#cancelve|cancelve]] would offer to remove them. You want to say '''no''' to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.
Then go to backup2 and move the dirs. So you’d do something like this:
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/

We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive.
If moving to the same drive, you can safely preserve hardlinks and move all files with:
sh
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done

To move everyone off a system, you’d do:
for f in `vl`; do migrate <ip> $f; done

Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.

=== vzmigrate: src=2.6.1 -> dst>=2.6.0 ===

This version of vzmigrate works properly with regard to handling ips. It will not notify ve owners of moves as in the above example. Other than that it’s essentially the same.

<pre>[root@virt12 sbin]# vzmigrate 10.1.4.64 -r no 1212:1212:/vz/private/1212:/vz/root/1212
migrating on 10.1.4.64
Connection to destination HN (10.1.4.64) is successfully established
Moving/copying VE#1212 -> VE#1212, [/vz/private/1212], [/vz/root/1212] ...
Syncing private area '/vz1/private/1212'
- 100% |*************************************************|
done
Successfully completed
Adding port redirection to VE(1): 4643 8443
Adding IP address(es) to pool: 69.55.229.84
Saved parameters for VE 1212
Starting VE ...
VE is mounted
Adding port redirection to VE(1): 4643 8443
Adding IP address(es): 69.55.229.84
Hostname for VE set: fourmajor.com
File resolv.conf was modified
VE start in progress...
[root@virt12 sbin]#</pre>

Confirm that the system is up and running on the dst virt. Try to ssh to it from another machine (backup2 or mail).
Cancel the ve (first we have to rename things which vzmigrate changed so cancelve will find them):

<pre>[root@virt12 sbin]# mv /vzconf/1212.conf.migrated /vzconf/1212.conf
[root@virt12 sbin]# mv /vz1/private/1212.migrated /vz1/private/1212

[root@virt12 sbin]# cancelve 1212
v stop 1212
v set 1212 --offline_management=no --save
Delete port redirection
Deleting IP address(es) from pool: 69.55.229.84
Saved parameters for VE 1212
mv /vzconf/1212.conf /vzconf/deprecated-1212
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414
don't forget to remove firewall rules and domains!
[root@virt12 sbin]#</pre>

Note: if the system had backups, <tt>cancelve</tt> would offer to remove them. You want to say no to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.
Then go to backup2 and move the dirs. So you’d do something like this:
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/

We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive.
If moving to the same drive, you can safely preserve hardlinks and move all files with:
sh
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done

Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.

=== src=2.5.x ===

First, go to the private dir:

cd /vz1/private/

Stop the VE - make sure it stops totally cleanly.

vzctl stop 1212

Then you’d use vemove - a script written to copy over the config, create tarballs of the ve’s data on the destination virt, and cancel the ve on the source system (in this example we’re going to put a ve that was in /vz1/private on the src virt, in /vz/private on the dst virt):

<pre>[root@virt12 sbin]# vemove
ERROR: Usage: vemove veid target_ip target_path_dir
[root@virt12 sbin]# vemove 1212 10.1.4.64 /vz/private/1212
tar cfpP - 1212 --ignore-failed-read | (ssh -2 -c arcfour 10.1.4.64 "split - -b 1024m /vz/private/1212.tar" )
scp /vzconf/1212.conf 10.1.4.64:/vzconf
cancelve 1212
v stop 1212
v set 1212 --offline_management=no --save
Delete port redirection
Deleting IP address(es) from pool: 69.55.229.84
Saved parameters for VE 1212
mv /vzconf/1212.conf /vzconf/deprecated-1212
mv /vz1/private/1212 /vz1/private/old-1212-cxld-20050414
don't forget to remove firewall rules and domains!
[root@virt12 sbin]#</pre>

Note: if the system had backups, cancelve would offer to remove them. You want to say no to this option – doing so would mean that the backups would have to be recreated on the dst virt. Instead, copy over the backup configs (make note of path changes in this example) from backup.config on the src virt to backup.config on the dst virt.
Then go to backup2 and move the dirs. So you’d do something like this:
mv /mnt/data1/virt12/0/vz1/private/1212 /mnt/data3/virt14/0/vz/private/

We don’t bother with the other dirs since there’s no harm in leaving them and eventually they’ll drop out. Besides, moving harlinked files across a filesystem as in the example above will create actual files and consume lots more space on the target drive.
If moving to the same drive, you can safely preserve hardlinks and move all files with:
sh
for f in 0 1 2 3 4 5 6; do mv /mnt/data1/virt12/$f/vz1/private/1212 /mnt/data1/virt14/$f/vz/private/; done

When you are done, go to /vz/private on the dst virt you will have files like this:

<pre>1212.taraa
1212.tarab
1212.tarac</pre>

Each one 1024m (or less, for the last one) in size.

on the dst server and run:

cat 1212.tar?? | tar xpPBf -

and after 20 mins or so it will be totally untarred. Now since the conf
file is already there, you can go ahead and start the system.

vzctl start 1212

Update the customer’s systems by clicking the “move” link on the moved system, update the system, template (should be pre-selected as the same), and the shut down date.

NOTE: you MUST tar the system up using the virtuozzo version of tar that
is on all the virt systems, and further you MUST untar the tarball with
the virtuozzo tar, using these options: `<tt>tar xpPBf -</tt>`

If you tar up an entire VE and move it to a non-virtuozzo machine, that is
ok, and you can untar it there with normal tar commands, but do not untar
it and then repack it with a normal tar and expect it to work - you need
to use virtuozzo tar commands on virtuozzo tarballs to make it work.

The backups are sort of an exception, since we are just (usually)
restoring user data that was created after we gave them the system, and
therefore has nothing to do with magic symlinks or vz-rpms, etc.

== Moving a VE on the same virt ==

Easy way: 
Scenario 1: ve 123 is to be renamed 1231 and moved from vz1 to vz

vzmlocal 123:1231:/vz/private/1231:/vz/root/1231

Scenario 2: ve 123 is to be moved vz1 to vz

vzmlocal 123:123:/vz/private/123:/vz/root/123

vzmlocal will reboot the ve at the end of the move

Manual/old way:

1) <tt>vzctl stop 123</tt> 
2) <tt>mv /vz1/private/123 /vz/private/.</tt> 
(or cp -a if you want to copy)
3) in <tt>/etc/sysconfig/vz-scripts/123.conf</tt> change value 
of '<tt>VE_PRIVATE</tt>' variable to point to a new private area location
4) <tt>vzctl start 123</tt> 
5) update backups if needed: <tt>mvbackups 123 virtX virt1 vz</tt> 
6) update management scerens 

Notes: a) absolute path to private area is stored in quota file <tt>/var/vzquota/quota.123</tt> - so during first startup quota will be recalculated. 
b) if you're going to write some script to do a job, you MUST be sure that $VEID won't be expanded to '' in ve config file - ie. you need to escape '$'. Otherwise you might have:

VE_PRIVATE="/vz/private/"

in config, and 'vzctl destroy' for this VE ID '''will remove everything under /vz/private/ directory'''.

== Adding a veth device to a VE ==

Not totally sure what this is, but a customer asked for it and here's what we did (as instructed by vz support):

<pre>v set 99 --netif_add eth99 --save
ipdel 99 69.55.230.58
v set 99 --ifname eth99 --ipadd 69.55.230.58 --save
v set 99 --ifname eth99 --gateway 69.55.230.1 --save

vznetcfg net new veth_net

# vznetcfg net list
Network ID Status Master Interface Slave Interfaces
net99 active eth0 veth77.77,veth99.99
veth_net active

# vznetcfg if list
Name Type Network ID Addresses
br0 bridge veth_net
br99 bridge net99
veth99.99 veth net99
eth1 nic 10.1.4.62/24
eth0 nic net99 69.55.227.70/24

vznetcfg br attach br0 eth0

(will remove 99 from orig net and move to veth_net)
vznetcfg net addif veth_net veth99.99

# vznetcfg net list
Network ID Status Master Interface Slave Interfaces
net99 active
veth_net active eth0 veth77.77,veth99.99

(delete the old crap)
vznetcfg net del net99

Then, to add another device in

v set 77 --netif_add eth77 --save
ipdel 77 69.55.230.78
v set 77 --ifname eth77 --ipadd 69.55.230.78 --save
v set 77 --ifname eth77 --gateway 69.55.230.1 --save
v set 77 --save --ifname eth77 --network veth_net

# vznetcfg if list
Name Type Network ID Addresses
veth77.77 veth
br0 bridge veth_net
veth99.99 veth veth_net
eth1 nic 10.1.4.62/24
eth0 nic veth_net 69.55.227.70/24

vznetcfg net addif veth_net veth77.77

# vznetcfg if list
Name Type Network ID Addresses
veth77.77 veth veth_net
br0 bridge veth_net
veth99.99 veth veth_net
eth1 nic 10.1.4.62/24
eth0 nic veth_net 69.55.227.70/24

# vznetcfg net list
Network ID Status Master Interface Slave Interfaces
veth_net active eth0 veth77.77,veth99.99

another example

v set 1182 --netif_add eth1182 --save
ipdel 1182 69.55.236.217
v set 1182 --ifname eth1182 --ipadd 69.55.236.217 --save
v set 1182 --ifname eth1182 --gateway 69.55.236.1 --save
vznetcfg net addif veth_net veth1182.1182
v set 1182 --save --ifname eth1182 --network veth_net

unused/not working commands:
ifconfig veth99.0 0
vznetcfg net list
vznetcfg br new br99 net99
vznetcfg br attach br99 eth0
vznetcfg br show
vznetcfg if list
vznetcfg net addif net99 veth99.99
vznetcfg net addif eth0 net99

---------

vznetcfg br new br1182 net1182

vznetcfg br attach br99 eth0
vznetcfg if list
vznetcfg net addif net99 veth99.99
vznetcfg net addif eth0 net99

vznetcfg net addif eth0 net1182

# vznetcfg net addif veth99.0 net99
--- 8< ---

vznetcfg net new net
# vznetcfg net addif eth0 net99

# vzctl set 99 --save --netif_add eth0 (at this stage veth99.0 interface have to appear
on node)
# vzctl set 99 --save --ifname eth0 --ipadd 69.55.230.58 (and probably few more arguments
here - see 'man vzctl')
# vznetcfg net addif veth99.0 net99</pre>

== Assigning/remove ip from a VE ==

1. Add or remove ips:
ipdel 1234 1.1.1.1 2.2.2.2
ipadd 1234 1.1.1.1 2.2.2.2

2. update Mgmt screens

3. offer to update any DNS we do for them

4. check to see if we had rules for old IP in firwall

== Enabling tun device for a ve ==
Note, there’s a command for this: [[#addtun|addtun]]

For posterity:
Make sure the tun.o module is already loaded before Virtuozzo is started:
lsmod
Allow the VPS to use the TUN/TAP device:
vzctl set 101 --devices c:10:200:rw --save
Create the corresponding device inside the VPS and set the proper permissions:
vzctl exec 101 mkdir -p /dev/net
vzctl exec 101 mknod /dev/net/tun c 10 200
vzctl exec 101 chmod 600 /dev/net/tun

== Remaking a system (on same virt) ==

1. [[#cancelve|cancelve]] (or v destroy x - ONLY if you're POSITIVE no data needs to be saved)

2. [[#vemake|vemake]] using same veid

3. [[#mvbackups|mvbackups]] or [[#vb|vb]] (if new mount point)

4. update mgmt with new dir/ip

5. update firewall if changed ip

== Re-initialize quota for a VE ==

There’s a commamd for this now: [[#clearquota|clearquota]]

For posterity:

vzctl stop 1
vzquota drop 1
vzctl start 1

== Traffic accounting on linux ==

DEPRECATED - all tracking is done via bwdb now. This is how we used to track traffic.

TODO: update for diff versions of vz

Unlike FreeBSD, where we have to add firewall count rules to the system to count the traffic, on virtuozzo counts the traffic for us. You an see the current traffic stats by running `vznetstat`:

<pre># vznetstat
VEID Net.Class Output(bytes) Input(bytes)
-----------------------------------------------
24218 1 484M 39M
24245 1 463M 143M
2451 1 2224M 265M
2454 1 2616M 385M
4149 1 125M 68M
418 1 1560M 34M
472 1 1219M 315M
726 1 628M 317M
763 1 223M 82M
771 1 4234M 437M
-----------------------------------------------
Total: 13780M 2090M</pre>

As you can see the VEID is on a line with the in and out bytes. So, we simply run a cron job:

4,9,14,19,24,29,34,39,44,49,55,59 * * * * /root/vztrafdump.sh

Just like we do on FreeBSD - this one goes through all the VEs in /vz/private and greps the line from vznetstat that matches them and dumps it in /jc_traffic_dump on their system. Then it does it again for all the VEs in /vz1/private. It is important to note that vznetstat runs only once, and the grepping is done from a temporary file that contains that output - we do this because running vznetstat once for each VE that we read out of /vz/private and /vz1/private would take way too long and be too intensive.

You do not need to do anything to facilitate this other than make sure that that cron job is running - the vznetstat counters are always running, and any new VEs that are added to the system will be accounted for automatically.

Traffic resetting no longer works with vz 2.6, so we disable the vztrafdump.sh on those virts.

== Watchdog script ==

On some of the older virts, we have a watchdog running that kills procs that are deemed bad per the following:

/root/watchdog from quar1

<pre>if echo $line | awk '{print $(NF-3)}' | grep [5-9]...
then
# 50-90%
if echo $line | awk '{print $(NF-1)}' | grep "...:.."
then
# running for > 99min
echo $line >> /root/wd.log
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log
kill -9 $pid

fi
if echo $line | awk '{print $(NF-1)}' | grep "....m"
then
# running for > 1000min
echo $line >> /root/wd.log
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log
kill -9 $pid

fi
fi

if echo $line | awk '{print $(NF-3)}' | grep [1-9]...
then
# running for 10-90 percent
if echo $line | awk '{print $NF}' | egrep 'cfusion|counter|vchkpw'
then

if echo $line | awk '{print $(NF-1)}' | grep "[2-9]:.."
then
# between 2-9min
echo $line >> /root/wd.log
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log
kill -9 $pid

elif echo $line | awk '{print $(NF-1)}' | grep "[0-9][0-9]:.."
then
# up to 99min
echo $line >> /root/wd.log
/usr/sbin/vzpid $pid | tail -1 >> /root/wd.log
kill -9 $pid

fi
fi
fi</pre>

== Misc Linux Items ==

We are overselling hard drive space ... when you configure a linux system with a certain amount of disk space (the default is 4gigs) you do not actually use up 4gigs of space on the system. The diskspace setting for a user is simply a cap, and they only use up as much space on the actual disk drive as they are actually using.

When you create a new linux system, even though there are some 300 RPMs or so installed, if you run `df -k` you will see that the entire 4gig partition is empty - no space is being used. This is because the files in their system are "magic symlinks" to the template for their OS that is in /vz/template - however, any changes to any of those files will "disconnect" them and they will immediately begin using space in their system. Further, any new files uploaded (even if those new files overwrite existing files) will take up space on the partition.

if you see this:

<pre>[root@virt8 root]# vzctl stop 160 ; vzctl start 160
VE is not running
Starting VE ...
VE is unmounted
VE is mounted
Adding IP address(es): 69.55.226.83 69.55.226.84
bash ERROR: Can't change file /etc/sysconfig/network
Deleting IP address(es): 69.55.226.83 69.55.226.84
VE is unmounted
[root@virt8 root]#</pre>

it probably means they no longer have /bin/bash - copy one in for them

ALSO: another possibility is that they have removed the `ed` RPM from their system - it needs to be reinstalled into their system. But since their system is down, this is tricky ...

VE startup scripts used by 'vzctl' want package 'ed' to be available inside VE. So if package 'ed' will be enabled in OS template config and OS template itself VE #827 is based on - this error should be fixed.

yes, it is possible to add RPM to VE while it not running.
Try to do following:

<pre># cd /vz/template/<OS_template_with_ed_package>/
# vzctl mount 827
# rpm -Uvh --root /vz/root/827 --veid 827 ed-0.2-25.i386.vz.rpm</pre>

Usually theres an error, but its ok

Note: replace 'ed-0.2-25.i386.vz.rpm' in last command with actual
version of 'ed' package you have.

----

So how do I know what template the user has ? cat their conf file and it is listed in there. For example, if the conf file has:

<pre>[root@virt12 sbin]# vc 1103
…snip…
OSTEMPLATE="debian-3.0/20030822"
TEMPLATES="mod_perl-deb30/20030707 mod_ssl-deb30/20030703 mysql-deb30/20030707 proftpd-deb30/20030703 webmin-deb30/20030823 "
…</pre>

then they are on debian 3.0, all of their system RPMs are in /vz/template/debian-3.0, and they are using version 20030822 of that debian 3.0 template. Also, they’ve also got additional packages installed (mod_perl, mod_ssl, etc). Those are also found under /vz/template

----

Edits needed to run java:

When we first created the VEs, the default setting for privvmpages was 93000:94000 ... which was high enough that most people never had problems ... however, you can;t run java or jdk or tomcat or anything java related with that setting. We have found that by setting privvmpages to 610000:615000 that java runs just fine. That is now the default setting. It is exceedingly rare that anyone needs it higher than that, although we have seen it once or twice.

Any problems with java at all - the first thing you need to do is see if the failcnt has raised for privvmpages.

<pre># vzctl start 160
Starting VE ...
vzquota : (error) Quota on syscall for 160: Device or resource busy
Running vzquota on failed for VE 160 [3]</pre>

This is because my pwd is _in_ their private directory - you can't start it until you move out

People seem to have trouble with php if they are clueless newbies. Here are two common problems/solutions:

no... but i figured it out myself. problem was the php.ini file that came
vanilla with the account was not configured to work with apache (the
ENGINE directive was set to off).

everything else seems fine now.

----

the problem was in the php.ini file. I noticed that is wasnt showing
the code when it was in an html file so I looked at the php.ini file
and had to change it so it recognized <? tags aswell as <?php tags.

Also, make sure added to httpd.conf
AddType application/x-httpd-php .php

----

You can set the time zone:

You can change the timezone by doing this:

ln -sf /usr/share/zoneinfo/<zone> /etc/localtime

where <zone> is the zone you want in the /usr/share/zoneinfo/ directory.

----

Failing shm_open calls:

first, please check if /dev/shm is mounted inside VE.
'cat /proc/mounts' command should show something like this:
tmpfs /dev/shm tmpfs rw 0 0

If /dev/shm is not mounted you have 2 ways to solve issue:
1. execute following command inside VE (doesn't require VE reboot):
mount -t tmpfs none /dev/shm
2. add following string to /etc/fstab inside VE and reboot it:
tmpfs /dev/shm tmpfs defaults 0 0

----

You can have a mounted but not running ve
Just:
vzctl mount <veid>

----

When a debian sys can’t get on the network, and you try:

<pre>vzctl set 1046 --ipadd 69.55.227.117
Adding IP address(es): 69.55.227.117
Failed to bring up lo.
Failed to bring up venet0.</pre>

They probably removed iproute package, which must be the one from swsoft. To restore:
<pre># dpkg -i --veid=1046 --admindir=/vz1/private/1046/root/var/lib/dpkg --instdir=/vz1/private/1046/root/ /vz/template/debian-3.0/iproute_20010824-8_i386.vz.deb
(Reading database ... 16007 files and directories currently installed.)
Preparing to replace iproute 20010824-8 (using .../iproute_20010824-8_i386.vz.deb) ...
Unpacking replacement iproute ...
Setting up iproute (20010824-8) ...</pre>

Then restart their ve

----

in a ve i do:

<pre>cd /
du -h .</pre>

and get: 483M .

i do:

<pre>bash-2.05a# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/vzfs 4.0G 2.3G 1.7G 56% /</pre>

how can this be?

Is it possible that quota file was corrupted somehow? Please try to:
<pre>vzctl stop <VEID>
vzquota drop <VEID>
vzquota init <VEID>
vzctl start <VEID></pre>

----

How to stop vz from starting after reboot:

VIRTUOZZO=no
in
/etc/sysconfig/vz

To start:
service vz start
(after setting VIRTUOZZO=yes in /etc/sysconfig/vz)

service vz restart will do some kind of 'soft reboot' -- restart all
VPSes and reload modules without rebooting the node

if you need to shut down all VPSes really really fast, run killall -9 init

----

Postfix tip:

You may want to tweak settings: default_process_limit=10

----

* uptime/load that vz users see isn’t what we see- it's their own load
* Vz overwrites resolve.conf on every boot to what’s in ve config file. Also overwrites hostname
* Quotas in vz: set Gid_limit in vz config file and restart ve. Errors leading up to this will be out of files but no inode or space issues. set to 2000 in conf, and restart vz

= Virtuozzo VPS Management Tools =

== vm ==

To create a new Linux VPS

* get an IP address from mgmt screen
* run df -h to figure out which disk has enough space
* run vm with no arguments to get syntax and exact template name
* use col0xxxx for hostname if they don’t give you a hostname
* copy over dir, veid, ip and password to pending customer screen

vm CID IP hostname /vz[1|2] email[,email] template ( <LB|LBP|LS|LM|LMP|LP> [disk] | <gb_disk/mb_ram/gb_burst> )

== disable ==
To disable a customers Virtuozzo VPS server.

* stop the server
vzctl stop <veid>
* prevent restart on server reboot and user restart via Control Panel
vzctl set <veid> --disabled=yes --offline_management=no --save

== cancelve ==
When a customer cancels all service.

cancelve <veid>
this will:
* stop a ve
* check for backups (offer to remove them from the backup server
and the backup.config)
* rename the private dir
* check for PTR, provide the commands to reset to default
* and rename the ve’s config
* remind you to remove firewall rules
* remind you to remove DNS entries

== ipadd ==
ipadd <veid> <ip> [ip] [ip]
add’s ip(s) to a ve

== ipdel ==
ipdel <veid> <ip> [ip] [ip]
removes ip(s) from a ve

== vc ==
vc <veid>

display the contents of the vz config file for the <veid>

the equivalent of: <tt>cat /vzconf/<veid>.conf</tt>

== vl ==
vl

List all virtual environments running on this machine.

will displays a list of ve #’s, 1 per line. (ostensibly to use in a for loop)

== vp ==
vp <veid>

show processes running under this <veid>.

the equivalent of: <tt>vzps auxww –E <veid></tt>

== vpe ==
DEPRECATED
vpe <veid>
this will allow you to do a vp when a ve is running out of control, the equivalent of (deprecated since vp operates outside the VPS):
<pre>vzctl set <veid> --kmemsize 2100000:2200000
vzctl exec <veid> ps auxw
vzctl set <veid> --kmemsize (ve’s orig lvalue):(ve’s orig hvalue)</pre>

== vt ==
vt <veid>
the equivalent of: <tt>vztop –E <veid></tt>

== vr ==
vr <veid>

Restart a <veid>.

the equivalent of: <tt>vzctl stop <veid>; vzctl start <veid></tt>
You can run this even if the ve is down - the stop command will just fail

== vs ==
vs [veid]
displays status (output of <tt>vzctl status <veid></tt>) on each ve configured on the system (as determined by <tt>/vzconf/*.conf</tt>)
If passed an argument, gives the status for just that ve.
A running system looks like:
<tt>VEID 16066 exist mounted running</tt>

A ve which is not running (but does exist) looks like:
<tt>VEID 9990 exist unmounted down</tt>

A ve which is not running and doesn’t exist looks like:
<tt>VEID 421 deleted unmounted down</tt>

== vs2 ==
vs2 [veid]
this is similar to vs in that it displays status (output of <tt>vzctl status <veid></tt>) on each ve,
but the difference is it’s list comes from doing an ls on the data dirs. This was meant to catch
the rare case where a ve configured but exists.

== vw ==
vw [veid]
displays the output of ‘<tt>w</tt>’ (the equivalent of <tt>vzctl exec <veid> w</tt>) for each configured ve (as determined by <tt>/vzconf/*.conf</tt>). Useful for determing which ve is contributing to a heavily-loaded system.
If passed an argument, gives ‘<tt>w</tt>‘ output for just that ve.
Ex:
<pre>[root@virt2 etc]# vw
134
10:52pm up 79 days, 6:14, 0 users, load average: 0.02, 0.02, 0.00
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
16027
2:52pm up 7 days, 19:54, 0 users, load average: 0.00, 0.00, 0.00
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
16055
2:52pm up 79 days, 6:38, 0 users, load average: 0.00, 0.04, 0.07
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT</pre>

== vwe ==
vwe [constraint]
just like <tt>vw</tt>, but takes a constraint as an argument, only show’s ve’s with loads >= the constraint provided. If no constraint is provided, 1 is used by default

== vzs ==
vzs [veid]
displays the beancounter status for all ve’s, or a particular ve if an argument is passed

== ve ==
ve <veid>

enter a <veid> as root.

the equivalent of: <tt>vzctl enter <veid></tt>

== vx ==
vx <veid> <command>

execute a single command in the <veid> environment.

the equivalent of: <tt>/usr/sbin/vzctl exec <veid> <command></tt>

== dprocs ==
dprocs [count]
a script which outputs a continuous report (or a certain number of reports if an option is passed) of processes stuck in the D state and which VPS’s those procs belong to.

== setmem ==
setmem VEID <256|512|768|1024|1280|1536|2048> [burst GB]
adjusts the memory resources for the VE. If this is on VZ ver >4, you will also be prompted to provide a burst ram amount. Sample usage:

<pre># setmem 21671 1024 4
/usr/sbin/vzctl set 21671 --ram 1024m --swap 3072m --save
</pre>

== afacheck.sh ==
afacheck.sh
displays the health/status of containers and mirrors on an adaptec card (currently quar1, tempvirt1-2, virt9, virt10)- all other are LSI

== backup ==
backup
backup script called nightly to update virt scripts and do customer backups

== checkload.pl ==
checkload.pl
this was intended to be setup as a cronjob to watch processes on a virt when the load
rises above a certain level. Not currently in use.

== findbackuppigs.pl ==
findbackuppigs.pl
looks for files larger than 50MB which customers have asked us to backup. Emails matches
to linux@johncompanies.com

== gatherlinux.pl ==
gatherlinux.pl
gathers up data about ve’s configured and writes to a file. Used for audits against the db

== gb ==
gb <search>
greps backup.config for the given search parameter

== gbg ==
gbg <search>
greps backup.config for the given search parameter and presents just the directories (for clean pasting)

== linuxtrafficgather.pl ==
linuxtrafficgather.pl [yy-mm]
sends a traffic usage summary by ve to support@johncomapnies.com and payments@johncompanies.com.
Optional arguments are year and month (must be in the past). If not passed, assumes last month. Relies on
traffic logs created by netstatreset and netstatbackup

== linuxtrafficwatch.pl ==
linuxtrafficwatch.pl
checks traffic usage and emails support@johncomapnies.com when a ve reaches the warning level (35G) and
the limit (40G). Works on virtuozzo versions <= 2.6.x. We really aren’t using this anymore now that we have netflow.

== linuxtrafficwatch2.pl ==
linuxtrafficwatch2.pl
checks traffic usage and emails support@johncomapnies.com when a ve reaches the warning level (35G) and
the limit (40G). Works on virtuozzo version 2.6.x. We really aren’t using this anymore now that we have netflow.

== load.pl ==
load.pl
feeds info to load mrtg - executed by inetd.

== mb (linux) ==
mb <mount|umount>
(nfs) mounts and umounts dirs to backup2. Shortcuts are mbm and mbu to mount and unmount.

== migrate ==
migrate <ip of node migrating to> <veid> <target_veid> [target dir: vz | vz1 | vz2]
this is basically a wrapper for <tt>vzmigrate</tt> – vzmigrate is a util to seamlessly move a ve from one host to another. This wrapper was written cause vz virtuozzo version 2.6 had a bug where the ve’s ip(s) on the src system were not properly removed from arp/route tables. This script mitigates that. Since it makes multiple ssh connections to the target host, it’s a good idea to put the pub key for the src system in the authorized_keys file on the target host. In addition, it emails ve owners when their migration starts and stops (if they place email addresses in a file on their system: /migrate_notify). To move everyone off a system, you’d do:
for f in `vl`; do migrate <ip> $f; done

== migrateonline ==
migrateonline <ip of node migrating to> <veid> <target_veid> [target dir: vz | vz1 | vz2]
this is the same as migrate but will migrate a ve in <tt>–online</tt> mode which means it won’t be shut down at the end of the migration. This only works when migrating ve’s between 2 machines running a 2.6 kernel (currently tempvirt1-2. virt16-19, virt12). If you get an error that the machine you’re trying to migrate to has a different CPU or features, etc, then you have to edit the file and add the –f switch to the vzmigrate line- you can basically ignore this kind of warning (but never ignore a warning about missing templates on the destination node). NOTE: This edit (if made to migrateonline) will be overwritten by the base script during each night’s backup.

== netstatbackup ==
DEPRECATED
netstatbackup
writes traffic count data to a logfile. Works on virtuozzo versions 2.5.x.

== netstatbackup2 ==
DEPRECATED
netstatbackup2
writes traffic count data to a logfile. Works on virtuozzo versions 2.6.x.

== netstatreset ==
DEPRECATED
netstatreset
writes traffic count data to a logfile and resets counters to 0. Works on virtuozzo versions 2.5.x

== netstatreset2 ==
DEPRECATED
netstatreset2
writes traffic count data to a logfile. Works on virtuozzo versions 2.6.x.

== orphanedbackupwatchlinux ==
orphanedbackupwatchlinux
looks for directories on backup2 which aren’t configured in backup.config and offers to
delete them

== rsync.backup (linux) ==
rsync.backup
does customer backups (relies on backup.config)

== startvirt.pl ==
startvirt.pl
forks off start ve commands – keeps 6 running at a time. This is not to be used on systems where fastboot is enabled as it circumvents the benefit of the fastboot. The script will occasionally not exit gracefully and will continue to use up CPU, so it should be watched. Also, don’t exit from the script till you’re sure all ve’s are started – if you do you need to start them manually and may have to free up locks. On some systems, the startvirt script doesn’t exit cleanly and you have to ^C out of it. Be careful though- doing so can leave some VE’s in an odd bootup state and you may need to ‘vr’ them manually. You should check to see which ve’s aren’t running and/or confirm all have started when ^C’ing out of startvirt.

== taskdone (linux) ==
taskdone
when called will email support@johncompanies.com with the hostname of the machine from which it was
executed as the subject

== vb (linux) ==
vb
the equivalent of: <tt>vi /usr/local/sbin/backup.config</tt>

== vemakeXX ==
DEPRECATED
vemakerh9
ve create script for RH9 (see vemake)

vemakedebian30
ve create script for debian 3.0 (Woody) (see vemake)

vemakedebian31
ve create script for debian 3.1 (Sarge) (see vemake)

vemakedebian40
ve create script for debian 4.0 (Etch) (see vemake)

vemakefedora, vemakefedora2, vemakefedora4, vemakefedora5, vemakefedora6, vemakefedora7
ve create script for fedora core 1, 2, 4, 5, 6, 7 respectively (see vemake)

vemakecentos3, vemakecentos4
ve create script for centos 3, 4 respectively (see vemake)

vemakesuse, vemakesuse93, vemakesuse100
ve create script for suse 9.2, 9.3, 10.0 respectively (see vemake)

vemakeubuntu5, vemakeubuntu606, vemakeubuntu606 vemakeubuntu704
ve create script for ubuntu 5.10, 6.06, 6.10, 7.04 respectively (see vemake)

== vemove ==
DEPRECATED
vemove <veid> <target_ip> </vz/private/123>
this script simplifies the old way of moving ve’s from one system to another - in short moving a ve to or from a virt running virtuozzo < 2.6.x
It’s the equivalent of:
<tt>tar cfpP - <veid> --ignore-failed-read | (ssh -2 -c arcfour <target_ip> "split - -b 1024m </vz/private/123>.tar" )</tt>This should only be used if migrate/vzmigrate can’t be used.

== vim.watchdog ==
vim.watchdog
looks for and kills procs matching “vi|vim|nano|pine|elm” that are running for a long time & consuming lots of cpu. Works on virtuozzo versions 2.5.x

== vim.watchdog2 ==
vim.watchdog2
looks for and kills procs matching “vi|vim|nano|pine|elm” that are running for a long time & consuming lots of cpu.
Works on virtuozzo versions 2.6.x.

== vzmigrate ==
vzmigrate <target_ip> -r no <veid>:[dst veid]:[dst /vzX/private/veid]:[dst /vzX/root/veid]
(this is the raw command “wrapped” by migrate/migrateonline) this will seamlessly move a ve from one host to another. The ve will run for the duration of the migration till the very end when it’s shut down, ip moved and started up on the target system. The filesystem on the src will remain. This should be watched – occasionally the move will timeout and leave the system shut down. If target private and root aren’t specified it just puts it in /vz. Only works when both systems are running virtuozzo 2.6.x

== vztrafdump.sh ==
DEPRECATED
vztrafdump.sh
writes traffic usage info by ve to a file called jc_traffic_dump in each ve’s / dir. Works on virtuozzo versions <= 2.5.x.

== vztrafdump2.sh ==
DEPRECATED
vztrafdump2.sh
writes traffic usage info by ve to a file called jc_traffic_dump in each ve’s / dir. Works on virtuozzo versions 2.6.x.

== addtun ==
addtun <veid>
Add’s tun device to ve.

== bwcap ==
bwcap <veid> <kbps>
Ex: <tt>bwcap 1234 512</tt>
Caps a VE’s bandwidth to the amount given

== setdisk ==
setdisk <veid> <diskspace in GB>
Ex: <tt>setdisk 1234 5</tt>
Gives a VE’s a given amount of disk space

== vdf ==
vdf <veid>
the equivalent of: <tt>vzctl exec <veid> df –h</tt>

== vdff ==
vdff
runs a (condensed) vdf for all ve’s in your pwd (must be run from /vz/privateN)

== mvbackups ==
mvbackups <veid> <target_machine> (virt1) <target_dir> (vz1)
moves backups from one location to another on the backup server, and provides you with option to remove entries from current backup.config, and simple paste command to add the config to backup.config on the target server

== checkquota ==
checkquota
for all the ve’s in the cwd (run from /vz/private, /vz1/private, etc) reports what vz quota says they’re using and what the actual usage is (as reported by du)

== clearquota ==
clearquota <veid>
Recalculates a ve’s quota, prints out the usage before and after. The equivalent of:
<tt>vdf <veid>; v stop <veid>; vzquota drop <veid>; v start <veid>; vdf <veid></tt>

== dprocs ==
dprocs
Sometimes the server’s have a large number of processes get stuck in the D state- this script shows (every 3 secs) which VE’s have D procs, which procs
are stuck and a running average of the top “offenders”

== vzstat ==
vstat
sort of like top for VZ. sort VEs by CPU usage by pressing 'o' and then 'c' keys

== stopvirt ==
stopvirt
will stop VEs as fast as it can, 6 at a time. May not exit when complete so you should watch [[#vzstat|vzstat]] in another window.

Cabinetmap

2014-03-30T10:53:34Z

72.207.101.92:

<table cellpadding=5>
<tr>
<td valign="top"><table border=1>

<tr><th>u</th> <th style="background:#FFFF99">1 </th><th>sw port</th> <th>ats port</th></tr>
<tr><td>42</td><td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p25&path= c2950 P25]
</td><td class=p>1</td> <td class=p>9</td></tr>
<tr><td>41</td><td>&nbsp </td><td class=p> </td> <td class=p></td></tr>
<tr><td>40</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>39</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>38</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>37</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>36</td><td>col02113 </td><td class=p>19 </td> <td class=p>1</td></tr>
<tr><td>35</td><td rowspan=2>col01402 RJclients </td><td class=p rowspan=2>2</td> <td class=p rowspan=2></td></tr>
<tr><td>34</td></tr>
<tr><td>33</td><td rowspan=2>col1500 </td><td class=p rowspan=2>3</td> <td class=p rowspan=2>15</td></tr>
<tr><td>32</td></tr>
<tr><td>31</td><td rowspan=2>col00138 data4 OFF </td><td class=p rowspan=2>4</td> <td class=p rowspan=2>14</td> </tr>
<tr><td>30</td></tr>
<tr><td>29</td><td rowspan=2>col01899 </td><td class=p rowspan=2>5,21 (RMM)</td> <td class=p rowspan=2>13</td></tr>
<tr><td>28</td></tr>
<tr><td>27</td><td rowspan=2>col01741 </td><td class=p rowspan=2>6</td> <td class=p rowspan=2>12</td></tr>
<tr><td>26</td></tr>
<tr><td>25</td><td rowspan=2>col01369 logos JC-08017 </td><td class=p rowspan=2>7</td> <td class=p rowspan=2>11</td></tr>
<tr><td>24</td></tr>
<tr><td>23</td><td>col01114 </td><td class=p>8</td> <td class=p >10</td></tr>
<tr><td>22</td><td> </td><td class=p>9</td> <td class=p >2-1</td></tr>
<tr><td>21</td><td colspan=3>  </td></tr>
<tr><td>20</td><td style="text-align:center; background:#cecece">[https://ats1.johncompanies.com ATS-1]
</td><td class=p>24</td> <td class=p></td></tr>
<tr><td>19</td><td colspan=3>  </td></tr>
<tr><td>18</td><td rowspan=2>col00574 </td><td class=p rowspan=2>9</td> <td class=p rowspan=2>7</td></tr>
<tr><td>17</td></tr>
<tr><td>16</td><td rowspan=2>col02210 JC-13009 </td><td class=p rowspan=2>10,11(RMM)</td> <td class=p rowspan=2>6</td></tr>
<tr><td>15</td></tr>
<tr><td>14</td><td rowspan=2>col02078 raven.nostrum.com </td><td class=p rowspan=2>17</td> <td class=p rowspan=2>5</td></tr>
<tr><td>13</td></tr>
<tr><td>12</td><td rowspan=2>col02184 JC-08028 </td><td class=p rowspan=2>12 </td> <td class=p rowspan=2>4</td></tr>
<tr><td>11</td></tr>
<tr><td>10</td><td rowspan=2> </td><td class=p rowspan=2></td> <td class=p rowspan=2>3</td></tr>
<tr><td>9 </td></tr>
<tr><td>8 </td><td rowspan=2>col00726 </td><td class=p rowspan=2>15</td> <td class=p rowspan=2>2</td></tr>
<tr><td>7 </td></tr>
<tr><td>6 </td><td rowspan=2>col01257 JC-13004 </td><td class=p rowspan=2>14 16 (RMM)</td> <td class=p rowspan=2>8</td></tr>
<tr><td>5 </td><td>  </td><td class=p></td> <td class=p></td></tr>

</table></td>

<td valign="top"><table border=1>

<tr><th>u</th> <th style="background:#FFFF99">2 </th><th>sw port</th> <th>ats port</th></tr>
<tr><td>42</td><td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p24&path= c2950 P24] </td><td class=p>1</td> <td class=p>9</td></tr>
<tr><td>41</td><td>&nbsp </td><td class=p> </td> <td class=p></td></tr>
<tr><td>40</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>39</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>38</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>37</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>36</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>35</td><td rowspan=2>col01378 </td><td class=p rowspan=2>2,15</td> <td class=p rowspan=2>15</td></tr>
<tr><td>34</td></tr>
<tr><td>33</td><td rowspan=2>col01629 newserver </td><td class=p rowspan=2>3</td> <td class=p rowspan=2>14</td></tr>
<tr><td>32</td></tr>
<tr><td>31</td><td rowspan=2>col01025 zfs </td><td class=p rowspan=2>4</td> <td class=p rowspan=2>13</td></tr>
<tr><td>30</td></tr>
<tr><td>29</td><td rowspan=2>col01631 </td><td class=p rowspan=2>5</td> <td class=p rowspan=2>12</td></tr>
<tr><td>28</td></tr>
<tr><td>27</td><td rowspan=2>col01621 agony JC-08013 </td><td class=p rowspan=2>6</td> <td class=p rowspan=2>11</td></tr>
<tr><td>26</td></tr>
<tr><td>25</td><td rowspan=2>col01567 KPSurplus OFF </td><td class=p rowspan=2>7</td> <td class=p rowspan=2>10</td></tr>
<tr><td>24</td></tr>
<tr><td>23</td><td rowspan=2>col02216 JC-06044 webengineer OFF </td><td class=p rowspan=2>18</td> <td class=p rowspan=2>8</td></tr>
<tr><td>22</td></tr>
<tr><td>21</td><td colspan=3>  </td></tr>
<tr><td>20</td><td style="text-align:center; background:#cecece">[https://ats2.johncompanies.com ATS-2]
</td><td class=p>24</td> <td class=p></td></tr>
<tr><td>19</td><td colspan=3>  </td></tr>
<tr><td>18</td><td rowspan=2>col01996 JC-08003 </td><td class=p rowspan=2>9</td> <td class=p rowspan=2>7</td></tr>
<tr><td>17</td></tr>
<tr><td>16</td><td rowspan=2>col01711 JC-07030 * </td><td class=p rowspan=2>10</td> <td class=p rowspan=2>6</td></tr>
<tr><td>15</td></tr>
<tr><td>14</td><td rowspan=2>col01788 JC-06021 </td><td class=p rowspan=2>11</td> <td class=p rowspan=2>5</td></tr>
<tr><td>13</td></tr>
<tr><td>12</td><td rowspan=2>col00226 JC-07037 </td><td class=p rowspan=2>12</td> <td class=p rowspan=2>4</td></tr>
<tr><td>11</td></tr>
<tr><td>10</td><td rowspan=2>col00633 JC-08044 </td><td class=p rowspan=2>13</td> <td class=p rowspan=2>3</td></tr>
<tr><td>9</td></tr>
<tr><td>8</td><td rowspan=2>col00433 djc-jc JC-07015 </td><td class=p rowspan=2>14</td> <td class=p rowspan=2>2</td></tr>
<tr><td>7</td></tr>
<tr><td>4</td><td rowspan=2>col00433 Dell 2950 </td><td class=p rowspan=2>16 DRAC 17</td> <td class=p rowspan=2>2</td></tr>
<tr><td>5</td></tr>

</table></td>

<td valign="top"><table border=1>

<tr><th>u</th> <th style="background:#FFFF99">3 </th><th>sw port</th> <th>ats port</th></tr>
<tr><td>42</td><td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p23&path= c2950 P23] </td><td class=p>1</td> <td class=p>9</td></tr>
<tr><td>41</td><td>&nbsp </td><td class=p> </td> <td class=p></td></tr>
<tr><td>40</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>39</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>38</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>37</td><td rowspan=2>col01727 JC-10019 </td><td class=p rowspan=2>19, 23(RMM)</td> <td class=p rowspan=2>1</td></tr>
<tr><td>36</td><td>
<tr><td>35</td><td rowspan=2>col01408 issoasis.com </td><td class=p rowspan=2>2 16 (RMM)</td> <td class=p rowspan=2>15</td></tr>
<tr><td>34</td></tr>
<tr><td>33</td><td rowspan=2>col00789 on001 </td><td class=p rowspan=2>3</td> <td class=p rowspan=2>14</td></tr>
<tr><td>32</td></tr>
<tr><td>31</td><td rowspan=2>col00789 laertes </td><td class=p rowspan=2>4</td> <td class=p rowspan=2>13</td></tr>
<tr><td>30</td></tr>
<tr><td>29</td><td rowspan=2>col02152 NO LABEL </td><td class=p rowspan=2>5</td> <td class=p rowspan=2>12</td></tr>
<tr><td>28</td></tr>
<tr><td>27</td><td rowspan=2>col02219 JC-0811 </td><td class=p rowspan=2>6</td> <td class=p rowspan=2>11</td></tr>
<tr><td>26</td></tr>
<tr><td>25</td><td rowspan=2>col01333 AJA Video OFF </td><td class=p rowspan=2>P24-22</td> <td class=p rowspan=2>10</td></tr>
<tr><td>24</td></tr>
<tr><td>23</td><td rowspan=2>col01741 cyrus.base </td><td class=p rowspan=2>15, 22 (RMM)</td> <td class=p rowspan=2>8</td></tr>
<tr><td>22</td></tr>
<tr><td>21</td><td colspan=3>  </td></tr>
<tr><td>20</td><td style="text-align:center; background:#cecece">[https://ats3.johncompanies.com ATS-3]
</td><td class=p>24</td> <td class=p></td></tr>
<tr><td>19</td><td colspan=3>  </td></tr>
<tr><td>18</td><td rowspan=2>col00789 bb001 </td><td class=p rowspan=2>9, 20 (RMM)</td> <td class=p rowspan=2>7</td></tr>
<tr><td>17</td></tr>
<tr><td>16</td><td rowspan=2>col01224 pariotrial.sea-incorporated </td><td class=p rowspan=2>10, 21 (RMM)</td> <td class=p rowspan=2>6</td></tr>
<tr><td>15</td></tr>
<tr><td>14</td><td rowspan=2>col01777 </td><td class=p rowspan=2>11</td> <td class=p rowspan=2>5</td></tr>
<tr><td>13</td></tr>
<tr><td>12</td><td rowspan=2>col01224 fuji </td><td class=p rowspan=2>12</td> <td class=p rowspan=2>4</td></tr>
<tr><td>11</td></tr>
<tr><td>10</td><td rowspan=2>col01817 </td><td class=p rowspan=2>13</td> <td class=p rowspan=2>3</td></tr>
<tr><td>9</td></tr>
<tr><td>8</td><td rowspan=2>col01332 vs4 </td><td class=p rowspan=2>14</td> <td class=p rowspan=2>2</td></tr>
<tr><td>7</td></tr>
<tr><td>6</td><td rowspan=2>col02202 JC-13006 OFF </td><td class=p rowspan=2>8 17 (RMM)</td> <td class=p rowspan=2>16</td></tr>
<tr><td>5</td><td>  </td><td class=p></td> <td class=p></td></tr>

</table></td>

<td valign="top"><table border=1>

<tr><th>u</th> <th style="background:#FFFF99">4 </th><th>sw port</th> <th>ats port</th></tr>
<tr><td>42</td><td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p22&path= c2950 P22] </td><td class=p>1</td> <td class=p>9</td></tr>
<tr><td>41</td><td>&nbsp </td><td class=p> </td> <td class=p></td></tr>
<tr><td>40</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>39</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>38</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>37</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>36</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>35</td><td rowspan=2>col01224 cisco (no rails) </td><td class=p rowspan=2>2</td> <td class=p rowspan=2>16</td></tr>
<tr><td>34</td></tr>
<tr><td>33</td><td rowspan=2>col01224 obc-c </td><td class=p rowspan=2> </td> <td class=p rowspan=2>15</td></tr>
<tr><td>32</td></tr>
<tr><td>31</td><td rowspan=2>col01224 obc-b </td><td class=p rowspan=2> </td> <td class=p rowspan=2>14</td></tr>
<tr><td>30</td></tr>
<tr><td>29</td><td rowspan=2>col01224 obc-a </td><td class=p rowspan=2> </td> <td class=p rowspan=2>13</td></tr>
<tr><td>28</td></tr>
<tr><td>27</td><td rowspan=2>col02220 OFF </td><td class=p rowspan=2></td> <td class=p rowspan=2>12</td></tr>
<tr><td>26</td></tr>
<tr><td>25</td><td>col01332 emailmarketer </td><td class=p>4</td> <td class=p>11</td></tr>
<tr><td>24</td><td>col01025 backup </td><td class=p>13</td> <td class=p>3</td></tr>
<tr><td>23</td><td rowspan=2>col01224 hydro4gehosting </td><td class=p rowspan=2>6</td> <td class=p rowspan=2>8</td></tr>
<tr><td>22</td></tr>
<tr><td>21</td><td colspan=3>  </td></tr>
<tr><td>20</td><td style="text-align:center; background:#cecece">[https://ats4.johncompanies.com ATS-4]
</td><td class=p>24</td> <td class=p></td></tr>
<tr><td>19</td><td colspan=3>  </td></tr>
<tr><td>18</td><td rowspan=2>col01182 dev01 </td><td class=p rowspan=2>7</td> <td class=p rowspan=2>7</td></tr>
<tr><td>17</td></tr>
<tr><td>16</td><td rowspan=2>col01182 demo2 </td><td class=p rowspan=2>8</td> <td class=p rowspan=2>6</td></tr>
<tr><td>15</td></tr>
<tr><td>14</td><td rowspan=2>col00336 rb3 </td><td class=p rowspan=2>9</td> <td class=p rowspan=2>5</td></tr>
<tr><td>13</td></tr>
<tr><td>12</td><td rowspan=2>col02221 </td><td class=p rowspan=2>10</td> <td class=p rowspan=2>4</td></tr>
<tr><td>11</td></tr>
<tr><td>10</td><td rowspan=2>col01710 </td><td class=p rowspan=2>11</td> <td class=p rowspan=2>3</td></tr>
<tr><td>9</td></tr>
<tr><td>8</td><td rowspan=2>col00336 coach </td><td class=p rowspan=2>12</td> <td class=p rowspan=2>2</td></tr>
<tr><td>7</td></tr>
<tr><td>6</td><td rowspan=2>col00534 malkum.pglaf </td><td class=p rowspan=2>18 RMM 19</td> <td class=p rowspan=2>10</td></tr>
<tr><td>5</td><td></tr>

</table></td>

<td valign="top"><table border=1>

<tr><th>u</th> <th style="background:#FFFF99">5 </th><th>sw port</th> <th>ats port</th></tr>
<tr><td>42</td><td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p21&path= c2950 P21]
</td><td class=p>g1</td> <td class=p>9</td></tr>
<tr><td>41</td><td>&nbsp </td><td class=p> </td> <td class=p></td></tr>
<tr><td>40</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>39</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>38</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>37</td><td rowspan=2>col01849 snacktime </td><td class=p rowspan=2>16 17 (RMM)</td> <td class=p rowspan=2>2</td></tr>
<tr><td>36</td><td>
<tr><td>35</td><td rowspan=2>col01224 ewing2 </td><td class=p rowspan=2>1</td> <td class=p rowspan=2></td></tr>
<tr><td>34</td></tr>
<tr><td>33</td><td rowspan=2>col01224 ewing </td><td class=p rowspan=2>2</td> <td class=p rowspan=2>15</td></tr>
<tr><td>32</td></tr>
<tr><td>31</td><td rowspan=2>col01224 luzerne </td><td class=p rowspan=2>11</td> <td class=p rowspan=2>14</td></tr>
<tr><td>30</td></tr>
<tr><td>29</td><td rowspan=2>col02215 JC-08019 </td><td class=p rowspan=2>20</td> <td class=p rowspan=2>13</td></tr>
<tr><td>28</td></tr>
<tr><td>27</td><td rowspan=2>col01556 </td><td class=p rowspan=2>5</td> <td class=p rowspan=2>12</td></tr>
<tr><td>26</td></tr>
<tr><td>25</td><td rowspan=2>col02138 webengineers </td><td class=p rowspan=2>7, 21 (RMM)</td> <td class=p rowspan=2>11</td></tr>
<tr><td>24</td></tr>
<tr><td>23</td><td rowspan=2>col01495 beany </td><td class=p rowspan=2>22</td> <td class=p rowspan=2>10</td></tr>
<tr><td>22</td></tr>
<tr><td>21</td><td colspan=3>  </td></tr>
<tr><td>20</td><td style="text-align:center; background:#cecece">[https://ats5.johncompanies.com ATS-5]
</td><td class=p>24</td> <td class=p></td></tr>
<tr><td>19</td><td colspan=3>  </td></tr>
<tr><td>18</td><td rowspan=2>col01841 paradigm </td><td class=p rowspan=2>8</td> <td class=p rowspan=2>6</td></tr>
<tr><td>17</td></tr>
<tr><td>16</td><td rowspan=2>col01080 helium OFF </td><td class=p rowspan=2>9</td> <td class=p rowspan=2>5</td></tr>
<tr><td>15</td></tr>
<tr><td>14</td><td rowspan=2>col01629 anitaborg mail </td><td class=p rowspan=2>10</td> <td class=p rowspan=2>7</td></tr>
<tr><td>13</td></tr>
<tr><td>12</td><td rowspan=2>col02205 OFF </td><td class=p rowspan=2>5 3 (RMM)</td> <td class=p rowspan=2>3</td></tr>
<tr><td>11</td></tr>
<tr><td>10</td><td rowspan=2>col01841 smartypants * </td><td class=p rowspan=2>12,13 (DRAC)</td> <td class=p rowspan=2>8</td></tr>
<tr><td>9</td></tr>
<tr><td>8</td><td rowspan=2>col01408 pepclub </td><td class=p rowspan=2>6, 15 (RMM)</td> <td class=p rowspan=2>4</td></tr>
<tr><td>7</td></tr>

</table></td>
</tr>
</table>

 

<table cellpadding=5>
<tr>
<td valign="top"><table border=1>

<tr><th>u</th> <th style="background:#FFFF99">6 </th><th>sw port</th> <th>ats port</th></tr>
<tr><td>42</td><td class="jc">digi3 </td><td class=p></td><td class=p>10</td></tr>
<tr><td>41</td><td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p20&path= c2950 P20]
</td><td class=p></td><td class=p>9</td></tr>
<tr><td>40</td><td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p12&path= c2950 P12 private]
</td><td class=p></td><td class=p>2</td></tr>
<tr><td>39</td><td>  c2960G P30 </td><td class=p></td><td class=p> </td></tr>
<tr><td>38</td><td>  </td><td class=p></td><td class=p> </td></tr>
<tr><td>37</td><td>isys </td><td class=p></td><td class=p>3</td></tr>
<tr><td>36</td><td rowspan=2>firewall2 </td><td class=p rowspan=2>p30-5-6</td><td class=p rowspan=2>8 (A), 7 (B)</td></tr>
<tr><td>35</td></tr>
<tr><td>34</td><td rowspan=2>firewall3 </td><td class=p rowspan=2>p12</td><td class=p rowspan=2>14</tr>
<tr><td>33</td></tr>
<tr><td>32</td><td rowspan=2>nat2 </td><td class=p rowspan=2>10</td><td class=p rowspan=2>6</td></tr>
<tr><td>31</td></tr>
<tr><td>30</td><td rowspan=2>bwdb2 </td><td class=p rowspan=2>p30-3</td><td class=p rowspan=2>5</td></tr>
<tr><td>29</td></tr>
<tr><td>28</td><td rowspan=2>old backup3 </td><td class=p rowspan=2>21</td><td class=p rowspan=2>4</td></tr>
<tr><td>27</td></tr>
<tr><td>26</td><td rowspan=2>jail3 (col01737) </td><td class=p rowspan=2>p21-13</td><td class=p rowspan=2>11</td></tr>
<tr><td>25</td></tr>
<tr><td>24</td><td rowspan=2>Magrathea2 </td><td class=p rowspan=2>p22-3</td><td class=p rowspan=2>16</td></tr>
<tr><td>23</td></tr>
<tr><td>22</td></tr>
<tr><td>21</td></tr>
<tr><td>20</td><td style="text-align:center; background:#cecece">[https://ats6.johncompanies.com ATS-6]
</td><td class=p></td><td class=p></td></tr>
<tr><td>19</td><td>  </td><td class=p></td><td class=p></td></tr>
<tr><td>18</td><td rowspan=2> </td><td class=p rowspan=2></td><td class=p rowspan=2></td></tr>
<tr><td>17</td></tr>
<tr><td>16</td><td rowspan=2>magrathea </td><td class=p rowspan=2>11</td><td class=p rowspan=2>15</td></tr>
<tr><td>15</td></tr>
<tr><td>14</td><td> col02015 cisco </td><td class=p>p21-18</td><td class=p>13</td></tr>
<tr><td>13</td><td>  </td><td class=p></td><td class=p></td></tr>
<tr><td>12</td><td rowspan=2> col02015 dell </td><td class=p rowspan=2>p21-15 drac</td><td class=p rowspan=2>12</td></tr>
<tr><td>11</td></tr>
<tr><td>10</td><td>  </td><td class=p></td><td class=p></td></tr>
<tr><td>9</td><td rowspan=3>new backup3 3U </td><td class=p rowspan=3>p30-7</td><td class=p rowspan=3>1</tr>
<tr><td>8</td></tr>
<tr><td>7</td></tr>

</table></td>

<td valign="top"><table border=1>

<tr><th>u</th> <th style="background:#FFFF99">7 </th><th>sw port</th> <th>ats port</th></tr>
<tr><td>42</td><td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p26&path= c2950 P26]
</td><td class=p>1</td> <td class=p>9</td></tr>
<tr><td>41</td><td>&nbsp </td><td class=p> </td> <td class=p></td></tr>
<tr><td>40</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>39</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>38</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>37</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>36</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>35</td><td rowspan=2> </td><td class=p rowspan=2>?</td> <td class=p rowspan=2>15</td></tr>
<tr><td>34</td></tr>
<tr><td>33</td><td rowspan=2> </td><td class=p rowspan=2>3</td> <td class=p rowspan=2>14</td></tr>
<tr><td>32</td></tr>
<tr><td>31</td><td rowspan=2>col01224 brandgage </td><td class=p rowspan=2>4</td> <td class=p rowspan=2>13</td></tr>
<tr><td>30</td></tr>
<tr><td>29</td><td rowspan=2> </td><td class=p rowspan=2>5</td> <td class=p rowspan=2>12</td></tr>
<tr><td>28</td></tr>
<tr><td>27</td><td rowspan=2> </td><td class=p rowspan=2>6</td> <td class=p rowspan=2>11</td></tr>
<tr><td>26</td></tr>
<tr><td>25</td><td rowspan=2> </td><td class=p rowspan=2>7</td> <td class=p rowspan=2>10</td></tr>
<tr><td>24</td></tr>
<tr><td>23</td><td rowspan=2> </td><td class=p rowspan=2>8</td> <td class=p rowspan=2>8</td></tr>
<tr><td>22</td></tr>
<tr><td>21</td><td colspan=3>  </td></tr>
<tr><td>20</td><td style="text-align:center; background:#cecece">[https://ats7.johncompanies.com ATS-7]
</td><td class=p>24</td> <td class=p></td></tr>
<tr><td>19</td><td colspan=3> </td></tr>
<tr><td>18</td><td rowspan=2>cloud1 Dell2950 </td><td class=p rowspan=2>9</td> <td class=p rowspan=2>7</td></tr>
<tr><td>17</td></tr>
<tr><td>16</td><td rowspan=2>col00875 backup OFF </td><td class=p rowspan=2></td> <td class=p rowspan=2>6</td></tr>
<tr><td>15</td></tr>
<tr><td>14</td><td rowspan=2> </td><td class=p rowspan=2>10</td> <td class=p rowspan=2>5</td></tr>
<tr><td>13</td></tr>
<tr><td>12</td><td rowspan=2>col02171 </td><td class=p rowspan=2>11</td> <td class=p rowspan=2>4</td></tr>
<tr><td>11</td></tr>
<tr><td>10</td><td rowspan=2>jc11013 label </td><td class=p rowspan=2>12</td> <td class=p rowspan=2>3</td></tr>
<tr><td>9</td></tr>
<tr><td>8</td><td rowspan=2>col02108 </td><td class=p rowspan=2>13</td> <td class=p rowspan=2>2</td></tr>
<tr><td>7</td></tr>
<tr><td>6</td><td rowspan=4> On Floor -- col00336 fumble.fantasybowl.com (Atom) and col02108 JC-11009 (Atom)</td><td class=p>12</td> <td class=p>1</td></tr>
<tr><td>5</td><td>  </td><td class=p></td> <td class=p></td></tr>
<tr><td>4</td><td>  </td><td class=p></td> <td class=p></td></tr>
<tr><td>3</td><td>  </td><td class=p></td> <td class=p></td></tr>

</table></td>

<td valign="top"><table border=1>

<tr><th>u</th> <th style="background:#FFFF99">8 </th><th>sw port</th> <th>ats port</th></tr>
<tr><td>42</td><td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p27&path= c2950 P27]
</td><td class=p>1</td> <td class=p>9</td></tr>
<tr><td>41</td><td>&nbsp </td><td class=p> </td> <td class=p></td></tr>
<tr><td>40</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>39</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>38</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>37</td><td rowspan=2>col01719 (1U) </td><td class=p rowspan=2>16</td> <td class=p rowspan=2>16</td></tr>
<tr><td>36</td></tr>
<tr><td>35</td><td rowspan=2> </td><td class=p rowspan=2>2</td> <td class=p rowspan=2>15</td></tr>
<tr><td>34</td></tr>
<tr><td>33</td><td rowspan=2>col01224 idgroup </td><td class=p rowspan=2>3</td> <td class=p rowspan=2>14</td></tr>
<tr><td>32</td></tr>
<tr><td>31</td><td rowspan=2> </td><td class=p rowspan=2>4</td> <td class=p rowspan=2>13</td></tr>
<tr><td>30</td></tr>
<tr><td>29</td><td rowspan=2>col02150 </td><td class=p rowspan=2>5, 21 (RMM)</td> <td class=p rowspan=2>12</td></tr>
<tr><td>28</td></tr>
<tr><td>27</td><td rowspan=2>col01860 </td><td class=p rowspan=2>6</td> <td class=p rowspan=2>11</td></tr>
<tr><td>26</td></tr>
<tr><td>25</td><td rowspan=2>col01025 feros </td><td class=p rowspan=2>7, 18 (RMM)</td> <td class=p rowspan=2>10</td></tr>
<tr><td>24</td></tr>
<tr><td>23</td><td rowspan=2> </td><td class=p rowspan=2>8</td> <td class=p rowspan=2>8</td></tr>
<tr><td>22</td></tr>
<tr><td>21</td><td colspan=3>  </td></tr>
<tr><td>20</td><td style="text-align:center; background:#cecece">[https://ats8.johncompanies.com ATS-8]
</td><td class=p>24</td> <td class=p></td></tr>
<tr><td>19</td><td colspan=3> </td></tr>
<tr><td>18</td><td rowspan=2>col01941 </td><td class=p rowspan=2>9</td> <td class=p rowspan=2>7</td></tr>
<tr><td>17</td></tr>
<tr><td>16</td><td rowspan=2>col01511 * </td><td class=p rowspan=2>10</td> <td class=p rowspan=2>6</td></tr>
<tr><td>15</td></tr>
<tr><td>14</td><td rowspan=2>col00138 data2 OFF </td><td class=p rowspan=2>15</td> <td class=p rowspan=2>5</td></tr>
<tr><td>13</td></tr>
<tr><td>12</td><td rowspan=2>col01567 prod OFF </td><td class=p rowspan=2>12</td> <td class=p rowspan=2>4</td></tr>
<tr><td>11</td></tr>
<tr><td>10</td><td rowspan=3> </td><td class=p rowspan=3></td> <td class=p rowspan=3>3</td></tr>
<tr><td>9</td></tr>
<tr><td>8</td></tr>
<tr><td>7</td><td rowspan=2>col02001 attic1jc </td><td class=p rowspan=2>11</td> <td class=p rowspan=2>2</td></tr>
<tr><td>6</td></tr>

</table></td>

<td valign="top"><table border=1>

<tr><th>u</th> <th style="background:#FFFF99">9 </th><th>sw port</th> <th>ats port</th></tr>
<tr><td>42</td><td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p28&path= c2950 P28]
</td><td class=p>1</td> <td class=p>9</td></tr>
<tr><td>41</td><td>&nbsp </td><td class=p> </td> <td class=p></td></tr>
<tr><td>40</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>39</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>38</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>37</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>36</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>35</td><td rowspan=2> </td><td class=p rowspan=2>2</td> <td class=p rowspan=2>15</td></tr>
<tr><td>34</td></tr>
<tr><td>33</td><td rowspan=2> </td><td class=p rowspan=2>3 <td> <td class=p rowspan=2>14</td></tr>
<tr><td>32</td></tr>
<tr><td>31</td><td rowspan=2> </td><td class=p rowspan=2>4</td> <td class=p rowspan=2>13</td></tr>
<tr><td>30</td></tr>
<tr><td>29</td><td rowspan=2> </td><td class=p rowspan=2>5</td> <td class=p rowspan=2>12</td></tr>
<tr><td>28</td></tr>
<tr><td>27</td><td rowspan=2> </td><td class=p rowspan=2>6</td> <td class=p rowspan=2>11</td></tr>
<tr><td>26</td></tr>
<tr><td>25</td><td rowspan=2> </td><td class=p rowspan=2>7</td> <td class=p rowspan=2>10</td></tr>
<tr><td>24</td></tr>
<tr><td>23</td><td rowspan=2> </td><td class=p rowspan=2>8</td> <td class=p rowspan=2>8</td></tr>
<tr><td>22</td></tr>
<tr><td>21</td><td colspan=3>  </td></tr>
<tr><td>20</td><td style="text-align:center; background:#cecece">[https://ats9.johncompanies.com ATS-9]
</td><td class=p>24</td> <td class=p></td></tr>
<tr><td>19</td><td colspan=3> </td></tr>
<tr><td>18</td><td rowspan=2> </td><td class=p rowspan=2>9</td> <td class=p rowspan=2>7</td></tr>
<tr><td>17</td></tr>
<tr><td>16</td><td rowspan=2> </td><td class=p rowspan=2>10</td> <td class=p rowspan=2>6</td></tr>
<tr><td>15</td></tr>
<tr><td>14</td><td rowspan=2> </td><td class=p rowspan=2>11</td> <td class=p rowspan=2>5</td></tr>
<tr><td>13</td></tr>
<tr><td>12</td><td rowspan=2> </td><td class=p rowspan=2>12</td> <td class=p rowspan=2>4</td></tr>
<tr><td>11</td></tr>
<tr><td>10</td><td rowspan=2> </td><td class=p rowspan=2>13</td> <td class=p rowspan=2>3</td></tr>
<tr><td>9</td></tr>
<tr><td>8</td><td rowspan=2> * </td><td class=p rowspan=2>14</td> <td class=p rowspan=2>2</td></tr>
<tr><td>7</td></tr>

</table></td>

</tr>
</table>

 
<table border=0 cellpadding=5>
<tr>
<td valign="top">
<table border=1>

<tr><th>U</th> <th style="background:#FFFF99">3-5 </th> <th>port</th></tr>
<tr><td class=u>129-127</td> <td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p3&path= c2924 P3] </td> <td class=p></td></tr>
<tr><td class=u>126-124</td> <td class=digi>digi2 </td> <td class=p></td></tr>
<tr><td class=u>123 118</td> <td>2U </td> <td class=p></td></tr>
<tr><td class=u>117 112</td> <td style="background:#f5ac81">virt14 * Dell 2950 </td> <td class=p>13</td></tr>
<tr><td class=u>111 106</td> <td style="background:#8df595">jail19 * </td> <td class=p></td></tr>
<tr><td class=u>105 100</td> <td style="background:#f5ac81">virt15 * </td> <td class=p></td></tr>
<tr><td class=u>99 94</td> <td>col00779 wendlink 233.77 </td> <td class=p></td></tr>
<tr><td class=u>93 88</td> <td style="background:#8df595">jail2 * </td> <td class=p></td></tr>
<tr><td class=u>87 82</td> <td style="background:#f5ac81">virt13 * Dell 2950 </td> <td class=p></td></tr>
<tr><td class=u>81 76</td> <td style="background:#8df595">old jail2 * </td> <td class=p></td></tr>
<tr><td class=u>75 70</td> <td style="background:#8df595">mx2 * </td> <td class=p></td></tr>
<tr><td class=u>69 64</td> <td>2U </td> <td class=p></td></tr>
<tr><td class=u>63 58</td> <td>col01372 cache2 </td> <td class=p></td></tr>
<tr><td class=u>57 52</td> <td>2U </td> <td class=p></td></tr>
<tr><td class=u>51 46</td> <td>col00499 </td> <td class=p></td></tr>
<tr><td class=u>45 40</td> <td>col02204 NIC P3-12 RMM P3-16 </td> <td class=p></td></tr>
<tr><td class=u>39 34</td> <td class=c>2U </td> <td class=p></td></tr>
<tr><td class=u>33 28</td> <td>col01727 - REMOVED 9/22/13 </td> <td class=p></td></tr>
</table>
</td>

<td valign="top">
<table border=1>

<tr><th>U</th> <th style="background:#FFFF99">3-6 </th> <th>port</th></tr>
<tr><td class=u>129-127</td> <td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p4&path= c2924 P4] </td> <td class=p></td></tr>
<tr><td class=u>126-124</td> <td>1U </td> <td class=p></td></tr>
<tr><td class=u>123 118</td> <td style="background:#f5ac81">virt11 * </td> <td class=p>4</td></tr>
<tr><td class=u>117 112</td> <td style="background:#f5ac81">virt17 * </td> <td class=p>7</td></tr>
<tr><td class=u>111-109</td> <td>1U </td> <td class=p></td></tr>
<tr><td class=u>108-106</td> <td>2U </td> <td class=p></td></tr>
<tr><td class=u>105 100</td> <td style="background:#8df595">jail8 * </td> <td class=p>10</td></tr>
<tr><td class=u>99 94</td> <td style="background:#f5ac81">virt19 * </td> <td class=p>12</td></tr>
<tr><td class=u>93 88</td> <td style="background:#8df595">jail9 * </td> <td class=p></td></tr>
<tr><td class=u>87 82</td> <td>col01372 cache4 </td> <td class=p></td></tr>
<tr><td class=u>81 76</td> <td>col01372 dbmsg.bookoo </td> <td class=p></td></tr>
<tr><td class=u>75 70</td> <td>2U </td> <td class=p></td></tr>
<tr><td class=u>69 64</td> <td>2U </td> <td class=p></td></tr>
<tr><td class=u>63 58</td> <td>2U </td> <td class=p></td></tr>
<tr><td class=u>57 52</td> <td>2U </td> <td class=p></td></tr>
<tr><td class=u>51 46</td> <td>2U </td> <td class=p></td></tr>
<tr><td class=u>45 40</td> <td>col00985 OFF </td> <td class=p></td></tr>
<tr><td class=u>39 34</td> <td>col02204 devsvr NIC P4-12, RMM P4-16 </td> <td class=p></td></tr>
<tr><td class=u>33 28</td> <td> </td> <td class=p></td></tr>

</table>
</td>

<td valign="top">
<table border=1>

<tr><th>U</th> <th style="background:#FFFF99">3-7 </th> <th>port</th></tr>
<tr><td class=u>129-127</td> <td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p17&path= c2924 P17] </td> <td class=p></td></tr>
<tr><td class=u>126-124</td> <td class=c>1U </td> <td class=p></td></tr>
<tr><td class=u>123-121</td> <td class=digi>digi1 (no rails)</td> <td class=p></td></tr>
<tr><td class=u>120-118</td> <td style="background:#a5e9f3">switch-m2 </td> <td class=p></td></tr>
<tr><td class=u>117-115</td> <td style="text-align:center; background:#cecece">ATS </td> <td class=p></td></tr>
<tr><td class=u>114-112</td> <td style="text-align:center; background:#cecece">ATS </td> <td class=p></td></tr>
<tr><td class=u>111 106</td> <td style="background:#8df595">jail11 * </td> <td class=p></td></tr>
<tr><td class=u>105 100</td> <td class=c>col01372 solr7.bookoo </td> <td class=p></td></tr>
<tr><td class=u>99 94</td> <td>col01372 solr8.bookoo </td> <td class=p></td></tr>
<tr><td class=u>93 88</td> <td style="background:#f5ac81">virt12 * </td> <td class=p></td></tr>
<tr><td class=u>87 82</td> <td style="background:#8df595">jail7 * </td> <td class=p></td></tr>
<tr><td class=u>81<div class=s3></div>73</td> <td style="background:#f5ac81">virt14 </td> <td class=p></td></tr>
<tr><td class=u>72-70</td> <td style="background:#cecece"> bwdb1 (no rails) </td> <td class=p></td></tr>
<tr><td class=u>69 64</td> <td style="background:#f5ac81">virt16 * </td> <td class=p></td></tr>
<tr><td class=u>63-61</td> <td>col00991 </td> <td class=p></td></tr>
<tr><td class=u>60<div class=s4></div>49</td> <td style="background:#cecece">backup2 *</td> <td class=p></td></tr>
<tr><td class=u>48 43</td> <td style="background:#8df595">jail18 * </td> <td class=p></td></tr>
<tr><td class=u>42 37</td> <td style="background:#8df595">mx1 * </td> <td class=p></td></tr>
<tr><td class=u>36-34</td> <td class=c>1U </td> <td class=p></td></tr>
<tr><td class=u>33-31</td> <td>col00994 </td> <td class=p></td></tr>
<tr><td class=u>30 25</td> <td style="background:#f5ac81">virt9 * </td> <td class=p></td></tr>
<tr><td class=u>24 19</td> <td>2U </td> <td class=p></td></tr>
<tr><td class=u>18-16</td> <td>col0460 </td> <td class=p></td></tr>
<tr><td class=u>15-13</td> <td class=c>1U </td> <td class=p></td></tr>
<tr><td class=u>12-10</td> <td style="background:#cecece">nat </td> <td class=p></td></tr>

</table>
</td>

<td valign="top">
<table border=1>

<tr><th>U</th> <th style="background:#FFFF99">3-8 </th> <th>port</th></tr>
<tr><td class=u>129-127</td> <td class=c>1U </td> <td class=p></td></tr>
<tr><td class=u>126-124</td> <td style="background:#a5e9f3">c3750 </td> <td class=p></td></tr>
<tr><td class=u>122 118</td> <td style="background:#a5e9f3">c3550 </td> <td class=p></td></tr>
<tr><td class=u>117-115</td> <td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p1&path= c2924 switch-p1] </td> <td class=p></td></tr>
<tr><td class=u>114-112</td> <td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p1a&path= c2960 switch-p1a] </td> <td class=p></td></tr>
<tr><td class=u>111-109</td> <td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p1b&path= c2960 switch-p1b] </td> <td class=p></td></tr>
<tr><td class=u>108 103</td> <td style="background:#cecece">firewall * </td> <td class=p></td></tr>
<tr><td class=u>102-100</td> <td style="background:#a5e9f3">ASA5500 </td> <td class=p></td></tr>
<tr><td class=u>99-97</td> <td style="background:#a5e9f3">M1 </td> <td class=p></td></tr>
<tr><td class=u>96 91</td> <td class=c>col02179 staging.issisystems.com </td> <td class=p></td></tr>
<tr><td class=u>90-88</td> <td style="background:#cecece">gate (old f/w) </td> <td class=p></td></tr>
<tr><td class=u>87 82</td> <td style="background:#f5ac81">quar1 * (no rails)</td> <td class=p></td></tr>
<tr><td class=u>81-79</td> <td>[shelf] </td> <td class=p></td></tr>
<tr><td class=u>78-76</td> <td class=c>1U </td> <td class=p></td></tr>
<tr><td class=u>75-73</td> <td style="text-align:center; background:#cecece">ATS </td> <td class=p></td></tr>
<tr><td class=u>72-70</td> <td style="text-align:center; background:#cecece">ATS </td> <td class=p></td></tr>
<tr><td class=u>69 64</td> <td style="background:#8df595">jail4</td> <td class=p>p1-16</td></tr>
<tr><td class=u>63 58</td> <td style="background:#8df595">jail1 * (no rails)</td> <td class=p></td></tr>
<tr><td class=u>57-55</td> <td class=c>1U </td> <td class=p></td></tr>
<tr><td class=u>54 52</td> <td style="background:#cecece">console (no rails) </td> <td class=p></td></tr>
<tr><td class=u>51 46</td> <td style="background:#cecece">mail * (no rails) </td> <td class=p></td></tr>
<tr><td class=u>45 40</td> <td class=c>col01372 dbemail.bookoo </td> <td class=p></td></tr>
<tr><td class=u>39 34</td> <td class=>col01943 * </td> <td class=p></td></tr>
<tr><td class=u>33-31</td> <td>[shelf] </td> <td class=p></td></tr>
<tr><td class=u>30 25</td> <td class=c>2U </td> <td class=p></td></tr>
<tr><td class=u>24 19</td> <td style="background:#cecece">backup1 </td> <td class=p></td></tr>
<tr><td class=u>18 13</td> <td style="background:#8df595">jail17 * </td> <td class=p></td></tr>

</table>
</td>

</tr>
</table>
 
<table border=0 cellpadding=3>
<tr>

<td valign="top">
<table border=1>

<tr><th>U</th> <th style="background:#FFFF99">6-16 </th> <th>port</th></tr>
<tr><td class=u>129-127</td> <td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p13&path= c2924 P13] </td> <td class=p></td></tr>
<tr><td class=u>126 121</td> <td class=c>2U </td> <td class=p></td></tr>
<tr><td class=u>120-118</td> <td>1372 gig sw </td> <td class=p></td></tr>
<tr><td class=u>117 112</td> <td class=c>col01372 cache5 bookoo Dell 1U </td> <td class=p></td></tr>
<tr><td class=u>111 106</td> <td>col01372 solr5.bookoo </td> <td class=p></td></tr>
<tr><td class=u>105 100</td> <td>col01372 monitor.bookoo </td> <td class=p></td></tr>
<tr><td class=u>99 94</td> <td>col01372 cache 3</td> <td class=p></td></tr>
<tr><td class=u>93 88</td> <td>col01372 web3.bookoo </td> <td class=p></td></tr>
<tr><td class=u>87 82</td> <td>col01372 web1.bookoounderground.com</td> <td class=p></td></tr>
<tr><td class=u>81 76</td> <td class=c>col01372 cache6 bookoo Dell 1U </td> <td class=p></td></tr>
<tr><td class=u>75-73</td> <td style="text-align:center; background:#cecece">PS </td> <td class=p></td></tr>
<tr><td class=u>72-70</td> <td style="text-align:center; background:#cecece">PS </td> <td class=p></td></tr>
<tr><td class=u>69 64</td> <td>col01372 dbstats.bookoo </td> <td class=p></td></tr>
<tr><td class=u>63 58</td> <td>col01372 db1.bookoounderground OFF *</td> <td class=p></td></tr>
<tr><td class=u>57 52</td> <td>col01372 backup.clker </td> <td class=p></td></tr>
<tr><td class=u>51 46</td> <td>col01372 dbbackup.bookoo </td> <td class=p></td></tr>
<tr><td class=u>45 40</td> <td>col01372 dbpv.bookoo </td> <td class=p></td></tr>
<tr><td class=u>39 34</td> <td>col01372 web7.bookoo </td> <td class=p></td></tr>
<tr><td class=u>33 28</td> <td>col01372 dbm2.bookoo * </td> <td class=p></td></tr>
<tr><td class=u>27 22</td> <td>col01372 dbsense.bookoo </td> <td class=p></td></tr>

</table>
</td>

<td valign="top">
<table border=1>

<tr><th>U</th> <th style="background:#FFFF99">6-17 </th> <th>port</th></tr>
<tr><td class=u>129-127</td> <td>1U</td></tr>
<tr><td class=u>126-124</td> <td>col01372 gig sw </td> <td class=p></td></tr>
<tr><td class=u>123 118</td> <td class=c>2U </td> <td class=p></td></tr>
<tr><td class=u>117 112</td> <td>col01372 db1.clker </td> <td class=p>14</td></tr>
<tr><td class=u>111 106</td> <td>col01372 solr6.bookoo </td> <td class=p></td></tr>
<tr><td class=u>105 100</td> <td>col01372 solr4.bookoo </td> <td class=p>18</td></tr>
<tr><td class=u>99 94</td> <td>col01372 web5.bookoo </td> <td class=p>19</td></tr>
<tr><td class=u>93 88</td> <td>col01372 sso.bookoo </td> <td class=p>4</td></tr>
<tr><td class=u>87-85</td> <td>1U </td> <td class=p>9</td></tr>
<tr><td class=u>84-82</td> <td>col01372 balancer </td> <td class=p>3,4</td></tr>
<tr><td class=u>81-79</td> <td>1U </td> <td class=p></td></tr>
<tr><td class=u>78-76</td> <td style="text-align:center; background:#cecece">PS </td> <td class=p></td></tr>
<tr><td class=u>75-73</td> <td style="text-align:center; background:#cecece">PS </td> <td class=p></td></tr>
<tr><td class=u>72-70</td> <td style="text-align:center; background:#cecece">PS </td> <td class=p></td></tr>
<tr><td class=u>69 64</td> <td>col01372 solr2.bookoo </td> <td class=p></td></tr>
<tr><td class=u>63 58</td> <td>col01372 web6.bookoo </td> <td class=p></td></tr>
<tr><td class=u>57 52</td> <td>col01372 web8.bookoo </td> <td class=p></td></tr>
<tr><td class=u>51 46</td> <td>col01372 web2.bookoo </td> <td class=p></td></tr>
<tr><td class=u>45 40</td> <td>col01372 web1.bookoo </td> <td class=p></td></tr>
<tr><td class=u>39 34</td> <td>col01372 web4.bookoo </td> <td class=p></td></tr>
<tr><td class=u>33 28</td> <td>2U </td> <td class=p></td></tr>
<tr><td class=u>27 22</td> <td>col01372 dbm1.bookoo * </td> <td class=p></td></tr>
<tr><td class=u>21-19</td> <td>1U </td> <td class=p></td></tr>
<tr><td class=u>18 13</td> <td>col01372 web1.cliker * </td> <td class=p></td></tr>

</table>
</td>

</tr>
</table>

Private IP Mapping

2014-03-02T21:18:44Z

72.207.101.92: /* i2b */

= Castle =
<pre>
10.1.4.1 = nat
10.1.4.3 = backup2
10.1.4.4 = sun/console
10.1.4.5 = mail
10.1.4.8 = backup1
10.1.4.10 = digi1
10.1.4.11 = digi2?
10.1.4.59 = virt9
10.1.4.60-10.1.4.69 = virt10 - virt19
10.1.4.90-10.1.4.91 = tempvirt1 - tempvirt2
10.1.4.101-10.1.4.119 = jail1 - jail19
10.1.4.151 = quar1
10.1.4.160 = switch-p1a
10.1.4.161 = switch-p1b
10.1.4.162 =
10.1.4.163 = switch-p3
10.1.4.164 = switch-p4
10.1.4.165 =
10.1.4.166 = switch-p6
10.1.4.167 = switch-p7
10.1.4.169 = switch-p9
10.1.4.160 = switch-p10
10.1.4.161 = switch-p11
10.1.4.162 = switch-p12
10.1.4.163 = switch-p13
10.1.4.164 = switch-p14
10.1.4.165 = switch-p15
10.1.4.166 = switch-p16
10.1.4.167 = switch-p17
10.1.4.168 =
10.1.4.169 =
10.1.4.170 = 3550
10.1.4.171 = 3750
10.1.4.172 = ASA
10.1.4.201 = mx1
10.1.4.202 = mx2
10.1.4.203 = bwdb
10.1.4.211 = DRAC virt11
10.1.4.212 = DRAC virt12
10.1.4.213 = DRAC virt13
10.1.4.217 = DRAC virt17
10.1.4.209 = DRAC virt19
10.1.4.223 = gateway
10.1.4.232 = DRAC jail2
10.1.4.237 = DRAC jail7
10.1.4.238 = DRAC jail8
10.1.4.239 = DRAC jail9
</pre>

= i2b =
<pre>
10.1.2.1 = nat2
10.1.2.2 = firewall2
10.1.2.3 = backup3
10.1.2.4 = bwdb2
10.1.2.5 = firewall3
10.1.2.10 = digi3
10.1.2.50 = switch-p20
10.1.2.51 = switch-p21
10.1.2.52 = switch-p22
10.1.2.53 = switch-p23
10.1.2.54 = switch-p24
10.1.2.55 = switch-p25
10.1.2.56 = switch-p26
10.1.2.57 = switch-p27
10.1.2.58 = switch-p28
10.1.2.60 = switch-p30
10.1.2.71 = ats-1
10.1.2.72 = ats-2
10.1.2.73 = ats-3
10.1.2.74 = ats-4
10.1.2.75 = ats-5
10.1.2.76 = ats-6
10.1.2.77 = ats-7
10.1.2.78 = ats-8
10.1.2.79 = ats-9
10.1.2.103 = jail3
</pre>

= Office =
<pre>
10.1.6.1 = Motorola Gateway router
</pre>

Infrastructure Machines

2013-12-23T07:27:22Z

72.207.101.92: /* management scripts */

= jails =

== jail1 ==

* Location: castle, cab 3-8
* OS: FreeBSD 6.2 i386
* Networking: Priv IP: 10.1.4.101 (PCI nic), Pub IP: 69.55.230.107 (onboard)
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 74 GB (4 x 74GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.
* Remote management: none
* Disk accounting: gvinum

== jail2 ==

* Location: castle, cab 3-5
* OS: FreeBSD 7.2 amd64
* Networking: Priv IP: 10.1.4.102, Pub IP: 69.55.228.53 (2 onboard nics)
* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: one 146 GB (2 x 146GB) RAID1 array, two 300 GB (4 x 300GB) RAID1 arrays running on an LSI-based, Dell-branded (PERC 6/i) RAID card.
* Remote management: [[DRAC/RMM|DRAC]] @ 10.1.4.232
* Disk accounting: md

== jail3 ==
* Location: i2b, cab 6
* OS: FreeBSD 8.3 amd64
* Networking: Priv IP: 10.1.2.103, Pub IP: 69.55.229.7 (2 onboard nics)
* Hardware: Supermicro (custom build). 6 SATA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 300 GB (2 x 300GB) RAID1 array running on a 3ware 8006-2LP RAID card.
* Remote management: none
* Disk accounting: md

=== Notes ===
* '''We should not add users to this server since it is at I2B'''
* must be ssh'd to from nat2
* is a super jail for customer col01737

== jail4 ==

* Location: castle, cab 3-8
* OS: FreeBSD 9.1 x86_64
* Networking: Priv IP: 10.1.4.104, Pub IP: 69.55.228.104 (2 onboard nics)
* Hardware: Dell 2850. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: one 1.4 TB RAID 5 array Dell-branded (PERC 4e)LSI megarc RAID card.
* Remote management: None
* Disk accounting: md

=== Notes ===
Only FreeBSD 9.1 jail

== jail7 ==

* Location: castle, cab 3-7
* OS: FreeBSD 6.3 i386
* Networking: Priv IP: 10.1.4.107, Pub IP: 69.55.230.108 (2 onboard nics)
* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 146 GB (4 x 146GB) RAID1 arrays, one 74 GB (2 x 74GB) RAID1 array running on an LSI-based, Dell-branded (PERC 6/i) RAID card.
* Remote management: [[DRAC/RMM|DRAC]] @ 10.1.4.237
* Disk accounting: gvinum

=== Notes ===
Do not run a verify while OS/jails running, will crash.

== jail8 ==

* Location: castle, cab 3-6
* OS: FreeBSD 8.0 amd64
* Networking: Priv IP: 10.1.4.108, Pub IP: 69.55.234.2 (2 onboard nics)
* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: one 146 GB (2 x 146GB) RAID1 array, one 300 GB (2 x 300GB) RAID1 array, one 400 GB (2 x 400GB) RAID1 array, running on an LSI-based, Dell-branded (PERC 6/i) RAID card.
* Remote management: [[DRAC/RMM|DRAC]] @ 10.1.4.238
* Disk accounting: md

== jail9 ==

* Location: castle, cab 3-6
* OS: FreeBSD 8.2 amd64
* Networking: Priv IP: 10.1.4.109, Pub IP: 69.55.232.36 (2 onboard nics)
* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: one 146 GB (2 x 146GB) RAID1 array, one 400 GB (2 x 300GB) RAID1 array running on an LSI-based, Dell-branded (PERC 5/i) RAID card.
* Remote management: [[DRAC/RMM|DRAC]] @ 10.1.4.239
* Disk accounting: md

== jail11 ==

* Location: castle, cab 3-7
* OS: FreeBSD 4.7 i386
* Networking: Priv IP: 10.1.4.111 (PCI nic), Pub IP: 69.55.236.92 (onboard)
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 74 GB (4 x 74GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.
* Remote management: none
* Disk accounting: vinum

== mx1 ==

* Location: castle, cab 3-7
* OS: FreeBSD 4.11 i386
* Networking: Priv IP: 10.1.4.201 (PCI nic), Pub IP: 69.55.237.3 (onboard)
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: one 36 GB (2 x 36GB) RAID1 array, one 74 GB (2 x 74GB) RAID1 array running on an Adaptec-based, Dell-branded (perc) RAID card.
* Remote management: none
* Disk accounting: vinum

=== Notes ===
* is our (old) backup mail/dns vps service host

== mx2 ==

* Location: castle, cab 3-5
* OS: FreeBSD 7.1 i386
* Networking: Priv IP: 10.1.4.202 (PCI nic), Pub IP: 69.55.237.90 (onboard)
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 74 GB (4 x 74GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.
* Disk accounting: gvinum

=== Notes ===
* is our latest backup mail/dns vps service host

== jail17 ==
* Location: castle, cab 3-8
* OS: FreeBSD 4.10 i386
* Networking: Priv IP: 10.1.4.117 (PCI nic), Pub IP: 69.55.228.2 (onboard nics)
* Hardware: Supermicro (custom build). 6 SCA SCSI drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 74 GB (4 x 74GB) RAID1 arrays, one 146 GB (2 x 146GB) RAID1 array, running on an Adaptec 2120S RAID card.
* Remote management: none
* Disk accounting: vinum

== jail18 ==
* Location: castle, cab 3-7
* OS: FreeBSD 4.10 i386
* Networking: Priv IP: 10.1.4.117 (PCI nic), Pub IP: 69.55.228.2 (onboard nics)
* Hardware: Supermicro (custom build). 6 SCA SCSI drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 74 GB (4 x 74GB) RAID1 arrays, one 146 GB (2 x 146GB) RAID1 array, running on an Adaptec 2120S RAID card.
* Remote management: none
* Disk accounting: vinum

== jail19 ==
* Location: castle, cab 3-6
* OS: FreeBSD 6.1 i386
* Networking: Priv IP: 10.1.4.119 (PCI nic), Pub IP: 69.55.228.200 (onboard nics)
* Hardware: Supermicro (custom build). 6 SCA SCSI drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: one 74 GB (2 x 74GB) RAID1 array, one 146 GB (2 x 146GB) RAID1 array, running on an Adaptec 2120S RAID card.
* Remote management: none
* Disk accounting: gvinum

= virts =

== quar1 ==

* Location: castle, cab 3-8
* OS: RedHat 7.3 x86
* Networking: Priv IP: 10.1.4.151 (PCI nic), Pub IP: 69.55.227.2 (onboard nic)
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: one 36 GB (2 x 36GB) RAID1 array, one 74 GB (2 x 74GB) RAID1 array, running on an Adaptec-based, Dell-branded (perc) RAID card.
* Remote management: none
* Virtuozzo version: 2.6.1
* VZ license: hwid=23C0.C0E1.6FDD.08BA.8971.8E1C.EBD5.1EDC serial=0DE6.903E.E239.E23F.470C.4369.4104.A5A4

=== Notes ===
* used to be the home of customers who's VE's would just run out of control/badly
* has a max of 10 VE's allowed to run

== virt9 ==

* Location: castle, cab 3-7
* OS: RedHat 7.3 x86
* Networking: Priv IP: 10.1.4.59 (PCI nic), Pub IP: 69.55.226.161 (onboard nic)
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 74 GB (2 x 74GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.
* Remote management: none
* Virtuozzo version: 2.6.1
* VZ license: hwid=BC15.B4D6.0D25.A5FE.F3BA.D518.E351.AE3F serial=F6AD.B6B4.5650.8869.C97C.73EE.AF65.FA8B

== virt11 ==

* Location: castle, cab 3-6
* OS: CentOS 5.4 x86
* Networking: Priv IP: 10.1.4.61, Pub IP: 69.55.238.3, 2 onboard nics
* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: one 146 GB (2 x 146GB) RAID1 array, one 400 GB (2 x 400GB) RAID1 array, one 146 GB (2 x 146GB) RAID1 array, running on an LSI-based, Dell-branded (perc 6/i) RAID card.
* Remote management: [[DRAC/RMM|DRAC]] @ 10.1.4.211
* Virtuozzo version: 4.0.0
* VZ license: hwid="029D.A187.78E1.480F.49E3.E20A.7389.7F79" serial="163C.F3E2.195F.96B5.2D38.8937.9600.4A05"

== virt12 ==

* Location: castle, cab 3-6
* OS: CentOS 5.2 x86
* Networking: Priv IP: 10.1.4.62, Pub IP: 69.55.227.70, 2 onboard nics
* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 300 GB (2 x 300GB) RAID1 arrays one 400 GB (2 x 400GB) RAID1 array, running on an LSI-based, Dell-branded (perc 6/i) RAID card.
* Remote management: [[DRAC/RMM|DRAC]] @ 10.1.4.212
* Virtuozzo version: 4.0.0
* VZ license: hwid="0C53.A413.E095.B4F4.51BC.D740.6919.A77B" serial="84E5.9498.3759.E683.E24B.2514.CA72.DC31"

== virt13 ==

* Location: castle, cab 3-5
* OS: CentOS 6.2 x86_64
* Networking: Priv IP: 10.1.4.63, Pub IP: 69.55.226.2, 2 onboard nics
* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: one 146 GB (2 x 146GB) RAID1 array, one 600 GB (2 x 600GB) RAID1 array, running on an LSI-based, Dell-branded (perc 6/i) RAID card.
* Remote management: [[DRAC/RMM|DRAC]] @ 10.1.4.213
* Virtuozzo version: 4.7.0
* VZ license: hwid="7D07.93BE.0B1F.7D2B.B039.4B5B.48B6.453B" serial="60A4.A94C.44BB.DCD6.8D03.1778.605B.10FE"

=== Notes ===
* home to our latest/current signups
* currently the only 64bit vz host

== virt14 ==

* Location: castle, cab 3-5
* Switch Port: p3-13
* OS: CentOS 6.4 x86_64
* Networking: Priv IP: 10.1.4.64 Pub IP: 69.55.225.14 2 onboard nics
* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* CPU: 2 x Xeon 5140 Dual Core @ 2.33GHz
* RAM: 16 GB
* Drives: one 146 GB (2 x 146 GB SAS) RAID1 array, and one 1TB RAID1 array (2 x 1 TB SATA), running on an LSI-based, Dell-branded (perc 5/i) RAID card.
* Remote management: [[DRAC/RMM|DRAC]] @ 10.1.4.214
* Virtuozzo version: 4.7.0
* VZ license: hwid="" serial=""

=== Notes ===
* new host not yet in production
* Temp server to offload Virt13 till we can get a Cloud going.
* virt 13 and 14 currently the only 64bit vz hosts

== virt15 ==

* Location: castle, cab 3-5
* OS: RedHat 9 x86
* Networking: Priv IP: 10.1.4.65, Pub IP: 69.55.232.160 (2 onboard nics)
* Hardware: Supermicro (custom build). 6 SCSI SCA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: one 74 GB (2 x 74GB) RAID1 array, two 146 GB (2 x 146GB) RAID1 arrays, running on an LSI MegaRAID SCSI 320-1 RAID card.
* Remote management: none
* Virtuozzo version: 2.6.2
* VZ license: hwid=A90F.6F48.E723.D8BA.3025.184A.5B73.D11E serial=E94B.5164.C1E6.A67F.67D1.7D96.0B6C.5524

== virt16 ==

* Location: castle, cab 3-7
* OS: Fedora Core 4 x86
* Networking: Priv IP: 10.1.4.66, Pub IP: 69.55.232.2 (2 onboard nics)
* Hardware: Supermicro (custom build). 6 SCSI SCA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: one 74 GB (2 x 74GB) RAID1 array, one 146 GB (2 x 146GB) RAID1 array, running on an LSI MegaRAID SCSI 320-1 RAID card.
* Remote management: none
* Virtuozzo version: 3.0.0
* VZ license: hwid=DEFA.A325.7230.BBC8.9715.8B52.3FD7.27BE serial=66C0.41EA.3FBB.11D3.9CC6.55C7.09AE.14AB

== virt17 ==

* Location: castle, cab 3-6
* OS: CentOS 4.4 x86
* Networking: Priv IP: 10.1.4.67, Pub IP: 69.55.232.162, 2 onboard nics
* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 146 GB (2 x 146GB) RAID1 arrays running on an LSI-based, Dell-branded (perc 5/i) RAID card.
* Remote management: [[DRAC/RMM|DRAC]] @ 10.1.4.217
* Virtuozzo version: 3.0.0
* VZ license: hwid=2E14.AED9.70B8.C26E.D99F.B0D3.BCD2.229C serial=2A11.DAD0.61DB.E889.8DF4.9AF7.CF82.3C37

== virt19 ==

* Location: castle, cab 3-6
* OS: CentOS 5.2 x86
* Networking: Priv IP: 10.1.4.69, Pub IP: 69.55.236.2, 2 onboard nics
* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: one 146 GB (2 x 146GB) RAID1 array, one 300 GB (2 x 300GB) RAID1 array, running on an LSI-based, Dell-branded (perc 5/i) RAID card.
* Remote management: [[DRAC/RMM|DRAC]] @ 10.1.4.219
* Virtuozzo version: 3.0.0
* VZ license: hwid=3968.13F7.B2AC.8952.8E19.13A9.6EF5.5822 serial=061D.84CD.CCE5.B213.15B5.C061.D6A7.B034

= mail =
== Summary ==
This machine (mail) is the swiss army knife of the company, playing host to many services and functions.

* Location: castle, cab 3-8
* OS: FreeBSD 4.10 x86
* Networking: Priv IP: 10.1.4.5, Pub IPs: 69.55.230.2, 69.55.225.225 (ns1c jail), 69.55.230.9. 1 onboard and 1 PCI
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 36 GB (2 x 36GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.

== Services Provided ==
* mail
* web
* mysql
* bigbrother server/pager
* snmp
* named in a jail (ns1c)

== email ==
This server hosts mail for johncompanies.com (mail.johncompanies.com). Sendmail 8.13.6/8.13.6 is listening on 69.55.230.2 port 25 for incoming mail. Relaying is allowed per /etc/mail/relay-domains
Other addresses (aliases) are defined per /etc/mail/aliases

The following active users have mail hosted on this server:
* dave
* linux
* support
* payments
* sales
* tech1
* info

Traditionally, mail is checked via shell apps (pine). qpopper (pop3s) is running to allow mail downloading. Checking mail in this way causes an opened INBOX in pine to lock read-only. For this reason, we tee incoming mail to support and linux to tech1.

Procmail rules are setup to filter spam and send text messages. They are enabled for info, support, linux, tech1, dave and can be found in ~/Procmail/, for example:
<pre># more ~support/Procmail/rc.emergency
:0c # use c only if you want to forward a copy and file the original later
* ^Subject:.*\<emergency\>
* ! ^Subject:\<re\>
{
:0h
FROMANDSUBJECT=|formail -XFrom: -XSubject:

:0fwh
| /usr/local/bin/formail -I"Subject: " -I"To: pager@johncompanies.com" ; echo $FROMANDSUBJECT ; echo

:0
! -t
}

</pre>

control: <tt>cd /etc/mail; make stop</tt> (stop), <tt>cd /etc/mail; make start</tt> (start)

The following aliases are also in place:

<pre>debian: linux
jobs: info
careers: info
#reboot: 6128102202@txt.att.net
#reboot: 8582298897@vtext.com
reboot: pager
#pager: 8582298897@vtext.com
pager: 4158718324@txt.att.net
tech1on: "| /usr/local/sbin/tech1on.sh"
tech1off: "| /usr/local/sbin/tech1off.sh"</pre>

To change them, edit <tt>/etc/aliases</tt> and then run <tt>newaliases</tt>

Note on tech1: this address was setup as a read-only address to be mirrored on all email coming into support and linux. We set this up so we could easily check support mail via a pop client- popping email locks out the user in pine so checking support/linux directly via pop was not an option. When checking and responding to email that comes into tech1, care should be taken to make sure it is sent as/under an address other than tech1. This is cause tech1 is not monitored by support staff as closely as email to support/linux. Further, the tech on call may not be checking tech1. Lastly, because of the nature of the copying, you will sometimes notice certain automated email/notices are received 2x in support- this is because of/related to the tech1 mirror.

To enable it (on mail, run):

<tt>~support/tech1on.sh</tt>

To disable

<tt>~support/tech1off.sh</tt>

Or via email:

<tt>tech1on@johncompanies.com
tech1off@johncompanies.com</tt>

== web ==

See [[Management_System_/_Public_Website_/_Signup|Management System / Public Website / Signup]]

== mysql ==
mysql 4.1.22 is running on port 3306

* datadir: <tt>/mnt/data1/db/mysql/</tt>
* config: <tt>/etc/my.cnf</tt>
* database: <tt>jc</tt>
* control: <tt>/usr/local/etc/rc.d/mysql-server.sh stop</tt> (stop), <tt>/usr/local/etc/rc.d/mysql-server.sh start</tt> (start)

== bigbrother ==
There is a client running on mail (which monitors the services running on mail and mail itself), installed under <tt>/usr/home/bb/bbc1.9e-btf</tt> 
And the big brother pager/server (which displays information gathered from all bb-monitored machines, including mail) is installed under <tt>/usr/home/bb/bbsrc/bb1.9i-btf</tt>

Both are running under the user <tt>bb</tt>

Refer to [[BigBrother]] for more about use.

== DNS (ns1c.johncompanies.com) ==
ns1c is a jail running on the mail server, who's IP is 69.55.225.225

It's running from <tt>/mnt/data1/ns1c-dir</tt>

See [[DNS]] for more details

== Usage and Notes ==
* always mounted to backup1 and backup2 via nfs:
<pre>backup2:/mnt/data1 on /backup (nfs)
backup2:/mnt/data2 on /backup2 (nfs)
backup2:/mnt/data3 on /backup3 (nfs)
backup2:/mnt/data4 on /backup4 (nfs)
backup1:/data on /backup1 (nfs)
</pre>

== Cronjobs ==
* * * * * /usr/local/www/mgmt/mrtg/mrtg.sh > /dev/null 2>&1
Gathers up data for our mrtg/load graphs

*/5 * * * * /usr/local/bin/rsync -a root@nat2:/mnt/data1/mrtg/data/ /usr/local/www/mgmt/mrtg/data/
Gathers up data from i2b servers for our mrtg/load graphs

40 0 * * * /usr/local/bin/rsync -a root@nat2:"/mnt/data1/mrtg/*.cfg" /usr/local/www/mgmt/mrtg
Gathers up mrtg configuration (port names) from i2b switches for our mrtg/load graphs

41 0 * * * for f in `grep -l "mnt\/data1" /usr/local/www/mgmt/mrtg/switch-p*.cfg`; do cat $f | sed s#\/mnt\/data1#\/usr\/local\/www\/mgmt# > $f.new; mv $f.new $f; done
Gathers up mrtg configuration (port names) from castle switches for our mrtg/load graphs

1 0 1 * * cp /usr/local/www/mgmt/html/top20ip /usr/local/www/mgmt/html/top20ip_last
1 0 1 * * cp /usr/local/www/mgmt/html/top20customers /usr/local/www/mgmt/html/top20customers_last
2 * * * * /usr/local/www/cronjobs/top20ip.pl > /dev/null 2>&1
15 * * * * /usr/local/www/cronjobs/top20customer.pl > /dev/null 2>&1
1 0 1 * * rm /usr/local/www/mgmt/html/bandtrack
Archiving and generation of bandwidth statistics presented in mgmt -> Reference -> Bandwidth

1 0 * * * /usr/local/etc/rsync.backup
Nightly backup script

0 1 * * * /usr/local/www/mgmt/awstats/wwwroot/cgi-bin/awstats.pl -config=jcpub -update
Public web traffic stats

15 0 * * * rm /usr/local/www/mgmt/bwgraphs/*.png
16 0 * * * rm /usr/local/www/am/bwgraphs/*
Cleanup for graph-related temp data generated by customers using the bandwidth reports via the AM

10 0 1 * * /usr/local/www/cronjobs/monthly_bandwidth_report.pl
Monthly bandwidth overage report

*/3 * * * * /usr/local/www/cronjobs/bbcheck.pl
Updates mgmt with bb monitoring issues

5 0 * * * /usr/local/www/cronjobs/shutdownreminder.pl
Emails customers reminding them of upcoming shutdown date

7 0 * * * /usr/local/www/cronjobs/invoice_email.pl
Emails customers who have invoices and are set to auto-email (currently no customer gets these)

8 */4 * * * /usr/local/www/cronjobs/mysqlrepchk.pl
Checking that we are properly replicating (mysql) traffic data from bwdb to backup1

16 0 1 * * /usr/local/www/cronjobs/purge_traffic.pl
Removed old traffic data from the traffic database (running on backup1)

*/5 * * * * chmod 0700 /usr/local/www/ccard_orders/* && mv /usr/local/www/ccard_orders/* /usr/local/www/ccard_orders/done
Secure credit card data: set root-read-only

25 0 * * * /usr/local/www/cronjobs/biller.pl
Enters service charges in customer billing ledgers

10 13 * * * /usr/local/www/cronjobs/pfp_batch_gather.pl
Looks for customers with balance due and active credit card on file, prepares a payflow batch

10 14 * * * /usr/local/www/cronjobs/pfp_batch_process.pl
Tries to collect ccard funds for items in payflow batch - communicates with payflow

15 13 * * * /usr/local/www/cronjobs/pb_batch_gather.pl
Looks for customers with balance due and active paypal billing agreement on file, prepares a paypal batch

15 14 * * * /usr/local/www/cronjobs/pb_batch_process.pl
Tries to collect paypal funds for items in paypal batch - communicates with paypal

0 7 * * 1 /usr/local/www/cronjobs/email_pmt_reminder.pl
Emails customers in arrears, reminding them to pay

0 0 1 * * /usr/bin/mail -s 'archive sent mail in pine' support@johncompanies.com < /dev/null
Reminds us to archive sent mail

0 3 * * * /usr/local/bin/rsync -a isys.e-monitoring.net:/var/mail /backup2/isys; /usr/local/bin/rsync -a isys.e-monitoring.net:/usr/home /backup2/isys
Backup data on isys

== Regular maintenance ==
*[[Routine_Maintenance#Adaptec_Controllers|Check RAID array]]

= nat =
== Summary ==
This is the main machine to which we ssh and runs all our screen sessions. Further, it's ip runs in a special block which is not routed through the firewall and this is somewhat immune to DoS attacks which hobble our firewall. Lastly, it acts as a nat server for certain/random devices on the private network.

* Location: castle, cab 3-7
* OS: FreeBSD 9.1 i386
* Networking: Priv IP: 10.1.4.1, Pub IPs: 69.55.233.195, 69.55.233.196, 69.55.233.197, 69.55.233.198, 69.55.233.199. 1 onboard and 1 PCI
* Hardware: Custom 1U. single power supply.
* Drives: one 8 GB IDE drive

== Services Provided ==
* nat

== nat control ==
All rules are contained in and look like:

<pre>cat /etc/ipnat.rules
# www (was 69.55.230.12)
# virt19
#bimap fxp0 10.1.4.209/32 -> 69.55.233.198/32
# virt18
#bimap fxp0 10.1.4.208/32 -> 69.55.233.196/32
# virt13
#bimap fxp0 10.1.4.213/32 -> 69.55.233.196/32
# virt12
#bimap fxp0 10.1.4.212/32 -> 69.55.233.196/32
# virt17
bimap fxp0 10.1.4.217/32 -> 69.55.233.196/32
# virt11
#bimap fxp0 10.1.4.211/32 -> 69.55.233.196/32
# ASA
#bimap fxp0 10.1.4.172/32 -> 69.55.233.196/32
# P1A
bimap fxp0 10.1.4.240/32 -> 69.55.233.197/32
#bimap fxp0 10.1.4.238/32 -> 69.55.233.197/32
# developer (was 69.55.230.17)
# jail2
#bimap fxp0 10.1.4.232/32 -> 69.55.233.198/32
# jail8
#bimap fxp0 10.1.4.238/32 -> 69.55.233.198/32
# jail9
#bimap fxp0 10.1.4.239/32 -> 69.55.233.198/32
# POLL
#BIMAP EM0 10.1.6.134/32 -> 69.55.230.20/32
# 1U SUN
#BIMAP EM0 10.1.4.4/32 -> 69.55.227.46/32
# ??
#BIMAP EM0 10.1.6.3/32 -> 69.55.230.100/32
# random machine
#bimap fxp0 10.1.6.13/32 -> 69.55.233.199/32
#bimap fxp0 10.1.4.232/32 -> 69.55.233.199/32
# OFFICE OUTBOUND TRAFFIC
#map fxp0 10.1.6.0/24 -> 0.0.0.0/32 proxy port ftp ftp/tcp
#map fxp0 10.1.6.0/24 -> 0.0.0.0/32</pre>

A simple entry looks like:
bimap fxp0 10.1.4.240/32 -> 69.55.233.197/32
Which essentially means make private IP 10.1.4.240 reachable on 69.55.233.197 and allow 10.1.4.240 to communicate with the public internet via 69.55.233.197

To reload new rule config:
ipnat -C -F -f /etc/ipnat.rules

You may want to setup natting, as above, when you need to reach a DRAC card's web interface, wherin the DRAC card only has a private IP.

= nat2 =
== Summary ==
This is the main machine to which we ssh and runs all our screen sessions at i2b, and runs ns3c (this is kind of the what mail is to castle). Further, it's ip runs in IP space provided by i2b: 66.181.18.1 - 66.181.18.30, which is not routed through the firewall and this is somewhat immune to DoS attacks which hobble our firewall. Lastly, it acts as a nat server for certain/random devices on the private network.

* Location: i2b, cab 6
* OS: FreeBSD 6.4 x86
* Networking: Priv IP: 10.1.2.1, Pub IPs: 69.55.229.2, 69.55.229.3, 66.181.18.4, 66.181.18.5, 66.181.18.6, 66.181.18.7, 66.181.18.8, 66.181.18.9, 66.181.18.10, 66.181.18.11, 66.181.18.12, 66.181.18.13, 66.181.18.14 1 onboard and 1 PCI
* Hardware: Custom 2U. 6 drive bays, non-hot-swappable. single power supply.
* Drives: one 150 GB (2 x 150GB) RAID1 array running on a 3ware 8006 RAID card.

== Services Provided ==
* nat
* bigbrother
* ns3c (jail)
* ntp

== nat config ==
Here's what's currently nat'd on nat2:
<pre>cat /etc/ipnat.rules
# sample entry
#ATS-9
bimap em0 10.1.2.79/32 -> 66.181.18.14/32
#ATS-8
bimap em0 10.1.2.78/32 -> 66.181.18.13/32
#ATS-7
bimap em0 10.1.2.77/32 -> 66.181.18.12/32
#ATS-6
bimap em0 10.1.2.76/32 -> 66.181.18.6/32
#ATS-5
bimap em0 10.1.2.75/32 -> 66.181.18.7/32
#ATS-4
bimap em0 10.1.2.74/32 -> 66.181.18.8/32
#ATS-3
bimap em0 10.1.2.73/32 -> 66.181.18.9/32
#ATS-2
bimap em0 10.1.2.72/32 -> 66.181.18.10/32
#ATS-1
bimap em0 10.1.2.71/32 -> 66.181.18.11/32
#bwdb2
bimap em0 10.1.2.4/32 -> 66.181.18.5/32

# spare

map em0 10.1.2.0/24 -> 0.0.0.0/32 proxy port ftp ftp/tcp

#bimap fxp0 10.1.6.49/32 -> 10.1.1.2/32
#bimap fxp0 10.1.4.208/32 -> 69.55.233.196/32
</pre>

== build ==

* partition map:
<pre>/ 512m
swap 1G
/var 256m
/tmp 256m
/usr 5g
/mnt/data1 ~</pre>

* edit /etc/make.conf
<pre>echo "WITHOUT_X11=yes \
KERNCONF=nat2 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf</pre>

* add settings to /boot/loader.conf and /boot.config

<pre>echo "-Dh" >> /boot.config

echo 'console="comconsole,vidconsole" \
boot_multicons="YES" \
boot_serial="YES" \
comconsole_speed="115200"' >> /boot/loader.conf</pre>

* turn off all ttyv's except 0 and 1 in /etc/ttys
also turn on ttyd0, change type to vt100:
<pre>vi /etc/ttys

ttyv2 "/usr/libexec/getty Pc" cons25 off secure
ttyv3 "/usr/libexec/getty Pc" cons25 off secure
ttyv4 "/usr/libexec/getty Pc" cons25 off secure
ttyv5 "/usr/libexec/getty Pc" cons25 off secure
ttyv6 "/usr/libexec/getty Pc" cons25 off secure
ttyv7 "/usr/libexec/getty Pc" cons25 off secure
# Serial terminals
# The 'dialup' keyword identifies dialin lines to login, fingerd etc.
ttyd0 "/usr/libexec/getty std.9600" vt100 on secure

kill -1 1</pre>

on console server:
vi /etc/remote
(rename port to jail8 depending on where and which digi plugged into)
test serial console

* populate hosts
<pre>echo "69.55.230.10 backup2" >> /etc/hosts
echo "69.55.230.11 backup1" >> /etc/hosts
echo "10.1.2.4 bwdb2" >> /etc/hosts
echo "10.1.2.3 backup3" >> /etc/hosts</pre>

* put key in authorized_keys on backup3
<pre>cd
ssh-keygen -t dsa -b 1024</pre>
(default location, leave password blank)

<pre>cat /root/.ssh/id_dsa.pub | ssh backup3 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'</pre>

confirm that you can ssh to backup3 and backup 2 without getting a login prompt

ssh backup3 hostname

ssh backup2 hostname

ssh backup1 hostname

* edit root's path and login script:
vi /root/.cshrc

Change alias entries (add G):
<pre>alias la ls -aG
alias lf ls -FAG
alias ll ls -lAG
alias ls ls -AG
alias mbm mb mount
alias mbu mb umount</pre>

and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "

* install cvsup
<pre>cd /usr/ports/net/cvsup-without-gui
make install clean; rehash; mail -s 'cvs installed' dave.boodman@vtext.com < /dev/null</pre>

* get latest sources for this release:
<pre>cd /usr/src
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_4\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

cvsup sup ; mail -s 'cvs sup done' dave.boodman@vtext.com < /dev/null</pre>

* configure new kernel.

cd /usr/src/sys/i386/conf
scp backup2:/mnt/data4/build/freebsd/nat2-6.4 ./nat2

* build, install kernel and world

<pre>cd /boot

mv kernel kernel.GENERIC
cd kernel.GENERIC
cd /usr/src
make buildkernel installkernel

make buildworld ; mail -s 'buildworld done' support@johncompanies.com < /dev/null
make installworld
mergemaster -i
</pre>

* populate /etc/rc.conf with IPs and NFS settings
<pre>vi /etc/rc.conf

hostname="nat2.johncompanies.com"
kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"

xntpd_enable="YES"
xntpd_flags="-A -p /var/run/ntpd.pid"

nfs_client_enable="YES"
nfs_reserved_port_only="YES"
ifconfig_em0="inet 10.1.6.50 netmask 255.255.255.0"
#ifconfig_em0="inet 69.55.229.2 netmask 255.255.255.0"
#ifconfig_em0_alias0="inet 69.55.229.229 netmask 255.255.255.255"
ifconfig_fxp0="inet 69.55.229.2 netmask 255.255.255.0"
ifconfig_fxp0_alias0="inet 69.55.229.3 netmask 255.255.255.255"
ifconfig_fxp1="inet 10.1.2.1 netmask 255.255.255.0"
defaultrouter="10.1.6.1"
#defaultrouter=" 66.181.14.250"
snmpd_enable="YES"
ipnat_enable="YES"
ipnat_rules="/etc/ipnat.rules"
gateway_enable="YES"

inetd_enable="YES"
inetd_flags="-wW -a 10.1.2.1"
fsck_y_enable="YES"
background_fsck="NO"
sshd_enable="YES"</pre>

* reboot. Confirm new kernel is loaded

uname -a

* update ports:
<pre>cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_4\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup

cvsup sup; mail -s 'cvs sup ports done' dave.boodman@vtext.com < /dev/null</pre>

* Install raid mgmt tool

<pre>cd /usr/local/sbin
fetch http://3ware.com/download/Escalade9690SA-Series/9.5.3/tw_cli-freebsd-x86-9.5.3.tgz
tar xzf tw_cli-freebsd-x86-9.5.3.tgz
rm tw_cli-freebsd-x86-9.5.3.tgz
chmod 0700 tw_cli</pre>

Test:
./tw_cli info c0

* install rsync from ports
<pre>cd /usr/ports/net/rsync
make install clean</pre>
choose default options

* install perl from ports
<pre>cd /usr/ports/lang/perl5.8
make install clean</pre>

* install screen from ports
<pre>cd /usr/ports/sysutils/screen
make install clean</pre>

* install bb client
<pre>adduser
Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username : bb
Password : <random>
Full Name : bb
Uid : 1984
Class :
Groups : bb
Home : /home/bb
Shell : /bin/sh
Locked : no
OK? (yes/no): yes

cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd.tar .
tar xvf bb-freebsd.tar</pre>
edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
<pre>echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.2.1 nat2.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
vi /home/bb/bbc1.9e-btf/ext/openfiles
MACHINE="nat2,johncompanies,com" # HAS TO BE IN A,B,C FORM

cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src
make; make install
cd ..

vi /usr/home/bb/bbc1.9e-btf/bin/bb-cpu.sh
$1 $TOPARGS > $BBTMP/TOP.$$
# /usr/local/jail/bin/jtop > $BBTMP/TOP.$$

./runbb.sh start
more BBOUT
(look for errors)
exit

echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh
</pre>

Punch a hole in the firewall to allow it to communicate with bb monitor (probably already exists):

ipfw add 96 allow ip from 66.181.18.0/27 to 69.55.230.2

* configure bb on mail:
<pre>vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
64.163.14.48 nat2.johncompanies.com # ssh

su bb
cd
bbsrc/bb/runbb.sh restart ; exit</pre>

* configure ntp
<pre>echo "server 69.55.230.2
server 0.pool.ntp.org
server 1.pool.ntp.org
server 2.pool.ntp.org
server 3.pool.ntp.org
server 66.187.233.4
server 217.204.76.170
server 64.112.189.11
server 66.69.112.130
server 80.85.129.25
server 80.237.234.15
server 130.60.7.44
server 134.99.176.3
server 198.144.202.250
server 202.74.170.194
server 204.17.42.199
server 204.87.183.6
server 213.15.3.1
server 213.239.178.33
server 217.114.97.97
server 69.55.230.2" > /etc/ntp.conf</pre>

<pre>/usr/sbin/ntpd -A -p /var/run/ntpd.pid
sleep 2; ntpq -p</pre>
(confirm it’s able to reach our time server)

echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh
chmod 0700 /usr/local/etc/rc.d/ntp.sh

* fwd and reverse lookups on ns1c
vr johncompanies.com
(edit the PTR too)

* setup backups, nfs mount

<pre>mkdir /backup3
echo 'backup3:/data /backup3 nfs rw,bg 0 0' >> /etc/fstab

echo '#\!/bin/sh\
backupdir=/data/nat2/current\
\
## ENTRY /etc ' > /usr/local/etc/backup.config</pre>

on backup3:
setup backup dirs:
ssh backup3 mkdir -p /data/nat2/current

on backup3, add the system to
vi /usr/local/sbin/snapshot_archive

<pre>scp 69.55.230.2:/usr/local/etc/rsync.backup /usr/local/etc/rsync.backup
vi /usr/local/etc/rsync.backup
backup1 > backup3</pre>

crontab -e
1 0 * * * /usr/local/etc/rsync.backup

* edit sshd_config for security
<pre>vi /etc/ssh/sshd_config
ListenAddress 66.181.18.1
ListenAddress 69.55.229.2
ListenAddress 10.1.2.1

kill -1 `cat /var/run/sshd.pid`</pre>

* raid chk

<pre>cat > /usr/local/sbin/lsiraidchk
#!/usr/bin/perl

my @out = split "\n", `megarc -ldInfo -a0 -Lall|grep Status:`;

foreach (@out) {
if ($_ =~ /DEGRADED/) { $date = `date`; chomp $date; `echo "$date: RAID ARRAY DEGRADED" >> /var/log/messages`; }
#print $_;
}</pre>

* netflow stuff
add crontab entries
<pre>crontab -e
30 3 * * * /usr/local/etc/rsync.backup
0 0 1 * * /sbin/ipfw zero
0 0 1 * * /sbin/ipfw del 3 4 5
59 23 30 * * /sbin/ipfw show > /tmp/ipfw_count
3 0 30 * * /sbin/ipfw show > /tmp/ipfw_count
0 3 * * * /etc/makefwrules.pl; /etc/makepiperules.pl;
*/5 * * * * /usr/local/sbin/lsiraidchk

#10 0 * * * rm /var/spool/clientmqueue/*</pre>

<pre>scp /etc/makefwrules.pl user@64.163.14.48:~
scp /etc/makepiperules.pl user@64.163.14.48:~
mv /home/user/makefwrules.pl /etc
mv /home/user/makepiperules.pl /etc
touch /etc/firewall.sh
mkdir /etc/oldrules/</pre>

other binaries

<pre>scp /usr/local/bin/rulemaker user@64.163.14.48:~
mv ~user/rulemaker /usr/local/sbin
scp ~user/Sendmail.pm user@64.163.14.48:~
scp ~user/doswatch.pl user@64.163.14.48:~</pre>

* add nat rules
<pre>vi /etc/ipnat.rules
# sample entry
bimap fxp0 10.1.6.70/32 -> 10.1.6.59/32
#bimap fxp0 10.1.4.208/32 -> 69.55.233.196/32

ipnat -C -f /etc/ipnat.rules</pre>

* shell for user
<pre>cp /root/.cshrc ~user/
vi ~user/</pre>
change # to $

* mrtg

<pre>cd /usr/ports/net-mgmt/mrtg
make install clean</pre>
(no FONTCONFIG, v3)

this didn't work cause of libtool incompat

so manually moved files:

<pre>scp /usr/local/bin/cfgmaker user@nat2:/usr/local/bin/cfgmaker
scp /usr/local/lib/perl5/site_perl/5.6.1/MRTG_lib.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/
scp /usr/local/lib/perl5/site_perl/5.6.1/SNMP_util.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/
scp /usr/local/lib/perl5/site_perl/5.6.1/BER.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/
scp /usr/local/lib/perl5/site_perl/5.6.1/SNMP_Session.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/
scp /usr/local/bin/mrtg root@nat2:/usr/local/bin/mrtg
scp /usr/local/lib/perl5/site_perl/5.6.1/locales_mrtg.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/locales_mrtg.pm
scp /usr/local/bin/rrdtool root@nat2:/usr/local/bin/rrdtool
scp /usr/local/lib/perl5/site_perl/5.6.1/mach/RRDs.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/mach/RRDs.pm
rsync -av /usr/local/lib/perl5/site_perl/5.6.1/mach/auto/RRDs/ root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/mach/auto/RRDs/
scp /usr/lib/libz.so.2 root@nat2:/usr/lib/libz.so.2
scp /usr/lib/libm.so.2 root@nat2:/usr/lib/libm.so.2
rsync -av /usr/local/lib/librrd* root@nat2:/usr/local/lib/
scp /usr/lib/libc.so.4 root@nat2:/usr/lib/libc.so.4

rsync -av /usr/ports/net/rrdtool root@nat2:/usr/ports/net
cd /usr/ports/net/rrdtool
make install

mkdir -p /mnt/data1/mrtg/data
scp /usr/local/www/mgmt/mrtg/template.pl root@nat2:/mnt/data1/mrtg/
scp /usr/local/www/mgmt/mrtg/host.pl root@nat2:/mnt/data1/mrtg/

cfgmaker --if-template=template.pl --show-op-down --global "options[_]: growright,bits" --global 'WorkDir: /mnt/data1/mrtg/data' --global 'Interval: 1' --global 'LogFormat: rrdtool' --global 'PathAdd: /usr/local/bin' --global 'LibAdd: /usr/local/lib' --host-template=host.pl jc292401@10.1.2.50 --output=switch-p20.cfg

cat > /mnt/data1/mrtg/mrtg.sh
#!/bin/sh
/usr/local/bin/mrtg /mnt/data1/mrtg/switch-p20.cfg

chmod 0700 /mnt/data1/mrtg/mrtg.sh

crontab -e
* * * * * /mnt/data1/mrtg/mrtg.sh 2>&1 > /dev/null</pre>

* snmp firewall block
<pre>cat > /usr/local/etc/rc.d/boot.sh
ipfw add 10 allow udp from 69.55.230.2 to any 161
ipfw add 10 allow udp from 10.1.2.1 to any 161
ipfw add 11 deny udp from any to any 161
chmod 0700 /usr/local/etc/rc.d/boot.sh</pre>

= bwdb =
== Summary ==
This machine tracks and stores network traffic (netflow) at castle. It is our means to monitor customer bandwidth usage.

* Location: castle, cab 3-7
* OS: FreeBSD 4.10 x86
* Networking: Priv IP: 10.1.4.203 There are 2 onboard nic's, one of which is the "listener"
* Hardware: Custom 1U. Single power supply.
* Drives: one 250 GB (2 x 250GB) RAID1 array running on a Promise IDE RAID card.

== Services Provided ==
* netflow
* mysql
* bigbrother
* snmp

== netflow ==

The main function of this server is to run netflow on an eth device in promiscuous mode so as to hear everything happening on the port (wherein all network traffic is mirrored to that port via the cisco swith). Every 15min, it creates a flow file under <tt>/usr/home/flows/</tt> (organized by date). The flow file contains all traffic data for a 15min increment of time.

A cronjob moves that flow file (or files if there are multiple due to some delay)
1,16,31,46 * * * * /usr/home/flowbin/queue.pl

into a processing queue:
<tt>/usr/home/working</tt>

Then a separate file processes whatever flow files it finds there, inserting the data into the local mysql database:
2,17,32,47 * * * * /usr/home/flowbin/processflows.pl

== mysql ==

The database storing all the traffic data is named <tt>traffic</tt>
Tables:
<pre>mysql> show tables;
+---------------------------+
| Tables_in_traffic |
+---------------------------+
| dailyIpTotals_69_55_224 |
| dailyIpTotals_69_55_225 |
| dailyIpTotals_69_55_226 |
| dailyIpTotals_69_55_227 |
| dailyIpTotals_69_55_228 |
| dailyIpTotals_69_55_229 |
| dailyIpTotals_69_55_230 |
| dailyIpTotals_69_55_231 |
| dailyIpTotals_69_55_232 |
| dailyIpTotals_69_55_233 |
| dailyIpTotals_69_55_234 |
| dailyIpTotals_69_55_235 |
| dailyIpTotals_69_55_236 |
| dailyIpTotals_69_55_237 |
| dailyIpTotals_69_55_238 |
| dailyIpTotals_69_55_239 |
| dailyPortTotals_69_55_224 |
| dailyPortTotals_69_55_225 |
| dailyPortTotals_69_55_226 |
| dailyPortTotals_69_55_227 |
| dailyPortTotals_69_55_228 |
| dailyPortTotals_69_55_229 |
| dailyPortTotals_69_55_230 |
| dailyPortTotals_69_55_231 |
| dailyPortTotals_69_55_232 |
| dailyPortTotals_69_55_233 |
| dailyPortTotals_69_55_234 |
| dailyPortTotals_69_55_235 |
| dailyPortTotals_69_55_236 |
| dailyPortTotals_69_55_237 |
| dailyPortTotals_69_55_238 |
| dailyPortTotals_69_55_239 |
| ipTotals_69_55_224 |
| ipTotals_69_55_225 |
| ipTotals_69_55_226 |
| ipTotals_69_55_227 |
| ipTotals_69_55_228 |
| ipTotals_69_55_229 |
| ipTotals_69_55_230 |
| ipTotals_69_55_231 |
| ipTotals_69_55_232 |
| ipTotals_69_55_233 |
| ipTotals_69_55_234 |
| ipTotals_69_55_235 |
| ipTotals_69_55_236 |
| ipTotals_69_55_237 |
| ipTotals_69_55_238 |
| ipTotals_69_55_239 |
| portTotals_69_55_224 |
| portTotals_69_55_225 |
| portTotals_69_55_226 |
| portTotals_69_55_227 |
| portTotals_69_55_228 |
| portTotals_69_55_229 |
| portTotals_69_55_230 |
| portTotals_69_55_231 |
| portTotals_69_55_232 |
| portTotals_69_55_233 |
| portTotals_69_55_234 |
| portTotals_69_55_235 |
| portTotals_69_55_236 |
| portTotals_69_55_237 |
| portTotals_69_55_238 |
| portTotals_69_55_239 |
+---------------------------+
</pre>

So as you see we store each class-C block in its own table, for efficiency. Further, we store and organize data in 4 ways: "daily" tables and 15-minute granularity tables, and for each of those we track simple IP traffic and port-specific traffic. The daily tables contains 2 entries (one for each direction) for each IP for each day. For the current day, the row data is incremented as the day goes on.

<pre>mysql> describe dailyIpTotals_69_55_224;
+-----------+-------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-----------+-------------+------+-----+---------+-------+
| id | varchar(23) | | PRI | | |
| date | date | YES | | NULL | |
| ip | varchar(15) | YES | MUL | NULL | |
| direction | tinyint(1) | YES | | NULL | |
| octets | bigint(12) | YES | | NULL | |
| packets | int(11) | YES | | NULL | |
+-----------+-------------+------+-----+---------+-------+

mysql> select * from dailyIpTotals_69_55_224 limit 1\G
*************************** 1. row ***************************
id: 6955224194-20100917-1
date: 2010-09-17
ip: 69.55.224.194
direction: 1
octets: 8821
packets: 91
</pre>

The <tt>id</tt> is a unique identifier (key), <tt>direction</tt> indicates incoming or outgoing traffic (outbound = 2, inbound = 1), <tt>octets</tt> are the amount of traffic in kilobytes, and <tt>packets</tt> is the total number of packets.

The 15-minute table has similar information, but it's organized in 15 minute increments:

<pre>mysql> describe ipTotals_69_55_224;
+-----------+------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-----------+------------+------+-----+---------+-------+
| date | datetime | YES | | NULL | |
| ip | char(15) | YES | MUL | NULL | |
| direction | tinyint(1) | YES | | NULL | |
| octets | bigint(20) | YES | | NULL | |
| packets | int(11) | YES | | NULL | |
+-----------+------------+------+-----+---------+-------+

mysql> select * from ipTotals_69_55_224 limit 2\G
*************************** 1. row ***************************
date: 2010-01-11 19:30:00
ip: 69.55.224.13
direction: 1
octets: 288
packets: 6
*************************** 2. row ***************************
date: 2010-01-11 19:30:00
ip: 69.55.224.12
direction: 1
octets: 216
packets: 4</pre>

So for a given IP, there will be 192 rows in a given day: 4 rows per hour, *2 for 2 directions, *24 for 24hours in a day. Obviously this table is large which is why we broke it down into a daily table for quick, easy, daily-summary access.

That covers the simple traffic tabulation tables. We also track traffic by port:

<pre>mysql> describe dailyPortTotals_69_55_224;
+-----------+-------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-----------+-------------+------+-----+---------+-------+
| id | varchar(28) | | PRI | | |
| date | date | YES | | NULL | |
| ip | varchar(15) | YES | MUL | NULL | |
| direction | tinyint(1) | YES | | NULL | |
| protocol | smallint(3) | YES | | NULL | |
| port | int(11) | YES | | NULL | |
| octets | bigint(11) | YES | | NULL | |
| packets | int(11) | YES | | NULL | |
+-----------+-------------+------+-----+---------+-------+
8 rows in set (0.00 sec)

mysql> select * from dailyPortTotals_69_55_224 limit 1\G
*************************** 1. row ***************************
id: 695522496-20091218-1-6-23
date: 2009-12-18
ip: 69.55.224.96
direction: 1
protocol: 6
port: 23
octets: 1796
packets: 30

mysql> select * from portTotals_69_55_224 limit 1\G
*************************** 1. row ***************************
date: 2010-09-07 18:45:00
ip: 69.55.224.254
direction: 1
protocol: 6
port: 99999
octets: 144
packets: 3

</pre>

This is largely the same with 2 more additions: <tt>protocol</tt> (1=ICMP, 6=TCP, 17=UDP), and <tt>port</tt> which we set to 99999 if the traffic is return traffic and the port is above 1024. Obviously the potential for number of rows grows quickly when you consider the addition of port and protocol tracking per IP.

== Regular maintenance ==
*[[Routine_Maintenance#Adaptec_Controllers|Check RAID array]]
* archive data from database
archive_daily.pl 2012 09
This will archive data for the given year and month from the daily summary tables. Generally we want to have a year of history in the database.

archive_15min.pl 2012 09
This will archive data for the given year and month from the 15min-increment tables. Generally, we want to have 6 months of history in the database.

* if space becomes tight, move flow files and exported data to a backup server, both located in <tt>/usr/home/flowbin/archive</tt> and <tt>/usr/home/exported</tt>, respectively

== Slaving ==

If we were going to setup traffic database slaving (we don't do this anymore), perhaps cause the bwdb machine gets busy and it cannot handle traffic requests and netflow, here's how it's done:

On the traffic master:

GRANT REPLICATION SLAVE ON *.* TO 'repl'@'10.1.4.8' IDENTIFIED BY 'qERUG8wf';

in my.cnf:
<pre>bin-log
server-id=1
max_binlog_size=500M
expire_logs_days = 3</pre>

on slave:
in my.cnf:
<pre>server-id = 2
master-host = 10.1.4.203
master-user = repl
master-password = qERUG8wf
master-connect-retry=60
replicate-wild-do-table=traffic.daily%
max_relay_log_size=500M
expire_logs_days = 3

replicate-wild-do-table=traffic.%</pre>

on master:
touch /usr/home/working/.lock
(make sure processflows not running)

<pre>FLUSH TABLES WITH READ LOCK;
cd /usr/home/database/traffic
tar -czf mysql-traffic-snapshot.tgz ./daily*
(~1G)
SHOW MASTER STATUS;
+-----------------+-----------+--------------+------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+-----------------+-----------+--------------+------------------+
| bwdb-bin.000039 | 154432615 | | |
+-----------------+-----------+--------------+------------------+

(write down info)
UNLOCK TABLES;
scp mysql-traffic-snapshot.tgz 10.1.4.5:/mnt/data1/db/mysql/traffic/</pre>

on slave:
<pre>mkdir /mnt/data1/db/mysql/traffic
cd /mnt/data1/db/mysql/traffic/
tar xzvf mysql-traffic-snapshot.tgz
(restart mysql)
CHANGE MASTER TO MASTER_HOST='10.1.4.203',MASTER_USER='repl',MASTER_PASSWORD='qERUG8wf',MASTER_LOG_FILE='bwdb-bin.000059',MASTER_LOG_POS=482502186;
START SLAVE;</pre>

<pre>cd /usr/home/database/traffic
scp *</pre>

<pre>optimize table dailyPortTotals_69_55_224;
optimize table dailyPortTotals_69_55_225;
optimize table dailyPortTotals_69_55_226;
optimize table dailyPortTotals_69_55_227;
optimize table dailyPortTotals_69_55_228;
optimize table dailyPortTotals_69_55_229;
optimize table dailyPortTotals_69_55_230;
optimize table dailyPortTotals_69_55_231;
optimize table dailyPortTotals_69_55_232;
optimize table dailyPortTotals_69_55_233;
optimize table dailyPortTotals_69_55_234;
optimize table dailyPortTotals_69_55_235;
optimize table dailyPortTotals_69_55_236;
optimize table dailyPortTotals_69_55_237;
optimize table dailyPortTotals_69_55_238;
optimize table dailyPortTotals_69_55_239;</pre>

== Build ==

=== BIOS Config ===
disable quiet boot

set to last state after power loss

set date/time to GMT

enable serial console output (baud rate 115200)

=== Install OS ===

Install FreeBSD 8.3 amd64

* partition map:
<pre>/ 500m
swap 4096m
/var 256m
/tmp 256m
/usr ~</pre>

* edit /etc/make.conf
Castle:
<pre>echo "WITHOUT_X11=yes \
KERNCONF=bwdb \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf</pre>

i2b:
<pre>echo "WITHOUT_X11=yes \
KERNCONF=bwdb2 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf</pre>

* add settings to /boot/loader.conf and /boot.config

<pre>echo "-Dh" >> /boot.config

echo 'console="comconsole,vidconsole" \
boot_multicons="YES" \
boot_serial="YES" \
comconsole_speed="115200"' >> /boot/loader.conf</pre>

* turn off all ttyv's except 0 and 1 in /etc/ttys
also turn on ttyu0, change type to vt100:
<pre>vi /etc/ttys

ttyv2 "/usr/libexec/getty Pc" cons25 off secure
ttyv3 "/usr/libexec/getty Pc" cons25 off secure
ttyv4 "/usr/libexec/getty Pc" cons25 off secure
ttyv5 "/usr/libexec/getty Pc" cons25 off secure
ttyv6 "/usr/libexec/getty Pc" cons25 off secure
ttyv7 "/usr/libexec/getty Pc" cons25 off secure
# Serial terminals
# The 'dialup' keyword identifies dialin lines to login, fingerd etc.
ttyu0 "/usr/libexec/getty std.9600" vt100 on secure

kill -1 1</pre>

on console server:
vi /etc/remote
(rename port to jail8 depending on where and which digi plugged into)
test serial console

* populate hosts
i2b:
<pre>echo "69.55.230.10 backup2" >> /etc/hosts
echo "69.55.230.11 backup1" >> /etc/hosts
echo "10.1.2.3 backup3" >> /etc/hosts</pre>

castle:
<pre>echo "10.1.4.3 backup2 backup2.johncompanies.com" >> /etc/hosts
echo "10.1.4.8 backup1 backup1.johncompanies.com" >> /etc/hosts
echo "10.1.4.4 mail mail.johncompanies.com" >> /etc/hosts
</pre>

* put key in authorized_keys on backup1 and backup2
cd
ssh-keygen -t dsa -b 1024
(default location, leave password blank)

castle:
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'

i2b:
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup3 'cat - >> /root/.ssh/authorized_keys'

confirm that you can ssh to backup2 and backup1 (and backup3 if at i2b) without getting a login prompt

ssh backup1 hostname
ssh backup2 hostname

* edit root's path and login script:
vi /root/.cshrc

Change alias entries (add G):
<pre>alias la ls -aG
alias lf ls -FAG
alias ll ls -lAG
alias ls ls -AG
</pre>

and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "

* install cvsup
cd /usr/ports/net/cvsup-without-gui
make install clean; rehash; mail -s 'cvs installed' support@johncompanies.com < /dev/null

* get latest sources for this release:
<pre>cd /usr/src
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_8_3\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

cvsup sup ; mail -s 'cvs sup done' support@johncompanies.com < /dev/null</pre>

* configure new kernel

cd /usr/src/sys/amd64/conf
scp backup2:/mnt/data4/build/freebsd/kern_config-bwdb-8.3-amd64 ./bwdb

Edit config and change name:
vi bwdb
ident bwdb

* build, install kernel and world

<pre>cd /boot

mv kernel kernel.GENERIC
cd kernel.GENERIC
cd /usr/src
make buildkernel installkernel

make buildworld ; mail -s 'buildworld done' support@johncompanies.com < /dev/null
(2450: 1:56min, supermicro: 59mins, 2950: 38mins)
make installworld
(2450: 3min, supermicro: 1min, 2950: :34)
mergemaster -i</pre>

* populate /etc/rc.conf with IPs and NFS settings

castle:
<pre>vi /etc/rc.conf

hostname="bwdb.johncompanies.com"
kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
ifconfig_fxp0="inet 10.1.4.203 netmask 255.255.255.0"
ifconfig_em0="up promisc"
defaultrouter="10.1.4.1"
snmpd_enable="YES"

inetd_enable="YES"
inetd_flags="-wW -a 10.1.4.203"
fsck_y_enable="YES"
background_fsck="NO"
sshd_enable="YES"
ipfw_load="YES"</pre>

i2b:
<pre>vi /etc/rc.conf

hostname="bwdb2.johncompanies.com"
kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
ifconfig_fxp0="inet 10.1.2.4 netmask 255.255.255.0"
ifconfig_em0="up promisc"
defaultrouter="10.1.2.1"
snmpd_enable="YES"

inetd_enable="YES"
inetd_flags="-wW -a 10.1.2.4"
fsck_y_enable="YES"
background_fsck="NO"
sshd_enable="YES"
ipfw_load="YES"</pre>

* reboot. Confirm new kernel is loaded

uname -a

* update ports:
<pre>cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_8_3\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup

cvsup sup; mail -s 'cvs sup ports done' support@johncompanies.com < /dev/null</pre>

* Install raid mgmt tool

<pre>cd /usr/local/sbin
scp backup2:/d4/build/3ware/tw_cli-freebsd-x86_64-9.5.0.1.tgz .
tar xzf tw_cli-freebsd-x86_64-9.5.0.1.tgz
rm tw_cli-freebsd-x86_64-9.5.0.1.tgz
chmod 0700 tw_cli</pre>

Test:
./tw_cli info c0

Grab raid check script:
scp backup1:/usr/local/sbin/3wraidchk /usr/local/etc

Setup cronjob:
<pre>crontab -e
*/5 * * * * /usr/local/etc/3wraidchk</pre>

* install rsync from ports
cd /usr/ports/net/rsync
make install clean

choose default options

* install perl from ports
cd /usr/ports/lang/perl5.8
make install clean

choose default options

* install bb client

Compiling from source on AMD64 will not work. So, we use a linux-compiled version and rely on linux compat. Linux compat won't install on 8.x - libtool 2.4 need. So, instead we copy(ed) over linux:
rsync -aSHv --exclude=proc --exclude=sys 10.1.4.108:/usr/compat/linux/ /usr/compat/linux/

adduser

Output/response:

<pre>Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username : bb
Password : <random>
Full Name : bb
Uid : 1984
Class :
Groups : bb
Home : /home/bb
Shell : /bin/sh
Locked : no
OK? (yes/no): yes</pre>

cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd_linuxcompat.tgz .
tar xzf bb-freebsd_linuxcompat.tgz

edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:

echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.4.203 bwdb.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

Edit for machine name and private IP.

if this machine is at i2b:
echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.2.4 bwdb2.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

<pre>vi /home/bb/bbc1.9e-btf/ext/openfiles

MACHINE="bwdb,johncompanies,com" # HAS TO BE IN A,B,C FORM</pre>
Edit for machine name.

Have bb watch for flow-capture, mysql
<pre>cat >> /home/bb/bbc1.9e-btf/etc/bb-proctab
localhost: flow-capture :
localhost: mysqld :</pre>

<pre>cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf
./runbb.sh start
more BBOUT
(look for errors)
exit</pre>

Put in script to start bb @ boot:
echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh

If this is at i2b, punch a hole in the firewall to allow it to communicate with bb monitor:

ipfw add 00096 allow tcp from 66.181.18.0/27 to 69.55.230.2

* configure bb on mail
<pre>vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
10.1.4.203 bwdb.johncompanies.com # ssh

su bb
cd
bbsrc/bb/runbb.sh restart ; exit</pre>

* configure ntp server
Castle:
echo "server 10.1.4.1" > /etc/ntp.conf

I2b:
echo "server 10.1.2.1" > /etc/ntp.conf

<pre>/usr/sbin/ntpd -p /var/run/ntpd.pid
sleep 2; ntpq -p</pre>
(confirm it’s able to reach our time server)

<pre>echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh
chmod 0700 /usr/local/etc/rc.d/ntp.sh</pre>

* fwd and reverse lookups on ns1c
vr johncompanies.com
(edit the PTR too)

* setup backups
<pre>echo '#\!/bin/sh\
backupdir=/data/bwdb/current\
server=backup1\
\
## ENTRY /etc\
## ENTRY /usr/home/flowbin\
## ENTRY /usr/home/database' > /usr/local/etc/backup.config</pre>

Castle:
setup backup dirs:
ssh backup1 mkdir -p /data/bwdb/current
on backup1, add the system to
vi /usr/local/sbin/snapshot_rotate

I2b:
setup backup dirs:
ssh backup3 mkdir -p /data/bwdb/current
on backup3, add the system to
vi /usr/local/sbin/snapshot_archive

Copy over the backup script:
scp backup2:/d4/bin/freebsd8.x/rsync.backup /usr/local/etc/

Edit rsync.backup and change <tt>config</tt> var to point to correct config file location: <tt>/usr/local/etc/backup.config</tt>

<pre>crontab -e
5 0 * * * /usr/local/etc/rsync.backup</pre>

* make /root/logs
mkdir /root/logs

* edit sshd_config for security

<pre>vi /etc/ssh/sshd_config
ListenAddress 10.1.4.203
PermitRootLogin yes

kill -1 `cat /var/run/sshd.pid`</pre>

Edit for private IP.

* snmp

(Before doing this you may need to take down the firewall and also add to resolv.conf 69.43.143.41)
<pre>
cd /usr/ports/net-mgmt/net-snmp
make install clean
(defaults)

cat >> /etc/rc.conf
snmpd_enable="YES"
snmpd_flags="-a"
snmpd_conffile="/usr/local/share/snmp/snmpd.conf"
snmptrapd_enable="YES"
snmptrapd_flags="-a -p /var/run/snmptrapd.pid"

cat > /usr/local/share/snmp/snmpd.conf
rocommunity jcread 10.1.4.5
rocommunity jcread 10.1.4.202
</pre>

=== netflow ===

Install flow tools:
<pre>cd /usr/ports/net-mgmt/flow-tools
make install clean</pre>
Defaults.

mkdir /usr/home/flows

Flow start script:
echo "/usr/local/bin/flow-capture -w /usr/home/flows -S5 -N -2 0/10.1.4.203/4444" > /usr/local/etc/rc.d/flow-capture.sh

chmod 0700 /usr/local/etc/rc.d/flow-capture.sh

Edit for private IP.

Netgraph start script:
<pre>
cat > /usr/local/etc/rc.d/netgraph.sh

/usr/sbin/ngctl -f- <<-SEQ
mkpeer em0: netflow lower iface0
name em0:lower netflow
connect em0: netflow: upper out0
mkpeer netflow: ksocket export inet/dgram/udp
msg netflow:export connect inet/10.1.4.203:4444
SEQ

#/usr/sbin/ngctl -f- <<-SEQ
#shutdown netflow:
#SEQ

chmod 0700 /usr/local/etc/rc.d/netgraph.sh</pre>
Edit for private IP.

Confirm netflow is running after running scripts:
<pre>newbwdb /usr/ports/net-mgmt/flow-tools# /usr/sbin/ngctl
Available commands:
config get or set configuration of node at <path>
connect Connects hook <peerhook> of the node at <relpath> to <hook>
debug Get/set debugging verbosity level
dot Produce a GraphViz (.dot) of the entire netgraph.
help Show command summary or get more help on a specific command
list Show information about all nodes
mkpeer Create and connect a new node to the node at "path"
msg Send a netgraph control message to the node at "path"
name Assign name <name> to the node at <path>
read Read and execute commands from a file
rmhook Disconnect hook "hook" of the node at "path"
show Show information about the node at <path>
shutdown Shutdown the node at <path>
status Get human readable status information from the node at <path>
types Show information about all installed node types
write Send a data packet down the hook named by "hook".
quit Exit program
+ show netflow:
Name: netflow Type: netflow ID: 00000004 Num hooks: 3
Local hook Peer name Peer type Peer ID Peer hook
---------- --------- --------- ------- ---------
export <unnamed> ksocket 00000005 inet/dgram/udp
out0 em0 ether 00000001 upper
iface0 em0 ether 00000001 lower
+
</pre>

We notice that sometimes flow-capture is failing due to swap exhaustion (even after adding more swap). So we crontab flow-capture to restart (it's ok to start if it's already running, it just quits):

<pre>
crontab -e
#restart flow-capture
*/15 * * * * /usr/local/etc/rc.d/flow-capture.sh
</pre>

==== process flow tools ====

<pre>mkdir /usr/home/flowbin
mkdir /usr/home/working</pre>

Install modules:
<pre>cd /usr/ports/devel/p5-Date-Calc
make install clean
cd /usr/ports/mail/p5-Mail-Sendmail
make install clean</pre>

Queue script:
<pre>
cat > /usr/home/flowbin/queue.pl
#!/usr/bin/perl

use strict;

BEGIN {
push @INC, "/usr/home/flowbin";
}

use date;

my $flowbase = "/usr/home/flows";
#my $flowqueue = "/usr/home/queue";
my $flowqueue = "/usr/home/working";

my ($date, $time) = date::CurrentDateTime();

my $flowdir = mkFlowDir($date);
`mv $flowdir/ft-* $flowqueue`;

if (date::DateWindow($date, $time, $date, "00:00:00", 600)) {
my $newdate = date::AddDays($date, -1);
my $flowdir = mkFlowDir($newdate);
`mv $flowdir/ft-* $flowqueue`;
}

sub mkFlowDir {
my $date = shift;
$date =~ /([0-9]{4}-[0-9]{2})/;
my $yearmonth = $1;
return "$flowbase/$yearmonth/$date";
}
</pre>

Date.pm module:
<pre>
cat > /usr/home/flowbin/date.pm
#!/usr/local/bin/perl
#
# $Header: /usr/cvs/newgw/lib/date.pm,v 1.2 2003/11/24 17:06:02 glenn Exp $
#
# Copyright (c) 2001, 2002, 2003
# e-Monitoring Networks, Inc. All rights reserved.
#
#
#
# date.pl - Higher level functions written on top of Date::Calc

package date;

use strict;
use Date::Calc qw(:all);

sub DayDiff { #calculate the difference in days from two dates
my $date1 = shift;
my $date2 = shift;
my ($year1, $month1, $day1) = &DateToymd($date1);
my ($year2, $month2, $day2) = &DateToymd($date2);
my $diff = &Delta_Days($year1, $month1, $day1, $year2, $month2, $day2);
return $diff;
}

sub AddDays { #adds specified number of days to the supplied date
my $date = shift;
my $days = shift;
my ($year, $month, $day) = &DateToymd($date);
my ($nyear, $nmonth, $nday) = &Add_Delta_Days($year, $month, $day, $days);
my $ndate = &ymdToDate($nyear, $nmonth, $nday);
return $ndate;
}

sub AddHours { #adds specified number of hours to the supplied date and time
my $date = shift;
my $time = shift;
my $addhours = shift;
my $adddays = 0;
if (abs($addhours / 24) >= 1) {
$adddays = int($addhours / 24);
$addhours -= $adddays * 24;
}
my ($year, $month, $day) = &DateToymd($date);
my ($hour, $minute, $second) = &TimeTohms($time);
my ($ny, $nm, $nd, $nh, $nmin, $ns) = &Add_Delta_DHMS($year, $month, $day,
$hour, $minute, $second,
$adddays, $addhours, 0, 0);
my $ndate = &ymdToDate($ny, $nm, $nd);
my $ntime = &hmsToTime($nh, $nmin, $ns);
return $ndate, $ntime;
}

sub AddMinutes {
my $date = shift;
my $time = shift;
my $minutes = shift;
my ($year, $month, $day) = &DateToymd($date);
my ($hour, $minute, $second) = &TimeTohms($time);
my ($ny, $nm, $nd, $nh, $nmin, $ns) = &Add_Delta_DHMS($year, $month, $day,
$hour, $minute, $second,
0, 0, $minutes, 0);
my $ndate = &ymdToDate($ny, $nm, $nd);
my $ntime = &hmsToTime($nh, $nmin, $ns);
return $ndate, $ntime;
}

sub CurrentDateTime { #return the current date and time
my ($y, $m, $d, $h, $min, $s, $z, $z, $z) = &System_Clock;
my $date = &ymdToDate($y, $m, $d);
my $time = &hmsToTime($h, $min, $s);
return $date, $time;
}

sub Currentymd { #return the current year, month and day as separate variables
my ($y, $m, $d, $h, $min, $s, $z, $z, $z) = &System_Clock;
return $y, $m, $d;
}

sub DateToymd { #takes a date and returns year, month, day as individual values
my $date = shift;
if ($date =~ /([0-9]{4})-([0-9]{2})-([0-9]{2})/) {
my $day = $3;
my $month = $2;
my $year = $1;
return $year, $month, $day;
}
return undef;
}

sub TimeTohms { #takes a time and return hours minutes and seconds as individual values
my $time = shift;
if ($time =~ /([0-9]{1,2}):([0-9]{1,2}):([0-9]{1,2})/) {
my $hour = $1;
my $minute = $2;
my $second = $3;
if ($hour !~ /[0-9]{2}/) { $hour = "0$hour"; }
if ($minute !~ /[0-9]{2}/) { $minute = "0$minute"; }
if ($second !~ /[0-9]{2}/) { $second = "0$second"; }
return $hour, $minute, $second;
}
return undef;
}

sub ymdToDate { #takes year, month, day and assembles them into our date format
my $year = shift;
my $month = shift;
my $day = shift;
if (defined($year) && defined($month) && defined ($day)) {
$month = sprintf("%02d", $month);
$day = sprintf("%02d", $day);
return "$year-$month-$day";
}
return undef;
}

sub hmsToTime { #takes hour minute and second and assembles them into our time format
my $hour = shift;
my $minute = shift;
my $second = shift;
if (defined($hour) && defined($minute) && defined ($second)) {
if ($hour !~ /[0-9]{2}/) { $hour = "0$hour"; }
if ($minute !~ /[0-9]{2}/) { $minute = "0$minute"; }
if ($second !~ /[0-9]{2}/) { $second = "0$second"; }
return sprintf ("%02d:%02d:%02d", $hour, $minute, $second);
}
return undef;
}

sub CompareDates { #compares two date and time pairs
my $date1 = shift;
my $time1 = shift;
my $date2 = shift;
my $time2 = shift;

my ($year1, $month1, $day1) = &DateToymd($date1);
my ($hour1, $minute1, $second1) = &TimeTohms($time1);
my ($year2, $month2, $day2) = &DateToymd($date2);
my ($hour2, $minute2, $second2) = &TimeTohms($time2);

# &debug("$year1, $month1, $day1, $year2, $month2, $day2");
my $days = &Delta_Days($year1, $month1, $day1, $year2, $month2, $day2);
if ($days > 0) { return 1;}
if ($days < 0) { return -1;}
if ($days == 0) { #same day, compare times
my $seconds1 = $second1 + (60 * $minute1) + (3600 * $hour1);
my $seconds2 = $second2 + (60 * $minute2) + (3600 * $hour2);
if ($seconds1 < $seconds2) { return 1;}
if ($seconds1 > $seconds2) { return -1;}
if ($seconds1 == $seconds2) { return 0;}
}
return undef;
}

sub DateWindow { #compares two date time pairs to see if they are < X seconds apart
my $date1 = shift;
my $time1 = shift;
my $date2 = shift;
my $time2 = shift;
my $window = shift;

my ($year1, $month1, $day1) = &DateToymd($date1);
my ($hour1, $minute1, $second1) = &TimeTohms($time1);
my ($year2, $month2, $day2) = &DateToymd($date2);
my ($hour2, $minute2, $second2) = &TimeTohms($time2);

my ($day, $hour, $minute, $second) =
&Delta_DHMS($year1, $month1, $day1, $hour1, $minute1, $second1,
$year2, $month2, $day2, $hour2, $minute2, $second2);
$minute *= 60;
$hour *= 3600;
$day *= 86400;
my $total = $second + $minute + $hour + $day;
if (abs($total) < $window) {
return 1;
}
return 0;
}

sub CheckDateOrder { #takes three dates/times, returns true if they are in chronological order
my $date1 = shift;
my $time1 = shift;
my $date2 = shift;
my $time2 = shift;
my $date3 = shift;
my $time3 = shift;
if (&CompareDates($date1, $time1, $date2, $time2) == -1) {
return 0;
}
if (&CompareDates($date2, $time2, $date3, $time3) == -1) {
return 0;
}
return 1;
}

sub EpochSeconds { #calculates number of seconds since the epoch for the given date/time
my $date = shift;
my $time = shift;
my ($year, $month, $day) = &DateToymd($date);
my ($hour, $minute, $second) = &TimeTohms($time);
my ($d, $h, $m, $s) = &Delta_DHMS(1970, 1, 1, 0, 0, 0,
$year, $month, $day, $hour, $minute, $second);
my $seconds = $s + (60 * $m) + (3600 * $h) + (86400 * $d);
return $seconds;
}

sub SecondsToDateTime { #converts seconds since epoch to date/time
my $seconds = shift;
my $days = int($seconds / 86400);
$seconds -= $days * 86400;
my $hours = int($seconds / 3600);
$seconds -= $hours * 3600;
my $minutes = int($seconds / 60);
$seconds -= $minutes * 60;
my ($year, $month, $day, $hour, $minute, $second) =
&Add_Delta_DHMS(1970, 1, 1, 0, 0, 0, $days, $hours, $minutes, $seconds);
$month = sprintf("%02d", $month);
$day = sprintf("%02d", $day);
$hour = sprintf("%02d", $hour);
$minute = sprintf("%02d", $minute);
$second = sprintf("%02d", $second);
return "$year-$month-$day", "$hour:$minute:$second";
}

sub DateToDayName {
my $date = shift;
my ($year, $month, $day) = &DateToymd($date);
my $name = &Day_of_Week_to_Text(&Day_of_Week($year, $month, $day));
$name =~ /^[A-Za-z]{3}/;
$name = $&;
return $name;
}

sub ValiDate {
return @_;

}

sub CheckBusinessDay { # checks to see if date is business day. 1=yes, 0=no
my $date = shift;
my ($year, $month, $day) = &DateToymd($date);
if (Day_of_Week($year,$month,$day) < 6) { return 1; }
else { return 0; }
}

1; #don't remove this line
</pre>

chmod 0700 /usr/home/flowbin/queue.pl

Setup cronjob:
<pre>crontab -e
#move flow data into the queue
1,16,31,46 * * * * /usr/home/flowbin/queue.pl</pre>

==== flow processing: i2b ====
<pre>cat > /usr/home/flowbin/processflows-sql.pl
#!/usr/bin/perl

#use strict;
#$debug=1;
#$dry=1;

my $log = '/usr/home/flowbin/discards.log';

use Data::Dumper;

BEGIN {
push @INC, "/usr/home/flowbin";
}

#my $queuedir = "/usr/home/queue";
my $queuedir = "/usr/home/working";
my $archivedir = "/usr/home/archive";
my $sqldir = "/usr/home/sql";
my $sqldirworking = "/usr/home/sql/tmp";

unless ($dry) {
if (-e "$queuedir/.lock") {
open(FILE, "$queuedir/.lock");
my $pid = <FILE>;
chomp($pid);
close(FILE);
if (kill(0, $pid)) {
#another process is using the queue, bail out
exit(0);
}
else {
#dead lock file, remove it
`rm $queuedir/.lock`;
}
}
open(FILE, "> $queuedir/.lock");
print FILE "$$\n";
close(FILE);
}

opendir(DIR, $queuedir);
my @files = readdir(DIR);
closedir(DIR);

foreach my $file (sort @files) {
unless($file =~ /^\./) {
$file =~ /([0-9]{4}-[0-9]{2}-[0-9]{2})\.([0-9]{2})([0-9]{2})([0-9]{2})/;
my $date = "$1 $2:$3:$4";
my $outfile = "$1-$2:$3.sql";
unless (open (SQL, "+> $sqldirworking/$outfile")) { die "cant open $sqldirworking/$outfile"; }
my $condensedDate = $1;
$condensedDate =~ s/-//g;
my $iptotal = {};
my $protototal = {};
my $porttotal = {};

&debug("started file $file at ");
&debug(`date`);
&debug("getting raw flow data (flow-print)");
`cat $queuedir/$file | /usr/local/bin/flow-print -f 5 > /usr/home/working/tmp-$file`;
&debug("aggregating data at ");
&debug(`date`);
unless (open(DATA, "/usr/home/working/tmp-$file")) { die "can't open: $!"; }
LOOP: while (my $line = readline DATA) {
my @d = split /[\s]+/, $line;
if ($d[0] ne '' && $d[0] ne 'Start') {
my $addr = 0;
my $port = 0;

#Start End Sif SrcIPaddress SrcP DIf DstIPaddress DstP P Fl Pkts Octets
#0 1 2 3 4 5 6 7 8 9 10 11
#|
# outbound = 2, inbound = 1

my (@src_ip) = split '\.', $d[3];
my (@dst_ip) = split '\.', $d[6];

if ($src_ip[0] == 69 && $src_ip[1] == 55 && ($src_ip[2] == 229 || $src_ip[2] == 231)) { # for i2b
$d[2] = 2;
# hack for outbound bulk traffic counted 2x
#if ($src_ip[2] == 231) { $d[11] /= 2; $d[10] /= 2; }
}
# note- this is where we filter out IPs only found at i2b
elsif ($dst_ip[0] == 69 && $dst_ip[1] == 55 && ($dst_ip[2] == 229 || $dst_ip[2] == 231)) { # for i2b
$d[2] = 1;
}
else { next LOOP; }

if ($d[2] == 2) {
$addr = $d[3];
# if the dst-port is low, store that
if ($d[7] <= 1024) { $port = $d[7]; }
# if the src-port is low, store that
elsif ($d[4] <= 1024) { $port = $d[4]; }
else { $port = 99999; }
}
elsif ($d[2] == 1) {
$addr = $d[6];
# if the dst-port is high, assume its return traffic, try to store src-port if low
if ($d[7] > 1024) {
if ($d[4] <= 1024) { $port = $d[4]; }
else { $port = 99999; }
} else {
$port = $d[7];
}
} else {
next LOOP;
}

my (@ip) = split '\.', $addr;
unless ($ip[0] == 69) { next LOOP; }
unless ($ip[1] == 55) { next LOOP; }
unless ($ip[2] == 229 || $ip[2] == 231) { next LOOP; }

my $classC = "$ip[0]_$ip[1]_$ip[2]";

# IP dir
# if ($d[10] < 2147483647) { $iptotal->{$classC}->{$addr}->{$d[2]}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $iptotal->{$classC}->{$addr}->{$d[2]}->{'octetTotal'} += $d[11]; }
#
# if ($d[10] < 2147483647) { $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'octetTotal'} += $d[11]; }
#
# if ($d[10] < 2147483647) { $porttotal->{$classC}->{$addr}->{$d[2]}->{$port}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $porttotal->{$classC}->{$addr}->{$d[2]}->{$port}->{'octetTotal'} += $d[11]; }
$iptotal->{$classC}->{$addr}->{$d[2]}->{'pktTotal'} += $d[10];
$iptotal->{$classC}->{$addr}->{$d[2]}->{'octetTotal'} += $d[11];

$protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'pktTotal'} += $d[10];
$protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'octetTotal'} += $d[11];

$porttotal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{$port}->{'pktTotal'} += $d[10];
$porttotal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{$port}->{'octetTotal'} += $d[11];
}
}
close(DATA);
`rm /usr/home/working/tmp-$file`;
&debug("processing ip totals at ");
&debug(`date`);
foreach my $classC (keys(%{$iptotal})) {
my @values;
foreach my $ip (keys(%{$iptotal->{$classC}})) {
foreach my $dir (keys(%{$iptotal->{$classC}->{$ip}})) {
my $octets = $iptotal->{$classC}->{$ip}->{$dir}->{'octetTotal'};
my $packets = $iptotal->{$classC}->{$ip}->{$dir}->{'pktTotal'};
# $packets = $packets > 2147483647 ? 0 : $packets;
if ($octets > 2147483647) {
my $ddir = $dir==1 ? 'in' : 'out';
#print SQL "$date $ip $ddir $octets\n";
# $octets = 0;
}
# dailyIpTotals.id = ip(no .'s)-yyyymmdd-direction
my $id = "$ip-$condensedDate-$dir";
$id =~ s/\.//g;
push @values, "('$date', '$ip', $dir, $octets, $packets)";
my $sql = "insert into dailyIpTotals_$classC values ('$id', '$date', '$ip', $dir, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
print "$sql\n" if $dry;
print SQL "$sql;\n";
# $db->query("insert into ipTotals values ('$date', '$ip', $dir, $octets, $packets)");
}
}

# break inserts into 100 records at a time
&debug("inserting $#values +1 values");
while ($#values > 0) {
my $sql = "insert into ipTotals_$classC values ";
my $max_index = $#values > 100 ? 100 : $#values;
for (my $i=0; $i<=$max_index; $i++) {
$sql .= shift @values;
$sql .= ',';
}
chop $sql;
print "$sql\n" if $dry;
print SQL "$sql;\n";
}
}

# &debug("processing protocol totals at ");
# &debug(`date`);
# foreach my $classC (keys(%{$protototal})) {
# $db->query("lock tables dailyProtoTotals_$classC write") unless $dry;
# my @values;
# foreach my $ip (keys(%{$protototal->{$classC}})) {
# foreach my $dir (keys(%{$protototal->{$classC}->{$ip}})) {
# foreach my $proto (keys(%{$protototal->{$classC}->{$ip}->{$dir}})) {
# my $octets = $protototal->{$classC}->{$ip}->{$dir}->{$proto}->{'octetTotal'};
# my $packets = $protototal->{$classC}->{$ip}->{$dir}->{$proto}->{'pktTotal'};
# # $octets = $octets > 2147483647 ? 0 : $octets;
# # $packets = $packets > 2147483647 ? 0 : $packets;
# # dailyProtoTotals.id = ip(no .'s)-yyyymmdd-direction-proto
# my $id = "$ip-$condensedDate-$dir-$proto";
# $id =~ s/\.//g;
# push @values, "('$date', '$ip', $dir, $proto, $octets, $packets)";
# my $sql = "insert into dailyProtoTotals_$classC values ('$id', '$date', '$ip', $dir, $proto, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
# print "$sql\n" if $dry;
# $db->query($sql) unless $dry;
# # $db->query("insert into protoTotals values ('$date', '$ip', $dir, $proto, $octets, $packets)");
# }
# }
# }
# $db->query("unlock tables") unless $dry;
# my $sql = "insert into protoTotals_$classC values ";
# $sql .= join ',', @values;
# $db->query("lock tables protoTotals_$classC write") unless $dry;
# print "$sql\n" if $dry;
# $db->query($sql) unless $dry;
# $db->query("unlock tables") unless $dry;
# }

&debug("processing port totals at ");
&debug(`date`);
foreach my $classC (keys(%{$porttotal})) {
my @values;
foreach my $ip (keys(%{$porttotal->{$classC}})) {
foreach my $dir (keys(%{$porttotal->{$classC}->{$ip}})) {
foreach my $proto (keys(%{$porttotal->{$classC}->{$ip}->{$dir}})) {
foreach my $port (keys(%{$porttotal->{$classC}->{$ip}->{$dir}->{$proto}})) {
my $octets = $porttotal->{$classC}->{$ip}->{$dir}->{$proto}->{$port}->{'octetTotal'};
my $packets = $porttotal->{$classC}->{$ip}->{$dir}->{$proto}->{$port}->{'pktTotal'};
# $octets = $octets > 2147483647 ? 0 : $octets;
# $packets = $packets > 2147483647 ? 0 : $packets;

# dailyPortTotals.id = ip(no .'s)-yyyymmdd-direction-protocol-port
my $id = "$ip-$condensedDate-$dir-$proto-$port";
$id =~ s/\.//g;
push @values, "('$date', '$ip', $dir, $proto, $port, $octets, $packets)";
my $sql = "insert into dailyPortTotals_$classC values ('$id', '$date', '$ip', $dir, $proto, $port, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
print "$sql\n" if $dry;
print SQL "$sql;\n";

# $db->query("insert into portTotals values ('$date', '$ip', $dir, $port, $octets, $packets)");
}
}
}
}

# break inserts into 100 records at a time
&debug("inserting $#values +1 values");
while ($#values > 0) {
my $sql = "insert into portTotals_$classC values ";
my $max_index = $#values > 100 ? 100 : $#values;
for (my $i=0; $i<=$max_index; $i++) {
$sql .= shift @values;
$sql .= ',';
}
chop $sql;
print "$sql\n" if $dry;
print SQL "$sql;\n";
}
}

# 12 1 8 1 1= 23
# dailyIpTotals.id = ip(no .'s)-yyyymmdd-direction
# 12 1 8 1 1 3=26
# dailyProtoTotals.id = ip(no .'s)-yyyymmdd-direction-proto
# 12 1 8 1 1 5=28
# dailyPortTotals.id = ip(no .'s)-yyyymmdd-direction-port
#print "finished at ";
#print `date`;
`mv $queuedir/$file $archivedir` unless $dry;
close(SQL);
`bzip2 $sqldirworking/$outfile`;
`mv $sqldirworking/$outfile.bz2 $sqldir/`;
}
}
`rm $queuedir/.lock` unless $dry;

sub debug {
my $message = shift;
if ($debug) {
print "$message\n";
}
}

# var full during ft-v05.2005-03-28.084500-0800 and
# 2005-02-24 69.55.226

# all port/daily totals before 2005-04-07
</pre>

This script sends the sql files to the traffic server for processing:
<pre>cat > /usr/home/flowbin/sendsql.pl
#!/usr/bin/perl

#use strict;
#$debug=1;
#$dry=1;

my $remote = "69.55.233.199";
my $sqldir = "/usr/home/sql";
my $archive = "/usr/home/archive";
my $sqldirremote = "/data/bwdb2/pending/";
my @err;
unless ($dry) {
if (-e "$sqldir/.lock") {
open(FILE, "$sqldir/.lock");
my $pid = <FILE>;
chomp($pid);
close(FILE);
if (kill(0, $pid)) {
#another process is using the queue, bail out
exit(0);
}
else {
#dead lock file, remove it
`rm $sqldir/.lock`;
}
}
open(FILE, "> $sqldir/.lock");
print FILE "$$\n";
close(FILE);
}

opendir(DIR, $sqldir);
my @files = readdir(DIR);
closedir(DIR);

foreach my $file (sort @files) {
next unless $file =~ /bz2$/;

my $r = `scp -Cq $sqldir/$file $remote:$sqldirremote 2>&1`;
# print "scp $sqldir/$file $remote:$sqldirremote";
unless ($?==0) {
push @err, "scp -Cq $sqldir/$file $remote:$sqldirremote ($r)";
}
else {
`mv $sqldir/$file $archive`;
`ssh $remote mv $sqldirremote/$file $sqldirremote/${file}.done`;
}
}

`rm $sqldir/.lock` unless $dry;

if (@err) {
email_support('bwdb2: sendsql.pl error',join "\n", @err);
}

sub email_support {
my $subj=shift;
my $body=shift;
use Mail::Sendmail;

# prepare message
my %mail = (
To => 'support@johncompanies.com,dave@johncompanies.com',
From => 'support@johncompanies.com',
Subject => $subj,
Message => $body,
smtp => 'mail.johncompanies.com',
);
sendmail(%mail) || warn "Error: $Mail::Sendmail::error";
}

sub debug {
my $message = shift;
if ($debug) {
print "$message\n";
}
}

# var full during ft-v05.2005-03-28.084500-0800 and
# 2005-02-24 69.55.226

# all port/daily totals before 2005-04-07
</pre>

<pre>crontab -e
#process flows
2,17,32,47 * * * * /usr/home/flowbin/processflows-sql.pl
#move sql commands to traffic db
8,23,38,53 * * * * /usr/home/flowbin/sendsql.pl
</pre>

==== flow processing: castle ====
<pre>
cat > /usr/home/flowbin/processflows.pl

#!/usr/bin/perl

#use strict;
#$debug=1;
#$dry=1;

my $log = '/usr/home/flowbin/discards.log';

use Data::Dumper;

BEGIN {
push @INC, "/usr/home/flowbin";
}

use db;

#my $queuedir = "/usr/home/queue";
my $queuedir = "/usr/home/working";
my $archivedir = "/usr/home/archive";

unless ($dry) {
if (-e "$queuedir/.lock") {
open(FILE, "$queuedir/.lock");
my $pid = <FILE>;
chomp($pid);
close(FILE);
if (kill(0, $pid)) {
#another process is using the queue, bail out
exit(0);
}
else {
#dead lock file, remove it
`rm $queuedir/.lock`;
}
}
open(FILE, "> $queuedir/.lock");
print FILE "$$\n";
close(FILE);
}

my $db = db->new();
$db->connect('traffic', '', 'root', '5over3') || die $db->{'error'};

opendir(DIR, $queuedir);
my @files = readdir(DIR);
closedir(DIR);

foreach my $file (sort @files) {
unless($file =~ /^\./) {
$file =~ /([0-9]{4}-[0-9]{2}-[0-9]{2})\.([0-9]{2})([0-9]{2})([0-9]{2})/;
my $date = "$1 $2:$3:$4";
my $condensedDate = $1;
$condensedDate =~ s/-//g;
my $iptotal = {};
my $protototal = {};
my $porttotal = {};

&debug("started file $file at ");
&debug(`date`);
&debug("getting raw flow data (flow-print)");
`cat $queuedir/$file | /usr/local/bin/flow-print -f 5 > /usr/home/working/tmp-$file`;
&debug("aggregating data at ");
&debug(`date`);
unless (open(DATA, "/usr/home/working/tmp-$file")) { die "can't open: $!"; }
LOOP: while (my $line = readline DATA) {
my @d = split /[\s]+/, $line;
if ($d[0] ne '' && $d[0] ne 'Start') {
my $addr = 0;
my $port = 0;

#Start End Sif SrcIPaddress SrcP DIf DstIPaddress DstP P Fl Pkts Octets
#0 1 2 3 4 5 6 7 8 9 10 11
#|
# outbound = 2, inbound = 1

my (@src_ip) = split '\.', $d[3];
my (@dst_ip) = split '\.', $d[6];

if ($src_ip[0] == 69 && $src_ip[1] == 55 &&
$src_ip[2] >= 224 && $src_ip[2] <= 239 &&
$src_ip[2] != 229 && $src_ip[2] != 231) { # for castle
# if ($src_ip[0] == 69 && $src_ip[1] == 55 && $src_ip[2] == 229) { # for i2b
$d[2] = 2;
# hack for outbound bulk traffic counted 2x
if ($dst_ip[2] == 234) { $d[11] /= 2; $d[10] /= 2; }
}
elsif ($dst_ip[0] == 69 && $dst_ip[1] == 55 &&
$dst_ip[2] >= 224 && $dst_ip[2] <= 239 &&
$dst_ip[2] != 229 && $dst_ip[2] != 231) { # for castle
# elsif ($dst_ip[0] == 69 && $dst_ip[1] == 55 && $dst_ip[2] == 229) { # for i2b
$d[2] = 1;
}
else { next LOOP; }

if ($d[2] == 2) {
$addr = $d[3];
# if the dst-port is low, store that
if ($d[7] <= 1024) { $port = $d[7]; }
# if the src-port is low, store that
elsif ($d[4] <= 1024) { $port = $d[4]; }
else { $port = 99999; }
}
elsif ($d[2] == 1) {
$addr = $d[6];
# if the dst-port is high, assume its return traffic, try to store src-port if low
if ($d[7] > 1024) {
if ($d[4] <= 1024) { $port = $d[4]; }
else { $port = 99999; }
} else {
$port = $d[7];
}
} else {
next LOOP;
}

my (@ip) = split '\.', $addr;
unless ($ip[0] == 69) { next LOOP; }
unless ($ip[1] == 55) { next LOOP; }
unless ($ip[2] >= 224 && $ip[2] <= 239 && $ip[2] != 229 && $ip[2] != 231) { next LOOP; }
# unless ($ip[2] == 229) { next LOOP; }

my $classC = "$ip[0]_$ip[1]_$ip[2]";

# IP dir
# if ($d[10] < 2147483647) { $iptotal->{$classC}->{$addr}->{$d[2]}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $iptotal->{$classC}->{$addr}->{$d[2]}->{'octetTotal'} += $d[11]; }
#
# if ($d[10] < 2147483647) { $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'octetTotal'} += $d[11]; }
#
# if ($d[10] < 2147483647) { $porttotal->{$classC}->{$addr}->{$d[2]}->{$port}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $porttotal->{$classC}->{$addr}->{$d[2]}->{$port}->{'octetTotal'} += $d[11]; }
$iptotal->{$classC}->{$addr}->{$d[2]}->{'pktTotal'} += $d[10];
$iptotal->{$classC}->{$addr}->{$d[2]}->{'octetTotal'} += $d[11];

$protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'pktTotal'} += $d[10];
$protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'octetTotal'} += $d[11];

$porttotal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{$port}->{'pktTotal'} += $d[10];
$porttotal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{$port}->{'octetTotal'} += $d[11];
}
}
close(DATA);
`rm /usr/home/working/tmp-$file`;
&debug("processing ip totals at ");
&debug(`date`);
foreach my $classC (keys(%{$iptotal})) {
$db->query("lock tables dailyIpTotals_$classC write") unless $dry;
my @values;
foreach my $ip (keys(%{$iptotal->{$classC}})) {
foreach my $dir (keys(%{$iptotal->{$classC}->{$ip}})) {
my $octets = $iptotal->{$classC}->{$ip}->{$dir}->{'octetTotal'};
my $packets = $iptotal->{$classC}->{$ip}->{$dir}->{'pktTotal'};
# $packets = $packets > 2147483647 ? 0 : $packets;
if ($octets > 2147483647) {
my $ddir = $dir==1 ? 'in' : 'out';
`echo "$date $ip $ddir $octets\n" >> $log`;
# $octets = 0;
}
# dailyIpTotals.id = ip(no .'s)-yyyymmdd-direction
my $id = "$ip-$condensedDate-$dir";
$id =~ s/\.//g;
push @values, "('$date', '$ip', $dir, $octets, $packets)";
my $sql = "insert into dailyIpTotals_$classC values ('$id', '$date', '$ip', $dir, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
print "$sql\n" if $dry;
$db->query($sql) unless $dry;
# $db->query("insert into ipTotals values ('$date', '$ip', $dir, $octets, $packets)");
}
}
$db->query("unlock tables") unless $dry;

$db->query("lock tables ipTotals_$classC write") unless $dry;
# break inserts into 100 records at a time
&debug("inserting $#values +1 values");
while ($#values > 0) {
my $sql = "insert into ipTotals_$classC values ";
my $max_index = $#values > 100 ? 100 : $#values;
for (my $i=0; $i<=$max_index; $i++) {
$sql .= shift @values;
$sql .= ',';
}
chop $sql;
print "$sql\n" if $dry;
$db->query($sql) unless $dry;
}
$db->query("unlock tables") unless $dry;
}

sleep 20;
# &debug("processing protocol totals at ");
# &debug(`date`);
# foreach my $classC (keys(%{$protototal})) {
# $db->query("lock tables dailyProtoTotals_$classC write") unless $dry;
# my @values;
# foreach my $ip (keys(%{$protototal->{$classC}})) {
# foreach my $dir (keys(%{$protototal->{$classC}->{$ip}})) {
# foreach my $proto (keys(%{$protototal->{$classC}->{$ip}->{$dir}})) {
# my $octets = $protototal->{$classC}->{$ip}->{$dir}->{$proto}->{'octetTotal'};
# my $packets = $protototal->{$classC}->{$ip}->{$dir}->{$proto}->{'pktTotal'};
# # $octets = $octets > 2147483647 ? 0 : $octets;
# # $packets = $packets > 2147483647 ? 0 : $packets;
# # dailyProtoTotals.id = ip(no .'s)-yyyymmdd-direction-proto
# my $id = "$ip-$condensedDate-$dir-$proto";
# $id =~ s/\.//g;
# push @values, "('$date', '$ip', $dir, $proto, $octets, $packets)";
# my $sql = "insert into dailyProtoTotals_$classC values ('$id', '$date', '$ip', $dir, $proto, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
# print "$sql\n" if $dry;
# $db->query($sql) unless $dry;
# # $db->query("insert into protoTotals values ('$date', '$ip', $dir, $proto, $octets, $packets)");
# }
# }
# }
# $db->query("unlock tables") unless $dry;
# my $sql = "insert into protoTotals_$classC values ";
# $sql .= join ',', @values;
# $db->query("lock tables protoTotals_$classC write") unless $dry;
# print "$sql\n" if $dry;
# $db->query($sql) unless $dry;
# $db->query("unlock tables") unless $dry;
# }

&debug("processing port totals at ");
&debug(`date`);
foreach my $classC (keys(%{$porttotal})) {
$db->query("lock tables dailyPortTotals_$classC write") unless $dry;
my @values;
foreach my $ip (keys(%{$porttotal->{$classC}})) {
foreach my $dir (keys(%{$porttotal->{$classC}->{$ip}})) {
foreach my $proto (keys(%{$porttotal->{$classC}->{$ip}->{$dir}})) {
foreach my $port (keys(%{$porttotal->{$classC}->{$ip}->{$dir}->{$proto}})) {
my $octets = $porttotal->{$classC}->{$ip}->{$dir}->{$proto}->{$port}->{'octetTotal'};
my $packets = $porttotal->{$classC}->{$ip}->{$dir}->{$proto}->{$port}->{'pktTotal'};
# $octets = $octets > 2147483647 ? 0 : $octets;
# $packets = $packets > 2147483647 ? 0 : $packets;

# dailyPortTotals.id = ip(no .'s)-yyyymmdd-direction-protocol-port
my $id = "$ip-$condensedDate-$dir-$proto-$port";
$id =~ s/\.//g;
push @values, "('$date', '$ip', $dir, $proto, $port, $octets, $packets)";
my $sql = "insert into dailyPortTotals_$classC values ('$id', '$date', '$ip', $dir, $proto, $port, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
print "$sql\n" if $dry;
$db->query($sql) unless $dry;
# $db->query("insert into portTotals values ('$date', '$ip', $dir, $port, $octets, $packets)");
}
}
}
}
$db->query("unlock tables") unless $dry;

$db->query("lock tables portTotals_$classC write") unless $dry;
# break inserts into 100 records at a time
&debug("inserting $#values +1 values");
while ($#values > 0) {
my $sql = "insert into portTotals_$classC values ";
my $max_index = $#values > 100 ? 100 : $#values;
for (my $i=0; $i<=$max_index; $i++) {
$sql .= shift @values;
$sql .= ',';
}
chop $sql;
print "$sql\n" if $dry;
$db->query($sql) unless $dry;
}
$db->query("unlock tables") unless $dry;
sleep 10;
}

# 12 1 8 1 1= 23
# dailyIpTotals.id = ip(no .'s)-yyyymmdd-direction
# 12 1 8 1 1 3=26
# dailyProtoTotals.id = ip(no .'s)-yyyymmdd-direction-proto
# 12 1 8 1 1 5=28
# dailyPortTotals.id = ip(no .'s)-yyyymmdd-direction-port
#print "finished at ";
#print `date`;
`mv $queuedir/$file $archivedir` unless $dry;
}
}
`rm $queuedir/.lock` unless $dry;

sub debug {
my $message = shift;
if ($debug) {
print "$message\n";
}
}

# var full during ft-v05.2005-03-28.084500-0800 and
# 2005-02-24 69.55.226

# all port/daily totals before 2005-04-07
</pre>

<pre>
cat > /usr/home/flowbin/db.pm

#!/usr/bin/perl
#
# $Header: /usr/cvs/newgw/lib/db.pm,v 1.4 2003/06/05 18:20:01 glenn Exp $
#
# Copyright (c) 2003
# e-Monitoring Networks, Inc. All rights reserved.
#
#
package db;

use strict;
use DBI;

sub new {
my $class = shift;
my $self = {};

$self->{'debug'} = 0;
bless $self, $class;

return $self;
}

sub connect {
my $self = shift;
my $dbname = shift;
my $dbhost = shift;
my $dbuser = shift;
my $dbpass = shift;

my $host = '';
if (defined($dbhost)) {
$host = ";host=$dbhost";
}

eval {
$self->debug("connecting to: DBI:mysql:database=$dbname;$host", 1);
$self->{'dbh'} = DBI->connect("DBI:mysql:database=$dbname;$host", $dbuser, $dbpass);
};
if ($self->{'dbh'}) {
return 1;
}
$self->{'error'} = "Error connecting to database $@";
$self->debug("Error connecting to database $@");
return 0;
}

sub query {
my $self = shift;
my $query = shift;

$self->debug($query, 1);
my $sth;
eval {
$sth = $self->{'dbh'}->prepare($query);
};
unless ($sth) {
$self->{'error'} = "error preparing query $@";
$self->debug("error preparing query $@");
return undef;
}
my $qty;
eval {
$qty = $sth->execute;
};
unless ($qty) {
$self->{'error'} = "error executing query $@";
warn "error executing query $@ $query";
return undef;
}
$self->debug("returning $qty, $sth from query", 6);
return ($qty, $sth);
}

sub disconnect {
my $self = shift;

$self->{'dbh'}->disconnect;
return 0;
}

sub debug {
my $self = shift;
my $msg = shift;
my $level = shift || 0;

if ($level < $self->{'debug'}) {
print "$msg\n";
}
return 0;
}
1;
</pre>

mkdir /usr/home/archive
mkdir -p /usr/home/sql/tmp

<pre>crontab -e
#process flows
2,17,32,47 * * * * /usr/home/flowbin/processflows.pl</pre>

==== setup traffic db ====
* Install mysql:
<pre>cd /usr/ports/databases/mysql50-server
make install clean</pre>

cat >> /etc/rc.conf
mysql_enable="YES"

Move db data dir:
/usr/local/etc/rc.d/mysql-server stop
mkdir /usr/home/database/
mv /var/db/mysql/* /usr/home/database/
chown -R mysql:mysql /usr/home/database

Edit database location in startup script:
vi /usr/local/etc/rc.d/mysql-server
# : ${mysql_dbdir="/var/db/mysql"}
: ${mysql_dbdir="/usr/home/database"}

/usr/local/etc/rc.d/mysql-server start

* Install mysql perl database modules:
<pre>
cd /usr/ports/databases/p5-DBI
make install clean
cd /usr/ports/databases/p5-DBD-mysql50
make install clean
(no to SSL support)
</pre>

* Setting up database
<pre>
rehash
/usr/local/etc/rc.d/mysql-server start
mysql -u root
create database traffic;
grant all on *.* to root@localhost identified by '5over3';
grant all on traffic.* to jc@10.1.4.5 identified by '2gMKY3Wt';

</pre>

If this was a new server we'd setup new tables. See [[#mysql_2|mysql]] for how those tables would be setup.

We are assuming here we are moving data from an existing db, here's how that's done (from the current traffic db):
rsync -av --progress /usr/home/database/traffic/ 10.1.4.203:/usr/home/database/traffic/

When you're ready to do the cutover, shut down mysql on both hosts and do one last sync.

==== process flows from bwdb2 ====
On traffic database server (bwdb):

<pre>crontab -e
#import sql from bwdb2
10,25,40,55 * * * * /usr/home/flowbin/processsql.pl</pre>

Add access to mysql:
<pre>mysql -u root -p
grant all on traffic.* to bwdb2@localhost identified by 's1lver4d';
</pre>

<pre>cat > /usr/home/flowbin/processsql.pl

#!/usr/bin/perl

#use strict;
#$debug=1;
#$dry=1;

my $sqldir = "/usr/home/bwdb2/pending";
my $mysql = '/usr/local/bin/mysql';
my @err;
unless ($dry) {
if (-e "$sqldir/.lock") {
open(FILE, "$sqldir/.lock");
my $pid = <FILE>;
chomp($pid);
close(FILE);
if (kill(0, $pid)) {
#another process is using the queue, bail out
exit(0);
}
else {
#dead lock file, remove it
`rm $sqldir/.lock`;
}
}
open(FILE, "> $sqldir/.lock");
print FILE "$$\n";
close(FILE);
}

opendir(DIR, $sqldir);
my @files = readdir(DIR);
closedir(DIR);

foreach my $file (sort @files) {
next unless $file =~ /done$/;
my $r = `bzcat $sqldir/$file | $mysql -u bwdb2 -ps1lver4d traffic`;
unless ($?==0) {
push @err, "bzcat $sqldir/$file | $mysql -u bwdb2 -pxxxxx traffic ($r)";
}
else {
`rm $sqldir/$file`;
}
}

`rm $sqldir/.lock` unless $dry;

if (@err) {
email_support('bwdb: processsql.pl error',join "\n", @err);
}

sub email_support {
my $subj=shift;
my $body=shift;
use Mail::Sendmail;

# prepare message
my %mail = (
To => 'dave@johncompanies.com',
From => 'support@johncompanies.com',
Subject => $subj,
Message => $body,
smtp => 'mail.johncompanies.com',
);
sendmail(%mail) || warn "Error: $Mail::Sendmail::error";
}

sub debug {
my $message = shift;
if ($debug) {
print "$message\n";
}
}
</pre>

chmod 0700 /usr/home/flowbin/processsql.pl

Make sure bwdb is reachable from the outside only to bwdb2:

On nat, add to <tt>/etc/ipnat.rules</tt>
<pre># bwdb
bimap fxp0 10.1.4.203/32 -> 69.55.233.199/32</pre>

Reload:
ipnat -C -F -f /etc/ipnat.rules

Setup firewall rule on firewall:
ipfw add 00094 allow ip from 66.181.18.5 to 69.55.233.199 22
ipfw add 00094 deny ip from any to 69.55.233.199

Setup firewall on bwdb to restrict access now that it's nat'd:
<pre>
cat >> /usr/local/etc/rc.d/boot.sh
ipfw add 1 allow tcp from any to any established
ipfw add 2 allow ip from 10.1.4.0/24,66.181.18.5,69.55.233.195 to me 22
ipfw add 3 allow ip from 10.1.4.5 to me 3306
ipfw add 4 allow ip from 69.55.225.225 53 to me
ipfw add 5 allow ip from 69.55.230.2 25 to me
ipfw add 6 allow ip from me to me 4444
ipfw add 7 allow icmp from any to me
ipfw add 8 allow udp from 10.1.4.203 to 10.1.4.203 dst-port 4444
ipfw add 9 allow udp from 10.1.4.5 to me 161
ipfw add 100 deny ip from any to me
</pre>

chmod 0700 /usr/local/etc/rc.d/boot.sh

From bwdb2, add ssh key:
cat /root/.ssh/id_dsa.pub | ssh 69.55.233.199 'cat - >> /root/.ssh/authorized_keys'

Confirm no password access:
ssh 69.55.233.199 hostname

= bwdb2 =
== Summary ==
This machine tracks and stores network traffic (netflow) at i2b. It is our means to monitor customer bandwidth usage.

* Location: i2b, cab6
* OS: FreeBSD 6.4 x86
* Networking: Priv IP: 10.1.2.4 There are 2 onboard nic's, one of which is the "listener"
* Hardware: Custom 2U. Single power supply.
* Drives: two 150 GB (2 x 150GB) RAID1 arrays running on a 3ware 7006 RAID card.

== Services Provided ==
* netflow
* bigbrother

== netflow ==

The main function of this server is to run netflow on an eth device in promiscuous mode so as to hear everything happening on the port (wherein all network traffic is mirrored to that port via the cisco swith). Every 15min, it creates a flow file under <tt>/usr/home/flows/</tt> (organized by date). The flow file contains all traffic data for a 15min increment of time.

A cronjob moves that flow file (or files if there are multiple due to some delay)
1,16,31,46 * * * * /usr/home/flowbin/queue.pl

into a processing queue:
<tt>/usr/home/working</tt>

Then a separate file processes whatever flow files it finds there, and builds sql files ready for insertion into the traffic database:
2,17,32,47 * * * * /usr/home/flowbin/processflows-sql.pl

Then yet another process copies the sql files to the traffic database server for processing and insertion into the mysql database:
8,23,38,53 * * * * /usr/home/flowbin/sendsql.pl

== Regular maintenance ==
*[[Routine_Maintenance#Adaptec_Controllers|Check RAID array]]

* if space becomes tight, move sql files and flow files to backup server, both located in <tt>/usr/home/flowbin/archive</tt>

= firewall (newgateway) =

== Summary ==

This machine is the primary (only) firewall for the entire network at castle.

* Location: castle, cab 3-8
* OS: FreeBSD 4.11 x86
* Networking: Priv IP: 10.1.4.223, Pub IPs: 69.55.233.164 (external), 69.55.233.156 (internal). It has 3 network connections (2 onboard, 1 PCI) connecting to the external, internal and private networks. If you're looking at the back of the server, the internal-network-facing nic is on the right (em1), and the external-facing-network (3750) is on the left (em0).
* Hardware: 6 SCSI SCA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: 36 GB (2 x 36GB) RAID1 array running on an Adaptec 2120S PCI RAID card.

== Services Provided ==
* firewall (ipfw)
* snmp
* bigbrother

== Firewall Rule Configuration ==

See [[FreeBSD_Reference#Firewall_Rule_Configuration|Firewall Rule Configuration]] for more discussion on how to actually manipulate firewall rules.

== Disaster Recovery ==

If there is ever an outage with the firewall, the old firewall "gate" is located just below and is running with the proper network configuration, but with no firewall rules in place (to facilitate good throughput). Have castle move the cable on the left on the current firewall to the left port in the old firewall and the right cable to the right port.

Here's what you need to put in /etc/rc.conf to get a firewall going (as far as routes and IPs)

<pre>hostname="newgateway.johncompanies.com"
firewall_script="/etc/firewall.sh"
firewall_enable="NO"
sendmail_enable="NONE"
sshd_enable="YES"
inetd_enable="NO"
xntpd_enable="YES"
snmpd_enable="YES"
#snmpd_flags="-as -p /var/run/snmpd.pid"
#ipnat_enable="YES"
#ipnat_rules="/etc/ipnat.rules"
gateway_enable="YES"

defaultrouter="69.55.233.161"

ifconfig_xl0="inet 10.1.4.223 netmask 255.255.255.0"
ifconfig_em0="inet 69.55.233.164 netmask 255.255.255.248"

#
# Original JohnCompanies 69.55.224.0/20
#
ifconfig_em1="inet 69.55.233.156 netmask 255.255.255.248"

static_routes="route1 route2 route3 route4 route5 route6 route7 route8 route9 route10 route11 route1
2 route13 route14 route15 route16 route17 route18"

route_route1="-net 69.55.224.0 69.55.233.153"
route_route2="-net 69.55.225.0 69.55.233.153"
route_route3="-net 69.55.226.0 69.55.233.153"
route_route4="-net 69.55.227.0 69.55.233.153"
route_route5="-net 69.55.228.0 69.55.233.153"
route_route6="-net 69.55.229.0 69.55.233.153"
route_route7="-net 69.55.230.0 69.55.233.153"
route_route8="-net 69.55.231.0 69.55.233.153"
route_route9="-net 69.55.232.0 69.55.233.153"
route_route10="-net 69.55.233.0 69.55.233.153"
route_route11="-net 69.55.234.0 69.55.233.153"
route_route12="-net 69.55.235.0 69.55.233.153"
route_route13="-net 69.55.236.0 69.55.233.153"
route_route14="-net 69.55.237.0 69.55.233.153"
route_route15="-net 69.55.238.0 69.55.233.153"
route_route16="-net 69.55.239.0 69.55.233.153"
route_route17="-net 10.1.5.0 10.1.4.2"
route_route18="-net 10.1.6.0 10.1.4.2"

#In case of 3750 failure:
#defaultrouter="69.43.128.81"
#ifconfig_em0="inet 69.43.129.84 netmask 255.255.255.248"

#bind .1's here:
#ifconfig_em1="inet 69.55.224.1 netmask 255.255.255.0"
#ifconfig_em1_alias0="inet 69.55.225.1 netmask 255.255.255.0"
#ifconfig_em1_alias1="inet 69.55.226.1 netmask 255.255.255.0"
#ifconfig_em1_alias2="inet 69.55.227.1 netmask 255.255.255.0"
#ifconfig_em1_alias3="inet 69.55.228.1 netmask 255.255.255.0"
#ifconfig_em1_alias4="inet 69.55.229.1 netmask 255.255.255.0"
#ifconfig_em1_alias5="inet 69.55.230.1 netmask 255.255.255.0"
#ifconfig_em1_alias6="inet 69.55.231.1 netmask 255.255.255.0"
#ifconfig_em1_alias7="inet 69.55.232.1 netmask 255.255.255.0"
#ifconfig_em1_alias8="inet 69.55.233.1 netmask 255.255.255.0"
#ifconfig_em1_alias9="inet 69.55.234.1 netmask 255.255.255.0"
#ifconfig_em1_alias10="inet 69.55.235.1 netmask 255.255.255.0"
#ifconfig_em1_alias11="inet 69.55.236.1 netmask 255.255.255.0"
#ifconfig_em1_alias12="inet 69.55.237.1 netmask 255.255.255.0"
#ifconfig_em1_alias13="inet 69.55.238.1 netmask 255.255.255.0"
#ifconfig_em1_alias14="inet 69.55.239.1 netmask 255.255.255.0"

#bulk:
# reassign 69.55.231.1 to the int iface on the firewall
# set the DG on the firewall to 69.43.138.9
# set the ext firewall IP to 69.43.138.12, NM: 255.255.255.248</pre>

== Cronjobs ==
1 0 * * * /usr/local/etc/rsync.backup
Backup to backup1

0 0 1 * * /sbin/ipfw zero
0 0 1 * * /sbin/ipfw del 3 4 5 17331
Reset counters and remove pipe rules on the 1st of the month. Pay attention when setting up a rule as 3 4 5 (that's not a temporary traffic cap).

Inside <tt>/etc/daily.local</tt> you will see a call to <tt>/etc/makepiperules.pl</tt>
This script will create <tt>/etc/firewall.sh</tt> which contains all the firewall and pipe rules in place at the time the script was run.

== DOS attacks ==

See [[FreeBSD_Reference#Handling_a_DoS_attack|Handling a DoS attack]] regarding how to handle a DOS attack.

Theres a background process (running from user shell) that monitors the firewall for incoming UDP DoS attacks. When it notices packets above a certain level it will
# enter a rule that allows all UDP to go through
# send an emergency email to support and indicating an attack is in progress
# send an email to castle (nocstaff@castleaccess.com and jcsupport@castleaccess.com) telling them to investigate and put up a null if warranted
# wait for a couple minutes to see if the attack subsides- if so it will remove the pass-all UDP rule, if not it will repeat the process from #1
This file lives under /usr/home/user/doswatch.pl
To run:
cd /usr/home/user
./doswatch.pl &

To kill;
fg
^C

It writes its findings to /usr/home/user/doswatch.log

= backup1 =

== Summary ==

This machine acts as the primary backup location for all VPS-based customers. No customer directly accesses this server to perform their backups. We also store cancelled customers on this server.

* Location: castle, cab 3-8
* OS: Ubuntu 8.04.1 server x86
* Networking: Priv IP: 10.1.4.8, Pub IP: 69.55.230.11 (firewalled from all but JC infrastructure @ i2b)
* Hardware: 6 SATA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Single power supply.
* Drives: 4.5 TB (6 x 1TB) RAID5 array running on a 3ware 9650SE-8LPML (8-port) card

== Services provided ==
* backup via rsync
* mysql - traffic data
* nfs server - for backups
* snmp client - for big brother
* bigbrother client

== Usage and Notes ==
* all data is stored under /data
* virtually all jc infrastructure, and all VPS machines are setup to mount to backup1 via nfs (mountpoint: <tt>/backup1</tt>), and they all have their ssh keys setup to allow passwordless rsync's
* each virt or jail backs up each evening to backup1. Each server has it's own directory (named for the server). Under those directories are 7 daily snapshots (0-6)
* at the time of writing, the mysql server running here is replicating from (slave to) the mysql instance on bwdb. Requests for bandwidth data usage for customers (coming from management, account manager, and accounting scripts running on mail) all direct towards the database "traffic" running on this server.
* cancelled customer systems are compressed and stored under <tt>/data/deprecated</tt>
* archived bwdb2 flow files are stored under <tt>/data/bwdb2</tt>
* critical files from backup2 are stored under <tt>/data/backup2</tt>

== Cronjobs ==
<pre>
00 5 * * * /usr/local/sbin/backupwatch.pl 2>&1 > /dev/null
35 5 * * * /usr/local/sbin/usage_check; /usr/local/sbin/snapshot_archive; /usr/local/sbin/snapshot_rotate /data/backuplog.log</pre>
this runs daily the scripts to report on how much disk space each customer system occupies and how long their backups took. Then it rotates backups for each system, removing the oldest backup. It will email support@johncompanies.com at it’s conclusion. This email can be deleted, however note when it begins to take significantly longer to complete, ie runs past 2200 pm – this usually indicates a problem on the backup server.

<pre>10,25,40,55 * * * * /usr/local/sbin/processsql.pl
</pre>
this processes prepared sql command files sent from/by bwdb2 (@ i2b) and imports them into the traffic database.
<pre>0 0 * * * /usr/local/sbin/3wraidchk
</pre>
checks the health of the RAID array

== Regular maintenance ==
*[[Routine_Maintenance#Free_up_space_on_backup1|Remove old backups]]
*[[Routine_Maintenance#3ware|Check on auto-verify]]

== build ==

<pre>Setup raid5 with a boot vol of 12G 5.45tb
12G boot
4664 GB

Install ubuntu 8.04

Swap 4G

Don’t format data drive

http://www.unixgods.org/~tilo/linux_larger_2TB.html

parted /dev/sdb
print
mklabel gpt
print

#Disk /dev/sdb: 4987GB
#Sector size (logical/physical): 512B/512B
#Partition Table: gpt

#Number Start End Size File system Name Flags

mkpart primary ext3 0 4987GB
print

#Disk /dev/sdb: 5987GB
#Sector size (logical/physical): 512B/512B
#Partition Table: gpt

#Number Start End Size File system Name Flags
# 1 17.4kB 4987GB 4987GB primary

quit

mkfs.ext3 /dev/sdb1
#mke2fs 1.40.8 (13-Mar-2008)
#Filesystem label=
#OS type: Linux
#Block size=4096 (log=2)
#Fragment size=4096 (log=2)
#304390144 inodes, 1217544183 blocks
#60877209 blocks (5.00%) reserved for the super user
#First data block=0
#Maximum filesystem blocks=0
#37157 block groups
#32768 blocks per group, 32768 fragments per group
#8192 inodes per group
#Superblock backups stored on blocks:
# 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
# 4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
# 102400000, 214990848, 512000000, 550731776, 644972544
#
#Writing inode tables: 967/37157

mkdir /data

#root@backup1:~# df -h
#Filesystem Size Used Avail Use% Mounted on
#/dev/sda2 8.3G 540M 7.3G 7% /
#varrun 1013M 40K 1013M 1% /var/run
#varlock 1013M 0 1013M 0% /var/lock
#udev 1013M 56K 1013M 1% /dev
#devshm 1013M 0 1013M 0% /dev/shm
#/dev/sdb1 4.5T 192M 4.3T 1% /data

apt-get update
apt-get upgrade
apt-get install snmp snmpd ntp nfs-kernel-server

echo "\"\e[5~\": history-search-backward" >> ~/.inputrc
echo "\"\e[6~\": history-search-forward" >> ~/.inputrc

vi /etc/ntp.conf
server 10.1.4.5

scp root@10.1.4.3:/root/.ssh/authorized_keys /root/.ssh/
cd /root/
ssh-keygen -t dsa
echo "10.1.4.3 backup2" >> /etc/hosts

cat .ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'

ssh backup2

vi /root/.bashrc
export PS1="[\u@\h \w]# "
alias h='history'
alias vi='vim'
alias j='jobs'
export PS1="[\u@\h \w]# "
alias dr='screen -dr'
export EDITOR=vim
export GREP_OPTIONS='--color=auto'
export HISTFILESIZE=1000

source /root/.bashrc

echo "# ttyS0 - getty
#
# This service maintains a getty on ttyS0 from the point the system is
# started until it is shut down again.

start on runlevel 2
start on runlevel 3
start on runlevel 4
start on runlevel 5

stop on runlevel 0
stop on runlevel 1
stop on runlevel 6

respawn
exec /sbin/getty 38400 ttyS0" > /etc/event.d/ttyS0

vi /boot/grub/menu.lst

serial --unit=0 --speed=38400 --word=8 --parity=no --stop=1
terminal --timeout=15 serial console

append to kernel lines:
console=tty0 console=ttyS0,38400n8

show menu:
#hiddenmenu

echo 'rocommunity jcread 10.1.4.5
rocommunity jcread 10.1.4.3
agentaddress 10.1.4.8:161' > /etc/snmp/snmpd.conf

# to see which iface it is, on backup2:

snmpwalk -v 1 -c jcread 10.1.4.8 interface

echo "bb:x:1984:1984:Big Brother:/home/bb:/bin/bash" >> /etc/passwd

echo "bb:x:1984:" >> /etc/group

pwconv

mkdir /home/bb
chown bb.bb /home/bb

cd ~bb
scp backup2:/mnt/data4/build/bb/bb-linux.tar .

tar xf bb-linux.tar

cd /home/bb/bbc1.9e-btf/etc

echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
echo "10.1.4.8 backup1.johncompanies.com # ssh" >> /home/bb/bbc1.9e-btf/etc/bb-hosts

echo "/:90:95
/var:90:95
/data:85:99" > /home/bb/bbc1.9e-btf/etc/bb-dftab

vi /home/bb/bbc1.9e-btf/bin/bb-disk.sh
(remove all | SORT xxxx)

chmod +r /var/log/messages

./bbchkcfg.sh
#(y to questions)
./bbchkhosts.sh
#(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src

#make; make install
cd ..
./runbb.sh start
more BBOUT
(look for errors)
exit

vi /etc/rc.local
su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"

echo '/data 10.1.4.0/24(rw, no_root_squash,async,no_subtree_check)' >> /etc/exports

/etc/init.d/nfs-kernel-server restart

echo 'chmod o+r /var/log/messages' >> /etc/cron.weekly/sysklogd

echo '10.1.4.8 backup1' >> /etc/hosts
echo '/dev/sdb1 /data ext3 rw,noatime 0 0' >> /etc/fstab

to install digi drivers:

wget http://ftp1.digi.com/support/driver/40002086_n.tgz
apt-get install linux-image-2.6.24-19-server
apt-get install linux-source-2.6.24 (not needed?)
apt-get install linux-headers-2.6.24-19-server
apt-get install make
apt-get install gcc
apt-get install g++
apt-get install libncurses5-dev
apt-get install expect
apt-get install libdbi-perl libdate-calc-perl libdbd-mysql-perl

cd /usr/src; ln -s linux-headers-2.6.24-19-server linux
./configure
make all
make install
make postinstall

/usr/bin/dgrp_cfg_node -v -v init el 65.116.11.2 8

apt-get install mysql

mkdir /data/mysql
chown mysql:mysql /data/mysql
/etc/init.d/mysql stop
mv /var/lib/mysql/* /data/mysql/
mv /data/mysql/ib_* /var/lib/mysql/
vi /etc/mysql/my.cnf
(change datadir to /data/mysql)
vi /etc/apparmor.d/usr.sbin.mysqld
add:
/data/mysql/ r,
/data/mysql/** rwk,
Comment out:
# /var/lib/mysql/ r,
# /var/lib/mysql/** rwk,

/etc/init.d/apparmor restart
/etc/init.d/mysql start

tw_cli /c0/u0 set ignoreECC=on
tw_cli /c0/u0 set storsave=balance
tw_cli /c0/u0 set cache=on

0 0 * * * /usr/local/sbin/3wraidchk
</pre>

= backup2 =

== Summary ==

This machine is used for archiving data and is a backup server for colo customers. It was the former primary backup location for all VPS-based customers before backup1 was installed. Only dedicated customers directly accesses this server to perform their backups. NOTE: power button is broken, so the reset button (paper clip) was rewired to be the power button.

* Location: castle, cab 3-7
* OS: FreeBSD 6.1 x86
* Networking: Priv IP: 10.1.4.3, Pub IP: 69.55.230.10 (firewalled from all but JC infrastructure @ i2b)
* Hardware: 16 IDE drive bays (4 columns of 4, drive 0-0 top left, drive 0-1 just to the right TODO) all hot-swap. Triple power supply.
* Drives:
**3ware 7500-8:
***200 GB JBOD (1 x 200G) labeled 0-0
***500 GB RAID5 (3 x 250G) 0-1 thru 0-3
***700 GB RAID5 (4 x 250G) 0-4 thru 0-7
**3ware 7500-8:
***700 GB RAID5 (4 x 250G) 1-0 thru 1-3
***700 GB RAID5 (4 x 250G) 1-4 thru 1-7

All drives MUST be western digital IDE drives. Other brands will not fit.

In case of an outage, nfs will hang on all connected servers until the nfs service returns. If you can't get backup2 back online, you can get nfs running elsewhere and fake backup2's MAC's: priv: 00:0e:0c:59:c1:a6, pub: 00:07:e9:5b:c6:45

To configure:
ifconfig fxp0 link 00:90:27:f9:0a:d9

== Services provided ==
* backup via rsync and nfs
* samba
* nfs
* snmp
* bigbrother

== Usage ==
* all data is stored under 4 mount points, corresponding to the 4 large RAID5 arrays: <tt>/mnt/data1 /mnt/data2 /mnt/data3 /mnt/data4</tt>
* iso images provided for customers wanting to mount an ISO as a CDROM via the IPKVM are provided via samba on this server. Images live under <tt>/mnt/data2/iso</tt>
* this used to be our primary backup server so you will see old backups from virt and jails around- missing customer data though, just the machine's data
* this server serves as an archive for exported db data from bwdb and old flow files.
* isys backs up here
* customers are nfs-moutned under /mnt/data3/customers as file-backed md devices
* in <tt>/mnt/data4</tt> there are lots of useful things used for building our vps servers, customer servers, and management scripts:
** <tt>/bin</tt>: the master repository of scripts and custom binaries we use on jails and virts. Each night every virt and jail rsync's what's in here to update the local files. So any global updates to scripts would need to be made here (or will be overwritten with what's in here)
** <tt>/build</tt>: files we use for setting up big brother, 3ware cli and scripts for colo's, vzcp customized setup files and so on
** <tt>/vzrpms</tt>: contains the OS templates for many-to-most of the OS's we offer on vz systems

== Cronjobs ==
* backs itself up nightly to nfs-mounted backup1 (mountpoint: <tt>/backup2</tt>)

== Regular maintenance ==
*[[Routine_Maintenance#3ware|Check on health]]

= backup3 =
== Summary ==
This machine is used for archiving data, is a backup server for colo customers, runs a samba server to make available iso's to the IPKVMs, and allows us to connect to the digi serial multiplexer at i2b. Only dedicated customers directly accesses this server to perform their backups.

* Location: i2b, cab 6
* OS: Ubuntu 10.04.1 server amd64
* Networking: Priv IP: 10.1.2.3, Pub IPs: 69.55.229.4 AND 69.55.231.2
* Hardware: 16 drive SATA bays (4 columns of 4, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: 5 TB (6 x 1TB) RAID5 array running on an Areca Technology Corp. ARC-1160 16-Port

== Services provided ==
* backup via rsync and nfs
* samba
* nfs
* digi realport
* snmp
* bigbrother

== Usage ==
* all data is stored under /data
* iso images provided for customers wanting to mount an ISO as a CDROM via the IPKVM are provided via samba on this server. Images live under <tt>/data/iso</tt>
* this server serves as an archive for exported db data from bwdb and old flow files.
* inftrastructure machines at i2b back up here
* customers are nfs-moutned under /data/customers as file-backed loopback devices

== management scripts ==
* mkbackups

mkbackup <cid> GB <ip>

== Cronjobs ==
0 0 * * * /usr/local/sbin/arecaraidchk
RAID checks

35 4 * * * /usr/local/sbin/snapshot_archive
Rotate daily snapshots for infrastructure machine backups

== Regular maintenance ==
*[[Routine_Maintenance#Areca|Check on RAID health]]

== Build ==

=== BIOS Config ===
disable quiet boot

set to last state after power loss

set date/time to GMT

enable serial console output (baud rate 115200)

=== Install OS ===
<pre>Ubuntu 10.04.1 amd64 (couldn't get 12.04 to load cause the H/W was incompat)
10G / ext3
2G swap
~ /data ext4

Install packages:
openssh
samba</pre>

=== DNS and private IP ===

echo "nameserver 69.55.225.225" >> /etc/resolv.conf

Add a 2nd IP to eth0 and setup priv net
<pre>vi /etc/network/interfaces

auto eth0
iface eth0 inet static
address 69.55.229.4
netmask 255.255.255.0
network 69.55.229.0
broadcast 69.55.229.255
gateway 69.55.229.1
# dns-* options are implemented by the resolvconf package, if installed
dns-nameservers 69.55.229.3 66.181.0.2
dns-search johncompanies.com

auto eth0:1
iface eth0:1 inet static
address 69.55.231.2
netmask 255.255.255.0
network 69.55.231.0
broadcast 69.55.231.255

auto eth1
iface eth1 inet static
address 10.1.2.3
netmask 255.255.255.0
network 10.1.2.0
broadcast 10.1.2.255

</pre>

=== Install packages ===
<pre>apt-get update
apt-get upgrade
apt-get install gcc
apt-get install libssl-dev
apt-get install libncurses5-dev
apt-get install cu
apt-get install unzip
apt-get install snmp snmpd ntp nfs-kernel-server</pre>

=== tweak grub, enable serial ===

<pre>vi /etc/default/grub
#GRUB_HIDDEN_TIMEOUT=0
GRUB_CMDLINE_LINUX_DEFAULT="max_loop=64"
GRUB_CMDLINE_LINUX="console=tty0 console=ttyS0"
update-grub</pre>

<pre>echo "start on stopped rc RUNLEVEL=[2345]
stop on runlevel [!2345]
respawn
exec /sbin/getty -L ttyS0 38400 vt102" > /etc/init/ttyS0.conf</pre>

=== install realport (digi) driver ===

give the digi an ip with DgIpServ.exe

<pre>cd /usr/src/
wget ftp://ftp1.digi.com/support/beta/linux/dgrp/dgrp-1.9.tgz
tar xzf dgrp-1.9.tgz
cd dgrp-1.9/
./configure
make
make install
make postinstall
update-rc.d dgrp_daemon defaults</pre>

configure ports:
dgrp_cfg_node init el 10.1.2.10 16

try connecting with:
cu -l /dev/ttyel00 -s 38400

=== shell, ntp, ssh key, hosts ===

Shell autocompletion search:
<pre>echo "\"\e[5~\": history-search-backward" >> ~/.inputrc
echo "\"\e[6~\": history-search-forward" >> ~/.inputrc</pre>

Setup ntp:
<pre>vi /etc/ntp.conf
server 10.1.2.1
server ntp.ubuntu.com</pre>

Generate ssh keys:
<pre>cd /root/
ssh-keygen -t dsa</pre>
Defaults, no password

Setup hosts:
<pre>echo "69.55.230.10 backup2" >> /etc/hosts
echo "69.55.230.11 backup1" >> /etc/hosts
echo "10.1.2.4 bwdb2" >> /etc/hosts
echo "10.1.2.3 backup3" >> /etc/hosts</pre>

Copy keys to servers where we need passwordless login:
<pre>cat .ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat .ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'</pre>

Setup shell:
<pre>vi /root/.bashrc
(add to bottom)
alias h='history'
alias vi='vim'
alias j='jobs'
export PS1="[\u@\h \w]# "
alias dr='screen -dr'
export EDITOR=vim
export GREP_OPTIONS='--color=auto'
export HISTFILESIZE=1000

alias tip-switch-p20='cu -l ttyel00 -s 9600'
alias tip-switch-p21='cu -l ttyel15 -s 9600'
alias tip-switch-p22='cu -l ttyel14 -s 9600'
alias tip-switch-p23='cu -l ttyel05 -s 9600'
alias tip-switch-p24='cu -l ttyel06 -s 9600'
alias tip-switch-p25='cu -l ttyel09 -s 9600'
alias tip-switch-p26='cu -l ttyel07 -s 9600'
alias tip-switch-p27='cu -l ttyel08 -s 9600'
alias tip-firewall2='cu -l ttyel01 -s 115200'
alias tip-nat2='cu -l /dev/ttyel02 -s 115200'
alias tip-backup3='cu -l ttyel04 -s 38400'
alias tip-bwdb2='cu -l ttyel03 -s 115200'
alias tip-backup4='cu -l ttyel13 -s 115200'
alias tip-jail3='cu -l ttyel11 -s 115200'

Load new shell:
source /root/.bashrc

Setup snmpd (this is only valid for a server at castle):
echo 'rocommunity jcread 10.1.4.5
rocommunity jcread 10.1.4.3
agentaddress 10.1.4.8:161' > /etc/snmp/snmpd.conf

to see which iface it is, on backup2:

snmpwalk -v 1 -c jcread 10.1.4.8 interface

=== nfs ===

Allow mounts from private net:
echo '/data 10.1.2.0/24(rw,no_root_squash,async,no_subtree_check)' >> /etc/exports

Restart nfsd:
/etc/init.d/nfs-kernel-server restart

=== bb ===

Add user, group:
echo "bb:x:1984:1984:Big Brother:/home/bb:/bin/bash" >> /etc/passwd
echo "bb:x:1984:" >> /etc/group
pwconv

Create home:
mkdir /home/bb
chown bb.bb /home/bb
cd ~bb

Copy over and install files:
<pre>scp backup2:/mnt/data4/build/bb/bb-linux.tar .
tar xf bb-linux.tar
cd /home/bb/bbc1.9e-btf/etc</pre>

Configure main bb server:
echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
echo "10.1.2.3 backup3.johncompanies.com # ssh" >> /home/bb/bbc1.9e-btf/etc/bb-hosts

Configure low disk alerts:
<pre>echo "/:90:95
/var:90:95
/data:85:99" > /home/bb/bbc1.9e-btf/etc/bb-dftab</pre>

vi /home/bb/bbc1.9e-btf/bin/bb-disk.sh
(remove all | SORT xxxx since SORT is broken)

chmod +r /var/log/messages

./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
<pre>cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src</pre>

<pre>make; make install
cd ..
./runbb.sh start
more BBOUT</pre>
(look for errors)
exit

<pre>vi /etc/rc.local
su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"
</pre>
(before the exit 0)

echo 'chmod o+r /var/log/messages' >> /etc/cron.weekly/sysklogd

Add f/w rule:
ipfw add 00096 allow ip from { 69.55.229.4 or 69.55.229.3 } to 69.55.230.2 1984

<pre>vi ~bb/bbc1.9e-btf/etc/bbdef-client.sh
DFWARN=199
DFPANIC=199</pre>

=== raid check ===

==== 3ware ====
<pre>
scp backup1:/usr/local/sbin/tw_cli /usr/local/sbin/tw_cli
scp backup1:/usr/local/sbin/checkraid.sh /usr/local/sbin/checkraid.sh
scp backup1:/usr/local/sbin/3wraidchk /usr/local/sbin/3wraidchk
vi /usr/local/sbin/checkraid.sh
:%s/c0/c2/g

crontab -e
0 0 * * * /usr/local/sbin/3wraidchk</pre>

==== areca ====
<pre>
cd /tmp
wget http://www.areca.us/support/s_linux/cli/linuxcli_V1.10.0_120815.zip
unzip linuxcli_V1.10.0_120815.zip
cp linuxcli_V1.10.0_120815/x86_64/cli64 /usr/local/sbin/
chmod 0700 /usr/local/sbin/cli64
cli64 rsf info
</pre>

<pre>scp backup2:/data4/bin/arecaraidchk /usr/local/sbin
scp backup1:/usr/local/sbin/Sendmail.pm /usr/local/sbin

crontab -e
0 0 * * * /usr/local/sbin/arecaraidchk
</pre>

<pre>cat > /root/verify.sh
cli64 vsf info
cli64 rsf info
cli64 disk info
cli64 event info
echo press enter when ready to run verify ; read x

cli64 vsf check vol=1
</pre>

=== misc binaries ===

scp backup1:/usr/local/sbin/snapshot_archive /usr/local/sbin/snapshot_archive
vi /usr/local/sbin/snapshot_archive
(remove entries)

crontab -e
35 4 * * * /usr/local/sbin/snapshot_archive

scp backup1:/usr/local/sbin/pagedave /usr/local/sbin/pagedave
scp backup1:/usr/local/sbin/taskdone /usr/local/sbin/taskdone

Since installing /bin/mail requires all sorts of packages (lame) we write a simple one here...which can only email johncompanies.com addr's unless you add relaying for this host:

<pre>
cat > /bin/mail
#!/usr/bin/perl
use strict;
use warnings;

use lib '/usr/local/sbin';
use Sendmail qw(sendmail);

my $sub = $ARGV[1];
my $to = $ARGV[2];

my %mail = (
To => $to,
From => $to,
Subject => $sub,
Message => '',
smtp => 'mail.johncompanies.com'
);
sendmail(%mail) || print "Error: $Sendmail::error";

</pre>

chmod 0700 /bin/mail

=== mkbackup ===

mkdir /data/customers

<pre>cat > /usr/local/sbin/mkbackup
#!/bin/sh

if test $1; then
cid=$1
else
echo "ERROR: Usage: mkbackup cid GB ip Terminating."
exit
fi

if test $2; then
gb=$2
else
echo "ERROR: Usage: mkbackup cid GB ip Terminating."
exit
fi

if test $3; then
ip=$3
else
echo "ERROR: Usage: mkbackup cid GB ip Terminating."
exit
fi

if test -e /data/customers/${cid}-file; then
echo "ERROR: /data/customers/${cid}-file exists"
exit
else
echo "touch /data/customers/${cid}-file"
touch /data/customers/${cid}-file
count=`echo $gb|awk '{print $1*1000}'`
echo "dd if=/dev/zero of=/data/customers/${cid}-file bs=1024K count=$count"
dd if=/dev/zero of=/data/customers/${cid}-file bs=1024K count=$count
echo "/sbin/mkfs -t ext3 -F -j -q /data/customers/${cid}-file"
/sbin/mkfs -t ext3 -F -j -q /data/customers/${cid}-file
fi

if test -e /data/customers/$cid; then
echo "ERROR: /data/customers/$cid exists"
exit
else
echo "mkdir /data/customers/${cid}"
mkdir /data/customers/${cid}
echo "mount -o loop /data/customers/${cid}-file /data/customers/$cid"
mount -o loop /data/customers/${cid}-file /data/customers/$cid
df -h /data/customers/$cid

echo "fsck -y /data/customers/${cid}-file" >> /etc/nfs_backup_mounts.sh
echo "mount -o loop /data/customers/${cid}-file /data/customers/$cid" >> /etc/nfs_backup_mounts.sh
echo "" >> /etc/nfs_backup_mounts.sh

echo "/data/customers/$cid $ip/32(rw,no_root_squash,async,no_subtree_check)" >> /etc/exports
/etc/init.d/nfs-kernel-server restart
tail /var/log/messages
fi</pre>

chmod 0700 /usr/local/sbin/mkbackup

vi /etc/rc.local
add:
/etc/nfs_backup_mounts.sh

=== samba ===

apt-get install samba

vi /etc/samba/smb.conf

; comment out any mounts, add:

<pre>[data]
read only = yes
locking = no
path = /data/iso
guest ok = yes</pre>

/etc/init.d/smbd restart

mkdir /data/iso

Bring over some stuff from backup2

<pre>cd /data/iso
scp backup2:/d2/iso/3wfirmware.iso .
scp backup2:/d2/iso/MD5SUMS .
scp backup2:/d2/iso/bootimg.iso .
scp backup2:/d2/iso/systemrescuecd-x86-0.2.19.iso .
scp backup2:/d2/iso/win98bootcd.iso .
scp backup2:/d2/iso/acronis_bootdisk.iso .
scp backup2:/d2/iso/memtest86-3.2.iso .</pre>

=== Moving from one server to another ===

Here are the steps you would take to move settings and data from one server to a new backup server:

* rsync over all /data/customers (we do this cause if we didn't use *-file it would copy over the files AND the data in the mountpoint)
rsync -av --progress --ignore-times *-file root@10.1.2.33:/data/customers/
after umounting all the customers, copy over the (empty) directories separately:
for f in `find . -type d`; do rsync -av $f root@69.55.229.25:/data/customers; done

* copy mount script
[root@backup3 /data/customers]# scp /etc/nfs_backup_mounts.sh root@69.55.229.25:/etc/nfs_backup_mounts.sh

* copy rc.local
[root@backup3 /data/customers]# scp /etc/rc.local root@69.55.229.25:/etc/rc.local

* copy /etc/exports
[root@backup3 /data/customers]# scp /etc/exports root@69.55.229.25:/etc/exports

* edit /etc/hostname on both machines (set current to oldbackup3)

* edit /etc/network/interfaces (swap IPs).

* stop mounts from mounting on old and new servers so it doesnt start with reboot right away:
chmod 000 /etc/nfs_backup_mounts.sh

* reboot both servers @ same time

* check everything out

* run /etc/nfs_backup_mounts.sh on new server

* if switch port changed update mrtg to reflect correct port pub nic is on (on p20):
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg

= console =

== Summary ==
This box's only purpose is to serve as a means to connect to the digi serial multiplexer boxes at castle. Connect to it using the blue (cisco) ribbon cable with the beige RJ-45 to serial connector, 9600 8N1.

* Location: castle, cab 3-8
* OS: SunOS 5.8 (solaris)
* Networking: Priv IP: 10.1.4.4
* Hardware: Sun Netra

To connect to consoles, ssh in as user 'console' and use the <tt>tip</tt> command to connect to devices listed in <tt>/etc/remote</tt>

i.e.
tip switch-p1
tip jail1

== Configuring digi/ports ==

=== /etc/remote ===
This is where the configuration/mapping for ports and custom names which we use along with the tip command to connect to various ports on the digi switches.

We have 2 digi's at castle we connect to:

#3-7 10.1.4.10
virt15:dv=/dev/dty/CO001s:br#38400:el=^C^S^Q^U^D:ie=%$:oe=^D:
virt13:dv=/dev/dty/CO002s:br#115200:el=^C^S^Q^U^D:ie=%$:oe=^D:hf:

and

#3-6 10.1.4.11
jail4:dv=/dev/dty/CP001s:br#9600:el=^C^S^Q^U^D:ie=%$:oe=^D:hf:
jail16:dv=/dev/dty/CP002s:br#9600:el=^C^S^Q^U^D:ie=%$:oe=^D:hf:

The only things you need to edit are the first part (i.e. <tt>jail4</tt>) and the speed (i.e. <tt>9600</tt>). You can decipher which port on the digi each line corresponds to by the <tt>CP001s or CO001s</tt> (port 1 on digi1 and digi2), <tt>CP002s or CO002s</tt> (port 2 on digi1 and digi2)

=== drpadmin ===
The tool you use to configure a device to a digi box is drpadmin:

<pre>bash-2.03$ su
Password:
# drpadmin

Please select an option (a)dd (d)elete (s)how (r)eset (q)uit : s
0 10.1.4.10 32 CO 771 never 1027
1 10.1.4.11 32 CP 771 never 1027
2 65.116.11.2 8 el 771 never 1027

Please select an option (a)dd (d)elete (s)how (r)eset (q)uit :</pre>

Use those commands above to modify the devices available.

== Switching IP/hostname ==

Edit:
/etc/defaultrouter
/etc/hosts
/etc/hostname.hme0
/etc/nodename
Maybe needed to run: # ifconfig hme0 10.1.4.4 up

= devweb =

We do web development on devweb.johncompanies.com

Currently this is a jail running on jail17 / 69.55.230.8

If the jail is restarted, you will need to manually restart the web service with:
httpsdctl restart

All website development work should be done here first. It works exactly like and is setup like our [[Management_System_/_Public_Website_/_Signup_/_Account_Manager|main site]].

= firewall2 =

== Summary ==

This machine is the primary firewall for the entire network at i2b. firewall3 is a hot standby replacement for
firewall2. Both firewall2 and firewall3 should not be connected at the same time since they use the same internal
and external IP addresses.

* Location: i2b, cab 6
* OS: FreeBSD 6.4 x86
* Networking: Priv IP: 10.1.2.2, Pub IPs: 66.181.18.3 (external), 69.55.229.1 & 69.55.231.1 (internal). It has 3 network connections (2 onboard, 1 PCI) connecting to the external, internal and private networks. TODO: describe NIC location/orientation

* Hardware: 2 SCSI SCA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: 73 GB (2 x 73GB) RAID1 array running on an LSI MegaRAID SCSI 320 PCI RAID card.

== Services Provided ==
* firewall (ipfw)
* bigbrother for customer machines

== Firewall Rule Configuration ==

See [[FreeBSD_Reference#Firewall_Rule_Configuration|Firewall Rule Configuration]] for more discussion on how to actually manipulate firewall rules.

== Disaster Recovery ==

TODO: need backup f/w and instructions on how to move cables.

Here's what you need to put in /etc/rc.conf to get a firewall going (as far as routes and IPs)

TODO

Here's the config on the live firewall:

<pre>kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"
gateway_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.2.2"

fsck_y_enable="YES"
background_fsck="NO"

defaultrouter="66.181.18.2"
hostname="firewall2.johncompanies.com"
ifconfig_bge0="inet 66.181.18.3 netmask 255.255.255.224"
ifconfig_bge1="inet 69.55.229.1 netmask 255.255.255.0"
ifconfig_bge1_alias0="inet 69.55.231.1 netmask 255.255.255.0"
ifconfig_bge1_alias1="inet 65.50.228.1 netmask 255.255.255.0"
ifconfig_bge1_alias2="inet 65.50.229.1 netmask 255.255.255.0"
ifconfig_bge1_alias3="inet 65.50.230.1 netmask 255.255.255.0"
ifconfig_bge1_alias4="inet 65.50.231.1 netmask 255.255.255.0"
ifconfig_bge1_alias5="inet 65.50.232.1 netmask 255.255.255.0"
ifconfig_bge1_alias6="inet 65.50.233.1 netmask 255.255.255.0"
ifconfig_bge1_alias7="inet 65.50.234.1 netmask 255.255.255.0"
ifconfig_bge1_alias8="inet 65.50.235.1 netmask 255.255.255.0"
ifconfig_fxp0="inet 10.1.2.2 netmask 255.255.255.0"
sshd_enable="YES"
usbd_enable="YES"
</pre>

== Cronjobs ==
30 3 * * * /usr/local/etc/rsync.backup
Backup to backup3

0 0 1 * * /sbin/ipfw zero
0 0 1 * * /sbin/ipfw del 3 4 5
Reset counters and remove pipe rules on the 1st of the month. Pay attention when setting up a rule as 3 4 5 (that's not a temporary traffic cap).

59 23 30 * * /sbin/ipfw show > /tmp/ipfw_count
3 0 30 * * /sbin/ipfw show > /tmp/ipfw_count
Capture counts periodically

0 3 * * * /etc/makefwrules.pl; /etc/makepiperules.pl;
This script will create <tt>/etc/firewall.sh</tt> which contains all the firewall and pipe rules in place at the time the script was run.

*/5 * * * * /usr/local/sbin/lsiraidchk
Checking the health of the RAID array

== DOS attacks ==

See [[FreeBSD_Reference#Handling_a_DoS_attack|Handling a DoS attack]] regarding how to handle a DOS attack.

== build ==

<pre>partition map:
/ 58g
swap 4g
/var 512m
/tmp 512m
/usr 5.5g

4. edit /etc/make.conf
echo "WITHOUT_X11=yes \
KERNCONF=firewall2 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf

5. add settings to /boot/loader.conf and /boot.config

echo "-Dh" >> /boot.config

echo 'console="comconsole,vidconsole" \
boot_multicons="YES" \
boot_serial="YES" \
comconsole_speed="115200"' >> /boot/loader.conf

6. turn off all ttyv's except 0 and 1 in /etc/ttys
also turn on ttyd0, change type to vt100:
vi /etc/ttys

ttyv2 "/usr/libexec/getty Pc" cons25 off secure
ttyv3 "/usr/libexec/getty Pc" cons25 off secure
ttyv4 "/usr/libexec/getty Pc" cons25 off secure
ttyv5 "/usr/libexec/getty Pc" cons25 off secure
ttyv6 "/usr/libexec/getty Pc" cons25 off secure
ttyv7 "/usr/libexec/getty Pc" cons25 off secure
# Serial terminals
# The 'dialup' keyword identifies dialin lines to login, fingerd etc.
ttyd0 "/usr/libexec/getty std.9600" vt100 on secure

kill -1 1

on console server:
vi /etc/remote
(rename port to jail8 depending on where and which digi plugged into)
test serial console

7. populate hosts
echo "69.55.230.10 backup2" >> /etc/hosts
echo "69.55.230.11 backup1" >> /etc/hosts
echo "10.1.2.3 backup3" >> /etc/hosts

8. put key in authorized_keys on backup3
cd
ssh-keygen -t dsa -b 1024
(default location, leave password blank)

Punch a hole in firewall1 to allow traffic to backup servers @ castle:

ipfw add 99 allow ip from 66.181.18.0/27 to 69.55.230.10 22
ipfw add 95 allow ip from 66.181.18.0/27 to 69.55.230.11 22

cat /root/.ssh/id_dsa.pub | ssh backup3 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'

confirm that you can ssh to backup3 and backup 2 without getting a login prompt

ssh backup3 hostname

ssh backup2 hostname

ssh backup1 hostname

10. edit root's path and login script:
vi /root/.cshrc

Change alias entries (add G):
alias la ls -aG
alias lf ls -FAG
alias ll ls -lAG
alias ls ls -AG
alias mbm mb mount
alias mbu mb umount

and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "

11. install cvsup
cd /usr/ports/net/cvsup-without-gui
make install clean; rehash; mail -s 'cvs installed' dave.boodman@vtext.com < /dev/null

12. get latest sources for this release:
cd /usr/src
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_4\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

cvsup sup ; mail -s 'cvs sup done' dave.boodman@vtext.com < /dev/null

13. configure new kernel.

cd /usr/src/sys/i386/conf
scp backup2:/mnt/data4/build/freebsd/firewall2-6.4 ./firewall2

15. build, install kernel and world

cd /boot

mv kernel kernel.GENERIC
cd kernel.GENERIC
cd /usr/src
make buildkernel installkernel

make buildworld ; mail -s 'buildworld done' dave.boodman@vtext.com < /dev/null
(2450: 1:56min, supermicro: 59mins, 2950: 38mins)
make installworld
(2450: 3min, supermicro: 1min, 2950: :34)
mergemaster -i

17. populate /etc/rc.conf with IPs and NFS settings
vi /etc/rc.conf

kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"
gateway_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.2.2"

ifconfig_bce1="inet 10.1.2.2 netmask 255.255.255.0"
fsck_y_enable="YES"
background_fsck="NO"

defaultrouter="66.181.18.2"
hostname="firewall2.johncompanies.com"
ifconfig_bge0="inet 66.181.18.3 netmask 255.255.255.224"
ifconfig_bge1="inet 69.55.229.1 netmask 255.255.255.0"
ifconfig_fxp0="inet 10.1.2.2 netmask 255.255.255.0"
sshd_enable="YES"
usbd_enable="YES"

20. reboot. Confirm new kernel is loaded

uname -a

21. update ports:
cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_4\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup

cvsup sup; mail -s 'cvs sup ports done' dave.boodman@vtext.com < /dev/null

22. Install raid mgmt tool

# linux base
cd /usr/ports/devel/libtool22
make install base

cd /usr/ports/emulators/linux_base-fc4
make install clean

#linux-megamgr-5.20
cd /usr/ports/sysutils/linux-megamgr
make install clean

# megarc-1.51
cd /usr/ports/sysutils/megarc
make install clean

Test:
rehash; megarc -ldInfo -a0 -l0

23. install rsync from ports
cd /usr/ports/net/rsync
make install clean

choose default options

25. install bb client
adduser
Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username : bb
Password : <random>
Full Name : bb
Uid : 1984
Class :
Groups : bb
Home : /home/bb
Shell : /bin/sh
Locked : no
OK? (yes/no): yes

cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd.tar .
tar xvf bb-freebsd.tar

edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.2.1 firewall2.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

vi /home/bb/bbc1.9e-btf/ext/openfiles
MACHINE="firewall2,johncompanies,com" # HAS TO BE IN A,B,C FORM

cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src
make; make install
cd ..

vi /usr/home/bb/bbc1.9e-btf/bin/bb-cpu.sh
$1 $TOPARGS > $BBTMP/TOP.$$
# /usr/local/jail/bin/jtop > $BBTMP/TOP.$$

./runbb.sh start
more BBOUT
(look for errors)
exit

echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh

Punch a hole in the firewall to allow it to communicate with bb monitor:

ipfw add 00096 allow ip from 66.181.18.0/27 to 69.55.230.2

27. configure bb on mail:
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
66.181.18.3 firewall2.johncompanies.com # ssh

su bb
cd
bbsrc/bb/runbb.sh restart ; exit

29. configure ntp
echo "server 10.1.2.1" > /etc/ntp.conf

/usr/sbin/ntpd -p /var/run/ntpd.pid
sleep 2; ntpq -p
(confirm it’s able to reach our time server)

echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh
chmod 0700 /usr/local/etc/rc.d/ntp.sh

30. fwd and reverse lookups on ns1c
vr johncompanies.com
(edit the PTR too)

33. setup backups
echo '#\!/bin/sh\
backupdir=/data/firewall2/current\
\
## ENTRY /etc ' > /usr/local/etc/backup.config

on backup3:
setup backup dirs:
ssh backup3 mkdir -p /data/firewall2/current

on backup3, add the system to
vi /usr/local/sbin/snapshot_archive

scp 69.55.230.2:/usr/local/etc/rsync.backup /usr/local/etc/rsync.backup
vi /usr/local/etc/rsync.backup
backup1 > backup3

crontab -e
1 0 * * * /usr/local/etc/rsync.backup

34. mkdir /root/logs

35. edit sshd_config for security
vi /etc/ssh/sshd_config
ListenAddress 66.181.18.3
ListenAddress 10.1.2.1

kill -1 `cat /var/run/sshd.pid`

35. raid chk

cat > /usr/local/sbin/lsiraidchk
#!/usr/bin/perl

my @out = split "\n", `megarc -ldInfo -a0 -Lall|grep Status:`;

foreach (@out) {
if ($_ =~ /DEGRADED/) { $date = `date`; chomp $date; `echo "$date: RAID ARRAY DEGRADED" >> /var/log/messages`; }
#print $_;
}

36. add crontab entries
crontab -e
30 3 * * * /usr/local/etc/rsync.backup
0 0 1 * * /sbin/ipfw zero
0 0 1 * * /sbin/ipfw del 3 4 5
59 23 30 * * /sbin/ipfw show > /tmp/ipfw_count
3 0 30 * * /sbin/ipfw show > /tmp/ipfw_count
0 3 * * * /etc/makefwrules.pl; /etc/makepiperules.pl;
*/5 * * * * /usr/local/sbin/lsiraidchk

#10 0 * * * rm /var/spool/clientmqueue/*

scp /etc/makefwrules.pl user@64.163.14.48:~
scp /etc/makepiperules.pl user@64.163.14.48:~
mv /home/user/makefwrules.pl /etc
mv /home/user/makepiperules.pl /etc
touch /etc/firewall.sh
mkdir /etc/oldrules/

other binaries

scp /usr/local/bin/rulemaker user@64.163.14.48:~
mv ~user/rulemaker /usr/local/sbin
scp ~user/Sendmail.pm user@64.163.14.48:~
scp ~user/doswatch.pl user@64.163.14.48:~

Setup basic ruleset

ipfw add 00009 count udp from any to any
ipfw add 00010 allow tcp from any to any established
ipfw add 00012 deny tcp from any to any tcpflags syn tcpoptions !mss
ipfw add 00012 deny icmp from any to any icmptypes 4,5,9,10,12,13,14,15,16,17,18
ipfw add 00012 deny tcp from any to any tcpflags syn,fin
ipfw add 00012 deny tcp from any to any tcpflags fin,psh,rst,urg
ipfw add 00012 allow icmp from any to any
ipfw add 00014 deny tcp from any to any dst-port 135
ipfw add 00150 skipto 65535 ip from any to any via em1 in

IPKVM3:
00098 allow ip from { 69.55.230.6 or 69.55.230.7 } to 69.55.230.10 dst-port 139
00098 deny ip from any to 69.55.230.10 dst-port 139
</pre>

= firewall3 =

== Summary ==

This machine is the backup firewall for the network at i2b.

* Location: i2b, cab ?
* OS: FreeBSD 9.1 amd64
* Networking: Priv IP: 10.1.2.5, Pub IPs: 66.181.18.3 (external), 69.55.229.1 & 69.55.231.1 (internal). It has 3 network connections (2 onboard, 1 PCI) connecting to the external, internal and private networks.

<pre>

The internal network NIC is the left one on the motherboard (69.55.229.1/24, ...).
The external network NIC is the right one on the motherboard (66.181.18.3/28).
The PCI ethernet card is connected to our private network (10.1.2.5/24).

</pre>

* Hardware: 2 SCSI SCA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: 160 GB (2 x 160GB) RAID1 array running on an LSI MegaRAID SCSI 320 PCI RAID card.

== Services Provided ==
* firewall (ipfw)
* bigbrother

== Firewall Rule Configuration ==

See [[FreeBSD_Reference#Firewall_Rule_Configuration|Firewall Rule Configuration]] for more discussion on how to actually manipulate firewall rules.

== Disaster Recovery ==

'''To put the backup firewall3 into service:'''

<pre>
Move the internal cable (to our networks) from firewall2 to em1 which is the left most ethernet port (69.55.229.1).
Move the external cable (to outside world) from firewall2 to em0 which is the port to the right on the motherboard (66.181.18.3).
The PCI ethernet port (fxp0) should already be connected to private network (10.1.2.5).
</pre>

Here's what you need to put in /etc/rc.conf to get a firewall going (as far as routes and IPs)

<pre>
kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"
gateway_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.2.5"

fsck_y_enable="YES"
background_fsck="NO"

defaultrouter="66.181.18.2"
hostname="firewall3.johncompanies.com"
ifconfig_em0="inet 66.181.18.3 netmask 255.255.255.224"

ifconfig_em1="inet 69.55.229.1 netmask 255.255.255.0"
ifconfig_em1_alias0="inet 69.55.231.1 netmask 255.255.255.0"

# ifconfig_em1_alias1="inet 65.50.228.1 netmask 255.255.255.0"
# ifconfig_em1_alias2="inet 65.50.229.1 netmask 255.255.255.0"
# ifconfig_em1_alias3="inet 65.50.230.1 netmask 255.255.255.0"
# ifconfig_em1_alias4="inet 65.50.231.1 netmask 255.255.255.0"
# ifconfig_em1_alias5="inet 65.50.232.1 netmask 255.255.255.0"
# ifconfig_em1_alias6="inet 65.50.233.1 netmask 255.255.255.0"
# ifconfig_em1_alias7="inet 65.50.234.1 netmask 255.255.255.0"
# ifconfig_em1_alias8="inet 65.50.235.1 netmask 255.255.255.0"

ifconfig_fxp0="inet 10.1.2.5 netmask 255.255.255.0"

sshd_enable="YES"
usbd_enable="YES"
</pre>

== Cronjobs ==
30 3 * * * /usr/local/etc/rsync.backup
Backup to backup3

0 0 1 * * /sbin/ipfw zero
0 0 1 * * /sbin/ipfw del 3 4 5
Reset counters and remove pipe rules on the 1st of the month. Pay attention when setting up a rule as 3 4 5 (that's not a temporary traffic cap).

59 23 30 * * /sbin/ipfw show > /tmp/ipfw_count
3 0 30 * * /sbin/ipfw show > /tmp/ipfw_count
Capture counts periodically

0 3 * * * /etc/makefwrules.pl; /etc/makepiperules.pl;
This script will create <tt>/etc/firewall.sh</tt> which contains all the firewall and pipe rules in place at the time the script was run.

*/5 * * * * /usr/local/sbin/lsiraidchk
Checking the health of the RAID array

== DOS attacks ==

See [[FreeBSD_Reference#Handling_a_DoS_attack|Handling a DoS attack]] regarding how to handle a DOS attack.

== build ==

<pre>partition map:
/ 58g
swap 4g
/var 512m
/tmp 512m
/usr 5.5g

4. edit /etc/make.conf
echo "WITHOUT_X11=yes \
KERNCONF=firewall3 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf

5. add settings to /boot/loader.conf and /boot.config

echo "-Dh" >> /boot.config

echo 'console="comconsole,vidconsole" \
boot_multicons="YES" \
boot_serial="YES" \
comconsole_speed="115200"' >> /boot/loader.conf

6. turn off all ttyv's except 0 and 1 in /etc/ttys
also turn on ttyd0, change type to vt100:
vi /etc/ttys

ttyv2 "/usr/libexec/getty Pc" cons25 off secure
ttyv3 "/usr/libexec/getty Pc" cons25 off secure
ttyv4 "/usr/libexec/getty Pc" cons25 off secure
ttyv5 "/usr/libexec/getty Pc" cons25 off secure
ttyv6 "/usr/libexec/getty Pc" cons25 off secure
ttyv7 "/usr/libexec/getty Pc" cons25 off secure
# Serial terminals
# The 'dialup' keyword identifies dialin lines to login, fingerd etc.
ttyd0 "/usr/libexec/getty std.9600" vt100 on secure

kill -1 1

on console server:
vi /etc/remote
(rename port to jail8 depending on where and which digi plugged into)
test serial console

7. populate hosts
echo "69.55.230.10 backup2" >> /etc/hosts
echo "69.55.230.11 backup1" >> /etc/hosts
echo "10.1.2.3 backup3" >> /etc/hosts

8. put key in authorized_keys on backup3
cd
ssh-keygen -t dsa -b 1024
(default location, leave password blank)

Punch a hole in firewall1 to allow traffic to backup servers @ castle:

ipfw add 99 allow ip from 66.181.18.0/27 to 69.55.230.10 22
ipfw add 95 allow ip from 66.181.18.0/27 to 69.55.230.11 22

cat /root/.ssh/id_dsa.pub | ssh backup3 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'

confirm that you can ssh to backup3 and backup 2 without getting a login prompt

ssh backup3 hostname

ssh backup2 hostname

ssh backup1 hostname

10. edit root's path and login script:
vi /root/.cshrc

Change alias entries (add G):
alias la ls -aG
alias lf ls -FAG
alias ll ls -lAG
alias ls ls -AG
alias mbm mb mount
alias mbu mb umount

and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "

11. install cvsup
cd /usr/ports/net/cvsup-without-gui
make install clean; rehash; mail -s 'cvs installed' 8583619553@vtext.com < /dev/null

12. get latest sources for this release:
cd /usr/src
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_9_1\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

cvsup sup ; mail -s 'cvs sup done' 8583619553@vtext.com < /dev/null

13. configure new kernel.

cd /usr/src/sys/amd64/conf
scp backup2:/mnt/data4/build/freebsd/firewall3-9.1 ./firewall3

15. build, install kernel and world

cd /boot

mv kernel kernel.GENERIC
cd kernel.GENERIC
cd /usr/src
make buildkernel installkernel

make buildworld ; mail -s 'buildworld done' 8583619553@vtext.com < /dev/null
(supermicro: 2:15 mins, 2950: 38? mins)
make installworld
(2450: 3min, supermicro: 1min, 2950: :34)
mergemaster -i

17. populate /etc/rc.conf with IPs and NFS settings
vi /etc/rc.conf

kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"
gateway_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"

fsck_y_enable="YES"
background_fsck="NO"

hostname="firewall3.johncompanies.com"
# external network
ifconfig_em0="inet 66.181.18.3 netmask 255.255.255.224"

# internal network
ifconfig_em1="inet 69.55.229.1 netmask 255.255.255.0"
ifconfig_em1_alias0="inet 69.55.231.1 netmask 255.255.255.0"

ifconfig_em1_alias1="inet 65.50.228.1 netmask 255.255.255.0"
ifconfig_em1_alias2="inet 65.50.229.1 netmask 255.255.255.0"
ifconfig_em1_alias3="inet 65.50.230.1 netmask 255.255.255.0"
ifconfig_em1_alias4="inet 65.50.231.1 netmask 255.255.255.0"
ifconfig_em1_alias5="inet 65.50.232.1 netmask 255.255.255.0"
ifconfig_em1_alias6="inet 65.50.233.1 netmask 255.255.255.0"
ifconfig_em1_alias7="inet 65.50.234.1 netmask 255.255.255.0"
ifconfig_em1_alias8="inet 65.50.235.1 netmask 255.255.255.0"

defaultrouter="66.181.18.2"

# private network
ifconfig_fxp0="inet 10.1.2.5 netmask 255.255.255.0"

inetd_enable="YES"
inetd_flags="-wW -a 10.1.2.5"

sshd_enable="YES"
usbd_enable="YES"
ntpd_enable="YES"
# powerd_enable="YES"

20. reboot. Confirm new kernel is loaded

uname -a

21. update ports:
cd /usr/ports

echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_9_1\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup

cvsup sup; mail -s 'cvs sup ports done' 8583619553@vtext.com < /dev/null

22. Install raid mgmt tool

# linux base
cd /usr/ports/devel/libtool22
make install base

cd /usr/ports/emulators/linux_base-fc4
make install clean

scp backup2:/d4/build/3ware/tw_cli-freebsd-x86_64-9.5.0.1.tgz /usr/local/sbin
cd /usr/local/sbin
tar xzvf tw_cli-freebsd-x86_64-9.5.0.1.tgz
rm tw_cli-freebsd-x86_64-9.5.0.1.tgz

23. install rsync from ports
cd /usr/ports/net/rsync
make install clean

choose default options

25. install bb client
adduser
Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username : bb
Password : <random>
Full Name : bb
Uid : 1984
Class :
Groups : bb
Home : /home/bb
Shell : /bin/sh
Locked : no
OK? (yes/no): yes

cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd.tar .
tar xvf bb-freebsd.tar

edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.2.5 firewall3.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

vi /home/bb/bbc1.9e-btf/ext/openfiles
MACHINE="firewall3,johncompanies,com" # HAS TO BE IN A,B,C FORM

cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src
make; make install
cd ..

vi /usr/home/bb/bbc1.9e-btf/bin/bb-cpu.sh
$1 $TOPARGS > $BBTMP/TOP.$$
# /usr/local/jail/bin/jtop > $BBTMP/TOP.$$

./runbb.sh start
more BBOUT
(look for errors)
exit

echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh

Punch a hole in the firewall to allow it to communicate with bb monitor:

ipfw add 00096 allow ip from 66.181.18.0/27 to 69.55.230.2

27. configure bb on mail:
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
66.181.18.3 firewall3.johncompanies.com # ssh

su bb
cd
bbsrc/bb/runbb.sh restart ; exit

29. configure ntp
echo "server 10.1.2.1" > /etc/ntp.conf

/usr/sbin/ntpd -p /var/run/ntpd.pid
sleep 2; ntpq -p
(confirm it’s able to reach our time server)

echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh
chmod 0700 /usr/local/etc/rc.d/ntp.sh

30. fwd and reverse lookups on ns1c
vr johncompanies.com
(edit the PTR too)

33. setup backups
echo '#\!/bin/sh\
backupdir=/data/firewall2/current\
\
## ENTRY /etc ' > /usr/local/etc/backup.config

on backup3:
setup backup dirs:
ssh backup3 mkdir -p /data/firewall2/current

on backup3, add the system to
vi /usr/local/sbin/snapshot_archive

scp 69.55.230.2:/usr/local/etc/rsync.backup /usr/local/etc/rsync.backup
vi /usr/local/etc/rsync.backup
backup1 > backup3

crontab -e
1 0 * * * /usr/local/etc/rsync.backup

34. mkdir /root/logs

35. edit sshd_config for security
vi /etc/ssh/sshd_config
ListenAddress 66.181.18.3
ListenAddress 10.1.2.5

kill -1 `cat /var/run/sshd.pid`

35. raid chk

cat > /usr/local/sbin/lsiraidchk
#!/usr/bin/perl

my @out = split "\n", `megarc -ldInfo -a0 -Lall|grep Status:`;

foreach (@out) {
if ($_ =~ /DEGRADED/) { $date = `date`; chomp $date; `echo "$date: RAID ARRAY DEGRADED" >> /var/log/messages`; }
#print $_;
}

36. add crontab entries
crontab -e
30 3 * * * /usr/local/etc/rsync.backup
0 0 1 * * /sbin/ipfw zero
0 0 1 * * /sbin/ipfw del 3 4 5
59 23 30 * * /sbin/ipfw show > /tmp/ipfw_count
3 0 30 * * /sbin/ipfw show > /tmp/ipfw_count
0 3 * * * /etc/makefwrules.pl; /etc/makepiperules.pl;
*/5 * * * * /usr/local/sbin/lsiraidchk

#10 0 * * * rm /var/spool/clientmqueue/*

scp /etc/makefwrules.pl user@64.163.14.48:~
scp /etc/makepiperules.pl user@64.163.14.48:~
mv /home/user/makefwrules.pl /etc
mv /home/user/makepiperules.pl /etc
touch /etc/firewall.sh
mkdir /etc/oldrules/

other binaries

scp /usr/local/bin/rulemaker user@64.163.14.48:~
mv ~user/rulemaker /usr/local/sbin
scp ~user/Sendmail.pm user@64.163.14.48:~
scp ~user/doswatch.pl user@64.163.14.48:~

Setup basic ruleset

ipfw add 00009 count udp from any to any
ipfw add 00010 allow tcp from any to any established
ipfw add 00012 deny tcp from any to any tcpflags syn tcpoptions !mss
ipfw add 00012 deny icmp from any to any icmptypes 4,5,9,10,12,13,14,15,16,17,18
ipfw add 00012 deny tcp from any to any tcpflags syn,fin
ipfw add 00012 deny tcp from any to any tcpflags fin,psh,rst,urg
ipfw add 00012 allow icmp from any to any
ipfw add 00014 deny tcp from any to any dst-port 135
ipfw add 00150 skipto 65535 ip from any to any via em1 in

IPKVM3:
00098 allow ip from { 69.55.230.6 or 69.55.230.7 } to 69.55.230.10 dst-port 139
00098 deny ip from any to 69.55.230.10 dst-port 139
</pre>

= wiki =

The wiki (mediawiki) runs on nat2 in a jail running off 69.55.229.8

The backup wiki lives on virt13 in CT 5 / 69.55.230.18

== Setup jail ==

<pre>
mkdir /mnt/data1/wiki-dir
cd /usr/src
make installworld DESTDIR=/mnt/data1/wiki-dir
cd etc
make distribution DESTDIR=/mnt/data1/wiki-dir

mount -t devfs devfs /mnt/data1/wiki-dir/dev
devfs -m /mnt/data1/wiki-dir/dev rule -s 3 applyset

cd /mnt/data1/wiki-dir

ln -sf dev/null kernel

scp jail9:/usr/local/sbin/jkill /mnt/data1/wiki-dir/sbin

jail /mnt/data1/wiki-dir wiki.johncompanies.com 69.55.229.8 /bin/sh
csh

touch /etc/fstab
echo 'network_interfaces=""\
hostname="wiki.johncompanies.com"\
kern_securelevel_enable="NO"\
sendmail_enable="YES"\
sshd_enable="YES"' > /etc/rc.conf

echo "nameserver 69.55.229.3\
nameserver 69.55.225.225" >> /etc/resolv.conf

vi /etc/crontab

(remove the adjkerntz lines )

vi /etc/periodic/security/100.chksetuid

replace: MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`
with: MP='/' (use single quotes)

mkdir -p /usr/compat/linux/dev

adduser

Username: user
Full name: user
Uid (Leave empty for default):
Login group [user]:
Login group is user. Invite user into other groups? []: wheel
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/user]:
Home directory permissions (Leave empty for default):
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: y
Lock out the account after creation? [no]:
Username : user
Password : <random>
Full Name : user
Uid : 1001
Class :
Groups : user
Home : /home/user
Home Mode :
Shell : /bin/sh
Locked : no
OK? (yes/no): y
adduser: INFO: Successfully added (user) to the user database.
adduser: INFO: Password for (user) is: 901gmYjO
Add another user? (yes/no): n
Goodbye!

vi /usr/home/user/.profile
TERM=vt100; export TERM

tzsetup

newaliases

rm /sbin/halt /sbin/reboot
ln /sbin/jkill /sbin/halt
ln /sbin/jkill /sbin/reboot

vi /etc/syslog.conf
#*.err;kern.warning;auth.notice;mail.crit /dev/console
*.err;kern.warning;auth.notice;mail.crit /var/log/messages

exit
exit

cd libexec
chflags noschg ld-elf32.so.1
chflags noschg ld-elf.so.1
mv ld-elf32.so.1 ld-elf32.so.1-orig
ln ld-elf.so.1 ld-elf32.so.1
chflags schg ld-elf.so.1
chflags schg ld-elf32.so.1

cp -r /usr/ports /mnt/data1/wiki-dir/usr

cat > /usr/local/etc/rc.d/wiki.sh
mount -t devfs devfs /mnt/data1/wiki-dir/dev/
devfs -m /mnt/data1/wiki-dir/dev rule -s 3 applyset
jail /mnt/data1/wiki-dir wiki.johncompanies.com 69.55.229.8 /bin/sh /etc/rc

chmod 0700 /usr/local/etc/rc.d/wiki.sh
</pre>

== mediawiki setup ==

<pre>

cd /usr/ports/net/rsync
make install clean

cd /usr/ports/distfiles/
fetch http://downloads.mysql.com/archives/mysql-5.5/mysql-5.5.4-m3.tar.gz
cd /usr/ports/databases/mysql55-server
make install clean

cd /usr/ports/distfiles/
fetch http://downloads.php.net/johannes/php-5.3.2.tar.bz2
cd /usr/ports/lang/php52
make install clean
(build apache module)

cd /usr/ports/lang/php5-extensions
make install clean

cd /usr/ports/www/apache22
make install clean

cd /usr/local/www/
fetch http://download.wikimedia.org/mediawiki/1.19/mediawiki-1.19.2.tar.gz
tar xzf mediawiki-1.19.2.tar.gz
mv mediawiki-1.19.2 wiki

vi /usr/local/etc/apache22/httpd.conf
DocumentRoot "/usr/local/www/"

Include etc/apache22/extra/vhost-wiki.conf
Listen 443

<IfModule mod_php5.c>
<FilesMatch "\.ph(p3?|tml)$">
SetHandler application/x-httpd-php
</FilesMatch>
<FilesMatch "\.phps$">
SetHandler application/x-httpd-php-source
</FilesMatch>
# To re-enable php in user directories comment the following lines
# (from <IfModule ...> to </IfModule>.) Do NOT set it to On as it
# prevents .htaccess files from disabling it.
<IfModule mod_userdir.c>
<Directory /home/*/public_html>
php_admin_value engine Off
</Directory>
</IfModule>
</IfModule>

cat > /usr/local/etc/apache22/extra/vhost-wiki.conf
<VirtualHost *:443>
ServerAdmin support@johncompanies.com

DocumentRoot /usr/local/www/wiki
# <Directory />
# Options FollowSymLinks
# AllowOverride None
# Order deny,allow
# </Directory>
<Directory /usr/local/www/wiki>
Options Indexes FollowSymLinks MultiViews
Deny from all
AllowOverride AuthConfig
Order allow,deny
DirectoryIndex index.php
#Allow from 69.55.233.195
#Allow from boody.dyndns.org
</Directory>

ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/
<Directory "/usr/lib/cgi-bin">
AllowOverride None
Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
Order allow,deny
Allow from all
</Directory>

ErrorLog /var/log/httpd-error.log

# Possible values include: debug, info, notice, warn, error, crit,
# alert, emerg.
LogLevel warn

CustomLog /var/log/httpd-access.log combined

Alias /doc/ "/usr/share/doc/"
<Directory "/usr/share/doc/">
Options Indexes MultiViews FollowSymLinks
AllowOverride None
Order deny,allow
Deny from all
Allow from 127.0.0.0/255.0.0.0 ::1/128
</Directory>

SSLEngine on
SSLCertificateFile /usr/local/etc/apache22/ssl/server.crt
SSLCertificateKeyFile /usr/local/etc/apache22/ssl/server.key

</VirtualHost>

mkdir ssl
cd ssl

openssl req -days 1999 -new -x509 -nodes -out server.crt -keyout server.key
US
CA
San Diego
johncompanies.com
johncompanies.com
wiki.johncompanies.com
support@johncompanies.com

cat > /usr/local/www/wiki/.htaccess
AuthType Basic
AuthUserFile /usr/local/etc/apache22/wiki.passwd
AuthName wiki
require valid-user
satisfy any

cd /usr/local/etc/apache22
htpasswd -c wiki.passwd admin

https://69.55.229.8/index.php

use mysql (innodb)
wiki name: JCWiki
Support / (mail pass) / support@johncompanies.com

cat > /usr/local/www/wiki/LocalSettings.php

<?php
# This file was automatically generated by the MediaWiki 1.19.2
# installer. If you make manual changes, please keep track in case you
# need to recreate them later.
#
# See includes/DefaultSettings.php for all configurable settings
# and their default values, but don't forget to make changes in _this_
# file, not there.
#
# Further documentation for configuration settings may be found at:
# http://www.mediawiki.org/wiki/Manual:Configuration_settings

# Protect against web entry
if ( !defined( 'MEDIAWIKI' ) ) {
exit;
}

## Uncomment this to disable output compression
# $wgDisableOutputCompression = true;

$wgSitename = "JCWiki";

## The URL base path to the directory containing the wiki;
## defaults for all runtime URL paths are based off of this.
## For more information on customizing the URLs please see:
## http://www.mediawiki.org/wiki/Manual:Short_URL
$wgScriptPath = "";
$wgScriptExtension = ".php";

## The protocol and server name to use in fully-qualified URLs
$wgServer = "https://69.55.229.8";

## The relative URL path to the skins directory
$wgStylePath = "$wgScriptPath/skins";

## The relative URL path to the logo. Make sure you change this from the default,
## or else you'll overwrite your logo when you upgrade!
#$wgLogo = "$wgStylePath/common/images/wiki.png";
$wgLogo = "$wgStylePath/common/images/jclogo.gif";

## UPO means: this is also a user preference option

$wgEnableEmail = true;
$wgEnableUserEmail = true; # UPO

$wgEmergencyContact = "apache@69.55.229.8";
$wgPasswordSender = "apache@69.55.229.8";

$wgEnotifUserTalk = false; # UPO
$wgEnotifWatchlist = false; # UPO
$wgEmailAuthentication = true;

## Database settings
$wgDBtype = "mysql";
$wgDBserver = "localhost";
$wgDBname = "my_wiki";
$wgDBuser = "root";
$wgDBpassword = "";

# MySQL specific settings
$wgDBprefix = "";

# MySQL table options to use during installation or update
$wgDBTableOptions = "ENGINE=InnoDB, DEFAULT CHARSET=binary";

# Experimental charset support for MySQL 5.0.
$wgDBmysql5 = false;

## Shared memory settings
$wgMainCacheType = CACHE_NONE;
$wgMemCachedServers = array();

## To enable image uploads, make sure the 'images' directory
## is writable, then set this to true:
$wgEnableUploads = false;
#$wgUseImageMagick = true;
#$wgImageMagickConvertCommand = "/usr/bin/convert";

# InstantCommons allows wiki to use images from http://commons.wikimedia.org
$wgUseInstantCommons = false;

## If you use ImageMagick (or any other shell command) on a
## Linux server, this will need to be set to the name of an
## available UTF-8 locale
$wgShellLocale = "en_US.utf8";

## If you want to use image uploads under safe mode,
## create the directories images/archive, images/thumb and
## images/temp, and make them all writable. Then uncomment
## this, if it's not already uncommented:
#$wgHashedUploadDirectory = false;

## Set $wgCacheDirectory to a writable directory on the web server
## to make your wiki go slightly faster. The directory should not
## be publically accessible from the web.
#$wgCacheDirectory = "$IP/cache";

# Site language code, should be one of the list in ./languages/Names.php
$wgLanguageCode = "en";

$wgSecretKey = "abc699ef26890b49b4055430f8ebbd25e84cce21a7e53aeaec4d4313af4c9739";

# Site upgrade key. Must be set to a string (default provided) to turn on the
# web installer while LocalSettings.php is in place
$wgUpgradeKey = "3196710f4a7d7332";

## Default skin: you can change the default skin. Use the internal symbolic
## names, ie 'standard', 'nostalgia', 'cologneblue', 'monobook', 'vector':
$wgDefaultSkin = "vector";

## For attaching licensing metadata to pages, and displaying an
## appropriate copyright notice / icon. GNU Free Documentation
## License and Creative Commons licenses are supported so far.
$wgRightsPage = ""; # Set to the title of a wiki page that describes your license/copyright
$wgRightsUrl = "";
$wgRightsText = "";
$wgRightsIcon = "";

# Path to the GNU diff3 utility. Used for conflict resolution.
$wgDiff3 = "/usr/bin/diff3";

# Query string length limit for ResourceLoader. You should only set this if
# your web server has a query string length limit (then set it to that limit),
# or if you have suhosin.get.max_value_length set in php.ini (then set it to
# that value)
$wgResourceLoaderMaxQueryLength = -1;

# End of automatically generated settings.
# Add more configuration options below.

</pre>

== copy/backup wiki ==
on main/primary wiki:
<pre>
/usr/local/etc/rc.d/mysql-server stop
ssh 69.55.230.18 "/etc/init.d/mysql stop"
rsync -av /var/db/mysql/my_wiki/ 69.55.230.18:/var/lib/mysql/my_wiki/
rsync -av /var/db/mysql/ib* 69.55.230.18:/var/lib/mysql/
/usr/local/etc/rc.d/mysql-server start
ssh 69.55.230.18 "/etc/init.d/mysql start"

</pre>