VPS Management: Difference between revisions
| Line 409: | Line 409: | ||
== jt == | == jt == | ||
== vwe == | == vwe == | ||
Revision as of 17:48, 28 February 2013
FreeBSD VPS Management tools
These files are located in /usr/local/jail/rc.d and /usr/local/jail/bin
jailps
jailps [hostname]
DEPRECATED FOR jps: displays processes belonging to/running inside a jail. The command takes one (optional) argument – the hostname of the jail you wish to query. If you don’t supply an argument, all processes on the machine are listed and grouped by jail.
jps
jps [hostname]
displays processes belonging to/running inside a jail. The command takes one (optional) argument – the hostname or ID of the jail you wish to query.
jailkill
jailkill <hostname>
stops all process running in a jail.
You can also run:
jailkill <JID>
problems
Occasionally you will hit an issue where jail will not kill off:
jail9# jailkill www.domain.com www.domain.com .. killed: none jail9#
Because no processes are running under that hostname. You cannot use jailps.pl either:
jail9# jailps www.domain.com www.domain.com doesn’t exist on this server jail9#
The reasons for this are usually:
- the jail is no longer running
- the jail's hostname has changed
In this case,
>=6.x: run a jls|grep <jail's IP> to find the correct hostname, then update the quad file, then kill the jail.
<6.x: the first step is to cat their /etc/rc.conf file to see if you can tell what they set the new hostname to. This very often works. For example:
cat /mnt/data2/198.78.65.136-col00261-DIR/etc/rc.conf
But maybe they set the hostname with the hostname command, and the original hostname is still in /etc/rc.conf.
The welcome email clearly states that they should tell us if they change their hostname, so there is no problem in just emailing them and asking them what they set the new hostname to.
Once you know the new hostname OR if a customer simply emails to inform you that they have set the hostname to something different, you need to edit the quad and safe files that their system is in to input the new hostname.
However, if push comes to shove and you cannot find out the hostname from them or from their system, then you need to start doing some detective work.
The easiest thing to do is run jailps looking for a hostname similar to their original hostname. Or you could get into the /bin/sh shell by running:
/bin/sh
and then looking at every hostname of every process:
for f in `ls /proc` ; do cat /proc/$f/status ; done
and scanning for a hostname that is either similar to their original hostname, or that you don't see in any of the quad safe files.
This is very brute force though, and it is possible that catting every file in /proc is dangerous - I don't recommend it. A better thing would be to identify any processes that you know belong to this system – perhaps the reason you are trying to find this system is because they are running something bad - and just catting the status from only that PID.
Somewhere there’s a jail where there may be 2 systems named www. Look at /etc/rc.conf and make sure they’re both really www. If they are, jailkill www, jailps www to make sure not running. Then immediately restart the other one, as the fqdn (as found from a rev nslookup)
- on >=6.x the hostname may not yet be hashed:
jail9 /# jls JID Hostname Path IP Address(es) 1 bitnet.dgate.org /mnt/data1/69.55.232.50-col02094-DIR 69.55.232.50 2 ns3.hctc.net /mnt/data1/69.55.234.52-col01925-DIR 69.55.234.52 3 bsd1 /mnt/data1/69.55.232.44-col00155-DIR 69.55.232.44 4 let2.bbag.org /mnt/data1/69.55.230.92-col00202-DIR 69.55.230.92 5 post.org /mnt/data2/69.55.232.51-col02095-DIR 69.55.232.51 ... 6 ns2 /mnt/data1/69.55.232.47-col01506-DIR 69.55.232.47 ... 7 arlen.server.net /mnt/data1/69.55.232.52-col01171-DIR 69.55.232.52 8 deskfood.com /mnt/data1/69.55.232.71-col00419-DIR 69.55.232.71 9 mirage.confluentforms.com /mnt/data1/69.55.232.54-col02105-DIR 69.55.232.54 ... 10 beachmember.com /mnt/data1/69.55.232.59-col02107-DIR 69.55.232.59 11 www.agottem.com /mnt/data1/69.55.232.60-col02109-DIR 69.55.232.60 12 sdhobbit.myglance.org /mnt/data1/69.55.236.82-col01708-DIR 69.55.236.82 13 ns1.jnielsen.net /mnt/data1/69.55.234.48-col00204-DIR 69.55.234.48 ... 14 ymt.rollingegg.net /mnt/data2/69.55.236.71-col01678-DIR 69.55.236.71 15 verse.unixlore.net /mnt/data1/69.55.232.58-col02131-DIR 69.55.232.58 16 smcc-mail.org /mnt/data2/69.55.232.68-col02144-DIR 69.55.232.68 17 kasoutsuki.w4jdh.net /mnt/data2/69.55.232.46-col02147-DIR 69.55.232.46 18 dili.thium.net /mnt/data2/69.55.232.80-col01901-DIR 69.55.232.80 20 www.tekmarsis.com /mnt/data2/69.55.232.66-col02155-DIR 69.55.232.66 21 vps.yoxel.net /mnt/data2/69.55.236.67-col01673-DIR 69.55.236.67 22 smitty.twitalertz.com /mnt/data2/69.55.232.84-col02153-DIR 69.55.232.84 23 deliver4.klatha.com /mnt/data2/69.55.232.67-col02160-DIR 69.55.232.67 24 nideffer.com /mnt/data2/69.55.232.65-col00412-DIR 69.55.232.65 25 usa.hanyuan.com /mnt/data2/69.55.232.57-col02163-DIR 69.55.232.57 26 daifuku.ppbh.com /mnt/data2/69.55.236.91-col01720-DIR 69.55.236.91 27 collins.greencape.net /mnt/data2/69.55.232.83-col01294-DIR 69.55.232.83 28 ragebox.com /mnt/data2/69.55.230.104-col01278-DIR 69.55.230.104 29 outside.mt.net /mnt/data2/69.55.232.72-col02166-DIR 69.55.232.72 30 vps.payneful.ca /mnt/data2/69.55.234.98-col01999-DIR 69.55.234.98 31 higgins /mnt/data2/69.55.232.87-col02165-DIR 69.55.232.87 ... 32 ozymandius /mnt/data2/69.55.228.96-col01233-DIR 69.55.228.96 33 trusted.realtors.org /mnt/data2/69.55.238.72-col02170-DIR 69.55.238.72 34 jc1.flanderous.com /mnt/data2/69.55.239.22-col01504-DIR 69.55.239.22 36 guppylog.com /mnt/data2/69.55.238.73-col00036-DIR 69.55.238.73 40 haliohost.com /mnt/data2/69.55.234.41-col01916-DIR 69.55.234.41 ... 41 satyr.jorge.cc /mnt/data1/69.55.232.70-col01963-DIR 69.55.232.70 jail9 /# jailkill satyr.jorge.cc ERROR: jail_: jail "satyr,jorge,cc" not found
Note how it's saying satyr,jorge,cc is not found, and not satyr.jorge.cc.
The jail subsystem tracks things using comma-delimited hostnames. That is created every few hours:
jail9 /# crontab -l 0 0,6,12,18 * * * /usr/local/jail/bin/sync_jail_names
So if we run this manually:
jail9 /# /usr/local/jail/bin/sync_jail_names
Then kill the jail:
jail9 /# jailkill satyr.jorge.cc
successfully killed: satyr,jorge,cc
It worked.
jailpsall
jailpsall
will run a jailps on all jails configured in the quad files (this is different from jailps with no arguments as it won’t help you find a “hidden” system)
jailpsw
jailpsw
will run a jailps with an extra -w to provide wider output
jt (>=7.x)
jt
displays the top 20 processes on the server (the top 20 processes from top) and which jail owns them. This is very helpful for determining who is doing what when the server is very busy.
jtop (>=7.x)
jtop
a wrapper for top displaying processes on the server and which jail owns them. Constantly updates, like top.
jtop (<7.x)
jtop
displays the top 20 processes on the server (the top 20 processes from top) and which jail owns them. This is very helpful for determining who is doing what when the server is very busy.
stopjail
stopjail <hostname> [1]
this will jailkill, umount and vnconfig –u a jail. If passed an optional 2nd argument, it will not exit before umounting and un-vnconfig’ing in the event jailkill returns no processes killed. This is useful if you just want to umount and vnconfig –u a jail you’ve already killed. It is intelligent in that it won’t try to umount or vnconfig –u if it’s not necessary.
startjail
startjail <hostname>
this will start vnconfig, mount (including linprocfs and null-mounts), and start a jail. Essentially, it reads the jail’s relevant block from the right quad file and executes it. It is intelligent in that it won’t try to mount or vnconfig if it’s not necessary.
jpid
jpid <pid>
displays information about a process – including which jail owns it. It’s the equivalent of running cat /proc/<pid>/status
canceljail
canceljail <hostname> [1]
this will stop a jail (the equivalent of stopjail), check for backups (offer to remove them from the backup server and the backup.config), rename the vnfile, remove the dir, and edit quad/safe. If passed an optional 2nd argument, it will not exit upon failing to kill and processes owned by the jail. This is useful if you just want to cancel a jail which is already stopped.
jls
jls [-v]
Lists all jails running:
JID #REF IP Address Hostname Path 101 135 69.55.224.148 mail.pc9.org /mnt/data2/69.55.224.148-col01034-DIR
- REF is the number of references or procs(?) running
Running with -v will give you all IPs assigned to each jail (7.2 up)
JID #REF Hostname Path IP Address(es) 101 139 mail.pc9.org /mnt/data2/69.55.224.148-col01034-DIR 69.55.224.14869.55.234.85
startalljails
startalljails
7.2+ only. This will parse through quad1 and start all jails. It utilizes lockfiles so it won’t try to start a jail more than once- therefore multiple instances can be running in parallel without fear of starting a jail twice. If a jail startup gets stuck, you can ^C without fear of killing the script. IMPORTANT- before running startalljails you should make sure you ran preboot once as it will clear out all the lockfiles and enable startalljails to work properly.
aaccheck.sh
aaccheck.sh
displayes the output of container list and task list from aaccli
backup
backup
backup script called nightly to update jail scripts and do customer backups
buildsafe
buildsafe
creates safe files based on quads (automatically removing the fsck’s). This will destructively overwrite safe files
checkload.pl
checkload.pl
this was intended to be setup as a cronjob to watch processes on a jail when the load rises above a certain level. Not currently in use.
checkprio.pl
checkprio.pl
will look for any process (other than the current shell’s csh, sh, sshd procs) with a non-normal priority and normalize it
diskusagemon
diskusagemon <mount point> <1k blocks>
watches a mount point’s disk use, when it reaches the level specified in the 2nd argument, it exits. This is useful when doing a restore and you want to be paged as it’s nearing completion. Best used as: diskusagemon /asd/asd 1234; pagexxx
dumprestore
dumprestore <dumpfile>
this is a perl expect script which automatically enters ‘1’ and ‘y’. It seems to cause restore to fail to set owner permissions on large restores.
g
g <search>
greps the quad/safe files for the given search parameter
gather.pl
gather.pl
gathers up data about jails configured and writes to a file. Used for audits against the db
gb
gb <search>
greps backup.config for the given search parameter
gbg
gbg <search>
greps backup.config for the given search parameter and presents just the directories (for clean pasting)
ipfwbackup
ipfwbackup
writes ipfw traffic count data to a logfile
ipfwreset
ipfwreset
writes ipfw traffic count data to a logfile and resets counters to 0
js
js
output varies by OS version, but generally provides information about the base jail: - which vn’s are in use - disk usage - info about the contents of quads - the # of inodes represented by the jails contained in the group (133.2 in the example below), and how many jails per data mount, as well as subtotals - ips bound to the base machine but not in use by a jail - free gvinum volumes, or unused vn’s or used md’s
/usr/local/jail/rc.d/quad1:
/mnt/data1 133.2 (1)
/mnt/data2 1040.5 (7)
total 1173.7 (8)
/usr/local/jail/rc.d/quad2:
/mnt/data1 983.4 (6)
total 983.4 (6)
/usr/local/jail/rc.d/quad3:
/mnt/data1 693.4 (4)
/mnt/data2 371.6 (3)
total 1065 (7)
/usr/local/jail/rc.d/quad4:
/mnt/data1 466.6 (3)
/mnt/data2 882.2 (5)
total 1348.8 (8)
/mnt/data1: 2276.6 (14)
/mnt/data2: 2294.3 (15)
Available IPs:
69.55.230.11 69.55.230.13 69.55.228.200
Available volumes:
v78 /mnt/data2 2G
v79 /mnt/data2 2G
v80 /mnt/data2 2G
load.pl
load.pl
feeds info to load mrtg - executed by inetd.
makevirginjail
makevirginjail
Only on some systems, makes an empty jail (doesn't do restore step)
mb
mb <mount|umount>
(nfs) mounts and umounts dirs to backup2. Shortcuts are mbm and mbu to mount and unmount.
notify.sh
notify.sh
emails reboot@johncompanies.com – intended to be called at boot time to alert us to a machine which panics and reboots and isn’t caught by bb or castle.
orphanedbackupwatch
orphanedbackupwatch
looks for directories on backup2 which aren’t configured in backup.config and offers to delete them
postboot
postboot
to be run after a machine reboot and quad/safe’s are done executing. It will:
- do chmod 666 on each jail’s /dev/null
- add ipfw counts
- run jailpsall (so you can see if a configured jail isn’t running)
preboot
preboot
to be run before running quad/safe – checks for misconfigurations:
- a jail configured in a quad but not a safe
- a jail is listed more than once in a quad
- the ip assigned to a jail isn’t configured on the machine
- alias numbering skips in the rc.conf (resulting in the above)
- orphaned vnfile's that aren't mentioned in a quad/safe
- ip mismatches between dir/vnfile name and the jail’s ip
- dir/vnfiles's in quad/safe that don’t exist
quadanalyze.pl
quadanalyze.pl
called by js, produces the info (seen above with js explanation) about the contents of quad (inode count, # of jails, etc.)
rsync.backup
rsync.backup
does customer backups (relies on backup.config)
taskdone
taskdone
when called will email support@johncompanies.com with the hostname of the machine from which it was executed as the subject
topten
topten
summarizes the top 10 traffic users (called by ipfwreset)
trafficgather.pl
trafficgather.pl [yy-mm]
sends a traffic usage summary by jail to support@johncomapnies.com and payments@johncompanies.com. Optional arguments are year and month (must be in the past). If not passed, assumes last month. Relies on traffic logs created by ipfwreset and ipfwbackup
trafficwatch.pl
trafficwatch.pl
checks traffic usage and emails support@johncomapnies.com when a jail reaches the warning level (35G) and the limit (40G). We really aren’t using this anymore now that we have netflow.
trafstats
trafstats
writes ipfw traffic usage info by jail to a file called jc_traffic_dump in each jail’s / dir
truncate_jailmake
truncate_jailmake
a version of jailmake which creates truncated vnfiles.
vb
vb
the equivalent of: vi /usr/local/jail/bin/backup.config
vs (freebsd)
vs<n>
the equivalent of: vi /usr/local/jail/rc.d/safe<n>
vq<n> the equivalent of: vi /usr/local/jail/rc.d/quad<n>
dumpremote
dumpremote <user@machine> </remote/location/file-dump> <vnX>
ex: dumpremote user@10.1.4.117 /mnt/data3/remote.echoditto.com-dump 7 this will dump a vn filesystem to a remote machine and location
oversellcheck
oversellcheck
displays how much a disk is oversold or undersold taking into account truncated vn files. Only for use on 4.x systems
mvbackups (freebsd)
mvbackups <dir> (1.1.1.1-col00001-DIR) <target_machine> (jail1) <target_dir> (data1)
moves backups from one location to another on the backup server, and provides you with option to remove entries from current backup.config, and simple paste command to add the config to backup.config on the target server
jailnice
jailnice <hostname>
applies renice 19 [PID] and rtprio 31 –[PID] to each process in the given jail
dumpremoterestore
dumpremoterestore <device> <ip of target machine> <dir on target machine>
ex: dumpremoterestore /dev/vn51 10.1.4.118 /mnt/data2/69.55.239.45-col00688-DIR dumps a device and restores it to a directory on a remote machine. Requires that you enable root ssh on the remote machine.
psj shows just the procs running on the base system – a ps auxw but without jail’d procs present
perc5iraidchk checks for degraded arrays on Dell 2950 systems with Perc5/6 controllers
perc4eraidchk checks for degraded arrays on Dell 2850 systems with Perc4e/Di controllers