Virtuozzo Server Install: Difference between revisions

From JCWiki
Jump to navigation Jump to search
Created page with "= 4.7 = = Older versions = == 4.0 == Last updated 2010-03-11 <pre>1. enable post info to console 2950: Serial comm.: on with cons. Redir via com1 Failsafe: 115200 Redir a..."
 
 
(16 intermediate revisions by 2 users not shown)
Line 1: Line 1:
= 4.7 =
= 4.7 =
Server-specific changes in <span style="color:red">red</span>
http://www.parallels.com/download/pvc/lin
http://www.parallels.com/support/virtualization-suite/pvc/
== BIOS ==
* enable post info to console
2950:<br>
Serial comm.: on with cons. Redir via com1<br>
Failsafe: 115200<br>
Redir after boot: enabled
* Tweak DRAC settings (see [[DRAC/RMM#DRAC_setup|DRAC setup]])
== Install OS ==
CentOS6.x (get latest from http://mirrors.kernel.org/centos/6.4/isos/x86_64/CentOS-6.4-x86_64-netinstall.iso)
May have to disable DRAC virtual media to make sure we get sda as our first device
Set partitions:
<pre>Disk 1:
/ 10240M (ext4)
9216M swap
/vz (the rest) (ext4)
Disk 2:
9216M Swap
/vz1 (the rest) (ext4)</pre>
Basic server
Use grub, install to MBR
== Configure system, install virtuozzo ==
* add ssh
yum install openssh-clients
* create ssh key, add swsoft key to authorized_keys
(enter to everything)
<pre>cd /root/
ssh-keygen -t dsa
Add vz support key:
echo "ssh-dss AAAAB3NzaC1kc3MAAACBAIB7XHmpcsa1D/vsKS+mzEvBVo7udew/aRahbjnjywL4YXT7ruWzpGo7R3UA8GKIgVHoQGUbuXrJz5ICtBEXOlbgXVEz7Bu5nXOG2C9i/8pmFtUw6GwACII8uZE7t/Zq5CRj+5zN1Rv7n/4X4RAjzk5BHXoZ62a32FTbS4tPvyPrAAAAFQDiUi7YEiffNzGvB544lQpRdn29TwAAAIBkUchLTBE1+sX11iC80yhNh9EGeCMLdkT8UqHsTSaD0myQPxGALuNY4rO31q+25uKqSRDTXd6ZWXp4bplA6Ls6jl7r7XVR9loJ7WvRMqc4Dv68TZ7h2UBqAhu4det2UF4LHim3DNf1gKn9DzWNkmI/N2FvVVAL4aM3sPunVIvUYgAAAIAJX38INABVgnCEBKqpv5FMLm3k9MgMn54oQFmylBbgM0w4SUeuDj3jLrkJf8dT9w8ByEDb8xviexpWrPsbz2DeThle4xgzcyKrUkt2Wugtmm4DwS0S/68BAA9wXoWD/vZ67GBoCKH7uq8VAjUVcCbJ/7IydljnFAZoJpktdwN7iQ== russup@support.sw-soft.com" >> /root/.ssh/authorized_keys</pre>
* setup access to backup2:
<pre>echo "10.1.4.3    backup2" >> /etc/hosts
echo "10.1.4.8        backup1" >> /etc/hosts</pre>
# Copy id_dsa.pub to backup2 and add it to the end of ~root/.ssh/authorized_keys:
<pre>cat .ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat .ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'</pre>
<pre>ssh backup2 hostname
ssh backup1 hostname</pre>
(confirm you can connect w/o entering a password)
* add bash backward history search
<pre>echo "\"\M-z\": history-search-backward
\"\M-x\": history-search-backward " > /root/.inputrc
echo "export GREP_OPTIONS='--color=auto'" >> ~/.bashrc
echo "\"\e[5~\": history-search-backward" >> ~/.inputrc
echo "\"\e[6~\": history-search-forward" >> ~/.inputrc</pre>
* add to /root/.bashrc, create log dir
<pre>vi /root/.bashrc
alias v='vzctl'
alias mbm='mb mount'
alias mbu='mb umount'
export PS1="[\u@\h \w]# "
alias cv='cd /vz/private'
alias cv1='cd /vz1/private'
alias cv2='cd /vz2/private'
alias pg='ps auxw|grep $1'</pre>
source /root/.bashrc
mkdir /root/logs
* make sure serial console and kernel messages are working
add these 2 lines to the top:
<pre>vi /boot/grub/menu.lst
serial --unit=0 --speed= 115200
terminal --timeout=10 serial console</pre>
and the kernel line should look like:
kernel /boot/vmlinuz-2.4.20-020stab009.24.777-enterprise ro root=/dev/sda1 console=tty0 console=ttyS0,115200
Remove or comment out these lines:
<pre>hiddensplashmenu
splashimage=(hd0,0)/boot/grub/splash.xpm.gz</pre>
add to /etc/inittab and /etc/securetty
<pre>echo "s0:2345:respawn:/sbin/agetty -L ttyS0 115200 linux" >> /etc/inittab
echo "ttyS0" >> /etc/securetty</pre>
kill -1 1
* make sure the hostname on the system is properly set - in the form virt4.johncompanies.com - and make sure it is set properly:
cat /etc/sysconfig/network
* Check/update fstab
vi /etc/fstab
/etc/fstab entries should look something like (note noatime option):
<pre>/dev/sda3              /vz                    ext4    defaults,noatime 1 2
/dev/sdb2              /vz1                    ext4    defaults,noatime 1 2</pre>
make sure swap is configured in fstab
Use real devices instead of LABEL
* Install Virtuozzo 4.7
Have a license on hand - request a day ahead of the install. Go to:
http://www.parallels.com/download/pvc47/
<pre>yum install compat-libstdc++-33
yum install wget
yum install ftp
yum install pciutils
yum install ed bc
cd /vz
wget http://download.parallels.com/pvc/47/lin/vzinstall-linux-x86_64.bin
chmod 0700 vzinstall-linux-x86_64.bin
./vzinstall-linux-x86_64.bin</pre>
download and install
change wd's to /vz
Ignore OS version error
Custom install (don’t install templates)
Install PVA Agent
Install PVA Management Node
reboot
Get on KVM to watch PVA install (won't happen on console)
vzup2date
(run again. till no more updates offered)
reboot
<DEPRECATED>
Run vzup2date (make sure to choose enterprise kernel) and/or install latest kernel:
NOTE: skip over the manual kernel load below if you can get the 16G recognized before running vzup2date- then it will dl the ent kernel
NOTE: go to /vz/vzup2date/virtuozzo/linux/i386/4.0.0/CU-2.6.18-028stab068.3 and install kern there (downloaded by vzup2date)
vi /boot/grub/menu.lst
(set default kernel to new kernel)
</DEPRECATED>
* Install ntp
yum install ntp
(if not already installed)
rm -f /etc/ntp.conf
Create a new file with one line:
echo "server 10.1.4.5" > /etc/ntp.conf
Then set ntpd so it will start at boot:
chkconfig --level 3 ntpd on
service ntpd start
Make sure it’s talking to the ntp server:
ntpq -p
Correct output should have something unfer the refid col and a '+' next to the remote server:
    <pre>remote          refid      st t when poll reach  delay  offset  jitter
==============================================================================
10.1.4.105+    clock1.redhat.c  2 u    -  64    1    0.370  -706677  0.008</pre>
* copy over binaries/scripts from backup2
scp backup2:/mnt/data4/bin/linux/* /usr/local/sbin/
scp <span style="color:red">10.1.4.62</span>:"/vzconf/jctmpl*" /etc/sysconfig/vz-scripts/
scp <span style="color:red">10.1.4.62</span>:/vzconf/ve-slm* /etc/sysconfig/vz-scripts/
* setup root crontab (alter backup time):
<pre>crontab -e
1 1 * * * /usr/local/sbin/backup
*/15 * * * * /usr/local/sbin/vim.watchdog.30 > /dev/null
#56 4,16 * * * root /usr/sbin/vzkeyupdate -c
*/5 * * * * /usr/local/sbin/perc5iraidchk64</pre>
* setup backups
<pre>
echo '#!/bin/sh
<tt>backupdir=/data/<span style="color:red">virt13</span> </tt>
server=backup1
## ENTRY /etc
## ENTRY /usr/local/sbin
## ENTRY /root/logs
' > /usr/local/sbin/backup.config
</pre>
* setup bigbrother
edit passwd and add the bb user:
echo "bb:x:1984:1984:Big Brother:/home/bb:/bin/bash" >> /etc/passwd
edit group and add the bb group:
echo "bb:x:1984:" >> /etc/group
rebuild the password database:
pwconv
Create bb’s home dir and set ownership:
mkdir /home/bb
chown bb.bb /home/bb
Go to bb’s home and fetch the client tarball:
<pre>cd ~bb
scp backup2:/mnt/data4/build/bb/GOOD-bb-linux-bbc1.9i-btf.tgz .
tar xvf GOOD-bb-linux-bbc1.9i-btf.tgz
yum install glibc-2.12-1.47.el6_2.5.i686
vi etc/bbsys.sh
SORT="/bin/sort"
vi bin/bb-disk.sh
eval "$DFCMD | $EGREP \"$DFUSE\" | $EGREP -v \"$DFEXCLUDE\" | $SORT \"-k${DFSORT}\" > $BBTMP/DFOUT.$$"
wclen1=`eval "$DFCMD | $EGREP \"$DFUSE\" | $EGREP -v \"$DFEXCLUDE\" | $SORT \"-k${DFSORT}\" | $WC"`
cd bbc1.9i-btf/etc/
echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh" > /home/bb/bbc1.9i-btf/etc/bb-hosts
echo "10.1.4.63 virt13.johncompanies.com # ssh" >> /home/bb/bbc1.9i-btf/etc/bb-hosts
(^^^change to correct host)
chmod +r /var/log/messages
cd ..
su bb
./runbb.sh start
exit</pre>
Add this line to /etc/rc.d/rc.local so bb will start at boot:
echo "su - bb -c \"cd /home/bb/bbc1.9i-btf; ./runbb.sh start\"" >> /etc/rc.d/rc.local
* Uncomment/add lines with the appropriate IP addresses in /etc/ssh/sshd_config:
vi /etc/ssh/sshd_config
<tt>ListenAddress <span style="color:red">69.55.226.2</span>
ListenAddress <span style="color:red">10.1.4.63</span></tt>
<DEPRECATED>
And make sure to uncomment
Protocol 2,1
</DEPRECATED>
* setup backup1/backup2 nfs mounts
<DEPRECATED>
setup portmap to start with server, and start it:
chkconfig --level 3 portmap on
service portmap start
</DEPRECATED>
mb mount
run df to confirm that nfs mounts are mounted
* make alias for /vzconf
ln -s /etc/sysconfig/vz-scripts /vzconf
* edit /etc/sysconfig/vz
vi /etc/sysconfig/vz
calc cpu to reserver for HN. take 5% of power:
vzcpucheck
Current CPU utilization: 27000
Power of the node: 1018320
so in this case:
VE0CPUUNITS=50916
enable fastboot:
VZFASTBOOT=yes
Disable auto mount
#VZMOUNTS="/vz"
Enable slm:
SLM=yes
<DEPRECATED>
Enable IPV6
IPV6=yes
</DEPRECATED>
add/edit to traffic parameters section:
TRAFFIC_SHAPING=yes
BANDWIDTH="eth0:102400"
TOTALRATE="eth0:1:102400"
RATE="eth0:1:50000"
for 2 nics, uncomment this line:
VE_ROUTE_SRC_DEV="eth0"
Make sure the IPTABLES section looks like:
<pre>IPTABLES="ip_tables iptable_filter iptable_mangle ipt_limit ipt_multiport ipt_tos ipt_TOS ipt_REJECT ipt_tcpmss ipt_ttl ipt_LOG ipt_length ip_conntrack ip_conntrack_ftp  ip_conntrack_irc ipt_conntrack ipt_state ipt_helper iptable_nat ip_nat_ftp ip_nat_irc ipt_REDIRECT ipt_multiport ipt_TCPMSS"</pre>
* disable periodic fsck's, lower the root padding on the /vz and /vz1 filesystems to 100MB
<pre>
tune2fs -i 0 /dev/sda3
tune2fs -c -1 /dev/sda3
tune2fs -r 100000 /dev/sda3
tune2fs -i 0 /dev/sdb2
tune2fs -c -1 /dev/sdb2
tune2fs -r 100000 /dev/sdb2</pre>
* setup vzagent with pub IP (to allow migrations from 3.x):
vzctl set 50 --ipdel all --ipadd <span style="color:red">69.55.226.3</span> --userpasswd vzagent0:k0zub1k --save
vzagent_ctl restart
* edit ve config files:
From split of 80 take
<tt>CPUUNITS</tt>
And put into files in /vzconf/base_ct_conf
along with:
<pre>VERSION="2"
ONBOOT="yes"
KMEMSIZE="2147483647:2147483647"
LOCKEDPAGES="2147483647:2147483647"
PRIVVMPAGES="2147483647:2147483647"
SHMPAGES="2147483647:2147483647"
NUMPROC="32567:32567"
VMGUARPAGES="2147483647:2147483647"
OOMGUARPAGES="2147483647:2147483647"
NUMTCPSOCK="2147483647:2147483647"
NUMFLOCK="2147483647:2147483647"
NUMPTY="255:255"
NUMSIGINFO="1024:1024"
TCPSNDBUF="2147483647:2147483647"
TCPRCVBUF="2147483647:2147483647"
OTHERSOCKBUF="2147483647:2147483647"
DGRAMRCVBUF="2147483647:2147483647"
NUMOTHERSOCK="2147483647:2147483647"
DCACHESIZE="2147483647:2147483647"
NUMFILE="2147483647:2147483647"
NUMIPTENT="2147483647:2147483647"
DISKSPACE=""
DISKINODES=""
QUOTATIME="600"
IOPRIO="4"
CPULIMIT="75"
OFFLINE_MANAGEMENT="yes"
ARCH="x86"
PLATFORM="linux"
IP_ADDRESS="0.0.0.0"
NAMESERVER="69.55.225.225 69.55.230.3"
CPUUNITS="12313"
IOLIMIT="10485760"
IOPSLIMIT="100"
</pre>
* customize the control panel:
rm -fr /vz/backups
vi /var/opt/pva/pp/xsl/lib/vt/backups-forms.xsl
<pre><!--No backups created yet. You can create a maximum of %max backup(s). -->
                                        <span style="font-size:13px; font-weight:bold">NOTE: You cannot create or view backups via this control panel- IT WILL NOT WORK. Please contact linux@johncompanies.com to setup or inquire about backups for your server.</span>
/var/opt/pva/pp/xsl/lib/vt/backups-list.xsl
                <!--
                                <ui:button tip="backup_ve" sandclock="false" icon="backup_new"
                                        submit-href="{$baseurl}/vt/backups/new?cmd=backup&amp;env-set=current"
                                        disabled="{boolean(not($new-allowed))}" require-items="false"
                                        new-obj-title="{sw:text('Backup')}"
                                >
                                        <sw:text>New Backup</sw:text>
                                </ui:button>
--></pre>
cd  /var/opt/pva/pp/static/vz/skins
for f in `ls`; do scp backup2:/d4/build/vzcp/var/vzcp/static/vz/skins/winxp.new/images/logo_main_4.7.gif $f/images/logo_main.gif; done
* template install
To install new templates:
vzup2date -z
Copy over templates from old server. Ex: ubu 8.04 from virt19:
rsync -a -e ssh <span style="color:red">10.1.4.69</span>:/vz/template/ubuntu/8.04 /vz/template/ubuntu/8.04
install ubuntu 8.04 templates (get latest from vz)
<pre>cd /backup4/vzrpms/distro/ubuntu/8.04
rpm -ivh *
vzpkg update cache ubuntu-8.04-x86</pre>
If you install a tempalte from vzup2date it will download an old cache as well. Better to download the OS rpm from VZ site then create cache then use vzup2date to install the application templates.
* add traffic shaping support
Re-define class 0:
<pre>vi /etc/sysconfig/vz-scripts/networks_classes
0 69.55.224.0/20
0 10.1.4.0/24</pre>
make sure 1 is defined (0.0.0.0) and no other classes are needed
Then start it
/etc/init.d/vz shaperrestart
* udp dos fix
echo "/usr/local/sbin/iptables.sh" >> /etc/rc.d/rc.local
* make rc.local start before vz
mv /etc/rc.d/rc3.d/S99local /etc/rc.d/rc3.d/S95local
* disable unwanted services/daemons
Before:
<pre>abrt-ccpp      0:off  1:off  2:off  3:on    4:off  5:on    6:off
abrt-oops      0:off  1:off  2:off  3:on    4:off  5:on    6:off
abrtd          0:off  1:off  2:off  3:on    4:off  5:on    6:off
acpid          0:off  1:off  2:on    3:on    4:on    5:on    6:off
atd            0:off  1:off  2:off  3:on    4:on    5:on    6:off
auditd          0:off  1:off  2:on    3:on    4:on    5:on    6:off
autofs          0:off  1:off  2:off  3:on    4:on    5:on    6:off
cpuspeed        0:off  1:on    2:on    3:on    4:on    5:on    6:off
crond          0:off  1:off  2:on    3:on    4:on    5:on    6:off
cups            0:off  1:off  2:on    3:on    4:on    5:on    6:off
haldaemon      0:off  1:off  2:off  3:on    4:on    5:on    6:off
ip6tables      0:off  1:off  2:on    3:on    4:on    5:on    6:off
iptables        0:off  1:off  2:on    3:on    4:on    5:on    6:off
irqbalance      0:off  1:off  2:off  3:on    4:on    5:on    6:off
kdump          0:off  1:off  2:off  3:on    4:on    5:on    6:off
lvm2-monitor    0:off  1:on    2:on    3:on    4:on    5:on    6:off
mcelogd        0:off  1:off  2:off  3:on    4:off  5:on    6:off
mdmonitor      0:off  1:off  2:on    3:on    4:on    5:on    6:off
messagebus      0:off  1:off  2:on    3:on    4:on    5:on    6:off
netfs          0:off  1:off  2:off  3:on    4:on    5:on    6:off
network        0:off  1:off  2:on    3:on    4:on    5:on    6:off
nfslock        0:off  1:off  2:off  3:on    4:on    5:on    6:off
ntpd            0:off  1:off  2:off  3:on    4:off  5:off  6:off
portreserve    0:off  1:off  2:on    3:on    4:on    5:on    6:off
postfix        0:off  1:off  2:on    3:on    4:on    5:on    6:off
pvaagentd      0:off  1:off  2:on    3:on    4:on    5:on    6:off
pvapp          0:off  1:off  2:on    3:on    4:on    5:on    6:off
qpidd          0:off  1:off  2:on    3:on    4:on    5:on    6:off
rpcbind        0:off  1:off  2:on    3:on    4:on    5:on    6:off
rpcgssd        0:off  1:off  2:off  3:on    4:on    5:on    6:off
rpcidmapd      0:off  1:off  2:off  3:on    4:on    5:on    6:off
rsyslog        0:off  1:off  2:on    3:on    4:on    5:on    6:off
sshd            0:off  1:off  2:on    3:on    4:on    5:on    6:off
sysstat        0:off  1:on    2:on    3:on    4:on    5:on    6:off
udev-post      0:off  1:on    2:on    3:on    4:on    5:on    6:off
vz              0:off  1:off  2:on    3:on    4:on    5:on    6:off
vzlmond        0:off  1:off  2:on    3:on    4:on    5:on    6:off
vznetcfg        0:off  1:off  2:on    3:on    4:on    5:on    6:off
xinetd          0:off  1:off  2:off  3:on    4:on    5:on    6:off</pre>
<pre>chkconfig --level 3 cpuspeed off
chkconfig --level 3 cups off
chkconfig --level 3 haldaemon off
chkconfig --level 3 ip6tables off
chkconfig --level 3 iptables off
chkconfig --level 3 irqbalance off
chkconfig --level 3 lvm2-monitor off
chkconfig --level 3 mdmonitor off
chkconfig --level 3 messagebus off
chkconfig --level 3 netfs off
chkconfig --level 3 nfslock off
chkconfig --level 3 qpidd off
chkconfig --level 3 rpcgssd off
chkconfig --level 3 rpcidmapd off</pre>
* load mrtg graphs
yum install xinetd
create /etc/xinetd.d/load with the following contents:
<pre>echo "service load
{
        disable = no
        flags          = REUSE
        socket_type    = stream
        wait            = no
        user            = root
        server          = /usr/local/sbin/load.pl
}" > /etc/xinetd.d/load</pre>
Add this line to /etc/services:
echo "load            12384/tcp" >> /etc/services
* Tasks to do on backup1:
setup backup dirs:
mkdir -p /data/<span style="color:red">virt13</span>/0
for f in 0 1 2 3 4 5 6; do mkdir -p /data/<span style="color:red">virt13</span>/$f/vz/private; mkdir -p /data/<span style="color:red">virt13</span>/$f/vz1/private; mkdir -p /data/<span style="color:red">virt13</span>/$f/vz2/private; done
add the system to
vi /usr/local/sbin/snapshot_rotate
* Tasks to do on mail:
Update backup graph:
vi /usr/local/www/mgmt/cgi/backupgraph.pl
(add hostname)
add to bbhosts:
<pre>vi /usr/home/bb/bbsrc/bb1.9i-btf/etc/bb-hosts
su bb
cd
bbsrc/bb/runbb.sh restart ; exit</pre>
add to load graph config:
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg
(add new section for new machine)
* add to management db (on mail and devweb) jc.ref_machines and jc.ref_templates
insert new server:
insert into ref_machines values (null,'virt19','virt19.johncompanies.com',0,'l');
get new machine_id back out:
select * from ref_machines;
insert as a system to jc customer col00001:
insert into systems values ('', 'col00001',30,null,null,100,'virt11.johncompanies.com',null,'2010-03-11',null,null,1,0,null,null,null);
get sysid:
select * from systems where hostname='virt19.johncompanies.com';
update ipmap (main IP and IP used for VZCP):
update ipmap set cid='col00001', sysid=3772, assign_date='2008-01-07', status=1 where ip='69.55.236.2';
update ipmap set cid='col00001', sysid=3772, assign_date='2008-01-07', status=1 where ip='69.55.230.109';
add templates (or do via mgmt):
<pre>
insert into ref_templates values ('','fedora-core-1/20050725',33,'Fedora Core 1',1);
insert into ref_templates values ('','fedora-core-2/20050725',33,'Fedora Core 2',1);
insert into ref_templates values ('','debian-3.1/20050927',33,'Debian 3.1',1);</pre>
* Make sure that ns1c has proper forward and reverse entries for the new system (both private and public addresses), so that new system emailing will work.
<pre>vr johncompanies.com
vi internal.johncompanies.com
rndc reload johncompanies.com IN private</pre>
(edit the PTR too)
* add an outside blocking rule to the firewall, so this machine can only be reached from inside the firewall.
Follow example already in firewall virt19 is:
<pre>ipfw add 00143 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 62.213.72.101 or 64.131.90.27 or 82.200.65.190 or 91.204.24.0/22 or 194.44.160.178 or 195.214.232.10 or 195.214.233.10 or 208.251.212.1 } to 69.55.226.2
ipfw add 00143 deny ip from any to 69.55.226.2</pre>
(virt12 would be 00142)
* add to [[Cabinetmap|Cabinetmap]]
* make /vz1/private
<pre>mkdir /vz1/private ; chmod 0755 /vz1/private
mkdir /vz1/root ; chmod 0755 /vz1/root</pre>
* put config files in a place they’ll get backed up
<pre>mkdir /vzconf/confs
crontab -e
0 0 * * * rm -f /vzconf/confs/*.conf; cp  /vzconf/*.conf /vzconf/confs</pre>
* fix pass/vzpp and localse issue with ubu 8.10
vi /vz/template/ubuntu/8.10/x86/config/os/default/post-install
sed '/^password.*pam_unix.so.*sha512/ s~sha512~md5~' -i etc/pam.d/common-password
localedef -i en_US -c -f UTF-8 en_US.UTF-8
* add tun device support
echo "modprobe tun" >> /etc/rc.local
* reinstall VZA/CT #1/CT #50
After the install, the PP was broken. So needed to reinstall everything. Did the pva_setup --uninstall and removed CT 50 and then setup the pva per http://kb.parallels.com/113071 then setup the CT 50 per the other article- centos6 had issues - and then authenticated with virt13 to get that into the management list.
http://kb.parallels.com/113072
http://kb.parallels.com/113071
* config logrotate (otherwise it creates weekly empty logs)
vi /etc/logrotate.conf
(add to end)
<pre>/var/log/messages {
        rotate 12
        monthly
        size 10M
        missingok
        compress
        delaycompress
        notifempty
        sharedscripts
        postrotate
                echo `date` > /var/log/messages
        endscript
}</pre>
* install megacli64 (can we just move the bin and not install the lib?)
<pre>scp backup2: /d4/build/3ware/MegaCli-8.00.40-1.i386.rpm /root/MegaCli-8.00.40-1.i386.rpm
scp backup2: /d4/build/3ware/Lib_Utils-1.00-08.noarch.rpm /root/ Lib_Utils-1.00-08.noarch.rpm
rpm -ivh Lib_Utils-1.00-08.noarch.rpm
rpm -ivh MegaCli-8.00.40-1.i386.rpm
cp opt/MegaRAID/MegaCli/MegaCli64 /usr/local/sbin/megacli64</pre>


= Older versions =
= Older versions =

Latest revision as of 10:56, 7 December 2013

4.7[edit]

Server-specific changes in red

http://www.parallels.com/download/pvc/lin

http://www.parallels.com/support/virtualization-suite/pvc/

BIOS[edit]

  • enable post info to console

2950:
Serial comm.: on with cons. Redir via com1
Failsafe: 115200
Redir after boot: enabled



Install OS[edit]

CentOS6.x (get latest from http://mirrors.kernel.org/centos/6.4/isos/x86_64/CentOS-6.4-x86_64-netinstall.iso)

May have to disable DRAC virtual media to make sure we get sda as our first device

Set partitions: 

Disk 1:
/ 10240M (ext4)
9216M swap
/vz (the rest) (ext4)
Disk 2: 
9216M Swap
/vz1 (the rest) (ext4)

Basic server

Use grub, install to MBR

Configure system, install virtuozzo[edit]

  • add ssh
yum install openssh-clients
  • create ssh key, add swsoft key to authorized_keys

(enter to everything) 

cd /root/
ssh-keygen -t dsa

Add vz support key:
echo "ssh-dss AAAAB3NzaC1kc3MAAACBAIB7XHmpcsa1D/vsKS+mzEvBVo7udew/aRahbjnjywL4YXT7ruWzpGo7R3UA8GKIgVHoQGUbuXrJz5ICtBEXOlbgXVEz7Bu5nXOG2C9i/8pmFtUw6GwACII8uZE7t/Zq5CRj+5zN1Rv7n/4X4RAjzk5BHXoZ62a32FTbS4tPvyPrAAAAFQDiUi7YEiffNzGvB544lQpRdn29TwAAAIBkUchLTBE1+sX11iC80yhNh9EGeCMLdkT8UqHsTSaD0myQPxGALuNY4rO31q+25uKqSRDTXd6ZWXp4bplA6Ls6jl7r7XVR9loJ7WvRMqc4Dv68TZ7h2UBqAhu4det2UF4LHim3DNf1gKn9DzWNkmI/N2FvVVAL4aM3sPunVIvUYgAAAIAJX38INABVgnCEBKqpv5FMLm3k9MgMn54oQFmylBbgM0w4SUeuDj3jLrkJf8dT9w8ByEDb8xviexpWrPsbz2DeThle4xgzcyKrUkt2Wugtmm4DwS0S/68BAA9wXoWD/vZ67GBoCKH7uq8VAjUVcCbJ/7IydljnFAZoJpktdwN7iQ== russup@support.sw-soft.com" >> /root/.ssh/authorized_keys


  • setup access to backup2:
echo "10.1.4.3    backup2" >> /etc/hosts
echo "10.1.4.8        backup1" >> /etc/hosts
  1. Copy id_dsa.pub to backup2 and add it to the end of ~root/.ssh/authorized_keys:
cat .ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys' 
cat .ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'
ssh backup2 hostname
ssh backup1 hostname

(confirm you can connect w/o entering a password)


  • add bash backward history search
echo "\"\M-z\": history-search-backward
\"\M-x\": history-search-backward " > /root/.inputrc
echo "export GREP_OPTIONS='--color=auto'" >> ~/.bashrc
echo "\"\e[5~\": history-search-backward" >> ~/.inputrc
echo "\"\e[6~\": history-search-forward" >> ~/.inputrc


  • add to /root/.bashrc, create log dir
vi /root/.bashrc
alias v='vzctl'
alias mbm='mb mount'
alias mbu='mb umount'
export PS1="[\u@\h \w]# "
alias cv='cd /vz/private'
alias cv1='cd /vz1/private'
alias cv2='cd /vz2/private'
alias pg='ps auxw|grep $1'
source /root/.bashrc

mkdir /root/logs


  • make sure serial console and kernel messages are working

add these 2 lines to the top: 

vi /boot/grub/menu.lst
serial --unit=0 --speed= 115200
terminal --timeout=10 serial console

and the kernel line should look like: 

kernel /boot/vmlinuz-2.4.20-020stab009.24.777-enterprise ro root=/dev/sda1 console=tty0 console=ttyS0,115200

Remove or comment out these lines: 

hiddensplashmenu 
splashimage=(hd0,0)/boot/grub/splash.xpm.gz

add to /etc/inittab and /etc/securetty 

echo "s0:2345:respawn:/sbin/agetty -L ttyS0 115200 linux" >> /etc/inittab
echo "ttyS0" >> /etc/securetty
kill -1 1


  • make sure the hostname on the system is properly set - in the form virt4.johncompanies.com - and make sure it is set properly:
cat /etc/sysconfig/network


  • Check/update fstab
vi /etc/fstab

/etc/fstab entries should look something like (note noatime option): 

/dev/sda3               /vz                     ext4    defaults,noatime 1 2
/dev/sdb2               /vz1                    ext4    defaults,noatime 1 2

make sure swap is configured in fstab

Use real devices instead of LABEL


  • Install Virtuozzo 4.7

Have a license on hand - request a day ahead of the install. Go to: http://www.parallels.com/download/pvc47/ 

yum install compat-libstdc++-33
yum install wget
yum install ftp
yum install pciutils
yum install ed bc 


cd /vz
wget http://download.parallels.com/pvc/47/lin/vzinstall-linux-x86_64.bin
chmod 0700 vzinstall-linux-x86_64.bin
./vzinstall-linux-x86_64.bin

download and install

change wd's to /vz

Ignore OS version error

Custom install (don’t install templates)

Install PVA Agent Install PVA Management Node 

reboot

Get on KVM to watch PVA install (won't happen on console) 

vzup2date

(run again. till no more updates offered) 

reboot

<DEPRECATED> Run vzup2date (make sure to choose enterprise kernel) and/or install latest kernel:

NOTE: skip over the manual kernel load below if you can get the 16G recognized before running vzup2date- then it will dl the ent kernel

NOTE: go to /vz/vzup2date/virtuozzo/linux/i386/4.0.0/CU-2.6.18-028stab068.3 and install kern there (downloaded by vzup2date) 

vi /boot/grub/menu.lst

(set default kernel to new kernel) </DEPRECATED>


  • Install ntp
yum install ntp

(if not already installed) 

rm -f /etc/ntp.conf

Create a new file with one line: 

echo "server 10.1.4.5" > /etc/ntp.conf

Then set ntpd so it will start at boot: 

chkconfig --level 3 ntpd on
service ntpd start

Make sure it’s talking to the ntp server: 

ntpq -p

Correct output should have something unfer the refid col and a '+' next to the remote server: 

remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 10.1.4.105+     clock1.redhat.c  2 u    -   64    1    0.370  -706677   0.008


  • copy over binaries/scripts from backup2


scp backup2:/mnt/data4/bin/linux/* /usr/local/sbin/
scp 10.1.4.62:"/vzconf/jctmpl*" /etc/sysconfig/vz-scripts/
scp 10.1.4.62:/vzconf/ve-slm* /etc/sysconfig/vz-scripts/


  • setup root crontab (alter backup time):
crontab -e
1 1 * * * /usr/local/sbin/backup
*/15 * * * * /usr/local/sbin/vim.watchdog.30 > /dev/null
#56 4,16 * * * root /usr/sbin/vzkeyupdate -c
*/5 * * * * /usr/local/sbin/perc5iraidchk64


  • setup backups
echo '#!/bin/sh
<tt>backupdir=/data/<span style="color:red">virt13</span> </tt>

server=backup1

## ENTRY /etc
## ENTRY /usr/local/sbin
## ENTRY /root/logs
' > /usr/local/sbin/backup.config


  • setup bigbrother

edit passwd and add the bb user: 

echo "bb:x:1984:1984:Big Brother:/home/bb:/bin/bash" >> /etc/passwd

edit group and add the bb group: 

echo "bb:x:1984:" >> /etc/group

rebuild the password database: 

pwconv

Create bb’s home dir and set ownership: 

mkdir /home/bb
chown bb.bb /home/bb

Go to bb’s home and fetch the client tarball: 

cd ~bb
scp backup2:/mnt/data4/build/bb/GOOD-bb-linux-bbc1.9i-btf.tgz .

tar xvf GOOD-bb-linux-bbc1.9i-btf.tgz

yum install glibc-2.12-1.47.el6_2.5.i686

vi etc/bbsys.sh
SORT="/bin/sort"

vi bin/bb-disk.sh
eval "$DFCMD | $EGREP \"$DFUSE\" | $EGREP -v \"$DFEXCLUDE\" | $SORT \"-k${DFSORT}\" > $BBTMP/DFOUT.$$"
wclen1=`eval "$DFCMD | $EGREP \"$DFUSE\" | $EGREP -v \"$DFEXCLUDE\" | $SORT \"-k${DFSORT}\" | $WC"`

cd bbc1.9i-btf/etc/

echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh" > /home/bb/bbc1.9i-btf/etc/bb-hosts
echo "10.1.4.63 virt13.johncompanies.com # ssh" >> /home/bb/bbc1.9i-btf/etc/bb-hosts
(^^^change to correct host)

chmod +r /var/log/messages

cd ..
su bb
./runbb.sh start
exit

Add this line to /etc/rc.d/rc.local so bb will start at boot: 

echo "su - bb -c \"cd /home/bb/bbc1.9i-btf; ./runbb.sh start\"" >> /etc/rc.d/rc.local


  • Uncomment/add lines with the appropriate IP addresses in /etc/ssh/sshd_config:
vi /etc/ssh/sshd_config

ListenAddress 69.55.226.2 ListenAddress 10.1.4.63

<DEPRECATED> And make sure to uncomment 

Protocol 2,1

</DEPRECATED>


  • setup backup1/backup2 nfs mounts

<DEPRECATED> setup portmap to start with server, and start it: 

chkconfig --level 3 portmap on
service portmap start

</DEPRECATED> 

mb mount

run df to confirm that nfs mounts are mounted


  • make alias for /vzconf
ln -s /etc/sysconfig/vz-scripts /vzconf


  • edit /etc/sysconfig/vz

vi /etc/sysconfig/vz

calc cpu to reserver for HN. take 5% of power: 

vzcpucheck
Current CPU utilization: 27000
Power of the node: 1018320

so in this case: 

VE0CPUUNITS=50916

enable fastboot: 

VZFASTBOOT=yes

Disable auto mount 

#VZMOUNTS="/vz"

Enable slm: 

SLM=yes

<DEPRECATED> Enable IPV6 IPV6=yes </DEPRECATED>

add/edit to traffic parameters section: 

TRAFFIC_SHAPING=yes
BANDWIDTH="eth0:102400"
TOTALRATE="eth0:1:102400"
RATE="eth0:1:50000"

for 2 nics, uncomment this line: 

VE_ROUTE_SRC_DEV="eth0"

Make sure the IPTABLES section looks like: 

IPTABLES="ip_tables iptable_filter iptable_mangle ipt_limit ipt_multiport ipt_tos ipt_TOS ipt_REJECT ipt_tcpmss ipt_ttl ipt_LOG ipt_length ip_conntrack ip_conntrack_ftp  ip_conntrack_irc ipt_conntrack ipt_state ipt_helper iptable_nat ip_nat_ftp ip_nat_irc ipt_REDIRECT ipt_multiport ipt_TCPMSS"


  • disable periodic fsck's, lower the root padding on the /vz and /vz1 filesystems to 100MB
tune2fs -i 0 /dev/sda3
tune2fs -c -1 /dev/sda3
tune2fs -r 100000 /dev/sda3

tune2fs -i 0 /dev/sdb2
tune2fs -c -1 /dev/sdb2
tune2fs -r 100000 /dev/sdb2


  • setup vzagent with pub IP (to allow migrations from 3.x):
vzctl set 50 --ipdel all --ipadd 69.55.226.3 --userpasswd vzagent0:k0zub1k --save
vzagent_ctl restart


  • edit ve config files:

From split of 80 take CPUUNITS

And put into files in /vzconf/base_ct_conf along with: 

VERSION="2"

ONBOOT="yes"

KMEMSIZE="2147483647:2147483647"
LOCKEDPAGES="2147483647:2147483647"
PRIVVMPAGES="2147483647:2147483647"
SHMPAGES="2147483647:2147483647"
NUMPROC="32567:32567"
VMGUARPAGES="2147483647:2147483647"
OOMGUARPAGES="2147483647:2147483647"
NUMTCPSOCK="2147483647:2147483647"
NUMFLOCK="2147483647:2147483647"
NUMPTY="255:255"
NUMSIGINFO="1024:1024"
TCPSNDBUF="2147483647:2147483647"
TCPRCVBUF="2147483647:2147483647"
OTHERSOCKBUF="2147483647:2147483647"
DGRAMRCVBUF="2147483647:2147483647"
NUMOTHERSOCK="2147483647:2147483647"
DCACHESIZE="2147483647:2147483647"
NUMFILE="2147483647:2147483647"
NUMIPTENT="2147483647:2147483647"

DISKSPACE=""
DISKINODES=""
QUOTATIME="600"
IOPRIO="4"
CPULIMIT="75"
OFFLINE_MANAGEMENT="yes"
ARCH="x86"
PLATFORM="linux"

IP_ADDRESS="0.0.0.0"
NAMESERVER="69.55.225.225 69.55.230.3"
CPUUNITS="12313"
IOLIMIT="10485760"
IOPSLIMIT="100"


  • customize the control panel:
rm -fr /vz/backups

vi /var/opt/pva/pp/xsl/lib/vt/backups-forms.xsl

<!--No backups created yet. You can create a maximum of %max backup(s). -->
                                        <span style="font-size:13px; font-weight:bold">NOTE: You cannot create or view backups via this control panel- IT WILL NOT WORK. Please contact linux@johncompanies.com to setup or inquire about backups for your server.</span>

/var/opt/pva/pp/xsl/lib/vt/backups-list.xsl
                <!--
                                <ui:button tip="backup_ve" sandclock="false" icon="backup_new"
                                        submit-href="{$baseurl}/vt/backups/new?cmd=backup&env-set=current"
                                        disabled="{boolean(not($new-allowed))}" require-items="false"
                                        new-obj-title="{sw:text('Backup')}"
                                >
                                        <sw:text>New Backup</sw:text>
                                </ui:button>
-->
cd  /var/opt/pva/pp/static/vz/skins
for f in `ls`; do scp backup2:/d4/build/vzcp/var/vzcp/static/vz/skins/winxp.new/images/logo_main_4.7.gif $f/images/logo_main.gif; done


  • template install

To install new templates: 

vzup2date -z

Copy over templates from old server. Ex: ubu 8.04 from virt19: 

rsync -a -e ssh 10.1.4.69:/vz/template/ubuntu/8.04 /vz/template/ubuntu/8.04

install ubuntu 8.04 templates (get latest from vz) 

cd /backup4/vzrpms/distro/ubuntu/8.04
rpm -ivh *
vzpkg update cache ubuntu-8.04-x86

If you install a tempalte from vzup2date it will download an old cache as well. Better to download the OS rpm from VZ site then create cache then use vzup2date to install the application templates.


  • add traffic shaping support

Re-define class 0: 

vi /etc/sysconfig/vz-scripts/networks_classes
0 69.55.224.0/20
0 10.1.4.0/24

make sure 1 is defined (0.0.0.0) and no other classes are needed

Then start it 

/etc/init.d/vz shaperrestart


  • udp dos fix
echo "/usr/local/sbin/iptables.sh" >> /etc/rc.d/rc.local


  • make rc.local start before vz
mv /etc/rc.d/rc3.d/S99local /etc/rc.d/rc3.d/S95local


  • disable unwanted services/daemons

Before: 

abrt-ccpp       0:off   1:off   2:off   3:on    4:off   5:on    6:off
abrt-oops       0:off   1:off   2:off   3:on    4:off   5:on    6:off
abrtd           0:off   1:off   2:off   3:on    4:off   5:on    6:off
acpid           0:off   1:off   2:on    3:on    4:on    5:on    6:off
atd             0:off   1:off   2:off   3:on    4:on    5:on    6:off
auditd          0:off   1:off   2:on    3:on    4:on    5:on    6:off
autofs          0:off   1:off   2:off   3:on    4:on    5:on    6:off
cpuspeed        0:off   1:on    2:on    3:on    4:on    5:on    6:off
crond           0:off   1:off   2:on    3:on    4:on    5:on    6:off
cups            0:off   1:off   2:on    3:on    4:on    5:on    6:off
haldaemon       0:off   1:off   2:off   3:on    4:on    5:on    6:off
ip6tables       0:off   1:off   2:on    3:on    4:on    5:on    6:off
iptables        0:off   1:off   2:on    3:on    4:on    5:on    6:off
irqbalance      0:off   1:off   2:off   3:on    4:on    5:on    6:off
kdump           0:off   1:off   2:off   3:on    4:on    5:on    6:off
lvm2-monitor    0:off   1:on    2:on    3:on    4:on    5:on    6:off
mcelogd         0:off   1:off   2:off   3:on    4:off   5:on    6:off
mdmonitor       0:off   1:off   2:on    3:on    4:on    5:on    6:off
messagebus      0:off   1:off   2:on    3:on    4:on    5:on    6:off
netfs           0:off   1:off   2:off   3:on    4:on    5:on    6:off
network         0:off   1:off   2:on    3:on    4:on    5:on    6:off
nfslock         0:off   1:off   2:off   3:on    4:on    5:on    6:off
ntpd            0:off   1:off   2:off   3:on    4:off   5:off   6:off
portreserve     0:off   1:off   2:on    3:on    4:on    5:on    6:off
postfix         0:off   1:off   2:on    3:on    4:on    5:on    6:off
pvaagentd       0:off   1:off   2:on    3:on    4:on    5:on    6:off
pvapp           0:off   1:off   2:on    3:on    4:on    5:on    6:off
qpidd           0:off   1:off   2:on    3:on    4:on    5:on    6:off
rpcbind         0:off   1:off   2:on    3:on    4:on    5:on    6:off
rpcgssd         0:off   1:off   2:off   3:on    4:on    5:on    6:off
rpcidmapd       0:off   1:off   2:off   3:on    4:on    5:on    6:off
rsyslog         0:off   1:off   2:on    3:on    4:on    5:on    6:off
sshd            0:off   1:off   2:on    3:on    4:on    5:on    6:off
sysstat         0:off   1:on    2:on    3:on    4:on    5:on    6:off
udev-post       0:off   1:on    2:on    3:on    4:on    5:on    6:off
vz              0:off   1:off   2:on    3:on    4:on    5:on    6:off
vzlmond         0:off   1:off   2:on    3:on    4:on    5:on    6:off
vznetcfg        0:off   1:off   2:on    3:on    4:on    5:on    6:off
xinetd          0:off   1:off   2:off   3:on    4:on    5:on    6:off
chkconfig --level 3 cpuspeed off
chkconfig --level 3 cups off
chkconfig --level 3 haldaemon off
chkconfig --level 3 ip6tables off
chkconfig --level 3 iptables off
chkconfig --level 3 irqbalance off
chkconfig --level 3 lvm2-monitor off
chkconfig --level 3 mdmonitor off
chkconfig --level 3 messagebus off
chkconfig --level 3 netfs off
chkconfig --level 3 nfslock off
chkconfig --level 3 qpidd off
chkconfig --level 3 rpcgssd off
chkconfig --level 3 rpcidmapd off


  • load mrtg graphs
yum install xinetd

create /etc/xinetd.d/load with the following contents: 

echo "service load
{
        disable = no
        flags           = REUSE
        socket_type     = stream
        wait            = no
        user            = root
        server          = /usr/local/sbin/load.pl
}" > /etc/xinetd.d/load

Add this line to /etc/services: 

echo "load            12384/tcp" >> /etc/services


  • Tasks to do on backup1:

setup backup dirs: 

mkdir -p /data/virt13/0

for f in 0 1 2 3 4 5 6; do mkdir -p /data/virt13/$f/vz/private; mkdir -p /data/virt13/$f/vz1/private; mkdir -p /data/virt13/$f/vz2/private; done

add the system to 

vi /usr/local/sbin/snapshot_rotate


  • Tasks to do on mail:

Update backup graph: 

vi /usr/local/www/mgmt/cgi/backupgraph.pl

(add hostname)

add to bbhosts: 

vi /usr/home/bb/bbsrc/bb1.9i-btf/etc/bb-hosts
su bb
cd
bbsrc/bb/runbb.sh restart ; exit

add to load graph config: 

vi /usr/local/www/mgmt/mrtg/mrtg1.cfg

(add new section for new machine)


  • add to management db (on mail and devweb) jc.ref_machines and jc.ref_templates

insert new server: 

insert into ref_machines values (null,'virt19','virt19.johncompanies.com',0,'l');

get new machine_id back out: select * from ref_machines;

insert as a system to jc customer col00001: 

insert into systems values (, 'col00001',30,null,null,100,'virt11.johncompanies.com',null,'2010-03-11',null,null,1,0,null,null,null);

get sysid: select * from systems where hostname='virt19.johncompanies.com';

update ipmap (main IP and IP used for VZCP): 

update ipmap set cid='col00001', sysid=3772, assign_date='2008-01-07', status=1 where ip='69.55.236.2';
update ipmap set cid='col00001', sysid=3772, assign_date='2008-01-07', status=1 where ip='69.55.230.109';

add templates (or do via mgmt): 

insert into ref_templates values ('','fedora-core-1/20050725',33,'Fedora Core 1',1);
insert into ref_templates values ('','fedora-core-2/20050725',33,'Fedora Core 2',1);
insert into ref_templates values ('','debian-3.1/20050927',33,'Debian 3.1',1);


  • Make sure that ns1c has proper forward and reverse entries for the new system (both private and public addresses), so that new system emailing will work.
vr johncompanies.com
vi internal.johncompanies.com
rndc reload johncompanies.com IN private

(edit the PTR too)


  • add an outside blocking rule to the firewall, so this machine can only be reached from inside the firewall.

Follow example already in firewall virt19 is: 

ipfw add 00143 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 62.213.72.101 or 64.131.90.27 or 82.200.65.190 or 91.204.24.0/22 or 194.44.160.178 or 195.214.232.10 or 195.214.233.10 or 208.251.212.1 } to 69.55.226.2
ipfw add 00143 deny ip from any to 69.55.226.2

(virt12 would be 00142)



  • make /vz1/private
mkdir /vz1/private ; chmod 0755 /vz1/private
mkdir /vz1/root ; chmod 0755 /vz1/root


  • put config files in a place they’ll get backed up
mkdir /vzconf/confs
crontab -e
0 0 * * * rm -f /vzconf/confs/*.conf; cp  /vzconf/*.conf /vzconf/confs


  • fix pass/vzpp and localse issue with ubu 8.10
vi /vz/template/ubuntu/8.10/x86/config/os/default/post-install

sed '/^password.*pam_unix.so.*sha512/ s~sha512~md5~' -i etc/pam.d/common-password

localedef -i en_US -c -f UTF-8 en_US.UTF-8


  • add tun device support
echo "modprobe tun" >> /etc/rc.local


  • reinstall VZA/CT #1/CT #50

After the install, the PP was broken. So needed to reinstall everything. Did the pva_setup --uninstall and removed CT 50 and then setup the pva per http://kb.parallels.com/113071 then setup the CT 50 per the other article- centos6 had issues - and then authenticated with virt13 to get that into the management list.

http://kb.parallels.com/113072

http://kb.parallels.com/113071


  • config logrotate (otherwise it creates weekly empty logs)
vi /etc/logrotate.conf

(add to end) 

/var/log/messages {
        rotate 12
        monthly
        size 10M
        missingok
        compress
        delaycompress
        notifempty
        sharedscripts
        postrotate
                echo `date` > /var/log/messages
        endscript
}


  • install megacli64 (can we just move the bin and not install the lib?)
scp backup2: /d4/build/3ware/MegaCli-8.00.40-1.i386.rpm /root/MegaCli-8.00.40-1.i386.rpm
scp backup2: /d4/build/3ware/Lib_Utils-1.00-08.noarch.rpm /root/ Lib_Utils-1.00-08.noarch.rpm

rpm -ivh Lib_Utils-1.00-08.noarch.rpm 
rpm -ivh MegaCli-8.00.40-1.i386.rpm

cp opt/MegaRAID/MegaCli/MegaCli64 /usr/local/sbin/megacli64

Older versions[edit]

4.0[edit]

Last updated 2010-03-11 

1. enable post info to console

2950:
Serial comm.: on with cons. Redir via com1
Failsafe: 115200
Redir after boot: enabled

Server
Custom- desel all but base
/root/install.log


2. Install OS

CentOS5.x (get latest from http://mirrors.kernel.org/centos/5.4/isos/i386/)


Set partitions:
Disk 1:
/ 3072M
6144M swap
/vz (the rest)
Disk 2: 
6144M Swap

Use grub, install to MBR
No firewall
Disable SElinux
Install only base


3. create ssh key, add swsoft key to authorized_keys

(enter to everything)

cd /root/
ssh-keygen -t dsa

echo "ssh-dss AAAAB3NzaC1kc3MAAACBAIB7XHmpcsa1D/vsKS+mzEvBVo7udew/aRahbjnjywL4YXT7ruWzpGo7R3UA8GKIgVHoQGUbuXrJz5ICtBEXOlbgXVEz7Bu5nXOG2C9i/8pmFtUw6GwACII8uZE7t/Zq5CRj+5zN1Rv7n/4X4RAjzk5BHXoZ62a32FTbS4tPvyPrAAAAFQDiUi7YEiffNzGvB544lQpRdn29TwAAAIBkUchLTBE1+sX11iC80yhNh9EGeCMLdkT8UqHsTSaD0myQPxGALuNY4rO31q+25uKqSRDTXd6ZWXp4bplA6Ls6jl7r7XVR9loJ7WvRMqc4Dv68TZ7h2UBqAhu4det2UF4LHim3DNf1gKn9DzWNkmI/N2FvVVAL4aM3sPunVIvUYgAAAIAJX38INABVgnCEBKqpv5FMLm3k9MgMn54oQFmylBbgM0w4SUeuDj3jLrkJf8dT9w8ByEDb8xviexpWrPsbz2DeThle4xgzcyKrUkt2Wugtmm4DwS0S/68BAA9wXoWD/vZ67GBoCKH7uq8VAjUVcCbJ/7IydljnFAZoJpktdwN7iQ== russup@support.sw-soft.com" >> /root/.ssh/authorized_keys

4. setup access to backup2:

echo "10.1.4.3        backup2" >> /etc/hosts
echo "10.1.4.8         backup1" >> /etc/hosts

# Copy id_dsa.pub to backup2 and add it to the end of ~root/.ssh/authorized_keys:

cat .ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys' 
cat .ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys' 

ssh backup2 hostname
ssh backup1 hostname

(confirm you can connect w/o entering a password)

5. add bash backward history search 
echo "\"\M-z\": history-search-backward
\"\M-x\": history-search-backward " > /root/.inputrc
echo "export GREP_OPTIONS='--color=auto'" >> ~/.bashrc
echo "\"\e[5~\": history-search-backward" >> ~/.inputrc
echo "\"\e[6~\": history-search-forward" >> ~/.inputrc

6. add to /root/.bashrc, create log dir

vi /root/.bashrc
alias v='vzctl'
alias mbm='mb mount'
alias mbu='mb umount'
export PS1="[\u@\h \w]# "
alias cv='cd /vz/private'
alias cv1='cd /vz1/private'
alias cv2='cd /vz2/private'

source /root/.bashrc

mkdir /root/logs

7. make sure serial console and kernel messages are working

add these 2 lines to the top:

vi /boot/grub/menu.lst
serial --unit=0 --speed= 115200
terminal --timeout=10 serial console

and the kernel line should look like:

kernel /boot/vmlinuz-2.4.20-020stab009.24.777-enterprise ro root=/dev/sda1 console=tty0 console=ttyS0,115200

Remove or comment out these lines:
hiddensplashmenu 
splashimage=(hd0,0)/boot/grub/splash.xpm.gz

add to /etc/inittab and /etc/securetty

echo "s0:2345:respawn:/sbin/agetty -L ttyS0 115200 linux" >> /etc/inittab
echo "ttyS0" >> /etc/securetty

kill -1 1

DEPRECATED 8. configuring eth1 for management net:

echo "10.1.5/24 via 10.1.4.2
10.1.6/24 via 10.1.4.2" >> /etc/sysconfig/network-scripts/route-eth1

reboot the machine to make sure that the interface is configured properly.

also verify that the routes are in place like so:

netstat -rn | grep 10.1.4.2
10.1.5.0        10.1.4.2        255.255.255.0   UG       40 0          0 eth1
10.1.6.0        10.1.4.2        255.255.255.0   UG       40 0          0 eth1

9. make sure the hostname on the system is properly set - in the form virt4.johncompanies.com - and make sure it is set properly: 
cat /etc/sysconfig/network

10. Check/update fstab

vi /etc/fstab

/etc/fstab entries should look something like (note noatime option):

/dev/sda3               /vz                     ext3    defaults,noatime 1 2
/dev/sdb2               /vz1                    ext3    defaults,noatime 1 2

make sure swap is configured in fstab

Use real devices instead of LABEL

11. Install Virtuozzo 4.0

umount /vz
umount /vz1

mke2fs -i1024 -b1024 -j /dev/sda3 &
mke2fs -i1024 -b1024 -j /dev/sdb2 &

Experimentally ran this for sdc on virt12:
mke2fs -i2048 -b2048 -j /dev/sdb1

mount -a

scp backup2:/d4/vzrpms/iso/vzinstall-linux.bin /vz 
cd /vz
chmod 0700 vzinstall-linux.bin
./vzinstall-linux.bin

change wd to /vz
Ignore OS version error
Custom install (don’t install templates)

reboot

vzup2date

 (run again. till no more updates offered)

reboot

Run vzup2date (make sure to choose enterprise kernel) and/or install latest kernel:

NOTE: skip over the manual kernel load below if you can get the 16G recognized before running vzup2date- then it will dl the ent kernel

NOTE: go to /vz/vzup2date/virtuozzo/linux/i386/4.0.0/CU-2.6.18-028stab068.3 and install kern there (downloaded by vzup2date)

<DEPREACATED>
ftp downloads.swsoft.com
User: 
gdawson
Pass: 
qjdK1R

cd /updates/Rel400/20080307
bin
get vzkernel-ent-2.6.18-028stab053.10.i686.rpm
get vzmodules-ent-2.6.18-028stab053.10.i686.rpm
bye

rpm -ivh vzkernel-ent-2.6.18-028stab053.10.i686.rpm \
vzmodules-ent-2.6.18-028stab053.10.i686.rpm
</DEPREACATED>

vi /boot/grub/menu.lst
(set default kernel to new kernel)

12. Install ntp

yum install ntp

rm -f /etc/ntp.conf

# Create a new file with one line:

echo "server 10.1.4.5" > /etc/ntp.conf 

# Then set ntpd so it will start at boot:

chkconfig --level 3 ntpd on
service ntpd start 

# Make sure it’s talking to the ntp server:

ntpq -p

Correct output should have * or + to the left of the IP:

     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 10.1.4.105      clock1.redhat.c  2 u    -   64    1    0.370  -706677   0.008


13. copy over binaries/scripts from backup2

scp backup2:/mnt/data4/bin/linux/* /usr/local/sbin/
scp 10.1.4.62:/vzconf/jctmpl* /etc/sysconfig/vz-scripts/
scp 10.1.4.62:/vzconf/ve-slm* /etc/sysconfig/vz-scripts/

14. setup root crontab (alter backup time): 
crontab -e
1 1 * * * /usr/local/sbin/backup
*/15 * * * * /usr/local/sbin/vim.watchdog.30 > /dev/null
#56 4,16 * * * root /usr/sbin/vzkeyupdate -c
*/5 * * * * /usr/local/sbin/perc5iraidchk

15. setup backups 

echo '#!/bin/sh
backupdir=/data/virt11
server=backup1

## ENTRY /etc
## ENTRY /usr/local/sbin
## ENTRY /root/logs
' > /usr/local/sbin/backup.config

16. setup bb 

edit passwd and add the bb user:
echo "bb:x:1984:1984:Big Brother:/home/bb:/bin/bash" >> /etc/passwd

# edit group and add the bb group:
echo "bb:x:1984:" >> /etc/group

# rebuild the password database:

pwconv

# Create bb’s home dir and set ownership:

mkdir /home/bb
chown bb.bb /home/bb

# Go to bb’s home and fetch the client tarball:

cd ~bb
scp backup2:/mnt/data4/build/bb/bb-linux.tar .

tar xf bb-linux.tar

cd bbc1.9e-btf/etc/

echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
echo "10.1.4.61 virt11.johncompanies.com # ssh" >> /home/bb/bbc1.9e-btf/etc/bb-hosts

chmod +r /var/log/messages

cd ..
su bb
./runbb.sh start
exit

# Add this line to /etc/rc.d/rc.local so bb will start at boot:

echo "su - bb -c \"cd /home/bb/bbc1.9e-btf; ./runbb.sh start\"" >> /etc/rc.d/rc.local 

17. Uncomment/add lines with the appropriate IP addresses in /etc/ssh/sshd_config:

vi /etc/ssh/sshd_config 
ListenAddress 69.55.238.3
ListenAddress 10.1.4.61

And make sure to uncomment
Protocol 2,1

18. setup backup2 nfs mounts

setup portmap to start with server, and start it: 

chkconfig --level 3 portmap on
service portmap start

mb mount

run df to confirm that nfs mounts are mounted

19. make alias for /vzconf
ln -s /etc/sysconfig/vz-scripts /vzconf

DEPRECATED 20. copy over the /etc/sysconfig/vz-scripts/4unlimited template file from the last virt server and put it in /etc/sysconfig/vz-scripts

scp 10.1.4.68:/etc/sysconfig/vz-scripts/4unlimited /etc/sysconfig/vz-scripts/

NOTE: if this system is more powerful (RAM, CPU) then the previous, you will want to create a new one using vzsplit:

cd /etc/sysconfig/vz-scripts/
vzsplit -f 4unlimited

21. edit /etc/sysconfig/vz 

vi /etc/sysconfig/vz 

enable fastboot:
VZFASTBOOT=yes

Disable auto mount
#VZMOUNTS="/vz"

Enable IPV6
IPV6=yes

add/edit to traffic parameters section:
TRAFFIC_SHAPING=yes
BANDWIDTH="eth0:102400"
TOTALRATE="eth0:1:102400"
RATE="eth0:1:50000"

for 2 nics, uncomment this line:
VE_ROUTE_SRC_DEV="eth0"

Make sure the IPTABLES section looks like:
IPTABLES="ip_tables iptable_filter iptable_mangle ipt_limit ipt_multiport ipt_tos ipt_TOS ipt_REJECT ipt_tcpmss ipt_ttl ipt_LOG ipt_length ip_conntrack ip_conntrack_ftp  ip_conntrack_irc ipt_conntrack ipt_state ipt_helper iptable_nat ip_nat_ftp ip_nat_irc ipt_REDIRECT ipt_multiport ipt_TCPMSS"

22. disable periodic fsck's, lower the root padding on the /vz and /vz1 filesystems to 100MB

tune2fs -i 0 /dev/sda3
tune2fs -i 0 /dev/sdb2
tune2fs -c -1 /dev/sda3
tune2fs -c -1 /dev/sdb2
tune2fs -r 100000 /dev/sda3
tune2fs -r 100000 /dev/sdb2

23. setup vzagent with pub IP (to allow migrations from 3.x):


vzctl set 1 --ipdel all --ipadd 69.55.230.57 --userpasswd vzagent0:k0zub1k --save
vzagent_ctl restart


24. edit ve config files:

From split of 80 take
CPUUNITS

And put into files in /usr/local/sbin/ve-slm.1024MB.conf  et. al.


25. customize the control panel:

vzmlocal 1:1:/vz/private/1:/vz/root/1
cd  /vz/private/1/fs/root/var/vzcp/static/vz/skins
for f in `ls`; do scp backup2:/d4/build/vzcp/var/vzcp/static/vz/skins/winxp.new/images/logo_main.gif $f/images/logo_pim.gif; done

1) comment 'Backup' tab in /etc/vzcp/pp/tabs.xml:

  <tab-bar id="maintance" obj-type="env.core.vt">
<!--    <tab href="$baseurl/vz/ves/backups/index-vzpp" title="Backups" id="maintance.backups"
tab="backups" tip="tab_env_maintenance" cred="vzcp.env.backups.list"
security-obj-type="ve.vz.vt"/> -->

2) change 'href' parameter if 'Maintentance' item in /etc/vzcp/pp/menu.xml:

    <!-- <item id="management.maintenance"
href="$baseurl/vz/ves/backups/index-vzpp?branch=env-home,vzpp" icon="backup"
tip="maintenance_screen"> -->
    <item id="management.maintenance" href="/vz/cp/vzdir/infrman/envs/repair?branch=env-home,vzpp" icon="backup" tip="maintenance_screen">


3) /etc/vzcp/pp/dashboard.xml
<!--    <item id="management.maintenanance" href="$baseurl/vz/ves/backups/index-vzpp" title="Maintenance" icon="env_backup" tip="maintenance_screen" cred="vzcp.env.backups.list" security-obj-type="ve.vz.vt">-->
    <item id="management.maintenanance" href="$baseurl/vzdir/infrman/envs/repair" title="Maintenance" icon="env_backup" tip="maintenance_screen" cred="vzcp.env.backups.list" security-obj-type="ve.vz.vt">


URL for 'href' parameter can be obtained from tabs.xml.

vi /var/vzcp/xsl/screens/vzdir/monitman/traffic.xsl
                                        <sw:text>Traffic for selected period and class (NOTE: this information is not certified by JohnCompanies and is not used to calculate actual usage or charges)</sw:text>

<DEPREACATED>
/vz/private/1/root/var/vzcp/static/vz/skins/
/vz/private/1/root/etc/vzcp/pp/menu.xml
/vz/private/1/root/etc/vzcp/pp/dashboard.xml
pp/tabs.xml
/vz/private/1/root/etc/vzcp/vzcpcon.conf

eg (2.6.1 or 3.0):
rsync -va -e ssh backup2:/d4/build/vzcp/var/vzcp/static/vz/skins/  /vz/private/1/root/var/vzcp/static/vz/skins/
rsync -va -e ssh backup2:/d4/build/vzcp/etc/vzcp/pp/menu.xml  /vz/private/1/root/etc/vzcp/pp/menu.xml
rsync -va -e ssh backup2:/d4/build/vzcp/etc/vzcp/pp/dashboard.xml /vz/private/1/root/etc/vzcp/pp/dashboard.xml
rsync -va -e ssh backup2:/d4/build/vzcp/etc/vzcp/vzcpcon.conf  /vz/private/1/root/etc/vzcp/vzcpcon.conf
</DEPREACATED>

24. template install

To install new templates:

vzup2date –z

Copy over ubu 8.04 from virt19

rsync -a -e ssh 10.1.4.69:/vz/template/ubuntu/8.04 /vz/template/ubuntu/8.04 

install ubuntu 8.04 templates (get latest from vz)

cd /backup4/vzrpms/distro/ubuntu/8.04
rpm -ivh *
vzpkg update cache ubuntu-8.04-x86


26. add traffic shaping support

Re-define class 0:

vi /etc/sysconfig/vz-scripts/networks_classes
0 69.55.224.0/20
0 10.1.4.0/24

make sure 1 is defined (0.0.0.0) and no other classes are needed

Then start it

/etc/init.d/vz shaperrestart

27. udp dos fix

echo "/usr/local/sbin/iptables.sh" >> /etc/rc.d/rc.local 

28. make rc.local start before vz

mv /etc/rc.d/rc3.d/S99local /etc/rc.d/rc3.d/S95local

29. disable unwanted services/daemons

chkconfig --level 3 avahi-daemon off
chkconfig --level 3 bluetooth off
chkconfig --level 3 cpuspeed off
chkconfig --level 3 cups off
chkconfig --level 3 firstboot off
chkconfig --level 3 gpm off
chkconfig --level 3 haldaemon off
chkconfig --level 3 hidd off
chkconfig --level 3 ip6tables off
chkconfig --level 3 iptables off
chkconfig --level 3 irqbalance off
chkconfig --level 3 isdn off
chkconfig --level 3 kudzu off
chkconfig --level 3 lvm2-monitor off
chkconfig --level 3 mcstrans off
chkconfig --level 3 mdmonitor off
chkconfig --level 3 messagebus off
chkconfig --level 3 netfs off
chkconfig --level 3 nfslock off
chkconfig --level 3 pcscd off
chkconfig --level 3 rawdevices off
chkconfig --level 3 readahead_early off
chkconfig --level 3 restorecond off
chkconfig --level 3 rpcgssd off
chkconfig --level 3 rpcidmapd off
chkconfig --level 3 smartd off
chkconfig --level 3 yum-updatesd off

30. load mrtg graphs


yum install xinetd

create /etc/xinetd.d/load with the following contents:

echo "service load
{
        disable = no
        flags           = REUSE
        socket_type     = stream
        wait            = no
        user            = root
        server          = /usr/local/sbin/load.pl
}" > /etc/xinetd.d/load

# Add this line to /etc/services:
echo "load            12384/tcp" >> /etc/services


31. Tasks to do on backup1:

setup backup dirs:
mkdir -p /data/virt11/0

add the system to 
vi /usr/local/sbin/snapshot_rotate

32. Tasks to do on mail:

Update backup graph:
vi /usr/local/www/mgmt/cgi/backupgraph.pl
(add hostname)

add to bbhosts:
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
su bb
cd
bbsrc/bb/runbb.sh restart ; exit

add to load graph config:
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg
(add new section for new machine)

33. add to management db (on mail and devweb) jc.ref_machines and jc.ref_templates

insert into ref_machines values (null,'virt19','virt19.johncompanies.com',0,'l');
select * from ref_machines;
insert into systems values ('', 'col00001',30,null,null,100,'virt11.johncompanies.com',null,'2010-03-11',null,null,1,0,null,null,null);
select * from systems where hostname='virt19.johncompanies.com';
update ipmap set cid='col00001', sysid=3772, assign_date='2008-01-07', status=1 where ip='69.55.236.2';
update ipmap set cid='col00001', sysid=3772, assign_date='2008-01-07', status=1 where ip='69.55.230.109';
insert into ref_templates values ('','fedora-core-1/20050725',33,'Fedora Core 1',1);
insert into ref_templates values ('','fedora-core-2/20050725',33,'Fedora Core 2',1);
insert into ref_templates values ('','debian-3.1/20050927',33,'Debian 3.1',1);

34. Make sure that ns1c has proper forward and reverse entries for the new system (both private and public addresses), so that new system emailing will work.
vr johncompanies.com
vi internal.johncompanies.com
rndc reload johncompanies.com IN private
(edit the PTR too)


35. add an outside blocking rule to the firewall, so this machine can only be reached from inside the firewall. 

Follow example already in firewall virt19 is:

ipfw add 00149 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 62.213.72.101 or 64.131.90.27 or 82.200.65.190 or 91.204.24.0/22 or 194.44.160.178 or 195.214.232.10 or 195.214.233.10 or 208.251.212.1 } to 69.55.236.2
ipfw add 00149 deny ip from any to 69.55.236.2

virt12 would be 00142

36. add to server/cabinet map

37. make /vz1/private
mkdir /vz1/private ; chmod 0755 /vz1/private
mkdir /vz1/root ; chmod 0755 /vz1/root

38. put config files in a place they’ll get backed up
mkdir /vzconf/confs
crontab -e
0 0 * * * rm -f /vzconf/confs/*.conf; cp  /vzconf/*.conf /vzconf/confs

39. fix pass/vzpp and localse issue with ubu 8.10

vi /vz/template/ubuntu/8.10/x86/config/os/default/post-install

sed '/^password.*pam_unix.so.*sha512/ s~sha512~md5~' -i etc/pam.d/common-password

localedef -i en_US -c -f UTF-8 en_US.UTF-8

40. add tun device support

echo "modprobe tun" >> /etc/rc.local


3.x[edit]

Last updated 2007-04-5 

1. enable post info to console
Supermicro:
Console redirection:
Com port addr: on-board COM A
Baud: 38400
Console type: vt100
Flow control: none
Console connection: direct
Continue cr after post: off

2450:
Make sure running bios A09
Console Redirection: VT100/VT220

2950:
Serial comm.: on with cons. Redir via com1
Failsafe: 115200
Redir after boot: enabled

6144
Server
Custom- desel all but base
/root/install.log


2. Install OS
Set partitions:
/ 3072M
Disk 1: Swap 4096M
Disk 2: Swap 4096M
/vz (the rest)

No firewall
Disable SElinux

FC4
Choose a custom install, the “minimal” package

CentOS4 Server CD (only option for 2950)
Choose the “minimal” package

3. create ssh key, add swsoft key to authorized_keys

(enter to everything)

cd /root/
ssh-keygen -t dsa

echo "ssh-dss AAAAB3NzaC1kc3MAAACBAIB7XHmpcsa1D/vsKS+mzEvBVo7udew/aRahbjnjywL4YXT7ruWzpGo7R3UA8GKIgVHoQGUbuXrJz5ICtBEXOlbgXVEz7Bu5nXOG2C9i/8pmFtUw6GwACII8uZE7t/Zq5CRj+5zN1Rv7n/4X4RAjzk5BHXoZ62a32FTbS4tPvyPrAAAAFQDiUi7YEiffNzGvB544lQpRdn29TwAAAIBkUchLTBE1+sX11iC80yhNh9EGeCMLdkT8UqHsTSaD0myQPxGALuNY4rO31q+25uKqSRDTXd6ZWXp4bplA6Ls6jl7r7XVR9loJ7WvRMqc4Dv68TZ7h2UBqAhu4det2UF4LHim3DNf1gKn9DzWNkmI/N2FvVVAL4aM3sPunVIvUYgAAAIAJX38INABVgnCEBKqpv5FMLm3k9MgMn54oQFmylBbgM0w4SUeuDj3jLrkJf8dT9w8ByEDb8xviexpWrPsbz2DeThle4xgzcyKrUkt2Wugtmm4DwS0S/68BAA9wXoWD/vZ67GBoCKH7uq8VAjUVcCbJ/7IydljnFAZoJpktdwN7iQ== russup@support.sw-soft.com" >> /root/.ssh/authorized_keys

4. setup access to backup2:

echo "10.1.4.3        backup2" >> /etc/hosts

Copy id_dsa.pub to backup2 and add it to the end of ~root/.ssh/authorized_keys:

cat .ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys' 

ssh backup2

(confirm you can connect w/o entering a password)

5. add bash backward history search 
echo "\"\M-z\": history-search-backward
\"\M-x\": history-search-backward " > /root/.inputrc
echo "export GREP_OPTIONS='--color=auto'" >> ~/.bashrc
echo "\"\e[5~\": history-search-backward" >> ~/.inputrc
echo "\"\e[6~\": history-search-forward" >> ~/.inputrc

6. add to /root/.bashrc, create log dir

vi /root/.bashrc
alias v='vzctl'
alias mbm='mb mount'
alias mbu='mb umount'
export PS1="[\u@\h \w]# "
alias cv='cd /vz/private'
alias cv1='cd /vz1/private'
alias cv2='cd /vz2/private'

mkdir /root/logs

7. make sure serial console and kernel messages are working

grub

add these 2 lines to the top:

vi /boot/grub/menu.lst
serial --unit=0 --speed= 115200
terminal --timeout=10 serial console

and the kernel line should look like:

kernel /boot/vmlinuz-2.4.20-020stab009.24.777-enterprise ro root=/dev/sda1 console=tty0 console=ttyS0,115200

Remove the line:
hiddensplashmenu

lilo

Make sure the append line in /etc/lilo.conf looks like this:

append="console=tty console=ttyS0,38400"

If you edit /etc/lilo.conf don’t forget to run lilo

# lilo
Added linux
Added linux-up
Added linux+virtuozzo *

(note: the last console listed is the one that gets output from the rc scripts.)



add to /etc/inittab and /etc/securetty

echo "s0:2345:respawn:/sbin/agetty -L ttyS0 115200 linux" >> /etc/inittab
echo "ttyS0" >> /etc/securetty

kill -1 1

8. configuring eth1 for management net:

echo "10.1.5/24 via 10.1.4.2
10.1.6/24 via 10.1.4.2" >> /etc/sysconfig/network-scripts/route-eth1

reboot the machine to make sure that the interface is configured properly.

also verify that the routes are in place like so:

netstat -rn | grep 10.1.4.2
10.1.5.0        10.1.4.2        255.255.255.0   UG       40 0          0 eth1
10.1.6.0        10.1.4.2        255.255.255.0   UG       40 0          0 eth1

9. make sure the hostname on the system is properly set - in the form virt4.johncompanies.com - and make sure it is set properly: 
cat /etc/sysconfig/network

10. Check/update fstab

vi /etc/fstab

/etc/fstab entries should look something like (note noatime option):

/dev/sda3               /vz                     ext3    defaults,noatime 1 2
/dev/sdb1               /vz1                    ext3    defaults,noatime 1 2

make sure swap is configured in fstab

Use real devices instead of LABEL

11. Install Virtuozzo 3.0

If it wasn’t done at install time:

setenforce 0

umount /vz
umount /vz1

mke2fs -i1024 -b1024 -j /dev/sda3 &
mke2fs -i1024 -b1024 -j /dev/sdb2 &

On CentOS 4:
yum install rpm-build; yum update iproute

mount -a

scp backup2:/d4/vzrpms/iso/Rel300.tar.gz /vz/.

cd /vz

tar xzf Rel300.tar.gz

cd Rel300/HW

./install

Choose Enterprise kernel
Vzagent password: k0zub1k
DON’T reboot

vzup2date
User: 
gdawson
Pass: 
qjdK1R
(run again. till no more updates offered)

reboot

Run vzup2date (make sure to choose enterprise kernel) and/or install latest kernel:

ftp downloads.swsoft.com
cd /updates/Rel300/20061206
bin
get vzkernel-enterprise-2.6.9-023stab033.9.i686.rpm
get vzmodules-enterprise-2.6.9-023stab033.9.i686.rpm
bye

rpm -ivh vzkernel-enterprise-2.6.9-023stab033.9.i686.rpm \
vzmodules-enterprise-2.6.9-023stab033.9.i686.rpm

vi /boot/grub/menu.lst
(set default kernel to new kernel)

12. Install various RPMs 

scp backup2:/mnt/data4/linuxrpms/* /tmp

install them in this order:

12a. http fetcher

rpm -ivh /tmp/http_fetcher-1.1.0-1.i386.rpm
rpm -ivh /tmp/fetch-1.0-1.i386.rpm

(ignore error on CentOS4)

Now the fetch command, which vemake calls to email the new system owner, will work (it downloads the welcome email from www.johncompanies.com

12b. ntp client

sometimes ntp isn’t included in which case:

FCx:

rpm -ivh /tmp/libcap-1.10-22.i386.rpm
rpm -ivh /tmp/compat-readline43-4.3-2.i386.rpm
rpm -ivh /tmp/ntp-4.1.2-5.i386.rpm

CentOS4:

rpm -ivh /tmp/ntp-4.1.2-5.i386.rpm


rm -f /etc/ntp.conf

Create a new file with one line:

echo "server 10.1.4.5" > /etc/ntp.conf 

Then set ntpd so it will start at boot:

chkconfig --level 3 ntpd on
service ntpd start 

Make sure it’s talking to the ntp server:

ntpq -p

Correct output should have something unfer the refid col:

     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 10.1.4.105      clock1.redhat.c  2 u    -   64    1    0.370  -706677   0.008

12c. adaptec machines only: install aaccli for raid controller (LSI card util is with bin files):

rpm -ivh /tmp/aacapps-4.1-0.i386.rpm

May have to run:

cd /dev
./MAKEDEV.aac aac0

13. copy over binaries/scripts from backup2

scp backup2:/mnt/data4/bin/linux/* /usr/local/sbin/

14. setup root crontab (alter backup time): 
crontab -e
1 1 * * * /usr/local/sbin/backup
*/15 * * * * /usr/local/sbin/vim.watchdog.30 > /dev/null
#56 4,16 * * * root /usr/sbin/vzkeyupdate -c

On Perc5/i machines:

*/5 * * * * /usr/local/sbin/perc5iraidchk

15. setup backups 

echo '#!/bin/sh
backupdir=/mnt/data4/virt19

## ENTRY /etc
## ENTRY /usr/local/sbin
## ENTRY /root/logs
' > /usr/local/sbin/backup.config

16. setup bb 

edit passwd and add the bb user:
echo "bb:x:1984:1984:Big Brother:/home/bb:/bin/bash" >> /etc/passwd

edit group and add the bb group:
echo "bb:x:1984:" >> /etc/group

rebuild the password database:

pwconv

Create bb’s home dir and set ownership:

mkdir /home/bb
chown bb.bb /home/bb

Go to bb’s home and fetch the client tarball:

cd ~bb
scp backup2:/mnt/data4/build/bb/bb-linux.tar .

tar xf bb-linux.tar

cd bbc1.9e-btf/etc/

echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
echo "10.1.4.69 virt19.johncompanies.com # ssh" >> /home/bb/bbc1.9e-btf/etc/bb-hosts

chmod +r /var/log/messages

cd ..
su bb
./runbb.sh start
exit

Add this line to /etc/rc.d/rc.local so bb will start at boot:

echo "su - bb -c \"cd /home/bb/bbc1.9e-btf; ./runbb.sh start\"" >> /etc/rc.d/rc.local 

17. Uncomment/add lines with the appropriate IP addresses in /etc/ssh/sshd_config:

vi /etc/ssh/sshd_config 
ListenAddress 69.55.232.162
ListenAddress 10.1.4.66

And make sure to uncomment
Protocol 2,1

18. setup backup2 nfs mounts

setup portmap to start with server, and start it: 

chkconfig --level 3 portmap on
service portmap start

mb mount

run df to confirm that nfs mounts are mounted

19. m	ake alias for /vzconf
ln -s /etc/sysconfig/vz-scripts /vzconf

20. copy over the /etc/sysconfig/vz-scripts/4unlimited template file from the last virt server and put it in /etc/sysconfig/vz-scripts

scp 10.1.4.68:/etc/sysconfig/vz-scripts/4unlimited /etc/sysconfig/vz-scripts/

NOTE: if this system is more powerful (RAM, CPU) then the previous, you will want to create a new one using vzsplit:

cd /etc/sysconfig/vz-scripts/
vzsplit -f 4unlimited

21. edit /etc/sysconfig/vz 

vi /etc/sysconfig/vz 

enable fastboot:
VZFASTBOOT=yes

add/edit to traffic parameters section:
TRAFFIC_SHAPING=yes
BANDWIDTH="eth0:102400"
TOTALRATE="eth0:1:102400"

for 2 nics, uncomment this line:
VE_ROUTE_SRC_DEV="eth0"

Make sure the IPTABLES section looks like:
IPTABLES="ip_tables iptable_filter iptable_mangle ipt_limit ipt_multiport ipt_tos ipt_TOS ipt_REJECT ipt_tcpmss ipt_ttl ipt_LOG ipt_length ip_conntrack ip_conntrack_ftp  ip_conntrack_irc ipt_conntrack ipt_state ipt_helper iptable_nat ip_nat_ftp ip_nat_irc ipt_REDIRECT"

22. disable periodic fsck's, lower the root padding on the /vz and /vz1 filesystems to 100MB

tune2fs -i 0 /dev/sda3
tune2fs -i 0 /dev/sdb2
tune2fs -c -1 /dev/sda3
tune2fs -c -1 /dev/sdb2
tune2fs -r 100000 /dev/sda3
tune2fs -r 100000 /dev/sdb2

23. copy over our versions of the vzpp files:

/vz/private/1/root/var/vzcp/static/vz/skins/
/vz/private/1/root/etc/vzcp/pp/menu.xml
/vz/private/1/root/etc/vzcp/pp/dashboard.xml
/vz/private/1/root/etc/vzcp/vzcpcon.conf

eg (2.6.1 or 3.0):
rsync -va -e ssh backup2:/d4/build/vzcp/var/vzcp/static/vz/skins/  /vz/private/1/root/var/vzcp/static/vz/skins/
rsync -va -e ssh backup2:/d4/build/vzcp/etc/vzcp/pp/menu.xml  /vz/private/1/root/etc/vzcp/pp/menu.xml
rsync -va -e ssh backup2:/d4/build/vzcp/etc/vzcp/pp/dashboard.xml /vz/private/1/root/etc/vzcp/pp/dashboard.xml
rsync -va -e ssh backup2:/d4/build/vzcp/etc/vzcp/vzcpcon.conf  /vz/private/1/root/etc/vzcp/vzcpcon.conf

To disable backups from scratch:

In svcve edit /etc/vzcp/vzcpcon.conf

In section:

<auth>
	<type>VEUser</type>
	<user>root</user>

Add into/below:
<cred>
	<id>maintenance</id>
</cred>

The following:
<cred>
	<id>show</id>
</cred>
<cred>
	<id>reinstall</id>
</cred>
<cred>
	<id>repair_mode</id>
</cred>

Will finally look like:

<cred>
	<id>maintenance</id>
		<cred>
			<id>show</id>
		</cred>
		<cred>
			<id>reinstall</id>
		</cred>
		<cred>
			<id>repair_mode</id>
		</cred>
	</cred>
</cred>

24. template install

mb mount

install debian 3.0 templates

cd /backup4/vzrpms/distro/debian/3.0/p1/
for f in `cat /backup4/vzrpms/distro/debian/3.0/install_list`; do rpm -ivh $f; done

install debian 3.1 templates

cd /backup4/vzrpms/distro/debian/3.1/p3/
for f in `ls`; do rpm -ivh $f; done

install redhat templates

cd /backup4/vzrpms/distro/rh9/current/
for f in `cat /backup4/vzrpms/distro/rh9/install_list`; do rpm -ivh $f; done

install fedora1 templates

cd /backup4/vzrpms/distro/fc1/current/
for f in `cat /backup4/vzrpms/distro/fc1/install_list`; do rpm -ivh $f; done

change 'krb5-devel-1.3.4-5' to 'krb5-devel-1.3.3-6' in (applies to pre-p5)
/vz/template/fedora-core-devel-1/conf/fedora-core-devel-1.conf.20050314

install fedora2 templates

cd /backup4/vzrpms/distro/fc2/current/
for f in `cat /backup4/vzrpms/distro/fc2/install_list`; do rpm -ivh $f; done

install suse9.2 templates

cd /backup4/vzrpms/distro/suse/current/
for f in `cat /backup4/vzrpms/distro/suse/install_list`; do rpm -ivh $f; done

install centos templates

cd /backup4/vzrpms/distro/centos/current/
for f in `cat /backup4/vzrpms/distro/centos/install_list`; do rpm -ivh $f; done

install fedora4 templates

cd /backup4/vzrpms/distro/fc4/current/
for f in `cat /backup4/vzrpms/distro/fc2/install_list`; do rpm -ivh $f; done

vzpkgcache

To pull templates from another machine (before hand stop ve 1 and move it’s template so you preserve the latest template files just setup by vzup2date):

rsync -a -e ssh 10.1.4.68:/vz/template/ /vz/template/

Make sure ubuntu source is defined:

UBU_SERVER=http://archive.ubuntu.com 

And change FC source:
FC_SERVER=http://rhold.fedoraproject.org

In /etc/vztt/vztt.conf

rsync -a -e ssh 10.1.4.68:/etc/vztt/vztt.conf  /etc/vztt/


To update the cache:

vzpkg update cache ubuntu-6.10-x86

25. add tun device support

Add to /etc/modules.conf:
echo "alias char-major-10-200 tun" >> /etc/modules.conf

depmod -a

echo "modprobe tun" >> /etc/rc.local

26. add traffic shaping support

Re-define class 0:

vi /etc/sysconfig/vz-scripts/networks_classes
0 69.55.224.0/20
0 10.1.4.0/24

make sure 1 is defined (0.0.0.0) and no other classes are needed

Then start it

/etc/init.d/vz shaperrestart

27. RAIDmon (only applies to LSI 320 PCI card)

cd /tmp
cp /backup4/build/megaraid/linuxmon30.tgz .
tar xzf linuxmon30.tgz
cd linuxmon30
./install
(say no)
mv –f raidmon.init /etc/init.d/raidmon
service raidmon start
tail -f /var/log/messages
(check for output)

28. udp dos fix

echo "/usr/local/sbin/iptables.sh" >> /etc/rc.d/rc.local 

29. make rc.local start before vz

mv /etc/rc.d/rc3.d/S99local /etc/rc.d/rc3.d/S95local

30. disable unwanted services/daemons

On CentOS:
chkconfig --del kudzu
chkconfig --del cpuspeed

This should be taken care of by vz install so probably safe to skip

chkconfig --del kudzu
chkconfig --del cups
chkconfig --del gpm
chkconfig --del isdn
chkconfig --del pcmcia
chkconfig --del apmd
chkconfig --del rhnsd
chkconfig --del mdmonitor

31. load mrtg graphs

On FCx xinetd not installed so:

rpm -ivh /tmp/xinetd-2.3.11-1.7x.i386.rpm

create /etc/xinetd.d/load with the following contents:

echo "service load
{
        disable = no
        flags           = REUSE
        socket_type     = stream
        wait            = no
        user            = root
        server          = /usr/local/sbin/load.pl
}" > /etc/xinetd.d/load

Add this line to /etc/services:
echo "load            12384/tcp" >> /etc/services

On FCx: if xinetd isn’t running, configure it for startup at boot time:

chkconfig --level 3 xinetd on
service xinetd start

32. Tasks to do on backup2:

setup backup dirs:
mkdir -p /mnt/data4/virt19/0

add the system to 
vi /mnt/data4/bin/rsync_houseclean 

33. Tasks to do on mail:

Update backup graph:
vi /usr/local/www/mgmt/cgi/backupgraph.pl
(add hostname)

add to bbhosts:
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
su bb
cd
bbsrc/bb/runbb.sh restart ; exit

add to load graph config:
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg
(add new section for new machine)

34. add to management db (on mail and devweb) jc.ref_machines and jc.ref_templates

insert into ref_machines values (null,'virt19','virt19.johncompanies.com',0,'l');
select * from ref_machines;
insert into systems values ('', 'col00001',43,null,null,100,'virt19.johncompanies.com',null,'2008-01-07',null,null,1,0,null,null,null);
select * from systems where hostname='virt19.johncompanies.com';
update ipmap set cid='col00001', sysid=3772, assign_date='2008-01-07', status=1 where ip='69.55.236.2';
update ipmap set cid='col00001', sysid=3772, assign_date='2008-01-07', status=1 where ip='69.55.230.109';
insert into ref_templates values ('','fedora-core-1/20050725',33,'Fedora Core 1',1);
insert into ref_templates values ('','fedora-core-2/20050725',33,'Fedora Core 2',1);
insert into ref_templates values ('','debian-3.1/20050927',33,'Debian 3.1',1);

35. Make sure that ns1c has proper forward and reverse entries for the new system (both private and public addresses), so that new system emailing will work.
vr johncompanies.com
vi internal.johncompanies.com
rndc reload johncompanies.com IN private
(edit the PTR too)


36. add an outside blocking rule to the firewall, so this machine can only be reached from inside the firewall. 

Follow example already in firewall virt19 is:
ipfw add 00149 allow ip from { 69.55.225.225 or 69.55.230.2 or 69.55.230.10 or 69.55.230.3 or 62.213.72.101 or 64.131.90.27 or 80.89.128.178 or 82.200.65.190 or 84.237.120.254 or 89.251.97.2 or 195.214.232.10 or 195.214.233.10 or 208.251.212.1 or 213.130.28.26 } to 69.55.236.2
ipfw add 00149 deny ip from any to 69.55.236.2

virt12 would be 00142

37. add to server/cabinet map

39. make /vz1/private
mkdir /vz1/private ; chmod 0755 /vz1/private
mkdir /vz1/root ; chmod 0755 /vz1/root
Retrieved from "https://wiki.jcihosting.com/index.php?title=Virtuozzo_Server_Install&oldid=1386"