Editing Virtuozzo Server Install (section)

== 3.x ==

Last updated 2007-04-5

<pre>1. enable post info to console
Supermicro:
Console redirection:
Com port addr: on-board COM A
Baud: 38400
Console type: vt100
Flow control: none
Console connection: direct
Continue cr after post: off

2450:
Make sure running bios A09
Console Redirection: VT100/VT220

2950:
Serial comm.: on with cons. Redir via com1
Failsafe: 115200
Redir after boot: enabled

6144
Server
Custom- desel all but base
/root/install.log


2. Install OS
Set partitions:
/ 3072M
Disk 1: Swap 4096M
Disk 2: Swap 4096M
/vz (the rest)

No firewall
Disable SElinux

FC4
Choose a custom install, the “minimal” package

CentOS4 Server CD (only option for 2950)
Choose the “minimal” package

3. create ssh key, add swsoft key to authorized_keys

(enter to everything)

cd /root/
ssh-keygen -t dsa

echo "ssh-dss AAAAB3NzaC1kc3MAAACBAIB7XHmpcsa1D/vsKS+mzEvBVo7udew/aRahbjnjywL4YXT7ruWzpGo7R3UA8GKIgVHoQGUbuXrJz5ICtBEXOlbgXVEz7Bu5nXOG2C9i/8pmFtUw6GwACII8uZE7t/Zq5CRj+5zN1Rv7n/4X4RAjzk5BHXoZ62a32FTbS4tPvyPrAAAAFQDiUi7YEiffNzGvB544lQpRdn29TwAAAIBkUchLTBE1+sX11iC80yhNh9EGeCMLdkT8UqHsTSaD0myQPxGALuNY4rO31q+25uKqSRDTXd6ZWXp4bplA6Ls6jl7r7XVR9loJ7WvRMqc4Dv68TZ7h2UBqAhu4det2UF4LHim3DNf1gKn9DzWNkmI/N2FvVVAL4aM3sPunVIvUYgAAAIAJX38INABVgnCEBKqpv5FMLm3k9MgMn54oQFmylBbgM0w4SUeuDj3jLrkJf8dT9w8ByEDb8xviexpWrPsbz2DeThle4xgzcyKrUkt2Wugtmm4DwS0S/68BAA9wXoWD/vZ67GBoCKH7uq8VAjUVcCbJ/7IydljnFAZoJpktdwN7iQ== russup@support.sw-soft.com" >> /root/.ssh/authorized_keys

4. setup access to backup2:

echo "10.1.4.3        backup2" >> /etc/hosts

Copy id_dsa.pub to backup2 and add it to the end of ~root/.ssh/authorized_keys:

cat .ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys' 

ssh backup2

(confirm you can connect w/o entering a password)

5. add bash backward history search 
echo "\"\M-z\": history-search-backward
\"\M-x\": history-search-backward " > /root/.inputrc
echo "export GREP_OPTIONS='--color=auto'" >> ~/.bashrc
echo "\"\e[5~\": history-search-backward" >> ~/.inputrc
echo "\"\e[6~\": history-search-forward" >> ~/.inputrc

6. add to /root/.bashrc, create log dir

vi /root/.bashrc
alias v='vzctl'
alias mbm='mb mount'
alias mbu='mb umount'
export PS1="[\u@\h \w]# "
alias cv='cd /vz/private'
alias cv1='cd /vz1/private'
alias cv2='cd /vz2/private'

mkdir /root/logs

7. make sure serial console and kernel messages are working

grub

add these 2 lines to the top:

vi /boot/grub/menu.lst
serial --unit=0 --speed= 115200
terminal --timeout=10 serial console

and the kernel line should look like:

kernel /boot/vmlinuz-2.4.20-020stab009.24.777-enterprise ro root=/dev/sda1 console=tty0 console=ttyS0,115200

Remove the line:
hiddensplashmenu

lilo

Make sure the append line in /etc/lilo.conf looks like this:

append="console=tty console=ttyS0,38400"

If you edit /etc/lilo.conf don’t forget to run lilo

# lilo
Added linux
Added linux-up
Added linux+virtuozzo *

(note: the last console listed is the one that gets output from the rc scripts.)



add to /etc/inittab and /etc/securetty

echo "s0:2345:respawn:/sbin/agetty -L ttyS0 115200 linux" >> /etc/inittab
echo "ttyS0" >> /etc/securetty

kill -1 1

8. configuring eth1 for management net:

echo "10.1.5/24 via 10.1.4.2
10.1.6/24 via 10.1.4.2" >> /etc/sysconfig/network-scripts/route-eth1

reboot the machine to make sure that the interface is configured properly.

also verify that the routes are in place like so:

netstat -rn | grep 10.1.4.2
10.1.5.0        10.1.4.2        255.255.255.0   UG       40 0          0 eth1
10.1.6.0        10.1.4.2        255.255.255.0   UG       40 0          0 eth1

9. make sure the hostname on the system is properly set - in the form virt4.johncompanies.com - and make sure it is set properly: 
cat /etc/sysconfig/network

10. Check/update fstab

vi /etc/fstab

/etc/fstab entries should look something like (note noatime option):

/dev/sda3               /vz                     ext3    defaults,noatime 1 2
/dev/sdb1               /vz1                    ext3    defaults,noatime 1 2

make sure swap is configured in fstab

Use real devices instead of LABEL

11. Install Virtuozzo 3.0

If it wasn’t done at install time:

setenforce 0

umount /vz
umount /vz1

mke2fs -i1024 -b1024 -j /dev/sda3 &
mke2fs -i1024 -b1024 -j /dev/sdb2 &

On CentOS 4:
yum install rpm-build; yum update iproute

mount -a

scp backup2:/d4/vzrpms/iso/Rel300.tar.gz /vz/.

cd /vz

tar xzf Rel300.tar.gz

cd Rel300/HW

./install

Choose Enterprise kernel
Vzagent password: k0zub1k
DON’T reboot

vzup2date
User: 
gdawson
Pass: 
qjdK1R
(run again. till no more updates offered)

reboot

Run vzup2date (make sure to choose enterprise kernel) and/or install latest kernel:

ftp downloads.swsoft.com
cd /updates/Rel300/20061206
bin
get vzkernel-enterprise-2.6.9-023stab033.9.i686.rpm
get vzmodules-enterprise-2.6.9-023stab033.9.i686.rpm
bye

rpm -ivh vzkernel-enterprise-2.6.9-023stab033.9.i686.rpm \
vzmodules-enterprise-2.6.9-023stab033.9.i686.rpm

vi /boot/grub/menu.lst
(set default kernel to new kernel)

12. Install various RPMs 

scp backup2:/mnt/data4/linuxrpms/* /tmp

install them in this order:

12a. http fetcher

rpm -ivh /tmp/http_fetcher-1.1.0-1.i386.rpm
rpm -ivh /tmp/fetch-1.0-1.i386.rpm

(ignore error on CentOS4)

Now the fetch command, which vemake calls to email the new system owner, will work (it downloads the welcome email from www.johncompanies.com

12b. ntp client

sometimes ntp isn’t included in which case:

FCx:

rpm -ivh /tmp/libcap-1.10-22.i386.rpm
rpm -ivh /tmp/compat-readline43-4.3-2.i386.rpm
rpm -ivh /tmp/ntp-4.1.2-5.i386.rpm

CentOS4:

rpm -ivh /tmp/ntp-4.1.2-5.i386.rpm


rm -f /etc/ntp.conf

Create a new file with one line:

echo "server 10.1.4.5" > /etc/ntp.conf 

Then set ntpd so it will start at boot:

chkconfig --level 3 ntpd on
service ntpd start 

Make sure it’s talking to the ntp server:

ntpq -p

Correct output should have something unfer the refid col:

     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 10.1.4.105      clock1.redhat.c  2 u    -   64    1    0.370  -706677   0.008

12c. adaptec machines only: install aaccli for raid controller (LSI card util is with bin files):

rpm -ivh /tmp/aacapps-4.1-0.i386.rpm

May have to run:

cd /dev
./MAKEDEV.aac aac0

13. copy over binaries/scripts from backup2

scp backup2:/mnt/data4/bin/linux/* /usr/local/sbin/

14. setup root crontab (alter backup time): 
crontab -e
1 1 * * * /usr/local/sbin/backup
*/15 * * * * /usr/local/sbin/vim.watchdog.30 > /dev/null
#56 4,16 * * * root /usr/sbin/vzkeyupdate -c

On Perc5/i machines:

*/5 * * * * /usr/local/sbin/perc5iraidchk

15. setup backups 

echo '#!/bin/sh
backupdir=/mnt/data4/virt19

## ENTRY /etc
## ENTRY /usr/local/sbin
## ENTRY /root/logs
' > /usr/local/sbin/backup.config

16. setup bb 

edit passwd and add the bb user:
echo "bb:x:1984:1984:Big Brother:/home/bb:/bin/bash" >> /etc/passwd

edit group and add the bb group:
echo "bb:x:1984:" >> /etc/group

rebuild the password database:

pwconv

Create bb’s home dir and set ownership:

mkdir /home/bb
chown bb.bb /home/bb

Go to bb’s home and fetch the client tarball:

cd ~bb
scp backup2:/mnt/data4/build/bb/bb-linux.tar .

tar xf bb-linux.tar

cd bbc1.9e-btf/etc/

echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
echo "10.1.4.69 virt19.johncompanies.com # ssh" >> /home/bb/bbc1.9e-btf/etc/bb-hosts

chmod +r /var/log/messages

cd ..
su bb
./runbb.sh start
exit

Add this line to /etc/rc.d/rc.local so bb will start at boot:

echo "su - bb -c \"cd /home/bb/bbc1.9e-btf; ./runbb.sh start\"" >> /etc/rc.d/rc.local 

17. Uncomment/add lines with the appropriate IP addresses in /etc/ssh/sshd_config:

vi /etc/ssh/sshd_config 
ListenAddress 69.55.232.162
ListenAddress 10.1.4.66

And make sure to uncomment
Protocol 2,1

18. setup backup2 nfs mounts

setup portmap to start with server, and start it: 

chkconfig --level 3 portmap on
service portmap start

mb mount

run df to confirm that nfs mounts are mounted

19. m	ake alias for /vzconf
ln -s /etc/sysconfig/vz-scripts /vzconf

20. copy over the /etc/sysconfig/vz-scripts/4unlimited template file from the last virt server and put it in /etc/sysconfig/vz-scripts

scp 10.1.4.68:/etc/sysconfig/vz-scripts/4unlimited /etc/sysconfig/vz-scripts/

NOTE: if this system is more powerful (RAM, CPU) then the previous, you will want to create a new one using vzsplit:

cd /etc/sysconfig/vz-scripts/
vzsplit -f 4unlimited

21. edit /etc/sysconfig/vz 

vi /etc/sysconfig/vz 

enable fastboot:
VZFASTBOOT=yes

add/edit to traffic parameters section:
TRAFFIC_SHAPING=yes
BANDWIDTH="eth0:102400"
TOTALRATE="eth0:1:102400"

for 2 nics, uncomment this line:
VE_ROUTE_SRC_DEV="eth0"

Make sure the IPTABLES section looks like:
IPTABLES="ip_tables iptable_filter iptable_mangle ipt_limit ipt_multiport ipt_tos ipt_TOS ipt_REJECT ipt_tcpmss ipt_ttl ipt_LOG ipt_length ip_conntrack ip_conntrack_ftp  ip_conntrack_irc ipt_conntrack ipt_state ipt_helper iptable_nat ip_nat_ftp ip_nat_irc ipt_REDIRECT"

22. disable periodic fsck's, lower the root padding on the /vz and /vz1 filesystems to 100MB

tune2fs -i 0 /dev/sda3
tune2fs -i 0 /dev/sdb2
tune2fs -c -1 /dev/sda3
tune2fs -c -1 /dev/sdb2
tune2fs -r 100000 /dev/sda3
tune2fs -r 100000 /dev/sdb2

23. copy over our versions of the vzpp files:

/vz/private/1/root/var/vzcp/static/vz/skins/
/vz/private/1/root/etc/vzcp/pp/menu.xml
/vz/private/1/root/etc/vzcp/pp/dashboard.xml
/vz/private/1/root/etc/vzcp/vzcpcon.conf

eg (2.6.1 or 3.0):
rsync -va -e ssh backup2:/d4/build/vzcp/var/vzcp/static/vz/skins/  /vz/private/1/root/var/vzcp/static/vz/skins/
rsync -va -e ssh backup2:/d4/build/vzcp/etc/vzcp/pp/menu.xml  /vz/private/1/root/etc/vzcp/pp/menu.xml
rsync -va -e ssh backup2:/d4/build/vzcp/etc/vzcp/pp/dashboard.xml /vz/private/1/root/etc/vzcp/pp/dashboard.xml
rsync -va -e ssh backup2:/d4/build/vzcp/etc/vzcp/vzcpcon.conf  /vz/private/1/root/etc/vzcp/vzcpcon.conf

To disable backups from scratch:

In svcve edit /etc/vzcp/vzcpcon.conf

In section:

<auth>
	<type>VEUser</type>
	<user>root</user>

Add into/below:
<cred>
	<id>maintenance</id>
</cred>

The following:
<cred>
	<id>show</id>
</cred>
<cred>
	<id>reinstall</id>
</cred>
<cred>
	<id>repair_mode</id>
</cred>

Will finally look like:

<cred>
	<id>maintenance</id>
		<cred>
			<id>show</id>
		</cred>
		<cred>
			<id>reinstall</id>
		</cred>
		<cred>
			<id>repair_mode</id>
		</cred>
	</cred>
</cred>

24. template install

mb mount

install debian 3.0 templates

cd /backup4/vzrpms/distro/debian/3.0/p1/
for f in `cat /backup4/vzrpms/distro/debian/3.0/install_list`; do rpm -ivh $f; done

install debian 3.1 templates

cd /backup4/vzrpms/distro/debian/3.1/p3/
for f in `ls`; do rpm -ivh $f; done

install redhat templates

cd /backup4/vzrpms/distro/rh9/current/
for f in `cat /backup4/vzrpms/distro/rh9/install_list`; do rpm -ivh $f; done

install fedora1 templates

cd /backup4/vzrpms/distro/fc1/current/
for f in `cat /backup4/vzrpms/distro/fc1/install_list`; do rpm -ivh $f; done

change 'krb5-devel-1.3.4-5' to 'krb5-devel-1.3.3-6' in (applies to pre-p5)
/vz/template/fedora-core-devel-1/conf/fedora-core-devel-1.conf.20050314

install fedora2 templates

cd /backup4/vzrpms/distro/fc2/current/
for f in `cat /backup4/vzrpms/distro/fc2/install_list`; do rpm -ivh $f; done

install suse9.2 templates

cd /backup4/vzrpms/distro/suse/current/
for f in `cat /backup4/vzrpms/distro/suse/install_list`; do rpm -ivh $f; done

install centos templates

cd /backup4/vzrpms/distro/centos/current/
for f in `cat /backup4/vzrpms/distro/centos/install_list`; do rpm -ivh $f; done

install fedora4 templates

cd /backup4/vzrpms/distro/fc4/current/
for f in `cat /backup4/vzrpms/distro/fc2/install_list`; do rpm -ivh $f; done

vzpkgcache

To pull templates from another machine (before hand stop ve 1 and move it’s template so you preserve the latest template files just setup by vzup2date):

rsync -a -e ssh 10.1.4.68:/vz/template/ /vz/template/

Make sure ubuntu source is defined:

UBU_SERVER=http://archive.ubuntu.com 

And change FC source:
FC_SERVER=http://rhold.fedoraproject.org

In /etc/vztt/vztt.conf

rsync -a -e ssh 10.1.4.68:/etc/vztt/vztt.conf  /etc/vztt/


To update the cache:

vzpkg update cache ubuntu-6.10-x86

25. add tun device support

Add to /etc/modules.conf:
echo "alias char-major-10-200 tun" >> /etc/modules.conf

depmod -a

echo "modprobe tun" >> /etc/rc.local

26. add traffic shaping support

Re-define class 0:

vi /etc/sysconfig/vz-scripts/networks_classes
0 69.55.224.0/20
0 10.1.4.0/24

make sure 1 is defined (0.0.0.0) and no other classes are needed

Then start it

/etc/init.d/vz shaperrestart

27. RAIDmon (only applies to LSI 320 PCI card)

cd /tmp
cp /backup4/build/megaraid/linuxmon30.tgz .
tar xzf linuxmon30.tgz
cd linuxmon30
./install
(say no)
mv –f raidmon.init /etc/init.d/raidmon
service raidmon start
tail -f /var/log/messages
(check for output)

28. udp dos fix

echo "/usr/local/sbin/iptables.sh" >> /etc/rc.d/rc.local 

29. make rc.local start before vz

mv /etc/rc.d/rc3.d/S99local /etc/rc.d/rc3.d/S95local

30. disable unwanted services/daemons

On CentOS:
chkconfig --del kudzu
chkconfig --del cpuspeed

This should be taken care of by vz install so probably safe to skip

chkconfig --del kudzu
chkconfig --del cups
chkconfig --del gpm
chkconfig --del isdn
chkconfig --del pcmcia
chkconfig --del apmd
chkconfig --del rhnsd
chkconfig --del mdmonitor

31. load mrtg graphs

On FCx xinetd not installed so:

rpm -ivh /tmp/xinetd-2.3.11-1.7x.i386.rpm

create /etc/xinetd.d/load with the following contents:

echo "service load
{
        disable = no
        flags           = REUSE
        socket_type     = stream
        wait            = no
        user            = root
        server          = /usr/local/sbin/load.pl
}" > /etc/xinetd.d/load

Add this line to /etc/services:
echo "load            12384/tcp" >> /etc/services

On FCx: if xinetd isn’t running, configure it for startup at boot time:

chkconfig --level 3 xinetd on
service xinetd start

32. Tasks to do on backup2:

setup backup dirs:
mkdir -p /mnt/data4/virt19/0

add the system to 
vi /mnt/data4/bin/rsync_houseclean 

33. Tasks to do on mail:

Update backup graph:
vi /usr/local/www/mgmt/cgi/backupgraph.pl
(add hostname)

add to bbhosts:
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
su bb
cd
bbsrc/bb/runbb.sh restart ; exit

add to load graph config:
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg
(add new section for new machine)

34. add to management db (on mail and devweb) jc.ref_machines and jc.ref_templates

insert into ref_machines values (null,'virt19','virt19.johncompanies.com',0,'l');
select * from ref_machines;
insert into systems values ('', 'col00001',43,null,null,100,'virt19.johncompanies.com',null,'2008-01-07',null,null,1,0,null,null,null);
select * from systems where hostname='virt19.johncompanies.com';
update ipmap set cid='col00001', sysid=3772, assign_date='2008-01-07', status=1 where ip='69.55.236.2';
update ipmap set cid='col00001', sysid=3772, assign_date='2008-01-07', status=1 where ip='69.55.230.109';
insert into ref_templates values ('','fedora-core-1/20050725',33,'Fedora Core 1',1);
insert into ref_templates values ('','fedora-core-2/20050725',33,'Fedora Core 2',1);
insert into ref_templates values ('','debian-3.1/20050927',33,'Debian 3.1',1);

35. Make sure that ns1c has proper forward and reverse entries for the new system (both private and public addresses), so that new system emailing will work.
vr johncompanies.com
vi internal.johncompanies.com
rndc reload johncompanies.com IN private
(edit the PTR too)


36. add an outside blocking rule to the firewall, so this machine can only be reached from inside the firewall. 

Follow example already in firewall virt19 is:
ipfw add 00149 allow ip from { 69.55.225.225 or 69.55.230.2 or 69.55.230.10 or 69.55.230.3 or 62.213.72.101 or 64.131.90.27 or 80.89.128.178 or 82.200.65.190 or 84.237.120.254 or 89.251.97.2 or 195.214.232.10 or 195.214.233.10 or 208.251.212.1 or 213.130.28.26 } to 69.55.236.2
ipfw add 00149 deny ip from any to 69.55.236.2

virt12 would be 00142

37. add to server/cabinet map

39. make /vz1/private
mkdir /vz1/private ; chmod 0755 /vz1/private
mkdir /vz1/root ; chmod 0755 /vz1/root</pre>