Editing Jail Server Install (section)

= FreeBSD 6.1 =

Last updated 2006-05-09

<pre>All time extimates below assume disks aren’t scrubbing

1. make sure bios is setup for bios console redirect
Supermicro:
Console redirection:
Com port addr: on-board COM A
Baud: 38400
Console type: vt100
Flow control: none
Console connection: direct
Continue cr after post: off

2450:
Make sure running bios A09
Console Redirection: VT100/VT220

2. assuming mirrors (or at least disks) created (if not, refer to this), boot to disk 1 of 5.4
skip kernel config (enter)
custom install

partition ->
move cursor to amrd0, hit space (takes you to partition map screen)
a for entire disk
q to quit and save
standard mbr (no boot manager)
space to unselect aacd0
cursor over aacd1
space
a for entire disk
q to quit and save
none (leave untouched)

cursor over aacd0
space
(takes you into part. Screen again) q to exit
none
Make sure both are checked and tab to ok

Label ->
Make sure aacd0 is highlighted
a to start with defaults

john likes:
/1g
/var 256m

Glenn likes:
/128
/var 256
/usr 3g
/tmp 256

We do:
delete everything
/ 128M
swap 2G
/var 256M
/tmp 256M
/usr 3G
/mnt/data1 remaining space

Make sure to toggle S for soft updates on all (should look like UFS2+S Y under the Newfs column)

move cursor to aacd1
swap 2G
/mnt/data2 remaining space

q to save and exit

distributions ->
developer
yes to install ports
exit

media ->
cd

commit ->
yes
(2450: 14mins, supermicro: 12mins)

yes to "visit general config" ->
Set root pwd

Add user ‘user’ member group is wheel, set password

Set tz

Networking->interfaces->Fxp0 ->
No IPV6
dhcp=no
Set hostname & domain
Enable sshd

exit...
exit install ->
yes

take the cd out and let the machine reboot
when it comes back up, enter junk when it asks for key seed


3. double check the date/time

4. edit /etc/make.conf (only add the console speed line if this is a supermicro capable of outputting BIOS redirect at that speed – 2450’s can only do 9600)
echo "WITHOUT_X11=yes \
KERNCONF=jail19 \
BOOT_COMCONSOLE_SPEED=38400" >> /etc/make.conf

5. add console="comconsole" to /boot/loader.conf
echo "console=""comconsole""" >> /boot/loader.conf

6. turn off all ttyv's except 0 and 1 in /etc/ttys
also turn on ttyd0, change type to vt100:
vi /etc/ttys
ttyd0   "/usr/libexec/getty std.9600"  vt100   on secure

kill -1 1

on console server:
vi /etc/remote
(rename port to jail18 depending on where and which digi plugged into)
test serial console

7. install cvsup
cd /usr/ports/net/cvsup-without-gui 
make install clean; rehash
(stay close for gettext options, 2450: 27mins, supermicro: 17mins)

8. get latest sources for this release:
cd /usr/src 
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_1\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

cvsup sup 
(2450, ~12mins, supermicro, 27mins)

9. populate hosts
echo "10.1.4.3 backup2" >> /etc/hosts

10. put key in authorized_keys on backup2
cd
ssh-keygen -t dsa -b 1024 
(default location, leave password blank)
scp /root/.ssh/id_dsa.pub user@backup2:/tmp/jailkey

cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'

confirm that you can ssh to backup2 without getting a login prompt

ssh backup2

11. configure new kernel. Get config from similar machine or there may be a master copy somewhere under /mnt/data4/build/freebsd (name the kernel config the same as the jail, ex jail18):
cd /usr/src/sys/i386/conf 
scp backup2:/mnt/data4/build/freebsd/kern_config-6.1 ./jail19

edit the kernel config and change ident to be the name of the jail:
vi jail14
ident           jail14

edit /sys/conf/newvers.sh to add –jc1 to the end of the BRANCH string (RELEASE-jc1)
vi /sys/conf/newvers.sh

12. bring over patches from backup2
The various patches are in /mnt/data4/build/freebsd/patches on backup2. There are dirs for each version. Not all dirs are populated, but patches for later versions work on older ones unless there is a new patch in the older dir.
cd /usr/src
scp backup2:"/mnt/data4/build/freebsd/patches/6.1/*" .
jail_proc_patch      100% |*************************************************|  2593    00:00
restore-patch        100% |*************************************************|  3295    00:00
54ps-jail-patch      100% |*************************************************|  2602    00:00
jail_rtprio_patch    100% |*************************************************|   301    00:00
udp-patch            100% |*************************************************|   594    00:00

Apply patches:
patch < 54ps-jail-patch
patch < jail_proc_patch
patch < restore-patch
patch < jail_rtprio_patch
patch < udp-patch

13. build, install kernel and world
cd /usr/src
make buildworld buildkernel installkernel 
(2450: 1:56min, supermicro::59mins)
make installworld 
(2450: 3min, supermicro: 1min)
mergemaster -i
delete /var/tmp/temproot
delete bsnmpd
delete temporary ./etc/hosts
delete temporary ./etc/motd
delete /var/tmp/temproot

14. reboot. Confirm new kernel is loaded

uname –a

15. update ports:
cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_1\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup

cvsup sup 
(2450: 18mins, supermicro: 19mins)

18. (only applies if adaptec card installed)
install linux_base:
cd /usr/ports/emulators/linux_base
make install clean 
(2450: 7min, supermicro: 3mins)

reibuild rpmdb cause we had probs installing aacapps
cd /compat/linux/bin
./rpm --initdb
./rpm --rebuilddb

install aacapps-4.1-0.i386.rpm
scp backup2:/mnt/data4/build/freebsd/aacapps-4.1-0.i386.rpm /tmp/.
/compat/linux/bin/rpm -ivh --excludepath=/dev /tmp/aacapps-4.1-0.i386.rpm

test out;
/compat/linux/usr/sbin/aaccli


20. edit root's path and login script:
vi /root/.cshrc
add to path: 
/usr/local/jail/bin 
(if adaptec card installed, also add /compat/linux/usr/sbin)

Change alias entries (add G):
alias la        ls -aG
alias lf        ls -FAG
alias ll        ls -lAG
alias ls        ls -AG
alias mbm       mb mount
alias mbu       mb umount

and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "

at the bottom of the file add:
set sshtty=`who am i|awk '{print $2}'`
/usr/sbin/rtprio 3 -`psj | grep $sshtty | awk '{print $2}'`

set shortty=`who am i | awk '{print $2}' | sed -E 's/.*(..)$/\1/'`
foreach x (`psj | grep sh | grep $shortty | awk '{print $2}'`)
/usr/sbin/rtprio 2 -$x
end
 
21. install rsync from ports
cd /usr/ports/net/rsync
make install clean

choose default options

21. install perl from ports
PROB NOT NECESSARY – INSTALLED WITH LINUX I THINK
cd /usr/ports/lang/perl5.8/
make install clean; rehash
(supermicro: 5min)

22. create & populate binaries/scripts dirs
mkdir -p /usr/local/jail/bin
mkdir -p /usr/local/jail/rc.d
mkdir /mnt/data1
mkdir /mnt/data2
scp backup2:"/mnt/data4/bin/freebsd6.x/*" /usr/local/jail/bin
cd /usr/local/jail/rc.d/
touch quad1
touch quad2
touch quad3
touch quad4
touch safe1
touch safe2
touch safe3
touch safe4
chmod +x *
cd /usr/local/jail/bin
ln -s /usr/local/jail/rc.d/quad1 quad1
ln -s /usr/local/jail/rc.d/quad2 quad2
ln -s /usr/local/jail/rc.d/quad3 quad3
ln -s /usr/local/jail/rc.d/quad4 quad4
ln -s /usr/local/jail/rc.d/safe1 safe1
ln -s /usr/local/jail/rc.d/safe2 safe2
ln -s /usr/local/jail/rc.d/safe3 safe3
ln -s /usr/local/jail/rc.d/safe4 safe4

rehash

23. configure inetd to respond to mrtg load queries
echo "load    stream  tcp     nowait  user    /usr/local/jail/bin/load.pl  load.pl" >> /etc/inetd.conf

echo "load            12384/tcp" >> /etc/services

26. install bb client
adduser
cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd.tar .
tar xvf bb-freebsd.tar

edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
echo "10.1.4.105 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.4.103 jail3.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

vi /home/bb/bbc1.9e-btf/ext/openfiles 
MACHINE="jail19,johncompanies,com"      # HAS TO BE IN A,B,C FORM

cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh (y to questions)
./bbchkhosts.sh (ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src
make; make install
cd ..
./runbb.sh start
more BBOUT 
(look for errors)
exit

echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh

modify bb-msgtab to look for raid failures

24. configure load mrtg, on mail
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg 
(add new entry to file following existing format)

25. configure bb on mail:
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
10.1.4.119 jail19.johncompanies.com # ssh

su bb
cd
bbsrc/bb/runbb.sh restart ; exit

27. remove reserve space, enable softupdates (probably already set, so not necessary)
NOT APPLICABLE IF USING GVINUM
cd
umount /mnt/data1
umount /mnt/data2
tunefs -m 0 /mnt/data1
tunefs -m 0 /mnt/data2
mount -a

28. configure ntp
echo "server 10.1.4.105" > /etc/ntp.conf

/usr/sbin/ntpd -p /var/run/ntpd.pid
sleep 2; ntpq -p
(confirm it’s able to reach our time server)

29. mrtg switch graphs

31. fwd and reverse lookups on ns1c
vr johncompanies.com
vi internal.johncompanies.com
rndc reload johncompanies.com IN private
(edit the PTR too)

33. populate /etc/rc.conf with IPs and NFS settings
vi /etc/rc.conf

kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
sshd_enable="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.4.119"
devfs_system_ruleset="devfsrules_show_all"

ifconfig_xl0="inet 10.1.4.118 netmask 255.255.255.0"
ifconfig_fxp0="inet 69.55.228.101 netmask 255.255.255.0"
defaultrouter="69.55.228.1"
ifconfig_fxp0_alias0="inet 69.55.2xx.xx netmask 255.255.255.0"

static_routes="t1 office"
route_t1="-net 10.1.5 10.1.4.2"
route_office="-net 10.1.6 10.1.4.2"
gvinum_enable="YES"
fsck_y_enable="YES"
background_fsck="NO"

34. make sure sysctls are set and preserved after reboot
echo "kern.consmute=0\
kern.ipc.shm_use_phys=1\
kern.ipc.shmall=65535\
kern.ipc.shmmax=134217728\
net.inet.tcp.syncookies=0\
kern.maxfiles=32768\
kern.fallback_elf_brand=3\
kern.maxprocperuid=4000\
security.jail.sysvipc_allowed=1\
security.jail.max_procs_per_jail: 1026\
security.jail.allow_raw_sockets=1\
security.jail.socket_unixiproute_only=1\
security.jail.chflags_allowed=0" >> /etc/sysctl.conf

35. mount procfs
echo "proc                    /proc           procfs  rw              0       0" >> /etc/fstab

36. enable noatime option
NOT APPLICABLE IF RUNNING GVINUM
data1 and data2 should look something like:
/dev/amrd0s1g           /mnt/data1      ufs     rw,noatime      2       2

36. populate devfs ruleset
scp backup2:/mnt/data4/build/freebsd/devfs.rules /etc

35. reboot
Check rules:
devfs rule showsets
devfs rule -s 3 show

36. create gvinum volumes

Make a g partition:

bsdlabel -e /dev/aacd0s1

given:
# /dev/aacd0s1:
8 partitions:
#        size   offset    fstype   [fsize bsize bps/cpg]
  a:   262144        0    4.2BSD     2048 16384 16392
  b:  4194304   262144      swap
  c: 143363997        0    unused        0     0         # "raw" part, don't edit
  d:   524288  4456448    4.2BSD     2048 16384 32776
  e:   524288  4980736    4.2BSD     2048 16384 32776
  f:  6291456  5505024    4.2BSD     2048 16384 28552

new offset = 6291456 + 5505024 = 11796480
new size is size for 'c' partition minus the new start from above
143363997 - 11796480 = 131567517
So:
g: 131567517 11796480 unused 0 0

For a 73G drive (after OS), we can fit 31 2G volumes so:

echo 'drive data1 device /dev/aacd0s1g' > /tmp/cgv
sh
for f in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31;\
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 2g drive data1' >> /tmp/cgv; done; exit

gvinum create /tmp/cgv

For a 146G drive (-4G for swap), we can fit 66 2G volumes so:

echo 'drive data2 device /dev/aacd1s1g' > /tmp/cgv
sh
for f in 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97;\
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 2g drive data2' >> /tmp/cgv; done; exit

gvinum create /tmp/cgv

For 3rd 73G drive (after 2G swap), we can fit 33 2G volumes so:
Label should be:
# /dev/aacd2s1:
8 partitions:
#        size   offset    fstype   [fsize bsize bps/cpg]
  c: 143299737        0    unused        0     0         # "raw" part, don't edit
  g: 143299721        16    unused        0     0

echo 'drive data3 device /dev/aacd2s1g' > /tmp/cgv
sh
for f in 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131;\
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 2g drive data3' >> /tmp/cgv; done

gvinum create /tmp/cgv

For a 2nd 73G drive (after 2G swap), we can fit 33 2G volumes so:

echo 'drive data2 device /dev/aacd1s1g' > /tmp/cgv
sh
for f in 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64;\
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 2g drive data2' >> /tmp/cgv; done; exit

gvinum create /tmp/cgv


36. create the jail template

mkdir /mnt/jail
newfs /dev/gvinum/v1
mount /dev/gvinum/v1 /mnt/jail

cd /usr/src
make clean
rm -fr /usr/obj/
make world DESTDIR=/mnt/jail
(2450: 2:28mins, supermicro: 55mins)
cd etc
make distribution DESTDIR=/mnt/jail
mount_devfs devfs /mnt/jail/dev
devfs -m /mnt/jail/dev rule -s 3 applyset 
cd /mnt/jail
ln -sf dev/null kernel

jail /mnt/jail testhostname 192.168.11.100 /bin/sh
csh
touch /etc/fstab
echo 'network_interfaces=""\
hostname="newsystem"\
kern_securelevel_enable="NO"\
sendmail_enable="YES"\
sshd_enable="YES"' > /etc/rc.conf

echo "nameserver 69.55.225.225\
nameserver 69.55.230.3" >> /etc/resolv.conf

vi /etc/crontab
remove the adjkerntz lines
comment out periodic’s and put this line above them:
# DO NOT UNCOMMENT THESE

rm -rf /etc/periodic/daily/400.status-disks

check /tmp for crap

vi /etc/periodic/security/100.chksetuid
replace: MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`
with: MP='/' (use single quotes)

mkdir -p /usr/compat/linux/dev

adduser (Add account for user, make sure in wheel group)

user/root passwd: 8ico2987

Set root password

vi /usr/home/user/.profile (and add to the file):
TERM=vt100;     export TERM

tzsetup

newaliases 

#cd /dev
#rm console
#ln -s null console

exit
exit

cd /usr/ports
make -DNOCLEANDEPENDS clean 
(2450: 15mins , supermicro: 29mins)
rm -fr /usr/ports/distfiles/*
cp -r /usr/ports /mnt/jail/usr (2450: 2:00 mins , supermicro: 15mins)

rm /mnt/jail/root/.history

cd 
mkdir /usr/local/jail/template/
dump -0a -f /usr/local/jail/template/61template /dev/gvinum/v1

umount /mnt/jail/dev
umount /dev/gvinum/v1

rm -fr /mnt/jail

37. setup backups
echo '#\!/bin/sh\
backupdir=/mnt/data3jail3_rsync\
\
## ENTRY /etc\
## ENTRY /usr/local/etc\
## ENTRY /usr/local/jail\
## ENTRY /root/logs' > /usr/local/jail/bin/backup.config

on backup2:
setup backup dirs:
mkdir -p /mnt/data3/jail3/0

add the system to 
vi /mnt/data4/bin/snapshot_rotate

on mail:
vi /usr/local/www/mgmt/cgi/backupgraph.pl
(add hostname)

38. mkdir /root/logs

39. edit sshd_config for security
vi /etc/ssh/sshd_config
ListenAddress 69.55.228.101
ListenAddress 10.1.4.118

kill -1 `cat /var/run/sshd.pid`

40. add crontab entries
crontab -e
5 0 * * * /usr/local/jail/bin/backup
1 0 1 * * /usr/local/jail/bin/ipfwreset
0 18 * * * /usr/local/jail/bin/ipfwbackup
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /usr/local/jail/bin/trafstats
55 10,23 * * * /usr/local/jail/bin/trafficwatch.pl


41. Reboot notify script
ln -s /usr/local/jail/bin/notify.sh /usr/local/etc/rc.d/notify.sh 

42. copy jailmake from prev system
scp user@10.1.4.119:/usr/local/jail/bin/jailmake /usr/local/jail/bin
rehash
NOTE: remove df altering code from jailmake since we put the correct df in the template, and make sure path to template file is right

43. add to management db (on mail and devweb) jc.ref_machines and jc.ref_templates

uname -r
5.4-RELEASE-p2-jc2

insert into ref_machines values (null,'jail19','jail19.johncompanies.com',0,'l');
select machine_id from ref_machines where host='jail19';
+------------+
| machine_id |
+------------+
|         35 |
+------------+
insert into ref_templates values ('',' 6.2-RELEASE-jc1',5,'FreeBSD 6.2',0);

44. add to server/cabinet map. On mail:
vi /usr/local/www/mgmt/html/cabinetmap.html

45. add an outside blocking rule to the firewall, so this machine can only be reached from inside the firewall. Follow example already in firewall jail17 is:
 
00119 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.228.200
00119 deny ip from any to 69.55.228.200

jail19 would be 00119...
ipfw add 00119 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.228.200
ipfw add 00119 deny ip from any to 69.55.228.200

46. select customers for probe map

47. install raid monitor

cd /usr/ports/sysutils/asr-utils
make install clean

48. make gv start on boot


scp backup2:/mnt/data4/build/freebsd/gvinum /etc/rc.d/gvinum




gconcat label -v somelabel /dev/gvinum/a /dev/gvinum/b
bsdlabel -r -w /dev/concat/somelabel
newfs /dev/concat/somelabela
mount /dev/concat/somelabel /mount/point

umount /dev/concat/somelabel
gconcat stop somelabel
gconcat label -v /dev/gvinum/a /dev/gvinum/b /dev/gvinum/c /dev/gvinum/d
growfs /dev/concat/somelabel
mount /dev/concat/somelabel /mount/point

volume f
        plex org concat
        sd length 30449m drive data1

where f is 

D data1                 State: up       /dev/aacd0s1g   A: 30449/64241 MB (47%)

gvinum rm -r f


so i setup 2 machines with 6.1. A had 2 logical drives and gv's created across both. B had 1 logical drive and gv's across the 1 drive.  
the labeling for the gv's was v1-vN (till i ran out of space) 
when i moved aac1 from A to B, the gv's on A's aac1, took precedence over the similarly labeled gv's on B's aac0. in other words. B's aac0 used to have v1-v30. A's aac1 had vn6-vn30. when A's aac1 was moved to B, v1-v5 were linked to B's aac0, and v6-v30 linked to A's aac1 
i relabeled B's aac0 gv's to something different (not v1-vN) and tried again and they all showed up.  
i tried the experiment again this time moving A's aac0 to B. at that point nothing worked. i moved A's aac0 back to A and renamed the device (data1) to something else, then tried the test again. still no beans.  
it's hazy what i did after that cause the machine's weren't cooperating and i was trying to rename the device but basically i think i need to repeat the test and see if i can re-define the device, and probably also give it a unique name and i bet it would have worked.  
also interesting- A's aac1 contained data about A's aac0 which showed up when i put aac1 into B 
skeeter:	as for the gv stuff, it definitely sounds like using some sort of serial numbering scheme would be the way to go if you want to be able to move disks around.... 
that overlap is still an issue  (aac1's device was 'data2')  when i had A's aac0 in B nothing worked both disks gv devices were called data1 
skeeter:	I suppose you could serialize those names as well...</pre>