Editing Jail Server Install (section)

= FreeBSD 4.11 =

<pre>
4.11
Last updated 2006-1-26

All time extimates below assume disks aren’t scrubbing

1. make sure bios is setup for bios console redirect
Supermicro:
Console redirection:
Com port addr: on-board COM A
Baud: 38400
Console type: vt100
Flow control: none
Console connection: direct
Continue cr after post: off

2450:
Make sure running bios A09
Console Redirection: VT100/VT220
	
2. assuming mirrors (or at least disks) created (if not, refer to this), boot to disk 1 of 4.11 
skip kernel config (enter)
custom install
partition
move cursor to aacd0, hit space (takes you to partition map screen)
a for entire disk
q to quit and save
standard mbr (no boot manager)
space to unselect aacd0
cursor over aacd1
space
a for entire disk
q to quit and save
none (leave untouched)

cursor over aacd0
space
(takes you into part. Screen again) q to exit
none
Make sure both are checked and tab to ok

label
Make sure aacd0 is highlighted
a to start with defaults

john likes:
/1g
/var 256m

Glenn likes:
/128
/var 256
/usr 3g
/tmp 256

We do:
delete everything
/ 128M
swap 4G
/var 256M
/tmp 256M
/usr 3G
/mnt/data1 remaining space

Make sure to toggle S for soft updates on all (should look like UFS+S Y under the Newfs column)
Set the noatime option

move cursor to aacd1
swap 4G
/mnt/data2 remaining space
Set the noatime option

q to save and exit

distributions
developer
yes to install ports
exit
media
cd
commit
yes
(2450: 16mins, supermicro: 11mins)
yes to "visit general config"
Set root pwd
Add user ‘user’ member group is wheel, set password
Set tz
Networking->interfaces->Fxp0
No IPV6
dhcp=yes
Set hostname & domain

Startup services:
Disble usbd

exit...
exit install
yes
take the cd out and let the machine reboot

3. put some temp settings in /etc/rc.conf:
usbd_enable="NO"
sendmail_enable="NO"

4. reboot

5. double check the date/time

6. edit /etc/make.conf (only add the console speed line if this is a supermicro capable of outputting BIOS redirect at that speed – 2450’s can only do 9600)
cat >> /etc/make.conf
WITHOUT_X11=yes
KERNCONF=jail18
BOOT_COMCONSOLE_SPEED=38400

7. install cvsup
cd /usr/ports/net/cvsup-without-gui 
make install clean (stay close for gettext options, 2450: 21mins, supermicro: 14mins)
rehash

8. get latest sources for this release:
cd /usr/src 
cat > sup
*default host=cvsup4.freebsd.org
*default base=/usr
*default prefix=/usr
*default release=cvs tag=RELENG_4_11
*default delete use-rel-suffix
*default compress
src-all

cvsup sup (2450, 4.10: 13mins, supermicro, 4.11: 11mins)

9. populate hosts
cat >> /etc/hosts
10.1.4.3	backup2

10. put key in authorized_keys on backup2
ssh-keygen -t dsa -b 1024 (default location, leave password blank)
scp /root/.ssh/id_dsa.pub user@backup2:/tmp/jail18pub
on backup2:
cat /tmp/jail18pub >> /root/.ssh/authorized_keys

confirm that you can ssh to backup2 without getting a login prompt

11. configure new kernel. Get config from similar machine or there may be a master copy somewhere under /mnt/data4/build (name the kernel config the same as the jail, ex jail4):
cd /usr/src/sys/i386/conf 
scp backup2:/mnt/data4/build/freebsd/kern_config-4.11 ./jail18

edit the kernel config and change ident to be the name of the jail:
ident           jail4

IMPORTANT CUSTOMIZATION:
for machines with >4G RAM, add to the config:
options         PAE

for supermicro mobo’s with broadcom nics, add to the config:
device          bge             # Broadcom BCM570x (``Tigon III'')

for machines where lots of postgres might be running, change SHMMAXPGS:
options         SHMMAXPGS=40960

edit /sys/conf/newvers.sh to add –jc2 to the end of the BRANCH string (RELEASE-p9-jc2)

12. bring over patches from backup2
The various patches are in /mnt/data4/build/freebsd/patches on backup2. There are dirs for each version. Not all dirs are populated, but patches for later versions work on older ones unless there is a new patch in the older dir.
cd /usr/src
scp backup2:"/mnt/data4/build/freebsd/patches/4.11/*" .
jail_proc_patch      100% |*************************************************|  2593    00:00
restore-patch        100% |*************************************************|  3295    00:00
411ps-jail-patch     100% |*************************************************|  2602    00:00
jail_rtprio_patch    100% |*************************************************|   301    00:00
udp-patch            100% |*************************************************|   594    00:00

Apply patches:
patch < 411ps-jail-patch
patch < jail_proc_patch
patch < restore-patch
patch < jail_rtprio_patch
patch < udp-patch

13. build, install kernel and world
make buildworld buildkernel installkernel (2450: 48min, supermicro: 20mins)
(Any compile errors can be looked up in /usr/include/sys/signal.h, other errors, do a rm -R /usr/obj/*)
make installworld (2450: 2min, supermicro: 1mins)
mergemaster -i
(answer no to most of it)

14. reboot. Confirm new kernel is loaded (uname -a)

15. update ports:
cd /usr/ports
cat > sup
*default host=cvsup4.FreeBSD.org
*default base=/usr
*default prefix=/usr
*default release=cvs tag=RELENG_4
*default delete use-rel-suffix
*default compress
ports-all tag=.

cvsup sup (2450: 26mins, supermicro: 26mins)

16. add console="comconsole" to /boot/loader.conf
cat >> /boot/loader.conf
console="comconsole"

17. edit /etc/ttys and turn off all ttyv's except 0 and 1
also turn on ttyd0, change type to vt100:
ttyd0   "/usr/libexec/getty std.9600"  vt100   on secure

kill -1 1

on console:
vi /etc/remote
(rename port to jail4 depending on where and which digi plugged into)
test serial console

18. install linux_base:
cd /usr/ports/emulators/linux_base
make install clean (2450: 7min, supermicro: 2mins)

reibuild rpmdb cause we had probs installing aacapps
cd /compat/linux/bin
./rpm --initdb
./rpm --rebuilddb

19. install aacapps-4.1-0.i386.rpm
scp backup2:/mnt/data4/build/freebsd/aacapps-4.1-0.i386.rpm /tmp/.
/compat/linux/bin/rpm -ivh /tmp/aacapps-4.1-0.i386.rpm
cd /dev
sh MAKEDEV aac0

test out /compat/linux/usr/sbin/aaccli

20. edit root's path and login script:
vi /root/.cshrc
add to path: /usr/local/jail/bin /compat/linux/usr/sbin

Change alias entries (add G):
alias la        ls -aG
alias lf        ls -FAG
alias ll        ls -lAG
alias ls        ls -AG
alias mbm       mb mount
alias mbu       mb umount

and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "

at the bottom of the file add:
set sshtty=`who am i|awk '{print $2}'`
/usr/sbin/rtprio 3 -`ps auxwJ | grep $sshtty | awk '{print $2}'`

set shortty=`who am i | awk '{print $2}' | sed -E 's/.*(..)$/\1/'`
foreach x (`ps cauxJ | grep sh | grep $shortty | awk '{print $2}'`)
/usr/sbin/rtprio 2 -$x
end
 
21. install rsync from ports
cd /usr/ports/net/rsync
make install clean

22. create & populate binaries/scripts dirs
mkdir -p /usr/local/jail/bin
mkdir -p /usr/local/jail/rc.d
scp backup2:"/mnt/data4/bin/freebsd/*" /usr/local/jail/bin
cd /usr/local/jail/rc.d/
touch quad1
touch quad2
touch quad3
touch quad4
touch safe1
touch safe2
touch safe3
touch safe4
chmod +x *
cd /usr/local/jail/bin
ln -s /usr/local/jail/rc.d/quad1 quad1
ln -s /usr/local/jail/rc.d/quad2 quad2
ln -s /usr/local/jail/rc.d/quad3 quad3
ln -s /usr/local/jail/rc.d/quad4 quad4
ln -s /usr/local/jail/rc.d/safe1 safe1
ln -s /usr/local/jail/rc.d/safe2 safe2
ln -s /usr/local/jail/rc.d/safe3 safe3
ln -s /usr/local/jail/rc.d/safe4 safe4
rehash

23. configure inetd to respond to mrtg load queries
cat >> /etc/inetd.conf
load    stream  tcp     nowait  user    /usr/local/jail/bin/load.pl  load.pl

cat >> /etc/services
load            12384/tcp

kill -HUP `cat /var/run/inetd.pid`

24. configure load mrtg, on mail
vi /usr/local/etc/mrtg/mrtg1.cfg
(add new entry to file following existing format)

25. configure bb on mail:
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
10.1.4.104 jail4.johncompanies.com # ssh

su bb
cd /usr/home/bb/bbsrc/bb1.9e-btf
./runbb.sh stop
./runbb.sh start
exit

26. install bb client
adduser -group 1984 -shell /bin/csh -uid 1984 bb
cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd.tar .
tar xvf bb-freebsd.tar

cat > /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
10.1.4.105 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh
10.1.4.118 jail18.johncompanies.com # ssh

vi /home/bb/bbc1.9e-btf/ext/openfiles   and change:
MACHINE="jail18,johncompanies,com"      # HAS TO BE IN A,B,C FORM

cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
./bbchkhosts.sh (ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd bbc1.9e-btf/
./runbb.sh start
more BBOUT (look for errors)
exit

cat > /usr/local/etc/rc.d/bb.sh
su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"

chmod +x /usr/local/etc/rc.d/bb.sh

27. remove reserve space, enable softupdates (probably already set, so not necessary)
cd
umount /mnt/data1
umount /mnt/data2
tunefs -m 0 /mnt/data1
tunefs -m 0 /mnt/data2

mount -a

28. configure ntp
cat > /etc/ntp.conf
server 10.1.4.105

/usr/sbin/ntpd -p /var/run/ntpd.pid
ntpq -p
(confirm it’s able to reach our time server)

29. mrtg switch graphs

31. fwd and reverse lookups on ns1c
vi johncompanies.com
rr johncompanies.com
vi internal.johncompanies.com
rndc reload johncompanies.com IN private
(edit the PTR too)

32. create all /dev/vn and /dev/pty files in /dev
cat > /tmp/runme.sh
#!/bin/sh
cd /dev
for i in 1 2 3 4 5 6 7 ; do sh MAKEDEV pty$i ; done
for i in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 ; do sh MAKEDEV vn$i ; done

NOTE: there are only 4 lines in the above output - however the lines may be wrapped on your terminal - make sure that there are only four lines in the script you make.

chmod +x /tmp/runme.sh
/tmp/runme.sh

rm /tmp/runme.sh

ls /dev/vn*|wc -l (make sure the output is a high number – larger than 128)
ls /dev/pty*|wc -l (make sure the output is exactly 256)

33. populate /etc/rc.conf with IPs and NFS settings
vi /etc/rc.conf

kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="NO"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
sshd_enable="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.4.104"

ifconfig_xl0="inet 10.1.4.104 netmask 255.255.255.0"
ifconfig_fxp0="inet 69.55.228.101 netmask 255.255.255.0"
defaultrouter="69.55.228.1"
ifconfig_fxp0_alias0="inet 69.55.2xx.xx netmask 255.255.255.0"

static_routes="t1 office"
route_t1="-net 10.1.5 10.1.4.2"
route_office="-net 10.1.6 10.1.4.2"

34. make sure sysctls are set and preserved after reboot
cat >> /etc/sysctl.conf
kern.consmute=0
jail.sysvipc_allowed=1
kern.ipc.shm_use_phys=1
kern.ipc.shmall=65535
kern.ipc.shmmax=134217728
net.inet.tcp.syncookies=0
kern.maxfiles=32768
kern.fallback_elf_brand=3
kern.maxprocperuid=4000
jail.max_procs_per_jail: 1026

35. reboot

36. create the jail template
vnconfig -T -S 1g -Z -s labels -c /dev/vn1 /mnt/data1/jail
disklabel -r -w vn1 auto
newfs /dev/vn1c
cd /usr/src
mkdir -p /mnt/data1/jail-DIR
mount /dev/vn1c /mnt/data1/jail-DIR
make world DESTDIR=/mnt/data1/jail-DIR (2450: 45mins, supermicro: 19mins)
cd etc
make distribution DESTDIR=/mnt/data1/jail-DIR -DNO_MAKEDEV_RUN
cd /mnt/data1/jail-DIR/dev
sh MAKEDEV jail
cd /mnt/data1/jail-DIR
ln -sf dev/null kernel

jail /mnt/data1/jail-DIR testhostname 192.168.11.100 /bin/sh
csh
touch /etc/fstab
cat > /etc/rc.conf
portmap_enable="NO"
network_interfaces=""
hostname="newsystem"
kern_securelevel_enable="NO"
sendmail_enable="YES"
sshd_enable="YES"

cat >> /etc/resolv.conf
nameserver 69.55.225.225
nameserver 69.55.230.3

vi /etc/crontab
remove the adjkerntz lines
comment out periodic’s and put this line above them:
# DO NOT UNCOMMENT THESE

rm -rf /etc/periodic/daily/400.status-disks

mv /bin/df /bin/df_

cat > /bin/df
#!/bin/sh
/bin/df_ $* .

chmod +x /bin/df

cat > /sbin/mount
echo `df | tail -1 | awk '{print $1 " on " $6 " (ufs, local)"}'`

check /tmp for crap

vi /etc/periodic/security/100.chksetuid
replace: MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`
with: MP='/'

mkdir -p /usr/compat/linux/dev

adduser (Add account for user)

put user in wheel group
vi /etc/group

vi /usr/home/user/.profile (and add to the file):
TERM=vt100;     export TERM

cd /etc
vipw -d .
root:$1$krszPxhk$xkCepSnz3mIikT3vCtJCt0:0:0::0:0:Charlie &:/root:/bin/csh
user:$1$Mx9p5Npk$QdMU6c8YQqp2FW2M3irEh/:1001:1001::0:0:User &:/home/user:/bin/sh

tzsetup

newaliases 

chflags schg /dev/*mem

cd /dev
rm console
ln -s null console

exit
exit

cd /usr/ports
make -DNOCLEANDEPENDS clean (2450: 47mins , supermicro: 17mins)
rm -fr /usr/ports/distfiles/*
cp -r /usr/ports /mnt/data1/jail-DIR/usr

cd /mnt/data1/jail-DIR/dev
rm kmem
mknod kmem c 2 1 root:kmem
chmod 640 kmem
rm mem
mknod mem c 2 0 root:kmem
chmod 640 mem

rm /mnt/data1/jail-DIR/root/.history

sh
for i in 1 2 3 4 5 6 7 ; do sh MAKEDEV pty$i ; done
exit

cd /mnt/data1/jail-DIR/usr/compat/linux/dev
mknod null c 2 2
mknod random c 2 3

cd 
mkdir /usr/local/jail/template/
dump -0a -f /usr/local/jail/template/411template /dev/vn1

umount /dev/vn1c
vnconfig -u /dev/vn1

rm /mnt/data1/jail
rm -fr /mnt/data1/jail-DIR

37. setup backups
cat > /usr/local/jail/bin/backup.config
#!/bin/sh
backupdir=/mnt/data2/jail18_rsync

## ENTRY /etc
## ENTRY /usr/local/etc
## ENTRY /usr/local/jail
## ENTRY /root/logs

on backup2:
setup backup dirs:
mkdir -p /mnt/data2/jail18_rsync/0_today
mkdir -p /mnt/data2/jail18_rsync/1_yesterday
mkdir -p /mnt/data2/jail18_rsync/2_two_day

add the system to /mnt/data1/bin/rsync_houseclean 

on mail:
vi /usr/local/www/mgmt/cgi/backupgraph.pl
(add hostname)

38. mkdir /root/logs

39. edit sshd_config for security
vi /etc/ssh/sshd_config
ListenAddress 69.55.228.101
ListenAddress 10.1.4.104

kill -1 `cat /var/run/sshd.pid`

40. add crontab entries
crontab -e
5 0 * * * /usr/local/jail/bin/backup
1 0 1 * * /usr/local/jail/bin/ipfwreset
0 18 * * * /usr/local/jail/bin/ipfwbackup
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /usr/local/jail/bin/trafstats
55 10,23 * * * /usr/local/jail/bin/trafficwatch.pl


41. Reboot notify script
cat > /usr/local/etc/rc.d/notify.sh
echo "`/bin/hostname` rebooted" | /usr/bin/mail reboot@johncompanies.com

chmod +x /usr/local/etc/rc.d/notify.sh

42. copy jailmake from prev system
scp user@10.1.4.118:/usr/local/jail/bin/jailmake /usr/local/jail/bin
rehash
NOTE: remove df altering code from jailmake since we put the correct df in the template, and make sure path to template file is right

43. add to templates via mgmt system

44. add to server/cabinet map. On mail:
vi /usr/local/www/mgmt/html/cabinetmap.html

45. add an outside blocking rule to the firewall, so this machine can only be reached from inside the firewall. Follow example already in firewall jail17 is:
 
00117 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.228.2
00117 deny ip from any to 69.55.228.2

jail4 would be 00104...
ipfw add 00118 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.228.140
ipfw add 00118 deny ip from any to 69.55.228.140

46. select customers for probe map
</pre>