Jail Server Install
8.x
Assumptions
Setup instructions below assume this is DELL 2950 with an LSI-based SAS RAID card.
Server is at castle, connected to pub, private, serial and DRAC
Assuming OS loading done via IPKVM with ISO mounted via USB
Assumes at 4 drives, 2 mirrors
Configure server BIOS
setup console redirect, speed 115200
set LCD string to name of server "jail8"
set date to GMT
go into RAID bios and setup mirrors
configure DRAC: TODO
Install OS (sysinstall)
boot to bootonly disk for AMD version of FreeBSD, i.e. FreeBSD-8.3-RELEASE-amd64-bootonly.iso
when the install menu appears, choose custom install
move cursor to mfid0, hit space (takes you to partition map screen). If there is only 1 mirror, there will be no option to select a specifit drive: mfid0 will be selected.
type 'a' to use entire disk
type 'q' to quit and save
choose 'freebsd standard mbr'
space to unselect mfid0
cursor down to mfid1
hit space
type 'a' to use entire disk
type 'q' to quit and save
choose 'none' for boot mgr (leave untouched)
cursor over mfid0
space
(takes you into part. Screen again) 'q' to exit
none for boot mgr
Make sure both drives (mfid0 and mfid1) are checked and tab to ok
Make sure mfid0 is highlighted at the top of the screen, setup the following partitions
/ 512M
swap 6G
/var 256M
/tmp 256M
/usr 5G
/mnt/data1 remaining space
All partitions except / should be setup for soft updates. If not, type 's' to enable for soft updates on all except for / (should look like UFS2+S Y under the Newfs column)
move cursor to mfid1 at the top of the screen
swap 8G (or 4G if there’s a 3rd drive)
/mnt/data2 remaining space
'q' to save and exit
distributions
Choose the following distribudions
- developer (ok to install ports)
- custom -> lib32
exit
media
if you are installing via a cd, no need to enter this menu or change anything. Otherwise, choose ftp to install via ftp. You will be prompted to setup networking. You will need to choose a nic (typically bce0 or bce1). Say no to DHCP and IPv6. Hopefully the public nic cable was installed in bce0 so start with that nic and provide the hostname, (public) IP, netmask, gateway and DNS. When configured, it should start pinging. If it doesn't, have the NOC swap cables. Select any FTP server, usually Main or ftp4.
commit
this usually takes 12mins
during the process you may need to select a new ftp mirror, this is not a problem.
at the conclution of the install you will be prompted to enter the root password (2x) and returned to the configuration menu.
add user
Add user 'user'. Defaults for everything is fine, just remember to enter 'wheel' in the member group field. Do set the password.
Setup timezone
PT
Networking
page down to the bottom and enable '[X]' sshd
If you installed via cd, you will need to visit:
interfaces->bce0->
No IPV6
dhcp=no
Set hostname, IP, DNS, gateway
(i.e. setup the nic as indicated above)
Exit the install and if you installed via CD, take it out and let the machine reboot
Configure OS, kernel, userland, jail
double check the date/time
populate /etc/resolv.conf
echo "nameserver 69.55.225.225\ nameserver 69.55.230.3 nameserver 69.55.229.3" > /etc/resolv.conf
edit /etc/make.conf
echo "WITHOUT_X11=yes \ KERNCONF=jail3 \ BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf
setup bootloader for console, etc
add settings to /boot/loader.conf and /boot.config:
echo "-Dh" >> /boot.config
echo 'console="comconsole,vidconsole" \ boot_multicons="YES" \ boot_serial="YES" \ mfi_linux_load="YES" \ comconsole_speed="115200"' >> /boot/loader.conf
enable login via serial console
turn off all ttyv's except 0 and 1 in /etc/ttys and turn on ttyd0, change type to vt100:
vi /etc/ttys
The changed lines should look like:
ttyv2 "/usr/libexec/getty Pc" cons25 off secure ttyv3 "/usr/libexec/getty Pc" cons25 off secure ttyv4 "/usr/libexec/getty Pc" cons25 off secure ttyv5 "/usr/libexec/getty Pc" cons25 off secure ttyv6 "/usr/libexec/getty Pc" cons25 off secure ttyv7 "/usr/libexec/getty Pc" cons25 off secure # Serial terminals # The 'dialup' keyword identifies dialin lines to login, fingerd etc. ttyu0 "/usr/libexec/getty std.9600" vt100 on secure
Restart init
kill -1 1
At this point you should have a login on console.
To configure serial console access, login to the console server as root and run:
# vi /etc/remote
following examples there, rename port to server's hostname, depending on where and which digi box this server is plugged into. Make sure to get speed right too: 115200
populate hosts
If server is at castle:
echo "10.1.4.3 backup2" >> /etc/hosts echo "10.1.4.8 backup1" >> /etc/hosts
If server is at i2b:
echo "69.55.230.10 backup2" >> /etc/hosts echo "10.1.2.3 backup3" >> /etc/hosts echo "69.55.230.11 backup1" >> /etc/hosts
create ssh key, upload to backup servers
cd ssh-keygen -t dsa -b 1024
(default location, leave password blank)
If server is at castle:
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys' cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'
If server is at i2b:
cat /root/.ssh/id_dsa.pub | ssh backup3 'cat - >> /root/.ssh/authorized_keys' cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys' cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'
confirm that you can ssh to backup2 and backup1 without getting a login prompt
ssh backup2 hostname ssh backup1 hostname ssh backup3 hostname
create & populate binaries/scripts dirs
mkdir -p /usr/local/jail/bin mkdir -p /usr/local/jail/rc.d mkdir -p /usr/local/jail/template/ mkdir /mnt/data1 mkdir /mnt/data2 scp backup2:"/mnt/data4/bin/freebsd8.x/*" /usr/local/jail/bin cd /usr/local/jail/rc.d/ touch quad1 touch deprecated chmod +x * cd /usr/local/jail/bin ln -s /usr/local/jail/rc.d/quad1 quad1 ln -s /usr/local/jail/bin/jailmake_md jailmake ln -s /usr/local/jail/bin/js_md js ln -s /usr/local/jail/bin/canceljail_md canceljail ln -s /usr/local/jail/bin/jailmakeempty_md jailmakeempty ln -s /usr/local/jail/bin/postboot_md postboot ln -s /usr/local/jail/bin/preboot_md preboot ln -s /usr/local/jail/bin/startjail_md startjail ln -s /usr/local/jail/bin/stopjail_md stopjail rehash
edit root's path and login script
vi /root/.cshrc
Change alias entries (add G):
alias la ls -aG alias lf ls -FAG alias ll ls -lAG alias ls ls -AG alias mbm mb mount alias mbu mb umount alias cjb cd /usr/local/jail/bin alias cd1 cd /mnt/data1 alias cd2 cd /mnt/data2 alias cd3 cd /mnt/data3 alias jtop jtop lj alias j jobs
add to path be careful to leave a space after bin and make sure the wrapping isn't broken:
/usr/local/jail/bin
alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "
at the bottom of the file add:
set sshtty=`who am i|awk '{print $2}'`
/usr/sbin/rtprio 3 -`psj | grep $sshtty | awk '{print $2}'`
set shortty=`who am i | awk '{print $2}' | sed -E 's/.*(..)$/\1/'`
foreach x (`psj | grep sh | grep $shortty | awk '{print $2}'`)
/usr/sbin/rtprio 2 -$x
end
Make the new settings active in current shell:
source /root/.cshrc
install cvsup
cd /usr/ports/net/cvsup-without-gui make install clean; rehash; mail -s 'cvs installed' support@johncompanies.com < /dev/null
stand by for gettext options (use defaults). this process takes approx 22mins- hence the email/page notice above.
get latest sources for this release
cd /usr/src echo "*default host=cvsup4.freebsd.org\ *default base=/usr\ *default prefix=/usr\ *default release=cvs tag=RELENG_8_3\ *default delete use-rel-suffix\ *default compress\ src-all" > sup
If you need to run stable (cause release is broken or some other reason) make the sup file look like:
echo "*default host=cvsup4.freebsd.org\ *default base=/usr\ *default prefix=/usr\ *default release=cvs tag=RELENG_8\ *default delete use-rel-suffix\ *default compress\ src-all" > sup
cvsup sup ; mail -s 'cvs sup done' support@johncompanies.com < /dev/null
time varies, 10-20mins
configure new kernel
Pull down the kernel config we are using for this distribution. In this case we use an 8.2 kernel config on 8.3, which is valid. The local file should be the same name as host- jail3 in this example
cd /usr/src/sys/amd64/conf scp backup2:/mnt/data4/build/freebsd/kern_config-8.2-amd64 ./jail3
edit the kernel config and change ident to be the name of the jail:
vi jail3 ident jail3
Optional, edit /sys/conf/newvers.sh to add –jc2 to the end of the BRANCH string (RELEASE-jc2)
vi /sys/conf/newvers.sh
notes on kernel configuring: http://www.FreeBSD.org/doc/en_US.ISO8859-1/books/handbook/kernelconfig-config.html
install patches
The various patches are in /mnt/data4/build/freebsd/patches on backup2. There are dirs for each version. Not all dirs are populated, but patches for later versions work on older ones unless there is a new patch in the older dir.
there are no patches we use for 8.x, but here would be the commands:
cd /usr/src scp backup2:"/mnt/data4/build/freebsd/patches/8.0/*" .
Apply patches, i.e. the jls-patch:
patch -l < jls-patch
build, install kernel and world
Rename current generic kernel so it will always be available to boot from. Save room by removing non-needed kernel modules:
cd /boot mv kernel kernel.GENERIC cd kernel.GENERIC mkdir hold mv mfi_linux.ko hold/ mv linux.ko hold/ mv linprocfs.ko hold/ mv linsysfs.ko hold/ mv geom_vinum.ko hold/ mv geom_concat.ko hold/ mv zfs.* hold/ mv opensolaris* hold/ rm *.ko rm *.symbols mv hold/* . rmdir hold/ cd /usr/src make buildkernel installkernel make buildworld ; mail -s 'buildworld done' dave.boodman@vtext.com < /dev/null
~38mins
make installworld
~34min
mergemaster -i
You will be prompted to merge, replace or ignore files changed by the src update. In most cases you can delete the temp (new) files.
ONLY if this will be a zfs system (not currently used in 8.x):
cd /sys/modules/zfs make make install cd /sys/modules/opensolaris make make install
populate devfs ruleset
scp backup2:/mnt/data4/build/freebsd/devfs.rules.8x /etc/devfs.rules
populate /etc/rc.conf with IPs and service settings
vi /etc/rc.conf kern_securelevel_enable="NO" portmap_enable="NO" sendmail_enable="NO" usbd_enable="YES" nfs_client_enable="YES" nfs_reserved_port_only="YES" inetd_enable="YES" inetd_flags="-wW -a 10.1.2.103" devfs_system_ruleset="devfsrules_show_all" ifconfig_bce1="inet 10.1.2.103 netmask 255.255.255.0" ifconfig_bce0="inet 69.55.229.7 netmask 255.255.255.0" #ifconfig_bce0_alias0="inet 69.55.2xx.xx netmask 255.255.255.0" fsck_y_enable="YES" background_fsck="NO" #rc_mfi_raid_tty_log="YES" #zfs_enable="YES"
Modify IPs, hostname, gateway for this box.
make sure sysctls are set and preserved upon boot
echo "kern.consmute=0\ kern.ipc.shm_use_phys=1\ kern.ipc.shmall=131070\ kern.ipc.shmmax=134217728\ net.inet.tcp.syncookies=0\ kern.maxfiles=32768\ kern.fallback_elf_brand=3\ kern.maxprocperuid=4000\ security.jail.sysvipc_allowed=1\ security.jail.allow_raw_sockets=1\ security.jail.socket_unixiproute_only=1\ security.jail.chflags_allowed=0\ dev.amr.0.allow_volume_configure=1\ compat.linux.osrelease=2.6.12\ vm.pmap.shpgperproc=500\ security.bsd.unprivileged_read_msgbuf=0\ kern.maxvnodes=400000" >> /etc/sysctl.conf
Tuning note: watch vfs.numvnodes while the server is live to get guidance on where to set maxvnodes
mount procfs
echo "proc /proc procfs rw 0 0" >> /etc/fstab echo "linprocfs /usr/compat/linux/proc linprocfs rw 0 0" >> /etc/fstab echo "linsysfs /usr/compat/linux/sys linsysfs rw 0 0" >> /etc/fstab mkdir -p /usr/compat/linux/proc mkdir -p /usr/compat/linux/sys
enable noatime option
data1 and data2 should look something like (add ',noatime' after 'rw'):
/dev/mfid0s1g /mnt/data1 ufs rw,noatime 2 2 /dev/mfid1s1d /mnt/data2 ufs rw,noatime 2 2
reboot. Confirm new kernel is loaded, devfs in place
uname -a
Check devfs rules
devfs rule showsets devfs rule -s 3 show
Should see:
# devfs rule showsets 1 2 3 4 # devfs rule -s 3 show 100 include 1 207 path pts* unhide 217 path fd unhide 218 path fd/* unhide 300 path ttyp* unhide 301 path ttyq* unhide 302 path ttyr* unhide 303 path ttys* unhide 304 path ttyP* unhide 305 path ttyQ* unhide 306 path ttyR* unhide 307 path ttyS* unhide 400 path null unhide 500 path zero unhide 600 path random unhide 610 path urandom unhide 700 path mem unhide 710 path kmem unhide 810 path mdctl unhide 900 path stdin unhide 910 path stdout unhide 920 path stderr unhide
update ports
cd /usr/ports echo "*default host=cvsup4.FreeBSD.org\ *default base=/usr\ *default prefix=/usr\ *default release=cvs tag=RELENG_8_3\ *default delete use-rel-suffix\ *default compress\ ports-all tag=." > sup cvsup sup; mail -s 'cvs sup ports done' support@johncompanies.com < /dev/null
~24mins
Install raid mgmt tool
Perc5/i, 6/i
Pull over cli from previous system (jail9)
scp /usr/local/sbin/mega* root@10.1.4.109:/usr/local/sbin/ scp /usr/local/libexec/MegaCli root@10.1.4.109:/usr/local/libexec/MegaCli
These are linux-based tools. This will require linux base which we would normally install from ports, but since it's failing lately, we just pull in the compat libraries from another system:
rsync -aSHv --exclude=proc --exclude=sys 10.1.4.109:/usr/compat/linux/ /usr/compat/linux/
Test:
rehash; megacli ldinfo lall a0
DEPRECATED: Assuming it worked, here's how we used to install linux_base:
cd /usr/ports/emulators/linux_base-fc4 make install clean
(didnt succeed due to libtool requirement)
cd /usr/ports/distfiles fetch http://www.lsi.com/DistributionSystem/AssetDocument/support/downloads/megaraid/miscellaneous/linux/2.00.15_Linux_MegaCLI.zip cd /usr/ports/sysutils/linux-megacli make install clean
(also failed due to libtool)
2850 PERC 4e/Di- no linux
cd /usr/ports/distfiles/ fetch http://backup01.best-hosting.ru/pub/FreeBSD/ports/distfiles/dr_freebsd_1.51.zip cd /usr/ports/sysutils/megarc make install clean megarc -dispCfg -a0
install rsync from ports
cd /usr/ports/net/rsync make install clean
choose default options
configure inetd to respond to mrtg load queries
echo "load stream tcp nowait user /usr/local/jail/bin/load.pl load.pl" >> /etc/inetd.conf echo "load 12384/tcp" >> /etc/services
install perl
cd /usr/ports/lang/perl5.12 make install clean
choose defaults
install bb client
Compiling from source on AMD64 will not work. So, we use a linux-compiled version and rely on linux compat. Per above, linux compat won't install on 8.x - libtool 2.4 need. So, instead we copy(ed) over linux:
rsync -aSHv --exclude=proc --exclude=sys 10.1.4.108:/usr/compat/linux/ /usr/compat/linux/
adduser
Output/response:
Username: bb Full name: bb Uid (Leave empty for default): 1984 Login group [bb]: Login group is bb. Invite bb into other groups? []: Login class [default]: Shell (sh csh tcsh nologin) [sh]: Home directory [/home/bb]: Use password-based authentication? [yes]: Use an empty password? (yes/no) [no]: Use a random password? (yes/no) [no]: yes Lock out the account after creation? [no]: Username : bb Password : <random> Full Name : bb Uid : 1984 Class : Groups : bb Home : /home/bb Shell : /bin/sh Locked : no OK? (yes/no): yes
cd /usr/home/bb scp backup2:/mnt/data4/build/bb/bb-freebsd_linuxcompat.tgz . tar xzf bb-freebsd_linuxcompat.tgz
edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \ 10.1.4.103 jail3.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
Edit for machine name and private IP.
if this machine is at i2b:
echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \ 69.55.229.7 jail3.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
vi /home/bb/bbc1.9e-btf/ext/openfiles MACHINE="jail3,johncompanies,com" # HAS TO BE IN A,B,C FORM
Edit for machine name.
cd /usr/home/bb/bbc1.9e-btf/etc ./bbchkcfg.sh (y to questions) ./bbchkhosts.sh (ignore ssh errors) cd ../.. chown -R bb . su bb cd cd bbc1.9e-btf ./runbb.sh start more BBOUT (look for errors) exit
Put in script to start bb @ boot:
echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh chmod +x /usr/local/etc/rc.d/bb.sh
remove reserve space
cd umount /mnt/data1 umount /mnt/data2 tunefs -m 0 /mnt/data1 tunefs -m 0 /mnt/data2 mount -a
setup rdate
cd /usr/ports/sysutils/rdate make install clean
crontab -e 0 0 * * * /usr/local/sbin/rdate -s utcnist.colorado.edu
/usr/local/sbin/rdate -s utcnist.colorado.edu
We used to use ntpd, however it listens on jail IPs which is a security risk. So we stopped. Here's the old instructions: install new ntp from ports
/usr/ports/net/ntp make install clean
echo "server 10.1.4.5" > /etc/ntp.conf /usr/sbin/ntpd -p /var/run/ntpd.pid sleep 2; ntpq -p
(confirm it’s able to reach our time server)
make a data partition
IF you didn't format the data partition during sysinstall:
Create a g partition on 2nd mirror – bsdlabel no longer works (below shows d partition made with sysinstall):
jail8 /usr/home/bb# gpart show
=> 63 285474735 mfid0 MBR (136G)
63 285458922 1 freebsd [active] (136G)
285458985 15813 - free - (7.7M)
=> 0 285458922 mfid0s1 BSD (136G)
0 524288 1 freebsd-ufs (256M)
524288 12582912 2 freebsd-swap (6.0G)
13107200 524288 4 freebsd-ufs (256M)
13631488 524288 5 freebsd-ufs (256M)
14155776 8388608 6 freebsd-ufs (4.0G)
22544384 262914538 7 freebsd-ufs (125G)
=> 63 584843175 mfid1 MBR (279G)
63 584830197 1 freebsd [active] (279G)
584830260 12978 - free - (6.3M)
=> 0 584830197 mfid1s1 BSD (279G)
0 16777216 2 freebsd-swap (8.0G)
16777216 568052981 4 freebsd-ufs (271G)
jail8 /usr/home/bb# gpart show mfid1s1
=> 0 584830197 mfid1s1 BSD (279G)
0 16777216 2 freebsd-swap (8.0G)
16777216 568052981 4 freebsd-ufs (271G)
# gpart list mfid1s1
Geom name: mfid1s1
fwheads: 255
fwsectors: 63
last: 584830196
first: 0
entries: 8
scheme: BSD
Providers:
1. Name: mfid1s1b
Mediasize: 8589934592 (8.0G)
Sectorsize: 512
Mode: r1w1e0
rawtype: 1
length: 8589934592
offset: 0
type: freebsd-swap
index: 2
end: 16777215
start: 0
2. Name: mfid1s1d
Mediasize: 290843126272 (271G)
Sectorsize: 512
Mode: r0w0e0
rawtype: 7
length: 290843126272
offset: 8589934592
type: freebsd-ufs
index: 4
end: 584830196
start: 16777216
Consumers:
1. Name: mfid1s1
Mediasize: 299433060864 (279G)
Sectorsize: 512
Mode: r1w1e1
# gpart delete -i 4 mfid1s1
mfid1s1d deleted
jail8 /usr/home/bb# gpart list mfid1s1
Geom name: mfid1s1
fwheads: 255
fwsectors: 63
last: 584830196
first: 0
entries: 8
scheme: BSD
Providers:
1. Name: mfid1s1b
Mediasize: 8589934592 (8.0G)
Sectorsize: 512
Mode: r1w1e0
rawtype: 1
length: 8589934592
offset: 0
type: freebsd-swap
index: 2
end: 16777215
start: 0
Consumers:
1. Name: mfid1s1
Mediasize: 299433060864 (279G)
Sectorsize: 512
Mode: r1w1e1
# gpart show mfid1s1
=> 0 584830197 mfid1s1 BSD (279G)
0 16777216 2 freebsd-swap (8.0G)
16777216 568052981 - free - (271G)
# gpart add -t freebsd-ufs -i 7 mfid1s1
mfid1s1g added
# gpart show mfid1s1
=> 0 584830197 mfid1s1 BSD (279G)
0 16777216 2 freebsd-swap (8.0G)
16777216 568052981 7 freebsd-ufs (271G)
Here's how we USED to do it with bsdlabel:
bsdlabel -e /dev/mfid0s1
given:
# /dev/aacd0s1: 8 partitions: # size offset fstype [fsize bsize bps/cpg] a: 262144 0 4.2BSD 2048 16384 16392 b: 4194304 262144 swap c: 143363997 0 unused 0 0 # "raw" part, don't edit d: 524288 4456448 4.2BSD 2048 16384 32776 e: 524288 4980736 4.2BSD 2048 16384 32776 f: 6291456 5505024 4.2BSD 2048 16384 28552
new offset = 6291456 + 5505024 = 11796480
new size is size for 'c' partition minus the new start from above
143363997 - 11796480 = 131567517
So:
g: 131567517 11796480 unused 0 0
create the jail template
cd /usr/ports/sysutils/jailutils make install clean
Create an md device to hold the jail:
touch /mnt/data1/jail-template20g mdconfig -a -t vnode -s 20g -f /mnt/data1/jail-template20g -u 0 newfs -O 1 /dev/md0 mkdir /mnt/data1/jail-DIR mount /dev/md0 /mnt/data1/jail-DIR
Build world into the jail:
cd /usr/src make world DESTDIR=/mnt/data1/jail-DIR; taskdone
~1hr
Make /etc into the jail, mount dev, copy in jkill:
cd etc make distribution DESTDIR=/mnt/data1/jail-DIR mount -t devfs devfs /mnt/data1/jail-DIR/dev devfs -m /mnt/data1/jail-DIR/dev rule -s 3 applyset cd /mnt/data1/jail-DIR ln -sf dev/null kernel cp /usr/local/sbin/jkill /mnt/data1/jail-DIR/sbin
Enter into jail to do configuration:
jail /mnt/data1/jail-DIR testhostname 192.168.11.100 /bin/sh csh
Create fstab:
touch /etc/fstab echo 'network_interfaces=""\ hostname="newsystem"\ kern_securelevel_enable="NO"\ sendmail_enable="YES"\ sshd_enable="YES"' > /etc/rc.conf echo "nameserver 69.55.225.225\ nameserver 69.55.230.3" >> /etc/resolv.conf
Edit crontab:
vi /etc/crontab
remove the adjkerntz lines comment out periodic’s and put this line above them:
# IF YOU UNCOMMENT THESE, PLEASE ADJUST THEIR RUN TIME
rm -rf /etc/periodic/daily/400.status-disks
check and remove any crap in /tmp
vi /etc/periodic/security/100.chksetuid
replace: MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`
with: MP='/' (use single quotes)
mkdir -p /usr/compat/linux/dev
Add account for user. Output/response:
adduser Username: user Full name: user Uid (Leave empty for default): Login group [user]: Login group is user. Invite user into other groups? []: wheel Login class [default]: Shell (sh csh tcsh nologin) [sh]: Home directory [/home/user]: Home directory permissions (Leave empty for default): Use password-based authentication? [yes]: Use an empty password? (yes/no) [no]: Use a random password? (yes/no) [no]: y Lock out the account after creation? [no]: Username : user Password : <random> Full Name : user Uid : 1001 Class : Groups : user Home : /home/user Home Mode : Shell : /bin/sh Locked : no OK? (yes/no): y adduser: INFO: Successfully added (user) to the user database. adduser: INFO: Password for (user) is: 901gmYjO Add another user? (yes/no): n Goodbye!
set TERM:
vi /usr/home/user/.profile TERM=vt100; export TERM
Set time zone to PT:
tzsetup
Reload aliases:
newaliases
Replace reboot/halt:
rm /sbin/halt /sbin/reboot ln /sbin/jkill /sbin/halt ln /sbin/jkill /sbin/reboot
Redirect console output, comment out console and move to /var/log/messages:
vi /etc/syslog.conf #*.err;kern.warning;auth.notice;mail.crit /dev/console *.err;kern.warning;auth.notice;mail.crit /var/log/messages
exit exit
Lib32 compat library:
cd libexec chflags noschg ld-elf32.so.1 chflags noschg ld-elf.so.1 mv ld-elf32.so.1 ld-elf32.so.1-orig ln ld-elf.so.1 ld-elf32.so.1 chflags schg ld-elf.so.1 chflags schg ld-elf32.so.1
Replace traceroute:
mv /mnt/data1/jail-DIR/usr/sbin/traceroute /mnt/data1/jail-DIR/usr/sbin/_traceroute echo '#\!/bin/sh\ /usr/sbin/_traceroute -i bce0 $1' >> /mnt/data1/jail-DIR/usr/sbin/traceroute chmod +x /mnt/data1/jail-DIR/usr/sbin/traceroute
Modify 'bce0' to reflect whichever nic is public on this hardware.
Clean out ports before copying into the jail (~30mins):
cd /usr/ports make -DNOCLEANDEPENDS clean rm -fr /usr/ports/distfiles/* cp -r /usr/ports /mnt/data1/jail-DIR/usr
rm /mnt/data1/jail-DIR/root/.history
Umount the jail and dump it:
cd / umount /mnt/data1/jail-DIR/dev dump -0a -f /usr/local/jail/template/template /dev/md0 umount /dev/md0 rmdir /mnt/data1/jail-DIR mdconfig -d -u 0
setup backups
echo '#\!/bin/sh\ backupdir=/data/jail3\ server=backup1\ \ ## ENTRY /etc\ ## ENTRY /usr/local/etc\ ## ENTRY /usr/local/jail\ ## ENTRY /root/logs' > /usr/local/jail/bin/backup.config
Edit to reflect backup server and jail hostname
On backup server, setup backup dirs:
ssh backup1 mkdir -p /data/jail3/0
On backup server, setup backup dirs:
backup1# vi /usr/local/sbin/snapshot_rotate
Edit /usr/local/jail/bin/backup.xxx to use the right drives and copy to /usr/local/jail/bin/backup
vi /usr/local/jail/bin/backup.md
adjust df so it includes all relevant drives, currently df > /etc/df.bak is fine. Also, make sure the binary/script source is pulling from the right dir on backup2:
/usr/local/bin/rsync -a backup2:/mnt/data4/bin/freebsd8.x/ /usr/local/jail/bin/
cp /usr/local/jail/bin/backup.md /usr/local/jail/bin/backup
create /root/logs
mkdir /root/logs
edit sshd_config
vi /etc/ssh/sshd_config ListenAddress 69.55.229.7 ListenAddress 10.1.2.103
Adjust to pub/private IP for jail.
Restart sshd:
kill -1 `cat /var/run/sshd.pid`
add crontab entries
crontab -e 5 0 * * * /usr/local/jail/bin/backup 1 0 1 * * /usr/local/jail/bin/ipfwreset 0 18 * * * /usr/local/jail/bin/ipfwbackup 4,9,14,19,24,29,34,39,44,49,55,59 * * * * /usr/local/jail/bin/trafstats 0 0,6,12,18 * * * /usr/local/jail/bin/sync_jail_names
On 2950:
*/5 * * * * /usr/local/jail/bin/perc5iraidchk
On 3ware-based servers:
0 0 * * * /usr/local/jail/bin/3wraidchk
Reboot notify script
ln -s /usr/local/jail/bin/notify.sh /usr/local/etc/rc.d/notify.sh
patch jail against starting jails with rtprio
mv /usr/sbin/jail /usr/sbin/jail_ echo '#\!/bin/sh\ /usr/sbin/rtprio -t /usr/sbin/jail_ $*' > /usr/sbin/jail chmod +x /usr/sbin/jail
make sure mail works
If there are map errors:
cd /etc/mail; make maps
recover space on /usr (optional)
rm -fr /usr/obj
wrapper for jps
mv /usr/local/sbin/jps /usr/local/sbin/jps_
wrapper for jls
mv /usr/sbin/jls /usr/sbin/jls_
wrapper for jexec
mv /usr/sbin/jexec /usr/sbin/jexec_
install jtop
cd /usr/ports/sysutils/jtop make install clean
block jails from reaching private net
echo 'ipfw add 1 deny ip from 69.55.224.0/20 to 10.1.4.0/24' > /usr/local/etc/rc.d/ipfw.sh chmod 0700 /usr/local/etc/rc.d/ipfw.sh
add to management infrastructure
add to management db
tables: jc.ref_machines and jc.ref_templates
on jail run:
uname -r
Which shows something like:
8.3-RELEASE-p2
Insert into db:
insert into ref_machines values (null,'jail3','mx3.johncompanies.com',0,'f8');
select machine_id from ref_machines where host='jail3';
+------------+
| machine_id |
+------------+
| 35 |
+------------+
insert into ref_templates values ('','8.3-RELEASE-jc2',35,'FreeBSD 8.3',1);
add to bb server
vi /usr/home/bb/bbsrc/bb1.9i-btf/etc/bb-hosts 10.1.4.109 jail9.johncompanies.com # ssh
In the case of an i2b server, use real ip:
69.55.229.7 jail3.johncompanies.com # ssh
su bb cd bbsrc/bb/runbb.sh restart ; exit
Update backupgraph
vi /usr/local/www/mgmt/cgi/backupgraph.pl
(add hostname)
Update load mrtg
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg
(add new entry to file following existing format)
ns1c
fwd and reverse lookups:
vr johncompanies.com
vi internal.johncompanies.com rndc reload johncompanies.com IN private
ptr 69.55.227.x
wiki
add to server/cabinet map
firewall
add an outside blocking rule to the firewall, so this machine can only be reached from inside the firewall. Follow example already in firewall jail17 is:
00117 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.228.200 00117 deny ip from any to 69.55.228.200
jail19 would be 00119... ipfw add 00109 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 } to 69.55.232.3 ipfw add 00109 deny ip from any to 69.55.232.3