Jail Server Install
9.x
Assumptions
Setup instructions below assume this is DELL 2950 with an LSI-based SAS RAID card.
Server is at castle, connected to pub, private, serial and DRAC
Assuming OS loading done via IPKVM with ISO mounted via USB
Assumes at 4 drives, 2 mirrors
Configure server BIOS
setup console redirect, speed 115200
set LCD string to name of server "jail8"
set date to GMT
go into RAID bios and setup mirrors
Setup DRAC
Install OS (sysinstall)
boot to bootonly disk for AMD version of FreeBSD, i.e. FreeBSD-8.3-RELEASE-amd64-bootonly.iso
when the install menu appears, choose custom install
move cursor to mfid0, hit space (takes you to partition map screen). If there is only 1 mirror, there will be no option to select a specifit drive: mfid0 will be selected.
type 'a' to use entire disk
type 'q' to quit and save
choose 'freebsd standard mbr'
space to unselect mfid0
cursor down to mfid1
hit space
type 'a' to use entire disk
type 'q' to quit and save
choose 'none' for boot mgr (leave untouched)
cursor over mfid0
space
(takes you into part. Screen again) 'q' to exit
none for boot mgr
Make sure both drives (mfid0 and mfid1) are checked and tab to ok
Make sure mfid0 is highlighted at the top of the screen, setup the following partitions
/ 512M
swap 6G
/var 1G
/tmp 256M
/usr 8G
/mnt/data1 remaining space
All partitions except / should be setup for soft updates. If not, type 's' to enable for soft updates on all except for / (should look like UFS2+S Y under the Newfs column)
move cursor to mfid1 at the top of the screen
swap 8G (or 4G if there’s a 3rd drive)
/mnt/data2 remaining space
'q' to save and exit
distributions
Choose the following distribudions
- developer (ok to install ports)
- custom -> lib32
exit
media
if you are installing via a cd, no need to enter this menu or change anything. Otherwise, choose ftp to install via ftp. You will be prompted to setup networking. You will need to choose a nic (typically bce0 or bce1). Say no to DHCP and IPv6. Hopefully the public nic cable was installed in bce0 so start with that nic and provide the hostname, (public) IP, netmask, gateway and DNS. When configured, it should start pinging. If it doesn't, have the NOC swap cables. Select any FTP server, usually Main or ftp4.
commit
this usually takes 12mins
during the process you may need to select a new ftp mirror, this is not a problem.
at the conclution of the install you will be prompted to enter the root password (2x) and returned to the configuration menu.
add user
Add user 'user'. Defaults for everything is fine, just remember to enter 'wheel' in the member group field. Do set the password.
Setup timezone
PT
Networking
page down to the bottom and enable '[X]' sshd
If you installed via cd, you will need to visit:
interfaces->bce0->
No IPV6
dhcp=no
Set hostname, IP, DNS, gateway
(i.e. setup the nic as indicated above)
Exit the install and if you installed via CD, take it out and let the machine reboot
Configure OS, kernel, userland, jail
double check the date/time
populate /etc/resolv.conf
echo "search johncompanies.com \ nameserver 69.55.225.225 \ nameserver 69.55.230.3 \ nameserver 69.55.229.3" > /etc/resolv.conf
edit /etc/make.conf
echo "WITHOUT_X11=yes \ KERNCONF=jail4 \ BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf
setup bootloader for console, etc
add settings to /boot/loader.conf and /boot.config:
echo "-Dh" >> /boot.config
echo 'console="comconsole,vidconsole" \ boot_multicons="YES" \ boot_serial="YES" \ mfi_linux_load="YES" \ comconsole_speed="115200"' >> /boot/loader.conf
enable login via serial console
turn off all ttyv's except 0 and 1 in /etc/ttys and turn on ttyd0, change type to vt100:
vi /etc/ttys
The changed lines should look like:
ttyv2 "/usr/libexec/getty Pc" cons25 off secure ttyv3 "/usr/libexec/getty Pc" cons25 off secure ttyv4 "/usr/libexec/getty Pc" cons25 off secure ttyv5 "/usr/libexec/getty Pc" cons25 off secure ttyv6 "/usr/libexec/getty Pc" cons25 off secure ttyv7 "/usr/libexec/getty Pc" cons25 off secure # Serial terminals # The 'dialup' keyword identifies dialin lines to login, fingerd etc. ttyu0 "/usr/libexec/getty std.9600" vt100 on secure
Restart init
kill -1 1
At this point you should have a login on console.
To configure serial console access, login to the console server as root and run:
# vi /etc/remote
following examples there, rename port to server's hostname, depending on where and which digi box this server is plugged into. Make sure to get speed right too: 115200
populate hosts
If server is at castle:
echo "10.1.4.3 backup2" >> /etc/hosts echo "10.1.4.8 backup1" >> /etc/hosts
If server is at i2b:
echo "69.55.230.10 backup2" >> /etc/hosts echo "10.1.2.3 backup3" >> /etc/hosts echo "69.55.230.11 backup1" >> /etc/hosts
create ssh key, upload to backup servers
cd ssh-keygen -t dsa -b 1024
(default location, leave password blank)
If server is at castle:
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys' cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'
If server is at i2b:
cat /root/.ssh/id_dsa.pub | ssh backup3 'cat - >> /root/.ssh/authorized_keys' cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys' cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'
confirm that you can ssh to backup2 and backup1 without getting a login prompt
ssh backup2 hostname ssh backup1 hostname ssh backup3 hostname
create & populate binaries/scripts dirs
mkdir -p /usr/local/jail/bin mkdir -p /usr/local/jail/rc.d mkdir -p /usr/local/jail/template/ mkdir /mnt/data1 mkdir /mnt/data2 scp backup2:"/mnt/data4/bin/freebsd8.x/*" /usr/local/jail/bin cd /usr/local/jail/rc.d/ touch quad1 touch deprecated chmod +x * cd /usr/local/jail/bin ln -s /usr/local/jail/rc.d/quad1 quad1 ln -s /usr/local/jail/bin/jailmake_md jailmake ln -s /usr/local/jail/bin/js_md js ln -s /usr/local/jail/bin/canceljail_md canceljail ln -s /usr/local/jail/bin/jailmakeempty_md jailmakeempty ln -s /usr/local/jail/bin/postboot_md postboot ln -s /usr/local/jail/bin/preboot_md preboot ln -s /usr/local/jail/bin/startjail_md startjail ln -s /usr/local/jail/bin/stopjail_md stopjail rehash
edit root's path and login script
vi /root/.cshrc
Change alias entries (add G):
alias la ls -aG alias lf ls -FAG alias ll ls -lAG alias ls ls -AG alias mbm mb mount alias mbu mb umount alias cjb cd /usr/local/jail/bin alias cd1 cd /mnt/data1 alias cd2 cd /mnt/data2 alias cd3 cd /mnt/data3 alias jtop jtop lj alias j jobs
add to path be careful to leave a space after bin and make sure the wrapping isn't broken:
/usr/local/jail/bin
alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "
at the bottom of the file add:
set sshtty=`who am i|awk '{print $2}'`
/usr/sbin/rtprio 3 -`psj | grep $sshtty | awk '{print $2}'`
set shortty=`who am i | awk '{print $2}' | sed -E 's/.*(..)$/\1/'`
foreach x (`psj | grep sh | grep $shortty | awk '{print $2}'`)
/usr/sbin/rtprio 2 -$x
end
Make the new settings active in current shell:
source /root/.cshrc
update ports
portsnap fetch portsnap extract
To update later on:
portsnap fetch portsnap update
install svn
setenv PACKAGESITE "ftp://ftp4.freebsd.org/pub/FreeBSD/ports/amd64/packages-9-current/Latest/" pkg_add subversion
get latest sources for this release
cd /usr mv src/ src.orig svn checkout svn://svn.FreeBSD.org/base/stable/9 /usr/src
To update:
make update SVN_UPDATE=yes
configure new kernel
Pull down the kernel config we are using for this distribution. In this case we use an 8.2 kernel config on 8.3, which is valid. The local file should be the same name as host- jail3 in this example
cd /usr/src/sys/amd64/conf scp backup2:/mnt/data4/build/freebsd/kern_config-8.2-amd64 ./jail3
edit the kernel config and change ident to be the name of the jail:
vi jail3 ident jail3
Optional, edit /sys/conf/newvers.sh to add –jc2 to the end of the BRANCH string (RELEASE-jc2)
vi /sys/conf/newvers.sh
notes on kernel configuring: http://www.FreeBSD.org/doc/en_US.ISO8859-1/books/handbook/kernelconfig-config.html
install patches
We don't have any patches right now. Refer to older FreeBSD version build docs on how that is/was done.
build, install kernel and world
Rename current generic kernel so it will always be available to boot from. Save room by removing non-needed kernel modules:
cd /boot mv kernel kernel.GENERIC cd kernel.GENERIC mkdir hold mv mfi_linux.ko hold/ mv linux.ko hold/ mv linprocfs.ko hold/ mv linsysfs.ko hold/ mv geom_vinum.ko hold/ mv geom_concat.ko hold/ mv zfs.* hold/ mv opensolaris* hold/ rm *.ko rm *.symbols mv hold/* . rmdir hold/ Note on -DWITHOUT_CLANG: try to do it without including that directive, it may work for you. cd /usr/src make buildworld KERNCONF=jail4; mail -s 'buildworld done' dave.boodman@vtext.com < /dev/null make -DWITHOUT_CLANG buildworld KERNCONF=jail4; mail -s 'buildworld done' dave.boodman@vtext.com < /dev/null
~38mins
cd /usr/src make buildkernel installkernel
mergemaster -p
You will be prompted to merge, replace or ignore files changed by the src update. In most cases you can delete the temp (new) files.
make -DWITHOUT_CLANG installworld
~34min
ONLY if this will be a zfs system (not currently used in 8.x):
cd /sys/modules/zfs make make install cd /sys/modules/opensolaris make make install
populate devfs ruleset
scp backup2:/mnt/data4/build/freebsd/devfs.rules.8x /etc/devfs.rules
populate /etc/rc.conf with IPs and service settings
vi /etc/rc.conf kern_securelevel_enable="NO" portmap_enable="NO" sendmail_enable="NO" usbd_enable="YES" nfs_client_enable="YES" nfs_reserved_port_only="YES" inetd_enable="YES" inetd_flags="-wW -a 10.1.2.103" devfs_system_ruleset="devfsrules_show_all" ifconfig_bce1="inet 10.1.2.103 netmask 255.255.255.0" ifconfig_bce0="inet 69.55.229.7 netmask 255.255.255.0" #ifconfig_bce0_alias0="inet 69.55.2xx.xx netmask 255.255.255.0" fsck_y_enable="YES" background_fsck="NO" #rc_mfi_raid_tty_log="YES" #zfs_enable="YES"
Modify IPs, hostname, gateway for this box.
make sure sysctls are set and preserved upon boot
echo "kern.consmute=0\ kern.ipc.shm_use_phys=1\ kern.ipc.shmall=131070\ kern.ipc.shmmax=134217728\ net.inet.tcp.syncookies=0\ kern.maxfiles=32768\ kern.fallback_elf_brand=3\ kern.maxprocperuid=4000\ security.jail.sysvipc_allowed=1\ security.jail.allow_raw_sockets=1\ security.jail.socket_unixiproute_only=1\ security.jail.chflags_allowed=0\ dev.amr.0.allow_volume_configure=1\ compat.linux.osrelease=2.6.12\ vm.pmap.shpgperproc=500\ security.bsd.unprivileged_read_msgbuf=0\ kern.maxvnodes=400000" >> /etc/sysctl.conf
Tuning note: watch vfs.numvnodes while the server is live to get guidance on where to set maxvnodes
mount procfs
echo "proc /proc procfs rw 0 0" >> /etc/fstab echo "linprocfs /usr/compat/linux/proc linprocfs rw 0 0" >> /etc/fstab echo "linsysfs /usr/compat/linux/sys linsysfs rw 0 0" >> /etc/fstab mkdir -p /usr/compat/linux/proc mkdir -p /usr/compat/linux/sys
enable noatime option
data1 and data2 should look something like (add ',noatime' after 'rw'):
/dev/mfid0s1g /mnt/data1 ufs rw,noatime 2 2 /dev/mfid1s1d /mnt/data2 ufs rw,noatime 2 2
reboot. Confirm new kernel is loaded, devfs in place
uname -a
Check devfs rules
devfs rule showsets devfs rule -s 3 show
Should see:
# devfs rule showsets 1 2 3 4 # devfs rule -s 3 show 100 include 1 207 path pts* unhide 217 path fd unhide 218 path fd/* unhide 300 path ttyp* unhide 301 path ttyq* unhide 302 path ttyr* unhide 303 path ttys* unhide 304 path ttyP* unhide 305 path ttyQ* unhide 306 path ttyR* unhide 307 path ttyS* unhide 400 path null unhide 500 path zero unhide 600 path random unhide 610 path urandom unhide 700 path mem unhide 710 path kmem unhide 810 path mdctl unhide 900 path stdin unhide 910 path stdout unhide 920 path stderr unhide
Install raid mgmt tool
Perc5/i, 6/i
Pull over cli from previous system (jail9)
scp root@10.1.4.109:"/usr/local/sbin/mega*" /usr/local/sbin/ scp root@10.1.4.109:/usr/local/libexec/MegaCli /usr/local/libexec/MegaCli
These are linux-based tools. This will require linux base...which you might install via:
pkg_add -r linux_base
Test:
rehash; megacli ldinfo lall a0
or
megarc -ldInfo -a0 -Lall
(2850)
however, linux does seem to be installed already so we don't need to pkg_add or port install or rsync anything over from a current system.
2850 PERC 4e/Di- no linux
cd /usr/ports/distfiles/ fetch http://backup01.best-hosting.ru/pub/FreeBSD/ports/distfiles/dr_freebsd_1.51.zip cd /usr/ports/sysutils/megarc make install clean megarc -dispCfg -a0
install rsync from ports
cd /usr/ports/net/rsync make install clean
choose default options
configure inetd to respond to mrtg load queries
echo "load stream tcp nowait user /usr/local/jail/bin/load.pl load.pl" >> /etc/inetd.conf echo "load 12384/tcp" >> /etc/services
install perl
cd /usr/ports/lang/perl5.12 make install clean
choose defaults
install bb client
Compiling from source on AMD64 will not work. So, we use a linux-compiled version and rely on linux compat.
adduser
Output/response:
Username: bb Full name: bb Uid (Leave empty for default): 1984 Login group [bb]: Login group is bb. Invite bb into other groups? []: Login class [default]: Shell (sh csh tcsh nologin) [sh]: Home directory [/home/bb]: Use password-based authentication? [yes]: Use an empty password? (yes/no) [no]: Use a random password? (yes/no) [no]: yes Lock out the account after creation? [no]: Username : bb Password : <random> Full Name : bb Uid : 1984 Class : Groups : bb Home : /home/bb Shell : /bin/sh Locked : no OK? (yes/no): yes
cd /usr/home/bb scp backup2:/mnt/data4/build/bb/bb-freebsd_linuxcompat.tgz . tar xzf bb-freebsd_linuxcompat.tgz
edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \ 10.1.4.103 jail3.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
Edit for machine name and private IP.
if this machine is at i2b:
echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \ 69.55.229.7 jail3.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
vi /home/bb/bbc1.9e-btf/ext/openfiles MACHINE="jail3,johncompanies,com" # HAS TO BE IN A,B,C FORM
Edit for machine name.
cd /usr/home/bb/bbc1.9e-btf/etc ./bbchkcfg.sh (y to questions) ./bbchkhosts.sh (ignore ssh errors) cd ../.. chown -R bb . su bb cd cd bbc1.9e-btf ./runbb.sh start more BBOUT (look for errors) exit
Put in script to start bb @ boot:
echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh chmod +x /usr/local/etc/rc.d/bb.sh
remove reserve space
cd umount /mnt/data1 umount /mnt/data2 tunefs -m 0 /mnt/data1 tunefs -m 0 /mnt/data2 mount -a
setup rdate
cd /usr/ports/sysutils/rdate make install clean
crontab -e 0 0 * * * /usr/local/sbin/rdate -s utcnist.colorado.edu
/usr/local/sbin/rdate -s utcnist.colorado.edu
We used to use ntpd, however it listens on jail IPs which is a security risk. So we stopped. Here's the old instructions: install new ntp from ports
/usr/ports/net/ntp make install clean
echo "server 10.1.4.5" > /etc/ntp.conf /usr/sbin/ntpd -p /var/run/ntpd.pid sleep 2; ntpq -p
(confirm it’s able to reach our time server)
make a data partition
IF you didn't format the data partition during sysinstall:
Create a g partition on 2nd mirror – bsdlabel no longer works (below shows d partition made with sysinstall):
jail8 /usr/home/bb# gpart show
=> 63 285474735 mfid0 MBR (136G)
63 285458922 1 freebsd [active] (136G)
285458985 15813 - free - (7.7M)
=> 0 285458922 mfid0s1 BSD (136G)
0 524288 1 freebsd-ufs (256M)
524288 12582912 2 freebsd-swap (6.0G)
13107200 524288 4 freebsd-ufs (256M)
13631488 524288 5 freebsd-ufs (256M)
14155776 8388608 6 freebsd-ufs (4.0G)
22544384 262914538 7 freebsd-ufs (125G)
=> 63 584843175 mfid1 MBR (279G)
63 584830197 1 freebsd [active] (279G)
584830260 12978 - free - (6.3M)
=> 0 584830197 mfid1s1 BSD (279G)
0 16777216 2 freebsd-swap (8.0G)
16777216 568052981 4 freebsd-ufs (271G)
jail8 /usr/home/bb# gpart show mfid1s1
=> 0 584830197 mfid1s1 BSD (279G)
0 16777216 2 freebsd-swap (8.0G)
16777216 568052981 4 freebsd-ufs (271G)
# gpart list mfid1s1
Geom name: mfid1s1
fwheads: 255
fwsectors: 63
last: 584830196
first: 0
entries: 8
scheme: BSD
Providers:
1. Name: mfid1s1b
Mediasize: 8589934592 (8.0G)
Sectorsize: 512
Mode: r1w1e0
rawtype: 1
length: 8589934592
offset: 0
type: freebsd-swap
index: 2
end: 16777215
start: 0
2. Name: mfid1s1d
Mediasize: 290843126272 (271G)
Sectorsize: 512
Mode: r0w0e0
rawtype: 7
length: 290843126272
offset: 8589934592
type: freebsd-ufs
index: 4
end: 584830196
start: 16777216
Consumers:
1. Name: mfid1s1
Mediasize: 299433060864 (279G)
Sectorsize: 512
Mode: r1w1e1
# gpart delete -i 4 mfid1s1
mfid1s1d deleted
jail8 /usr/home/bb# gpart list mfid1s1
Geom name: mfid1s1
fwheads: 255
fwsectors: 63
last: 584830196
first: 0
entries: 8
scheme: BSD
Providers:
1. Name: mfid1s1b
Mediasize: 8589934592 (8.0G)
Sectorsize: 512
Mode: r1w1e0
rawtype: 1
length: 8589934592
offset: 0
type: freebsd-swap
index: 2
end: 16777215
start: 0
Consumers:
1. Name: mfid1s1
Mediasize: 299433060864 (279G)
Sectorsize: 512
Mode: r1w1e1
# gpart show mfid1s1
=> 0 584830197 mfid1s1 BSD (279G)
0 16777216 2 freebsd-swap (8.0G)
16777216 568052981 - free - (271G)
# gpart add -t freebsd-ufs -i 7 mfid1s1
mfid1s1g added
# gpart show mfid1s1
=> 0 584830197 mfid1s1 BSD (279G)
0 16777216 2 freebsd-swap (8.0G)
16777216 568052981 7 freebsd-ufs (271G)
Here's how we USED to do it with bsdlabel:
bsdlabel -e /dev/mfid0s1
given:
# /dev/aacd0s1: 8 partitions: # size offset fstype [fsize bsize bps/cpg] a: 262144 0 4.2BSD 2048 16384 16392 b: 4194304 262144 swap c: 143363997 0 unused 0 0 # "raw" part, don't edit d: 524288 4456448 4.2BSD 2048 16384 32776 e: 524288 4980736 4.2BSD 2048 16384 32776 f: 6291456 5505024 4.2BSD 2048 16384 28552
new offset = 6291456 + 5505024 = 11796480
new size is size for 'c' partition minus the new start from above
143363997 - 11796480 = 131567517
So:
g: 131567517 11796480 unused 0 0
create the jail template
cd /usr/ports/sysutils/jailutils make install clean
Create an md device to hold the jail:
touch /mnt/data1/jail-template20g mdconfig -a -t vnode -s 20g -f /mnt/data1/jail-template20g -u 0 newfs -O 1 /dev/md0 mkdir /mnt/data1/jail-DIR mount /dev/md0 /mnt/data1/jail-DIR
Build world into the jail:
cd /usr/src make world DESTDIR=/mnt/data1/jail-DIR; taskdone
~1hr
Make /etc into the jail, mount dev, copy in jkill:
cd etc make distribution DESTDIR=/mnt/data1/jail-DIR mount -t devfs devfs /mnt/data1/jail-DIR/dev devfs -m /mnt/data1/jail-DIR/dev rule -s 3 applyset cd /mnt/data1/jail-DIR ln -sf dev/null kernel cp /usr/local/sbin/jkill /mnt/data1/jail-DIR/sbin
Enter into jail to do configuration:
jail /mnt/data1/jail-DIR testhostname 192.168.11.100 /bin/sh csh
Create fstab:
touch /etc/fstab echo 'network_interfaces=""\ hostname="newsystem"\ kern_securelevel_enable="NO"\ sendmail_enable="YES"\ sshd_enable="YES"' > /etc/rc.conf echo "nameserver 69.55.225.225\ nameserver 69.55.230.3" >> /etc/resolv.conf
Edit crontab:
vi /etc/crontab
remove the adjkerntz lines comment out periodic’s and put this line above them:
# IF YOU UNCOMMENT THESE, PLEASE ADJUST THEIR RUN TIME
rm -rf /etc/periodic/daily/400.status-disks
check and remove any crap in /tmp
vi /etc/periodic/security/100.chksetuid
replace: MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`
with: MP='/' (use single quotes)
mkdir -p /usr/compat/linux/dev
Add account for user. Output/response:
adduser Username: user Full name: user Uid (Leave empty for default): Login group [user]: Login group is user. Invite user into other groups? []: wheel Login class [default]: Shell (sh csh tcsh nologin) [sh]: Home directory [/home/user]: Home directory permissions (Leave empty for default): Use password-based authentication? [yes]: Use an empty password? (yes/no) [no]: Use a random password? (yes/no) [no]: y Lock out the account after creation? [no]: Username : user Password : <random> Full Name : user Uid : 1001 Class : Groups : user Home : /home/user Home Mode : Shell : /bin/sh Locked : no OK? (yes/no): y adduser: INFO: Successfully added (user) to the user database. adduser: INFO: Password for (user) is: 901gmYjO Add another user? (yes/no): n Goodbye!
set TERM:
vi /usr/home/user/.profile TERM=vt100; export TERM
Set time zone to PT:
tzsetup
Reload aliases:
newaliases
Replace reboot/halt:
rm /sbin/halt /sbin/reboot ln /sbin/jkill /sbin/halt ln /sbin/jkill /sbin/reboot
Redirect console output, comment out console and move to /var/log/messages:
vi /etc/syslog.conf #*.err;kern.warning;auth.notice;mail.crit /dev/console *.err;kern.warning;auth.notice;mail.crit /var/log/messages
exit exit
Lib32 compat library:
cd libexec chflags noschg ld-elf32.so.1 chflags noschg ld-elf.so.1 mv ld-elf32.so.1 ld-elf32.so.1-orig ln ld-elf.so.1 ld-elf32.so.1 chflags schg ld-elf.so.1 chflags schg ld-elf32.so.1
Replace traceroute:
mv /mnt/data1/jail-DIR/usr/sbin/traceroute /mnt/data1/jail-DIR/usr/sbin/_traceroute echo '#\!/bin/sh\ /usr/sbin/_traceroute -i bce0 $1' >> /mnt/data1/jail-DIR/usr/sbin/traceroute chmod +x /mnt/data1/jail-DIR/usr/sbin/traceroute
Modify 'bce0' to reflect whichever nic is public on this hardware.
Clean out ports before copying into the jail (~30mins):
cd /usr/ports make -DNOCLEANDEPENDS clean rm -fr /usr/ports/distfiles/* cp -r /usr/ports /mnt/data1/jail-DIR/usr
rm /mnt/data1/jail-DIR/root/.history
Umount the jail and dump it:
cd / umount /mnt/data1/jail-DIR/dev dump -0a -f /usr/local/jail/template/template /dev/md0 umount /dev/md0 rmdir /mnt/data1/jail-DIR mdconfig -d -u 0
setup backups
echo '#\!/bin/sh\ backupdir=/data/jail3\ server=backup1\ \ ## ENTRY /etc\ ## ENTRY /usr/local/etc\ ## ENTRY /usr/local/jail\ ## ENTRY /root/logs' > /usr/local/jail/bin/backup.config
Edit to reflect backup server and jail hostname
On backup server, setup backup dirs:
ssh backup1 mkdir -p /data/jail3/0
On backup server, setup backup dirs:
backup1# vi /usr/local/sbin/snapshot_rotate
Edit /usr/local/jail/bin/backup.xxx to use the right drives and copy to /usr/local/jail/bin/backup
vi /usr/local/jail/bin/backup.md
adjust df so it includes all relevant drives, currently df > /etc/df.bak is fine. Also, make sure the binary/script source is pulling from the right dir on backup2:
/usr/local/bin/rsync -a backup2:/mnt/data4/bin/freebsd8.x/ /usr/local/jail/bin/
cp /usr/local/jail/bin/backup.md /usr/local/jail/bin/backup
create /root/logs
mkdir /root/logs
edit sshd_config
vi /etc/ssh/sshd_config ListenAddress 69.55.229.7 ListenAddress 10.1.2.103
Adjust to pub/private IP for jail.
Restart sshd:
kill -1 `cat /var/run/sshd.pid`
add crontab entries
crontab -e 5 0 * * * /usr/local/jail/bin/backup 1 0 1 * * /usr/local/jail/bin/ipfwreset 0 18 * * * /usr/local/jail/bin/ipfwbackup 4,9,14,19,24,29,34,39,44,49,55,59 * * * * /usr/local/jail/bin/trafstats 0 0,6,12,18 * * * /usr/local/jail/bin/sync_jail_names
On 2950:
*/5 * * * * /usr/local/jail/bin/perc5iraidchk
On 3ware-based servers:
0 0 * * * /usr/local/jail/bin/3wraidchk
Reboot notify script
ln -s /usr/local/jail/bin/notify.sh /usr/local/etc/rc.d/notify.sh
patch jail against starting jails with rtprio
mv /usr/sbin/jail /usr/sbin/jail_ echo '#\!/bin/sh\ /usr/sbin/rtprio -t /usr/sbin/jail_ $*' > /usr/sbin/jail chmod +x /usr/sbin/jail
make sure mail works
If there are map errors:
cd /etc/mail; make maps
recover space on /usr (optional)
rm -fr /usr/obj
wrapper for jps
mv /usr/local/sbin/jps /usr/local/sbin/jps_
wrapper for jls
mv /usr/sbin/jls /usr/sbin/jls_
wrapper for jexec
mv /usr/sbin/jexec /usr/sbin/jexec_
install jtop
cd /usr/ports/sysutils/jtop make install clean
block jails from reaching private net
echo 'ipfw add 1 deny ip from 69.55.224.0/20 to 10.1.4.0/24' > /usr/local/etc/rc.d/ipfw.sh chmod 0700 /usr/local/etc/rc.d/ipfw.sh
add to management infrastructure
add to management db
tables: jc.ref_machines and jc.ref_templates
on jail run:
uname -r
Which shows something like:
8.3-RELEASE-p2
Insert into db:
insert into ref_machines values (null,'jail3','mx3.johncompanies.com',0,'f8');
select machine_id from ref_machines where host='jail3';
+------------+
| machine_id |
+------------+
| 35 |
+------------+
insert into ref_templates values ('','8.3-RELEASE-jc2',35,'FreeBSD 8.3',1);
add to bb server
vi /usr/home/bb/bbsrc/bb1.9i-btf/etc/bb-hosts 10.1.4.109 jail9.johncompanies.com # ssh
In the case of an i2b server, use real ip:
69.55.229.7 jail3.johncompanies.com # ssh
su bb cd bbsrc/bb/runbb.sh restart ; exit
Update backupgraph
vi /usr/local/www/mgmt/cgi/backupgraph.pl
(add hostname)
Update load mrtg
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg
(add new entry to file following existing format)
ns1c
fwd and reverse lookups:
vr johncompanies.com
vi internal.johncompanies.com rndc reload johncompanies.com IN private
ptr 69.55.227.x
wiki
add to server/cabinet map
firewall
add an outside blocking rule to the firewall, so this machine can only be reached from inside the firewall. Follow example already in firewall jail17 is:
00117 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.228.200
00117 deny ip from any to 69.55.228.200
jail19 would be 00119...
ipfw add 00109 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 } to 69.55.232.3
ipfw add 00109 deny ip from any to 69.55.232.3
select some customers for castle probe map
8.x
Assumptions
Setup instructions below assume this is DELL 2950 with an LSI-based SAS RAID card.
Server is at castle, connected to pub, private, serial and DRAC
Assuming OS loading done via IPKVM with ISO mounted via USB
Assumes at 4 drives, 2 mirrors
Configure server BIOS
setup console redirect, speed 115200
set LCD string to name of server "jail8"
set date to GMT
go into RAID bios and setup mirrors
configure DRAC: TODO
Install OS (sysinstall)
boot to bootonly disk for AMD version of FreeBSD, i.e. FreeBSD-8.3-RELEASE-amd64-bootonly.iso
when the install menu appears, choose custom install
move cursor to mfid0, hit space (takes you to partition map screen). If there is only 1 mirror, there will be no option to select a specifit drive: mfid0 will be selected.
type 'a' to use entire disk
type 'q' to quit and save
choose 'freebsd standard mbr'
space to unselect mfid0
cursor down to mfid1
hit space
type 'a' to use entire disk
type 'q' to quit and save
choose 'none' for boot mgr (leave untouched)
cursor over mfid0
space
(takes you into part. Screen again) 'q' to exit
none for boot mgr
Make sure both drives (mfid0 and mfid1) are checked and tab to ok
Make sure mfid0 is highlighted at the top of the screen, setup the following partitions
/ 512M
swap 6G
/var 256M
/tmp 256M
/usr 5G
/mnt/data1 remaining space
All partitions except / should be setup for soft updates. If not, type 's' to enable for soft updates on all except for / (should look like UFS2+S Y under the Newfs column)
move cursor to mfid1 at the top of the screen
swap 8G (or 4G if there’s a 3rd drive)
/mnt/data2 remaining space
'q' to save and exit
distributions
Choose the following distribudions
- developer (ok to install ports)
- custom -> lib32
exit
media
if you are installing via a cd, no need to enter this menu or change anything. Otherwise, choose ftp to install via ftp. You will be prompted to setup networking. You will need to choose a nic (typically bce0 or bce1). Say no to DHCP and IPv6. Hopefully the public nic cable was installed in bce0 so start with that nic and provide the hostname, (public) IP, netmask, gateway and DNS. When configured, it should start pinging. If it doesn't, have the NOC swap cables. Select any FTP server, usually Main or ftp4.
commit
this usually takes 12mins
during the process you may need to select a new ftp mirror, this is not a problem.
at the conclution of the install you will be prompted to enter the root password (2x) and returned to the configuration menu.
add user
Add user 'user'. Defaults for everything is fine, just remember to enter 'wheel' in the member group field. Do set the password.
Setup timezone
PT
Networking
page down to the bottom and enable '[X]' sshd
If you installed via cd, you will need to visit:
interfaces->bce0->
No IPV6
dhcp=no
Set hostname, IP, DNS, gateway
(i.e. setup the nic as indicated above)
Exit the install and if you installed via CD, take it out and let the machine reboot
Configure OS, kernel, userland, jail
double check the date/time
populate /etc/resolv.conf
echo "nameserver 69.55.225.225\ nameserver 69.55.230.3 nameserver 69.55.229.3" > /etc/resolv.conf
edit /etc/make.conf
echo "WITHOUT_X11=yes \ KERNCONF=jail3 \ BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf
setup bootloader for console, etc
add settings to /boot/loader.conf and /boot.config:
echo "-Dh" >> /boot.config
echo 'console="comconsole,vidconsole" \ boot_multicons="YES" \ boot_serial="YES" \ mfi_linux_load="YES" \ comconsole_speed="115200"' >> /boot/loader.conf
enable login via serial console
turn off all ttyv's except 0 and 1 in /etc/ttys and turn on ttyd0, change type to vt100:
vi /etc/ttys
The changed lines should look like:
ttyv2 "/usr/libexec/getty Pc" cons25 off secure ttyv3 "/usr/libexec/getty Pc" cons25 off secure ttyv4 "/usr/libexec/getty Pc" cons25 off secure ttyv5 "/usr/libexec/getty Pc" cons25 off secure ttyv6 "/usr/libexec/getty Pc" cons25 off secure ttyv7 "/usr/libexec/getty Pc" cons25 off secure # Serial terminals # The 'dialup' keyword identifies dialin lines to login, fingerd etc. ttyu0 "/usr/libexec/getty std.9600" vt100 on secure
Restart init
kill -1 1
At this point you should have a login on console.
To configure serial console access, login to the console server as root and run:
# vi /etc/remote
following examples there, rename port to server's hostname, depending on where and which digi box this server is plugged into. Make sure to get speed right too: 115200
populate hosts
If server is at castle:
echo "10.1.4.3 backup2" >> /etc/hosts echo "10.1.4.8 backup1" >> /etc/hosts
If server is at i2b:
echo "69.55.230.10 backup2" >> /etc/hosts echo "10.1.2.3 backup3" >> /etc/hosts echo "69.55.230.11 backup1" >> /etc/hosts
create ssh key, upload to backup servers
cd ssh-keygen -t dsa -b 1024
(default location, leave password blank)
If server is at castle:
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys' cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'
If server is at i2b:
cat /root/.ssh/id_dsa.pub | ssh backup3 'cat - >> /root/.ssh/authorized_keys' cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys' cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'
confirm that you can ssh to backup2 and backup1 without getting a login prompt
ssh backup2 hostname ssh backup1 hostname ssh backup3 hostname
create & populate binaries/scripts dirs
mkdir -p /usr/local/jail/bin mkdir -p /usr/local/jail/rc.d mkdir -p /usr/local/jail/template/ mkdir /mnt/data1 mkdir /mnt/data2 scp backup2:"/mnt/data4/bin/freebsd8.x/*" /usr/local/jail/bin cd /usr/local/jail/rc.d/ touch quad1 touch deprecated chmod +x * cd /usr/local/jail/bin ln -s /usr/local/jail/rc.d/quad1 quad1 ln -s /usr/local/jail/bin/jailmake_md jailmake ln -s /usr/local/jail/bin/js_md js ln -s /usr/local/jail/bin/canceljail_md canceljail ln -s /usr/local/jail/bin/jailmakeempty_md jailmakeempty ln -s /usr/local/jail/bin/postboot_md postboot ln -s /usr/local/jail/bin/preboot_md preboot ln -s /usr/local/jail/bin/startjail_md startjail ln -s /usr/local/jail/bin/stopjail_md stopjail rehash
edit root's path and login script
vi /root/.cshrc
Change alias entries (add G):
alias la ls -aG alias lf ls -FAG alias ll ls -lAG alias ls ls -AG alias mbm mb mount alias mbu mb umount alias cjb cd /usr/local/jail/bin alias cd1 cd /mnt/data1 alias cd2 cd /mnt/data2 alias cd3 cd /mnt/data3 alias jtop jtop lj alias j jobs
add to path be careful to leave a space after bin and make sure the wrapping isn't broken:
/usr/local/jail/bin
alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "
at the bottom of the file add:
set sshtty=`who am i|awk '{print $2}'`
/usr/sbin/rtprio 3 -`psj | grep $sshtty | awk '{print $2}'`
set shortty=`who am i | awk '{print $2}' | sed -E 's/.*(..)$/\1/'`
foreach x (`psj | grep sh | grep $shortty | awk '{print $2}'`)
/usr/sbin/rtprio 2 -$x
end
Make the new settings active in current shell:
source /root/.cshrc
install cvsup
cd /usr/ports/net/cvsup-without-gui make install clean; rehash; mail -s 'cvs installed' support@johncompanies.com < /dev/null
stand by for gettext options (use defaults). this process takes approx 22mins- hence the email/page notice above.
get latest sources for this release
cd /usr/src echo "*default host=cvsup4.freebsd.org\ *default base=/usr\ *default prefix=/usr\ *default release=cvs tag=RELENG_8_3\ *default delete use-rel-suffix\ *default compress\ src-all" > sup
If you need to run stable (cause release is broken or some other reason) make the sup file look like:
echo "*default host=cvsup4.freebsd.org\ *default base=/usr\ *default prefix=/usr\ *default release=cvs tag=RELENG_8\ *default delete use-rel-suffix\ *default compress\ src-all" > sup
cvsup sup ; mail -s 'cvs sup done' support@johncompanies.com < /dev/null
time varies, 10-20mins
configure new kernel
Pull down the kernel config we are using for this distribution. In this case we use an 8.2 kernel config on 8.3, which is valid. The local file should be the same name as host- jail3 in this example
cd /usr/src/sys/amd64/conf scp backup2:/mnt/data4/build/freebsd/kern_config-8.2-amd64 ./jail3
edit the kernel config and change ident to be the name of the jail:
vi jail3 ident jail3
Optional, edit /sys/conf/newvers.sh to add –jc2 to the end of the BRANCH string (RELEASE-jc2)
vi /sys/conf/newvers.sh
notes on kernel configuring: http://www.FreeBSD.org/doc/en_US.ISO8859-1/books/handbook/kernelconfig-config.html
install patches
The various patches are in /mnt/data4/build/freebsd/patches on backup2. There are dirs for each version. Not all dirs are populated, but patches for later versions work on older ones unless there is a new patch in the older dir.
there are no patches we use for 8.x, but here would be the commands:
cd /usr/src scp backup2:"/mnt/data4/build/freebsd/patches/8.0/*" .
Apply patches, i.e. the jls-patch:
patch -l < jls-patch
build, install kernel and world
Rename current generic kernel so it will always be available to boot from. Save room by removing non-needed kernel modules:
cd /boot mv kernel kernel.GENERIC cd kernel.GENERIC mkdir hold mv mfi_linux.ko hold/ mv linux.ko hold/ mv linprocfs.ko hold/ mv linsysfs.ko hold/ mv geom_vinum.ko hold/ mv geom_concat.ko hold/ mv zfs.* hold/ mv opensolaris* hold/ rm *.ko rm *.symbols mv hold/* . rmdir hold/ cd /usr/src make buildkernel installkernel make buildworld ; mail -s 'buildworld done' dave.boodman@vtext.com < /dev/null
~38mins
make installworld
~34min
mergemaster -i
You will be prompted to merge, replace or ignore files changed by the src update. In most cases you can delete the temp (new) files.
ONLY if this will be a zfs system (not currently used in 8.x):
cd /sys/modules/zfs make make install cd /sys/modules/opensolaris make make install
populate devfs ruleset
scp backup2:/mnt/data4/build/freebsd/devfs.rules.8x /etc/devfs.rules
populate /etc/rc.conf with IPs and service settings
vi /etc/rc.conf kern_securelevel_enable="NO" portmap_enable="NO" sendmail_enable="NO" usbd_enable="YES" nfs_client_enable="YES" nfs_reserved_port_only="YES" inetd_enable="YES" inetd_flags="-wW -a 10.1.2.103" devfs_system_ruleset="devfsrules_show_all" ifconfig_bce1="inet 10.1.2.103 netmask 255.255.255.0" ifconfig_bce0="inet 69.55.229.7 netmask 255.255.255.0" #ifconfig_bce0_alias0="inet 69.55.2xx.xx netmask 255.255.255.0" fsck_y_enable="YES" background_fsck="NO" #rc_mfi_raid_tty_log="YES" #zfs_enable="YES"
Modify IPs, hostname, gateway for this box.
make sure sysctls are set and preserved upon boot
echo "kern.consmute=0\ kern.ipc.shm_use_phys=1\ kern.ipc.shmall=131070\ kern.ipc.shmmax=134217728\ net.inet.tcp.syncookies=0\ kern.maxfiles=32768\ kern.fallback_elf_brand=3\ kern.maxprocperuid=4000\ security.jail.sysvipc_allowed=1\ security.jail.allow_raw_sockets=1\ security.jail.socket_unixiproute_only=1\ security.jail.chflags_allowed=0\ dev.amr.0.allow_volume_configure=1\ compat.linux.osrelease=2.6.12\ vm.pmap.shpgperproc=500\ security.bsd.unprivileged_read_msgbuf=0\ kern.maxvnodes=400000" >> /etc/sysctl.conf
Tuning note: watch vfs.numvnodes while the server is live to get guidance on where to set maxvnodes
mount procfs
echo "proc /proc procfs rw 0 0" >> /etc/fstab echo "linprocfs /usr/compat/linux/proc linprocfs rw 0 0" >> /etc/fstab echo "linsysfs /usr/compat/linux/sys linsysfs rw 0 0" >> /etc/fstab mkdir -p /usr/compat/linux/proc mkdir -p /usr/compat/linux/sys
enable noatime option
data1 and data2 should look something like (add ',noatime' after 'rw'):
/dev/mfid0s1g /mnt/data1 ufs rw,noatime 2 2 /dev/mfid1s1d /mnt/data2 ufs rw,noatime 2 2
reboot. Confirm new kernel is loaded, devfs in place
uname -a
Check devfs rules
devfs rule showsets devfs rule -s 3 show
Should see:
# devfs rule showsets 1 2 3 4 # devfs rule -s 3 show 100 include 1 207 path pts* unhide 217 path fd unhide 218 path fd/* unhide 300 path ttyp* unhide 301 path ttyq* unhide 302 path ttyr* unhide 303 path ttys* unhide 304 path ttyP* unhide 305 path ttyQ* unhide 306 path ttyR* unhide 307 path ttyS* unhide 400 path null unhide 500 path zero unhide 600 path random unhide 610 path urandom unhide 700 path mem unhide 710 path kmem unhide 810 path mdctl unhide 900 path stdin unhide 910 path stdout unhide 920 path stderr unhide
update ports
cd /usr/ports echo "*default host=cvsup4.FreeBSD.org\ *default base=/usr\ *default prefix=/usr\ *default release=cvs tag=RELENG_8_3\ *default delete use-rel-suffix\ *default compress\ ports-all tag=." > sup cvsup sup; mail -s 'cvs sup ports done' support@johncompanies.com < /dev/null
~24mins
Install raid mgmt tool
Perc5/i, 6/i
Pull over cli from previous system (jail9)
scp /usr/local/sbin/mega* root@10.1.4.109:/usr/local/sbin/ scp /usr/local/libexec/MegaCli root@10.1.4.109:/usr/local/libexec/MegaCli
These are linux-based tools. This will require linux base which we would normally install from ports, but since it's failing lately, we just pull in the compat libraries from another system:
rsync -aSHv --exclude=proc --exclude=sys 10.1.4.109:/usr/compat/linux/ /usr/compat/linux/
Test:
rehash; megacli ldinfo lall a0
DEPRECATED: Assuming it worked, here's how we used to install linux_base:
cd /usr/ports/emulators/linux_base-fc4 make install clean
(didnt succeed due to libtool requirement)
cd /usr/ports/distfiles fetch http://www.lsi.com/DistributionSystem/AssetDocument/support/downloads/megaraid/miscellaneous/linux/2.00.15_Linux_MegaCLI.zip cd /usr/ports/sysutils/linux-megacli make install clean
(also failed due to libtool)
2850 PERC 4e/Di- no linux
cd /usr/ports/distfiles/ fetch http://backup01.best-hosting.ru/pub/FreeBSD/ports/distfiles/dr_freebsd_1.51.zip cd /usr/ports/sysutils/megarc make install clean megarc -dispCfg -a0
install rsync from ports
cd /usr/ports/net/rsync make install clean
choose default options
configure inetd to respond to mrtg load queries
echo "load stream tcp nowait user /usr/local/jail/bin/load.pl load.pl" >> /etc/inetd.conf echo "load 12384/tcp" >> /etc/services
install perl
cd /usr/ports/lang/perl5.12 make install clean
choose defaults
install bb client
Compiling from source on AMD64 will not work. So, we use a linux-compiled version and rely on linux compat. Per above, linux compat won't install on 8.x - libtool 2.4 need. So, instead we copy(ed) over linux:
rsync -aSHv --exclude=proc --exclude=sys 10.1.4.108:/usr/compat/linux/ /usr/compat/linux/
adduser
Output/response:
Username: bb Full name: bb Uid (Leave empty for default): 1984 Login group [bb]: Login group is bb. Invite bb into other groups? []: Login class [default]: Shell (sh csh tcsh nologin) [sh]: Home directory [/home/bb]: Use password-based authentication? [yes]: Use an empty password? (yes/no) [no]: Use a random password? (yes/no) [no]: yes Lock out the account after creation? [no]: Username : bb Password : <random> Full Name : bb Uid : 1984 Class : Groups : bb Home : /home/bb Shell : /bin/sh Locked : no OK? (yes/no): yes
cd /usr/home/bb scp backup2:/mnt/data4/build/bb/bb-freebsd_linuxcompat.tgz . tar xzf bb-freebsd_linuxcompat.tgz
edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \ 10.1.4.103 jail3.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
Edit for machine name and private IP.
if this machine is at i2b:
echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \ 69.55.229.7 jail3.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
vi /home/bb/bbc1.9e-btf/ext/openfiles MACHINE="jail3,johncompanies,com" # HAS TO BE IN A,B,C FORM
Edit for machine name.
cd /usr/home/bb/bbc1.9e-btf/etc ./bbchkcfg.sh (y to questions) ./bbchkhosts.sh (ignore ssh errors) cd ../.. chown -R bb . su bb cd cd bbc1.9e-btf ./runbb.sh start more BBOUT (look for errors) exit
Put in script to start bb @ boot:
echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh chmod +x /usr/local/etc/rc.d/bb.sh
remove reserve space
cd umount /mnt/data1 umount /mnt/data2 tunefs -m 0 /mnt/data1 tunefs -m 0 /mnt/data2 mount -a
setup rdate
cd /usr/ports/sysutils/rdate make install clean
crontab -e 0 0 * * * /usr/local/sbin/rdate -s utcnist.colorado.edu
/usr/local/sbin/rdate -s utcnist.colorado.edu
We used to use ntpd, however it listens on jail IPs which is a security risk. So we stopped. Here's the old instructions: install new ntp from ports
/usr/ports/net/ntp make install clean
echo "server 10.1.4.5" > /etc/ntp.conf /usr/sbin/ntpd -p /var/run/ntpd.pid sleep 2; ntpq -p
(confirm it’s able to reach our time server)
make a data partition
IF you didn't format the data partition during sysinstall:
Create a g partition on 2nd mirror – bsdlabel no longer works (below shows d partition made with sysinstall):
jail8 /usr/home/bb# gpart show
=> 63 285474735 mfid0 MBR (136G)
63 285458922 1 freebsd [active] (136G)
285458985 15813 - free - (7.7M)
=> 0 285458922 mfid0s1 BSD (136G)
0 524288 1 freebsd-ufs (256M)
524288 12582912 2 freebsd-swap (6.0G)
13107200 524288 4 freebsd-ufs (256M)
13631488 524288 5 freebsd-ufs (256M)
14155776 8388608 6 freebsd-ufs (4.0G)
22544384 262914538 7 freebsd-ufs (125G)
=> 63 584843175 mfid1 MBR (279G)
63 584830197 1 freebsd [active] (279G)
584830260 12978 - free - (6.3M)
=> 0 584830197 mfid1s1 BSD (279G)
0 16777216 2 freebsd-swap (8.0G)
16777216 568052981 4 freebsd-ufs (271G)
jail8 /usr/home/bb# gpart show mfid1s1
=> 0 584830197 mfid1s1 BSD (279G)
0 16777216 2 freebsd-swap (8.0G)
16777216 568052981 4 freebsd-ufs (271G)
# gpart list mfid1s1
Geom name: mfid1s1
fwheads: 255
fwsectors: 63
last: 584830196
first: 0
entries: 8
scheme: BSD
Providers:
1. Name: mfid1s1b
Mediasize: 8589934592 (8.0G)
Sectorsize: 512
Mode: r1w1e0
rawtype: 1
length: 8589934592
offset: 0
type: freebsd-swap
index: 2
end: 16777215
start: 0
2. Name: mfid1s1d
Mediasize: 290843126272 (271G)
Sectorsize: 512
Mode: r0w0e0
rawtype: 7
length: 290843126272
offset: 8589934592
type: freebsd-ufs
index: 4
end: 584830196
start: 16777216
Consumers:
1. Name: mfid1s1
Mediasize: 299433060864 (279G)
Sectorsize: 512
Mode: r1w1e1
# gpart delete -i 4 mfid1s1
mfid1s1d deleted
jail8 /usr/home/bb# gpart list mfid1s1
Geom name: mfid1s1
fwheads: 255
fwsectors: 63
last: 584830196
first: 0
entries: 8
scheme: BSD
Providers:
1. Name: mfid1s1b
Mediasize: 8589934592 (8.0G)
Sectorsize: 512
Mode: r1w1e0
rawtype: 1
length: 8589934592
offset: 0
type: freebsd-swap
index: 2
end: 16777215
start: 0
Consumers:
1. Name: mfid1s1
Mediasize: 299433060864 (279G)
Sectorsize: 512
Mode: r1w1e1
# gpart show mfid1s1
=> 0 584830197 mfid1s1 BSD (279G)
0 16777216 2 freebsd-swap (8.0G)
16777216 568052981 - free - (271G)
# gpart add -t freebsd-ufs -i 7 mfid1s1
mfid1s1g added
# gpart show mfid1s1
=> 0 584830197 mfid1s1 BSD (279G)
0 16777216 2 freebsd-swap (8.0G)
16777216 568052981 7 freebsd-ufs (271G)
Here's how we USED to do it with bsdlabel:
bsdlabel -e /dev/mfid0s1
given:
# /dev/aacd0s1: 8 partitions: # size offset fstype [fsize bsize bps/cpg] a: 262144 0 4.2BSD 2048 16384 16392 b: 4194304 262144 swap c: 143363997 0 unused 0 0 # "raw" part, don't edit d: 524288 4456448 4.2BSD 2048 16384 32776 e: 524288 4980736 4.2BSD 2048 16384 32776 f: 6291456 5505024 4.2BSD 2048 16384 28552
new offset = 6291456 + 5505024 = 11796480
new size is size for 'c' partition minus the new start from above
143363997 - 11796480 = 131567517
So:
g: 131567517 11796480 unused 0 0
create the jail template
cd /usr/ports/sysutils/jailutils make install clean
Create an md device to hold the jail:
touch /mnt/data1/jail-template20g mdconfig -a -t vnode -s 20g -f /mnt/data1/jail-template20g -u 0 newfs -O 1 /dev/md0 mkdir /mnt/data1/jail-DIR mount /dev/md0 /mnt/data1/jail-DIR
Build world into the jail:
cd /usr/src make world DESTDIR=/mnt/data1/jail-DIR; taskdone
~1hr
Make /etc into the jail, mount dev, copy in jkill:
cd etc make distribution DESTDIR=/mnt/data1/jail-DIR mount -t devfs devfs /mnt/data1/jail-DIR/dev devfs -m /mnt/data1/jail-DIR/dev rule -s 3 applyset cd /mnt/data1/jail-DIR ln -sf dev/null kernel cp /usr/local/sbin/jkill /mnt/data1/jail-DIR/sbin
Enter into jail to do configuration:
jail /mnt/data1/jail-DIR testhostname 192.168.11.100 /bin/sh csh
Create fstab:
touch /etc/fstab echo 'network_interfaces=""\ hostname="newsystem"\ kern_securelevel_enable="NO"\ sendmail_enable="YES"\ sshd_enable="YES"' > /etc/rc.conf echo "nameserver 69.55.225.225\ nameserver 69.55.230.3" >> /etc/resolv.conf
Edit crontab:
vi /etc/crontab
remove the adjkerntz lines comment out periodic’s and put this line above them:
# IF YOU UNCOMMENT THESE, PLEASE ADJUST THEIR RUN TIME
rm -rf /etc/periodic/daily/400.status-disks
check and remove any crap in /tmp
vi /etc/periodic/security/100.chksetuid
replace: MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`
with: MP='/' (use single quotes)
mkdir -p /usr/compat/linux/dev
Add account for user. Output/response:
adduser Username: user Full name: user Uid (Leave empty for default): Login group [user]: Login group is user. Invite user into other groups? []: wheel Login class [default]: Shell (sh csh tcsh nologin) [sh]: Home directory [/home/user]: Home directory permissions (Leave empty for default): Use password-based authentication? [yes]: Use an empty password? (yes/no) [no]: Use a random password? (yes/no) [no]: y Lock out the account after creation? [no]: Username : user Password : <random> Full Name : user Uid : 1001 Class : Groups : user Home : /home/user Home Mode : Shell : /bin/sh Locked : no OK? (yes/no): y adduser: INFO: Successfully added (user) to the user database. adduser: INFO: Password for (user) is: 901gmYjO Add another user? (yes/no): n Goodbye!
set TERM:
vi /usr/home/user/.profile TERM=vt100; export TERM
Set time zone to PT:
tzsetup
Reload aliases:
newaliases
Replace reboot/halt:
rm /sbin/halt /sbin/reboot ln /sbin/jkill /sbin/halt ln /sbin/jkill /sbin/reboot
Redirect console output, comment out console and move to /var/log/messages:
vi /etc/syslog.conf #*.err;kern.warning;auth.notice;mail.crit /dev/console *.err;kern.warning;auth.notice;mail.crit /var/log/messages
exit exit
Lib32 compat library:
cd libexec chflags noschg ld-elf32.so.1 chflags noschg ld-elf.so.1 mv ld-elf32.so.1 ld-elf32.so.1-orig ln ld-elf.so.1 ld-elf32.so.1 chflags schg ld-elf.so.1 chflags schg ld-elf32.so.1
Replace traceroute:
mv /mnt/data1/jail-DIR/usr/sbin/traceroute /mnt/data1/jail-DIR/usr/sbin/_traceroute echo '#\!/bin/sh\ /usr/sbin/_traceroute -i bce0 $1' >> /mnt/data1/jail-DIR/usr/sbin/traceroute chmod +x /mnt/data1/jail-DIR/usr/sbin/traceroute
Modify 'bce0' to reflect whichever nic is public on this hardware.
Clean out ports before copying into the jail (~30mins):
cd /usr/ports make -DNOCLEANDEPENDS clean rm -fr /usr/ports/distfiles/* cp -r /usr/ports /mnt/data1/jail-DIR/usr
rm /mnt/data1/jail-DIR/root/.history
Umount the jail and dump it:
cd / umount /mnt/data1/jail-DIR/dev dump -0a -f /usr/local/jail/template/template /dev/md0 umount /dev/md0 rmdir /mnt/data1/jail-DIR mdconfig -d -u 0
setup backups
echo '#\!/bin/sh\ backupdir=/data/jail3\ server=backup1\ \ ## ENTRY /etc\ ## ENTRY /usr/local/etc\ ## ENTRY /usr/local/jail\ ## ENTRY /root/logs' > /usr/local/jail/bin/backup.config
Edit to reflect backup server and jail hostname
On backup server, setup backup dirs:
ssh backup1 mkdir -p /data/jail3/0
On backup server, setup backup dirs:
backup1# vi /usr/local/sbin/snapshot_rotate
Edit /usr/local/jail/bin/backup.xxx to use the right drives and copy to /usr/local/jail/bin/backup
vi /usr/local/jail/bin/backup.md
adjust df so it includes all relevant drives, currently df > /etc/df.bak is fine. Also, make sure the binary/script source is pulling from the right dir on backup2:
/usr/local/bin/rsync -a backup2:/mnt/data4/bin/freebsd8.x/ /usr/local/jail/bin/
cp /usr/local/jail/bin/backup.md /usr/local/jail/bin/backup
create /root/logs
mkdir /root/logs
edit sshd_config
vi /etc/ssh/sshd_config ListenAddress 69.55.229.7 ListenAddress 10.1.2.103
Adjust to pub/private IP for jail.
Restart sshd:
kill -1 `cat /var/run/sshd.pid`
add crontab entries
crontab -e 5 0 * * * /usr/local/jail/bin/backup 1 0 1 * * /usr/local/jail/bin/ipfwreset 0 18 * * * /usr/local/jail/bin/ipfwbackup 4,9,14,19,24,29,34,39,44,49,55,59 * * * * /usr/local/jail/bin/trafstats 0 0,6,12,18 * * * /usr/local/jail/bin/sync_jail_names
On 2950:
*/5 * * * * /usr/local/jail/bin/perc5iraidchk
On 3ware-based servers:
0 0 * * * /usr/local/jail/bin/3wraidchk
Reboot notify script
ln -s /usr/local/jail/bin/notify.sh /usr/local/etc/rc.d/notify.sh
patch jail against starting jails with rtprio
mv /usr/sbin/jail /usr/sbin/jail_ echo '#\!/bin/sh\ /usr/sbin/rtprio -t /usr/sbin/jail_ $*' > /usr/sbin/jail chmod +x /usr/sbin/jail
make sure mail works
If there are map errors:
cd /etc/mail; make maps
recover space on /usr (optional)
rm -fr /usr/obj
wrapper for jps
mv /usr/local/sbin/jps /usr/local/sbin/jps_
wrapper for jls
mv /usr/sbin/jls /usr/sbin/jls_
wrapper for jexec
mv /usr/sbin/jexec /usr/sbin/jexec_
install jtop
cd /usr/ports/sysutils/jtop make install clean
block jails from reaching private net
echo 'ipfw add 1 deny ip from 69.55.224.0/20 to 10.1.4.0/24' > /usr/local/etc/rc.d/ipfw.sh chmod 0700 /usr/local/etc/rc.d/ipfw.sh
add to management infrastructure
add to management db
tables: jc.ref_machines and jc.ref_templates
on jail run:
uname -r
Which shows something like:
8.3-RELEASE-p2
Insert into db:
insert into ref_machines values (null,'jail3','mx3.johncompanies.com',0,'f8');
select machine_id from ref_machines where host='jail3';
+------------+
| machine_id |
+------------+
| 35 |
+------------+
insert into ref_templates values ('','8.3-RELEASE-jc2',35,'FreeBSD 8.3',1);
add to bb server
vi /usr/home/bb/bbsrc/bb1.9i-btf/etc/bb-hosts 10.1.4.109 jail9.johncompanies.com # ssh
In the case of an i2b server, use real ip:
69.55.229.7 jail3.johncompanies.com # ssh
su bb cd bbsrc/bb/runbb.sh restart ; exit
Update backupgraph
vi /usr/local/www/mgmt/cgi/backupgraph.pl
(add hostname)
Update load mrtg
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg
(add new entry to file following existing format)
ns1c
fwd and reverse lookups:
vr johncompanies.com
vi internal.johncompanies.com rndc reload johncompanies.com IN private
ptr 69.55.227.x
wiki
add to server/cabinet map
firewall
add an outside blocking rule to the firewall, so this machine can only be reached from inside the firewall. Follow example already in firewall jail17 is:
00117 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.228.200
00117 deny ip from any to 69.55.228.200
jail19 would be 00119...
ipfw add 00109 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 } to 69.55.232.3
ipfw add 00109 deny ip from any to 69.55.232.3
select some customers for castle probe map
4.11
4.11
Last updated 2006-1-26
All time extimates below assume disks aren’t scrubbing
1. make sure bios is setup for bios console redirect
Supermicro:
Console redirection:
Com port addr: on-board COM A
Baud: 38400
Console type: vt100
Flow control: none
Console connection: direct
Continue cr after post: off
2450:
Make sure running bios A09
Console Redirection: VT100/VT220
2. assuming mirrors (or at least disks) created (if not, refer to this), boot to disk 1 of 4.11
skip kernel config (enter)
custom install
partition
move cursor to aacd0, hit space (takes you to partition map screen)
a for entire disk
q to quit and save
standard mbr (no boot manager)
space to unselect aacd0
cursor over aacd1
space
a for entire disk
q to quit and save
none (leave untouched)
cursor over aacd0
space
(takes you into part. Screen again) q to exit
none
Make sure both are checked and tab to ok
label
Make sure aacd0 is highlighted
a to start with defaults
john likes:
/1g
/var 256m
Glenn likes:
/128
/var 256
/usr 3g
/tmp 256
We do:
delete everything
/ 128M
swap 4G
/var 256M
/tmp 256M
/usr 3G
/mnt/data1 remaining space
Make sure to toggle S for soft updates on all (should look like UFS+S Y under the Newfs column)
Set the noatime option
move cursor to aacd1
swap 4G
/mnt/data2 remaining space
Set the noatime option
q to save and exit
distributions
developer
yes to install ports
exit
media
cd
commit
yes
(2450: 16mins, supermicro: 11mins)
yes to "visit general config"
Set root pwd
Add user ‘user’ member group is wheel, set password
Set tz
Networking->interfaces->Fxp0
No IPV6
dhcp=yes
Set hostname & domain
Startup services:
Disble usbd
exit...
exit install
yes
take the cd out and let the machine reboot
3. put some temp settings in /etc/rc.conf:
usbd_enable="NO"
sendmail_enable="NO"
4. reboot
5. double check the date/time
6. edit /etc/make.conf (only add the console speed line if this is a supermicro capable of outputting BIOS redirect at that speed – 2450’s can only do 9600)
cat >> /etc/make.conf
WITHOUT_X11=yes
KERNCONF=jail18
BOOT_COMCONSOLE_SPEED=38400
7. install cvsup
cd /usr/ports/net/cvsup-without-gui
make install clean (stay close for gettext options, 2450: 21mins, supermicro: 14mins)
rehash
8. get latest sources for this release:
cd /usr/src
cat > sup
*default host=cvsup4.freebsd.org
*default base=/usr
*default prefix=/usr
*default release=cvs tag=RELENG_4_11
*default delete use-rel-suffix
*default compress
src-all
cvsup sup (2450, 4.10: 13mins, supermicro, 4.11: 11mins)
9. populate hosts
cat >> /etc/hosts
10.1.4.3 backup2
10. put key in authorized_keys on backup2
ssh-keygen -t dsa -b 1024 (default location, leave password blank)
scp /root/.ssh/id_dsa.pub user@backup2:/tmp/jail18pub
on backup2:
cat /tmp/jail18pub >> /root/.ssh/authorized_keys
confirm that you can ssh to backup2 without getting a login prompt
11. configure new kernel. Get config from similar machine or there may be a master copy somewhere under /mnt/data4/build (name the kernel config the same as the jail, ex jail4):
cd /usr/src/sys/i386/conf
scp backup2:/mnt/data4/build/freebsd/kern_config-4.11 ./jail18
edit the kernel config and change ident to be the name of the jail:
ident jail4
IMPORTANT CUSTOMIZATION:
for machines with >4G RAM, add to the config:
options PAE
for supermicro mobo’s with broadcom nics, add to the config:
device bge # Broadcom BCM570x (``Tigon III'')
for machines where lots of postgres might be running, change SHMMAXPGS:
options SHMMAXPGS=40960
edit /sys/conf/newvers.sh to add –jc2 to the end of the BRANCH string (RELEASE-p9-jc2)
12. bring over patches from backup2
The various patches are in /mnt/data4/build/freebsd/patches on backup2. There are dirs for each version. Not all dirs are populated, but patches for later versions work on older ones unless there is a new patch in the older dir.
cd /usr/src
scp backup2:"/mnt/data4/build/freebsd/patches/4.11/*" .
jail_proc_patch 100% |*************************************************| 2593 00:00
restore-patch 100% |*************************************************| 3295 00:00
411ps-jail-patch 100% |*************************************************| 2602 00:00
jail_rtprio_patch 100% |*************************************************| 301 00:00
udp-patch 100% |*************************************************| 594 00:00
Apply patches:
patch < 411ps-jail-patch
patch < jail_proc_patch
patch < restore-patch
patch < jail_rtprio_patch
patch < udp-patch
13. build, install kernel and world
make buildworld buildkernel installkernel (2450: 48min, supermicro: 20mins)
(Any compile errors can be looked up in /usr/include/sys/signal.h, other errors, do a rm -R /usr/obj/*)
make installworld (2450: 2min, supermicro: 1mins)
mergemaster -i
(answer no to most of it)
14. reboot. Confirm new kernel is loaded (uname -a)
15. update ports:
cd /usr/ports
cat > sup
*default host=cvsup4.FreeBSD.org
*default base=/usr
*default prefix=/usr
*default release=cvs tag=RELENG_4
*default delete use-rel-suffix
*default compress
ports-all tag=.
cvsup sup (2450: 26mins, supermicro: 26mins)
16. add console="comconsole" to /boot/loader.conf
cat >> /boot/loader.conf
console="comconsole"
17. edit /etc/ttys and turn off all ttyv's except 0 and 1
also turn on ttyd0, change type to vt100:
ttyd0 "/usr/libexec/getty std.9600" vt100 on secure
kill -1 1
on console:
vi /etc/remote
(rename port to jail4 depending on where and which digi plugged into)
test serial console
18. install linux_base:
cd /usr/ports/emulators/linux_base
make install clean (2450: 7min, supermicro: 2mins)
reibuild rpmdb cause we had probs installing aacapps
cd /compat/linux/bin
./rpm --initdb
./rpm --rebuilddb
19. install aacapps-4.1-0.i386.rpm
scp backup2:/mnt/data4/build/freebsd/aacapps-4.1-0.i386.rpm /tmp/.
/compat/linux/bin/rpm -ivh /tmp/aacapps-4.1-0.i386.rpm
cd /dev
sh MAKEDEV aac0
test out /compat/linux/usr/sbin/aaccli
20. edit root's path and login script:
vi /root/.cshrc
add to path: /usr/local/jail/bin /compat/linux/usr/sbin
Change alias entries (add G):
alias la ls -aG
alias lf ls -FAG
alias ll ls -lAG
alias ls ls -AG
alias mbm mb mount
alias mbu mb umount
and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "
at the bottom of the file add:
set sshtty=`who am i|awk '{print $2}'`
/usr/sbin/rtprio 3 -`ps auxwJ | grep $sshtty | awk '{print $2}'`
set shortty=`who am i | awk '{print $2}' | sed -E 's/.*(..)$/\1/'`
foreach x (`ps cauxJ | grep sh | grep $shortty | awk '{print $2}'`)
/usr/sbin/rtprio 2 -$x
end
21. install rsync from ports
cd /usr/ports/net/rsync
make install clean
22. create & populate binaries/scripts dirs
mkdir -p /usr/local/jail/bin
mkdir -p /usr/local/jail/rc.d
scp backup2:"/mnt/data4/bin/freebsd/*" /usr/local/jail/bin
cd /usr/local/jail/rc.d/
touch quad1
touch quad2
touch quad3
touch quad4
touch safe1
touch safe2
touch safe3
touch safe4
chmod +x *
cd /usr/local/jail/bin
ln -s /usr/local/jail/rc.d/quad1 quad1
ln -s /usr/local/jail/rc.d/quad2 quad2
ln -s /usr/local/jail/rc.d/quad3 quad3
ln -s /usr/local/jail/rc.d/quad4 quad4
ln -s /usr/local/jail/rc.d/safe1 safe1
ln -s /usr/local/jail/rc.d/safe2 safe2
ln -s /usr/local/jail/rc.d/safe3 safe3
ln -s /usr/local/jail/rc.d/safe4 safe4
rehash
23. configure inetd to respond to mrtg load queries
cat >> /etc/inetd.conf
load stream tcp nowait user /usr/local/jail/bin/load.pl load.pl
cat >> /etc/services
load 12384/tcp
kill -HUP `cat /var/run/inetd.pid`
24. configure load mrtg, on mail
vi /usr/local/etc/mrtg/mrtg1.cfg
(add new entry to file following existing format)
25. configure bb on mail:
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
10.1.4.104 jail4.johncompanies.com # ssh
su bb
cd /usr/home/bb/bbsrc/bb1.9e-btf
./runbb.sh stop
./runbb.sh start
exit
26. install bb client
adduser -group 1984 -shell /bin/csh -uid 1984 bb
cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd.tar .
tar xvf bb-freebsd.tar
cat > /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
10.1.4.105 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh
10.1.4.118 jail18.johncompanies.com # ssh
vi /home/bb/bbc1.9e-btf/ext/openfiles and change:
MACHINE="jail18,johncompanies,com" # HAS TO BE IN A,B,C FORM
cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
./bbchkhosts.sh (ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd bbc1.9e-btf/
./runbb.sh start
more BBOUT (look for errors)
exit
cat > /usr/local/etc/rc.d/bb.sh
su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"
chmod +x /usr/local/etc/rc.d/bb.sh
27. remove reserve space, enable softupdates (probably already set, so not necessary)
cd
umount /mnt/data1
umount /mnt/data2
tunefs -m 0 /mnt/data1
tunefs -m 0 /mnt/data2
mount -a
28. configure ntp
cat > /etc/ntp.conf
server 10.1.4.105
/usr/sbin/ntpd -p /var/run/ntpd.pid
ntpq -p
(confirm it’s able to reach our time server)
29. mrtg switch graphs
31. fwd and reverse lookups on ns1c
vi johncompanies.com
rr johncompanies.com
vi internal.johncompanies.com
rndc reload johncompanies.com IN private
(edit the PTR too)
32. create all /dev/vn and /dev/pty files in /dev
cat > /tmp/runme.sh
#!/bin/sh
cd /dev
for i in 1 2 3 4 5 6 7 ; do sh MAKEDEV pty$i ; done
for i in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 ; do sh MAKEDEV vn$i ; done
NOTE: there are only 4 lines in the above output - however the lines may be wrapped on your terminal - make sure that there are only four lines in the script you make.
chmod +x /tmp/runme.sh
/tmp/runme.sh
rm /tmp/runme.sh
ls /dev/vn*|wc -l (make sure the output is a high number – larger than 128)
ls /dev/pty*|wc -l (make sure the output is exactly 256)
33. populate /etc/rc.conf with IPs and NFS settings
vi /etc/rc.conf
kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="NO"
xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
sshd_enable="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.4.104"
ifconfig_xl0="inet 10.1.4.104 netmask 255.255.255.0"
ifconfig_fxp0="inet 69.55.228.101 netmask 255.255.255.0"
defaultrouter="69.55.228.1"
ifconfig_fxp0_alias0="inet 69.55.2xx.xx netmask 255.255.255.0"
static_routes="t1 office"
route_t1="-net 10.1.5 10.1.4.2"
route_office="-net 10.1.6 10.1.4.2"
34. make sure sysctls are set and preserved after reboot
cat >> /etc/sysctl.conf
kern.consmute=0
jail.sysvipc_allowed=1
kern.ipc.shm_use_phys=1
kern.ipc.shmall=65535
kern.ipc.shmmax=134217728
net.inet.tcp.syncookies=0
kern.maxfiles=32768
kern.fallback_elf_brand=3
kern.maxprocperuid=4000
jail.max_procs_per_jail: 1026
35. reboot
36. create the jail template
vnconfig -T -S 1g -Z -s labels -c /dev/vn1 /mnt/data1/jail
disklabel -r -w vn1 auto
newfs /dev/vn1c
cd /usr/src
mkdir -p /mnt/data1/jail-DIR
mount /dev/vn1c /mnt/data1/jail-DIR
make world DESTDIR=/mnt/data1/jail-DIR (2450: 45mins, supermicro: 19mins)
cd etc
make distribution DESTDIR=/mnt/data1/jail-DIR -DNO_MAKEDEV_RUN
cd /mnt/data1/jail-DIR/dev
sh MAKEDEV jail
cd /mnt/data1/jail-DIR
ln -sf dev/null kernel
jail /mnt/data1/jail-DIR testhostname 192.168.11.100 /bin/sh
csh
touch /etc/fstab
cat > /etc/rc.conf
portmap_enable="NO"
network_interfaces=""
hostname="newsystem"
kern_securelevel_enable="NO"
sendmail_enable="YES"
sshd_enable="YES"
cat >> /etc/resolv.conf
nameserver 69.55.225.225
nameserver 69.55.230.3
vi /etc/crontab
remove the adjkerntz lines
comment out periodic’s and put this line above them:
# DO NOT UNCOMMENT THESE
rm -rf /etc/periodic/daily/400.status-disks
mv /bin/df /bin/df_
cat > /bin/df
#!/bin/sh
/bin/df_ $* .
chmod +x /bin/df
cat > /sbin/mount
echo `df | tail -1 | awk '{print $1 " on " $6 " (ufs, local)"}'`
check /tmp for crap
vi /etc/periodic/security/100.chksetuid
replace: MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`
with: MP='/'
mkdir -p /usr/compat/linux/dev
adduser (Add account for user)
put user in wheel group
vi /etc/group
vi /usr/home/user/.profile (and add to the file):
TERM=vt100; export TERM
cd /etc
vipw -d .
root:$1$krszPxhk$xkCepSnz3mIikT3vCtJCt0:0:0::0:0:Charlie &:/root:/bin/csh
user:$1$Mx9p5Npk$QdMU6c8YQqp2FW2M3irEh/:1001:1001::0:0:User &:/home/user:/bin/sh
tzsetup
newaliases
chflags schg /dev/*mem
cd /dev
rm console
ln -s null console
exit
exit
cd /usr/ports
make -DNOCLEANDEPENDS clean (2450: 47mins , supermicro: 17mins)
rm -fr /usr/ports/distfiles/*
cp -r /usr/ports /mnt/data1/jail-DIR/usr
cd /mnt/data1/jail-DIR/dev
rm kmem
mknod kmem c 2 1 root:kmem
chmod 640 kmem
rm mem
mknod mem c 2 0 root:kmem
chmod 640 mem
rm /mnt/data1/jail-DIR/root/.history
sh
for i in 1 2 3 4 5 6 7 ; do sh MAKEDEV pty$i ; done
exit
cd /mnt/data1/jail-DIR/usr/compat/linux/dev
mknod null c 2 2
mknod random c 2 3
cd
mkdir /usr/local/jail/template/
dump -0a -f /usr/local/jail/template/411template /dev/vn1
umount /dev/vn1c
vnconfig -u /dev/vn1
rm /mnt/data1/jail
rm -fr /mnt/data1/jail-DIR
37. setup backups
cat > /usr/local/jail/bin/backup.config
#!/bin/sh
backupdir=/mnt/data2/jail18_rsync
## ENTRY /etc
## ENTRY /usr/local/etc
## ENTRY /usr/local/jail
## ENTRY /root/logs
on backup2:
setup backup dirs:
mkdir -p /mnt/data2/jail18_rsync/0_today
mkdir -p /mnt/data2/jail18_rsync/1_yesterday
mkdir -p /mnt/data2/jail18_rsync/2_two_day
add the system to /mnt/data1/bin/rsync_houseclean
on mail:
vi /usr/local/www/mgmt/cgi/backupgraph.pl
(add hostname)
38. mkdir /root/logs
39. edit sshd_config for security
vi /etc/ssh/sshd_config
ListenAddress 69.55.228.101
ListenAddress 10.1.4.104
kill -1 `cat /var/run/sshd.pid`
40. add crontab entries
crontab -e
5 0 * * * /usr/local/jail/bin/backup
1 0 1 * * /usr/local/jail/bin/ipfwreset
0 18 * * * /usr/local/jail/bin/ipfwbackup
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /usr/local/jail/bin/trafstats
55 10,23 * * * /usr/local/jail/bin/trafficwatch.pl
41. Reboot notify script
cat > /usr/local/etc/rc.d/notify.sh
echo "`/bin/hostname` rebooted" | /usr/bin/mail reboot@johncompanies.com
chmod +x /usr/local/etc/rc.d/notify.sh
42. copy jailmake from prev system
scp user@10.1.4.118:/usr/local/jail/bin/jailmake /usr/local/jail/bin
rehash
NOTE: remove df altering code from jailmake since we put the correct df in the template, and make sure path to template file is right
43. add to templates via mgmt system
44. add to server/cabinet map. On mail:
vi /usr/local/www/mgmt/html/cabinetmap.html
45. add an outside blocking rule to the firewall, so this machine can only be reached from inside the firewall. Follow example already in firewall jail17 is:
00117 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.228.2
00117 deny ip from any to 69.55.228.2
jail4 would be 00104...
ipfw add 00118 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.228.140
ipfw add 00118 deny ip from any to 69.55.228.140
46. select customers for probe map
6.1
Last updated 2006-05-09
All time extimates below assume disks aren’t scrubbing
1. make sure bios is setup for bios console redirect
Supermicro:
Console redirection:
Com port addr: on-board COM A
Baud: 38400
Console type: vt100
Flow control: none
Console connection: direct
Continue cr after post: off
2450:
Make sure running bios A09
Console Redirection: VT100/VT220
2. assuming mirrors (or at least disks) created (if not, refer to this), boot to disk 1 of 5.4
skip kernel config (enter)
custom install
partition ->
move cursor to amrd0, hit space (takes you to partition map screen)
a for entire disk
q to quit and save
standard mbr (no boot manager)
space to unselect aacd0
cursor over aacd1
space
a for entire disk
q to quit and save
none (leave untouched)
cursor over aacd0
space
(takes you into part. Screen again) q to exit
none
Make sure both are checked and tab to ok
Label ->
Make sure aacd0 is highlighted
a to start with defaults
john likes:
/1g
/var 256m
Glenn likes:
/128
/var 256
/usr 3g
/tmp 256
We do:
delete everything
/ 128M
swap 2G
/var 256M
/tmp 256M
/usr 3G
/mnt/data1 remaining space
Make sure to toggle S for soft updates on all (should look like UFS2+S Y under the Newfs column)
move cursor to aacd1
swap 2G
/mnt/data2 remaining space
q to save and exit
distributions ->
developer
yes to install ports
exit
media ->
cd
commit ->
yes
(2450: 14mins, supermicro: 12mins)
yes to "visit general config" ->
Set root pwd
Add user ‘user’ member group is wheel, set password
Set tz
Networking->interfaces->Fxp0 ->
No IPV6
dhcp=no
Set hostname & domain
Enable sshd
exit...
exit install ->
yes
take the cd out and let the machine reboot
when it comes back up, enter junk when it asks for key seed
3. double check the date/time
4. edit /etc/make.conf (only add the console speed line if this is a supermicro capable of outputting BIOS redirect at that speed – 2450’s can only do 9600)
echo "WITHOUT_X11=yes \
KERNCONF=jail19 \
BOOT_COMCONSOLE_SPEED=38400" >> /etc/make.conf
5. add console="comconsole" to /boot/loader.conf
echo "console=""comconsole""" >> /boot/loader.conf
6. turn off all ttyv's except 0 and 1 in /etc/ttys
also turn on ttyd0, change type to vt100:
vi /etc/ttys
ttyd0 "/usr/libexec/getty std.9600" vt100 on secure
kill -1 1
on console server:
vi /etc/remote
(rename port to jail18 depending on where and which digi plugged into)
test serial console
7. install cvsup
cd /usr/ports/net/cvsup-without-gui
make install clean; rehash
(stay close for gettext options, 2450: 27mins, supermicro: 17mins)
8. get latest sources for this release:
cd /usr/src
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_1\
*default delete use-rel-suffix\
*default compress\
src-all" > sup
cvsup sup
(2450, ~12mins, supermicro, 27mins)
9. populate hosts
echo "10.1.4.3 backup2" >> /etc/hosts
10. put key in authorized_keys on backup2
cd
ssh-keygen -t dsa -b 1024
(default location, leave password blank)
scp /root/.ssh/id_dsa.pub user@backup2:/tmp/jailkey
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
confirm that you can ssh to backup2 without getting a login prompt
ssh backup2
11. configure new kernel. Get config from similar machine or there may be a master copy somewhere under /mnt/data4/build/freebsd (name the kernel config the same as the jail, ex jail18):
cd /usr/src/sys/i386/conf
scp backup2:/mnt/data4/build/freebsd/kern_config-6.1 ./jail19
edit the kernel config and change ident to be the name of the jail:
vi jail14
ident jail14
edit /sys/conf/newvers.sh to add –jc1 to the end of the BRANCH string (RELEASE-jc1)
vi /sys/conf/newvers.sh
12. bring over patches from backup2
The various patches are in /mnt/data4/build/freebsd/patches on backup2. There are dirs for each version. Not all dirs are populated, but patches for later versions work on older ones unless there is a new patch in the older dir.
cd /usr/src
scp backup2:"/mnt/data4/build/freebsd/patches/6.1/*" .
jail_proc_patch 100% |*************************************************| 2593 00:00
restore-patch 100% |*************************************************| 3295 00:00
54ps-jail-patch 100% |*************************************************| 2602 00:00
jail_rtprio_patch 100% |*************************************************| 301 00:00
udp-patch 100% |*************************************************| 594 00:00
Apply patches:
patch < 54ps-jail-patch
patch < jail_proc_patch
patch < restore-patch
patch < jail_rtprio_patch
patch < udp-patch
13. build, install kernel and world
cd /usr/src
make buildworld buildkernel installkernel
(2450: 1:56min, supermicro::59mins)
make installworld
(2450: 3min, supermicro: 1min)
mergemaster -i
delete /var/tmp/temproot
delete bsnmpd
delete temporary ./etc/hosts
delete temporary ./etc/motd
delete /var/tmp/temproot
14. reboot. Confirm new kernel is loaded
uname –a
15. update ports:
cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_1\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup
cvsup sup
(2450: 18mins, supermicro: 19mins)
18. (only applies if adaptec card installed)
install linux_base:
cd /usr/ports/emulators/linux_base
make install clean
(2450: 7min, supermicro: 3mins)
reibuild rpmdb cause we had probs installing aacapps
cd /compat/linux/bin
./rpm --initdb
./rpm --rebuilddb
install aacapps-4.1-0.i386.rpm
scp backup2:/mnt/data4/build/freebsd/aacapps-4.1-0.i386.rpm /tmp/.
/compat/linux/bin/rpm -ivh --excludepath=/dev /tmp/aacapps-4.1-0.i386.rpm
test out;
/compat/linux/usr/sbin/aaccli
20. edit root's path and login script:
vi /root/.cshrc
add to path:
/usr/local/jail/bin
(if adaptec card installed, also add /compat/linux/usr/sbin)
Change alias entries (add G):
alias la ls -aG
alias lf ls -FAG
alias ll ls -lAG
alias ls ls -AG
alias mbm mb mount
alias mbu mb umount
and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "
at the bottom of the file add:
set sshtty=`who am i|awk '{print $2}'`
/usr/sbin/rtprio 3 -`psj | grep $sshtty | awk '{print $2}'`
set shortty=`who am i | awk '{print $2}' | sed -E 's/.*(..)$/\1/'`
foreach x (`psj | grep sh | grep $shortty | awk '{print $2}'`)
/usr/sbin/rtprio 2 -$x
end
21. install rsync from ports
cd /usr/ports/net/rsync
make install clean
choose default options
21. install perl from ports
PROB NOT NECESSARY – INSTALLED WITH LINUX I THINK
cd /usr/ports/lang/perl5.8/
make install clean; rehash
(supermicro: 5min)
22. create & populate binaries/scripts dirs
mkdir -p /usr/local/jail/bin
mkdir -p /usr/local/jail/rc.d
mkdir /mnt/data1
mkdir /mnt/data2
scp backup2:"/mnt/data4/bin/freebsd6.x/*" /usr/local/jail/bin
cd /usr/local/jail/rc.d/
touch quad1
touch quad2
touch quad3
touch quad4
touch safe1
touch safe2
touch safe3
touch safe4
chmod +x *
cd /usr/local/jail/bin
ln -s /usr/local/jail/rc.d/quad1 quad1
ln -s /usr/local/jail/rc.d/quad2 quad2
ln -s /usr/local/jail/rc.d/quad3 quad3
ln -s /usr/local/jail/rc.d/quad4 quad4
ln -s /usr/local/jail/rc.d/safe1 safe1
ln -s /usr/local/jail/rc.d/safe2 safe2
ln -s /usr/local/jail/rc.d/safe3 safe3
ln -s /usr/local/jail/rc.d/safe4 safe4
rehash
23. configure inetd to respond to mrtg load queries
echo "load stream tcp nowait user /usr/local/jail/bin/load.pl load.pl" >> /etc/inetd.conf
echo "load 12384/tcp" >> /etc/services
26. install bb client
adduser
cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd.tar .
tar xvf bb-freebsd.tar
edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
echo "10.1.4.105 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.4.103 jail3.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
vi /home/bb/bbc1.9e-btf/ext/openfiles
MACHINE="jail19,johncompanies,com" # HAS TO BE IN A,B,C FORM
cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh (y to questions)
./bbchkhosts.sh (ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src
make; make install
cd ..
./runbb.sh start
more BBOUT
(look for errors)
exit
echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh
modify bb-msgtab to look for raid failures
24. configure load mrtg, on mail
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg
(add new entry to file following existing format)
25. configure bb on mail:
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
10.1.4.119 jail19.johncompanies.com # ssh
su bb
cd
bbsrc/bb/runbb.sh restart ; exit
27. remove reserve space, enable softupdates (probably already set, so not necessary)
NOT APPLICABLE IF USING GVINUM
cd
umount /mnt/data1
umount /mnt/data2
tunefs -m 0 /mnt/data1
tunefs -m 0 /mnt/data2
mount -a
28. configure ntp
echo "server 10.1.4.105" > /etc/ntp.conf
/usr/sbin/ntpd -p /var/run/ntpd.pid
sleep 2; ntpq -p
(confirm it’s able to reach our time server)
29. mrtg switch graphs
31. fwd and reverse lookups on ns1c
vr johncompanies.com
vi internal.johncompanies.com
rndc reload johncompanies.com IN private
(edit the PTR too)
33. populate /etc/rc.conf with IPs and NFS settings
vi /etc/rc.conf
kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"
xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
sshd_enable="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.4.119"
devfs_system_ruleset="devfsrules_show_all"
ifconfig_xl0="inet 10.1.4.118 netmask 255.255.255.0"
ifconfig_fxp0="inet 69.55.228.101 netmask 255.255.255.0"
defaultrouter="69.55.228.1"
ifconfig_fxp0_alias0="inet 69.55.2xx.xx netmask 255.255.255.0"
static_routes="t1 office"
route_t1="-net 10.1.5 10.1.4.2"
route_office="-net 10.1.6 10.1.4.2"
gvinum_enable="YES"
fsck_y_enable="YES"
background_fsck="NO"
34. make sure sysctls are set and preserved after reboot
echo "kern.consmute=0\
kern.ipc.shm_use_phys=1\
kern.ipc.shmall=65535\
kern.ipc.shmmax=134217728\
net.inet.tcp.syncookies=0\
kern.maxfiles=32768\
kern.fallback_elf_brand=3\
kern.maxprocperuid=4000\
security.jail.sysvipc_allowed=1\
security.jail.max_procs_per_jail: 1026\
security.jail.allow_raw_sockets=1\
security.jail.socket_unixiproute_only=1\
security.jail.chflags_allowed=0" >> /etc/sysctl.conf
35. mount procfs
echo "proc /proc procfs rw 0 0" >> /etc/fstab
36. enable noatime option
NOT APPLICABLE IF RUNNING GVINUM
data1 and data2 should look something like:
/dev/amrd0s1g /mnt/data1 ufs rw,noatime 2 2
36. populate devfs ruleset
scp backup2:/mnt/data4/build/freebsd/devfs.rules /etc
35. reboot
Check rules:
devfs rule showsets
devfs rule -s 3 show
36. create gvinum volumes
Make a g partition:
bsdlabel -e /dev/aacd0s1
given:
# /dev/aacd0s1:
8 partitions:
# size offset fstype [fsize bsize bps/cpg]
a: 262144 0 4.2BSD 2048 16384 16392
b: 4194304 262144 swap
c: 143363997 0 unused 0 0 # "raw" part, don't edit
d: 524288 4456448 4.2BSD 2048 16384 32776
e: 524288 4980736 4.2BSD 2048 16384 32776
f: 6291456 5505024 4.2BSD 2048 16384 28552
new offset = 6291456 + 5505024 = 11796480
new size is size for 'c' partition minus the new start from above
143363997 - 11796480 = 131567517
So:
g: 131567517 11796480 unused 0 0
For a 73G drive (after OS), we can fit 31 2G volumes so:
echo 'drive data1 device /dev/aacd0s1g' > /tmp/cgv
sh
for f in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31;\
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 2g drive data1' >> /tmp/cgv; done; exit
gvinum create /tmp/cgv
For a 146G drive (-4G for swap), we can fit 66 2G volumes so:
echo 'drive data2 device /dev/aacd1s1g' > /tmp/cgv
sh
for f in 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97;\
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 2g drive data2' >> /tmp/cgv; done; exit
gvinum create /tmp/cgv
For 3rd 73G drive (after 2G swap), we can fit 33 2G volumes so:
Label should be:
# /dev/aacd2s1:
8 partitions:
# size offset fstype [fsize bsize bps/cpg]
c: 143299737 0 unused 0 0 # "raw" part, don't edit
g: 143299721 16 unused 0 0
echo 'drive data3 device /dev/aacd2s1g' > /tmp/cgv
sh
for f in 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131;\
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 2g drive data3' >> /tmp/cgv; done
gvinum create /tmp/cgv
For a 2nd 73G drive (after 2G swap), we can fit 33 2G volumes so:
echo 'drive data2 device /dev/aacd1s1g' > /tmp/cgv
sh
for f in 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64;\
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 2g drive data2' >> /tmp/cgv; done; exit
gvinum create /tmp/cgv
36. create the jail template
mkdir /mnt/jail
newfs /dev/gvinum/v1
mount /dev/gvinum/v1 /mnt/jail
cd /usr/src
make clean
rm -fr /usr/obj/
make world DESTDIR=/mnt/jail
(2450: 2:28mins, supermicro: 55mins)
cd etc
make distribution DESTDIR=/mnt/jail
mount_devfs devfs /mnt/jail/dev
devfs -m /mnt/jail/dev rule -s 3 applyset
cd /mnt/jail
ln -sf dev/null kernel
jail /mnt/jail testhostname 192.168.11.100 /bin/sh
csh
touch /etc/fstab
echo 'network_interfaces=""\
hostname="newsystem"\
kern_securelevel_enable="NO"\
sendmail_enable="YES"\
sshd_enable="YES"' > /etc/rc.conf
echo "nameserver 69.55.225.225\
nameserver 69.55.230.3" >> /etc/resolv.conf
vi /etc/crontab
remove the adjkerntz lines
comment out periodic’s and put this line above them:
# DO NOT UNCOMMENT THESE
rm -rf /etc/periodic/daily/400.status-disks
check /tmp for crap
vi /etc/periodic/security/100.chksetuid
replace: MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`
with: MP='/' (use single quotes)
mkdir -p /usr/compat/linux/dev
adduser (Add account for user, make sure in wheel group)
user/root passwd: 8ico2987
Set root password
vi /usr/home/user/.profile (and add to the file):
TERM=vt100; export TERM
tzsetup
newaliases
#cd /dev
#rm console
#ln -s null console
exit
exit
cd /usr/ports
make -DNOCLEANDEPENDS clean
(2450: 15mins , supermicro: 29mins)
rm -fr /usr/ports/distfiles/*
cp -r /usr/ports /mnt/jail/usr (2450: 2:00 mins , supermicro: 15mins)
rm /mnt/jail/root/.history
cd
mkdir /usr/local/jail/template/
dump -0a -f /usr/local/jail/template/61template /dev/gvinum/v1
umount /mnt/jail/dev
umount /dev/gvinum/v1
rm -fr /mnt/jail
37. setup backups
echo '#\!/bin/sh\
backupdir=/mnt/data3jail3_rsync\
\
## ENTRY /etc\
## ENTRY /usr/local/etc\
## ENTRY /usr/local/jail\
## ENTRY /root/logs' > /usr/local/jail/bin/backup.config
on backup2:
setup backup dirs:
mkdir -p /mnt/data3/jail3/0
add the system to
vi /mnt/data4/bin/snapshot_rotate
on mail:
vi /usr/local/www/mgmt/cgi/backupgraph.pl
(add hostname)
38. mkdir /root/logs
39. edit sshd_config for security
vi /etc/ssh/sshd_config
ListenAddress 69.55.228.101
ListenAddress 10.1.4.118
kill -1 `cat /var/run/sshd.pid`
40. add crontab entries
crontab -e
5 0 * * * /usr/local/jail/bin/backup
1 0 1 * * /usr/local/jail/bin/ipfwreset
0 18 * * * /usr/local/jail/bin/ipfwbackup
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /usr/local/jail/bin/trafstats
55 10,23 * * * /usr/local/jail/bin/trafficwatch.pl
41. Reboot notify script
ln -s /usr/local/jail/bin/notify.sh /usr/local/etc/rc.d/notify.sh
42. copy jailmake from prev system
scp user@10.1.4.119:/usr/local/jail/bin/jailmake /usr/local/jail/bin
rehash
NOTE: remove df altering code from jailmake since we put the correct df in the template, and make sure path to template file is right
43. add to management db (on mail and devweb) jc.ref_machines and jc.ref_templates
uname -r
5.4-RELEASE-p2-jc2
insert into ref_machines values (null,'jail19','jail19.johncompanies.com',0,'l');
select machine_id from ref_machines where host='jail19';
+------------+
| machine_id |
+------------+
| 35 |
+------------+
insert into ref_templates values ('',' 6.2-RELEASE-jc1',5,'FreeBSD 6.2',0);
44. add to server/cabinet map. On mail:
vi /usr/local/www/mgmt/html/cabinetmap.html
45. add an outside blocking rule to the firewall, so this machine can only be reached from inside the firewall. Follow example already in firewall jail17 is:
00119 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.228.200
00119 deny ip from any to 69.55.228.200
jail19 would be 00119...
ipfw add 00119 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.228.200
ipfw add 00119 deny ip from any to 69.55.228.200
46. select customers for probe map
47. install raid monitor
cd /usr/ports/sysutils/asr-utils
make install clean
48. make gv start on boot
scp backup2:/mnt/data4/build/freebsd/gvinum /etc/rc.d/gvinum
gconcat label -v somelabel /dev/gvinum/a /dev/gvinum/b
bsdlabel -r -w /dev/concat/somelabel
newfs /dev/concat/somelabela
mount /dev/concat/somelabel /mount/point
umount /dev/concat/somelabel
gconcat stop somelabel
gconcat label -v /dev/gvinum/a /dev/gvinum/b /dev/gvinum/c /dev/gvinum/d
growfs /dev/concat/somelabel
mount /dev/concat/somelabel /mount/point
volume f
plex org concat
sd length 30449m drive data1
where f is
D data1 State: up /dev/aacd0s1g A: 30449/64241 MB (47%)
gvinum rm -r f
so i setup 2 machines with 6.1. A had 2 logical drives and gv's created across both. B had 1 logical drive and gv's across the 1 drive.
the labeling for the gv's was v1-vN (till i ran out of space)
when i moved aac1 from A to B, the gv's on A's aac1, took precedence over the similarly labeled gv's on B's aac0. in other words. B's aac0 used to have v1-v30. A's aac1 had vn6-vn30. when A's aac1 was moved to B, v1-v5 were linked to B's aac0, and v6-v30 linked to A's aac1
i relabeled B's aac0 gv's to something different (not v1-vN) and tried again and they all showed up.
i tried the experiment again this time moving A's aac0 to B. at that point nothing worked. i moved A's aac0 back to A and renamed the device (data1) to something else, then tried the test again. still no beans.
it's hazy what i did after that cause the machine's weren't cooperating and i was trying to rename the device but basically i think i need to repeat the test and see if i can re-define the device, and probably also give it a unique name and i bet it would have worked.
also interesting- A's aac1 contained data about A's aac0 which showed up when i put aac1 into B
skeeter: as for the gv stuff, it definitely sounds like using some sort of serial numbering scheme would be the way to go if you want to be able to move disks around....
that overlap is still an issue (aac1's device was 'data2') when i had A's aac0 in B nothing worked both disks gv devices were called data1
skeeter: I suppose you could serialize those names as well...
6.2
Last updated 2007-10-15
All time extimates below assume disks aren’t scrubbing. Setup instructions below are for LSI card:
1. make sure bios is setup for bios console redirect
Supermicro:
Console redirection:
Com port addr: on-board COM A
Baud: 38400
Console type: vt100
Flow control: none
Console connection: direct
Continue cr after post: on
2450:
Make sure running bios A09
Console Redirection: VT100/VT220
2. assuming mirrors (or at least disks) created (if not, refer to this), boot to disk 1 of 5.4
skip kernel config (enter)
custom install
partition ->
move cursor to amrd0, hit space (takes you to partition map screen)
a for entire disk
q to quit and save
standard mbr (no boot manager)
space to unselect amrd0
cursor over amrd1
space
a for entire disk
q to quit and save
none (leave untouched)
cursor over amrd0
space
(takes you into part. Screen again) q to exit
none
Make sure both are checked and tab to ok
Label ->
Make sure amrd0 is highlighted
a to start with defaults
john likes:
/1g
/var 256m
Glenn likes:
/128
/var 256
/usr 3g
/tmp 256
We do:
delete everything
/ 128M
swap 2G (for 2950 make it 4G)
/var 256M
/tmp 256M
/usr 3.5G (3584M)
/mnt/data1 remaining space (no need to newfs)
Make sure to toggle S for soft updates on all (should look like UFS2+S Y under the Newfs column)
move cursor to amrd1
swap 4G
/mnt/data2 remaining space (no need to newfs)
q to save and exit
distributions ->
developer
yes to install ports
exit
media ->
cd
commit ->
yes
(2450: 14mins, supermicro: 12mins)
yes to "visit general config" ->
Set root pwd
Add user ‘user’ member group is wheel, set password
Set tz
Networking->interfaces->Fxp0 ->
No IPV6
dhcp=no
Set hostname & domain
Enable sshd
exit...
exit install ->
yes
take the cd out and let the machine reboot
3. double check the date/time
4. edit /etc/make.conf (only add the console speed line if this is a supermicro capable of outputting BIOS redirect at that speed – 2450’s can only do 9600)
echo "WITHOUT_X11=yes \
KERNCONF=jail7 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf
5. add console="comconsole" to /boot/loader.conf
echo "console=""comconsole""" >> /boot/loader.conf
6. turn off all ttyv's except 0 and 1 in /etc/ttys
also turn on ttyd0, change type to vt100:
vi /etc/ttys
ttyd0 "/usr/libexec/getty std.9600" vt100 on secure
kill -1 1
on console server:
vi /etc/remote
(rename port to jail18 depending on where and which digi plugged into)
test serial console
7. populate hosts
echo "10.1.4.3 backup2" >> /etc/hosts
8. put key in authorized_keys on backup2
cd
ssh-keygen -t dsa -b 1024
(default location, leave password blank)
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
confirm that you can ssh to backup2 without getting a login prompt
ssh backup2
9. create & populate binaries/scripts dirs
mkdir -p /usr/local/jail/bin
mkdir -p /usr/local/jail/rc.d
mkdir -p /usr/local/jail/template/
mkdir /mnt/data1
mkdir /mnt/data2
scp backup2:"/mnt/data4/bin/freebsd6.x/*" /usr/local/jail/bin
cd /usr/local/jail/rc.d/
touch quad1
touch quad2
touch quad3
touch quad4
touch safe1
touch safe2
touch safe3
touch safe4
chmod +x *
cd /usr/local/jail/bin
ln -s /usr/local/jail/rc.d/quad1 quad1
ln -s /usr/local/jail/rc.d/quad2 quad2
ln -s /usr/local/jail/rc.d/quad3 quad3
ln -s /usr/local/jail/rc.d/quad4 quad4
ln -s /usr/local/jail/rc.d/safe1 safe1
ln -s /usr/local/jail/rc.d/safe2 safe2
ln -s /usr/local/jail/rc.d/safe3 safe3
ln -s /usr/local/jail/rc.d/safe4 safe4
rehash
10. edit root's path and login script:
vi /root/.cshrc
add to path:
/usr/local/jail/bin
(if adaptec card installed, also add /compat/linux/usr/sbin)
Change alias entries (add G):
alias la ls -aG
alias lf ls -FAG
alias ll ls -lAG
alias ls ls -AG
alias mbm mb mount
alias mbu mb umount
and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "
at the bottom of the file add:
set sshtty=`who am i|awk '{print $2}'`
/usr/sbin/rtprio 3 -`psj | grep $sshtty | awk '{print $2}'`
set shortty=`who am i | awk '{print $2}' | sed -E 's/.*(..)$/\1/'`
foreach x (`psj | grep sh | grep $shortty | awk '{print $2}'`)
/usr/sbin/rtprio 2 -$x
end
To load the new file:
source /root/.cshrc
11. install cvsup
cd /usr/ports/net/cvsup-without-gui
make install clean; rehash; mail -s 'cvs installed' dave.boodman@vtext.com < /dev/null
(stay close for gettext options, 2450: 27mins, supermicro: 17mins, 2950: 22mins)
12. get latest sources for this release:
cd /usr/src
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_2\
*default delete use-rel-suffix\
*default compress\
src-all" > sup
cvsup sup ; mail -s 'cvs sup done' dave.boodman@vtext.com < /dev/null
(2450, ~12mins, supermicro, 27mins, 2950: 7mins)
13. configure new kernel. Get config from similar machine or there may be a master copy somewhere under /mnt/data4/build/freebsd (name the kernel config the same as the jail, ex jail18):
cd /usr/src/sys/i386/conf
scp backup2:/mnt/data4/build/freebsd/kern_config-6.2 ./jail7
or for PAE
scp backup2:/mnt/data4/build/freebsd/kern_config-6.2-PAE ./jail7
edit the kernel config and change ident to be the name of the jail:
vi jail7
ident jail7
edit /sys/conf/newvers.sh to add –jc1 to the end of the BRANCH string (RELEASE-jc1)
vi /sys/conf/newvers.sh
14. bring over patches from backup2
The various patches are in /mnt/data4/build/freebsd/patches on backup2. There are dirs for each version. Not all dirs are populated, but patches for later versions work on older ones unless there is a new patch in the older dir.
cd /usr/src
scp backup2:"/mnt/data4/build/freebsd/patches/6.x/*" .
Apply patches:
patch -l < jls-patch
Apply these only to 2950 with PAE:
patch -p0 < mfi-patch
patch -p0 < gvinum-staticcompile-patch
patch -p0 < gvinum-bin-patch
15. build, install kernel and world
cd /usr/src
make buildworld buildkernel installkernel; mail -s 'kernel build done' dave.boodman@vtext.com < /dev/null
(2450: 1:56min, supermicro: 59mins, 2950: 38mins)
make installworld
(2450: 3min, supermicro: 1min, 2950: :34)
mergemaster -i
delete /var/tmp/temproot
delete bsnmpd
delete temporary ./etc/hosts
delete temporary ./etc/motd
delete /var/tmp/temproot
16. populate devfs ruleset
scp backup2:/mnt/data4/build/freebsd/devfs.rules /etc
17. populate /etc/rc.conf with IPs and NFS settings
vi /etc/rc.conf
kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"
xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
sshd_enable="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.4.119"
devfs_system_ruleset="devfsrules_show_all"
ifconfig_xl0="inet 10.1.4.118 netmask 255.255.255.0"
ifconfig_fxp0="inet 69.55.228.101 netmask 255.255.255.0"
defaultrouter="69.55.228.1"
ifconfig_fxp0_alias0="inet 69.55.2xx.xx netmask 255.255.255.0"
static_routes="t1 office"
route_t1="-net 10.1.5 10.1.4.2"
route_office="-net 10.1.6 10.1.4.2"
gvinum_enable="YES"
fsck_y_enable="YES"
background_fsck="NO"
18. make sure sysctls are set and preserved after reboot
echo "kern.consmute=0\
kern.ipc.shm_use_phys=1\
kern.ipc.shmall=65535\
kern.ipc.shmmax=134217728\
net.inet.tcp.syncookies=0\
kern.maxfiles=32768\
kern.fallback_elf_brand=3\
kern.maxprocperuid=4000\
security.jail.sysvipc_allowed=1\
security.jail.max_procs_per_jail: 1026\
security.jail.allow_raw_sockets=1\
security.jail.socket_unixiproute_only=1\
security.jail.chflags_allowed=0\
dev.amr.0.allow_volume_configure=1\
compat.linux.osrelease=2.6.12" >> /etc/sysctl.conf
19. mount procfs
echo "proc /proc procfs rw 0 0" >> /etc/fstab
For Dell 2950:
echo "linprocfs /usr/compat/linux/proc linprocfs rw 0 0" >> /etc/fstab
echo "linsysfs /usr/compat/linux/sys linsysfs rw 0 0" >> /etc/fstab
20. reboot. Confirm new kernel is loaded
uname –a
Check devfs rules:
devfs rule showsets
devfs rule -s 3 show
21. update ports:
cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_2\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup
cvsup sup; mail -s 'cvs sup ports done' dave.boodman@vtext.com < /dev/null
(2450: 18mins, supermicro: 19mins; 2950: 24mins)
22. Install raid mgmt tool
install linux_base:
cd /usr/ports/emulators/linux_base-fc4
make install clean
(2450: 7min, supermicro: 3mins, 2950: 14mins)
(for LSI)
cd /usr/ports/sysutils/linux-megamgr
make install clean
cd /usr/ports/sysutils/megarc
make install clean
(for Perc5/i)
cd /usr/ports/sysutils/linux-megacli
make install clean
Test:
rehash; megacli ldinfo lall a0
(for adaptec)
This didn’t work: reibuild rpmdb cause we had probs installing aacapps
cd /compat/linux/bin
./rpm --initdb
./rpm --rebuilddb
Install of linux-base lead to broken rpm on 6.2 so:
install aacapps-4.1-0.i386.rpm
scp backup2:/mnt/data4/build/freebsd/aacapps-4.1-0.i386.rpm /tmp/.
/compat/linux/bin/rpm -ivh --excludepath=/dev /tmp/aacapps-4.1-0.i386.rpm
scp user@10.1.4.107:/compat/linux/usr/sbin/aaccli /compat/linux/usr/sbin/aaccli
test out;
/compat/linux/usr/sbin/aaccli
rpm didn’t even install on latest so just scp’d over aaccli and it worked
23. install rsync from ports
cd /usr/ports/net/rsync
make install clean
choose default options
24. install perl from ports
Not necessary if linux_base is installed
cd /usr/ports/lang/perl5.8/
make install clean; rehash
(supermicro: 5min)
25. configure inetd to respond to mrtg load queries
echo "load stream tcp nowait user /usr/local/jail/bin/load.pl load.pl" >> /etc/inetd.conf
echo "load 12384/tcp" >> /etc/services
26. install bb client
adduser
Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username : bb
Password : <random>
Full Name : bb
Uid : 1984
Class :
Groups : bb
Home : /home/bb
Shell : /bin/sh
Locked : no
OK? (yes/no): yes
cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd.tar .
tar xvf bb-freebsd.tar
edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.4.107 jail7.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
vi /home/bb/bbc1.9e-btf/ext/openfiles
MACHINE="jail19,johncompanies,com" # HAS TO BE IN A,B,C FORM
cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src
make; make install
cd ..
./runbb.sh start
more BBOUT
(look for errors)
exit
echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh
27. configure load mrtg, on mail
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg
(add new entry to file following existing format)
28. configure bb on mail:
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
10.1.4.119 jail19.johncompanies.com # ssh
su bb
cd
bbsrc/bb/runbb.sh restart ; exit
29. remove reserve space, enable softupdates (probably already set, so not necessary)
NOT APPLICABLE IF USING GVINUM
cd
umount /mnt/data1
umount /mnt/data2
tunefs -m 0 /mnt/data1
tunefs -m 0 /mnt/data2
mount -a
30. configure ntp
echo "server 10.1.4.5" > /etc/ntp.conf
/usr/sbin/ntpd -p /var/run/ntpd.pid
sleep 2; ntpq -p
(confirm it’s able to reach our time server)
31. mrtg switch graphs
32. fwd and reverse lookups on ns1c
vr johncompanies.com
vi internal.johncompanies.com
rndc reload johncompanies.com IN private
(edit the PTR too)
33. enable noatime option
NOT APPLICABLE IF RUNNING GVINUM
data1 and data2 should look something like:
/dev/amrd0s1g /mnt/data1 ufs rw,noatime 2 2
reboot
34. create gvinum volumes
Make a g partition:
bsdlabel -e /dev/amrd0s1
given:
# /dev/aacd0s1:
8 partitions:
# size offset fstype [fsize bsize bps/cpg]
a: 262144 0 4.2BSD 2048 16384 16392
b: 4194304 262144 swap
c: 143363997 0 unused 0 0 # "raw" part, don't edit
d: 524288 4456448 4.2BSD 2048 16384 32776
e: 524288 4980736 4.2BSD 2048 16384 32776
f: 6291456 5505024 4.2BSD 2048 16384 28552
new offset = 6291456 + 5505024 = 11796480
new size is size for 'c' partition minus the new start from above
143363997 - 11796480 = 131567517
So:
g: 131567517 11796480 unused 0 0
bsdlabel -e /dev/amrd1s1
change d to g
For a 73G drive (after OS), we can fit 31 2G volumes so:
echo 'drive data1 device /dev/aacd0s1g' > /tmp/cgv
or
echo '#\!/bin/sh\
i="1"\
while [ $i -le 31 ]\
do\
echo "volume v$i" >> /tmp/cgv;\
echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 2g drive data1' >> /tmp/cgv;\
i=`expr $i + 1`\
done' > /tmp/mkcgv
sh /tmp/mkcgv
gvinum create /tmp/cgv
For a 146G drive (-4G for swap), we can fit 66 2G volumes so:
echo 'drive data2 device /dev/amrd1s1g' > /tmp/cgv
sh
for f in 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97;\
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 2g drive data2' >> /tmp/cgv; done; exit
gvinum create /tmp/cgv
For 3rd 73G drive (after 2G swap), we can fit 33 2G volumes so:
Label should be:
# /dev/aacd2s1:
8 partitions:
# size offset fstype [fsize bsize bps/cpg]
c: 143299737 0 unused 0 0 # "raw" part, don't edit
g: 143299721 16 unused 0 0
/dev/aacd1s1d
echo 'drive data3 device /dev/aacd2s1g' > /tmp/cgv
sh
for f in 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131;\
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 2g drive data3' >> /tmp/cgv; done
gvinum create /tmp/cgv
For a 2nd 73G drive (after 2G swap), we can fit 33 2G volumes so:
echo 'drive data2 device /dev/aacd1s1g' > /tmp/cgv
sh
for f in 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64;\
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 2g drive data2' >> /tmp/cgv; done; exit
gvinum create /tmp/cgv
For a 2nd 73G drive (after 4G swap), we can fit 32 2G volumes so:
echo 'drive data2 device /dev/aacd1s1g' > /tmp/cgv
sh
for f in 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63;\
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 2g drive data2' >> /tmp/cgv; done; exit
gvinum create /tmp/cgv
For a 3nd 73G drive, we can fit 34 2G volumes so:
echo 'drive data3 device /dev/mfid2s1g' > /tmp/cgv
sh
for f in 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163;\
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 2g drive data3' >> /tmp/cgv; done; exit
gvinum create /tmp/cgv
2950:
bsdlabel -e /dev/mfid0s1
bsdlabel -e /dev/mfid1s1
For 1st drive (146G)
echo 'drive data1 device /dev/mfid0s1g' > /tmp/cgv
sh
for f in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63;\
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 2g drive data1' >> /tmp/cgv; done; exit
Usually there’s 2040m leftover so create one more smaller vol:
echo 'volume v64\
plex org concat\
sd length 2040m drive data1' >> /tmp/cgv
gvinum create /tmp/cgv
For 2nd drive (146G)
echo 'drive data2 device /dev/mfid1s1g' > /tmp/cgv
sh
for f in 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130;\
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 2g drive data2' >> /tmp/cgv; done; exit
gvinum create /tmp/cgv
35. create the jail template
mkdir /mnt/jail
newfs /dev/gvinum/v1
mount /dev/gvinum/v1 /mnt/jail
cd /usr/src
make clean
rm -fr /usr/obj/
make buildworld
(2450: 2:28mins, supermicro: 53mins, 2950: 36min)
make world DESTDIR=/mnt/jail
(2450: 2:28mins, supermicro: 55mins, 2950: )
cd etc
make distribution DESTDIR=/mnt/jail
mount_devfs devfs /mnt/jail/dev
devfs -m /mnt/jail/dev rule -s 3 applyset
cd /mnt/jail
ln -sf dev/null kernel
jail /mnt/jail testhostname 192.168.11.100 /bin/sh
csh
touch /etc/fstab
echo 'network_interfaces=""\
kern_securelevel_enable="NO"\
sendmail_enable="YES"\
sshd_enable="YES"' > /etc/rc.conf
echo "nameserver 69.55.225.225\
nameserver 69.55.230.3" >> /etc/resolv.conf
vi /etc/crontab
remove the adjkerntz lines
comment out periodic’s and put this line above them:
# DO NOT UNCOMMENT THESE
rm -rf /etc/periodic/daily/400.status-disks
check /tmp for crap
vi /etc/periodic/security/100.chksetuid
replace: MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`
with: MP='/' (use single quotes)
mkdir -p /usr/compat/linux/dev
adduser (Add account for user, make sure in wheel group)
user/root passwd: 8ico2987
Set root password
vi /usr/home/user/.profile (and add to the file):
TERM=vt100; export TERM
tzsetup
newaliases
#cd /dev
#rm console
#ln -s null console
vi /etc/syslog.conf (comment out console and move to /var/log/messages):
#*.err;kern.warning;auth.notice;mail.crit /dev/console *.err;kern.warning;auth.notice;mail.crit /var/log/messages
exit
exit
cd /usr/ports
make -DNOCLEANDEPENDS clean
(2450: 15mins , supermicro: 29mins, 2950: 24mins)
rm -fr /usr/ports/distfiles/*
cp -r /usr/ports /mnt/jail/usr (2450: 2:00 mins , supermicro: 15mins, 2950: 3mins)
rm /mnt/jail/root/.history
cd
dump -0a -f /usr/local/jail/template/61template /dev/gvinum/v1
umount /mnt/jail/dev
umount /dev/gvinum/v1
rm -fr /mnt/jail
36. setup backups
echo '#\!/bin/sh\
backupdir=/mnt/data4/jail7\
\
## ENTRY /etc\
## ENTRY /usr/local/etc\
## ENTRY /usr/local/jai7\
## ENTRY /root/logs' > /usr/local/jail/bin/backup.config
on backup2:
setup backup dirs:
mkdir -p /mnt/data2/jail7/0
add the system to
vi /mnt/data4/bin/snapshot_rotate
on mail:
vi /usr/local/www/mgmt/cgi/backupgraph.pl
(add hostname)
Edit /usr/local/jail/bin/backup to use the right drives
37. mkdir /root/logs
38. edit sshd_config for security
vi /etc/ssh/sshd_config
ListenAddress 69.55.226.128
ListenAddress 10.1.4.114
kill -1 `cat /var/run/sshd.pid`
39. add crontab entries
crontab -e
5 0 * * * /usr/local/jail/bin/backup
1 0 1 * * /usr/local/jail/bin/ipfwreset
0 18 * * * /usr/local/jail/bin/ipfwbackup
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /usr/local/jail/bin/trafstats
55 10,23 * * * /usr/local/jail/bin/trafficwatch.pl
On 2950 with Perc5/i also add:
*/5 * * * * /usr/local/jail/bin/perc5iraidchk
40. Reboot notify script
ln -s /usr/local/jail/bin/notify.sh /usr/local/etc/rc.d/notify.sh
41. add to management db (on mail and devweb) jc.ref_machines and jc.ref_templates
uname -r
6.2-RELEASE-jc1
insert into ref_machines values (null,'jail19','jail19.johncompanies.com',0,'l');
select machine_id from ref_machines where host='jail19';
+------------+
| machine_id |
+------------+
| 35 |
+------------+
insert into ref_templates values ('',' 6.2-RELEASE-jc1',15,'FreeBSD 6.2',0);
42. add to server/cabinet map. On mail:
vi /usr/local/www/mgmt/html/cabinetmap.html
43. add an outside blocking rule to the firewall, so this machine can only be reached from inside the firewall. Follow example already in firewall jail17 is:
00117 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.228.200
00117 deny ip from any to 69.55.228.200
jail19 would be 00119...
ipfw add 00107 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.230.108
ipfw add 00107 deny ip from any to 69.55.220.108
44. select customers for probe map
45. install raid monitor (LSI 320 only):
scp backup2:/d4/build/megaraid/MegaMonitor1.02.tgz /tmp
pkg_add MegaMonitor1.02.tgz
edit line in /usr/local/etc/rc.d/megamonitor.sh to look like:
/usr/sbin/MegaCtrl -start -log /var/log/messages -disChkCon -SMART9999 > /megamonitor.out
comment out:
localhost: /var/log/messages : : : **Monitor** :
In /usr/home/bb/bbc1.9e-btf/etc/bb-msgstab
cd /usr/ports/sysutils/megarc
make install clean
46. make gv start on boot
scp backup2:/mnt/data4/build/freebsd/gvinum /etc/rc.d/gvinum
If they start stale:
echo '#\!/bin/sh\
i="1"\
while [ $i -le 64 ]\
do\
gvinum start v$i\
i=`expr $i + 1`\
done' > /tmp/startgv
sh /tmp/startgv
47. patch jail against starting jails with rtprio
mv /usr/sbin/jail /usr/sbin/jail_
echo '#\!/bin/sh\
/usr/sbin/rtprio -t /usr/sbin/jail_ $*' > /usr/sbin/jail
chmod +x /usr/sbin/jail
48. make sure mail works
If there are map errors:
cd /etc/mail; make maps
6.2 -> 6.3
Last updated 2008-08-07
1. remove old src cd /usr rm –fr src/* 2. get latest sources for this release: cd /usr/src echo "*default host=cvsup4.freebsd.org\ *default base=/usr\ *default prefix=/usr\ *default release=cvs tag=RELENG_6_3\ *default delete use-rel-suffix\ *default compress\ src-all" > sup cvsup sup ; pagedave (20min) 3. configure new kernel. Get config from similar machine or there may be a master copy somewhere under /mnt/data4/build/freebsd (name the kernel config the same as the jail, ex jail18): cd /usr/src/sys/i386/conf scp backup2:/mnt/data4/build/freebsd/kern_config-6.2 ./jail7 or for PAE scp backup2:/mnt/data4/build/freebsd/kern_config-6.2-PAE ./jail7 edit the kernel config and change ident to be the name of the jail: vi jail7 ident jail7 edit /sys/conf/newvers.sh to add –jc1 to the end of the BRANCH string (RELEASE-jc1) vi /sys/conf/newvers.sh 3. bring over patches from backup2 The various patches are in /mnt/data4/build/freebsd/patches on backup2. There are dirs for each version. Not all dirs are populated, but patches for later versions work on older ones unless there is a new patch in the older dir. cd /usr/src scp backup2:"/mnt/data4/build/freebsd/patches/6.x/*" . Apply patches: patch -l < jls-patch Apply these only to 2950 with PAE: patch -p0 < gvinum-staticcompile-patch patch -p0 < gvinum-bin-patch 4. build, install kernel cd /usr/src make buildkernel; pagedave (2min) cd /boot mv kernel.old kernel.old.old (optional move old kernel out of the way) cd /usr/src make installkernel cd /boot mv kernel.old kernel.6.2 5. take ½ of mirror and test (boot up) in new hardware 6. improved loader configs: /boot/loader.conf: console="comconsole,vidconsole" boot_serial="YES" boot_multicons="YES" 7. build userland cd /usr/src make buildworld; pagedave (33mins) make installworld mergemaster –i reboot 8. patch jail against starting jails with rtprio mv /usr/sbin/jail /usr/sbin/jail_ echo '#\!/bin/sh\ /usr/sbin/rtprio -t /usr/sbin/jail_ $*' > /usr/sbin/jail chmod +x /usr/sbin/jail
7.0
Last updated 2008-04-30
All time extimates below assume disks aren’t scrubbing. Setup instructions below are for LSI card:
1. make sure bios is setup for bios console redirect
Supermicro:
Console redirection:
Com port addr: on-board COM A
Baud: 38400
Console type: vt100
Flow control: none
Console connection: direct
Continue cr after post: on
2450:
Make sure running bios A09
Console Redirection: VT100/VT220
2. assuming mirrors (or at least disks) created (if not, refer to this), boot to disk 1 of 5.4
skip kernel config (enter)
custom install
partition ->
move cursor to amrd0, hit space (takes you to partition map screen)
a for entire disk
q to quit and save
standard mbr (no boot manager)
space to unselect amrd0
cursor over amrd1
space
a for entire disk
q to quit and save
none (leave untouched)
cursor over amrd0
space
(takes you into part. Screen again) q to exit
none
Make sure both are checked and tab to ok
Label ->
Make sure mfid0 is highlighted
/ 256M
swap 2G (for 2950 make it 4G)
/var 256M (4.6G)
/tmp 256M
/usr 4G
/mnt/data1 remaining space (no need to newfs)
Make sure to toggle S for soft updates on all (should look like UFS2+S Y under the Newfs column)
move cursor to mfid1
swap 8G
/mnt/data2 remaining space (no need to newfs)
q to save and exit
distributions ->
developer
yes to install ports
exit
media ->
cd
commit ->
yes
(2450: 14mins, supermicro: 12mins)
yes to "visit general config" ->
Set root pwd
Add user ‘user’ member group is wheel, set password
Set tz
Networking->interfaces->Fxp0 ->
No IPV6
dhcp=no
Set hostname & domain
Enable sshd
exit...
exit install ->
yes
take the cd out and let the machine reboot
3. double check the date/time
4. edit /etc/make.conf (only add the console speed line if this is a supermicro capable of outputting BIOS redirect at that speed – 2450’s can only do 9600)
echo "WITHOUT_X11=yes \
KERNCONF=jail2 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf
5. add settings to /boot/loader.conf and /boot.config
echo "-Dh" >> /boot.config
echo 'console="comconsole,vidconsole" \
boot_multicons="YES" \
boot_serial="YES" \
mfi_linux_load="YES" \
comconsole_speed="115200"' >> /boot/loader.conf
(leave out the speed and mfi lines for 2450s)
6. turn off all ttyv's except 0 and 1 in /etc/ttys
also turn on ttyd0, change type to vt100:
vi /etc/ttys
ttyd0 "/usr/libexec/getty std.9600" vt100 on secure
kill -1 1
on console server:
vi /etc/remote
(rename port to jail18 depending on where and which digi plugged into)
test serial console
7. populate hosts
echo "10.1.4.3 backup2" >> /etc/hosts
echo "10.1.4.8 backup1" >> /etc/hosts
8. put key in authorized_keys on backup2
cd
ssh-keygen -t dsa -b 1024
(default location, leave password blank)
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'
confirm that you can ssh to backup2 and backup1 without getting a login prompt
ssh backup2 hostname
ssh backup1 hostname
9. create & populate binaries/scripts dirs
mkdir -p /usr/local/jail/bin
mkdir -p /usr/local/jail/rc.d
mkdir -p /usr/local/jail/template/
mkdir /mnt/data1
mkdir /mnt/data2
scp backup2:"/mnt/data4/bin/freebsd7.x/*" /usr/local/jail/bin
cd /usr/local/jail/rc.d/
touch quad1
touch quad2
touch quad3
touch quad4
touch safe1
touch safe2
touch safe3
touch safe4
chmod +x *
cd /usr/local/jail/bin
ln -s /usr/local/jail/rc.d/quad1 quad1
ln -s /usr/local/jail/rc.d/quad2 quad2
ln -s /usr/local/jail/rc.d/quad3 quad3
ln -s /usr/local/jail/rc.d/quad4 quad4
ln -s /usr/local/jail/rc.d/safe1 safe1
ln -s /usr/local/jail/rc.d/safe2 safe2
ln -s /usr/local/jail/rc.d/safe3 safe3
ln -s /usr/local/jail/rc.d/safe4 safe4
ln -s /usr/local/jail/bin/jailmake_zfs jailmake
--OR--
ln -s /usr/local/jail/bin/jailmake_geom jailmake
ln -s /usr/local/jail/bin/js_zfs js
--OR--
ln -s /usr/local/jail/bin/js_geom js
rehash
10. edit root's path and login script:
vi /root/.cshrc
add to path:
/usr/local/jail/bin
(if adaptec card installed, also add /compat/linux/usr/sbin)
Change alias entries (add G):
alias la ls -aG
alias lf ls -FAG
alias ll ls -lAG
alias ls ls -AG
alias mbm mb mount
alias mbu mb umount
and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "
at the bottom of the file add:
set sshtty=`who am i|awk '{print $2}'`
/usr/sbin/rtprio 3 -`psj | grep $sshtty | awk '{print $2}'`
set shortty=`who am i | awk '{print $2}' | sed -E 's/.*(..)$/\1/'`
foreach x (`psj | grep sh | grep $shortty | awk '{print $2}'`)
/usr/sbin/rtprio 2 -$x
end
To load the new file:
source /root/.cshrc
11. install cvsup
cd /usr/ports/net/cvsup-without-gui
make install clean; rehash; mail -s 'cvs installed' dave.boodman@vtext.com < /dev/null
(stay close for gettext options, 2450: 27mins, supermicro: 17mins, 2950: 22mins)
12. get latest sources for this release:
cd /usr/src
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_7\
*default delete use-rel-suffix\
*default compress\
src-all" > sup
-OR-
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_7_1\
*default delete use-rel-suffix\
*default compress\
src-all" > sup
cvsup sup ; mail -s 'cvs sup done' dave.boodman@vtext.com < /dev/null
(2450, ~12mins, supermicro, 27mins, 2950: 7mins)
13. configure new kernel.
for i386:
cd /usr/src/sys/i386/conf
scp backup2:/mnt/data4/build/freebsd/kern_config-7.0-PAE ./mx2
-or-
for amd64:
cd /usr/src/sys/amd64/conf
scp backup2:/mnt/data4/build/freebsd/kern_config-7.0-amd64 ./jail2
------
edit the kernel config and change ident to be the name of the jail:
vi jail2
ident jail2
edit /sys/conf/newvers.sh to add –jc2 to the end of the BRANCH string (RELEASE-jc2)
vi /sys/conf/newvers.sh
notes: http://www.FreeBSD.org/doc/en_US.ISO8859-1/books/handbook/kernelconfig-config.html
14. bring over patches from backup2
The various patches are in /mnt/data4/build/freebsd/patches on backup2. There are dirs for each version. Not all dirs are populated, but patches for later versions work on older ones unless there is a new patch in the older dir.
cd /usr/src
scp backup2:"/mnt/data4/build/freebsd/patches/7.x/*" .
Apply patches:
patch -l < jls-patch
15. build, install kernel and world
cd /usr/src
make buildworld buildkernel installkernel; mail -s 'kernel build done' dave.boodman@vtext.com < /dev/null
(2450: 1:56min, supermicro: 59mins, 2950: 38mins)
make installworld
(2450: 3min, supermicro: 1min, 2950: :34)
mergemaster -i
delete /var/tmp/temproot
delete bsnmpd
delete temporary ./etc/hosts
delete temporary ./etc/motd
delete /var/tmp/temproot
cd /usr/src/sys/modules/zfs
make
make install
16. populate devfs ruleset
scp backup2:/mnt/data4/build/freebsd/devfs.rules /etc
17. populate /etc/rc.conf with IPs and NFS settings
vi /etc/rc.conf
kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"
xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
sshd_enable="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.4.102"
devfs_system_ruleset="devfsrules_show_all"
hostname="jail2.johncompanies.com"
ifconfig_xl0="inet 10.1.4.102 netmask 255.255.255.0"
ifconfig_fxp0="inet 69.55.228.53 netmask 255.255.255.0"
defaultrouter="69.55.228.1"
#ifconfig_fxp0_alias0="inet 69.55.2xx.xx netmask 255.255.255.0"
static_routes="t1 office"
route_t1="-net 10.1.5 10.1.4.2"
route_office="-net 10.1.6 10.1.4.2"
zfs_enable="YES"
18. make sure sysctls are set and preserved after reboot
echo "kern.consmute=0\
kern.ipc.shm_use_phys=1\
kern.ipc.shmall=65535\
kern.ipc.shmmax=134217728\
net.inet.tcp.syncookies=0\
kern.maxfiles=32768\
kern.fallback_elf_brand=3\
kern.maxprocperuid=4000\
security.jail.sysvipc_allowed=1\
security.jail.allow_raw_sockets=1\
security.jail.socket_unixiproute_only=1\
security.jail.chflags_allowed=0\
compat.linux.osrelease=2.6.12\
kern.maxvnodes=400000" >> /etc/sysctl.conf
19. mount procfs
echo "proc /proc procfs rw 0 0" >> /etc/fstab
For Dell 2950/2450:
echo "linprocfs /usr/compat/linux/proc linprocfs rw 0 0" >> /etc/fstab
For Dell 2950:
echo "linsysfs /usr/compat/linux/sys linsysfs rw 0 0" >> /etc/fstab
20. reboot. Confirm new kernel is loaded
uname –a
Check devfs rules:
devfs rule showsets
devfs rule -s 3 show
21. update ports:
cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_7_0\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup
cvsup sup; mail -s 'cvs sup ports done' dave.boodman@vtext.com < /dev/null
(2450: 18mins, supermicro: 19mins; 2950: 24mins)
22. Install raid mgmt tool
For LSI based cards:
install linux_base:
cd /usr/ports/emulators/linux_base-fc4
make install clean
(2450: 7min, supermicro: 3mins, 2950: 14mins)
(for LSI)
cd /usr/ports/sysutils/linux-megamgr
make install clean
cd /usr/ports/sysutils/megarc
make install clean
(for Perc5/i)
cd /usr/ports/sysutils/linux-megacli
make install clean
Test:
rehash; megacli ldinfo lall a0
For adaptec:
On jail18:
scp /compat/linux/usr/sbin/aaccli user@10.1.4.102:~
mv ~user/aaccli /compat/linux/usr/sbin/aaccli
test out;
/compat/linux/usr/sbin/aaccli
For 3w9690:
scp backup2:/d4/build/3ware/tw_cli-freebsd-x86_64-9.5.0.1.tgz /usr/local/sbin
cd /usr/local/sbin xzvf tw_cli-freebsd-x86_64-9.5.0.1.tgz
rm tw_cli-freebsd-x86_64-9.5.0.1.tgz
test out;
./tw_cli /c0 show allstatus
23. install rsync from ports
cd /usr/ports/net/rsync
make install clean
choose default options
24. install perl from ports
Not necessary if linux_base is installed
cd /usr/ports/lang/perl5.8/
make install clean; rehash
(supermicro: 5min)
25. configure inetd to respond to mrtg load queries
echo "load stream tcp nowait user /usr/local/jail/bin/load.pl load.pl" >> /etc/inetd.conf
echo "load 12384/tcp" >> /etc/services
26. install bb client
adduser
Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username : bb
Password : <random>
Full Name : bb
Uid : 1984
Class :
Groups : bb
Home : /home/bb
Shell : /bin/sh
Locked : no
OK? (yes/no): yes
cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd.tar .
tar xf bb-freebsd.tar
edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.4.108 jail2.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
vi /home/bb/bbc1.9e-btf/ext/openfiles
MACHINE="jail2,johncompanies,com" # HAS TO BE IN A,B,C FORM
cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src
make; make install
cd ..
./runbb.sh start
more BBOUT
(look for errors)
exit
echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh
NOTE: to get bb working on amd, had to copy over bin dir from linux dist
27. configure load mrtg, on mail
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg
(add new entry to file following existing format)
28. configure bb on mail:
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
10.1.4.108 jail2.johncompanies.com # ssh
su bb
cd
bbsrc/bb/runbb.sh restart ; exit
29. remove reserve space, enable softupdates (probably already set, so not necessary)
NOT APPLICABLE IF USING GVINUM
cd
umount /mnt/data1
umount /mnt/data2
tunefs -m 0 /mnt/data1
tunefs -m 0 /mnt/data2
mount -a
30. configure ntp
echo "server 10.1.4.5" > /etc/ntp.conf
/usr/sbin/ntpd -p /var/run/ntpd.pid
sleep 2; ntpq -p
(confirm it’s able to reach our time server)
31. mrtg switch graphs
32. fwd and reverse lookups on ns1c
vr johncompanies.com
vi internal.johncompanies.com
rndc reload johncompanies.com IN private
(edit the PTR too)
33. create zpools
Make a g partition:
bsdlabel -e /dev/mfid0s1
given:
# /dev/aacd0s1:
8 partitions:
# size offset fstype [fsize bsize bps/cpg]
a: 262144 0 4.2BSD 2048 16384 16392
b: 4194304 262144 swap
c: 143363997 0 unused 0 0 # "raw" part, don't edit
d: 524288 4456448 4.2BSD 2048 16384 32776
e: 524288 4980736 4.2BSD 2048 16384 32776
f: 6291456 5505024 4.2BSD 2048 16384 28552
new offset = 6291456 + 5505024 = 11796480
new size is size for 'c' partition minus the new start from above
143363997 - 11796480 = 131567517
So:
g: 131567517 11796480 unused 0 0
bsdlabel -e /dev/mfid1s1
change d to g
zpool create pool1 mfid0s1g
zpool create pool2 mfid1s1g
zfs set atime=off pool1
zfs set atime=off pool2
echo ' vfs.zfs.prefetch_disable="1" \
vfs.zfs.arc_min=16777216 \
vfs.zfs.arc_max=33554432 \
vm.kmem_size_max="1G" # for i386\
vm.kmem_size="1G" # for i386\
vm.kmem_size_max="1.5G" # for amd64\
vm.kmem_size="1.5G" # for amd64\
vfs.zfs.zil_disable="1" ' >> /boot/loader.conf
35. create the jail template
zfs create -o mountpoint=/mnt/data1/jail -o quota=4G pool1/jail
cd /usr/ports/sysutils/jailutils
make install clean
cd /usr/src
make world DESTDIR=/mnt/data1/jail
(2450: 2:28mins, supermicro: 55mins, 2950: 1h )
cd etc
make distribution DESTDIR=/mnt/data1/jail
mount -t devfs devfs /mnt/data1/jail/dev
devfs -m /mnt/data1/jail/dev rule -s 3 applyset
cd /mnt/data1/jail
ln -sf dev/null kernel
cp /usr/local/sbin/jkill /mnt/data1/jail/sbin
jail /mnt/data1/jail testhostname 192.168.11.100 /bin/sh
csh
touch /etc/fstab
echo 'network_interfaces=""\
hostname="newsystem"\
kern_securelevel_enable="NO"\
sendmail_enable="YES"\
sshd_enable="YES"' > /etc/rc.conf
echo "nameserver 69.55.225.225\
nameserver 69.55.230.3" >> /etc/resolv.conf
vi /etc/crontab
remove the adjkerntz lines
comment out periodic’s and put this line above them:
# IF YOU UNCOMMENT THESE, PLEASE ADJUST THEIR RUN TIME
rm -rf /etc/periodic/daily/400.status-disks
check /tmp for crap
vi /etc/periodic/security/100.chksetuid
replace: MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`
with: MP='/' (use single quotes)
mkdir -p /usr/compat/linux/dev
adduser (Add account for user, make sure in wheel group)
user/root passwd: 8ico2987
Set root password
vi /usr/home/user/.profile (and add to the file):
TERM=vt100; export TERM
tzsetup
newaliases
rm /sbin/halt /sbin/reboot
ln /sbin/jkill /sbin/halt
ln /sbin/jkill /sbin/reboot
#cd /dev
#rm console
#ln -s null console
vi /etc/syslog.conf (comment out console and move to /var/log/messages):
#*.err;kern.warning;auth.notice;mail.crit /dev/console *.err;kern.warning;auth.notice;mail.crit /var/log/messages
cd /libexec
mv ld-elf32.so.1 ld-elf32.so.1-orig
ln ld-elf.so.1 ld-elf32.so.1
exit
exit
cd /usr/ports
make -DNOCLEANDEPENDS clean
(2450: 15mins , supermicro: 29mins, 2950: 24mins)
rm -fr /usr/ports/distfiles/*
cp -r /usr/ports /mnt/data1/jail/usr (2450: 2:00 mins , supermicro: 15mins, 2950: 3mins)
rm /mnt/data1/jail/root/.history
cd
zfs snapshot pool1/jail@now
zfs send pool1/jail@now | cat > /usr/local/jail/template/70template
zfs destroy pool1/jail@now
zfs destroy pool1/jail
rmdir /mnt/data1/jail
f
36. setup backups
echo '#\!/bin/sh\
backupdir=/data/jail2\
server=backup1\
\
## ENTRY /etc\
## ENTRY /usr/local/etc\
## ENTRY /usr/local/jail\
## ENTRY /root/logs' > /usr/local/jail/bin/backup.config
on backup1:
setup backup dirs:
mkdir -p /data/jail2/0
add the system to
vi /usr/local/sbin/snapshot_rotate
on mail:
vi /usr/local/www/mgmt/cgi/backupgraph.pl
(add hostname)
Edit /usr/local/jail/bin/backup to use the right drives
37. mkdir /root/logs
38. edit sshd_config for security
vi /etc/ssh/sshd_config
ListenAddress 69.55.238.233
ListenAddress 10.1.4.108
kill -1 `cat /var/run/sshd.pid`
39. add crontab entries
crontab -e
5 0 * * * /usr/local/jail/bin/backup.zfs
1 0 1 * * /usr/local/jail/bin/ipfwreset
0 18 * * * /usr/local/jail/bin/ipfwbackup
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /usr/local/jail/bin/trafstats
On 2950 with Perc5/i also add:
*/5 * * * * /usr/local/jail/bin/perc5iraidchk
40. Reboot notify script
ln -s /usr/local/jail/bin/notify.sh /usr/local/etc/rc.d/notify.sh
41. add to management db (on mail and devweb) jc.ref_machines and jc.ref_templates
uname -r
6.2-RELEASE-jc1
insert into ref_machines values (null,'mx2','mx2.johncompanies.com',0,'m');
select machine_id from ref_machines where host='mx2';
+------------+
| machine_id |
+------------+
| 35 |
+------------+
insert into ref_templates values ('',' 7.1-RELEASE-jc2',44,'FreeBSD 7.1',0);
42. add to server/cabinet map. On mail:
vi /usr/local/www/mgmt/html/cabinetmap.html
43. add an outside blocking rule to the firewall, so this machine can only be reached from inside the firewall. Follow example already in firewall jail17 is:
00117 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.228.200
00117 deny ip from any to 69.55.228.200
jail19 would be 00119...
ipfw add 00107 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.230.108
ipfw add 00107 deny ip from any to 69.55.220.108
44. select customers for probe map
47. patch jail against starting jails with rtprio
mv /usr/sbin/jail /usr/sbin/jail_
echo '#\!/bin/sh\
/usr/sbin/rtprio -t /usr/sbin/jail_ $*' > /usr/sbin/jail
chmod +x /usr/sbin/jail
48. make sure mail works
If there are map errors:
cd /etc/mail; make maps
49. move and pare down generic kernel
mv kernel.old/ kernel.GENERIC
/ too full, remove old kernel modules except for:
Id Refs Address Size Name
1 14 0xffffffff80100000 ac6c08 kernel
2 1 0xffffffff80bc8000 1128 mfi_linux.ko
3 4 0xffffffff80bca000 39138 linux.ko
4 1 0xffffffffb48f8000 358c linprocfs.ko
5 1 0xffffffffb48fc000 9d3 linsysfs.ko
6 1 0xffffffffb490c000 80ee8 zfs.ko
50. ntp doesn’t seem to start from rc so,
echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh
chmod 0700 /usr/local/etc/rc.d/ntp.sh
7.1
Last updated 2009-02-16
All time extimates below assume disks aren’t scrubbing. Setup instructions below are for LSI card:
1. make sure bios is setup for bios console redirect
Supermicro:
Console redirection:
Com port addr: on-board COM A
Baud: 38400
Console type: vt100
Flow control: none
Console connection: direct
Continue cr after post: on
2450:
Make sure running bios A09
Console Redirection: VT100/VT220
2. assuming mirrors (or at least disks) created (if not, refer to this), boot to disk 1 of 5.4
skip kernel config (enter)
custom install
partition ->
move cursor to amrd0, hit space (takes you to partition map screen)
a for entire disk
q to quit and save
NOTE
For gvin probs made 2 slices, 1st 8960M
Freebsd boot mgr
standard mbr (no boot manager)
space to unselect amrd0
cursor over amrd1
space
a for entire disk
q to quit and save
none (leave untouched)
cursor over amrd0
space
(takes you into part. Screen again) q to exit
none
Make sure both are checked and tab to ok
Label ->
Make sure mfid0 is highlighted
/ 256M
swap 2G (for 2950 make it 4G)
/var 256M (4.6G)
/tmp 256M
/usr 4G
/mnt/data1 remaining space (no need to newfs)
Make sure to toggle S for soft updates on all (should look like UFS2+S Y under the Newfs column)
move cursor to mfid1
swap 8G
/mnt/data2 remaining space (no need to newfs)
q to save and exit
distributions ->
developer
yes to install ports
exit
media ->
cd
commit ->
yes
(2450: 14mins, supermicro: 12mins)
yes to "visit general config" ->
Set root pwd
Add user ‘user’ member group is wheel, set password
Set tz
Networking->interfaces->Fxp0 ->
No IPV6
dhcp=no
Set hostname & domain
Enable sshd
exit...
exit install ->
yes
take the cd out and let the machine reboot
3. double check the date/time
4. edit /etc/make.conf (only add the console speed line if this is a supermicro capable of outputting BIOS redirect at that speed – 2450’s can only do 9600)
echo "WITHOUT_X11=yes \
KERNCONF=jail2 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf
5. add settings to /boot/loader.conf and /boot.config
echo "-Dh" >> /boot.config
echo 'console="comconsole,vidconsole" \
boot_multicons="YES" \
boot_serial="YES" \
mfi_linux_load="YES" \
comconsole_speed="115200"' >> /boot/loader.conf
(leave out the speed and mfi lines for 2450s)
6. turn off all ttyv's except 0 and 1 in /etc/ttys
also turn on ttyd0, change type to vt100:
vi /etc/ttys
ttyv2 "/usr/libexec/getty Pc" cons25 off secure
ttyv3 "/usr/libexec/getty Pc" cons25 off secure
ttyv4 "/usr/libexec/getty Pc" cons25 off secure
ttyv5 "/usr/libexec/getty Pc" cons25 off secure
ttyv6 "/usr/libexec/getty Pc" cons25 off secure
ttyv7 "/usr/libexec/getty Pc" cons25 off secure
ttyd0 "/usr/libexec/getty std.9600" vt100 on secure
kill -1 1
on console server:
vi /etc/remote
(rename port to jail18 depending on where and which digi plugged into)
test serial console
7. populate hosts
echo "10.1.4.3 backup2" >> /etc/hosts
echo "10.1.4.8 backup1" >> /etc/hosts
8. put key in authorized_keys on backup2
cd
ssh-keygen -t dsa -b 1024
(default location, leave password blank)
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'
confirm that you can ssh to backup2 and backup1 without getting a login prompt
ssh backup2 hostname
ssh backup1 hostname
9. create & populate binaries/scripts dirs
mkdir -p /usr/local/jail/bin
mkdir -p /usr/local/jail/rc.d
mkdir -p /usr/local/jail/template/
mkdir /mnt/data1
mkdir /mnt/data2
scp backup2:"/mnt/data4/bin/freebsd7.x/*" /usr/local/jail/bin
cd /usr/local/jail/rc.d/
touch quad1
touch quad2
touch quad3
touch quad4
touch safe1
touch safe2
touch safe3
touch safe4
chmod +x *
cd /usr/local/jail/bin
ln -s /usr/local/jail/rc.d/quad1 quad1
ln -s /usr/local/jail/rc.d/quad2 quad2
ln -s /usr/local/jail/rc.d/quad3 quad3
ln -s /usr/local/jail/rc.d/quad4 quad4
ln -s /usr/local/jail/rc.d/safe1 safe1
ln -s /usr/local/jail/rc.d/safe2 safe2
ln -s /usr/local/jail/rc.d/safe3 safe3
ln -s /usr/local/jail/rc.d/safe4 safe4
ln -s /usr/local/jail/bin/jailmake_geom jailmake
ln -s /usr/local/jail/bin/js_geom js
ln -s /usr/local/jail/bin/canceljail_geom canceljail
ln -s /usr/local/jail/bin/jailmakeempty_geom jailmakeempty
ln -s /usr/local/jail/bin/jailmake_md jailmake
ln -s /usr/local/jail/bin/js_md js
ln -s /usr/local/jail/bin/canceljail_md canceljail
ln -s /usr/local/jail/bin/jailmakeempty_md jailmakeempty
ln -s /usr/local/jail/bin/postboot_md postboot
rehash
10. edit root's path and login script:
vi /root/.cshrc
Change alias entries (add G):
alias la ls -aG
alias lf ls -FAG
alias ll ls -lAG
alias ls ls -AG
alias mbm mb mount
alias mbu mb umount
add to path:
/usr/local/jail/bin
(if adaptec card installed, also add /compat/linux/usr/sbin)
and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "
at the bottom of the file add:
set sshtty=`who am i|awk '{print $2}'`
/usr/sbin/rtprio 3 -`psj | grep $sshtty | awk '{print $2}'`
set shortty=`who am i | awk '{print $2}' | sed -E 's/.*(..)$/\1/'`
foreach x (`psj | grep sh | grep $shortty | awk '{print $2}'`)
/usr/sbin/rtprio 2 -$x
end
To load the new file:
source /root/.cshrc
11. install cvsup
cd /usr/ports/net/cvsup-without-gui
make install clean; rehash; mail -s 'cvs installed' dave.boodman@vtext.com < /dev/null
(stay close for gettext options, 2450: 27mins, supermicro: 17mins, 2950: 22mins)
12. get latest sources for this release:
cd /usr/src
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_7_1\
*default delete use-rel-suffix\
*default compress\
src-all" > sup
-OR-
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_7\
*default delete use-rel-suffix\
*default compress\
src-all" > sup
(stable)
cvsup sup ; mail -s 'cvs sup done' dave.boodman@vtext.com < /dev/null
(2450, ~12mins, supermicro, 27mins, 2950: 7mins)
13. configure new kernel.
for i386:
cd /usr/src/sys/i386/conf
scp backup2:/mnt/data4/build/freebsd/kern_config-7.0-PAE ./mx2
-or-
for amd64:
cd /usr/src/sys/amd64/conf
scp backup2:/mnt/data4/build/freebsd/kern_config-7.1-amd64 ./jail2
-------
edit the kernel config and change ident to be the name of the jail:
vi jail2
ident jail2
edit /sys/conf/newvers.sh to add –jc2 to the end of the BRANCH string (RELEASE-jc2)
vi /sys/conf/newvers.sh
notes: http://www.FreeBSD.org/doc/en_US.ISO8859-1/books/handbook/kernelconfig-config.html
14. bring over patches from backup2
The various patches are in /mnt/data4/build/freebsd/patches on backup2. There are dirs for each version. Not all dirs are populated, but patches for later versions work on older ones unless there is a new patch in the older dir.
cd /usr/src
scp backup2:"/mnt/data4/build/freebsd/patches/7.x/*" .
Apply patches:
patch -l < jls-patch
15. build, install kernel and world
cd /boot
mv kernel kernel.GENERIC
cd kernel.GENERIC
mkdir hold
mv mfi_linux.ko hold/
mv linux.ko hold/
mv linprocfs.ko hold/
mv linsysfs.ko hold/
mv geom_vinum.ko hold/
mv geom_concat.ko hold/
rm *.ko
rm *.symbols
mv hold/* .
rmdir hold/
cd /usr/src
make buildkernel installkernel
make buildworld ; mail -s 'kernel build done' dave.boodman@vtext.com < /dev/null
(2450: 1:56min, supermicro: 59mins, 2950: 38mins)
make installworld
(2450: 3min, supermicro: 1min, 2950: :34)
mergemaster -i
delete /var/tmp/temproot
delete bsnmpd
delete temporary ./etc/hosts
delete temporary ./etc/motd
delete /var/tmp/temproot
cd /sys/modules/geom/geom_vinum
make
make install
16. populate devfs ruleset
scp backup2:/mnt/data4/build/freebsd/devfs.rules /etc
17. populate /etc/rc.conf with IPs and NFS settings
vi /etc/rc.conf
kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"
xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.4.102"
devfs_system_ruleset="devfsrules_show_all"
#ifconfig_xl0="inet 10.1.4.102 netmask 255.255.255.0"
#ifconfig_fxp0="inet 69.55.228.53 netmask 255.255.255.0"
#ifconfig_fxp0_alias0="inet 69.55.2xx.xx netmask 255.255.255.0"
static_routes="t1 office"
route_t1="-net 10.1.5 10.1.4.2"
route_office="-net 10.1.6 10.1.4.2"
gvinum_enable="YES"
fsck_y_enable="YES"
background_fsck="NO"
rc_mfi_raid_tty_log="YES"
18. make sure sysctls are set and preserved after reboot
echo "kern.consmute=0\
kern.ipc.shm_use_phys=1\
kern.ipc.shmall=131070\
kern.ipc.shmmax=134217728\
net.inet.tcp.syncookies=0\
kern.maxfiles=32768\
kern.fallback_elf_brand=3\
kern.maxprocperuid=4000\
security.jail.sysvipc_allowed=1\
security.jail.allow_raw_sockets=1\
security.jail.socket_unixiproute_only=1\
security.jail.chflags_allowed=0\
dev.amr.0.allow_volume_configure=1\
compat.linux.osrelease=2.6.12" >> /etc/sysctl.conf
19. mount procfs
echo "proc /proc procfs rw 0 0" >> /etc/fstab
For Dell 2950/2450:
echo "linprocfs /usr/compat/linux/proc linprocfs rw 0 0" >> /etc/fstab
For Dell 2950:
echo "linsysfs /usr/compat/linux/sys linsysfs rw 0 0" >> /etc/fstab
mkdir -p /usr/compat/linux/proc
mkdir -p /usr/compat/linux/sys
20. reboot. Confirm new kernel is loaded
uname -a
Check devfs rules:
devfs rule showsets
devfs rule -s 3 show
21. update ports:
cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_7_1\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup
cvsup sup; mail -s 'cvs sup ports done' dave.boodman@vtext.com < /dev/null
(2450: 18mins, supermicro: 19mins; 2950: 24mins)
22. Install raid mgmt tool
For LSI based cards:
install linux_base:
cd /usr/ports/emulators/linux_base-fc4
make install clean
(2450: 7min, supermicro: 3mins, 2950: 14mins)
(for LSI)
cd /usr/ports/sysutils/linux-megamgr
make install clean
cd /usr/ports/sysutils/megarc
make install clean
(for Perc5/i, 6/i)
cd /usr/ports/sysutils/linux-megacli2
make install clean
Test:
rehash; megacli ldinfo lall a0
For adaptec:
On jail18:
scp /compat/linux/usr/sbin/aaccli user@10.1.4.102:~
mv ~user/aaccli /compat/linux/usr/sbin/aaccli
test out;
/compat/linux/usr/sbin/aaccli
For 3w9690:
scp backup2:/d4/build/3ware/tw_cli-freebsd-x86_64-9.5.0.1.tgz /usr/local/sbin
cd /usr/local/sbin xzvf tw_cli-freebsd-x86_64-9.5.0.1.tgz
rm tw_cli-freebsd-x86_64-9.5.0.1.tgz
test out;
./tw_cli /c0 show allstatus
23. install rsync from ports
cd /usr/ports/net/rsync
make install clean
choose default options
25. configure inetd to respond to mrtg load queries
echo "load stream tcp nowait user /usr/local/jail/bin/load.pl load.pl" >> /etc/inetd.conf
echo "load 12384/tcp" >> /etc/services
26. install bb client
adduser
Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username : bb
Password : <random>
Full Name : bb
Uid : 1984
Class :
Groups : bb
Home : /home/bb
Shell : /bin/sh
Locked : no
OK? (yes/no): yes
cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd_linuxcompat.tgz .
tar xzf bb-freebsd_linuxcompat.tgz
edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.4.102 jail2.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
vi /home/bb/bbc1.9e-btf/ext/openfiles
MACHINE="jail2,johncompanies,com" # HAS TO BE IN A,B,C FORM
cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf
./runbb.sh start
more BBOUT
(look for errors)
exit
echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh
NOTE: to get bb working on amd, had to copy over bin dir from linux dist
27. configure load mrtg, on mail
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg
(add new entry to file following existing format)
28. configure bb on mail:
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
10.1.4.102 jail2.johncompanies.com # ssh
su bb
cd
bbsrc/bb/runbb.sh restart ; exit
29. remove reserve space, enable softupdates (probably already set, so not necessary)
NOT APPLICABLE IF USING GVINUM
cd
umount /mnt/data1
umount /mnt/data2
tunefs -m 0 /mnt/data1
tunefs -m 0 /mnt/data2
mount -a
30. configure ntp
echo "server 10.1.4.5" > /etc/ntp.conf
/usr/sbin/ntpd -p /var/run/ntpd.pid
sleep 2; ntpq -p
(confirm it’s able to reach our time server)
32. fwd and reverse lookups on ns1c
vr johncompanies.com
vi internal.johncompanies.com
rndc reload johncompanies.com IN private
(edit the PTR too)
33. create gvinum volumes
Make a g partition:
2950:
bsdlabel -e /dev/mfid0s1
bsdlabel -e /dev/mfid1s1
bsdlabel -e /dev/mfid2s1
given:
# /dev/aacd0s1:
8 partitions:
# size offset fstype [fsize bsize bps/cpg]
a: 262144 0 4.2BSD 2048 16384 16392
b: 4194304 262144 swap
c: 143363997 0 unused 0 0 # "raw" part, don't edit
d: 524288 4456448 4.2BSD 2048 16384 32776
e: 524288 4980736 4.2BSD 2048 16384 32776
f: 6291456 5505024 4.2BSD 2048 16384 28552
new offset = 6291456 + 5505024 = 11796480
new size is size for 'c' partition minus the new start from above
143363997 - 11796480 = 131567517
So:
g: 131567517 11796480 unused 0 0
bsdlabel -e /dev/amrd1s1
change d to g
For a 146G drive (after OS and 4G swap), we can fit 127 1G volumes so:
echo 'drive data1 device /dev/mfid0s1g' > /tmp/cgv
echo '#\!/bin/sh\
i="1"\
while [ $i -le 127 ]\
do\
echo "volume v$i" >> /tmp/cgv;\
echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 1g drive data1' >> /tmp/cgv;\
i=`expr $i + 1`\
done' > /tmp/mkcgv
sh /tmp/mkcgv
gvinum create /tmp/cgv
volume 1
plex org concat
sd length 3g drive data1
volume 2
plex org concat
sd length 5g drive data1
volume 3
plex org concat
sd length 8g drive data1
volume 4
plex org concat
sd length 6g drive data1
volume 5
plex org concat
sd length 10g drive data1
for f in 1 2 3 4 5; do bsdlabel -rw /dev/gvinum/$f; done
for f in 1 2 3 4 5; do newfs /dev/gvinum/${f}a; done
for f in 1 2 3 4 5; do mkdir /mnt/data1/$f; mount /dev/gvinum/${f}a /mnt/data1/$f;done
for f in 1 2 3 4 5; do dd if=/dev/zero of=/mnt/data1/$f/this_is_$f bs=1m count=$f; done
for f in 1 2 3 4 5; do ll -h /mnt/data1/$f/; done
For a 300G drive (after OS and 4G swap), we can fit 273 1G volumes so:
echo 'drive data2 device /dev/mfid1s1g' > /tmp/cgv
echo '#\!/bin/sh\
i="128"\
while [ $i -le 401 ]\
do\
echo "volume v$i" >> /tmp/cgv;\
echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 1g drive data2' >> /tmp/cgv;\
i=`expr $i + 1`\
done' > /tmp/mkcgv
sh /tmp/mkcgv
gvinum create /tmp/cgv
For a 300G drive (after OS and 4G swap), we can fit 273 1G volumes so:
echo 'drive data3 device /dev/mfid2s1g' > /tmp/cgv
echo '#\!/bin/sh\
i="402"\
while [ $i -le 675 ]\
do\
echo "volume v$i" >> /tmp/cgv;\
echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 1g drive data3' >> /tmp/cgv;\
i=`expr $i + 1`\
done' > /tmp/mkcgv
sh /tmp/mkcgv
gvinum create /tmp/cgv
Sometimes there’s 2040m leftover so create one more smaller vol:
echo 'volume v64\
plex org concat\
sd length 2040m drive data1' >> /tmp/cgv
To delete:
echo '#\!/bin/sh\
i="1"\
while [ $i -le 127 ]\
do\
echo "gvinum rm -r v$i" >> /tmp/dgv;\
i=`expr $i + 1`\
done' > /tmp/mkdgv
sh /tmp/mkdgv
sh /tmp/dgv
dd if=/dev/zero of=/dev/sdb seek=285474816 bs=512
dd if=/dev/zero of=/dev/mfid0s2 bs=1m count=1000 (do this to all drives)
35. create the jail template
touch /mnt/data1/jail
mdconfig -a -t vnode -s 1g -f /mnt/data1/jail -u 0
bsdlabel -rw md0
newfs4x /dev/md0c
mkdir /mnt/data1/jail-DIR
mount /dev/md0c /mnt/data1/jail-DIR
cd /usr/ports/sysutils/jailutils
make install clean
cd /usr/src
make world DESTDIR=/mnt/data1/jail-DIR
(2450: 2:28mins, supermicro: 55mins, 2950: 1h )
cd etc
make distribution DESTDIR=/mnt/data1/jail-DIR
mount -t devfs devfs /mnt/data1/jail-DIR/dev
devfs -m /mnt/data1/jail-DIR/dev rule -s 3 applyset
cd /mnt/data1/jail-DIR
ln -sf dev/null kernel
cp /usr/local/sbin/jkill /mnt/data1/jail-DIR/sbin
jail /mnt/data1/jail-DIR testhostname 192.168.11.100 /bin/sh
csh
touch /etc/fstab
echo 'network_interfaces=""\
hostname="newsystem"\
kern_securelevel_enable="NO"\
sendmail_enable="YES"\
sshd_enable="YES"' > /etc/rc.conf
echo "nameserver 69.55.225.225\
nameserver 69.55.230.3" >> /etc/resolv.conf
vi /etc/crontab
remove the adjkerntz lines
comment out periodic’s and put this line above them:
# IF YOU UNCOMMENT THESE, PLEASE ADJUST THEIR RUN TIME
rm -rf /etc/periodic/daily/400.status-disks
check /tmp for crap
vi /etc/periodic/security/100.chksetuid
replace: MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`
with: MP='/' (use single quotes)
mkdir -p /usr/compat/linux/dev
adduser (Add account for user, make sure in wheel group)
user/root passwd: 8ico2987
Set root password
vi /usr/home/user/.profile (and add to the file):
TERM=vt100; export TERM
tzsetup
newaliases
rm /sbin/halt /sbin/reboot
ln /sbin/jkill /sbin/halt
ln /sbin/jkill /sbin/reboot
#cd /dev
#rm console
#ln -s null console
vi /etc/syslog.conf (comment out console and move to /var/log/messages):
#*.err;kern.warning;auth.notice;mail.crit /dev/console *.err;kern.warning;auth.notice;mail.crit /var/log/messages
exit
exit
cd libexec
mv ld-elf32.so.1 ld-elf32.so.1-orig
ln ld-elf.so.1 ld-elf32.so.1
may have to run:
chflags noschg ld-elf32.so.1
chflags noschg ld-elf.so.1
then
chflags schg ld-elf.so.1
chflags schg ld-elf32.so.1
cd /usr/ports
make -DNOCLEANDEPENDS clean
(2450: 15mins , supermicro: 29mins, 2950: 18mins)
rm -fr /usr/ports/distfiles/*
cp -r /usr/ports /mnt/data1/jail-DIR/usr (2450: 2:00 mins , supermicro: 15mins, 2950: 3mins)
rm /mnt/data1/jail-DIR/root/.history
cd
dump -0a -f /usr/local/jail/template/71template /dev/md0c
umount /mnt/data1/jail-DIR/dev
umount /dev/md0c
rmdir /mnt/data1/jail-DIR
36. setup backups
echo '#\!/bin/sh\
backupdir=/data/jail2\
server=backup1\
\
## ENTRY /etc\
## ENTRY /usr/local/etc\
## ENTRY /usr/local/jail\
## ENTRY /root/logs' > /usr/local/jail/bin/backup.config
on backup1:
setup backup dirs:
ssh backup1 mkdir -p /data/jail2/0
add the system to
vi /usr/local/sbin/snapshot_rotate
on mail:
vi /usr/local/www/mgmt/cgi/backupgraph.pl
(add hostname)
Edit /usr/local/jail/bin/backup.xxx to use the right drives and copy to /usr/local/jail/bin/backup
37. mkdir /root/logs
38. edit sshd_config for security
vi /etc/ssh/sshd_config
ListenAddress 69.55.228.53
ListenAddress 10.1.4.102
kill -1 `cat /var/run/sshd.pid`
39. add crontab entries
crontab -e
5 0 * * * /usr/local/jail/bin/backup.md
1 0 1 * * /usr/local/jail/bin/ipfwreset
0 18 * * * /usr/local/jail/bin/ipfwbackup
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /usr/local/jail/bin/trafstats
On 2950 with Perc5/i also add:
*/5 * * * * /usr/local/jail/bin/perc5iraidchk
40. Reboot notify script
ln -s /usr/local/jail/bin/notify.sh /usr/local/etc/rc.d/notify.sh
41. add to management db (on mail and devweb) jc.ref_machines and jc.ref_templates
uname -r
6.2-RELEASE-jc1
insert into ref_machines values (null,'mx2','mx2.johncompanies.com',0,'m');
select machine_id from ref_machines where host='mx2';
+------------+
| machine_id |
+------------+
| 35 |
+------------+
insert into ref_templates values ('',' 7.1-RELEASE-jc2',44,'FreeBSD 7.1',0);
42. add to server/cabinet map. On mail:
vi /usr/local/www/mgmt/html/cabinetmap.html
43. add an outside blocking rule to the firewall, so this machine can only be reached from inside the firewall. Follow example already in firewall jail17 is:
00117 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.228.200
00117 deny ip from any to 69.55.228.200
jail19 would be 00119...
ipfw add 00107 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.230.108
ipfw add 00107 deny ip from any to 69.55.220.108
44. select customers for probe map
46. make gv start on boot
scp backup2:/mnt/data4/build/freebsd/gvinum /etc/rc.d/gvinum
If they start stale:
echo '#\!/bin/sh\
i="1"\
while [ $i -le 127 ]\
do\
gvinum start v$i\
i=`expr $i + 1`\
done' > /tmp/startgv
sh /tmp/startgv
47. patch jail against starting jails with rtprio
mv /usr/sbin/jail /usr/sbin/jail_
echo '#\!/bin/sh\
/usr/sbin/rtprio -t /usr/sbin/jail_ $*' > /usr/sbin/jail
chmod +x /usr/sbin/jail
48. make sure mail works
If there are map errors:
cd /etc/mail; make maps
49. move and pare down generic kernel
mv kernel.old/ kernel.GENERIC
/ too full, remove old kernel modules except for:
Id Refs Address Size Name
1 14 0xffffffff80100000 ac6c08 kernel
2 1 0xffffffff80bc8000 1128 mfi_linux.ko
3 4 0xffffffff80bca000 39138 linux.ko
4 1 0xffffffffb48f8000 358c linprocfs.ko
5 1 0xffffffffb48fc000 9d3 linsysfs.ko
6 1 0xffffffffb490c000 80ee8 zfs.ko
50. ntp doesn’t seem to start from rc so,
echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh
chmod 0700 /usr/local/etc/rc.d/ntp.sh
7.2
Last updated 2009-06-18
All time extimates below assume disks aren’t scrubbing. Setup instructions below are for LSI card:
1. make sure bios is setup for bios console redirect
2950:
Console redirection:
LCD string..
Date to GMT
2. assuming mirrors (or at least disks) created (if not, refer to this), boot to disk 1 of 7.2
skip kernel config (enter)
custom install
partition ->
move cursor to mfid0, hit space (takes you to partition map screen)
a for entire disk
q to quit and save
standard mbr (no boot manager)
space to unselect mfid0
cursor over mfid1
space
a for entire disk
q to quit and save
none (leave untouched)
cursor over mfid0
space
(takes you into part. Screen again) q to exit
none
Make sure both are checked and tab to ok
Label ->
Make sure mfid0 is highlighted
/ 256M
swap 2G (for 2950 make it 4G)
/var 256M (4.6G)
/tmp 256M
/usr 4G
/mnt/data1 remaining space (no need to newfs)
Make sure to toggle S for soft updates on all (should look like UFS2+S Y under the Newfs column)
move cursor to mfid1
swap 8G (or 4G if there’s a 3rd drive)
/mnt/data2 remaining space (no need to newfs)
q to save and exit
distributions ->
developer
yes to install ports
exit
media ->
cd
commit ->
yes
(2450: 14mins, supermicro: 12mins)
yes to "visit general config" ->
Set root pwd
Add user ‘user’ member group is wheel, set password
Set tz
Networking->interfaces->Fxp0 ->
No IPV6
dhcp=no
Set hostname & domain
Enable sshd
exit...
exit install ->
yes
take the cd out and let the machine reboot
3. double check the date/time
4. edit /etc/make.conf
echo "WITHOUT_X11=yes \
KERNCONF=jail8 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf
5. add settings to /boot/loader.conf and /boot.config
echo "-Dh" >> /boot.config
echo 'console="comconsole,vidconsole" \
boot_multicons="YES" \
boot_serial="YES" \
mfi_linux_load="YES" \
comconsole_speed="115200"' >> /boot/loader.conf
6. turn off all ttyv's except 0 and 1 in /etc/ttys
also turn on ttyd0, change type to vt100:
vi /etc/ttys
ttyv2 "/usr/libexec/getty Pc" cons25 off secure
ttyv3 "/usr/libexec/getty Pc" cons25 off secure
ttyv4 "/usr/libexec/getty Pc" cons25 off secure
ttyv5 "/usr/libexec/getty Pc" cons25 off secure
ttyv6 "/usr/libexec/getty Pc" cons25 off secure
ttyv7 "/usr/libexec/getty Pc" cons25 off secure
ttyd0 "/usr/libexec/getty std.9600" vt100 on secure
kill -1 1
on console server:
vi /etc/remote
(rename port to jail18 depending on where and which digi plugged into)
test serial console
7. populate hosts
echo "10.1.4.3 backup2" >> /etc/hosts
echo "10.1.4.8 backup1" >> /etc/hosts
8. put key in authorized_keys on backup2
cd
ssh-keygen -t dsa -b 1024
(default location, leave password blank)
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'
confirm that you can ssh to backup2 and backup1 without getting a login prompt
ssh backup2 hostname
ssh backup1 hostname
9. create & populate binaries/scripts dirs
mkdir -p /usr/local/jail/bin
mkdir -p /usr/local/jail/rc.d
mkdir -p /usr/local/jail/template/
mkdir /mnt/data1
mkdir /mnt/data2
scp backup2:"/mnt/data4/bin/freebsd7.x/*" /usr/local/jail/bin
cd /usr/local/jail/rc.d/
touch quad1
touch deprecated
chmod +x *
cd /usr/local/jail/bin
ln -s /usr/local/jail/rc.d/quad1 quad1
ln -s /usr/local/jail/bin/jailmake_md jailmake
ln -s /usr/local/jail/bin/js_md js
ln -s /usr/local/jail/bin/canceljail_md canceljail
ln -s /usr/local/jail/bin/jailmakeempty_md jailmakeempty
ln -s /usr/local/jail/bin/postboot_md postboot
rehash
10. edit root's path and login script:
vi /root/.cshrc
Change alias entries (add G):
alias la ls -aG
alias lf ls -FAG
alias ll ls -lAG
alias ls ls -AG
alias mbm mb mount
alias mbu mb umount
add to path:
/usr/local/jail/bin
(if adaptec card installed, also add /compat/linux/usr/sbin)
and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "
at the bottom of the file add:
set sshtty=`who am i|awk '{print $2}'`
/usr/sbin/rtprio 3 -`psj | grep $sshtty | awk '{print $2}'`
set shortty=`who am i | awk '{print $2}' | sed -E 's/.*(..)$/\1/'`
foreach x (`psj | grep sh | grep $shortty | awk '{print $2}'`)
/usr/sbin/rtprio 2 -$x
end
To load the new file:
source /root/.cshrc
11. install cvsup
cd /usr/ports/net/cvsup-without-gui
make install clean; rehash; mail -s 'cvs installed' dave.boodman@vtext.com < /dev/null
(stay close for gettext options, 2450: 27mins, supermicro: 17mins, 2950: 22mins)
12. get latest sources for this release:
cd /usr/src
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_7_2\
*default delete use-rel-suffix\
*default compress\
src-all" > sup
-OR-
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_7\
*default delete use-rel-suffix\
*default compress\
src-all" > sup
(stable)
cvsup sup ; mail -s 'cvs sup done' dave.boodman@vtext.com < /dev/null
(2450, ~12mins, supermicro, 27mins, 2950: 7mins)
13. configure new kernel.
cd /usr/src/sys/amd64/conf
scp backup2:/mnt/data4/build/freebsd/kern_config-7.1-amd64 ./jail8
edit the kernel config and change ident to be the name of the jail:
vi jail8
ident jail8
edit /sys/conf/newvers.sh to add –jc2 to the end of the BRANCH string (RELEASE-jc2)
vi /sys/conf/newvers.sh
notes: http://www.FreeBSD.org/doc/en_US.ISO8859-1/books/handbook/kernelconfig-config.html
14. bring over patches from backup2
The various patches are in /mnt/data4/build/freebsd/patches on backup2. There are dirs for each version. Not all dirs are populated, but patches for later versions work on older ones unless there is a new patch in the older dir.
cd /usr/src
scp backup2:"/mnt/data4/build/freebsd/patches/7.2/*" .
Apply patches:
patch -l < jls-patch
15. build, install kernel and world
cd /boot
mv kernel kernel.GENERIC
cd kernel.GENERIC
mkdir hold
mv mfi_linux.ko hold/
mv linux.ko hold/
mv linprocfs.ko hold/
mv linsysfs.ko hold/
mv geom_vinum.ko hold/
mv geom_concat.ko hold/
rm *.ko
rm *.symbols
mv hold/* .
rmdir hold/
cd /usr/src
make buildkernel installkernel
make buildworld ; mail -s 'buildworld done' dave.boodman@vtext.com < /dev/null
(2450: 1:56min, supermicro: 59mins, 2950: 38mins)
make installworld
(2450: 3min, supermicro: 1min, 2950: :34)
mergemaster -i
delete /var/tmp/temproot
delete bsnmpd
delete temporary ./etc/hosts
delete temporary ./etc/motd
delete /var/tmp/temproot
cd /sys/modules/geom/geom_vinum
make
make install
16. populate devfs ruleset
scp backup2:/mnt/data4/build/freebsd/devfs.rules /etc
17. populate /etc/rc.conf with IPs and NFS settings
vi /etc/rc.conf
kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"
xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.4.102"
devfs_system_ruleset="devfsrules_show_all"
#ifconfig_xl0="inet 10.1.4.102 netmask 255.255.255.0"
#ifconfig_fxp0="inet 69.55.228.53 netmask 255.255.255.0"
#ifconfig_fxp0_alias0="inet 69.55.2xx.xx netmask 255.255.255.0"
static_routes="t1 office"
route_t1="-net 10.1.5 10.1.4.2"
route_office="-net 10.1.6 10.1.4.2"
gvinum_enable="YES"
fsck_y_enable="YES"
background_fsck="NO"
rc_mfi_raid_tty_log="YES"
18. make sure sysctls are set and preserved after reboot
echo "kern.consmute=0\
kern.ipc.shm_use_phys=1\
kern.ipc.shmall=131070\
kern.ipc.shmmax=134217728\
net.inet.tcp.syncookies=0\
kern.maxfiles=32768\
kern.fallback_elf_brand=3\
kern.maxprocperuid=4000\
security.jail.sysvipc_allowed=1\
security.jail.allow_raw_sockets=1\
security.jail.socket_unixiproute_only=1\
security.jail.chflags_allowed=0\
dev.amr.0.allow_volume_configure=1\
compat.linux.osrelease=2.6.12\
vm.pmap.shpgperproc=300" >> /etc/sysctl.conf
19. mount procfs
echo "proc /proc procfs rw 0 0" >> /etc/fstab
For Dell 2950/2450:
echo "linprocfs /usr/compat/linux/proc linprocfs rw 0 0" >> /etc/fstab
For Dell 2950:
echo "linsysfs /usr/compat/linux/sys linsysfs rw 0 0" >> /etc/fstab
mkdir -p /usr/compat/linux/proc
mkdir -p /usr/compat/linux/sys
19. enable noatime option
NOT APPLICABLE IF RUNNING GVINUM
data1 and data2 should look something like:
/dev/amrd0s1g /mnt/data1 ufs rw,noatime 2 2
20. reboot. Confirm new kernel is loaded
uname -a
Check devfs rules:
devfs rule showsets
devfs rule -s 3 show
21. update ports:
cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_7_2\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup
cvsup sup; mail -s 'cvs sup ports done' dave.boodman@vtext.com < /dev/null
(2450: 18mins, supermicro: 19mins; 2950: 24mins)
22. Install raid mgmt tool
For LSI based cards:
install linux_base:
cd /usr/ports/emulators/linux_base-fc4
make install clean
(2450: 7min, supermicro: 3mins, 2950: 14mins)
(for LSI)
cd /usr/ports/sysutils/linux-megamgr
make install clean
cd /usr/ports/sysutils/megarc
make install clean
(for Perc5/i, 6/i)
cd /usr/ports/sysutils/linux-megacli2
make install clean
Test:
rehash; megacli ldinfo lall a0
For adaptec:
On jail18:
scp /compat/linux/usr/sbin/aaccli user@10.1.4.102:~
mv ~user/aaccli /compat/linux/usr/sbin/aaccli
test out;
/compat/linux/usr/sbin/aaccli
For 3w9690:
scp backup2:/d4/build/3ware/tw_cli-freebsd-x86_64-9.5.0.1.tgz /usr/local/sbin
cd /usr/local/sbin xzvf tw_cli-freebsd-x86_64-9.5.0.1.tgz
rm tw_cli-freebsd-x86_64-9.5.0.1.tgz
test out;
./tw_cli /c0 show allstatus
23. install rsync from ports
cd /usr/ports/net/rsync
make install clean
choose default options
25. configure inetd to respond to mrtg load queries
echo "load stream tcp nowait user /usr/local/jail/bin/load.pl load.pl" >> /etc/inetd.conf
echo "load 12384/tcp" >> /etc/services
26. install bb client
adduser
Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username : bb
Password : <random>
Full Name : bb
Uid : 1984
Class :
Groups : bb
Home : /home/bb
Shell : /bin/sh
Locked : no
OK? (yes/no): yes
cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd_linuxcompat.tgz .
tar xzf bb-freebsd_linuxcompat.tgz
edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.4.108 jail8.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
vi /home/bb/bbc1.9e-btf/ext/openfiles
MACHINE="jail8,johncompanies,com" # HAS TO BE IN A,B,C FORM
cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf
./runbb.sh start
more BBOUT
(look for errors)
exit
echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh
NOTE: to get bb working on amd, had to copy over bin dir from linux dist
27. configure load mrtg, on mail
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg
(add new entry to file following existing format)
28. configure bb on mail:
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
10.1.4.102 jail2.johncompanies.com # ssh
su bb
cd
bbsrc/bb/runbb.sh restart ; exit
29. remove reserve space, enable softupdates (probably already set, so not necessary)
NOT APPLICABLE IF USING GVINUM
cd
umount /mnt/data1
umount /mnt/data2
tunefs -m 0 /mnt/data1
tunefs -m 0 /mnt/data2
mount -a
30. configure ntp
echo "server 10.1.4.5" > /etc/ntp.conf
/usr/sbin/ntpd -p /var/run/ntpd.pid
sleep 2; ntpq -p
(confirm it’s able to reach our time server)
32. fwd and reverse lookups on ns1c
vr johncompanies.com
vi internal.johncompanies.com
rndc reload johncompanies.com IN private
(edit the PTR too)
35. create the jail template
touch /mnt/data1/jail
mdconfig -a -t vnode -s 1g -f /mnt/data1/jail -u 0
bsdlabel -rw md0
newfs4x -i 4096 /dev/md0c
mkdir /mnt/data1/jail-DIR
mount /dev/md0c /mnt/data1/jail-DIR
cd /usr/ports/sysutils/jailutils
make install clean
cd /usr/src
make world DESTDIR=/mnt/data1/jail-DIR; pagedave
(2450: 2:28mins, supermicro: 55mins, 2950: 1h )
cd etc
make distribution DESTDIR=/mnt/data1/jail-DIR
mount -t devfs devfs /mnt/data1/jail-DIR/dev
devfs -m /mnt/data1/jail-DIR/dev rule -s 3 applyset
cd /mnt/data1/jail-DIR
ln -sf dev/null kernel
cp /usr/local/sbin/jkill /mnt/data1/jail-DIR/sbin
jail /mnt/data1/jail-DIR testhostname 192.168.11.100 /bin/sh
csh
touch /etc/fstab
echo 'network_interfaces=""\
hostname="newsystem"\
kern_securelevel_enable="NO"\
sendmail_enable="YES"\
sshd_enable="YES"' > /etc/rc.conf
echo "nameserver 69.55.225.225\
nameserver 69.55.230.3" >> /etc/resolv.conf
vi /etc/crontab
remove the adjkerntz lines
comment out periodic’s and put this line above them:
# IF YOU UNCOMMENT THESE, PLEASE ADJUST THEIR RUN TIME
rm -rf /etc/periodic/daily/400.status-disks
check /tmp for crap
vi /etc/periodic/security/100.chksetuid
replace: MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`
with: MP='/' (use single quotes)
mkdir -p /usr/compat/linux/dev
adduser (Add account for user, make sure in wheel group)
Username: user
Full name: user
Uid (Leave empty for default):
Login group [user]:
Login group is user. Invite user into other groups? []: wheel
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/user]:
Home directory permissions (Leave empty for default):
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: y
Lock out the account after creation? [no]:
Username : user
Password : <random>
Full Name : user
Uid : 1001
Class :
Groups : user
Home : /home/user
Home Mode :
Shell : /bin/sh
Locked : no
OK? (yes/no): y
adduser: INFO: Successfully added (user) to the user database.
adduser: INFO: Password for (user) is: 901gmYjO
Add another user? (yes/no): n
Goodbye!
vi /usr/home/user/.profile (and add to the file):
TERM=vt100; export TERM
tzsetup
newaliases
rm /sbin/halt /sbin/reboot
ln /sbin/jkill /sbin/halt
ln /sbin/jkill /sbin/reboot
#cd /dev
#rm console
#ln -s null console
vi /etc/syslog.conf (comment out console and move to /var/log/messages):
#*.err;kern.warning;auth.notice;mail.crit /dev/console *.err;kern.warning;auth.notice;mail.crit /var/log/messages
exit
exit
cd libexec
chflags noschg ld-elf32.so.1
chflags noschg ld-elf.so.1
mv ld-elf32.so.1 ld-elf32.so.1-orig
ln ld-elf.so.1 ld-elf32.so.1
chflags schg ld-elf.so.1
chflags schg ld-elf32.so.1
cd /usr/ports
make -DNOCLEANDEPENDS clean
(2450: 15mins , supermicro: 29mins, 2950: 18mins)
rm -fr /usr/ports/distfiles/*
cp -r /usr/ports /mnt/data1/jail-DIR/usr (2450: 2:00 mins , supermicro: 15mins, 2950: 3mins)
rm /mnt/data1/jail-DIR/root/.history
cd
umount /mnt/data1/jail-DIR/dev
dump -0a -f /usr/local/jail/template/72template /dev/md0c
umount /dev/md0c
rmdir /mnt/data1/jail-DIR
36. setup backups
echo '#\!/bin/sh\
backupdir=/data/jail2\
server=backup1\
\
## ENTRY /etc\
## ENTRY /usr/local/etc\
## ENTRY /usr/local/jail\
## ENTRY /root/logs' > /usr/local/jail/bin/backup.config
on backup1:
setup backup dirs:
ssh backup1 mkdir -p /data/jail2/0
add the system to
vi /usr/local/sbin/snapshot_rotate
on mail:
vi /usr/local/www/mgmt/cgi/backupgraph.pl
(add hostname)
Edit /usr/local/jail/bin/backup.xxx to use the right drives and copy to /usr/local/jail/bin/backup
37. mkdir /root/logs
38. edit sshd_config for security
vi /etc/ssh/sshd_config
ListenAddress 69.55.228.53
ListenAddress 10.1.4.102
kill -1 `cat /var/run/sshd.pid`
39. add crontab entries
crontab -e
5 0 * * * /usr/local/jail/bin/backup.md
1 0 1 * * /usr/local/jail/bin/ipfwreset
0 18 * * * /usr/local/jail/bin/ipfwbackup
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /usr/local/jail/bin/trafstats
On 2950 with Perc5/i also add:
*/5 * * * * /usr/local/jail/bin/perc5iraidchk
40. Reboot notify script
ln -s /usr/local/jail/bin/notify.sh /usr/local/etc/rc.d/notify.sh
41. add to management db (on mail and devweb) jc.ref_machines and jc.ref_templates
uname -r
6.2-RELEASE-jc1
insert into ref_machines values (null,'mx2','mx2.johncompanies.com',0,'m');
select machine_id from ref_machines where host='mx2';
+------------+
| machine_id |
+------------+
| 35 |
+------------+
insert into ref_templates values ('',' 7.1-RELEASE-jc2',44,'FreeBSD 7.1',0);
42. add to server/cabinet map. On mail:
vi /usr/local/www/mgmt/html/cabinetmap.html
43. add an outside blocking rule to the firewall, so this machine can only be reached from inside the firewall. Follow example already in firewall jail17 is:
00117 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.228.200
00117 deny ip from any to 69.55.228.200
jail19 would be 00119...
ipfw add 00107 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.230.108
ipfw add 00107 deny ip from any to 69.55.220.108
44. select customers for probe map
47. patch jail against starting jails with rtprio
mv /usr/sbin/jail /usr/sbin/jail_
echo '#\!/bin/sh\
/usr/sbin/rtprio -t /usr/sbin/jail_ $*' > /usr/sbin/jail
chmod +x /usr/sbin/jail
48. make sure mail works
If there are map errors:
cd /etc/mail; make maps
50. ntp doesn’t seem to start from rc so,
echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh
chmod 0700 /usr/local/etc/rc.d/ntp.sh
51. recover space on /usr
rm -fr /usr/obj
52. setup fuse
cd /usr/ports/sysutils/fusefs-kmod/
make install
vi /etc/rc.conf
fusefs_enable="YES"
sysctl vfs.usermount=1
cd /usr/ports/sysutils/fusefs-sshfs
make install
sshfs 1005@usw-s009.rsync.net: /mnt/data1/69.55.234.68-col00001-DIR/mnt
Stress testing a 6.x jail
mkdir /mnt/jail newfs /dev/gvinum/v1 mount /dev/gvinum/v1 /mnt/jail cd /mnt/jail restore -r -f /path/to/61stress . cd .. umount /mnt/jail sh for f in 1 2 3 4 5 6 7 8 9 10; do mkdir /mnt/data1/$f; done gconcat label v3-v5 /dev/gvinum/v3 /dev/gvinum/v4 /dev/gvinum/v5 bsdlabel -r -w /dev/concat/v3-v5 newfs /dev/concat/v3-v5a mount /dev/concat/v3-v5a /mnt/data1/1 bsdlabel -r -w /dev/gvinum/v10 newfs /dev/gvinum/v10a mount /dev/gvinum/v10a /mnt/data1/2 gconcat label v23-v25 /dev/gvinum/v23 /dev/gvinum/v24 /dev/gvinum/v25 bsdlabel -r -w /dev/concat/v23-v25 newfs /dev/concat/v23-v25a mount /dev/concat/v23-v25a /mnt/data1/3 bsdlabel -r -w /dev/gvinum/v50 newfs /dev/gvinum/v50a mount /dev/gvinum/v50a /mnt/data1/4 gconcat label v63-v65 /dev/gvinum/v63 /dev/gvinum/v64 /dev/gvinum/v65 bsdlabel -r -w /dev/concat/v63-v65 newfs /dev/concat/v63-v65a mount /dev/concat/v63-v65a /mnt/data1/5 bsdlabel -r -w /dev/gvinum/v70 newfs /dev/gvinum/v70a mount /dev/gvinum/v70a /mnt/data1/6 gconcat label v83-v85 /dev/gvinum/v83 /dev/gvinum/v84 /dev/gvinum/v85 bsdlabel -r -w /dev/concat/v83-v85 newfs /dev/concat/v83-v85a mount /dev/concat/v83-v85a /mnt/data1/7 bsdlabel -r -w /dev/gvinum/v100 newfs /dev/gvinum/v100a mount /dev/gvinum//v100a /mnt/data1/8 gconcat label v123-v125 /dev/gvinum/v123 /dev/gvinum/v124 /dev/gvinum/v125 bsdlabel -r -w /dev/concat/v123-v125 mount /dev/concat/v123-v125a /mnt/data1/9 newfs /dev/concat/v123-v125a mount /dev/concat/v123-v125a /mnt/data1/9 bsdlabel -r -w /dev/gvinum/v130 newfs /dev/gvinum/v130a mount /dev/gvinum//v130a /mnt/data1/10 sh for f in 1 2 3 4 5 6 7 8 9 10; do cd /mnt/data1/$f; dump -0a -f - /dev/gvinum/v1| restore -r -f - ; done sh for f in 1 2 3 4 5 6 7 8 9 10; do mount_devfs devfs /mnt/data1/$f/dev; devfs -m /mnt/data1/$f/dev rule -s 3 applyset; done sh for f in 15 16 17 18 19 20 21 22 23 24; do ifconfig bce0 alias 10.1.6.$f netmask 255.255.255.255; done jail /mnt/data1/1 testhostname1 10.1.6.15 /bin/sh /etc/rc 2> /dev/null jail /mnt/data1/2 testhostname2 10.1.6.16 /bin/sh /etc/rc 2> /dev/null jail /mnt/data1/3 testhostname3 10.1.6.17 /bin/sh /etc/rc 2> /dev/null jail /mnt/data1/4 testhostname4 10.1.6.18 /bin/sh /etc/rc 2> /dev/null jail /mnt/data1/5 testhostname5 10.1.6.19 /bin/sh /etc/rc 2> /dev/null jail /mnt/data1/6 testhostname6 10.1.6.20 /bin/sh /etc/rc 2> /dev/null jail /mnt/data1/7 testhostname7 10.1.6.21 /bin/sh /etc/rc 2> /dev/null jail /mnt/data1/8 testhostname8 10.1.6.22 /bin/sh /etc/rc 2> /dev/null jail /mnt/data1/9 testhostname9 10.1.6.23 /bin/sh /etc/rc 2> /dev/null jail /mnt/data1/10 testhostname10 10.1.6.24 /bin/sh /etc/rc 2> /dev/null systat -vmstat
Stress testing zfs
zfs receive pool1/jail2 < /usr/local/jail/template/template zfs set mountpoint=/mnt/data1/jail2 pool1/jail2 zfs set quota=4G pool1/jail2 zfs destroy pool1/jail2@now
install apps that will run when jail started
cp -r /usr/src /mnt/data1/jail2/usr
rm -fr /mnt/data1/jail2/usr/obj
mount -t devfs devfs /mnt/data1/jail2/dev; devfs -m /mnt/data1/jail2/dev rule -s 3 applyset;
jail /mnt/data1/jail2 stress-test 69.55.234.86 /bin/sh
csh
cd /usr/ports/benchmarks/bonnie
make install clean
cd /usr/ports/sysutils/stress
make install clean
cd /usr/ports/net/rsync
make install clean
cd /usr/ports/lang/perl5.8
make install clean
cd /usr/local/etc/rc.d/
cat >> boot.sh
sleep 30 && sh /usr/local/etc/rc.d/buildworld &
sleep 30 && sh /usr/local/etc/rc.d/portindex &
sleep 30 && sh /usr/local/etc/rc.d/stress &
sleep 30 && sh /usr/local/etc/rc.d/bonnie &
cat > buildworld
#!/bin/sh
while (true); do cd /usr/src; make buildworld; done
cat > portindex
while (true); do cd /usr/ports/; make index; done
cat > stress
/usr/local/bin/stress -c 1 -i 1 -m 1 -d 1 --vm-bytes 768M --hdd-bytes 128M
cat > bonnie
#!/bin/sh
while (true); do /usr/local/bin/bonnie -s 2g; done
chmod +x *
exit
exit
cd
zfs snapshot pool1/jail2@now
sh
for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105; do zfs send pool1/jail2@now | zfs receive pool1/jail$f;
zfs set quota=10G pool1/jail$f;
zfs set mountpoint=/mnt/data1/jail$f pool1/jail$f;
done
for f in 106 107 108 114 115 116 117 118 119 120 121 122 123 125 126 127 128 129 130 131; do zfs send pool1/jail2@now | zfs receive pool2/jail$f
zfs set quota=10G pool2/jail$f;
zfs set mountpoint=/mnt/data2/jail$f pool2/jail$f;
done
for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101; do chmod 0 /mnt/data1/jail$f/usr/local/etc/rc.d/bonnie; done
for f in 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105; do chmod 0 /mnt/data1/jail$f/usr/local/etc/rc.d/stress; done
for f in 115 116 117 118 119 120 121 122 123 125 126 127; do chmod 0 /mnt/data2/jail$f/usr/local/etc/rc.d/bonnie; done
for f in 119 120 121 122 123 125 126 127 128 129 130 131; do chmod 0 /mnt/data2/jail$f/usr/local/etc/rc.d/stress; done
for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105; do mount -t devfs devfs /mnt/data1/jail$f/dev; devfs -m /mnt/data1/jail$f/dev rule -s 3 applyset; done
for f in 106 107 108 114 115 116 117 118 119 120 121 122 123 125 126 127 128 129 130 131; do mount -t devfs devfs /mnt/data2/jail$f/dev; devfs -m /mnt/data2/jail$f/dev rule -s 3 applyset; done
for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105;
do jail /mnt/data1/jail$f testhostname$f 69.55.234.$f /bin/sh /etc/rc 2> /dev/null; done
for f in 106 107 108 114 115 116 117 118 119 120 121 122 123 125 126 127 128 129 130 131;
do jail /mnt/data2/jail$f testhostname$f 69.55.234.$f /bin/sh /etc/rc 2> /dev/null; done
systat -vmstat
sysctl vfs.numvnodes
for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105; do umount /mnt/data1/jail$f/dev; zfs destroy -r pool1/jail$f; done
for f in 106 107 108 114 115 116 117 118 119 120 121 122 123 125 126 127 128 129 130 131; do umount /mnt/data2/jail$f/dev; zfs destroy -r pool2/jail$f; done
for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105; do touch /mnt/data1/jail$f;
mdconfig -a -t vnode -s 10g -f /mnt/data1/jail$f -u $f;
bsdlabel -r -w md$f auto;
newfs -O 1 /dev/md${f}a;
mkdir /mnt/data1/jail$f-DIR;
mount /dev/md${f}a /mnt/data1/jail$f-DIR;
rsync -aSH /mnt/data2/jail2/ /mnt/data1/jail$f-DIR/;
done
for f in 106 107 108 114 115 116 117 118 119 120 121 122 123 125 126 127 128 129 130 131; do touch /mnt/data2/jail$f;
mdconfig -a -t vnode -s 10g -f /mnt/data2/jail$f -u $f;
bsdlabel -r -w md$f auto;
newfs -O 1 /dev/md${f}a;
mkdir /mnt/data2/jail$f-DIR;
mount /dev/md${f}a /mnt/data2/jail$f-DIR;
rsync -aSH /mnt/data1/jail86-DIR/ /mnt/data2/jail$f-DIR/;
done
for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105;
do chmod 0700 /mnt/data1/jail$f-DIR/usr/local/etc/rc.d/bonnie; done
for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105;
do chmod 0700 /mnt/data1/jail$f-DIR/usr/local/etc/rc.d/stress; done
for f in 106 107 108 114 115 116 117 118 119 120 121 122 123 125 126 127 128 129 130 131;
do chmod 0700 /mnt/data2/jail$f-DIR/usr/local/etc/rc.d/bonnie; done
for f in 106 107 108 114 115 116 117 118 119 120 121 122 123 125 126 127 128 129 130 131;
do chmod 0700 /mnt/data2/jail$f-DIR/usr/local/etc/rc.d/stress; done
for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101; do chmod 0 /mnt/data1/jail$f-DIR/usr/local/etc/rc.d/bonnie; done
for f in 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105; do chmod 0 /mnt/data1/jail$f-DIR/usr/local/etc/rc.d/stress; done
for f in 115 116 117 118 119 120 121 122 123 125 126 127; do chmod 0 /mnt/data2/jail$f-DIR/usr/local/etc/rc.d/bonnie; done
for f in 119 120 121 122 123 125 126 127 128 129 130 131; do chmod 0 /mnt/data2/jail$f-DIR/usr/local/etc/rc.d/stress; done
for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105; do mount -t devfs devfs /mnt/data1/jail$f-DIR/dev; devfs -m /mnt/data1/jail$f-DIR/dev rule -s 3 applyset; done
for f in 106 107 108 114 115 116 117 118 119 120 121 122 123 125 126 127 128 129 130 131; do mount -t devfs devfs /mnt/data2/jail$f-DIR/dev; devfs -m /mnt/data2/jail$f-DIR/dev rule -s 3 applyset; done
for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105;
do jail /mnt/data1/jail$f-DIR testhostname$f 69.55.234.$f /bin/sh /etc/rc 2> /dev/null; done
for f in 106 107 108 114 115 116 117 118 119 120 121 122 123 125 126 127 128 129 130 131;
do jail /mnt/data2/jail$f-DIR testhostname$f 69.55.234.$f /bin/sh /etc/rc 2> /dev/null; done
for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105;
do cp /mnt/data1/boot.sh /mnt/data1/jail$f-DIR/usr/local/etc/rc.d/;
cp /mnt/data1/stress /mnt/data1/jail$f-DIR/usr/local/etc/rc.d/;
cp /mnt/data1/bonnie /mnt/data1/jail$f-DIR/usr/local/etc/rc.d/;
done
for f in 106 107 108 114 115 116 117 118 119 120 121 122 123 125 126 127 128 129 130 131;
do cp /mnt/data1/boot.sh /mnt/data2/jail$f-DIR/usr/local/etc/rc.d/;
cp /mnt/data1/stress /mnt/data2/jail$f-DIR/usr/local/etc/rc.d/;
cp /mnt/data1/bonnie /mnt/data2/jail$f-DIR/usr/local/etc/rc.d/;
done
for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105; do umount /mnt/data1/jail$f-DIR/dev; umount /mnt/data1/jail$f-DIR; mdconfig -d -u $f; done
for f in 106 107 108 114 115 116 117 118 119 120 121 122 123 125 126 127 128 129 130 131; do umount /mnt/data2/jail$f-DIR/dev; umount /mnt/data2/jail$f-DIR; mdconfig -d -u $f; done
for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105; do
mkdir /mnt/data1/jail$f-DIR/;
cd /mnt/data1/jail$f-DIR/;
dump -0a -f - /dev/md0a | restore -r -f -;
done
for f in 106 107 108 114 115 116 117 118 119 120 121 122 123 125 126 127 128 129 130 131; do mkdir /mnt/data2/jail$f-DIR/;
cd /mnt/data2/jail$f-DIR/;
dump -0a -f - /dev/md0a | restore -r -f -;
done
for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105; do chmod 0700 /mnt/data1/jail$f-DIR/usr/local/etc/rc.d/*; done
for f in 106 107 108 114 115 116 117 118 119 120 121 122 123 125 126 127 128 129 130 131; do chmod 0700 /mnt/data2/jail$f-DIR/usr/local/etc/rc.d/*; done
--vm-bytes 300M
Reloading FreeBSD (while preserving customer data)
2. assuming 2 mirrors, boot to disk 1 of 4.11 skip kernel config (enter) custom install (skip partition) label move cursor to aacd0, hit space (takes you to partition map screen) given this as the pre-existing partition map: Filesystem 1K-blocks Used Avail Capacity Mounted on /dev/aacd0s1a 128990 36888 81784 31% / /dev/aacd0s1h 30499106 4 28059174 0% /mnt/data1 /dev/aacd1s1e 69526550 4 63964422 0% /mnt/data2 /dev/aacd0s1f 257998 10 237350 0% /tmp /dev/aacd0s1g 3096462 852082 1996664 30% /usr /dev/aacd0s1e 257998 1198 236162 1% /var procfs 4 4 0 100% /proc relabel and all mount points according to prev settings or, delete all partitions except for data parts, re-create as necessary re-toggle newfs on all mount points except for /mnt/data1 - make sure under the Newfs column the data1 (and data2) partiotions look like: UFS+S Y make sure under the Newfs column the data1 (and data2) partiotions look like UFS+S N space to unselect aacd0 cursor over aacd1 space Space to select both drives and tab to ok Continue installation as per normal distributions developer yes to install ports exit media cd commit yes (2450: 16mins, supermicro: 11mins) yes to "visit general config" Set root pwd Add user ‘user’ member group is wheel, set password Set tz Networking->interfaces->Fxp0 No IPV6 dhcp=yes Set hostname & domain exit... exit install yes take the cd out and let the machine reboot