Revision as of 14:18, 15 March 2013

8.x

Assumptions

Setup instructions below assume this is DELL 2950 with an LSI-based SAS RAID card.

Server is at castle, connected to pub, private, serial and DRAC

Assuming OS loading done via IPKVM with ISO mounted via USB

Assumes at 4 drives, 2 mirrors

Configure server BIOS

setup console redirect, speed 115200

set LCD string to name of server "jail8"

set date to GMT

go into RAID bios and setup mirrors

configure DRAC: TODO

Install OS (sysinstall)

boot to bootonly disk for AMD version of FreeBSD, i.e. FreeBSD-8.3-RELEASE-amd64-bootonly.iso

when the install menu appears, choose custom install

partition menu

move cursor to mfid0, hit space (takes you to partition map screen). If there is only 1 mirror, there will be no option to select a specifit drive: mfid0 will be selected.

type 'a' to use entire disk
type 'q' to quit and save
choose 'freebsd standard mbr'

space to unselect mfid0
cursor down to mfid1
hit space

type 'a' to use entire disk
type 'q' to quit and save
choose 'none' for boot mgr (leave untouched)

cursor over mfid0
space
(takes you into part. Screen again) 'q' to exit
none for boot mgr

Make sure both drives (mfid0 and mfid1) are checked and tab to ok

Label menu

Make sure mfid0 is highlighted at the top of the screen, setup the following partitions

/ 512M
swap 6G
/var 256M
/tmp 256M
/usr 5G
/mnt/data1 remaining space

All partitions except / should be setup for soft updates. If not, type 's' to enable for soft updates on all except for / (should look like UFS2+S Y under the Newfs column)

move cursor to mfid1 at the top of the screen
swap 8G (or 4G if there’s a 3rd drive)
/mnt/data2 remaining space
'q' to save and exit

distributions

Choose the following distribudions

developer (ok to install ports)
custom -> lib32

exit

media

if you are installing via a cd, no need to enter this menu or change anything. Otherwise, choose ftp to install via ftp. You will be prompted to setup networking. You will need to choose a nic (typically bce0 or bce1). Say no to DHCP and IPv6. Hopefully the public nic cable was installed in bce0 so start with that nic and provide the hostname, (public) IP, netmask, gateway and DNS. When configured, it should start pinging. If it doesn't, have the NOC swap cables. Select any FTP server, usually Main or ftp4.

commit

this usually takes 12mins
during the process you may need to select a new ftp mirror, this is not a problem.
at the conclution of the install you will be prompted to enter the root password (2x) and returned to the configuration menu.

add user

Add user 'user'. Defaults for everything is fine, just remember to enter 'wheel' in the member group field. Do set the password.

Setup timezone

PT

Networking

page down to the bottom and enable '[X]' sshd

If you installed via cd, you will need to visit:
interfaces->bce0->
No IPV6
dhcp=no
Set hostname, IP, DNS, gateway
(i.e. setup the nic as indicated above)

Exit the install and if you installed via CD, take it out and let the machine reboot

Configure OS, kernel, userland, jail

double check the date/time

populate /etc/resolv.conf

echo "nameserver 69.55.225.225\
nameserver 69.55.230.3
nameserver 69.55.229.3" > /etc/resolv.conf

edit /etc/make.conf

echo "WITHOUT_X11=yes \
KERNCONF=jail3 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf

setup bootloader for console, etc

add settings to /boot/loader.conf and /boot.config:

echo "-Dh" >> /boot.config

echo 'console="comconsole,vidconsole" \
boot_multicons="YES" \
boot_serial="YES" \
mfi_linux_load="YES" \
comconsole_speed="115200"' >> /boot/loader.conf

enable login via serial console

turn off all ttyv's except 0 and 1 in /etc/ttys and turn on ttyd0, change type to vt100:

vi /etc/ttys

The changed lines should look like:

ttyv2   "/usr/libexec/getty Pc"         cons25  off secure
ttyv3   "/usr/libexec/getty Pc"         cons25  off secure
ttyv4   "/usr/libexec/getty Pc"         cons25  off secure
ttyv5   "/usr/libexec/getty Pc"         cons25  off secure
ttyv6   "/usr/libexec/getty Pc"         cons25  off secure
ttyv7   "/usr/libexec/getty Pc"         cons25  off secure
# Serial terminals
# The 'dialup' keyword identifies dialin lines to login, fingerd etc.
ttyu0   "/usr/libexec/getty std.9600"  vt100   on secure

Restart init

kill -1 1

At this point you should have a login on console.

To configure serial console access, login to the console server as root and run:

# vi /etc/remote

following examples there, rename port to server's hostname, depending on where and which digi box this server is plugged into. Make sure to get speed right too: 115200

populate hosts

If server is at castle:

echo "10.1.4.3 backup2" >> /etc/hosts
echo "10.1.4.8 backup1" >> /etc/hosts

If server is at i2b:

echo "69.55.230.10 backup2" >> /etc/hosts
echo "10.1.2.3 backup3" >> /etc/hosts
echo "69.55.230.11 backup1" >> /etc/hosts

create ssh key, upload to backup servers

cd
ssh-keygen -t dsa -b 1024

(default location, leave password blank)

If server is at castle:

cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys' 
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'

If server is at i2b:

cat /root/.ssh/id_dsa.pub | ssh backup3 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys' 
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'

confirm that you can ssh to backup2 and backup1 without getting a login prompt

ssh backup2 hostname
ssh backup1 hostname
ssh backup3 hostname

create & populate binaries/scripts dirs

mkdir -p /usr/local/jail/bin
mkdir -p /usr/local/jail/rc.d
mkdir -p /usr/local/jail/template/
mkdir /mnt/data1
mkdir /mnt/data2
scp backup2:"/mnt/data4/bin/freebsd8.x/*" /usr/local/jail/bin
cd /usr/local/jail/rc.d/
touch quad1
touch deprecated
chmod +x *
cd /usr/local/jail/bin
ln -s /usr/local/jail/rc.d/quad1 quad1
ln -s /usr/local/jail/bin/jailmake_md jailmake
ln -s /usr/local/jail/bin/js_md js
ln -s /usr/local/jail/bin/canceljail_md canceljail
ln -s /usr/local/jail/bin/jailmakeempty_md jailmakeempty
ln -s /usr/local/jail/bin/postboot_md postboot
ln -s /usr/local/jail/bin/preboot_md preboot
ln -s /usr/local/jail/bin/startjail_md startjail
ln -s /usr/local/jail/bin/stopjail_md stopjail

rehash

edit root's path and login script

vi /root/.cshrc

Change alias entries (add G):

alias la        ls -aG
alias lf        ls -FAG
alias ll        ls -lAG
alias ls        ls -AG
alias mbm       mb mount
alias mbu       mb umount
alias cjb       cd /usr/local/jail/bin
alias cd1       cd /mnt/data1
alias cd2       cd /mnt/data2
alias cd3       cd /mnt/data3
alias jtop      jtop lj
alias j         jobs

add to path be careful to leave a space after bin and make sure the wrapping isn't broken:

/usr/local/jail/bin

alter the prompt, set the following:

set prompt = "`/bin/hostname -s` %/# "

at the bottom of the file add:

set sshtty=`who am i|awk '{print $2}'`
/usr/sbin/rtprio 3 -`psj | grep $sshtty | awk '{print $2}'`

set shortty=`who am i | awk '{print $2}' | sed -E 's/.*(..)$/\1/'`
foreach x (`psj | grep sh | grep $shortty | awk '{print $2}'`)
/usr/sbin/rtprio 2 -$x
end

Make the new settings active in current shell:

source /root/.cshrc

install cvsup

cd /usr/ports/net/cvsup-without-gui 
make install clean; rehash; mail -s 'cvs installed' support@johncompanies.com < /dev/null

stand by for gettext options (use defaults). this process takes approx 22mins- hence the email/page notice above.

get latest sources for this release

cd /usr/src 
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_8_3\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

If you need to run stable (cause release is broken or some other reason) make the sup file look like:

echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_8\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

cvsup sup ; mail -s 'cvs sup done' support@johncompanies.com < /dev/null

time varies, 10-20mins

configure new kernel

Pull down the kernel config we are using for this distribution. In this case we use an 8.2 kernel config on 8.3, which is valid. The local file should be the same name as host- jail3 in this example

cd /usr/src/sys/amd64/conf 
scp backup2:/mnt/data4/build/freebsd/kern_config-8.2-amd64 ./jail3

edit the kernel config and change ident to be the name of the jail:

vi jail3
ident           jail3

Optional, edit /sys/conf/newvers.sh to add –jc2 to the end of the BRANCH string (RELEASE-jc2)

vi /sys/conf/newvers.sh

notes on kernel configuring: http://www.FreeBSD.org/doc/en_US.ISO8859-1/books/handbook/kernelconfig-config.html

install patches

The various patches are in /mnt/data4/build/freebsd/patches on backup2. There are dirs for each version. Not all dirs are populated, but patches for later versions work on older ones unless there is a new patch in the older dir.

there are no patches we use for 8.x, but here would be the commands:

cd /usr/src
scp backup2:"/mnt/data4/build/freebsd/patches/8.0/*" .

Apply patches, i.e. the jls-patch:

patch -l < jls-patch

build, install kernel and world

Rename current generic kernel so it will always be available to boot from. Save room by removing non-needed kernel modules:

cd /boot
mv kernel kernel.GENERIC
cd kernel.GENERIC
mkdir hold
mv mfi_linux.ko hold/
mv linux.ko hold/
mv linprocfs.ko hold/
mv linsysfs.ko hold/
mv geom_vinum.ko hold/
mv geom_concat.ko hold/
mv zfs.* hold/
mv opensolaris* hold/

rm *.ko
rm *.symbols
mv hold/* .
rmdir hold/

cd /usr/src
make buildkernel installkernel

make buildworld ; mail -s 'buildworld done' dave.boodman@vtext.com < /dev/null

~38mins

make installworld

~34min

mergemaster -i

You will be prompted to merge, replace or ignore files changed by the src update. In most cases you can delete the temp (new) files.

ONLY if this will be a zfs system (not currently used in 8.x):

cd /sys/modules/zfs
make 
make install
cd /sys/modules/opensolaris
make 
make install

populate devfs ruleset

scp backup2:/mnt/data4/build/freebsd/devfs.rules.8x /etc/devfs.rules

populate /etc/rc.conf with IPs and service settings

vi /etc/rc.conf

kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"

nfs_client_enable="YES"
nfs_reserved_port_only="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.2.103"
devfs_system_ruleset="devfsrules_show_all"

ifconfig_bce1="inet 10.1.2.103 netmask 255.255.255.0"
ifconfig_bce0="inet 69.55.229.7 netmask 255.255.255.0"
#ifconfig_bce0_alias0="inet 69.55.2xx.xx netmask 255.255.255.0"

fsck_y_enable="YES"
background_fsck="NO"
#rc_mfi_raid_tty_log="YES"
#zfs_enable="YES"

Modify IPs, hostname, gateway for this box.

make sure sysctls are set and preserved upon boot

echo "kern.consmute=0\
kern.ipc.shm_use_phys=1\
kern.ipc.shmall=131070\
kern.ipc.shmmax=134217728\
net.inet.tcp.syncookies=0\
kern.maxfiles=32768\
kern.fallback_elf_brand=3\
kern.maxprocperuid=4000\
security.jail.sysvipc_allowed=1\
security.jail.allow_raw_sockets=1\
security.jail.socket_unixiproute_only=1\
security.jail.chflags_allowed=0\
dev.amr.0.allow_volume_configure=1\
compat.linux.osrelease=2.6.12\
vm.pmap.shpgperproc=500\
security.bsd.unprivileged_read_msgbuf=0\
kern.maxvnodes=400000" >> /etc/sysctl.conf

Tuning note: watch vfs.numvnodes while the server is live to get guidance on where to set maxvnodes

mount procfs

echo "proc                    /proc           procfs  rw              0       0" >> /etc/fstab
echo "linprocfs               /usr/compat/linux/proc linprocfs rw     0       0" >> /etc/fstab
echo "linsysfs                /usr/compat/linux/sys linsysfs rw       0       0" >> /etc/fstab
mkdir -p /usr/compat/linux/proc
mkdir -p /usr/compat/linux/sys

enable noatime option

data1 and data2 should look something like (add ',noatime' after 'rw'):

/dev/mfid0s1g           /mnt/data1      ufs     rw,noatime      2       2
/dev/mfid1s1d           /mnt/data2      ufs     rw,noatime      2       2

reboot. Confirm new kernel is loaded, devfs in place

uname -a

Check devfs rules

devfs rule showsets
devfs rule -s 3 show

Should see:

#  devfs rule showsets
1
2
3
4

#  devfs rule -s 3 show
100 include 1
207 path pts* unhide
217 path fd unhide
218 path fd/* unhide
300 path ttyp* unhide
301 path ttyq* unhide
302 path ttyr* unhide
303 path ttys* unhide
304 path ttyP* unhide
305 path ttyQ* unhide
306 path ttyR* unhide
307 path ttyS* unhide
400 path null unhide
500 path zero unhide
600 path random unhide
610 path urandom unhide
700 path mem unhide
710 path kmem unhide
810 path mdctl unhide
900 path stdin unhide
910 path stdout unhide
920 path stderr unhide

update ports

cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_8_3\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup

cvsup sup; mail -s 'cvs sup ports done' support@johncompanies.com < /dev/null

~24mins

Install raid mgmt tool

Perc5/i, 6/i

Pull over cli from previous system (jail9)

scp /usr/local/sbin/mega* root@10.1.4.109:/usr/local/sbin/
scp /usr/local/libexec/MegaCli root@10.1.4.109:/usr/local/libexec/MegaCli

These are linux-based tools. This will require linux base which we would normally install from ports, but since it's failing lately, we just pull in the compat libraries from another system:

rsync -aSHv --exclude=proc --exclude=sys 10.1.4.109:/usr/compat/linux/ /usr/compat/linux/

Test:

rehash; megacli ldinfo lall a0

DEPRECATED: Assuming it worked, here's how we used to install linux_base:

cd /usr/ports/emulators/linux_base-fc4
make install clean

(didnt succeed due to libtool requirement)

cd /usr/ports/distfiles
fetch http://www.lsi.com/DistributionSystem/AssetDocument/support/downloads/megaraid/miscellaneous/linux/2.00.15_Linux_MegaCLI.zip
cd /usr/ports/sysutils/linux-megacli
make install clean

(also failed due to libtool)

2850 PERC 4e/Di- no linux

cd /usr/ports/distfiles/
fetch http://backup01.best-hosting.ru/pub/FreeBSD/ports/distfiles/dr_freebsd_1.51.zip
cd /usr/ports/sysutils/megarc
make install clean
megarc -dispCfg -a0

install rsync from ports

cd /usr/ports/net/rsync
make install clean

choose default options

configure inetd to respond to mrtg load queries

echo "load    stream  tcp     nowait  user    /usr/local/jail/bin/load.pl  load.pl" >> /etc/inetd.conf
echo "load            12384/tcp" >> /etc/services

install perl

cd /usr/ports/lang/perl5.12
make install clean

choose defaults

install bb client

Compiling from source on AMD64 will not work. So, we use a linux-compiled version and rely on linux compat. Per above, linux compat won't install on 8.x - libtool 2.4 need. So, instead we copy(ed) over linux:

rsync -aSHv --exclude=proc --exclude=sys 10.1.4.108:/usr/compat/linux/ /usr/compat/linux/

adduser

Output/response:

Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]: 
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username   : bb
Password   : <random>
Full Name  : bb
Uid        : 1984
Class      :
Groups     : bb
Home       : /home/bb
Shell      : /bin/sh
Locked     : no
OK? (yes/no): yes

cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd_linuxcompat.tgz .
tar xzf bb-freebsd_linuxcompat.tgz

edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:

echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.4.103 jail3.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

Edit for machine name and private IP.

if this machine is at i2b:

echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
69.55.229.7 jail3.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

vi /home/bb/bbc1.9e-btf/ext/openfiles 

MACHINE="jail3,johncompanies,com"      # HAS TO BE IN A,B,C FORM

Edit for machine name.

cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh 
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf
./runbb.sh start
more BBOUT 
(look for errors)
exit

Put in script to start bb @ boot:

echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh

remove reserve space

cd
umount /mnt/data1
umount /mnt/data2
tunefs -m 0 /mnt/data1
tunefs -m 0 /mnt/data2
mount -a

setup rdate

cd /usr/ports/sysutils/rdate
make install clean

crontab -e
0 0 * * * /usr/local/sbin/rdate -s utcnist.colorado.edu

/usr/local/sbin/rdate -s utcnist.colorado.edu

We used to use ntpd, however it listens on jail IPs which is a security risk. So we stopped. Here's the old instructions: install new ntp from ports

/usr/ports/net/ntp
make install clean

echo "server 10.1.4.5" > /etc/ntp.conf
/usr/sbin/ntpd -p /var/run/ntpd.pid
sleep 2; ntpq -p

(confirm it’s able to reach our time server)

make a data partition

IF you didn't format the data partition during sysinstall:

Create a g partition on 2nd mirror – bsdlabel no longer works (below shows d partition made with sysinstall):

jail8 /usr/home/bb# gpart show
=>       63  285474735  mfid0  MBR  (136G)
         63  285458922      1  freebsd  [active]  (136G)
  285458985      15813         - free -  (7.7M)

=>        0  285458922  mfid0s1  BSD  (136G)
          0     524288        1  freebsd-ufs  (256M)
     524288   12582912        2  freebsd-swap  (6.0G)
   13107200     524288        4  freebsd-ufs  (256M)
   13631488     524288        5  freebsd-ufs  (256M)
   14155776    8388608        6  freebsd-ufs  (4.0G)
   22544384  262914538        7  freebsd-ufs  (125G)

=>       63  584843175  mfid1  MBR  (279G)
         63  584830197      1  freebsd  [active]  (279G)
  584830260      12978         - free -  (6.3M)

=>        0  584830197  mfid1s1  BSD  (279G)
          0   16777216        2  freebsd-swap  (8.0G)
   16777216  568052981        4  freebsd-ufs  (271G)

jail8 /usr/home/bb# gpart show mfid1s1
=>        0  584830197  mfid1s1  BSD  (279G)
          0   16777216        2  freebsd-swap  (8.0G)
   16777216  568052981        4  freebsd-ufs  (271G)

# gpart list mfid1s1

Geom name: mfid1s1
fwheads: 255
fwsectors: 63
last: 584830196
first: 0
entries: 8
scheme: BSD
Providers:
1. Name: mfid1s1b
   Mediasize: 8589934592 (8.0G)
   Sectorsize: 512
   Mode: r1w1e0
   rawtype: 1
   length: 8589934592
   offset: 0
   type: freebsd-swap
   index: 2
   end: 16777215
   start: 0
2. Name: mfid1s1d
   Mediasize: 290843126272 (271G)
   Sectorsize: 512
   Mode: r0w0e0
   rawtype: 7
   length: 290843126272
   offset: 8589934592
   type: freebsd-ufs
   index: 4
   end: 584830196
   start: 16777216
Consumers:
1. Name: mfid1s1
   Mediasize: 299433060864 (279G)
   Sectorsize: 512
   Mode: r1w1e1

# gpart delete -i 4 mfid1s1
mfid1s1d deleted
jail8 /usr/home/bb# gpart list mfid1s1
Geom name: mfid1s1
fwheads: 255
fwsectors: 63
last: 584830196
first: 0
entries: 8
scheme: BSD
Providers:
1. Name: mfid1s1b
   Mediasize: 8589934592 (8.0G)
   Sectorsize: 512
   Mode: r1w1e0
   rawtype: 1
   length: 8589934592
   offset: 0
   type: freebsd-swap
   index: 2
   end: 16777215
   start: 0
Consumers:
1. Name: mfid1s1
   Mediasize: 299433060864 (279G)
   Sectorsize: 512
   Mode: r1w1e1

# gpart show mfid1s1
=>        0  584830197  mfid1s1  BSD  (279G)
          0   16777216        2  freebsd-swap  (8.0G)
   16777216  568052981           - free -  (271G)

# gpart add  -t freebsd-ufs -i 7 mfid1s1
mfid1s1g added

# gpart show mfid1s1
=>        0  584830197  mfid1s1  BSD  (279G)
          0   16777216        2  freebsd-swap  (8.0G)
   16777216  568052981        7  freebsd-ufs  (271G)

Here's how we USED to do it with bsdlabel:

bsdlabel -e /dev/mfid0s1

given:

# /dev/aacd0s1:
8 partitions:
#        size   offset    fstype   [fsize bsize bps/cpg]
  a:   262144        0    4.2BSD     2048 16384 16392
  b:  4194304   262144      swap
  c: 143363997        0    unused        0     0         # "raw" part, don't edit
  d:   524288  4456448    4.2BSD     2048 16384 32776
  e:   524288  4980736    4.2BSD     2048 16384 32776
  f:  6291456  5505024    4.2BSD     2048 16384 28552

new offset = 6291456 + 5505024 = 11796480
new size is size for 'c' partition minus the new start from above
143363997 - 11796480 = 131567517

So:

g: 131567517 11796480 unused 0 0

create the jail template

cd /usr/ports/sysutils/jailutils make install clean

Create an md device to hold the jail:

touch /mnt/data1/jail-template20g
mdconfig -a -t vnode -s 20g -f /mnt/data1/jail-template20g -u 0
newfs -O 1 /dev/md0
mkdir /mnt/data1/jail-DIR
mount /dev/md0 /mnt/data1/jail-DIR

Build world into the jail:

cd /usr/src
make world DESTDIR=/mnt/data1/jail-DIR; taskdone

~1hr

Make /etc into the jail, mount dev, copy in jkill:

cd etc
make distribution DESTDIR=/mnt/data1/jail-DIR
mount -t devfs devfs /mnt/data1/jail-DIR/dev
devfs -m /mnt/data1/jail-DIR/dev rule -s 3 applyset 
cd /mnt/data1/jail-DIR
ln -sf dev/null kernel
cp /usr/local/sbin/jkill /mnt/data1/jail-DIR/sbin

Enter into jail to do configuration:

jail /mnt/data1/jail-DIR testhostname 192.168.11.100 /bin/sh
csh

Create fstab:

touch /etc/fstab
echo 'network_interfaces=""\
hostname="newsystem"\
kern_securelevel_enable="NO"\
sendmail_enable="YES"\
sshd_enable="YES"' > /etc/rc.conf

echo "nameserver 69.55.225.225\
nameserver 69.55.230.3" >> /etc/resolv.conf

Edit crontab:

vi /etc/crontab

remove the adjkerntz lines comment out periodic’s and put this line above them:

# IF YOU UNCOMMENT THESE, PLEASE ADJUST THEIR RUN TIME

rm -rf /etc/periodic/daily/400.status-disks

check and remove any crap in /tmp

vi /etc/periodic/security/100.chksetuid

replace: MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`
with: MP='/' (use single quotes)

mkdir -p /usr/compat/linux/dev

Add account for user. Output/response:

adduser

Username: user
Full name: user
Uid (Leave empty for default):
Login group [user]:
Login group is user. Invite user into other groups? []: wheel
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/user]:
Home directory permissions (Leave empty for default):
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: y
Lock out the account after creation? [no]:
Username   : user
Password   : <random>
Full Name  : user
Uid        : 1001
Class      :
Groups     : user
Home       : /home/user
Home Mode  :
Shell      : /bin/sh
Locked     : no
OK? (yes/no): y
adduser: INFO: Successfully added (user) to the user database.
adduser: INFO: Password for (user) is: 901gmYjO
Add another user? (yes/no): n
Goodbye!

set TERM:

vi /usr/home/user/.profile 
TERM=vt100;     export TERM

Set time zone to PT:

tzsetup

Reload aliases:

newaliases

Replace reboot/halt:

rm /sbin/halt /sbin/reboot
ln /sbin/jkill /sbin/halt
ln /sbin/jkill /sbin/reboot

Redirect console output, comment out console and move to /var/log/messages:

vi /etc/syslog.conf
#*.err;kern.warning;auth.notice;mail.crit               /dev/console
*.err;kern.warning;auth.notice;mail.crit                /var/log/messages

exit
exit

Lib32 compat library:

cd libexec
chflags noschg ld-elf32.so.1
chflags noschg ld-elf.so.1
mv ld-elf32.so.1 ld-elf32.so.1-orig
ln ld-elf.so.1 ld-elf32.so.1
chflags schg ld-elf.so.1
chflags schg ld-elf32.so.1

Replace traceroute:

mv /mnt/data1/jail-DIR/usr/sbin/traceroute /mnt/data1/jail-DIR/usr/sbin/_traceroute
echo '#\!/bin/sh\
/usr/sbin/_traceroute -i bce0 $1' >> /mnt/data1/jail-DIR/usr/sbin/traceroute 
chmod +x /mnt/data1/jail-DIR/usr/sbin/traceroute

Modify 'bce0' to reflect whichever nic is public on this hardware.

Clean out ports before copying into the jail (~30mins):

cd /usr/ports
make -DNOCLEANDEPENDS clean 
rm -fr /usr/ports/distfiles/*
cp -r /usr/ports /mnt/data1/jail-DIR/usr

rm /mnt/data1/jail-DIR/root/.history

Umount the jail and dump it:

cd /
umount /mnt/data1/jail-DIR/dev
dump -0a -f /usr/local/jail/template/template /dev/md0
umount /dev/md0
rmdir /mnt/data1/jail-DIR
mdconfig -d -u 0

setup backups

echo '#\!/bin/sh\
backupdir=/data/jail3\
server=backup1\
\
## ENTRY /etc\
## ENTRY /usr/local/etc\
## ENTRY /usr/local/jail\
## ENTRY /root/logs' > /usr/local/jail/bin/backup.config

Edit to reflect backup server and jail hostname

On backup server, setup backup dirs:

ssh backup1 mkdir -p /data/jail3/0

On backup server, setup backup dirs:

backup1# vi /usr/local/sbin/snapshot_rotate

Edit /usr/local/jail/bin/backup.xxx to use the right drives and copy to /usr/local/jail/bin/backup

vi /usr/local/jail/bin/backup.md

adjust df so it includes all relevant drives, currently df > /etc/df.bak is fine. Also, make sure the binary/script source is pulling from the right dir on backup2:

/usr/local/bin/rsync -a backup2:/mnt/data4/bin/freebsd8.x/ /usr/local/jail/bin/

cp /usr/local/jail/bin/backup.md /usr/local/jail/bin/backup

create /root/logs

mkdir /root/logs

edit sshd_config

vi /etc/ssh/sshd_config
ListenAddress 69.55.229.7
ListenAddress 10.1.2.103

Adjust to pub/private IP for jail.

Restart sshd:

kill -1 `cat /var/run/sshd.pid`

add crontab entries

crontab -e
5 0 * * * /usr/local/jail/bin/backup
1 0 1 * * /usr/local/jail/bin/ipfwreset
0 18 * * * /usr/local/jail/bin/ipfwbackup
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /usr/local/jail/bin/trafstats
0 0,6,12,18 * * * /usr/local/jail/bin/sync_jail_names

On 2950:

*/5 * * * * /usr/local/jail/bin/perc5iraidchk

On 3ware-based servers:

0 0 * * * /usr/local/jail/bin/3wraidchk

Reboot notify script

ln -s /usr/local/jail/bin/notify.sh /usr/local/etc/rc.d/notify.sh

patch jail against starting jails with rtprio

mv /usr/sbin/jail /usr/sbin/jail_
echo '#\!/bin/sh\
/usr/sbin/rtprio -t /usr/sbin/jail_ $*' > /usr/sbin/jail
chmod +x /usr/sbin/jail

make sure mail works

If there are map errors:

cd /etc/mail; make maps

recover space on /usr (optional)

rm -fr /usr/obj

wrapper for jps

mv /usr/local/sbin/jps /usr/local/sbin/jps_

wrapper for jls

mv /usr/sbin/jls /usr/sbin/jls_

wrapper for jexec

mv /usr/sbin/jexec /usr/sbin/jexec_

install jtop

cd /usr/ports/sysutils/jtop
make install clean

block jails from reaching private net

echo 'ipfw add 1 deny ip from 69.55.224.0/20 to 10.1.4.0/24' > /usr/local/etc/rc.d/ipfw.sh
chmod 0700 /usr/local/etc/rc.d/ipfw.sh

add to management infrastructure

mail

add to management db

tables: jc.ref_machines and jc.ref_templates

on jail run:

uname -r

Which shows something like:

8.3-RELEASE-p2

Insert into db:

insert into ref_machines values (null,'jail3','mx3.johncompanies.com',0,'f8');
select machine_id from ref_machines where host='jail3';
+------------+
| machine_id |
+------------+
|         35 |
+------------+
insert into ref_templates values ('','8.3-RELEASE-jc2',35,'FreeBSD 8.3',1);

add to bb server

vi /usr/home/bb/bbsrc/bb1.9i-btf/etc/bb-hosts
10.1.4.109 jail9.johncompanies.com # ssh

In the case of an i2b server, use real ip:

69.55.229.7 jail3.johncompanies.com # ssh

su bb
cd
bbsrc/bb/runbb.sh restart ; exit

Update backupgraph

vi /usr/local/www/mgmt/cgi/backupgraph.pl

(add hostname)

Update load mrtg

vi /usr/local/www/mgmt/mrtg/mrtg1.cfg

(add new entry to file following existing format)

ns1c

fwd and reverse lookups:

vr johncompanies.com

vi internal.johncompanies.com
rndc reload johncompanies.com IN private

ptr 69.55.227.x

wiki

add to server/cabinet map

firewall

add an outside blocking rule to the firewall, so this machine can only be reached from inside the firewall. Follow example already in firewall jail17 is:

00117 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.228.200
00117 deny ip from any to 69.55.228.200

jail19 would be 00119...

ipfw add 00109 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 } to 69.55.232.3
ipfw add 00109 deny ip from any to 69.55.232.3

select some customers for castle probe map

4.11

4.11
Last updated 2006-1-26

All time extimates below assume disks aren’t scrubbing

1. make sure bios is setup for bios console redirect
Supermicro:
Console redirection:
Com port addr: on-board COM A
Baud: 38400
Console type: vt100
Flow control: none
Console connection: direct
Continue cr after post: off

2450:
Make sure running bios A09
Console Redirection: VT100/VT220
	
2. assuming mirrors (or at least disks) created (if not, refer to this), boot to disk 1 of 4.11 
skip kernel config (enter)
custom install
partition
move cursor to aacd0, hit space (takes you to partition map screen)
a for entire disk
q to quit and save
standard mbr (no boot manager)
space to unselect aacd0
cursor over aacd1
space
a for entire disk
q to quit and save
none (leave untouched)

cursor over aacd0
space
(takes you into part. Screen again) q to exit
none
Make sure both are checked and tab to ok

label
Make sure aacd0 is highlighted
a to start with defaults

john likes:
/1g
/var 256m

Glenn likes:
/128
/var 256
/usr 3g
/tmp 256

We do:
delete everything
/ 128M
swap 4G
/var 256M
/tmp 256M
/usr 3G
/mnt/data1 remaining space

Make sure to toggle S for soft updates on all (should look like UFS+S Y under the Newfs column)
Set the noatime option

move cursor to aacd1
swap 4G
/mnt/data2 remaining space
Set the noatime option

q to save and exit

distributions
developer
yes to install ports
exit
media
cd
commit
yes
(2450: 16mins, supermicro: 11mins)
yes to "visit general config"
Set root pwd
Add user ‘user’ member group is wheel, set password
Set tz
Networking->interfaces->Fxp0
No IPV6
dhcp=yes
Set hostname & domain

Startup services:
Disble usbd

exit...
exit install
yes
take the cd out and let the machine reboot

3. put some temp settings in /etc/rc.conf:
usbd_enable="NO"
sendmail_enable="NO"

4. reboot

5. double check the date/time

6. edit /etc/make.conf (only add the console speed line if this is a supermicro capable of outputting BIOS redirect at that speed – 2450’s can only do 9600)
cat >> /etc/make.conf
WITHOUT_X11=yes
KERNCONF=jail18
BOOT_COMCONSOLE_SPEED=38400

7. install cvsup
cd /usr/ports/net/cvsup-without-gui 
make install clean (stay close for gettext options, 2450: 21mins, supermicro: 14mins)
rehash

8. get latest sources for this release:
cd /usr/src 
cat > sup
*default host=cvsup4.freebsd.org
*default base=/usr
*default prefix=/usr
*default release=cvs tag=RELENG_4_11
*default delete use-rel-suffix
*default compress
src-all

cvsup sup (2450, 4.10: 13mins, supermicro, 4.11: 11mins)

9. populate hosts
cat >> /etc/hosts
10.1.4.3	backup2

10. put key in authorized_keys on backup2
ssh-keygen -t dsa -b 1024 (default location, leave password blank)
scp /root/.ssh/id_dsa.pub user@backup2:/tmp/jail18pub
on backup2:
cat /tmp/jail18pub >> /root/.ssh/authorized_keys

confirm that you can ssh to backup2 without getting a login prompt

11. configure new kernel. Get config from similar machine or there may be a master copy somewhere under /mnt/data4/build (name the kernel config the same as the jail, ex jail4):
cd /usr/src/sys/i386/conf 
scp backup2:/mnt/data4/build/freebsd/kern_config-4.11 ./jail18

edit the kernel config and change ident to be the name of the jail:
ident           jail4

IMPORTANT CUSTOMIZATION:
for machines with >4G RAM, add to the config:
options         PAE

for supermicro mobo’s with broadcom nics, add to the config:
device          bge             # Broadcom BCM570x (``Tigon III'')

for machines where lots of postgres might be running, change SHMMAXPGS:
options         SHMMAXPGS=40960

edit /sys/conf/newvers.sh to add –jc2 to the end of the BRANCH string (RELEASE-p9-jc2)

12. bring over patches from backup2
The various patches are in /mnt/data4/build/freebsd/patches on backup2. There are dirs for each version. Not all dirs are populated, but patches for later versions work on older ones unless there is a new patch in the older dir.
cd /usr/src
scp backup2:"/mnt/data4/build/freebsd/patches/4.11/*" .
jail_proc_patch      100% |*************************************************|  2593    00:00
restore-patch        100% |*************************************************|  3295    00:00
411ps-jail-patch     100% |*************************************************|  2602    00:00
jail_rtprio_patch    100% |*************************************************|   301    00:00
udp-patch            100% |*************************************************|   594    00:00

Apply patches:
patch < 411ps-jail-patch
patch < jail_proc_patch
patch < restore-patch
patch < jail_rtprio_patch
patch < udp-patch

13. build, install kernel and world
make buildworld buildkernel installkernel (2450: 48min, supermicro: 20mins)
(Any compile errors can be looked up in /usr/include/sys/signal.h, other errors, do a rm -R /usr/obj/*)
make installworld (2450: 2min, supermicro: 1mins)
mergemaster -i
(answer no to most of it)

14. reboot. Confirm new kernel is loaded (uname -a)

15. update ports:
cd /usr/ports
cat > sup
*default host=cvsup4.FreeBSD.org
*default base=/usr
*default prefix=/usr
*default release=cvs tag=RELENG_4
*default delete use-rel-suffix
*default compress
ports-all tag=.

cvsup sup (2450: 26mins, supermicro: 26mins)

16. add console="comconsole" to /boot/loader.conf
cat >> /boot/loader.conf
console="comconsole"

17. edit /etc/ttys and turn off all ttyv's except 0 and 1
also turn on ttyd0, change type to vt100:
ttyd0   "/usr/libexec/getty std.9600"  vt100   on secure

kill -1 1

on console:
vi /etc/remote
(rename port to jail4 depending on where and which digi plugged into)
test serial console

18. install linux_base:
cd /usr/ports/emulators/linux_base
make install clean (2450: 7min, supermicro: 2mins)

reibuild rpmdb cause we had probs installing aacapps
cd /compat/linux/bin
./rpm --initdb
./rpm --rebuilddb

19. install aacapps-4.1-0.i386.rpm
scp backup2:/mnt/data4/build/freebsd/aacapps-4.1-0.i386.rpm /tmp/.
/compat/linux/bin/rpm -ivh /tmp/aacapps-4.1-0.i386.rpm
cd /dev
sh MAKEDEV aac0

test out /compat/linux/usr/sbin/aaccli

20. edit root's path and login script:
vi /root/.cshrc
add to path: /usr/local/jail/bin /compat/linux/usr/sbin

Change alias entries (add G):
alias la        ls -aG
alias lf        ls -FAG
alias ll        ls -lAG
alias ls        ls -AG
alias mbm       mb mount
alias mbu       mb umount

and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "

at the bottom of the file add:
set sshtty=`who am i|awk '{print $2}'`
/usr/sbin/rtprio 3 -`ps auxwJ | grep $sshtty | awk '{print $2}'`

set shortty=`who am i | awk '{print $2}' | sed -E 's/.*(..)$/\1/'`
foreach x (`ps cauxJ | grep sh | grep $shortty | awk '{print $2}'`)
/usr/sbin/rtprio 2 -$x
end
 
21. install rsync from ports
cd /usr/ports/net/rsync
make install clean

22. create & populate binaries/scripts dirs
mkdir -p /usr/local/jail/bin
mkdir -p /usr/local/jail/rc.d
scp backup2:"/mnt/data4/bin/freebsd/*" /usr/local/jail/bin
cd /usr/local/jail/rc.d/
touch quad1
touch quad2
touch quad3
touch quad4
touch safe1
touch safe2
touch safe3
touch safe4
chmod +x *
cd /usr/local/jail/bin
ln -s /usr/local/jail/rc.d/quad1 quad1
ln -s /usr/local/jail/rc.d/quad2 quad2
ln -s /usr/local/jail/rc.d/quad3 quad3
ln -s /usr/local/jail/rc.d/quad4 quad4
ln -s /usr/local/jail/rc.d/safe1 safe1
ln -s /usr/local/jail/rc.d/safe2 safe2
ln -s /usr/local/jail/rc.d/safe3 safe3
ln -s /usr/local/jail/rc.d/safe4 safe4
rehash

23. configure inetd to respond to mrtg load queries
cat >> /etc/inetd.conf
load    stream  tcp     nowait  user    /usr/local/jail/bin/load.pl  load.pl

cat >> /etc/services
load            12384/tcp

kill -HUP `cat /var/run/inetd.pid`

24. configure load mrtg, on mail
vi /usr/local/etc/mrtg/mrtg1.cfg
(add new entry to file following existing format)

25. configure bb on mail:
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
10.1.4.104 jail4.johncompanies.com # ssh

su bb
cd /usr/home/bb/bbsrc/bb1.9e-btf
./runbb.sh stop
./runbb.sh start
exit

26. install bb client
adduser -group 1984 -shell /bin/csh -uid 1984 bb
cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd.tar .
tar xvf bb-freebsd.tar

cat > /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
10.1.4.105 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh
10.1.4.118 jail18.johncompanies.com # ssh

vi /home/bb/bbc1.9e-btf/ext/openfiles   and change:
MACHINE="jail18,johncompanies,com"      # HAS TO BE IN A,B,C FORM

cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
./bbchkhosts.sh (ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd bbc1.9e-btf/
./runbb.sh start
more BBOUT (look for errors)
exit

cat > /usr/local/etc/rc.d/bb.sh
su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"

chmod +x /usr/local/etc/rc.d/bb.sh

27. remove reserve space, enable softupdates (probably already set, so not necessary)
cd
umount /mnt/data1
umount /mnt/data2
tunefs -m 0 /mnt/data1
tunefs -m 0 /mnt/data2

mount -a

28. configure ntp
cat > /etc/ntp.conf
server 10.1.4.105

/usr/sbin/ntpd -p /var/run/ntpd.pid
ntpq -p
(confirm it’s able to reach our time server)

29. mrtg switch graphs

31. fwd and reverse lookups on ns1c
vi johncompanies.com
rr johncompanies.com
vi internal.johncompanies.com
rndc reload johncompanies.com IN private
(edit the PTR too)

32. create all /dev/vn and /dev/pty files in /dev
cat > /tmp/runme.sh
#!/bin/sh
cd /dev
for i in 1 2 3 4 5 6 7 ; do sh MAKEDEV pty$i ; done
for i in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 ; do sh MAKEDEV vn$i ; done

NOTE: there are only 4 lines in the above output - however the lines may be wrapped on your terminal - make sure that there are only four lines in the script you make.

chmod +x /tmp/runme.sh
/tmp/runme.sh

rm /tmp/runme.sh

ls /dev/vn*|wc -l (make sure the output is a high number – larger than 128)
ls /dev/pty*|wc -l (make sure the output is exactly 256)

33. populate /etc/rc.conf with IPs and NFS settings
vi /etc/rc.conf

kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="NO"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
sshd_enable="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.4.104"

ifconfig_xl0="inet 10.1.4.104 netmask 255.255.255.0"
ifconfig_fxp0="inet 69.55.228.101 netmask 255.255.255.0"
defaultrouter="69.55.228.1"
ifconfig_fxp0_alias0="inet 69.55.2xx.xx netmask 255.255.255.0"

static_routes="t1 office"
route_t1="-net 10.1.5 10.1.4.2"
route_office="-net 10.1.6 10.1.4.2"

34. make sure sysctls are set and preserved after reboot
cat >> /etc/sysctl.conf
kern.consmute=0
jail.sysvipc_allowed=1
kern.ipc.shm_use_phys=1
kern.ipc.shmall=65535
kern.ipc.shmmax=134217728
net.inet.tcp.syncookies=0
kern.maxfiles=32768
kern.fallback_elf_brand=3
kern.maxprocperuid=4000
jail.max_procs_per_jail: 1026

35. reboot

36. create the jail template
vnconfig -T -S 1g -Z -s labels -c /dev/vn1 /mnt/data1/jail
disklabel -r -w vn1 auto
newfs /dev/vn1c
cd /usr/src
mkdir -p /mnt/data1/jail-DIR
mount /dev/vn1c /mnt/data1/jail-DIR
make world DESTDIR=/mnt/data1/jail-DIR (2450: 45mins, supermicro: 19mins)
cd etc
make distribution DESTDIR=/mnt/data1/jail-DIR -DNO_MAKEDEV_RUN
cd /mnt/data1/jail-DIR/dev
sh MAKEDEV jail
cd /mnt/data1/jail-DIR
ln -sf dev/null kernel

jail /mnt/data1/jail-DIR testhostname 192.168.11.100 /bin/sh
csh
touch /etc/fstab
cat > /etc/rc.conf
portmap_enable="NO"
network_interfaces=""
hostname="newsystem"
kern_securelevel_enable="NO"
sendmail_enable="YES"
sshd_enable="YES"

cat >> /etc/resolv.conf
nameserver 69.55.225.225
nameserver 69.55.230.3

vi /etc/crontab
remove the adjkerntz lines
comment out periodic’s and put this line above them:
# DO NOT UNCOMMENT THESE

rm -rf /etc/periodic/daily/400.status-disks

mv /bin/df /bin/df_

cat > /bin/df
#!/bin/sh
/bin/df_ $* .

chmod +x /bin/df

cat > /sbin/mount
echo `df | tail -1 | awk '{print $1 " on " $6 " (ufs, local)"}'`

check /tmp for crap

vi /etc/periodic/security/100.chksetuid
replace: MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`
with: MP='/'

mkdir -p /usr/compat/linux/dev

adduser (Add account for user)

put user in wheel group
vi /etc/group

vi /usr/home/user/.profile (and add to the file):
TERM=vt100;     export TERM

cd /etc
vipw -d .
root:$1$krszPxhk$xkCepSnz3mIikT3vCtJCt0:0:0::0:0:Charlie &:/root:/bin/csh
user:$1$Mx9p5Npk$QdMU6c8YQqp2FW2M3irEh/:1001:1001::0:0:User &:/home/user:/bin/sh

tzsetup

newaliases 

chflags schg /dev/*mem

cd /dev
rm console
ln -s null console

exit
exit

cd /usr/ports
make -DNOCLEANDEPENDS clean (2450: 47mins , supermicro: 17mins)
rm -fr /usr/ports/distfiles/*
cp -r /usr/ports /mnt/data1/jail-DIR/usr

cd /mnt/data1/jail-DIR/dev
rm kmem
mknod kmem c 2 1 root:kmem
chmod 640 kmem
rm mem
mknod mem c 2 0 root:kmem
chmod 640 mem

rm /mnt/data1/jail-DIR/root/.history

sh
for i in 1 2 3 4 5 6 7 ; do sh MAKEDEV pty$i ; done
exit

cd /mnt/data1/jail-DIR/usr/compat/linux/dev
mknod null c 2 2
mknod random c 2 3

cd 
mkdir /usr/local/jail/template/
dump -0a -f /usr/local/jail/template/411template /dev/vn1

umount /dev/vn1c
vnconfig -u /dev/vn1

rm /mnt/data1/jail
rm -fr /mnt/data1/jail-DIR

37. setup backups
cat > /usr/local/jail/bin/backup.config
#!/bin/sh
backupdir=/mnt/data2/jail18_rsync

## ENTRY /etc
## ENTRY /usr/local/etc
## ENTRY /usr/local/jail
## ENTRY /root/logs

on backup2:
setup backup dirs:
mkdir -p /mnt/data2/jail18_rsync/0_today
mkdir -p /mnt/data2/jail18_rsync/1_yesterday
mkdir -p /mnt/data2/jail18_rsync/2_two_day

add the system to /mnt/data1/bin/rsync_houseclean 

on mail:
vi /usr/local/www/mgmt/cgi/backupgraph.pl
(add hostname)

38. mkdir /root/logs

39. edit sshd_config for security
vi /etc/ssh/sshd_config
ListenAddress 69.55.228.101
ListenAddress 10.1.4.104

kill -1 `cat /var/run/sshd.pid`

40. add crontab entries
crontab -e
5 0 * * * /usr/local/jail/bin/backup
1 0 1 * * /usr/local/jail/bin/ipfwreset
0 18 * * * /usr/local/jail/bin/ipfwbackup
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /usr/local/jail/bin/trafstats
55 10,23 * * * /usr/local/jail/bin/trafficwatch.pl


41. Reboot notify script
cat > /usr/local/etc/rc.d/notify.sh
echo "`/bin/hostname` rebooted" | /usr/bin/mail reboot@johncompanies.com

chmod +x /usr/local/etc/rc.d/notify.sh

42. copy jailmake from prev system
scp user@10.1.4.118:/usr/local/jail/bin/jailmake /usr/local/jail/bin
rehash
NOTE: remove df altering code from jailmake since we put the correct df in the template, and make sure path to template file is right

43. add to templates via mgmt system

44. add to server/cabinet map. On mail:
vi /usr/local/www/mgmt/html/cabinetmap.html

45. add an outside blocking rule to the firewall, so this machine can only be reached from inside the firewall. Follow example already in firewall jail17 is:
 
00117 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.228.2
00117 deny ip from any to 69.55.228.2

jail4 would be 00104...
ipfw add 00118 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.228.140
ipfw add 00118 deny ip from any to 69.55.228.140

46. select customers for probe map

6.1

Last updated 2006-05-09

All time extimates below assume disks aren’t scrubbing

1. make sure bios is setup for bios console redirect
Supermicro:
Console redirection:
Com port addr: on-board COM A
Baud: 38400
Console type: vt100
Flow control: none
Console connection: direct
Continue cr after post: off

2450:
Make sure running bios A09
Console Redirection: VT100/VT220

2. assuming mirrors (or at least disks) created (if not, refer to this), boot to disk 1 of 5.4
skip kernel config (enter)
custom install

partition ->
move cursor to amrd0, hit space (takes you to partition map screen)
a for entire disk
q to quit and save
standard mbr (no boot manager)
space to unselect aacd0
cursor over aacd1
space
a for entire disk
q to quit and save
none (leave untouched)

cursor over aacd0
space
(takes you into part. Screen again) q to exit
none
Make sure both are checked and tab to ok

Label ->
Make sure aacd0 is highlighted
a to start with defaults

john likes:
/1g
/var 256m

Glenn likes:
/128
/var 256
/usr 3g
/tmp 256

We do:
delete everything
/ 128M
swap 2G
/var 256M
/tmp 256M
/usr 3G
/mnt/data1 remaining space

Make sure to toggle S for soft updates on all (should look like UFS2+S Y under the Newfs column)

move cursor to aacd1
swap 2G
/mnt/data2 remaining space

q to save and exit

distributions ->
developer
yes to install ports
exit

media ->
cd

commit ->
yes
(2450: 14mins, supermicro: 12mins)

yes to "visit general config" ->
Set root pwd

Add user ‘user’ member group is wheel, set password

Set tz

Networking->interfaces->Fxp0 ->
No IPV6
dhcp=no
Set hostname & domain
Enable sshd

exit...
exit install ->
yes

take the cd out and let the machine reboot
when it comes back up, enter junk when it asks for key seed


3. double check the date/time

4. edit /etc/make.conf (only add the console speed line if this is a supermicro capable of outputting BIOS redirect at that speed – 2450’s can only do 9600)
echo "WITHOUT_X11=yes \
KERNCONF=jail19 \
BOOT_COMCONSOLE_SPEED=38400" >> /etc/make.conf

5. add console="comconsole" to /boot/loader.conf
echo "console=""comconsole""" >> /boot/loader.conf

6. turn off all ttyv's except 0 and 1 in /etc/ttys
also turn on ttyd0, change type to vt100:
vi /etc/ttys
ttyd0   "/usr/libexec/getty std.9600"  vt100   on secure

kill -1 1

on console server:
vi /etc/remote
(rename port to jail18 depending on where and which digi plugged into)
test serial console

7. install cvsup
cd /usr/ports/net/cvsup-without-gui 
make install clean; rehash
(stay close for gettext options, 2450: 27mins, supermicro: 17mins)

8. get latest sources for this release:
cd /usr/src 
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_1\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

cvsup sup 
(2450, ~12mins, supermicro, 27mins)

9. populate hosts
echo "10.1.4.3 backup2" >> /etc/hosts

10. put key in authorized_keys on backup2
cd
ssh-keygen -t dsa -b 1024 
(default location, leave password blank)
scp /root/.ssh/id_dsa.pub user@backup2:/tmp/jailkey

cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'

confirm that you can ssh to backup2 without getting a login prompt

ssh backup2

11. configure new kernel. Get config from similar machine or there may be a master copy somewhere under /mnt/data4/build/freebsd (name the kernel config the same as the jail, ex jail18):
cd /usr/src/sys/i386/conf 
scp backup2:/mnt/data4/build/freebsd/kern_config-6.1 ./jail19

edit the kernel config and change ident to be the name of the jail:
vi jail14
ident           jail14

edit /sys/conf/newvers.sh to add –jc1 to the end of the BRANCH string (RELEASE-jc1)
vi /sys/conf/newvers.sh

12. bring over patches from backup2
The various patches are in /mnt/data4/build/freebsd/patches on backup2. There are dirs for each version. Not all dirs are populated, but patches for later versions work on older ones unless there is a new patch in the older dir.
cd /usr/src
scp backup2:"/mnt/data4/build/freebsd/patches/6.1/*" .
jail_proc_patch      100% |*************************************************|  2593    00:00
restore-patch        100% |*************************************************|  3295    00:00
54ps-jail-patch      100% |*************************************************|  2602    00:00
jail_rtprio_patch    100% |*************************************************|   301    00:00
udp-patch            100% |*************************************************|   594    00:00

Apply patches:
patch < 54ps-jail-patch
patch < jail_proc_patch
patch < restore-patch
patch < jail_rtprio_patch
patch < udp-patch

13. build, install kernel and world
cd /usr/src
make buildworld buildkernel installkernel 
(2450: 1:56min, supermicro::59mins)
make installworld 
(2450: 3min, supermicro: 1min)
mergemaster -i
delete /var/tmp/temproot
delete bsnmpd
delete temporary ./etc/hosts
delete temporary ./etc/motd
delete /var/tmp/temproot

14. reboot. Confirm new kernel is loaded

uname –a

15. update ports:
cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_1\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup

cvsup sup 
(2450: 18mins, supermicro: 19mins)

18. (only applies if adaptec card installed)
install linux_base:
cd /usr/ports/emulators/linux_base
make install clean 
(2450: 7min, supermicro: 3mins)

reibuild rpmdb cause we had probs installing aacapps
cd /compat/linux/bin
./rpm --initdb
./rpm --rebuilddb

install aacapps-4.1-0.i386.rpm
scp backup2:/mnt/data4/build/freebsd/aacapps-4.1-0.i386.rpm /tmp/.
/compat/linux/bin/rpm -ivh --excludepath=/dev /tmp/aacapps-4.1-0.i386.rpm

test out;
/compat/linux/usr/sbin/aaccli


20. edit root's path and login script:
vi /root/.cshrc
add to path: 
/usr/local/jail/bin 
(if adaptec card installed, also add /compat/linux/usr/sbin)

Change alias entries (add G):
alias la        ls -aG
alias lf        ls -FAG
alias ll        ls -lAG
alias ls        ls -AG
alias mbm       mb mount
alias mbu       mb umount

and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "

at the bottom of the file add:
set sshtty=`who am i|awk '{print $2}'`
/usr/sbin/rtprio 3 -`psj | grep $sshtty | awk '{print $2}'`

set shortty=`who am i | awk '{print $2}' | sed -E 's/.*(..)$/\1/'`
foreach x (`psj | grep sh | grep $shortty | awk '{print $2}'`)
/usr/sbin/rtprio 2 -$x
end
 
21. install rsync from ports
cd /usr/ports/net/rsync
make install clean

choose default options

21. install perl from ports
PROB NOT NECESSARY – INSTALLED WITH LINUX I THINK
cd /usr/ports/lang/perl5.8/
make install clean; rehash
(supermicro: 5min)

22. create & populate binaries/scripts dirs
mkdir -p /usr/local/jail/bin
mkdir -p /usr/local/jail/rc.d
mkdir /mnt/data1
mkdir /mnt/data2
scp backup2:"/mnt/data4/bin/freebsd6.x/*" /usr/local/jail/bin
cd /usr/local/jail/rc.d/
touch quad1
touch quad2
touch quad3
touch quad4
touch safe1
touch safe2
touch safe3
touch safe4
chmod +x *
cd /usr/local/jail/bin
ln -s /usr/local/jail/rc.d/quad1 quad1
ln -s /usr/local/jail/rc.d/quad2 quad2
ln -s /usr/local/jail/rc.d/quad3 quad3
ln -s /usr/local/jail/rc.d/quad4 quad4
ln -s /usr/local/jail/rc.d/safe1 safe1
ln -s /usr/local/jail/rc.d/safe2 safe2
ln -s /usr/local/jail/rc.d/safe3 safe3
ln -s /usr/local/jail/rc.d/safe4 safe4

rehash

23. configure inetd to respond to mrtg load queries
echo "load    stream  tcp     nowait  user    /usr/local/jail/bin/load.pl  load.pl" >> /etc/inetd.conf

echo "load            12384/tcp" >> /etc/services

26. install bb client
adduser
cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd.tar .
tar xvf bb-freebsd.tar

edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
echo "10.1.4.105 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.4.103 jail3.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

vi /home/bb/bbc1.9e-btf/ext/openfiles 
MACHINE="jail19,johncompanies,com"      # HAS TO BE IN A,B,C FORM

cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh (y to questions)
./bbchkhosts.sh (ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src
make; make install
cd ..
./runbb.sh start
more BBOUT 
(look for errors)
exit

echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh

modify bb-msgtab to look for raid failures

24. configure load mrtg, on mail
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg 
(add new entry to file following existing format)

25. configure bb on mail:
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
10.1.4.119 jail19.johncompanies.com # ssh

su bb
cd
bbsrc/bb/runbb.sh restart ; exit

27. remove reserve space, enable softupdates (probably already set, so not necessary)
NOT APPLICABLE IF USING GVINUM
cd
umount /mnt/data1
umount /mnt/data2
tunefs -m 0 /mnt/data1
tunefs -m 0 /mnt/data2
mount -a

28. configure ntp
echo "server 10.1.4.105" > /etc/ntp.conf

/usr/sbin/ntpd -p /var/run/ntpd.pid
sleep 2; ntpq -p
(confirm it’s able to reach our time server)

29. mrtg switch graphs

31. fwd and reverse lookups on ns1c
vr johncompanies.com
vi internal.johncompanies.com
rndc reload johncompanies.com IN private
(edit the PTR too)

33. populate /etc/rc.conf with IPs and NFS settings
vi /etc/rc.conf

kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
sshd_enable="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.4.119"
devfs_system_ruleset="devfsrules_show_all"

ifconfig_xl0="inet 10.1.4.118 netmask 255.255.255.0"
ifconfig_fxp0="inet 69.55.228.101 netmask 255.255.255.0"
defaultrouter="69.55.228.1"
ifconfig_fxp0_alias0="inet 69.55.2xx.xx netmask 255.255.255.0"

static_routes="t1 office"
route_t1="-net 10.1.5 10.1.4.2"
route_office="-net 10.1.6 10.1.4.2"
gvinum_enable="YES"
fsck_y_enable="YES"
background_fsck="NO"

34. make sure sysctls are set and preserved after reboot
echo "kern.consmute=0\
kern.ipc.shm_use_phys=1\
kern.ipc.shmall=65535\
kern.ipc.shmmax=134217728\
net.inet.tcp.syncookies=0\
kern.maxfiles=32768\
kern.fallback_elf_brand=3\
kern.maxprocperuid=4000\
security.jail.sysvipc_allowed=1\
security.jail.max_procs_per_jail: 1026\
security.jail.allow_raw_sockets=1\
security.jail.socket_unixiproute_only=1\
security.jail.chflags_allowed=0" >> /etc/sysctl.conf

35. mount procfs
echo "proc                    /proc           procfs  rw              0       0" >> /etc/fstab

36. enable noatime option
NOT APPLICABLE IF RUNNING GVINUM
data1 and data2 should look something like:
/dev/amrd0s1g           /mnt/data1      ufs     rw,noatime      2       2

36. populate devfs ruleset
scp backup2:/mnt/data4/build/freebsd/devfs.rules /etc

35. reboot
Check rules:
devfs rule showsets
devfs rule -s 3 show

36. create gvinum volumes

Make a g partition:

bsdlabel -e /dev/aacd0s1

given:
# /dev/aacd0s1:
8 partitions:
#        size   offset    fstype   [fsize bsize bps/cpg]
  a:   262144        0    4.2BSD     2048 16384 16392
  b:  4194304   262144      swap
  c: 143363997        0    unused        0     0         # "raw" part, don't edit
  d:   524288  4456448    4.2BSD     2048 16384 32776
  e:   524288  4980736    4.2BSD     2048 16384 32776
  f:  6291456  5505024    4.2BSD     2048 16384 28552

new offset = 6291456 + 5505024 = 11796480
new size is size for 'c' partition minus the new start from above
143363997 - 11796480 = 131567517
So:
g: 131567517 11796480 unused 0 0

For a 73G drive (after OS), we can fit 31 2G volumes so:

echo 'drive data1 device /dev/aacd0s1g' > /tmp/cgv
sh
for f in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31;\
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 2g drive data1' >> /tmp/cgv; done; exit

gvinum create /tmp/cgv

For a 146G drive (-4G for swap), we can fit 66 2G volumes so:

echo 'drive data2 device /dev/aacd1s1g' > /tmp/cgv
sh
for f in 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97;\
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 2g drive data2' >> /tmp/cgv; done; exit

gvinum create /tmp/cgv

For 3rd 73G drive (after 2G swap), we can fit 33 2G volumes so:
Label should be:
# /dev/aacd2s1:
8 partitions:
#        size   offset    fstype   [fsize bsize bps/cpg]
  c: 143299737        0    unused        0     0         # "raw" part, don't edit
  g: 143299721        16    unused        0     0

echo 'drive data3 device /dev/aacd2s1g' > /tmp/cgv
sh
for f in 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131;\
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 2g drive data3' >> /tmp/cgv; done

gvinum create /tmp/cgv

For a 2nd 73G drive (after 2G swap), we can fit 33 2G volumes so:

echo 'drive data2 device /dev/aacd1s1g' > /tmp/cgv
sh
for f in 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64;\
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 2g drive data2' >> /tmp/cgv; done; exit

gvinum create /tmp/cgv


36. create the jail template

mkdir /mnt/jail
newfs /dev/gvinum/v1
mount /dev/gvinum/v1 /mnt/jail

cd /usr/src
make clean
rm -fr /usr/obj/
make world DESTDIR=/mnt/jail
(2450: 2:28mins, supermicro: 55mins)
cd etc
make distribution DESTDIR=/mnt/jail
mount_devfs devfs /mnt/jail/dev
devfs -m /mnt/jail/dev rule -s 3 applyset 
cd /mnt/jail
ln -sf dev/null kernel

jail /mnt/jail testhostname 192.168.11.100 /bin/sh
csh
touch /etc/fstab
echo 'network_interfaces=""\
hostname="newsystem"\
kern_securelevel_enable="NO"\
sendmail_enable="YES"\
sshd_enable="YES"' > /etc/rc.conf

echo "nameserver 69.55.225.225\
nameserver 69.55.230.3" >> /etc/resolv.conf

vi /etc/crontab
remove the adjkerntz lines
comment out periodic’s and put this line above them:
# DO NOT UNCOMMENT THESE

rm -rf /etc/periodic/daily/400.status-disks

check /tmp for crap

vi /etc/periodic/security/100.chksetuid
replace: MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`
with: MP='/' (use single quotes)

mkdir -p /usr/compat/linux/dev

adduser (Add account for user, make sure in wheel group)

user/root passwd: 8ico2987

Set root password

vi /usr/home/user/.profile (and add to the file):
TERM=vt100;     export TERM

tzsetup

newaliases 

#cd /dev
#rm console
#ln -s null console

exit
exit

cd /usr/ports
make -DNOCLEANDEPENDS clean 
(2450: 15mins , supermicro: 29mins)
rm -fr /usr/ports/distfiles/*
cp -r /usr/ports /mnt/jail/usr (2450: 2:00 mins , supermicro: 15mins)

rm /mnt/jail/root/.history

cd 
mkdir /usr/local/jail/template/
dump -0a -f /usr/local/jail/template/61template /dev/gvinum/v1

umount /mnt/jail/dev
umount /dev/gvinum/v1

rm -fr /mnt/jail

37. setup backups
echo '#\!/bin/sh\
backupdir=/mnt/data3jail3_rsync\
\
## ENTRY /etc\
## ENTRY /usr/local/etc\
## ENTRY /usr/local/jail\
## ENTRY /root/logs' > /usr/local/jail/bin/backup.config

on backup2:
setup backup dirs:
mkdir -p /mnt/data3/jail3/0

add the system to 
vi /mnt/data4/bin/snapshot_rotate

on mail:
vi /usr/local/www/mgmt/cgi/backupgraph.pl
(add hostname)

38. mkdir /root/logs

39. edit sshd_config for security
vi /etc/ssh/sshd_config
ListenAddress 69.55.228.101
ListenAddress 10.1.4.118

kill -1 `cat /var/run/sshd.pid`

40. add crontab entries
crontab -e
5 0 * * * /usr/local/jail/bin/backup
1 0 1 * * /usr/local/jail/bin/ipfwreset
0 18 * * * /usr/local/jail/bin/ipfwbackup
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /usr/local/jail/bin/trafstats
55 10,23 * * * /usr/local/jail/bin/trafficwatch.pl


41. Reboot notify script
ln -s /usr/local/jail/bin/notify.sh /usr/local/etc/rc.d/notify.sh 

42. copy jailmake from prev system
scp user@10.1.4.119:/usr/local/jail/bin/jailmake /usr/local/jail/bin
rehash
NOTE: remove df altering code from jailmake since we put the correct df in the template, and make sure path to template file is right

43. add to management db (on mail and devweb) jc.ref_machines and jc.ref_templates

uname -r
5.4-RELEASE-p2-jc2

insert into ref_machines values (null,'jail19','jail19.johncompanies.com',0,'l');
select machine_id from ref_machines where host='jail19';
+------------+
| machine_id |
+------------+
|         35 |
+------------+
insert into ref_templates values ('',' 6.2-RELEASE-jc1',5,'FreeBSD 6.2',0);

44. add to server/cabinet map. On mail:
vi /usr/local/www/mgmt/html/cabinetmap.html

45. add an outside blocking rule to the firewall, so this machine can only be reached from inside the firewall. Follow example already in firewall jail17 is:
 
00119 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.228.200
00119 deny ip from any to 69.55.228.200

jail19 would be 00119...
ipfw add 00119 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.228.200
ipfw add 00119 deny ip from any to 69.55.228.200

46. select customers for probe map

47. install raid monitor

cd /usr/ports/sysutils/asr-utils
make install clean

48. make gv start on boot


scp backup2:/mnt/data4/build/freebsd/gvinum /etc/rc.d/gvinum




gconcat label -v somelabel /dev/gvinum/a /dev/gvinum/b
bsdlabel -r -w /dev/concat/somelabel
newfs /dev/concat/somelabela
mount /dev/concat/somelabel /mount/point

umount /dev/concat/somelabel
gconcat stop somelabel
gconcat label -v /dev/gvinum/a /dev/gvinum/b /dev/gvinum/c /dev/gvinum/d
growfs /dev/concat/somelabel
mount /dev/concat/somelabel /mount/point

volume f
        plex org concat
        sd length 30449m drive data1

where f is 

D data1                 State: up       /dev/aacd0s1g   A: 30449/64241 MB (47%)

gvinum rm -r f


so i setup 2 machines with 6.1. A had 2 logical drives and gv's created across both. B had 1 logical drive and gv's across the 1 drive.  
the labeling for the gv's was v1-vN (till i ran out of space) 
when i moved aac1 from A to B, the gv's on A's aac1, took precedence over the similarly labeled gv's on B's aac0. in other words. B's aac0 used to have v1-v30. A's aac1 had vn6-vn30. when A's aac1 was moved to B, v1-v5 were linked to B's aac0, and v6-v30 linked to A's aac1 
i relabeled B's aac0 gv's to something different (not v1-vN) and tried again and they all showed up.  
i tried the experiment again this time moving A's aac0 to B. at that point nothing worked. i moved A's aac0 back to A and renamed the device (data1) to something else, then tried the test again. still no beans.  
it's hazy what i did after that cause the machine's weren't cooperating and i was trying to rename the device but basically i think i need to repeat the test and see if i can re-define the device, and probably also give it a unique name and i bet it would have worked.  
also interesting- A's aac1 contained data about A's aac0 which showed up when i put aac1 into B 
skeeter:	as for the gv stuff, it definitely sounds like using some sort of serial numbering scheme would be the way to go if you want to be able to move disks around.... 
that overlap is still an issue  (aac1's device was 'data2')  when i had A's aac0 in B nothing worked both disks gv devices were called data1 
skeeter:	I suppose you could serialize those names as well...

Jail Server Install: Difference between revisions