Jail Server Install: Difference between revisions

From JCWiki
Jump to navigation Jump to search
← Older edit
 
(61 intermediate revisions by 4 users not shown)
Line 1: Line 1:
= 8.x =
= FreeBSD 9.x =


== Assumptions ==
== Assumptions ==
Line 21: Line 21:
go into RAID bios and setup mirrors
go into RAID bios and setup mirrors


configure DRAC: TODO
 
 
== Setup DRAC ==
[[DRAC/RMM#DRAC_setup|DRAC setup]]


== Install OS (sysinstall) ==
== Install OS (sysinstall) ==
Line 57: Line 60:
/ 512M<br>
/ 512M<br>
swap 6G<br>
swap 6G<br>
/var 256M <br>
/var 1G <br>
/tmp 256M<br>
/tmp 256M<br>
/usr 5G<br>
/usr 8G<br>
/mnt/data1 remaining space  
/mnt/data1 remaining space  


Line 111: Line 114:
=== double check the date/time ===
=== double check the date/time ===


  date


=== populate /etc/resolv.conf ===
=== populate /etc/resolv.conf ===


  echo "nameserver 69.55.225.225\
  echo "search johncompanies.com \
  nameserver 69.55.230.3
nameserver 69.55.225.225 \
  nameserver 69.55.230.3 \
  nameserver 69.55.229.3" > /etc/resolv.conf
  nameserver 69.55.229.3" > /etc/resolv.conf


=== edit /etc/make.conf ===
=== edit /etc/make.conf ===
  echo "WITHOUT_X11=yes \
  echo "WITHOUT_X11=yes \
  KERNCONF=jail3 \
  KERNCONF=jail4 \
  BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf
  BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf


=== setup bootloader for console, etc ===
=== setup bootloader for console, etc ===
Line 206: Line 209:
mkdir /mnt/data1
mkdir /mnt/data1
mkdir /mnt/data2
mkdir /mnt/data2
scp backup2:"/mnt/data4/bin/freebsd8.x/*" /usr/local/jail/bin
scp backup2:"/mnt/data4/bin/freebsd9.x/*" /usr/local/jail/bin
cd /usr/local/jail/rc.d/
cd /usr/local/jail/rc.d/
touch quad1
touch quad1
Line 223: Line 226:


rehash</pre>
rehash</pre>


=== edit root's path and login script ===
=== edit root's path and login script ===
Line 248: Line 250:


alter the prompt, set the following:
alter the prompt, set the following:
  set prompt = "`/bin/hostname -s` %/# "
  set prompt = "$user@`/bin/hostname -s` %/# "


at the bottom of the file add:
at the bottom of the file add:
Line 262: Line 264:
  source /root/.cshrc
  source /root/.cshrc


=== update ports ===


=== install cvsup ===
portsnap fetch
portsnap extract


<pre>cd /usr/ports/net/cvsup-without-gui
To update later on:
make install clean; rehash; mail -s 'cvs installed' support@johncompanies.com < /dev/null</pre>
portsnap fetch
stand by for gettext options (use defaults). this process takes approx 22mins- hence the email/page notice above.
portsnap update




=== get latest sources for this release ===
=== install svn ===


<pre>cd /usr/src
<pre>
echo "*default host=cvsup4.freebsd.org\
setenv PACKAGESITE "ftp://ftp4.freebsd.org/pub/FreeBSD/ports/amd64/packages-9-current/Latest/"
*default base=/usr\
pkg_add subversion
*default prefix=/usr\
</pre>
*default release=cvs tag=RELENG_8_3\
*default delete use-rel-suffix\
*default compress\
src-all" > sup</pre>


If you need to run stable (cause release is broken or some other reason) make the sup file look like:
=== get latest sources for this release ===


<pre>echo "*default host=cvsup4.freebsd.org\
<pre>
*default base=/usr\
cd /usr
*default prefix=/usr\
*default release=cvs tag=RELENG_8\
*default delete use-rel-suffix\
*default compress\
src-all" > sup</pre>


cvsup sup ; mail -s 'cvs sup done' support@johncompanies.com < /dev/null
# mv src/ src.orig
 
tar cvzf src.orig.tgz src
time varies, 10-20mins
rm -fr src/*
svn checkout svn://svn.FreeBSD.org/base/stable/9 /usr/src
</pre>


To update:
make update SVN_UPDATE=yes


=== configure new kernel ===
=== configure new kernel ===


Pull down the kernel config we are using for this distribution. In this case we use an 8.2 kernel config on 8.3, which is valid. The local file should be the same name as host- <tt>jail3</tt> in this example
Pull down the kernel config we are using for this distribution. In this case we use an 8.2 kernel config on 8.3, which is valid. The local file should be the same name as host- <tt>jail4</tt> in this example


<pre>cd /usr/src/sys/amd64/conf  
<pre>cd /usr/src/sys/amd64/conf  
scp backup2:/mnt/data4/build/freebsd/kern_config-8.2-amd64 ./jail3</pre>
scp backup2:/mnt/data4/build/freebsd/kern_config-9.1-amd64 ./jail4</pre>


edit the kernel config and change ident to be the name of the jail:
edit the kernel config and change ident to be the name of the jail:
  vi jail3
  vi jail4
  ident          jail3
  ident          jail4


Optional, edit <tt>/sys/conf/newvers.sh</tt> to add <tt>–jc2</tt> to the end of the BRANCH string (RELEASE-jc2)
Optional, edit <tt>/sys/conf/newvers.sh</tt> to add <tt>–jc2</tt> to the end of the BRANCH string (RELEASE-jc2)
Line 311: Line 310:


notes on kernel configuring: http://www.FreeBSD.org/doc/en_US.ISO8859-1/books/handbook/kernelconfig-config.html
notes on kernel configuring: http://www.FreeBSD.org/doc/en_US.ISO8859-1/books/handbook/kernelconfig-config.html


=== install patches ===
=== install patches ===


The various patches are in /mnt/data4/build/freebsd/patches on backup2. There are dirs for each version. Not all dirs are populated, but patches for later versions work on older ones unless there is a new patch in the older dir.
We don't have any patches right now. Refer to older FreeBSD version build docs on how that is/was done.


there are no patches we use for 8.x, but here would be the commands:
=== build, install kernel and world ===
 
cd /usr/src
scp backup2:"/mnt/data4/build/freebsd/patches/8.0/*" .
 
Apply patches, i.e. the <tt>jls-patch</tt>:
patch -l < jls-patch
 
 
=== build, install kernel and world ===


Rename current generic kernel so it will always be available to boot from. Save room by removing non-needed kernel modules:
Rename current generic kernel so it will always be available to boot from. Save room by removing non-needed kernel modules:
Line 347: Line 336:
mv hold/* .
mv hold/* .
rmdir hold/
rmdir hold/
Note on -DWITHOUT_CLANG: try to do it without including that directive, it may work for you.


cd /usr/src
cd /usr/src
make buildkernel installkernel
make buildworld KERNCONF=jail4; mail -s 'buildworld done' dave.boodman@vtext.com < /dev/null
 
 
make -DWITHOUT_CLANG buildworld KERNCONF=jail4; mail -s 'buildworld done' dave.boodman@vtext.com < /dev/null
</pre>
~4.5hr
 
cd /usr/src
make buildkernel installkernel
 


make buildworld ; mail -s 'buildworld done' dave.boodman@vtext.com < /dev/null</pre>
mergemaster -p
~38mins
You will be prompted to merge, replace or ignore files changed by the src update. In most cases you can '''d'''elete the temp (new) files.


  make installworld  
  make -DWITHOUT_CLANG installworld
~34min
~34min


mergemaster -i
You will be prompted to merge, replace or ignore files changed by the src update. In most cases you can '''d'''elete the temp (new) files.


ONLY if this will be a zfs system (not currently used in 8.x):
ONLY if this will be a zfs system (not currently used in 8.x):
Line 367: Line 365:
make  
make  
make install</pre>
make install</pre>


===  populate devfs ruleset ===
===  populate devfs ruleset ===
Line 385: Line 382:
nfs_reserved_port_only="YES"
nfs_reserved_port_only="YES"
inetd_enable="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.2.103"
inetd_flags="-wW -a 10.1.4.XXX"
devfs_system_ruleset="devfsrules_show_all"
devfs_system_ruleset="devfsrules_show_all"


ifconfig_bce1="inet 10.1.2.103 netmask 255.255.255.0"
ifconfig_bce1="inet 10.1.4.XXX netmask 255.255.255.0"
ifconfig_bce0="inet 69.55.229.7 netmask 255.255.255.0"
ifconfig_bce0="inet 69.55.2XX.XXX netmask 255.255.255.0"
#ifconfig_bce0_alias0="inet 69.55.2xx.xx netmask 255.255.255.0"
#ifconfig_bce0_alias0="inet 69.55.2XX.XXX netmask 255.255.255.0"


fsck_y_enable="YES"
fsck_y_enable="YES"
Line 398: Line 395:


Modify IPs, hostname, gateway for this box.
Modify IPs, hostname, gateway for this box.


=== make sure sysctls are set and preserved upon boot ===
=== make sure sysctls are set and preserved upon boot ===
Line 432: Line 428:


=== enable noatime option ===
=== enable noatime option ===
grep data /etc/fstab
data1 and data2 should look something like (add ',noatime' after 'rw'):
data1 and data2 should look something like (add ',noatime' after 'rw'):
  /dev/mfid0s1g          /mnt/data1      ufs    rw,noatime      2      2
  /dev/mfid0s1g          /mnt/data1      ufs    rw,noatime      2      2
  /dev/mfid1s1d          /mnt/data2      ufs    rw,noatime      2      2
  /dev/mfid1s1d          /mnt/data2      ufs    rw,noatime      2      2


=== reboot. Confirm new kernel is loaded, devfs in place ===
=== reboot. Confirm new kernel is loaded, devfs in place ===
Line 475: Line 474:
910 path stdout unhide
910 path stdout unhide
920 path stderr unhide</pre>
920 path stderr unhide</pre>
=== update ports ===
<pre>cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_8_3\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup
cvsup sup; mail -s 'cvs sup ports done' support@johncompanies.com < /dev/null</pre>
~24mins




Line 496: Line 480:
==== Perc5/i, 6/i ====
==== Perc5/i, 6/i ====
Pull over cli from previous system (jail9)
Pull over cli from previous system (jail9)
  scp /usr/local/sbin/mega* root@10.1.4.109:/usr/local/sbin/
  scp root@10.1.4.109:"/usr/local/sbin/mega*" /usr/local/sbin/
scp /usr/local/libexec/MegaCli root@10.1.4.109:/usr/local/libexec/MegaCli
scp root@10.1.4.109:/usr/local/libexec/MegaCli /usr/local/libexec/MegaCli
These are linux-based tools. This will require linux base which we would normally install from ports, but since it's failing lately, we just pull in the compat libraries from another system:


rsync -aSHv --exclude=proc --exclude=sys 10.1.4.109:/usr/compat/linux/ /usr/compat/linux/
These are linux-based tools. This will require linux base...which you might install via:
 
pkg_add -r linux_base


Test:
Test:
  rehash; megacli ldinfo lall a0
  rehash; megacli ldinfo lall a0
or
megarc -ldInfo -a0 -Lall
(2850)


DEPRECATED: Assuming it worked, here's how we ''used to'' install linux_base:
however, linux does seem to be installed already so we don't need to pkg_add or port install or rsync anything over from a current system.
----
cd /usr/ports/emulators/linux_base-fc4
make install clean
(didnt succeed due to libtool requirement)
 
cd /usr/ports/distfiles
fetch http://www.lsi.com/DistributionSystem/AssetDocument/support/downloads/megaraid/miscellaneous/linux/2.00.15_Linux_MegaCLI.zip
cd /usr/ports/sysutils/linux-megacli
make install clean
(also failed due to libtool)
----
 


==== 2850 PERC 4e/Di- no linux ====
==== 2850 PERC 4e/Di- no linux ====
Line 548: Line 524:
=== install bb client ===
=== install bb client ===


Compiling from source on AMD64 will not work. So, we use a linux-compiled version and rely on linux compat. Per [[#Perc5.2Fi.2C_6.2Fi|above]], linux compat won't install on 8.x - libtool 2.4 need. So, instead we copy(ed) over linux:
Compiling from source on AMD64 will not work. So, we use a linux-compiled version and rely on linux compat.  
rsync -aSHv --exclude=proc --exclude=sys 10.1.4.108:/usr/compat/linux/ /usr/compat/linux/


  adduser
  adduser
Line 585: Line 560:


  echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
  echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
  10.1.4.103 jail3.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
  10.1.4.'''104 jail4'''.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts


Edit for machine name and private IP.
Edit for machine name and private IP.
Line 591: Line 566:
if this machine is at i2b:
if this machine is at i2b:
  echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
  echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
  69.55.229.7 jail3.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
  69.55.228.104 jail4.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts


<pre>vi /home/bb/bbc1.9e-btf/ext/openfiles  
<pre>vi /home/bb/bbc1.9e-btf/ext/openfiles  


MACHINE="jail3,johncompanies,com"      # HAS TO BE IN A,B,C FORM</pre>
MACHINE="jail4,johncompanies,com"      # HAS TO BE IN A,B,C FORM</pre>
Edit for machine name.
Edit for machine name.


Line 616: Line 591:
  echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
  echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
  chmod +x /usr/local/etc/rc.d/bb.sh
  chmod +x /usr/local/etc/rc.d/bb.sh


=== remove reserve space ===
=== remove reserve space ===
Line 782: Line 756:
=== create the jail template ===
=== create the jail template ===


cd /usr/ports/sysutils/jailutils
cd /usr/ports/sysutils/jailutils
make install clean
make install clean


Create an md device to hold the jail:
Create an md device to hold the jail:
Line 795: Line 769:
<pre>cd /usr/src
<pre>cd /usr/src
make world DESTDIR=/mnt/data1/jail-DIR; taskdone</pre>
make world DESTDIR=/mnt/data1/jail-DIR; taskdone</pre>
~1hr
~4.5 hr


Make /etc into the jail, mount dev, copy in jkill:
Make /etc into the jail, mount dev, copy in jkill:
Line 924: Line 898:
rmdir /mnt/data1/jail-DIR
rmdir /mnt/data1/jail-DIR
mdconfig -d -u 0</pre>
mdconfig -d -u 0</pre>


=== setup backups ===
=== setup backups ===
Line 958: Line 931:
=== edit sshd_config ===
=== edit sshd_config ===
  vi /etc/ssh/sshd_config
  vi /etc/ssh/sshd_config
  ListenAddress 69.55.229.7
  ListenAddress 69.55.'''229.7'''
  ListenAddress 10.1.2.103
  ListenAddress 10.1.4.'''103'''


Adjust to pub/private IP for jail.
Adjust to pub/private IP for jail.
Line 965: Line 938:
Restart sshd:
Restart sshd:
  kill -1 `cat /var/run/sshd.pid`
  kill -1 `cat /var/run/sshd.pid`


=== add crontab entries ===
=== add crontab entries ===
Line 1,056: Line 1,028:
==== add to bb server ====
==== add to bb server ====
  vi /usr/home/bb/bbsrc/bb1.9i-btf/etc/bb-hosts
  vi /usr/home/bb/bbsrc/bb1.9i-btf/etc/bb-hosts
  10.1.4.109 jail9.johncompanies.com # ssh
  10.1.4.'''109 jail9'''.johncompanies.com # ssh


In the case of an i2b server, use real ip:
In the case of an i2b server, use real ip:
  69.55.229.7 jail3.johncompanies.com # ssh
  69.55.'''229.7 jail3'''.johncompanies.com # ssh


<pre>su bb
<pre>su bb
Line 1,097: Line 1,069:
=== select some customers for castle probe map ===
=== select some customers for castle probe map ===


= 4.11 =
= FreeBSD 8.x =
 
== Assumptions ==


<pre>
Setup instructions below assume this is DELL 2950 with an LSI-based SAS RAID card.
4.11
 
Last updated 2006-1-26
Server is at castle, connected to pub, private, serial and DRAC


All time extimates below assume disks aren’t scrubbing
Assuming OS loading done via IPKVM with ISO mounted via USB


1. make sure bios is setup for bios console redirect
Assumes at 4 drives, 2 mirrors
Supermicro:
Console redirection:
Com port addr: on-board COM A
Baud: 38400
Console type: vt100
Flow control: none
Console connection: direct
Continue cr after post: off


2450:
== Configure server BIOS ==
Make sure running bios A09
 
Console Redirection: VT100/VT220
setup console redirect, speed 115200
2. assuming mirrors (or at least disks) created (if not, refer to this), boot to disk 1 of 4.11
skip kernel config (enter)
custom install
partition
move cursor to aacd0, hit space (takes you to partition map screen)
a for entire disk
q to quit and save
standard mbr (no boot manager)
space to unselect aacd0
cursor over aacd1
space
a for entire disk
q to quit and save
none (leave untouched)


cursor over aacd0
set LCD string to name of server "jail8"
space
(takes you into part. Screen again) q to exit
none
Make sure both are checked and tab to ok


label
set date to GMT
Make sure aacd0 is highlighted
a to start with defaults


john likes:
go into RAID bios and setup mirrors
/1g
/var 256m


Glenn likes:
configure DRAC: TODO
/128
/var 256
/usr 3g
/tmp 256


We do:
== Install OS (sysinstall) ==
delete everything
/ 128M
swap 4G
/var 256M
/tmp 256M
/usr 3G
/mnt/data1 remaining space


Make sure to toggle S for soft updates on all (should look like UFS+S Y under the Newfs column)
boot to bootonly disk for AMD version of FreeBSD, i.e. <tt>FreeBSD-8.3-RELEASE-amd64-bootonly.iso</tt>
Set the noatime option


move cursor to aacd1
when the install menu appears, choose <tt>custom install</tt>
swap 4G
/mnt/data2 remaining space
Set the noatime option


q to save and exit
=== partition menu ===
move cursor to mfid0, hit space (takes you to partition map screen). If there is only 1 mirror, there will be no option to select a specifit drive: mfid0 will be selected.


distributions
type 'a' to use entire disk<br>
developer
type 'q' to quit and save<br>
yes to install ports
choose 'freebsd standard mbr'
exit
media
cd
commit
yes
(2450: 16mins, supermicro: 11mins)
yes to "visit general config"
Set root pwd
Add user ‘user’ member group is wheel, set password
Set tz
Networking->interfaces->Fxp0
No IPV6
dhcp=yes
Set hostname & domain


Startup services:
space to '''un'''select mfid0<br>
Disble usbd
cursor down to mfid1<br>
hit space


exit...
type 'a' to use entire disk<br>
exit install
type 'q' to quit and save<br>
yes
choose 'none' for boot mgr (leave untouched)
take the cd out and let the machine reboot


3. put some temp settings in /etc/rc.conf:
cursor over mfid0<br>
usbd_enable="NO"
space<br>
sendmail_enable="NO"
(takes you into part. Screen again) 'q' to exit<br>
none for boot mgr<br>


4. reboot
Make sure both drives (mfid0 and mfid1) are checked and tab to ok


5. double check the date/time
=== Label menu ===


6. edit /etc/make.conf (only add the console speed line if this is a supermicro capable of outputting BIOS redirect at that speed – 2450’s can only do 9600)
Make sure mfid0 is highlighted at the top of the screen, setup the following partitions
cat >> /etc/make.conf
WITHOUT_X11=yes
KERNCONF=jail18
BOOT_COMCONSOLE_SPEED=38400


7. install cvsup
/ 512M<br>
cd /usr/ports/net/cvsup-without-gui
swap 6G<br>
make install clean (stay close for gettext options, 2450: 21mins, supermicro: 14mins)
/var 256M <br>
rehash
/tmp 256M<br>
/usr 5G<br>
/mnt/data1 remaining space


8. get latest sources for this release:
All partitions except / should be setup for soft updates. If not, type 's' to enable for soft updates on all except for / (should look like UFS2+S Y under the Newfs column)
cd /usr/src
cat > sup
*default host=cvsup4.freebsd.org
*default base=/usr
*default prefix=/usr
*default release=cvs tag=RELENG_4_11
*default delete use-rel-suffix
*default compress
src-all


cvsup sup (2450, 4.10: 13mins, supermicro, 4.11: 11mins)
move cursor to mfid1 at the top of the screen<br>
swap 8G (or 4G if there’s a 3rd drive)<br>
/mnt/data2 remaining space <br>
'q' to save and exit


9. populate hosts
=== distributions ===
cat >> /etc/hosts
Choose the following distribudions
10.1.4.3 backup2


10. put key in authorized_keys on backup2
* developer (ok to install ports)
ssh-keygen -t dsa -b 1024 (default location, leave password blank)
* custom -> lib32
scp /root/.ssh/id_dsa.pub user@backup2:/tmp/jail18pub
exit
on backup2:
cat /tmp/jail18pub >> /root/.ssh/authorized_keys


confirm that you can ssh to backup2 without getting a login prompt
=== media ===
if you are installing via a cd, no need to enter this menu or change anything. Otherwise, choose ftp to install via ftp. You will be prompted to setup networking. You will need to choose a nic (typically bce0 or bce1). Say no to DHCP and IPv6. Hopefully the public nic cable was installed in bce0 so start with that nic and provide the hostname, (public) IP, netmask, gateway and DNS. When configured, it should start pinging. If it doesn't, have the NOC swap cables. Select any FTP server, usually Main or ftp4.


11. configure new kernel. Get config from similar machine or there may be a master copy somewhere under /mnt/data4/build (name the kernel config the same as the jail, ex jail4):
=== commit ===
cd /usr/src/sys/i386/conf
scp backup2:/mnt/data4/build/freebsd/kern_config-4.11 ./jail18


edit the kernel config and change ident to be the name of the jail:
this usually takes 12mins<br>
ident          jail4
during the process you may need to select a new ftp mirror, this is not a problem.<br>
at the conclution of the install you will be prompted to enter the root password (2x) and returned to the configuration menu.


IMPORTANT CUSTOMIZATION:
=== add user ===
for machines with >4G RAM, add to the config:
options        PAE


for supermicro mobo’s with broadcom nics, add to the config:
'''Add''' user 'user'. Defaults for everything is fine, just remember to enter 'wheel' in the member group field.
device          bge            # Broadcom BCM570x (``Tigon III'')
Do set the password.


for machines where lots of postgres might be running, change SHMMAXPGS:
=== Setup timezone ===
options        SHMMAXPGS=40960


edit /sys/conf/newvers.sh to add –jc2 to the end of the BRANCH string (RELEASE-p9-jc2)
PT


12. bring over patches from backup2
=== Networking ===
The various patches are in /mnt/data4/build/freebsd/patches on backup2. There are dirs for each version. Not all dirs are populated, but patches for later versions work on older ones unless there is a new patch in the older dir.
page down to the bottom and enable '[X]' sshd
cd /usr/src
scp backup2:"/mnt/data4/build/freebsd/patches/4.11/*" .
jail_proc_patch      100% |*************************************************|  2593    00:00
restore-patch        100% |*************************************************|  3295    00:00
411ps-jail-patch    100% |*************************************************|  2602    00:00
jail_rtprio_patch    100% |*************************************************|  301    00:00
udp-patch            100% |*************************************************|  594    00:00


Apply patches:
If you installed via cd, you will need to visit:<br>
patch < 411ps-jail-patch
interfaces->bce0-><br>
patch < jail_proc_patch
No IPV6<br>
patch < restore-patch
dhcp=no<br>
patch < jail_rtprio_patch
Set hostname, IP, DNS, gateway<br>
patch < udp-patch
(i.e. setup the nic as indicated above)


13. build, install kernel and world
make buildworld buildkernel installkernel (2450: 48min, supermicro: 20mins)
(Any compile errors can be looked up in /usr/include/sys/signal.h, other errors, do a rm -R /usr/obj/*)
make installworld (2450: 2min, supermicro: 1mins)
mergemaster -i
(answer no to most of it)


14. reboot. Confirm new kernel is loaded (uname -a)
Exit the install and if you installed via CD, take it out and let the machine reboot


15. update ports:
== Configure OS, kernel, userland, jail ==
cd /usr/ports
cat > sup
*default host=cvsup4.FreeBSD.org
*default base=/usr
*default prefix=/usr
*default release=cvs tag=RELENG_4
*default delete use-rel-suffix
*default compress
ports-all tag=.


cvsup sup (2450: 26mins, supermicro: 26mins)
=== double check the date/time ===


16. add console="comconsole" to /boot/loader.conf
cat >> /boot/loader.conf
console="comconsole"


17. edit /etc/ttys and turn off all ttyv's except 0 and 1
=== populate /etc/resolv.conf ===
also turn on ttyd0, change type to vt100:
ttyd0  "/usr/libexec/getty std.9600"  vt100  on secure


kill -1 1
echo "nameserver 69.55.225.225\
nameserver 69.55.230.3
nameserver 69.55.229.3" > /etc/resolv.conf


on console:
vi /etc/remote
(rename port to jail4 depending on where and which digi plugged into)
test serial console


18. install linux_base:
=== edit /etc/make.conf ===
cd /usr/ports/emulators/linux_base
echo "WITHOUT_X11=yes \
make install clean (2450: 7min, supermicro: 2mins)
KERNCONF=jail3 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf


reibuild rpmdb cause we had probs installing aacapps
cd /compat/linux/bin
./rpm --initdb
./rpm --rebuilddb


19. install aacapps-4.1-0.i386.rpm
=== setup bootloader for console, etc ===
scp backup2:/mnt/data4/build/freebsd/aacapps-4.1-0.i386.rpm /tmp/.
/compat/linux/bin/rpm -ivh /tmp/aacapps-4.1-0.i386.rpm
cd /dev
sh MAKEDEV aac0


test out /compat/linux/usr/sbin/aaccli
add settings to /boot/loader.conf and /boot.config:


20. edit root's path and login script:
echo "-Dh" >> /boot.config
vi /root/.cshrc
add to path: /usr/local/jail/bin /compat/linux/usr/sbin


Change alias entries (add G):
echo 'console="comconsole,vidconsole" \
alias la        ls -aG
boot_multicons="YES" \
alias lf        ls -FAG
boot_serial="YES" \
alias ll        ls -lAG
mfi_linux_load="YES" \
alias ls        ls -AG
comconsole_speed="115200"' >> /boot/loader.conf
alias mbm      mb mount
alias mbu      mb umount


and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "


at the bottom of the file add:
=== enable login via serial console ===
set sshtty=`who am i|awk '{print $2}'`
turn off all ttyv's except 0 and 1 in /etc/ttys and turn on ttyd0, change type to vt100:
/usr/sbin/rtprio 3 -`ps auxwJ | grep $sshtty | awk '{print $2}'`


set shortty=`who am i | awk '{print $2}' | sed -E 's/.*(..)$/\1/'`
  vi /etc/ttys
foreach x (`ps cauxJ | grep sh | grep $shortty | awk '{print $2}'`)
/usr/sbin/rtprio 2 -$x
end
   
21. install rsync from ports
cd /usr/ports/net/rsync
make install clean


22. create & populate binaries/scripts dirs
The changed lines should look like:
mkdir -p /usr/local/jail/bin
mkdir -p /usr/local/jail/rc.d
scp backup2:"/mnt/data4/bin/freebsd/*" /usr/local/jail/bin
cd /usr/local/jail/rc.d/
touch quad1
touch quad2
touch quad3
touch quad4
touch safe1
touch safe2
touch safe3
touch safe4
chmod +x *
cd /usr/local/jail/bin
ln -s /usr/local/jail/rc.d/quad1 quad1
ln -s /usr/local/jail/rc.d/quad2 quad2
ln -s /usr/local/jail/rc.d/quad3 quad3
ln -s /usr/local/jail/rc.d/quad4 quad4
ln -s /usr/local/jail/rc.d/safe1 safe1
ln -s /usr/local/jail/rc.d/safe2 safe2
ln -s /usr/local/jail/rc.d/safe3 safe3
ln -s /usr/local/jail/rc.d/safe4 safe4
rehash


23. configure inetd to respond to mrtg load queries
ttyv2  "/usr/libexec/getty Pc"        cons25  off secure
cat >> /etc/inetd.conf
ttyv3  "/usr/libexec/getty Pc"        cons25  off secure
load    stream tcp    nowait user    /usr/local/jail/bin/load.pl load.pl
ttyv4  "/usr/libexec/getty Pc"        cons25  off secure
  ttyv5  "/usr/libexec/getty Pc"        cons25  off secure
  ttyv6  "/usr/libexec/getty Pc"        cons25  off secure
ttyv7  "/usr/libexec/getty Pc"        cons25  off secure
# Serial terminals
# The 'dialup' keyword identifies dialin lines to login, fingerd etc.
  ttyu0  "/usr/libexec/getty std.9600"  vt100  on secure


cat >> /etc/services
Restart init
load            12384/tcp
kill -1 1


kill -HUP `cat /var/run/inetd.pid`
At this point you should have a login on console.


24. configure load mrtg, on mail
To configure serial console access, login to the console server as root and run:
vi /usr/local/etc/mrtg/mrtg1.cfg
(add new entry to file following existing format)


25. configure bb on mail:
# vi /etc/remote
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
10.1.4.104 jail4.johncompanies.com # ssh


su bb
following examples there, rename port to server's hostname, depending on where and which digi box this server is plugged into. Make sure to get speed right too: 115200
cd /usr/home/bb/bbsrc/bb1.9e-btf
./runbb.sh stop
./runbb.sh start
exit


26. install bb client
adduser -group 1984 -shell /bin/csh -uid 1984 bb
cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd.tar .
tar xvf bb-freebsd.tar


cat > /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
=== populate hosts ===
10.1.4.105 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh
10.1.4.118 jail18.johncompanies.com # ssh


vi /home/bb/bbc1.9e-btf/ext/openfiles  and change:
If server is at castle:
MACHINE="jail18,johncompanies,com"     # HAS TO BE IN A,B,C FORM
echo "10.1.4.3 backup2" >> /etc/hosts
echo "10.1.4.8 backup1" >> /etc/hosts


cd /usr/home/bb/bbc1.9e-btf/etc
If server is at i2b:
./bbchkcfg.sh
echo "69.55.230.10 backup2" >> /etc/hosts
./bbchkhosts.sh (ignore ssh errors)
echo "10.1.2.3 backup3" >> /etc/hosts
cd ../..
echo "69.55.230.11 backup1" >> /etc/hosts
chown -R bb .
su bb
cd bbc1.9e-btf/
./runbb.sh start
more BBOUT (look for errors)
exit


cat > /usr/local/etc/rc.d/bb.sh
su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"


chmod +x /usr/local/etc/rc.d/bb.sh
=== create ssh key, upload to backup servers ===
cd
ssh-keygen -t dsa -b 1024
(default location, leave password blank)


27. remove reserve space, enable softupdates (probably already set, so not necessary)
If server is at castle:
cd
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
umount /mnt/data1
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'
umount /mnt/data2
tunefs -m 0 /mnt/data1
tunefs -m 0 /mnt/data2


mount -a
If server is at i2b:
cat /root/.ssh/id_dsa.pub | ssh backup3 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'


28. configure ntp
confirm that you can ssh to backup2 and backup1 without getting a login prompt
cat > /etc/ntp.conf
server 10.1.4.105


/usr/sbin/ntpd -p /var/run/ntpd.pid
ssh backup2 hostname
ntpq -p
ssh backup1 hostname
(confirm it’s able to reach our time server)
ssh backup3 hostname


29. mrtg switch graphs


31. fwd and reverse lookups on ns1c
=== create & populate binaries/scripts dirs ===
vi johncompanies.com
<pre>mkdir -p /usr/local/jail/bin
rr johncompanies.com
mkdir -p /usr/local/jail/rc.d
vi internal.johncompanies.com
mkdir -p /usr/local/jail/template/
rndc reload johncompanies.com IN private
mkdir /mnt/data1
(edit the PTR too)
mkdir /mnt/data2
scp backup2:"/mnt/data4/bin/freebsd8.x/*" /usr/local/jail/bin
cd /usr/local/jail/rc.d/
touch quad1
touch deprecated
chmod +x *
cd /usr/local/jail/bin
ln -s /usr/local/jail/rc.d/quad1 quad1
ln -s /usr/local/jail/bin/jailmake_md jailmake
ln -s /usr/local/jail/bin/js_md js
ln -s /usr/local/jail/bin/canceljail_md canceljail
ln -s /usr/local/jail/bin/jailmakeempty_md jailmakeempty
ln -s /usr/local/jail/bin/postboot_md postboot
ln -s /usr/local/jail/bin/preboot_md preboot
ln -s /usr/local/jail/bin/startjail_md startjail
ln -s /usr/local/jail/bin/stopjail_md stopjail


32. create all /dev/vn and /dev/pty files in /dev
rehash</pre>
cat > /tmp/runme.sh
#!/bin/sh
cd /dev
for i in 1 2 3 4 5 6 7 ; do sh MAKEDEV pty$i ; done
for i in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 ; do sh MAKEDEV vn$i ; done


NOTE: there are only 4 lines in the above output - however the lines may be wrapped on your terminal - make sure that there are only four lines in the script you make.


chmod +x /tmp/runme.sh
=== edit root's path and login script ===
/tmp/runme.sh


rm /tmp/runme.sh
vi /root/.cshrc


ls /dev/vn*|wc -l (make sure the output is a high number – larger than 128)
Change alias entries (add G):
ls /dev/pty*|wc -l (make sure the output is exactly 256)


33. populate /etc/rc.conf with IPs and NFS settings
<pre>alias la        ls -aG
vi /etc/rc.conf
alias lf        ls -FAG
alias ll        ls -lAG
alias ls        ls -AG
alias mbm      mb mount
alias mbu      mb umount
alias cjb      cd /usr/local/jail/bin
alias cd1      cd /mnt/data1
alias cd2      cd /mnt/data2
alias cd3      cd /mnt/data3
alias jtop      jtop lj
alias j        jobs</pre>


kern_securelevel_enable="NO"
add to path be careful to leave a space after bin and make sure the wrapping isn't broken: 
portmap_enable="NO"
/usr/local/jail/bin
sendmail_enable="NO"
 
usbd_enable="NO"
alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "
 
at the bottom of the file add:
<pre>set sshtty=`who am i|awk '{print $2}'`
/usr/sbin/rtprio 3 -`psj | grep $sshtty | awk '{print $2}'`


xntpd_enable="YES"
set shortty=`who am i | awk '{print $2}' | sed -E 's/.*(..)$/\1/'`
nfs_client_enable="YES"
foreach x (`psj | grep sh | grep $shortty | awk '{print $2}'`)
nfs_reserved_port_only="YES"
/usr/sbin/rtprio 2 -$x
sshd_enable="YES"
end</pre>
inetd_enable="YES"
inetd_flags="-wW -a 10.1.4.104"


ifconfig_xl0="inet 10.1.4.104 netmask 255.255.255.0"
Make the new settings active in current shell:
ifconfig_fxp0="inet 69.55.228.101 netmask 255.255.255.0"
source /root/.cshrc
defaultrouter="69.55.228.1"
ifconfig_fxp0_alias0="inet 69.55.2xx.xx netmask 255.255.255.0"


static_routes="t1 office"
route_t1="-net 10.1.5 10.1.4.2"
route_office="-net 10.1.6 10.1.4.2"


34. make sure sysctls are set and preserved after reboot
=== install cvsup ===
cat >> /etc/sysctl.conf
kern.consmute=0
jail.sysvipc_allowed=1
kern.ipc.shm_use_phys=1
kern.ipc.shmall=65535
kern.ipc.shmmax=134217728
net.inet.tcp.syncookies=0
kern.maxfiles=32768
kern.fallback_elf_brand=3
kern.maxprocperuid=4000
jail.max_procs_per_jail: 1026


35. reboot
<pre>cd /usr/ports/net/cvsup-without-gui
make install clean; rehash; mail -s 'cvs installed' support@johncompanies.com < /dev/null</pre>
stand by for gettext options (use defaults). this process takes approx 22mins- hence the email/page notice above.  


36. create the jail template
vnconfig -T -S 1g -Z -s labels -c /dev/vn1 /mnt/data1/jail
disklabel -r -w vn1 auto
newfs /dev/vn1c
cd /usr/src
mkdir -p /mnt/data1/jail-DIR
mount /dev/vn1c /mnt/data1/jail-DIR
make world DESTDIR=/mnt/data1/jail-DIR (2450: 45mins, supermicro: 19mins)
cd etc
make distribution DESTDIR=/mnt/data1/jail-DIR -DNO_MAKEDEV_RUN
cd /mnt/data1/jail-DIR/dev
sh MAKEDEV jail
cd /mnt/data1/jail-DIR
ln -sf dev/null kernel


jail /mnt/data1/jail-DIR testhostname 192.168.11.100 /bin/sh
=== get latest sources for this release ===
csh
touch /etc/fstab
cat > /etc/rc.conf
portmap_enable="NO"
network_interfaces=""
hostname="newsystem"
kern_securelevel_enable="NO"
sendmail_enable="YES"
sshd_enable="YES"


cat >> /etc/resolv.conf
<pre>cd /usr/src
nameserver 69.55.225.225
echo "*default host=cvsup4.freebsd.org\
nameserver 69.55.230.3
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_8_3\
*default delete use-rel-suffix\
*default compress\
src-all" > sup</pre>


vi /etc/crontab
If you need to run stable (cause release is broken or some other reason) make the sup file look like:
remove the adjkerntz lines
comment out periodic’s and put this line above them:
# DO NOT UNCOMMENT THESE


rm -rf /etc/periodic/daily/400.status-disks
<pre>echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_8\
*default delete use-rel-suffix\
*default compress\
src-all" > sup</pre>


mv /bin/df /bin/df_
cvsup sup ; mail -s 'cvs sup done' support@johncompanies.com < /dev/null


cat > /bin/df
time varies, 10-20mins
#!/bin/sh
/bin/df_ $* .


chmod +x /bin/df


cat > /sbin/mount
=== configure new kernel ===
echo `df | tail -1 | awk '{print $1 " on " $6 " (ufs, local)"}'`
 
Pull down the kernel config we are using for this distribution. In this case we use an 8.2 kernel config on 8.3, which is valid. The local file should be the same name as host- <tt>jail3</tt> in this example


check /tmp for crap
<pre>cd /usr/src/sys/amd64/conf
scp backup2:/mnt/data4/build/freebsd/kern_config-8.2-amd64 ./jail3</pre>


vi /etc/periodic/security/100.chksetuid
edit the kernel config and change ident to be the name of the jail:
replace: MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`
vi jail3
with: MP='/'
ident          jail3


mkdir -p /usr/compat/linux/dev
Optional, edit <tt>/sys/conf/newvers.sh</tt> to add <tt>–jc2</tt> to the end of the BRANCH string (RELEASE-jc2)
vi /sys/conf/newvers.sh


adduser (Add account for user)
notes on kernel configuring: http://www.FreeBSD.org/doc/en_US.ISO8859-1/books/handbook/kernelconfig-config.html


put user in wheel group
vi /etc/group


vi /usr/home/user/.profile (and add to the file):
=== install patches ===
TERM=vt100;    export TERM


cd /etc
The various patches are in /mnt/data4/build/freebsd/patches on backup2. There are dirs for each version. Not all dirs are populated, but patches for later versions work on older ones unless there is a new patch in the older dir.
vipw -d .
root:$1$krszPxhk$xkCepSnz3mIikT3vCtJCt0:0:0::0:0:Charlie &:/root:/bin/csh
user:$1$Mx9p5Npk$QdMU6c8YQqp2FW2M3irEh/:1001:1001::0:0:User &:/home/user:/bin/sh


tzsetup
there are no patches we use for 8.x, but here would be the commands:


newaliases
cd /usr/src
scp backup2:"/mnt/data4/build/freebsd/patches/8.0/*" .


chflags schg /dev/*mem
Apply patches, i.e. the <tt>jls-patch</tt>:
patch -l < jls-patch


cd /dev
rm console
ln -s null console


exit
=== build, install kernel and world ===
exit


cd /usr/ports
Rename current generic kernel so it will always be available to boot from. Save room by removing non-needed kernel modules:
make -DNOCLEANDEPENDS clean (2450: 47mins , supermicro: 17mins)
rm -fr /usr/ports/distfiles/*
cp -r /usr/ports /mnt/data1/jail-DIR/usr


cd /mnt/data1/jail-DIR/dev
<pre>cd /boot
rm kmem
mv kernel kernel.GENERIC
mknod kmem c 2 1 root:kmem
cd kernel.GENERIC
chmod 640 kmem
mkdir hold
rm mem
mv mfi_linux.ko hold/
mknod mem c 2 0 root:kmem
mv linux.ko hold/
chmod 640 mem
mv linprocfs.ko hold/
mv linsysfs.ko hold/
mv geom_vinum.ko hold/
mv geom_concat.ko hold/
mv zfs.* hold/
mv opensolaris* hold/
 
rm *.ko
rm *.symbols
mv hold/* .
rmdir hold/


rm /mnt/data1/jail-DIR/root/.history
cd /usr/src
make buildkernel installkernel


sh
make buildworld ; mail -s 'buildworld done' dave.boodman@vtext.com < /dev/null</pre>
for i in 1 2 3 4 5 6 7 ; do sh MAKEDEV pty$i ; done
~38mins
exit


cd /mnt/data1/jail-DIR/usr/compat/linux/dev
make installworld
mknod null c 2 2
~34min
mknod random c 2 3


cd
mergemaster -i
mkdir /usr/local/jail/template/
You will be prompted to merge, replace or ignore files changed by the src update. In most cases you can '''d'''elete the temp (new) files.
dump -0a -f /usr/local/jail/template/411template /dev/vn1


umount /dev/vn1c
ONLY if this will be a zfs system (not currently used in 8.x):
vnconfig -u /dev/vn1
<pre>cd /sys/modules/zfs
make
make install
cd /sys/modules/opensolaris
make
make install</pre>


rm /mnt/data1/jail
rm -fr /mnt/data1/jail-DIR


37. setup backups
===  populate devfs ruleset ===
cat > /usr/local/jail/bin/backup.config
scp backup2:/mnt/data4/build/freebsd/devfs.rules.8x /etc/devfs.rules
#!/bin/sh
backupdir=/mnt/data2/jail18_rsync


## ENTRY /etc
## ENTRY /usr/local/etc
## ENTRY /usr/local/jail
## ENTRY /root/logs


on backup2:
=== populate /etc/rc.conf with IPs and service settings ===
setup backup dirs:
mkdir -p /mnt/data2/jail18_rsync/0_today
mkdir -p /mnt/data2/jail18_rsync/1_yesterday
mkdir -p /mnt/data2/jail18_rsync/2_two_day


add the system to /mnt/data1/bin/rsync_houseclean
<pre>vi /etc/rc.conf


on mail:
kern_securelevel_enable="NO"
vi /usr/local/www/mgmt/cgi/backupgraph.pl
portmap_enable="NO"
(add hostname)
sendmail_enable="NO"
usbd_enable="YES"


38. mkdir /root/logs
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.4.103"
devfs_system_ruleset="devfsrules_show_all"


39. edit sshd_config for security
ifconfig_bce1="inet 10.1.4.103 netmask 255.255.255.0"
vi /etc/ssh/sshd_config
ifconfig_bce0="inet 69.55.229.7 netmask 255.255.255.0"
ListenAddress 69.55.228.101
#ifconfig_bce0_alias0="inet 69.55.2xx.xx netmask 255.255.255.0"
ListenAddress 10.1.4.104


kill -1 `cat /var/run/sshd.pid`
fsck_y_enable="YES"
background_fsck="NO"
#rc_mfi_raid_tty_log="YES"
#zfs_enable="YES"</pre>


40. add crontab entries
Modify IPs, hostname, gateway for this box.
crontab -e
5 0 * * * /usr/local/jail/bin/backup
1 0 1 * * /usr/local/jail/bin/ipfwreset
0 18 * * * /usr/local/jail/bin/ipfwbackup
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /usr/local/jail/bin/trafstats
55 10,23 * * * /usr/local/jail/bin/trafficwatch.pl


=== make sure sysctls are set and preserved upon boot ===


41. Reboot notify script
<pre>echo "kern.consmute=0\
cat > /usr/local/etc/rc.d/notify.sh
kern.ipc.shm_use_phys=1\
echo "`/bin/hostname` rebooted" | /usr/bin/mail reboot@johncompanies.com
kern.ipc.shmall=131070\
kern.ipc.shmmax=134217728\
net.inet.tcp.syncookies=0\
kern.maxfiles=32768\
kern.fallback_elf_brand=3\
kern.maxprocperuid=4000\
security.jail.sysvipc_allowed=1\
security.jail.allow_raw_sockets=1\
security.jail.socket_unixiproute_only=1\
security.jail.chflags_allowed=0\
dev.amr.0.allow_volume_configure=1\
compat.linux.osrelease=2.6.12\
vm.pmap.shpgperproc=500\
security.bsd.unprivileged_read_msgbuf=0\
kern.maxvnodes=400000" >> /etc/sysctl.conf</pre>


chmod +x /usr/local/etc/rc.d/notify.sh
Tuning note: watch vfs.numvnodes while the server is live to get guidance on where to set maxvnodes


42. copy jailmake from prev system
scp user@10.1.4.118:/usr/local/jail/bin/jailmake /usr/local/jail/bin
rehash
NOTE: remove df altering code from jailmake since we put the correct df in the template, and make sure path to template file is right


43. add to templates via mgmt system
=== mount procfs ===
echo "proc                    /proc          procfs  rw              0      0" >> /etc/fstab
echo "linprocfs              /usr/compat/linux/proc linprocfs rw    0      0" >> /etc/fstab
echo "linsysfs                /usr/compat/linux/sys linsysfs rw      0      0" >> /etc/fstab
mkdir -p /usr/compat/linux/proc
mkdir -p /usr/compat/linux/sys


44. add to server/cabinet map. On mail:
vi /usr/local/www/mgmt/html/cabinetmap.html


45. add an outside blocking rule to the firewall, so this machine can only be reached from inside the firewall. Follow example already in firewall jail17 is:
=== enable noatime option ===
   
data1 and data2 should look something like (add ',noatime' after 'rw'):
00117 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.228.2
  /dev/mfid0s1g          /mnt/data1      ufs    rw,noatime      2       2
00117 deny ip from any to 69.55.228.2
/dev/mfid1s1d          /mnt/data2      ufs    rw,noatime      2      2


jail4 would be 00104...
ipfw add 00118 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.228.140
ipfw add 00118 deny ip from any to 69.55.228.140


46. select customers for probe map
=== reboot. Confirm new kernel is loaded, devfs in place ===
</pre>


uname -a
Check devfs rules
devfs rule showsets
devfs rule -s 3 show


= 6.1 =
Should see:
<pre>#  devfs rule showsets
1
2
3
4


Last updated 2006-05-09
#  devfs rule -s 3 show
100 include 1
207 path pts* unhide
217 path fd unhide
218 path fd/* unhide
300 path ttyp* unhide
301 path ttyq* unhide
302 path ttyr* unhide
303 path ttys* unhide
304 path ttyP* unhide
305 path ttyQ* unhide
306 path ttyR* unhide
307 path ttyS* unhide
400 path null unhide
500 path zero unhide
600 path random unhide
610 path urandom unhide
700 path mem unhide
710 path kmem unhide
810 path mdctl unhide
900 path stdin unhide
910 path stdout unhide
920 path stderr unhide</pre>


<pre>All time extimates below assume disks aren’t scrubbing


1. make sure bios is setup for bios console redirect
=== update ports ===
Supermicro:
Console redirection:
Com port addr: on-board COM A
Baud: 38400
Console type: vt100
Flow control: none
Console connection: direct
Continue cr after post: off


2450:
<pre>cd /usr/ports
Make sure running bios A09
echo "*default host=cvsup4.FreeBSD.org\
Console Redirection: VT100/VT220
*default base=/usr\
 
*default prefix=/usr\
2. assuming mirrors (or at least disks) created (if not, refer to this), boot to disk 1 of 5.4
*default release=cvs tag=RELENG_8_3\
skip kernel config (enter)
*default delete use-rel-suffix\
custom install
*default compress\
ports-all tag=." > sup


partition ->
cvsup sup; mail -s 'cvs sup ports done' support@johncompanies.com < /dev/null</pre>
move cursor to amrd0, hit space (takes you to partition map screen)
~24mins
a for entire disk
q to quit and save
standard mbr (no boot manager)
space to unselect aacd0
cursor over aacd1
space
a for entire disk
q to quit and save
none (leave untouched)


cursor over aacd0
space
(takes you into part. Screen again) q to exit
none
Make sure both are checked and tab to ok


Label ->
=== Install raid mgmt tool ===
Make sure aacd0 is highlighted
a to start with defaults


john likes:
==== Perc5/i, 6/i ====
/1g
Pull over cli from previous system (jail9)
/var 256m
scp /usr/local/sbin/mega* root@10.1.4.109:/usr/local/sbin/
scp /usr/local/libexec/MegaCli root@10.1.4.109:/usr/local/libexec/MegaCli
These are linux-based tools. This will require linux base which we would normally install from ports, but since it's failing lately, we just pull in the compat libraries from another system:


Glenn likes:
rsync -aSHv --exclude=proc --exclude=sys 10.1.4.109:/usr/compat/linux/ /usr/compat/linux/
/128
/var 256
/usr 3g
/tmp 256


We do:
Test:
delete everything
rehash; megacli ldinfo lall a0
/ 128M
swap 2G
/var 256M
/tmp 256M
/usr 3G
/mnt/data1 remaining space


Make sure to toggle S for soft updates on all (should look like UFS2+S Y under the Newfs column)
DEPRECATED: Assuming it worked, here's how we ''used to'' install linux_base:
----
cd /usr/ports/emulators/linux_base-fc4
make install clean
(didnt succeed due to libtool requirement)


move cursor to aacd1
cd /usr/ports/distfiles
swap 2G
fetch http://www.lsi.com/DistributionSystem/AssetDocument/support/downloads/megaraid/miscellaneous/linux/2.00.15_Linux_MegaCLI.zip
/mnt/data2 remaining space
cd /usr/ports/sysutils/linux-megacli
make install clean
(also failed due to libtool)
----


q to save and exit


distributions ->
==== 2850 PERC 4e/Di- no linux ====
developer
<pre>cd /usr/ports/distfiles/
yes to install ports
fetch http://backup01.best-hosting.ru/pub/FreeBSD/ports/distfiles/dr_freebsd_1.51.zip
exit
cd /usr/ports/sysutils/megarc
make install clean
megarc -dispCfg -a0</pre>


media ->
cd


commit ->
=== install rsync from ports ===
yes
cd /usr/ports/net/rsync
(2450: 14mins, supermicro: 12mins)
make install clean


yes to "visit general config" ->
choose default options
Set root pwd


Add user ‘user’ member group is wheel, set password


Set tz
=== configure inetd to respond to mrtg load queries ===
echo "load    stream  tcp    nowait  user    /usr/local/jail/bin/load.pl  load.pl" >> /etc/inetd.conf
echo "load            12384/tcp" >> /etc/services


Networking->interfaces->Fxp0 ->
No IPV6
dhcp=no
Set hostname & domain
Enable sshd


exit...
=== install perl ===
exit install ->
cd /usr/ports/lang/perl5.12
yes
make install clean


take the cd out and let the machine reboot
choose defaults
when it comes back up, enter junk when it asks for key seed




3. double check the date/time
=== install bb client ===


4. edit /etc/make.conf (only add the console speed line if this is a supermicro capable of outputting BIOS redirect at that speed – 2450’s can only do 9600)
Compiling from source on AMD64 will not work. So, we use a linux-compiled version and rely on linux compat. Per [[#Perc5.2Fi.2C_6.2Fi|above]], linux compat won't install on 8.x - libtool 2.4 need. So, instead we copy(ed) over linux:
echo "WITHOUT_X11=yes \
rsync -aSHv --exclude=proc --exclude=sys 10.1.4.108:/usr/compat/linux/ /usr/compat/linux/
KERNCONF=jail19 \
BOOT_COMCONSOLE_SPEED=38400" >> /etc/make.conf


5. add console="comconsole" to /boot/loader.conf
adduser
echo "console=""comconsole""" >> /boot/loader.conf


6. turn off all ttyv's except 0 and 1 in /etc/ttys
Output/response:
also turn on ttyd0, change type to vt100:
vi /etc/ttys
ttyd0  "/usr/libexec/getty std.9600"  vt100  on secure


kill -1 1
<pre>Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username  : bb
Password  : <random>
Full Name  : bb
Uid        : 1984
Class      :
Groups    : bb
Home      : /home/bb
Shell      : /bin/sh
Locked    : no
OK? (yes/no): yes</pre>


on console server:
cd /usr/home/bb
vi /etc/remote
scp backup2:/mnt/data4/build/bb/bb-freebsd_linuxcompat.tgz .
(rename port to jail18 depending on where and which digi plugged into)
tar xzf bb-freebsd_linuxcompat.tgz
test serial console


7. install cvsup
edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
cd /usr/ports/net/cvsup-without-gui
make install clean; rehash
(stay close for gettext options, 2450: 27mins, supermicro: 17mins)


8. get latest sources for this release:
echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
cd /usr/src
10.1.4.'''103 jail3'''.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_1\
*default delete use-rel-suffix\
*default compress\
src-all" > sup


cvsup sup
Edit for machine name and private IP.
(2450, ~12mins, supermicro, 27mins)


9. populate hosts
if this machine is at i2b:
echo "10.1.4.3 backup2" >> /etc/hosts
echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
69.55.229.7 jail3.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts


10. put key in authorized_keys on backup2
<pre>vi /home/bb/bbc1.9e-btf/ext/openfiles
cd
ssh-keygen -t dsa -b 1024
(default location, leave password blank)
scp /root/.ssh/id_dsa.pub user@backup2:/tmp/jailkey


cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
MACHINE="jail3,johncompanies,com"      # HAS TO BE IN A,B,C FORM</pre>
Edit for machine name.


confirm that you can ssh to backup2 without getting a login prompt
<pre>cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf
./runbb.sh start
more BBOUT
(look for errors)
exit</pre>


ssh backup2
Put in script to start bb @ boot:
echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh
 
=== remove reserve space ===
<pre>cd
umount /mnt/data1
umount /mnt/data2
tunefs -m 0 /mnt/data1
tunefs -m 0 /mnt/data2
mount -a</pre>


11. configure new kernel. Get config from similar machine or there may be a master copy somewhere under /mnt/data4/build/freebsd (name the kernel config the same as the jail, ex jail18):
cd /usr/src/sys/i386/conf
scp backup2:/mnt/data4/build/freebsd/kern_config-6.1 ./jail19


edit the kernel config and change ident to be the name of the jail:
=== setup rdate ===
vi jail14
cd /usr/ports/sysutils/rdate
ident          jail14
make install clean


edit /sys/conf/newvers.sh to add –jc1 to the end of the BRANCH string (RELEASE-jc1)
crontab -e
vi /sys/conf/newvers.sh
0 0 * * * /usr/local/sbin/rdate -s utcnist.colorado.edu


12. bring over patches from backup2
/usr/local/sbin/rdate -s utcnist.colorado.edu
The various patches are in /mnt/data4/build/freebsd/patches on backup2. There are dirs for each version. Not all dirs are populated, but patches for later versions work on older ones unless there is a new patch in the older dir.
cd /usr/src
scp backup2:"/mnt/data4/build/freebsd/patches/6.1/*" .
jail_proc_patch      100% |*************************************************|  2593    00:00
restore-patch        100% |*************************************************|  3295    00:00
54ps-jail-patch      100% |*************************************************|  2602    00:00
jail_rtprio_patch    100% |*************************************************|  301    00:00
udp-patch            100% |*************************************************|  594    00:00


Apply patches:
We used to use ntpd, however it listens on jail IPs which is a security risk. So we stopped. Here's the old instructions:
patch < 54ps-jail-patch
install new ntp from ports
patch < jail_proc_patch
/usr/ports/net/ntp
patch < restore-patch
make install clean
patch < jail_rtprio_patch
patch < udp-patch


13. build, install kernel and world
<pre>echo "server 10.1.4.5" > /etc/ntp.conf
cd /usr/src
/usr/sbin/ntpd -p /var/run/ntpd.pid
make buildworld buildkernel installkernel
sleep 2; ntpq -p</pre>(confirm it’s able to reach our time server)
(2450: 1:56min, supermicro::59mins)
make installworld
(2450: 3min, supermicro: 1min)
mergemaster -i
delete /var/tmp/temproot
delete bsnmpd
delete temporary ./etc/hosts
delete temporary ./etc/motd
delete /var/tmp/temproot


14. reboot. Confirm new kernel is loaded


uname –a
=== make a data partition ===


15. update ports:
IF you didn't format the data partition during sysinstall:
cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_1\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup


cvsup sup
Create a g partition on 2nd mirror – bsdlabel no longer works (below shows d partition made with sysinstall):
(2450: 18mins, supermicro: 19mins)


18. (only applies if adaptec card installed)
<pre>jail8 /usr/home/bb# gpart show
install linux_base:
=>      63  285474735  mfid0  MBR  (136G)
cd /usr/ports/emulators/linux_base
        63  285458922      1  freebsd  [active]  (136G)
make install clean
  285458985      15813        - free -  (7.7M)
(2450: 7min, supermicro: 3mins)


reibuild rpmdb cause we had probs installing aacapps
=>        0  285458922  mfid0s1  BSD  (136G)
cd /compat/linux/bin
          0    524288        1  freebsd-ufs  (256M)
./rpm --initdb
    524288  12582912        2  freebsd-swap  (6.0G)
./rpm --rebuilddb
  13107200    524288        4  freebsd-ufs  (256M)
  13631488    524288        5  freebsd-ufs  (256M)
  14155776    8388608        6  freebsd-ufs  (4.0G)
  22544384  262914538        7  freebsd-ufs  (125G)


install aacapps-4.1-0.i386.rpm
=>      63  584843175  mfid1  MBR  (279G)
scp backup2:/mnt/data4/build/freebsd/aacapps-4.1-0.i386.rpm /tmp/.
        63  584830197      1 freebsd [active]  (279G)
/compat/linux/bin/rpm -ivh --excludepath=/dev /tmp/aacapps-4.1-0.i386.rpm
  584830260      12978        - free - (6.3M)


test out;
=>        0  584830197  mfid1s1  BSD  (279G)
/compat/linux/usr/sbin/aaccli
          0  16777216        2  freebsd-swap  (8.0G)
  16777216  568052981        4  freebsd-ufs  (271G)


jail8 /usr/home/bb# gpart show mfid1s1
=>        0  584830197  mfid1s1  BSD  (279G)
          0  16777216        2  freebsd-swap  (8.0G)
  16777216  568052981        4  freebsd-ufs  (271G)


20. edit root's path and login script:
# gpart list mfid1s1
vi /root/.cshrc
add to path:
/usr/local/jail/bin
(if adaptec card installed, also add /compat/linux/usr/sbin)


Change alias entries (add G):
Geom name: mfid1s1
alias la        ls -aG
fwheads: 255
alias lf        ls -FAG
fwsectors: 63
alias ll        ls -lAG
last: 584830196
alias ls        ls -AG
first: 0
alias mbm      mb mount
entries: 8
alias mbu      mb umount
scheme: BSD
 
Providers:
and alter the prompt, set the following:
1. Name: mfid1s1b
set prompt = "`/bin/hostname -s` %/# "
  Mediasize: 8589934592 (8.0G)
 
  Sectorsize: 512
at the bottom of the file add:
  Mode: r1w1e0
set sshtty=`who am i|awk '{print $2}'`
  rawtype: 1
/usr/sbin/rtprio 3 -`psj | grep $sshtty | awk '{print $2}'`
  length: 8589934592
 
  offset: 0
set shortty=`who am i | awk '{print $2}' | sed -E 's/.*(..)$/\1/'`
  type: freebsd-swap
foreach x (`psj | grep sh | grep $shortty | awk '{print $2}'`)
  index: 2
/usr/sbin/rtprio 2 -$x
  end: 16777215
end
  start: 0
2. Name: mfid1s1d
21. install rsync from ports
  Mediasize: 290843126272 (271G)
cd /usr/ports/net/rsync
  Sectorsize: 512
make install clean
  Mode: r0w0e0
  rawtype: 7
  length: 290843126272
  offset: 8589934592
  type: freebsd-ufs
  index: 4
  end: 584830196
  start: 16777216
Consumers:
1. Name: mfid1s1
  Mediasize: 299433060864 (279G)
  Sectorsize: 512
  Mode: r1w1e1


choose default options
# gpart delete -i 4 mfid1s1
mfid1s1d deleted
jail8 /usr/home/bb# gpart list mfid1s1
Geom name: mfid1s1
fwheads: 255
fwsectors: 63
last: 584830196
first: 0
entries: 8
scheme: BSD
Providers:
1. Name: mfid1s1b
  Mediasize: 8589934592 (8.0G)
  Sectorsize: 512
  Mode: r1w1e0
  rawtype: 1
  length: 8589934592
  offset: 0
  type: freebsd-swap
  index: 2
  end: 16777215
  start: 0
Consumers:
1. Name: mfid1s1
  Mediasize: 299433060864 (279G)
  Sectorsize: 512
  Mode: r1w1e1


21. install perl from ports
# gpart show mfid1s1
PROB NOT NECESSARY – INSTALLED WITH LINUX I THINK
=>        0  584830197  mfid1s1  BSD  (279G)
cd /usr/ports/lang/perl5.8/
          0  16777216        2  freebsd-swap  (8.0G)
make install clean; rehash
  16777216  568052981          - free -  (271G)
(supermicro: 5min)


22. create & populate binaries/scripts dirs
# gpart add  -t freebsd-ufs -i 7 mfid1s1
mkdir -p /usr/local/jail/bin
mfid1s1g added
mkdir -p /usr/local/jail/rc.d
mkdir /mnt/data1
mkdir /mnt/data2
scp backup2:"/mnt/data4/bin/freebsd6.x/*" /usr/local/jail/bin
cd /usr/local/jail/rc.d/
touch quad1
touch quad2
touch quad3
touch quad4
touch safe1
touch safe2
touch safe3
touch safe4
chmod +x *
cd /usr/local/jail/bin
ln -s /usr/local/jail/rc.d/quad1 quad1
ln -s /usr/local/jail/rc.d/quad2 quad2
ln -s /usr/local/jail/rc.d/quad3 quad3
ln -s /usr/local/jail/rc.d/quad4 quad4
ln -s /usr/local/jail/rc.d/safe1 safe1
ln -s /usr/local/jail/rc.d/safe2 safe2
ln -s /usr/local/jail/rc.d/safe3 safe3
ln -s /usr/local/jail/rc.d/safe4 safe4


rehash
# gpart show mfid1s1
=>        0  584830197  mfid1s1  BSD  (279G)
          0  16777216        2  freebsd-swap  (8.0G)
  16777216  568052981        7  freebsd-ufs  (271G)</pre>


23. configure inetd to respond to mrtg load queries
Here's how we USED to do it with bsdlabel:
echo "load    stream  tcp    nowait  user    /usr/local/jail/bin/load.pl  load.pl" >> /etc/inetd.conf


echo "load            12384/tcp" >> /etc/services
bsdlabel -e /dev/mfid0s1


26. install bb client
given:
adduser
<pre># /dev/aacd0s1:
cd /usr/home/bb
8 partitions:
scp backup2:/mnt/data4/build/bb/bb-freebsd.tar .
#        size  offset    fstype  [fsize bsize bps/cpg]
tar xvf bb-freebsd.tar
  a:  262144        0    4.2BSD    2048 16384 16392
  b: 4194304  262144      swap
  c: 143363997        0    unused        0    0        # "raw" part, don't edit
  d:  524288  4456448    4.2BSD    2048 16384 32776
  e:  524288  4980736    4.2BSD    2048 16384 32776
  f:  6291456  5505024    4.2BSD    2048 16384 28552</pre>


edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
new offset = 6291456 + 5505024 = 11796480<br>
echo "10.1.4.105 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
new size is size for 'c' partition minus the new start from above<br>
10.1.4.103 jail3.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
143363997 - 11796480 = 131567517


vi /home/bb/bbc1.9e-btf/ext/openfiles
So:<br>
MACHINE="jail19,johncompanies,com"      # HAS TO BE IN A,B,C FORM
g: 131567517 11796480 unused 0 0


cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh (y to questions)
./bbchkhosts.sh (ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src
make; make install
cd ..
./runbb.sh start
more BBOUT
(look for errors)
exit


echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
=== create the jail template ===
chmod +x /usr/local/etc/rc.d/bb.sh


modify bb-msgtab to look for raid failures
cd /usr/ports/sysutils/jailutils
make install clean


24. configure load mrtg, on mail
Create an md device to hold the jail:
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg
<pre>touch /mnt/data1/jail-template20g
(add new entry to file following existing format)
mdconfig -a -t vnode -s 20g -f /mnt/data1/jail-template20g -u 0
newfs -O 1 /dev/md0
mkdir /mnt/data1/jail-DIR
mount /dev/md0 /mnt/data1/jail-DIR</pre>


25. configure bb on mail:
Build world into the jail:
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
<pre>cd /usr/src
10.1.4.119 jail19.johncompanies.com # ssh
make world DESTDIR=/mnt/data1/jail-DIR; taskdone</pre>
~1hr


su bb
Make /etc into the jail, mount dev, copy in jkill:
cd
<pre>cd etc
bbsrc/bb/runbb.sh restart ; exit
make distribution DESTDIR=/mnt/data1/jail-DIR
mount -t devfs devfs /mnt/data1/jail-DIR/dev
devfs -m /mnt/data1/jail-DIR/dev rule -s 3 applyset
cd /mnt/data1/jail-DIR
ln -sf dev/null kernel
cp /usr/local/sbin/jkill /mnt/data1/jail-DIR/sbin</pre>


27. remove reserve space, enable softupdates (probably already set, so not necessary)
Enter into jail to do configuration:
NOT APPLICABLE IF USING GVINUM
jail /mnt/data1/jail-DIR testhostname 192.168.11.100 /bin/sh
cd
csh
umount /mnt/data1
umount /mnt/data2
tunefs -m 0 /mnt/data1
tunefs -m 0 /mnt/data2
mount -a


28. configure ntp
Create fstab:
echo "server 10.1.4.105" > /etc/ntp.conf
<pre>touch /etc/fstab
echo 'network_interfaces=""\
hostname="newsystem"\
kern_securelevel_enable="NO"\
sendmail_enable="YES"\
sshd_enable="YES"' > /etc/rc.conf


/usr/sbin/ntpd -p /var/run/ntpd.pid
echo "nameserver 69.55.225.225\
sleep 2; ntpq -p
nameserver 69.55.230.3" >> /etc/resolv.conf</pre>
(confirm it’s able to reach our time server)


29. mrtg switch graphs
Edit crontab:
vi /etc/crontab
remove the adjkerntz lines
comment out periodic’s and put this line above them:
# IF YOU UNCOMMENT THESE, PLEASE ADJUST THEIR RUN TIME


31. fwd and reverse lookups on ns1c
rm -rf /etc/periodic/daily/400.status-disks
vr johncompanies.com
vi internal.johncompanies.com
rndc reload johncompanies.com IN private
(edit the PTR too)


33. populate /etc/rc.conf with IPs and NFS settings
check and remove any crap in /tmp
vi /etc/rc.conf


kern_securelevel_enable="NO"
vi /etc/periodic/security/100.chksetuid
portmap_enable="NO"
replace: <tt>MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`</tt><br>
sendmail_enable="NO"
with: <tt>MP='/'</tt> (use single quotes)
usbd_enable="YES"


xntpd_enable="YES"
mkdir -p /usr/compat/linux/dev
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
sshd_enable="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.4.119"
devfs_system_ruleset="devfsrules_show_all"


ifconfig_xl0="inet 10.1.4.118 netmask 255.255.255.0"
Add account for user. Output/response:
ifconfig_fxp0="inet 69.55.228.101 netmask 255.255.255.0"
<pre>adduser
defaultrouter="69.55.228.1"
ifconfig_fxp0_alias0="inet 69.55.2xx.xx netmask 255.255.255.0"


static_routes="t1 office"
Username: user
route_t1="-net 10.1.5 10.1.4.2"
Full name: user
route_office="-net 10.1.6 10.1.4.2"
Uid (Leave empty for default):
gvinum_enable="YES"
Login group [user]:
fsck_y_enable="YES"
Login group is user. Invite user into other groups? []: wheel
background_fsck="NO"
Login class [default]:
 
Shell (sh csh tcsh nologin) [sh]:
34. make sure sysctls are set and preserved after reboot
Home directory [/home/user]:
echo "kern.consmute=0\
Home directory permissions (Leave empty for default):
kern.ipc.shm_use_phys=1\
Use password-based authentication? [yes]:
kern.ipc.shmall=65535\
Use an empty password? (yes/no) [no]:
kern.ipc.shmmax=134217728\
Use a random password? (yes/no) [no]: y
net.inet.tcp.syncookies=0\
Lock out the account after creation? [no]:
kern.maxfiles=32768\
Username  : user
kern.fallback_elf_brand=3\
Password  : <random>
kern.maxprocperuid=4000\
Full Name  : user
security.jail.sysvipc_allowed=1\
Uid        : 1001
security.jail.max_procs_per_jail: 1026\
Class      :
security.jail.allow_raw_sockets=1\
Groups    : user
security.jail.socket_unixiproute_only=1\
Home      : /home/user
security.jail.chflags_allowed=0" >> /etc/sysctl.conf
Home Mode  :
Shell      : /bin/sh
Locked    : no
OK? (yes/no): y
adduser: INFO: Successfully added (user) to the user database.
adduser: INFO: Password for (user) is: 901gmYjO
Add another user? (yes/no): n
Goodbye!</pre>


35. mount procfs
set TERM:
echo "proc                    /proc          procfs  rw              0      0" >> /etc/fstab
vi /usr/home/user/.profile
TERM=vt100;    export TERM


36. enable noatime option
Set time zone to PT:
NOT APPLICABLE IF RUNNING GVINUM
tzsetup
data1 and data2 should look something like:
/dev/amrd0s1g          /mnt/data1      ufs    rw,noatime      2      2


36. populate devfs ruleset
Reload aliases:
scp backup2:/mnt/data4/build/freebsd/devfs.rules /etc
newaliases


35. reboot
Replace reboot/halt:
Check rules:
rm /sbin/halt /sbin/reboot
devfs rule showsets
ln /sbin/jkill /sbin/halt
devfs rule -s 3 show
ln /sbin/jkill /sbin/reboot


36. create gvinum volumes
Redirect console output, comment out console and move to /var/log/messages:
vi /etc/syslog.conf
#*.err;kern.warning;auth.notice;mail.crit              /dev/console
*.err;kern.warning;auth.notice;mail.crit                /var/log/messages


Make a g partition:
exit
exit


bsdlabel -e /dev/aacd0s1
Lib32 compat library:
cd libexec
chflags noschg ld-elf32.so.1
chflags noschg ld-elf.so.1
mv ld-elf32.so.1 ld-elf32.so.1-orig
ln ld-elf.so.1 ld-elf32.so.1
chflags schg ld-elf.so.1
chflags schg ld-elf32.so.1


given:
Replace traceroute:
# /dev/aacd0s1:
mv /mnt/data1/jail-DIR/usr/sbin/traceroute /mnt/data1/jail-DIR/usr/sbin/_traceroute
8 partitions:
echo '#\!/bin/sh\
#       size  offset    fstype  [fsize bsize bps/cpg]
  /usr/sbin/_traceroute -i bce0 $1' >> /mnt/data1/jail-DIR/usr/sbin/traceroute
  a:  262144        0    4.2BSD    2048 16384 16392
  chmod +x /mnt/data1/jail-DIR/usr/sbin/traceroute
  b: 4194304  262144      swap
  c: 143363997        0    unused        0    0        # "raw" part, don't edit
  d:  524288 4456448    4.2BSD    2048 16384 32776
  e:  524288  4980736    4.2BSD    2048 16384 32776
  f:  6291456  5505024    4.2BSD    2048 16384 28552


new offset = 6291456 + 5505024 = 11796480
Modify 'bce0' to reflect whichever nic is public on this hardware.
new size is size for 'c' partition minus the new start from above
143363997 - 11796480 = 131567517
So:
g: 131567517 11796480 unused 0 0


For a 73G drive (after OS), we can fit 31 2G volumes so:
Clean out ports before copying into the jail (~30mins):
cd /usr/ports
make -DNOCLEANDEPENDS clean
rm -fr /usr/ports/distfiles/*
cp -r /usr/ports /mnt/data1/jail-DIR/usr


echo 'drive data1 device /dev/aacd0s1g' > /tmp/cgv
rm /mnt/data1/jail-DIR/root/.history
sh
for f in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31;\
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 2g drive data1' >> /tmp/cgv; done; exit


gvinum create /tmp/cgv
Umount the jail and dump it:
<pre>cd /
umount /mnt/data1/jail-DIR/dev
dump -0a -f /usr/local/jail/template/template /dev/md0
umount /dev/md0
rmdir /mnt/data1/jail-DIR
mdconfig -d -u 0</pre>


For a 146G drive (-4G for swap), we can fit 66 2G volumes so:


echo 'drive data2 device /dev/aacd1s1g' > /tmp/cgv
=== setup backups ===
sh
<pre>echo '#\!/bin/sh\
for f in 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97;\
backupdir=/data/jail3\
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
server=backup1\
echo 'sd length 2g drive data2' >> /tmp/cgv; done; exit
\
## ENTRY /etc\
## ENTRY /usr/local/etc\
## ENTRY /usr/local/jail\
## ENTRY /root/logs' > /usr/local/jail/bin/backup.config</pre>


gvinum create /tmp/cgv
Edit to reflect backup server and jail hostname


For 3rd 73G drive (after 2G swap), we can fit 33 2G volumes so:
On backup server, setup backup dirs:
Label should be:
ssh backup1 mkdir -p /data/jail3/0
# /dev/aacd2s1:
8 partitions:
#        size  offset    fstype  [fsize bsize bps/cpg]
  c: 143299737        0    unused        0    0        # "raw" part, don't edit
  g: 143299721        16    unused        0    0


echo 'drive data3 device /dev/aacd2s1g' > /tmp/cgv
On backup server, setup backup dirs:
sh
backup1# vi /usr/local/sbin/snapshot_rotate
for f in 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131;\
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 2g drive data3' >> /tmp/cgv; done


gvinum create /tmp/cgv
Edit /usr/local/jail/bin/backup.xxx to use the right drives and copy to /usr/local/jail/bin/backup
vi /usr/local/jail/bin/backup.md
adjust df so it includes all relevant drives, currently <tt>df > /etc/df.bak</tt> is fine. Also, make sure the binary/script source is pulling from the right dir on backup2:
/usr/local/bin/rsync -a backup2:/mnt/data4/bin/freebsd8.x/ /usr/local/jail/bin/


For a 2nd 73G drive (after 2G swap), we can fit 33 2G volumes so:
cp /usr/local/jail/bin/backup.md /usr/local/jail/bin/backup


echo 'drive data2 device /dev/aacd1s1g' > /tmp/cgv
sh
for f in 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64;\
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 2g drive data2' >> /tmp/cgv; done; exit


gvinum create /tmp/cgv
=== create /root/logs ===
mkdir /root/logs




36. create the jail template
=== edit sshd_config ===
vi /etc/ssh/sshd_config
ListenAddress 69.55.229.7
ListenAddress 10.1.4.103


mkdir /mnt/jail
'''''Adjust to pub/private IP for jail.'''
newfs /dev/gvinum/v1
''
mount /dev/gvinum/v1 /mnt/jail
Restart sshd:
kill -1 `cat /var/run/sshd.pid`


cd /usr/src
=== add crontab entries ===
make clean
<pre>crontab -e
rm -fr /usr/obj/
5 0 * * * /usr/local/jail/bin/backup
make world DESTDIR=/mnt/jail
1 0 1 * * /usr/local/jail/bin/ipfwreset
(2450: 2:28mins, supermicro: 55mins)
0 18 * * * /usr/local/jail/bin/ipfwbackup
cd etc
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /usr/local/jail/bin/trafstats
make distribution DESTDIR=/mnt/jail
0 0,6,12,18 * * * /usr/local/jail/bin/sync_jail_names</pre>
mount_devfs devfs /mnt/jail/dev
devfs -m /mnt/jail/dev rule -s 3 applyset
cd /mnt/jail
ln -sf dev/null kernel


jail /mnt/jail testhostname 192.168.11.100 /bin/sh
On 2950:
csh
*/5 * * * * /usr/local/jail/bin/perc5iraidchk
touch /etc/fstab
echo 'network_interfaces=""\
hostname="newsystem"\
kern_securelevel_enable="NO"\
sendmail_enable="YES"\
sshd_enable="YES"' > /etc/rc.conf


echo "nameserver 69.55.225.225\
On 3ware-based servers:
nameserver 69.55.230.3" >> /etc/resolv.conf
0 0 * * * /usr/local/jail/bin/3wraidchk


vi /etc/crontab
remove the adjkerntz lines
comment out periodic’s and put this line above them:
# DO NOT UNCOMMENT THESE


rm -rf /etc/periodic/daily/400.status-disks
=== Reboot notify script ===
ln -s /usr/local/jail/bin/notify.sh /usr/local/etc/rc.d/notify.sh


check /tmp for crap


vi /etc/periodic/security/100.chksetuid
=== patch jail against starting jails with rtprio ===
replace: MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`
with: MP='/' (use single quotes)


mkdir -p /usr/compat/linux/dev
<pre>mv /usr/sbin/jail /usr/sbin/jail_
echo '#\!/bin/sh\
/usr/sbin/rtprio -t /usr/sbin/jail_ $*' > /usr/sbin/jail
chmod +x /usr/sbin/jail</pre>


adduser (Add account for user, make sure in wheel group)


user/root passwd: 8ico2987
=== make sure mail works ===
If there are map errors:
cd /etc/mail; make maps


Set root password


vi /usr/home/user/.profile (and add to the file):
=== recover space on /usr (optional) ===
TERM=vt100;    export TERM


tzsetup
rm -fr /usr/obj


newaliases


#cd /dev
=== wrapper for jps ===
#rm console
#ln -s null console


exit
mv /usr/local/sbin/jps /usr/local/sbin/jps_
exit


cd /usr/ports
make -DNOCLEANDEPENDS clean
(2450: 15mins , supermicro: 29mins)
rm -fr /usr/ports/distfiles/*
cp -r /usr/ports /mnt/jail/usr (2450: 2:00 mins , supermicro: 15mins)


rm /mnt/jail/root/.history
=== wrapper for jls ===


cd
mv /usr/sbin/jls /usr/sbin/jls_
mkdir /usr/local/jail/template/
dump -0a -f /usr/local/jail/template/61template /dev/gvinum/v1


umount /mnt/jail/dev
umount /dev/gvinum/v1


rm -fr /mnt/jail
=== wrapper for jexec ===


37. setup backups
mv /usr/sbin/jexec /usr/sbin/jexec_
echo '#\!/bin/sh\
backupdir=/mnt/data3jail3_rsync\
\
## ENTRY /etc\
## ENTRY /usr/local/etc\
## ENTRY /usr/local/jail\
## ENTRY /root/logs' > /usr/local/jail/bin/backup.config


on backup2:
setup backup dirs:
mkdir -p /mnt/data3/jail3/0


add the system to
=== install jtop ===
vi /mnt/data4/bin/snapshot_rotate


on mail:
cd /usr/ports/sysutils/jtop
vi /usr/local/www/mgmt/cgi/backupgraph.pl
make install clean
(add hostname)


38. mkdir /root/logs


39. edit sshd_config for security
=== block jails from reaching private net ===
vi /etc/ssh/sshd_config
echo 'ipfw add 1 deny ip from 69.55.224.0/20 to 10.1.4.0/24' > /usr/local/etc/rc.d/ipfw.sh
ListenAddress 69.55.228.101
chmod 0700 /usr/local/etc/rc.d/ipfw.sh
ListenAddress 10.1.4.118


kill -1 `cat /var/run/sshd.pid`
== add to management infrastructure ==


40. add crontab entries
=== mail ===
crontab -e
5 0 * * * /usr/local/jail/bin/backup
1 0 1 * * /usr/local/jail/bin/ipfwreset
0 18 * * * /usr/local/jail/bin/ipfwbackup
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /usr/local/jail/bin/trafstats
55 10,23 * * * /usr/local/jail/bin/trafficwatch.pl


==== add to management db ====


41. Reboot notify script
tables: jc.ref_machines and jc.ref_templates
ln -s /usr/local/jail/bin/notify.sh /usr/local/etc/rc.d/notify.sh


42. copy jailmake from prev system
on jail run:
scp user@10.1.4.119:/usr/local/jail/bin/jailmake /usr/local/jail/bin
uname -r
rehash
NOTE: remove df altering code from jailmake since we put the correct df in the template, and make sure path to template file is right


43. add to management db (on mail and devweb) jc.ref_machines and jc.ref_templates
Which shows something like:
8.3-RELEASE-p2


uname -r
Insert into db:
5.4-RELEASE-p2-jc2


insert into ref_machines values (null,'jail19','jail19.johncompanies.com',0,'l');
<pre>insert into ref_machines values (null,'jail3','mx3.johncompanies.com',0,'f8');
select machine_id from ref_machines where host='jail19';
select machine_id from ref_machines where host='jail3';
+------------+
+------------+
| machine_id |
| machine_id |
Line 2,351: Line 2,119:
|        35 |
|        35 |
+------------+
+------------+
insert into ref_templates values ('',' 6.2-RELEASE-jc1',5,'FreeBSD 6.2',0);
insert into ref_templates values ('','8.3-RELEASE-jc2',35,'FreeBSD 8.3',1);</pre>


44. add to server/cabinet map. On mail:
==== add to bb server ====
vi /usr/local/www/mgmt/html/cabinetmap.html
vi /usr/home/bb/bbsrc/bb1.9i-btf/etc/bb-hosts
10.1.4.109 jail9.johncompanies.com # ssh


45. add an outside blocking rule to the firewall, so this machine can only be reached from inside the firewall. Follow example already in firewall jail17 is:
In the case of an i2b server, use real ip:
   
  69.55.229.7 jail3.johncompanies.com # ssh
00119 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.228.200
 
00119 deny ip from any to 69.55.228.200
<pre>su bb
cd
bbsrc/bb/runbb.sh restart ; exit</pre>


jail19 would be 00119...
==== Update backupgraph ====
ipfw add 00119 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.228.200
vi /usr/local/www/mgmt/cgi/backupgraph.pl
ipfw add 00119 deny ip from any to 69.55.228.200
(add hostname)


46. select customers for probe map
==== Update load mrtg ====
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg
(add new entry to file following existing format)


47. install raid monitor
=== ns1c ===
fwd and reverse lookups:
vr johncompanies.com


cd /usr/ports/sysutils/asr-utils
vi internal.johncompanies.com
make install clean
rndc reload johncompanies.com IN private


48. make gv start on boot
ptr 69.55.227.x


=== wiki ===
add to server/cabinet map


scp backup2:/mnt/data4/build/freebsd/gvinum /etc/rc.d/gvinum
=== firewall ===
add an outside blocking rule to the firewall, so this machine can only be reached from inside the firewall. Follow example already in firewall jail17 is:
00117 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.228.200
00117 deny ip from any to 69.55.228.200


jail19 would be 00119...
ipfw add 00109 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 } to 69.55.232.3
ipfw add 00109 deny ip from any to 69.55.232.3


=== select some customers for castle probe map ===


= FreeBSD 4.11 =


gconcat label -v somelabel /dev/gvinum/a /dev/gvinum/b
<pre>
bsdlabel -r -w /dev/concat/somelabel
4.11
newfs /dev/concat/somelabela
Last updated 2006-1-26
mount /dev/concat/somelabel /mount/point
 
umount /dev/concat/somelabel
gconcat stop somelabel
gconcat label -v /dev/gvinum/a /dev/gvinum/b /dev/gvinum/c /dev/gvinum/d
growfs /dev/concat/somelabel
mount /dev/concat/somelabel /mount/point
 
volume f
        plex org concat
        sd length 30449m drive data1
 
where f is
 
D data1                State: up      /dev/aacd0s1g  A: 30449/64241 MB (47%)
 
gvinum rm -r f


 
All time extimates below assume disks aren’t scrubbing
so i setup 2 machines with 6.1. A had 2 logical drives and gv's created across both. B had 1 logical drive and gv's across the 1 drive. 
the labeling for the gv's was v1-vN (till i ran out of space)
when i moved aac1 from A to B, the gv's on A's aac1, took precedence over the similarly labeled gv's on B's aac0. in other words. B's aac0 used to have v1-v30. A's aac1 had vn6-vn30. when A's aac1 was moved to B, v1-v5 were linked to B's aac0, and v6-v30 linked to A's aac1
i relabeled B's aac0 gv's to something different (not v1-vN) and tried again and they all showed up. 
i tried the experiment again this time moving A's aac0 to B. at that point nothing worked. i moved A's aac0 back to A and renamed the device (data1) to something else, then tried the test again. still no beans. 
it's hazy what i did after that cause the machine's weren't cooperating and i was trying to rename the device but basically i think i need to repeat the test and see if i can re-define the device, and probably also give it a unique name and i bet it would have worked. 
also interesting- A's aac1 contained data about A's aac0 which showed up when i put aac1 into B
skeeter: as for the gv stuff, it definitely sounds like using some sort of serial numbering scheme would be the way to go if you want to be able to move disks around....
that overlap is still an issue  (aac1's device was 'data2')  when i had A's aac0 in B nothing worked both disks gv devices were called data1
skeeter: I suppose you could serialize those names as well...</pre>
 
 
= 6.2 =
 
Last updated 2007-10-15
 
<pre>All time extimates below assume disks aren’t scrubbing. Setup instructions below are for LSI card:


1. make sure bios is setup for bios console redirect
1. make sure bios is setup for bios console redirect
Line 2,428: Line 2,180:
Flow control: none
Flow control: none
Console connection: direct
Console connection: direct
Continue cr after post: on
Continue cr after post: off


2450:
2450:
Make sure running bios A09
Make sure running bios A09
Console Redirection: VT100/VT220
Console Redirection: VT100/VT220
 
2. assuming mirrors (or at least disks) created (if not, refer to this), boot to disk 1 of 5.4
2. assuming mirrors (or at least disks) created (if not, refer to this), boot to disk 1 of 4.11
skip kernel config (enter)
skip kernel config (enter)
custom install
custom install
 
partition
partition ->
move cursor to aacd0, hit space (takes you to partition map screen)
move cursor to amrd0, hit space (takes you to partition map screen)
a for entire disk
a for entire disk
q to quit and save
q to quit and save
standard mbr (no boot manager)
standard mbr (no boot manager)
space to unselect amrd0
space to unselect aacd0
cursor over amrd1
cursor over aacd1
space
space
a for entire disk
a for entire disk
Line 2,450: Line 2,201:
none (leave untouched)
none (leave untouched)


cursor over amrd0
cursor over aacd0
space
space
(takes you into part. Screen again) q to exit
(takes you into part. Screen again) q to exit
Line 2,456: Line 2,207:
Make sure both are checked and tab to ok
Make sure both are checked and tab to ok


Label ->
label
Make sure amrd0 is highlighted
Make sure aacd0 is highlighted
a to start with defaults
a to start with defaults


Line 2,473: Line 2,224:
delete everything
delete everything
/ 128M
/ 128M
swap 2G (for 2950 make it 4G)
swap 4G
/var 256M
/var 256M
/tmp 256M
/tmp 256M
/usr 3.5G (3584M)
/usr 3G
/mnt/data1 remaining space (no need to newfs)
/mnt/data1 remaining space


Make sure to toggle S for soft updates on all (should look like UFS2+S Y under the Newfs column)
Make sure to toggle S for soft updates on all (should look like UFS+S Y under the Newfs column)
Set the noatime option


move cursor to amrd1
move cursor to aacd1
swap 4G
swap 4G
/mnt/data2 remaining space (no need to newfs)
/mnt/data2 remaining space
Set the noatime option


q to save and exit
q to save and exit


distributions ->
distributions
developer
developer
yes to install ports
yes to install ports
exit
exit
 
media
media ->
cd
cd
 
commit
commit ->
yes
yes
(2450: 14mins, supermicro: 12mins)
(2450: 16mins, supermicro: 11mins)
 
yes to "visit general config"
yes to "visit general config" ->
Set root pwd
Set root pwd
Add user ‘user’ member group is wheel, set password
Add user ‘user’ member group is wheel, set password
Set tz
Set tz
 
Networking->interfaces->Fxp0
Networking->interfaces->Fxp0 ->
No IPV6
No IPV6
dhcp=no
dhcp=yes
Set hostname & domain
Set hostname & domain
Enable sshd
 
Startup services:
Disble usbd


exit...
exit...
exit install ->
exit install
yes
yes
take the cd out and let the machine reboot
take the cd out and let the machine reboot


3. double check the date/time
3. put some temp settings in /etc/rc.conf:
usbd_enable="NO"
sendmail_enable="NO"


4. edit /etc/make.conf (only add the console speed line if this is a supermicro capable of outputting BIOS redirect at that speed – 2450’s can only do 9600)
4. reboot
echo "WITHOUT_X11=yes \
KERNCONF=jail7 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf


5. add console="comconsole" to /boot/loader.conf
5. double check the date/time
echo "console=""comconsole""" >> /boot/loader.conf


6. turn off all ttyv's except 0 and 1 in /etc/ttys
6. edit /etc/make.conf (only add the console speed line if this is a supermicro capable of outputting BIOS redirect at that speed – 2450’s can only do 9600)
also turn on ttyd0, change type to vt100:
cat >> /etc/make.conf
vi /etc/ttys
WITHOUT_X11=yes
ttyd0  "/usr/libexec/getty std.9600"  vt100  on secure
KERNCONF=jail18
BOOT_COMCONSOLE_SPEED=38400


kill -1 1
7. install cvsup
cd /usr/ports/net/cvsup-without-gui
make install clean (stay close for gettext options, 2450: 21mins, supermicro: 14mins)
rehash


on console server:
8. get latest sources for this release:
vi /etc/remote
cd /usr/src
(rename port to jail18 depending on where and which digi plugged into)
cat > sup
test serial console
*default host=cvsup4.freebsd.org
*default base=/usr
*default prefix=/usr
*default release=cvs tag=RELENG_4_11
*default delete use-rel-suffix
*default compress
src-all


7. populate hosts
cvsup sup (2450, 4.10: 13mins, supermicro, 4.11: 11mins)
echo "10.1.4.3 backup2" >> /etc/hosts


8. put key in authorized_keys on backup2
9. populate hosts
cd
cat >> /etc/hosts
ssh-keygen -t dsa -b 1024
10.1.4.3 backup2
(default location, leave password blank)


cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
10. put key in authorized_keys on backup2
ssh-keygen -t dsa -b 1024 (default location, leave password blank)
scp /root/.ssh/id_dsa.pub user@backup2:/tmp/jail18pub
on backup2:
cat /tmp/jail18pub >> /root/.ssh/authorized_keys


confirm that you can ssh to backup2 without getting a login prompt
confirm that you can ssh to backup2 without getting a login prompt


ssh backup2
11. configure new kernel. Get config from similar machine or there may be a master copy somewhere under /mnt/data4/build (name the kernel config the same as the jail, ex jail4):
cd /usr/src/sys/i386/conf
scp backup2:/mnt/data4/build/freebsd/kern_config-4.11 ./jail18


9. create & populate binaries/scripts dirs
edit the kernel config and change ident to be the name of the jail:
mkdir -p /usr/local/jail/bin
ident          jail4
mkdir -p /usr/local/jail/rc.d
mkdir -p /usr/local/jail/template/
mkdir /mnt/data1
mkdir /mnt/data2
scp backup2:"/mnt/data4/bin/freebsd6.x/*" /usr/local/jail/bin
cd /usr/local/jail/rc.d/
touch quad1
touch quad2
touch quad3
touch quad4
touch safe1
touch safe2
touch safe3
touch safe4
chmod +x *
cd /usr/local/jail/bin
ln -s /usr/local/jail/rc.d/quad1 quad1
ln -s /usr/local/jail/rc.d/quad2 quad2
ln -s /usr/local/jail/rc.d/quad3 quad3
ln -s /usr/local/jail/rc.d/quad4 quad4
ln -s /usr/local/jail/rc.d/safe1 safe1
ln -s /usr/local/jail/rc.d/safe2 safe2
ln -s /usr/local/jail/rc.d/safe3 safe3
ln -s /usr/local/jail/rc.d/safe4 safe4


rehash
IMPORTANT CUSTOMIZATION:
for machines with >4G RAM, add to the config:
options        PAE


10. edit root's path and login script:
for supermicro mobo’s with broadcom nics, add to the config:
vi /root/.cshrc
device          bge            # Broadcom BCM570x (``Tigon III'')
add to path:  
/usr/local/jail/bin
(if adaptec card installed, also add /compat/linux/usr/sbin)


Change alias entries (add G):
for machines where lots of postgres might be running, change SHMMAXPGS:
alias la        ls -aG
options        SHMMAXPGS=40960
alias lf        ls -FAG
alias ll        ls -lAG
alias ls        ls -AG
alias mbm      mb mount
alias mbu      mb umount


and alter the prompt, set the following:
edit /sys/conf/newvers.sh to add –jc2 to the end of the BRANCH string (RELEASE-p9-jc2)
set prompt = "`/bin/hostname -s` %/# "


at the bottom of the file add:
12. bring over patches from backup2
set sshtty=`who am i|awk '{print $2}'`
The various patches are in /mnt/data4/build/freebsd/patches on backup2. There are dirs for each version. Not all dirs are populated, but patches for later versions work on older ones unless there is a new patch in the older dir.
/usr/sbin/rtprio 3 -`psj | grep $sshtty | awk '{print $2}'`
cd /usr/src
scp backup2:"/mnt/data4/build/freebsd/patches/4.11/*" .
jail_proc_patch      100% |*************************************************|  2593    00:00
restore-patch        100% |*************************************************| 3295    00:00
411ps-jail-patch    100% |*************************************************|  2602    00:00
jail_rtprio_patch    100% |*************************************************|  301    00:00
udp-patch            100% |*************************************************|  594    00:00
 
Apply patches:
patch < 411ps-jail-patch
patch < jail_proc_patch
patch < restore-patch
patch < jail_rtprio_patch
patch < udp-patch


set shortty=`who am i | awk '{print $2}' | sed -E 's/.*(..)$/\1/'`
13. build, install kernel and world
foreach x (`psj | grep sh | grep $shortty | awk '{print $2}'`)
make buildworld buildkernel installkernel (2450: 48min, supermicro: 20mins)
/usr/sbin/rtprio 2 -$x
(Any compile errors can be looked up in /usr/include/sys/signal.h, other errors, do a rm -R /usr/obj/*)
end
make installworld (2450: 2min, supermicro: 1mins)
mergemaster -i
(answer no to most of it)


To load the new file:
14. reboot. Confirm new kernel is loaded (uname -a)
source /root/.cshrc


11. install cvsup
15. update ports:
cd /usr/ports/net/cvsup-without-gui
cd /usr/ports
make install clean; rehash; mail -s 'cvs installed' dave.boodman@vtext.com < /dev/null
cat > sup
*default host=cvsup4.FreeBSD.org
*default base=/usr
*default prefix=/usr
*default release=cvs tag=RELENG_4
*default delete use-rel-suffix
*default compress
ports-all tag=.


(stay close for gettext options, 2450: 27mins, supermicro: 17mins, 2950: 22mins)
cvsup sup (2450: 26mins, supermicro: 26mins)


12. get latest sources for this release:
16. add console="comconsole" to /boot/loader.conf
cd /usr/src
cat >> /boot/loader.conf
echo "*default host=cvsup4.freebsd.org\
console="comconsole"
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_2\
*default delete use-rel-suffix\
*default compress\
src-all" > sup


cvsup sup ; mail -s 'cvs sup done' dave.boodman@vtext.com < /dev/null
17. edit /etc/ttys and turn off all ttyv's except 0 and 1
also turn on ttyd0, change type to vt100:
ttyd0  "/usr/libexec/getty std.9600"  vt100  on secure


(2450, ~12mins, supermicro, 27mins, 2950: 7mins)
kill -1 1


on console:
vi /etc/remote
(rename port to jail4 depending on where and which digi plugged into)
test serial console


13. configure new kernel. Get config from similar machine or there may be a master copy somewhere under /mnt/data4/build/freebsd (name the kernel config the same as the jail, ex jail18):
18. install linux_base:
cd /usr/src/sys/i386/conf
cd /usr/ports/emulators/linux_base
scp backup2:/mnt/data4/build/freebsd/kern_config-6.2 ./jail7
make install clean (2450: 7min, supermicro: 2mins)


or for PAE
reibuild rpmdb cause we had probs installing aacapps
cd /compat/linux/bin
./rpm --initdb
./rpm --rebuilddb


scp backup2:/mnt/data4/build/freebsd/kern_config-6.2-PAE ./jail7
19. install aacapps-4.1-0.i386.rpm
scp backup2:/mnt/data4/build/freebsd/aacapps-4.1-0.i386.rpm /tmp/.
/compat/linux/bin/rpm -ivh /tmp/aacapps-4.1-0.i386.rpm
cd /dev
sh MAKEDEV aac0


edit the kernel config and change ident to be the name of the jail:
test out /compat/linux/usr/sbin/aaccli
vi jail7
ident          jail7


edit /sys/conf/newvers.sh to add –jc1 to the end of the BRANCH string (RELEASE-jc1)
20. edit root's path and login script:
vi /sys/conf/newvers.sh
vi /root/.cshrc
add to path: /usr/local/jail/bin /compat/linux/usr/sbin


14. bring over patches from backup2
Change alias entries (add G):
alias la        ls -aG
alias lf        ls -FAG
alias ll        ls -lAG
alias ls        ls -AG
alias mbm      mb mount
alias mbu      mb umount


The various patches are in /mnt/data4/build/freebsd/patches on backup2. There are dirs for each version. Not all dirs are populated, but patches for later versions work on older ones unless there is a new patch in the older dir.
and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "


cd /usr/src
at the bottom of the file add:
scp backup2:"/mnt/data4/build/freebsd/patches/6.x/*" .
set sshtty=`who am i|awk '{print $2}'`
/usr/sbin/rtprio 3 -`ps auxwJ | grep $sshtty | awk '{print $2}'`


Apply patches:
set shortty=`who am i | awk '{print $2}' | sed -E 's/.*(..)$/\1/'`
patch -l < jls-patch
foreach x (`ps cauxJ | grep sh | grep $shortty | awk '{print $2}'`)
/usr/sbin/rtprio 2 -$x
end
21. install rsync from ports
cd /usr/ports/net/rsync
make install clean


Apply these only to 2950 with PAE:
22. create & populate binaries/scripts dirs
patch -p0 < mfi-patch
mkdir -p /usr/local/jail/bin
patch -p0 < gvinum-staticcompile-patch
mkdir -p /usr/local/jail/rc.d
patch -p0 < gvinum-bin-patch
scp backup2:"/mnt/data4/bin/freebsd/*" /usr/local/jail/bin
cd /usr/local/jail/rc.d/
touch quad1
touch quad2
touch quad3
touch quad4
touch safe1
touch safe2
touch safe3
touch safe4
chmod +x *
cd /usr/local/jail/bin
ln -s /usr/local/jail/rc.d/quad1 quad1
ln -s /usr/local/jail/rc.d/quad2 quad2
ln -s /usr/local/jail/rc.d/quad3 quad3
ln -s /usr/local/jail/rc.d/quad4 quad4
ln -s /usr/local/jail/rc.d/safe1 safe1
ln -s /usr/local/jail/rc.d/safe2 safe2
ln -s /usr/local/jail/rc.d/safe3 safe3
ln -s /usr/local/jail/rc.d/safe4 safe4
rehash


15. build, install kernel and world
23. configure inetd to respond to mrtg load queries
cd /usr/src
cat >> /etc/inetd.conf
make buildworld buildkernel installkernel; mail -s 'kernel build done' dave.boodman@vtext.com < /dev/null
load    stream  tcp    nowait  user    /usr/local/jail/bin/load.pl  load.pl
(2450: 1:56min, supermicro: 59mins, 2950: 38mins)
make installworld
(2450: 3min, supermicro: 1min, 2950: :34)
mergemaster -i
delete /var/tmp/temproot
delete bsnmpd
delete temporary ./etc/hosts
delete temporary ./etc/motd
delete /var/tmp/temproot


16. populate devfs ruleset
cat >> /etc/services
scp backup2:/mnt/data4/build/freebsd/devfs.rules /etc
load            12384/tcp


17. populate /etc/rc.conf with IPs and NFS settings
kill -HUP `cat /var/run/inetd.pid`
vi /etc/rc.conf


kern_securelevel_enable="NO"
24. configure load mrtg, on mail
portmap_enable="NO"
vi /usr/local/etc/mrtg/mrtg1.cfg
sendmail_enable="NO"
(add new entry to file following existing format)
usbd_enable="YES"


xntpd_enable="YES"
25. configure bb on mail:
nfs_client_enable="YES"
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
nfs_reserved_port_only="YES"
10.1.4.104 jail4.johncompanies.com # ssh
sshd_enable="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.4.119"
devfs_system_ruleset="devfsrules_show_all"


ifconfig_xl0="inet 10.1.4.118 netmask 255.255.255.0"
su bb
ifconfig_fxp0="inet 69.55.228.101 netmask 255.255.255.0"
cd /usr/home/bb/bbsrc/bb1.9e-btf
defaultrouter="69.55.228.1"
./runbb.sh stop
ifconfig_fxp0_alias0="inet 69.55.2xx.xx netmask 255.255.255.0"
./runbb.sh start
exit


static_routes="t1 office"
26. install bb client
route_t1="-net 10.1.5 10.1.4.2"
adduser -group 1984 -shell /bin/csh -uid 1984 bb
route_office="-net 10.1.6 10.1.4.2"
cd /usr/home/bb
gvinum_enable="YES"
scp backup2:/mnt/data4/build/bb/bb-freebsd.tar .
fsck_y_enable="YES"
tar xvf bb-freebsd.tar
background_fsck="NO"


18. make sure sysctls are set and preserved after reboot
cat > /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
echo "kern.consmute=0\
10.1.4.105 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh
kern.ipc.shm_use_phys=1\
10.1.4.118 jail18.johncompanies.com # ssh
kern.ipc.shmall=65535\
kern.ipc.shmmax=134217728\
net.inet.tcp.syncookies=0\
kern.maxfiles=32768\
kern.fallback_elf_brand=3\
kern.maxprocperuid=4000\
security.jail.sysvipc_allowed=1\
security.jail.max_procs_per_jail: 1026\
security.jail.allow_raw_sockets=1\
security.jail.socket_unixiproute_only=1\
security.jail.chflags_allowed=0\
dev.amr.0.allow_volume_configure=1\
compat.linux.osrelease=2.6.12" >> /etc/sysctl.conf


19. mount procfs
vi /home/bb/bbc1.9e-btf/ext/openfiles  and change:
echo "proc                    /proc          procfs  rw              0      0" >> /etc/fstab
MACHINE="jail18,johncompanies,com"     # HAS TO BE IN A,B,C FORM


For Dell 2950:
cd /usr/home/bb/bbc1.9e-btf/etc
echo "linprocfs              /usr/compat/linux/proc linprocfs rw    0      0" >> /etc/fstab
./bbchkcfg.sh
echo "linsysfs                /usr/compat/linux/sys linsysfs rw      0      0" >> /etc/fstab
./bbchkhosts.sh (ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd bbc1.9e-btf/
./runbb.sh start
more BBOUT (look for errors)
exit


20. reboot. Confirm new kernel is loaded
cat > /usr/local/etc/rc.d/bb.sh
su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"


uname –a
chmod +x /usr/local/etc/rc.d/bb.sh


Check devfs rules:
27. remove reserve space, enable softupdates (probably already set, so not necessary)
devfs rule showsets
cd
devfs rule -s 3 show
umount /mnt/data1
umount /mnt/data2
tunefs -m 0 /mnt/data1
tunefs -m 0 /mnt/data2


21. update ports:
mount -a
cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_2\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup


cvsup sup; mail -s 'cvs sup ports done' dave.boodman@vtext.com < /dev/null
28. configure ntp
cat > /etc/ntp.conf
server 10.1.4.105


(2450: 18mins, supermicro: 19mins; 2950: 24mins)
/usr/sbin/ntpd -p /var/run/ntpd.pid
ntpq -p
(confirm it’s able to reach our time server)


22. Install raid mgmt tool
29. mrtg switch graphs
install linux_base:
cd /usr/ports/emulators/linux_base-fc4
make install clean
(2450: 7min, supermicro: 3mins, 2950: 14mins)


(for LSI)
31. fwd and reverse lookups on ns1c
cd /usr/ports/sysutils/linux-megamgr
vi johncompanies.com
make install clean
rr johncompanies.com
vi internal.johncompanies.com
rndc reload johncompanies.com IN private
(edit the PTR too)


cd /usr/ports/sysutils/megarc
32. create all /dev/vn and /dev/pty files in /dev
make install clean
cat > /tmp/runme.sh
#!/bin/sh
cd /dev
for i in 1 2 3 4 5 6 7 ; do sh MAKEDEV pty$i ; done
for i in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 ; do sh MAKEDEV vn$i ; done


(for Perc5/i)
NOTE: there are only 4 lines in the above output - however the lines may be wrapped on your terminal - make sure that there are only four lines in the script you make.
cd /usr/ports/sysutils/linux-megacli
make install clean


Test:
chmod +x /tmp/runme.sh
rehash; megacli ldinfo lall a0
/tmp/runme.sh


(for adaptec)
rm /tmp/runme.sh
This didn’t work: reibuild rpmdb cause we had probs installing aacapps
cd /compat/linux/bin
./rpm --initdb
./rpm --rebuilddb


Install of linux-base lead to broken rpm on 6.2 so:
ls /dev/vn*|wc -l (make sure the output is a high number – larger than 128)
install aacapps-4.1-0.i386.rpm
ls /dev/pty*|wc -l (make sure the output is exactly 256)
scp backup2:/mnt/data4/build/freebsd/aacapps-4.1-0.i386.rpm /tmp/.
/compat/linux/bin/rpm -ivh --excludepath=/dev /tmp/aacapps-4.1-0.i386.rpm
scp user@10.1.4.107:/compat/linux/usr/sbin/aaccli /compat/linux/usr/sbin/aaccli


test out;
33. populate /etc/rc.conf with IPs and NFS settings
/compat/linux/usr/sbin/aaccli
vi /etc/rc.conf


rpm didn’t even install on latest so just scp’d over aaccli and it worked
kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="NO"


23. install rsync from ports
xntpd_enable="YES"
cd /usr/ports/net/rsync
nfs_client_enable="YES"
make install clean
nfs_reserved_port_only="YES"
sshd_enable="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.4.104"


choose default options
ifconfig_xl0="inet 10.1.4.104 netmask 255.255.255.0"
ifconfig_fxp0="inet 69.55.228.101 netmask 255.255.255.0"
defaultrouter="69.55.228.1"
ifconfig_fxp0_alias0="inet 69.55.2xx.xx netmask 255.255.255.0"


24. install perl from ports
static_routes="t1 office"
Not necessary if linux_base is installed
route_t1="-net 10.1.5 10.1.4.2"
route_office="-net 10.1.6 10.1.4.2"


cd /usr/ports/lang/perl5.8/
34. make sure sysctls are set and preserved after reboot
make install clean; rehash
cat >> /etc/sysctl.conf
(supermicro: 5min)
kern.consmute=0
 
jail.sysvipc_allowed=1
25. configure inetd to respond to mrtg load queries
kern.ipc.shm_use_phys=1
echo "load    stream  tcp     nowait  user    /usr/local/jail/bin/load.pl  load.pl" >> /etc/inetd.conf
kern.ipc.shmall=65535
kern.ipc.shmmax=134217728
net.inet.tcp.syncookies=0
kern.maxfiles=32768
kern.fallback_elf_brand=3
kern.maxprocperuid=4000
jail.max_procs_per_jail: 1026


echo "load            12384/tcp" >> /etc/services
35. reboot


26. install bb client
36. create the jail template
adduser
vnconfig -T -S 1g -Z -s labels -c /dev/vn1 /mnt/data1/jail
Username: bb
disklabel -r -w vn1 auto
Full name: bb
newfs /dev/vn1c
Uid (Leave empty for default): 1984
cd /usr/src
Login group [bb]:
mkdir -p /mnt/data1/jail-DIR
Login group is bb. Invite bb into other groups? []:
mount /dev/vn1c /mnt/data1/jail-DIR
Login class [default]:
make world DESTDIR=/mnt/data1/jail-DIR (2450: 45mins, supermicro: 19mins)
Shell (sh csh tcsh nologin) [sh]:
cd etc
Home directory [/home/bb]:
make distribution DESTDIR=/mnt/data1/jail-DIR -DNO_MAKEDEV_RUN
Use password-based authentication? [yes]:
cd /mnt/data1/jail-DIR/dev
Use an empty password? (yes/no) [no]:
sh MAKEDEV jail
Use a random password? (yes/no) [no]: yes
cd /mnt/data1/jail-DIR
Lock out the account after creation? [no]:
ln -sf dev/null kernel
Username  : bb
Password  : <random>
Full Name  : bb
Uid        : 1984
Class      :
Groups    : bb
Home      : /home/bb
Shell      : /bin/sh
Locked    : no
OK? (yes/no): yes


cd /usr/home/bb
jail /mnt/data1/jail-DIR testhostname 192.168.11.100 /bin/sh
scp backup2:/mnt/data4/build/bb/bb-freebsd.tar .
csh
tar xvf bb-freebsd.tar
touch /etc/fstab
cat > /etc/rc.conf
portmap_enable="NO"
network_interfaces=""
hostname="newsystem"
kern_securelevel_enable="NO"
sendmail_enable="YES"
sshd_enable="YES"
 
cat >> /etc/resolv.conf
nameserver 69.55.225.225
nameserver 69.55.230.3


edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
vi /etc/crontab
echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
remove the adjkerntz lines
10.1.4.107 jail7.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
comment out periodic’s and put this line above them:
# DO NOT UNCOMMENT THESE


vi /home/bb/bbc1.9e-btf/ext/openfiles
rm -rf /etc/periodic/daily/400.status-disks
MACHINE="jail19,johncompanies,com"      # HAS TO BE IN A,B,C FORM


cd /usr/home/bb/bbc1.9e-btf/etc
mv /bin/df /bin/df_
./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src
make; make install
cd ..
./runbb.sh start
more BBOUT
(look for errors)
exit


echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
cat > /bin/df
chmod +x /usr/local/etc/rc.d/bb.sh
#!/bin/sh
/bin/df_ $* .


27. configure load mrtg, on mail
chmod +x /bin/df
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg
(add new entry to file following existing format)


28. configure bb on mail:
cat > /sbin/mount
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
echo `df | tail -1 | awk '{print $1 " on " $6 " (ufs, local)"}'`
10.1.4.119 jail19.johncompanies.com # ssh


su bb
check /tmp for crap
cd
bbsrc/bb/runbb.sh restart ; exit


29. remove reserve space, enable softupdates (probably already set, so not necessary)
vi /etc/periodic/security/100.chksetuid
NOT APPLICABLE IF USING GVINUM
replace: MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`
cd
with: MP='/'
umount /mnt/data1
umount /mnt/data2
tunefs -m 0 /mnt/data1
tunefs -m 0 /mnt/data2
mount -a


30. configure ntp
mkdir -p /usr/compat/linux/dev
echo "server 10.1.4.5" > /etc/ntp.conf


/usr/sbin/ntpd -p /var/run/ntpd.pid
adduser (Add account for user)
sleep 2; ntpq -p
(confirm it’s able to reach our time server)


31. mrtg switch graphs
put user in wheel group
vi /etc/group


32. fwd and reverse lookups on ns1c
vi /usr/home/user/.profile (and add to the file):
vr johncompanies.com
TERM=vt100;    export TERM
vi internal.johncompanies.com
rndc reload johncompanies.com IN private
(edit the PTR too)


cd /etc
vipw -d .
root:$1$krszPxhk$xkCepSnz3mIikT3vCtJCt0:0:0::0:0:Charlie &:/root:/bin/csh
user:$1$Mx9p5Npk$QdMU6c8YQqp2FW2M3irEh/:1001:1001::0:0:User &:/home/user:/bin/sh


33. enable noatime option
tzsetup
NOT APPLICABLE IF RUNNING GVINUM
data1 and data2 should look something like:
/dev/amrd0s1g          /mnt/data1      ufs    rw,noatime      2      2


newaliases


reboot
chflags schg /dev/*mem


34. create gvinum volumes
cd /dev
rm console
ln -s null console


Make a g partition:
exit
exit


bsdlabel -e /dev/amrd0s1
cd /usr/ports
make -DNOCLEANDEPENDS clean (2450: 47mins , supermicro: 17mins)
rm -fr /usr/ports/distfiles/*
cp -r /usr/ports /mnt/data1/jail-DIR/usr


given:
cd /mnt/data1/jail-DIR/dev
# /dev/aacd0s1:
rm kmem
8 partitions:
mknod kmem c 2 1 root:kmem
#        size  offset    fstype  [fsize bsize bps/cpg]
chmod 640 kmem
  a:  262144        0    4.2BSD    2048 16384 16392
rm mem
  b:  4194304  262144      swap
mknod mem c 2 0 root:kmem
  c: 143363997        0    unused        0    0         # "raw" part, don't edit
chmod 640 mem
  d:   524288  4456448    4.2BSD    2048 16384 32776
  e:  524288  4980736    4.2BSD    2048 16384 32776
  f:  6291456  5505024    4.2BSD    2048 16384 28552


new offset = 6291456 + 5505024 = 11796480
rm /mnt/data1/jail-DIR/root/.history
new size is size for 'c' partition minus the new start from above
143363997 - 11796480 = 131567517
So:
g: 131567517 11796480 unused 0 0


bsdlabel -e /dev/amrd1s1
sh
change d to g
for i in 1 2 3 4 5 6 7 ; do sh MAKEDEV pty$i ; done
exit


For a 73G drive (after OS), we can fit 31 2G volumes so:
cd /mnt/data1/jail-DIR/usr/compat/linux/dev
mknod null c 2 2
mknod random c 2 3


echo 'drive data1 device /dev/aacd0s1g' > /tmp/cgv
cd
mkdir /usr/local/jail/template/
dump -0a -f /usr/local/jail/template/411template /dev/vn1


or
umount /dev/vn1c
vnconfig -u /dev/vn1


echo '#\!/bin/sh\
rm /mnt/data1/jail
i="1"\
rm -fr /mnt/data1/jail-DIR
while [ $i -le 31 ]\
do\
  echo "volume v$i" >> /tmp/cgv;\
  echo 'plex org concat' >> /tmp/cgv;\
  echo 'sd length 2g drive data1' >> /tmp/cgv;\
i=`expr $i + 1`\
done' > /tmp/mkcgv
sh /tmp/mkcgv
gvinum create /tmp/cgv


37. setup backups
cat > /usr/local/jail/bin/backup.config
#!/bin/sh
backupdir=/mnt/data2/jail18_rsync


For a 146G drive (-4G for swap), we can fit 66 2G volumes so:
## ENTRY /etc
## ENTRY /usr/local/etc
## ENTRY /usr/local/jail
## ENTRY /root/logs


echo 'drive data2 device /dev/amrd1s1g' > /tmp/cgv
on backup2:
sh
setup backup dirs:
for f in 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97;\
mkdir -p /mnt/data2/jail18_rsync/0_today
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
mkdir -p /mnt/data2/jail18_rsync/1_yesterday
echo 'sd length 2g drive data2' >> /tmp/cgv; done; exit
mkdir -p /mnt/data2/jail18_rsync/2_two_day


gvinum create /tmp/cgv
add the system to /mnt/data1/bin/rsync_houseclean


For 3rd 73G drive (after 2G swap), we can fit 33 2G volumes so:
on mail:
Label should be:
vi /usr/local/www/mgmt/cgi/backupgraph.pl
# /dev/aacd2s1:
(add hostname)
8 partitions:
#        size  offset    fstype  [fsize bsize bps/cpg]
  c: 143299737        0    unused        0    0        # "raw" part, don't edit
  g: 143299721        16    unused        0    0


/dev/aacd1s1d
38. mkdir /root/logs
echo 'drive data3 device /dev/aacd2s1g' > /tmp/cgv
sh
for f in 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131;\
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 2g drive data3' >> /tmp/cgv; done


gvinum create /tmp/cgv
39. edit sshd_config for security
vi /etc/ssh/sshd_config
ListenAddress 69.55.228.101
ListenAddress 10.1.4.104


For a 2nd 73G drive (after 2G swap), we can fit 33 2G volumes so:
kill -1 `cat /var/run/sshd.pid`


echo 'drive data2 device /dev/aacd1s1g' > /tmp/cgv
40. add crontab entries
sh
crontab -e
for f in 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64;\
5 0 * * * /usr/local/jail/bin/backup
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
1 0 1 * * /usr/local/jail/bin/ipfwreset
echo 'sd length 2g drive data2' >> /tmp/cgv; done; exit
0 18 * * * /usr/local/jail/bin/ipfwbackup
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /usr/local/jail/bin/trafstats
55 10,23 * * * /usr/local/jail/bin/trafficwatch.pl


gvinum create /tmp/cgv


For a 2nd 73G drive (after 4G swap), we can fit 32 2G volumes so:
41. Reboot notify script
cat > /usr/local/etc/rc.d/notify.sh
echo "`/bin/hostname` rebooted" | /usr/bin/mail reboot@johncompanies.com


echo 'drive data2 device /dev/aacd1s1g' > /tmp/cgv
chmod +x /usr/local/etc/rc.d/notify.sh
sh
for f in 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63;\
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 2g drive data2' >> /tmp/cgv; done; exit


gvinum create /tmp/cgv
42. copy jailmake from prev system
scp user@10.1.4.118:/usr/local/jail/bin/jailmake /usr/local/jail/bin
rehash
NOTE: remove df altering code from jailmake since we put the correct df in the template, and make sure path to template file is right


For a 3nd 73G drive, we can fit 34 2G volumes so:
43. add to templates via mgmt system


echo 'drive data3 device /dev/mfid2s1g' > /tmp/cgv
44. add to server/cabinet map. On mail:
sh
vi /usr/local/www/mgmt/html/cabinetmap.html
for f in 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163;\
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 2g drive data3' >> /tmp/cgv; done; exit


gvinum create /tmp/cgv
45. add an outside blocking rule to the firewall, so this machine can only be reached from inside the firewall. Follow example already in firewall jail17 is:
00117 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.228.2
00117 deny ip from any to 69.55.228.2


2950:
jail4 would be 00104...
bsdlabel -e /dev/mfid0s1
ipfw add 00118 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.228.140
bsdlabel -e /dev/mfid1s1
ipfw add 00118 deny ip from any to 69.55.228.140


46. select customers for probe map
</pre>


= FreeBSD 6.1 =


For 1st drive (146G)
Last updated 2006-05-09
echo 'drive data1 device /dev/mfid0s1g' > /tmp/cgv
sh
for f in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63;\
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 2g drive data1' >> /tmp/cgv; done; exit


Usually there’s 2040m leftover so create one more smaller vol:
<pre>All time extimates below assume disks aren’t scrubbing
echo 'volume v64\
plex org concat\
sd length 2040m drive data1' >> /tmp/cgv


gvinum create /tmp/cgv
1. make sure bios is setup for bios console redirect
Supermicro:
Console redirection:
Com port addr: on-board COM A
Baud: 38400
Console type: vt100
Flow control: none
Console connection: direct
Continue cr after post: off


For 2nd drive (146G)
2450:
echo 'drive data2 device /dev/mfid1s1g' > /tmp/cgv
Make sure running bios A09
sh
Console Redirection: VT100/VT220
for f in 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130;\
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 2g drive data2' >> /tmp/cgv; done; exit
gvinum create /tmp/cgv


2. assuming mirrors (or at least disks) created (if not, refer to this), boot to disk 1 of 5.4
skip kernel config (enter)
custom install


35. create the jail template
partition ->
move cursor to amrd0, hit space (takes you to partition map screen)
a for entire disk
q to quit and save
standard mbr (no boot manager)
space to unselect aacd0
cursor over aacd1
space
a for entire disk
q to quit and save
none (leave untouched)


mkdir /mnt/jail
cursor over aacd0
newfs /dev/gvinum/v1
space
mount /dev/gvinum/v1 /mnt/jail
(takes you into part. Screen again) q to exit
none
Make sure both are checked and tab to ok


cd /usr/src
Label ->
make clean
Make sure aacd0 is highlighted
rm -fr /usr/obj/
a to start with defaults
make buildworld
(2450: 2:28mins, supermicro: 53mins, 2950: 36min)
make world DESTDIR=/mnt/jail
(2450: 2:28mins, supermicro: 55mins, 2950: )
cd etc
make distribution DESTDIR=/mnt/jail
mount_devfs devfs /mnt/jail/dev
devfs -m /mnt/jail/dev rule -s 3 applyset
cd /mnt/jail
ln -sf dev/null kernel


jail /mnt/jail testhostname 192.168.11.100 /bin/sh
john likes:
csh
/1g
touch /etc/fstab
/var 256m
echo 'network_interfaces=""\
kern_securelevel_enable="NO"\
sendmail_enable="YES"\
sshd_enable="YES"' > /etc/rc.conf


echo "nameserver 69.55.225.225\
Glenn likes:
nameserver 69.55.230.3" >> /etc/resolv.conf
/128
/var 256
/usr 3g
/tmp 256


vi /etc/crontab
We do:
remove the adjkerntz lines
delete everything
comment out periodic’s and put this line above them:
/ 128M
# DO NOT UNCOMMENT THESE
swap 2G
 
/var 256M
rm -rf /etc/periodic/daily/400.status-disks
/tmp 256M
/usr 3G
/mnt/data1 remaining space


check /tmp for crap
Make sure to toggle S for soft updates on all (should look like UFS2+S Y under the Newfs column)


vi /etc/periodic/security/100.chksetuid
move cursor to aacd1
replace: MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`
swap 2G
with: MP='/' (use single quotes)
/mnt/data2 remaining space


mkdir -p /usr/compat/linux/dev
q to save and exit


adduser (Add account for user, make sure in wheel group)
distributions ->
developer
yes to install ports
exit


user/root passwd: 8ico2987
media ->
cd


Set root password
commit ->
yes
(2450: 14mins, supermicro: 12mins)


vi /usr/home/user/.profile (and add to the file):
yes to "visit general config" ->
TERM=vt100;    export TERM
Set root pwd


tzsetup
Add user ‘user’ member group is wheel, set password


newaliases
Set tz


#cd /dev
Networking->interfaces->Fxp0 ->
#rm console
No IPV6
#ln -s null console
dhcp=no
Set hostname & domain
Enable sshd


vi /etc/syslog.conf (comment out console and move to /var/log/messages):
exit...
#*.err;kern.warning;auth.notice;mail.crit              /dev/console *.err;kern.warning;auth.notice;mail.crit                /var/log/messages
exit install ->
yes


take the cd out and let the machine reboot
when it comes back up, enter junk when it asks for key seed


exit
exit


cd /usr/ports
3. double check the date/time
make -DNOCLEANDEPENDS clean
(2450: 15mins , supermicro: 29mins, 2950: 24mins)
rm -fr /usr/ports/distfiles/*
cp -r /usr/ports /mnt/jail/usr (2450: 2:00 mins , supermicro: 15mins, 2950: 3mins)


rm /mnt/jail/root/.history
4. edit /etc/make.conf (only add the console speed line if this is a supermicro capable of outputting BIOS redirect at that speed – 2450’s can only do 9600)
echo "WITHOUT_X11=yes \
KERNCONF=jail19 \
BOOT_COMCONSOLE_SPEED=38400" >> /etc/make.conf


cd
5. add console="comconsole" to /boot/loader.conf
dump -0a -f /usr/local/jail/template/61template /dev/gvinum/v1
echo "console=""comconsole""" >> /boot/loader.conf


umount /mnt/jail/dev
6. turn off all ttyv's except 0 and 1 in /etc/ttys
umount /dev/gvinum/v1
also turn on ttyd0, change type to vt100:
vi /etc/ttys
ttyd0  "/usr/libexec/getty std.9600"  vt100  on secure


rm -fr /mnt/jail
kill -1 1


on console server:
vi /etc/remote
(rename port to jail18 depending on where and which digi plugged into)
test serial console


36. setup backups
7. install cvsup
echo '#\!/bin/sh\
cd /usr/ports/net/cvsup-without-gui
backupdir=/mnt/data4/jail7\
make install clean; rehash
\
(stay close for gettext options, 2450: 27mins, supermicro: 17mins)
## ENTRY /etc\
## ENTRY /usr/local/etc\
## ENTRY /usr/local/jai7\
## ENTRY /root/logs' > /usr/local/jail/bin/backup.config


on backup2:
8. get latest sources for this release:
setup backup dirs:
cd /usr/src
mkdir -p /mnt/data2/jail7/0
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_1\
*default delete use-rel-suffix\
*default compress\
src-all" > sup


add the system to
cvsup sup
vi /mnt/data4/bin/snapshot_rotate
(2450, ~12mins, supermicro, 27mins)


on mail:
9. populate hosts
vi /usr/local/www/mgmt/cgi/backupgraph.pl
echo "10.1.4.3 backup2" >> /etc/hosts
(add hostname)


Edit /usr/local/jail/bin/backup to use the right drives
10. put key in authorized_keys on backup2
cd
ssh-keygen -t dsa -b 1024
(default location, leave password blank)
scp /root/.ssh/id_dsa.pub user@backup2:/tmp/jailkey


37. mkdir /root/logs
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'


38. edit sshd_config for security
confirm that you can ssh to backup2 without getting a login prompt
vi /etc/ssh/sshd_config
ListenAddress 69.55.226.128
ListenAddress 10.1.4.114


kill -1 `cat /var/run/sshd.pid`
ssh backup2


39. add crontab entries
11. configure new kernel. Get config from similar machine or there may be a master copy somewhere under /mnt/data4/build/freebsd (name the kernel config the same as the jail, ex jail18):
crontab -e
cd /usr/src/sys/i386/conf
5 0 * * * /usr/local/jail/bin/backup
scp backup2:/mnt/data4/build/freebsd/kern_config-6.1 ./jail19
1 0 1 * * /usr/local/jail/bin/ipfwreset
0 18 * * * /usr/local/jail/bin/ipfwbackup
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /usr/local/jail/bin/trafstats
55 10,23 * * * /usr/local/jail/bin/trafficwatch.pl


On 2950 with Perc5/i also add:
edit the kernel config and change ident to be the name of the jail:
*/5 * * * * /usr/local/jail/bin/perc5iraidchk
vi jail14
ident          jail14


40. Reboot notify script
edit /sys/conf/newvers.sh to add –jc1 to the end of the BRANCH string (RELEASE-jc1)
ln -s /usr/local/jail/bin/notify.sh /usr/local/etc/rc.d/notify.sh  
vi /sys/conf/newvers.sh


41. add to management db (on mail and devweb) jc.ref_machines and jc.ref_templates
12. bring over patches from backup2
 
The various patches are in /mnt/data4/build/freebsd/patches on backup2. There are dirs for each version. Not all dirs are populated, but patches for later versions work on older ones unless there is a new patch in the older dir.
uname -r
cd /usr/src
6.2-RELEASE-jc1
scp backup2:"/mnt/data4/build/freebsd/patches/6.1/*" .
jail_proc_patch      100% |*************************************************|  2593    00:00
restore-patch        100% |*************************************************|  3295    00:00
54ps-jail-patch      100% |*************************************************|  2602    00:00
jail_rtprio_patch    100% |*************************************************|  301    00:00
udp-patch            100% |*************************************************|  594    00:00


insert into ref_machines values (null,'jail19','jail19.johncompanies.com',0,'l');
Apply patches:
select machine_id from ref_machines where host='jail19';
patch < 54ps-jail-patch
+------------+
patch < jail_proc_patch
| machine_id |
patch < restore-patch
+------------+
patch < jail_rtprio_patch
|        35 |
patch < udp-patch
+------------+
insert into ref_templates values ('',' 6.2-RELEASE-jc1',15,'FreeBSD 6.2',0);


42. add to server/cabinet map. On mail:
13. build, install kernel and world
vi /usr/local/www/mgmt/html/cabinetmap.html
cd /usr/src
make buildworld buildkernel installkernel
(2450: 1:56min, supermicro::59mins)
make installworld
(2450: 3min, supermicro: 1min)
mergemaster -i
delete /var/tmp/temproot
delete bsnmpd
delete temporary ./etc/hosts
delete temporary ./etc/motd
delete /var/tmp/temproot


43. add an outside blocking rule to the firewall, so this machine can only be reached from inside the firewall. Follow example already in firewall jail17 is:
14. reboot. Confirm new kernel is loaded
00117 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.228.200
00117 deny ip from any to 69.55.228.200


jail19 would be 00119...
uname –a
ipfw add 00107 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.230.108
ipfw add 00107 deny ip from any to 69.55.220.108


44. select customers for probe map
15. update ports:
cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_1\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup


45. install raid monitor (LSI 320 only):
cvsup sup
(2450: 18mins, supermicro: 19mins)
 
18. (only applies if adaptec card installed)
install linux_base:
cd /usr/ports/emulators/linux_base
make install clean
(2450: 7min, supermicro: 3mins)


scp backup2:/d4/build/megaraid/MegaMonitor1.02.tgz /tmp
reibuild rpmdb cause we had probs installing aacapps
pkg_add MegaMonitor1.02.tgz
cd /compat/linux/bin
./rpm --initdb
./rpm --rebuilddb


edit line in /usr/local/etc/rc.d/megamonitor.sh to look like:
install aacapps-4.1-0.i386.rpm
/usr/sbin/MegaCtrl -start -log /var/log/messages  -disChkCon -SMART9999 > /megamonitor.out
scp backup2:/mnt/data4/build/freebsd/aacapps-4.1-0.i386.rpm /tmp/.
/compat/linux/bin/rpm -ivh --excludepath=/dev /tmp/aacapps-4.1-0.i386.rpm


comment out:
test out;
localhost: /var/log/messages : : : **Monitor** :
/compat/linux/usr/sbin/aaccli
In /usr/home/bb/bbc1.9e-btf/etc/bb-msgstab


cd /usr/ports/sysutils/megarc
make install clean


20. edit root's path and login script:
vi /root/.cshrc
add to path:
/usr/local/jail/bin
(if adaptec card installed, also add /compat/linux/usr/sbin)


46. make gv start on boot
Change alias entries (add G):
alias la        ls -aG
alias lf        ls -FAG
alias ll        ls -lAG
alias ls        ls -AG
alias mbm      mb mount
alias mbu      mb umount


scp backup2:/mnt/data4/build/freebsd/gvinum /etc/rc.d/gvinum
and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "


If they start stale:
at the bottom of the file add:
set sshtty=`who am i|awk '{print $2}'`
/usr/sbin/rtprio 3 -`psj | grep $sshtty | awk '{print $2}'`


echo '#\!/bin/sh\
set shortty=`who am i | awk '{print $2}' | sed -E 's/.*(..)$/\1/'`
i="1"\
foreach x (`psj | grep sh | grep $shortty | awk '{print $2}'`)
while [ $i -le 64 ]\
/usr/sbin/rtprio 2 -$x
do\
end
        gvinum start v$i\
        i=`expr $i + 1`\
21. install rsync from ports
done' > /tmp/startgv
cd /usr/ports/net/rsync
make install clean
 
choose default options


sh /tmp/startgv
21. install perl from ports
PROB NOT NECESSARY – INSTALLED WITH LINUX I THINK
cd /usr/ports/lang/perl5.8/
make install clean; rehash
(supermicro: 5min)


47. patch jail against starting jails with rtprio
22. create & populate binaries/scripts dirs
mkdir -p /usr/local/jail/bin
mkdir -p /usr/local/jail/rc.d
mkdir /mnt/data1
mkdir /mnt/data2
scp backup2:"/mnt/data4/bin/freebsd6.x/*" /usr/local/jail/bin
cd /usr/local/jail/rc.d/
touch quad1
touch quad2
touch quad3
touch quad4
touch safe1
touch safe2
touch safe3
touch safe4
chmod +x *
cd /usr/local/jail/bin
ln -s /usr/local/jail/rc.d/quad1 quad1
ln -s /usr/local/jail/rc.d/quad2 quad2
ln -s /usr/local/jail/rc.d/quad3 quad3
ln -s /usr/local/jail/rc.d/quad4 quad4
ln -s /usr/local/jail/rc.d/safe1 safe1
ln -s /usr/local/jail/rc.d/safe2 safe2
ln -s /usr/local/jail/rc.d/safe3 safe3
ln -s /usr/local/jail/rc.d/safe4 safe4


mv /usr/sbin/jail /usr/sbin/jail_
rehash
echo '#\!/bin/sh\
/usr/sbin/rtprio -t /usr/sbin/jail_ $*' > /usr/sbin/jail
chmod +x /usr/sbin/jail


48. make sure mail works
23. configure inetd to respond to mrtg load queries
If there are map errors:
echo "load    stream  tcp    nowait  user    /usr/local/jail/bin/load.pl  load.pl" >> /etc/inetd.conf
cd /etc/mail; make maps</pre>


= 6.2 -> 6.3 =
echo "load            12384/tcp" >> /etc/services


Last updated 2008-08-07
26. install bb client
adduser
cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd.tar .
tar xvf bb-freebsd.tar


<pre>
edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
echo "10.1.4.105 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.4.103 jail3.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts


1. remove old src
vi /home/bb/bbc1.9e-btf/ext/openfiles
cd /usr
MACHINE="jail19,johncompanies,com"      # HAS TO BE IN A,B,C FORM
rm –fr src/*


2. get latest sources for this release:
cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh (y to questions)
./bbchkhosts.sh (ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src
make; make install
cd ..
./runbb.sh start
more BBOUT
(look for errors)
exit


cd /usr/src
echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
echo "*default host=cvsup4.freebsd.org\
chmod +x /usr/local/etc/rc.d/bb.sh
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_3\
*default delete use-rel-suffix\
*default compress\
src-all" > sup


cvsup sup ; pagedave
modify bb-msgtab to look for raid failures
(20min)


3. configure new kernel. Get config from similar machine or there may be a master copy somewhere under /mnt/data4/build/freebsd (name the kernel config the same as the jail, ex jail18):
24. configure load mrtg, on mail
cd /usr/src/sys/i386/conf
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg
scp backup2:/mnt/data4/build/freebsd/kern_config-6.2 ./jail7
(add new entry to file following existing format)


or for PAE
25. configure bb on mail:
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
10.1.4.119 jail19.johncompanies.com # ssh


scp backup2:/mnt/data4/build/freebsd/kern_config-6.2-PAE ./jail7
su bb
cd
bbsrc/bb/runbb.sh restart ; exit


edit the kernel config and change ident to be the name of the jail:
27. remove reserve space, enable softupdates (probably already set, so not necessary)
vi jail7
NOT APPLICABLE IF USING GVINUM
ident          jail7
cd
umount /mnt/data1
umount /mnt/data2
tunefs -m 0 /mnt/data1
tunefs -m 0 /mnt/data2
mount -a


edit /sys/conf/newvers.sh to add –jc1 to the end of the BRANCH string (RELEASE-jc1)
28. configure ntp
vi /sys/conf/newvers.sh
echo "server 10.1.4.105" > /etc/ntp.conf


3. bring over patches from backup2
/usr/sbin/ntpd -p /var/run/ntpd.pid
sleep 2; ntpq -p
(confirm it’s able to reach our time server)


The various patches are in /mnt/data4/build/freebsd/patches on backup2. There are dirs for each version. Not all dirs are populated, but patches for later versions work on older ones unless there is a new patch in the older dir.
29. mrtg switch graphs


cd /usr/src
31. fwd and reverse lookups on ns1c
scp backup2:"/mnt/data4/build/freebsd/patches/6.x/*" .
vr johncompanies.com
vi internal.johncompanies.com
rndc reload johncompanies.com IN private
(edit the PTR too)


Apply patches:
33. populate /etc/rc.conf with IPs and NFS settings
patch -l < jls-patch
vi /etc/rc.conf


Apply these only to 2950 with PAE:
kern_securelevel_enable="NO"
patch -p0 < gvinum-staticcompile-patch
portmap_enable="NO"
patch -p0 < gvinum-bin-patch
sendmail_enable="NO"
usbd_enable="YES"


4. build, install kernel
xntpd_enable="YES"
cd /usr/src
nfs_client_enable="YES"
make buildkernel;  pagedave
nfs_reserved_port_only="YES"
(2min)
sshd_enable="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.4.119"
devfs_system_ruleset="devfsrules_show_all"


cd /boot
ifconfig_xl0="inet 10.1.4.118 netmask 255.255.255.0"
mv kernel.old kernel.old.old
ifconfig_fxp0="inet 69.55.228.101 netmask 255.255.255.0"
(optional move old kernel out of the way)
defaultrouter="69.55.228.1"
ifconfig_fxp0_alias0="inet 69.55.2xx.xx netmask 255.255.255.0"


cd /usr/src
static_routes="t1 office"
make installkernel
route_t1="-net 10.1.5 10.1.4.2"
route_office="-net 10.1.6 10.1.4.2"
gvinum_enable="YES"
fsck_y_enable="YES"
background_fsck="NO"


cd /boot
34. make sure sysctls are set and preserved after reboot
mv kernel.old kernel.6.2
echo "kern.consmute=0\
kern.ipc.shm_use_phys=1\
kern.ipc.shmall=65535\
kern.ipc.shmmax=134217728\
net.inet.tcp.syncookies=0\
kern.maxfiles=32768\
kern.fallback_elf_brand=3\
kern.maxprocperuid=4000\
security.jail.sysvipc_allowed=1\
security.jail.max_procs_per_jail: 1026\
security.jail.allow_raw_sockets=1\
security.jail.socket_unixiproute_only=1\
security.jail.chflags_allowed=0" >> /etc/sysctl.conf


35. mount procfs
echo "proc                    /proc          procfs  rw              0      0" >> /etc/fstab


5. take ½ of mirror and test (boot up) in new hardware
36. enable noatime option
NOT APPLICABLE IF RUNNING GVINUM
data1 and data2 should look something like:
/dev/amrd0s1g          /mnt/data1      ufs    rw,noatime      2      2


6. improved loader configs:
36. populate devfs ruleset
scp backup2:/mnt/data4/build/freebsd/devfs.rules /etc


/boot/loader.conf:
35. reboot
console="comconsole,vidconsole"
Check rules:
boot_serial="YES"
devfs rule showsets
boot_multicons="YES"
devfs rule -s 3 show


7. build userland
36. create gvinum volumes
cd /usr/src
make buildworld; pagedave
(33mins)
make installworld


mergemaster –i
Make a g partition:
reboot
8. patch jail against starting jails with rtprio


mv /usr/sbin/jail /usr/sbin/jail_
bsdlabel -e /dev/aacd0s1
echo '#\!/bin/sh\
/usr/sbin/rtprio -t /usr/sbin/jail_ $*' > /usr/sbin/jail
chmod +x /usr/sbin/jail
</pre>


given:
# /dev/aacd0s1:
8 partitions:
#        size  offset    fstype  [fsize bsize bps/cpg]
  a:  262144        0    4.2BSD    2048 16384 16392
  b:  4194304  262144      swap
  c: 143363997        0    unused        0    0        # "raw" part, don't edit
  d:  524288  4456448    4.2BSD    2048 16384 32776
  e:  524288  4980736    4.2BSD    2048 16384 32776
  f:  6291456  5505024    4.2BSD    2048 16384 28552


= 7.0 =
new offset = 6291456 + 5505024 = 11796480
new size is size for 'c' partition minus the new start from above
143363997 - 11796480 = 131567517
So:
g: 131567517 11796480 unused 0 0


Last updated 2008-04-30
For a 73G drive (after OS), we can fit 31 2G volumes so:


<pre>
echo 'drive data1 device /dev/aacd0s1g' > /tmp/cgv
sh
for f in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31;\
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 2g drive data1' >> /tmp/cgv; done; exit


All time extimates below assume disks aren’t scrubbing. Setup instructions below are for LSI card:
gvinum create /tmp/cgv


1. make sure bios is setup for bios console redirect
For a 146G drive (-4G for swap), we can fit 66 2G volumes so:
Supermicro:
Console redirection:
Com port addr: on-board COM A
Baud: 38400
Console type: vt100
Flow control: none
Console connection: direct
Continue cr after post: on


2450:
echo 'drive data2 device /dev/aacd1s1g' > /tmp/cgv
Make sure running bios A09
sh
Console Redirection: VT100/VT220
for f in 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97;\
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 2g drive data2' >> /tmp/cgv; done; exit


2. assuming mirrors (or at least disks) created (if not, refer to this), boot to disk 1 of 5.4
gvinum create /tmp/cgv
skip kernel config (enter)
custom install


partition ->
For 3rd 73G drive (after 2G swap), we can fit 33 2G volumes so:
move cursor to amrd0, hit space (takes you to partition map screen)
Label should be:
a for entire disk
# /dev/aacd2s1:
q to quit and save
8 partitions:
standard mbr (no boot manager)
#        size  offset    fstype  [fsize bsize bps/cpg]
space to unselect amrd0
  c: 143299737        0    unused        0    0        # "raw" part, don't edit
cursor over amrd1
  g: 143299721        16    unused        0    0
space
a for entire disk
q to quit and save
none (leave untouched)


cursor over amrd0
echo 'drive data3 device /dev/aacd2s1g' > /tmp/cgv
space
sh
(takes you into part. Screen again) q to exit
for f in 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131;\
none
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
Make sure both are checked and tab to ok
echo 'sd length 2g drive data3' >> /tmp/cgv; done


Label ->
gvinum create /tmp/cgv
Make sure mfid0 is highlighted


/ 256M
For a 2nd 73G drive (after 2G swap), we can fit 33 2G volumes so:
swap 2G (for 2950 make it 4G)
/var 256M (4.6G)
/tmp 256M
/usr 4G
/mnt/data1 remaining space (no need to newfs)


Make sure to toggle S for soft updates on all (should look like UFS2+S Y under the Newfs column)
echo 'drive data2 device /dev/aacd1s1g' > /tmp/cgv
sh
for f in 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64;\
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 2g drive data2' >> /tmp/cgv; done; exit


move cursor to mfid1
gvinum create /tmp/cgv
swap 8G
/mnt/data2 remaining space (no need to newfs)


q to save and exit


distributions ->
36. create the jail template
developer
yes to install ports
exit


media ->
mkdir /mnt/jail
cd
newfs /dev/gvinum/v1
mount /dev/gvinum/v1 /mnt/jail


commit ->
cd /usr/src
yes
make clean
(2450: 14mins, supermicro: 12mins)
rm -fr /usr/obj/
make world DESTDIR=/mnt/jail
(2450: 2:28mins, supermicro: 55mins)
cd etc
make distribution DESTDIR=/mnt/jail
mount_devfs devfs /mnt/jail/dev
devfs -m /mnt/jail/dev rule -s 3 applyset
cd /mnt/jail
ln -sf dev/null kernel


yes to "visit general config" ->
jail /mnt/jail testhostname 192.168.11.100 /bin/sh
Set root pwd
csh
 
touch /etc/fstab
Add user ‘user’ member group is wheel, set password
echo 'network_interfaces=""\
hostname="newsystem"\
kern_securelevel_enable="NO"\
sendmail_enable="YES"\
sshd_enable="YES"' > /etc/rc.conf


Set tz
echo "nameserver 69.55.225.225\
nameserver 69.55.230.3" >> /etc/resolv.conf


Networking->interfaces->Fxp0 ->
vi /etc/crontab
No IPV6
remove the adjkerntz lines
dhcp=no
comment out periodic’s and put this line above them:
Set hostname & domain
# DO NOT UNCOMMENT THESE
Enable sshd


exit...
rm -rf /etc/periodic/daily/400.status-disks
exit install ->
yes


take the cd out and let the machine reboot
check /tmp for crap


3. double check the date/time
vi /etc/periodic/security/100.chksetuid
replace: MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`
with: MP='/' (use single quotes)


4. edit /etc/make.conf (only add the console speed line if this is a supermicro capable of outputting BIOS redirect at that speed – 2450’s can only do 9600)
mkdir -p /usr/compat/linux/dev
echo "WITHOUT_X11=yes \
KERNCONF=jail2 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf


adduser (Add account for user, make sure in wheel group)


user/root passwd: 8ico2987


5. add settings to /boot/loader.conf and /boot.config
Set root password


echo "-Dh" >> /boot.config
vi /usr/home/user/.profile (and add to the file):
TERM=vt100;    export TERM


echo 'console="comconsole,vidconsole" \
tzsetup
boot_multicons="YES" \
boot_serial="YES" \
mfi_linux_load="YES" \
comconsole_speed="115200"' >> /boot/loader.conf


(leave out the speed and mfi lines for 2450s)
newaliases


6. turn off all ttyv's except 0 and 1 in /etc/ttys
#cd /dev
also turn on ttyd0, change type to vt100:
#rm console
vi /etc/ttys
#ln -s null console
ttyd0  "/usr/libexec/getty std.9600"  vt100  on secure


kill -1 1
exit
exit


on console server:
cd /usr/ports
vi /etc/remote
make -DNOCLEANDEPENDS clean
(rename port to jail18 depending on where and which digi plugged into)
(2450: 15mins , supermicro: 29mins)
test serial console
rm -fr /usr/ports/distfiles/*
cp -r /usr/ports /mnt/jail/usr (2450: 2:00 mins , supermicro: 15mins)


7. populate hosts
rm /mnt/jail/root/.history
echo "10.1.4.3 backup2" >> /etc/hosts
echo "10.1.4.8 backup1" >> /etc/hosts


cd
mkdir /usr/local/jail/template/
dump -0a -f /usr/local/jail/template/61template /dev/gvinum/v1


8. put key in authorized_keys on backup2
umount /mnt/jail/dev
cd
umount /dev/gvinum/v1
ssh-keygen -t dsa -b 1024
(default location, leave password blank)


cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
rm -fr /mnt/jail
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'


confirm that you can ssh to backup2 and backup1 without getting a login prompt
37. setup backups
echo '#\!/bin/sh\
backupdir=/mnt/data3jail3_rsync\
\
## ENTRY /etc\
## ENTRY /usr/local/etc\
## ENTRY /usr/local/jail\
## ENTRY /root/logs' > /usr/local/jail/bin/backup.config


ssh backup2 hostname
on backup2:
setup backup dirs:
mkdir -p /mnt/data3/jail3/0


ssh backup1 hostname
add the system to
vi /mnt/data4/bin/snapshot_rotate


9. create & populate binaries/scripts dirs
on mail:
mkdir -p /usr/local/jail/bin
vi /usr/local/www/mgmt/cgi/backupgraph.pl
mkdir -p /usr/local/jail/rc.d
(add hostname)
mkdir -p /usr/local/jail/template/
mkdir /mnt/data1
mkdir /mnt/data2
scp backup2:"/mnt/data4/bin/freebsd7.x/*" /usr/local/jail/bin
cd /usr/local/jail/rc.d/
touch quad1
touch quad2
touch quad3
touch quad4
touch safe1
touch safe2
touch safe3
touch safe4
chmod +x *
cd /usr/local/jail/bin
ln -s /usr/local/jail/rc.d/quad1 quad1
ln -s /usr/local/jail/rc.d/quad2 quad2
ln -s /usr/local/jail/rc.d/quad3 quad3
ln -s /usr/local/jail/rc.d/quad4 quad4
ln -s /usr/local/jail/rc.d/safe1 safe1
ln -s /usr/local/jail/rc.d/safe2 safe2
ln -s /usr/local/jail/rc.d/safe3 safe3
ln -s /usr/local/jail/rc.d/safe4 safe4
ln -s /usr/local/jail/bin/jailmake_zfs jailmake
--OR--
ln -s /usr/local/jail/bin/jailmake_geom jailmake


ln -s /usr/local/jail/bin/js_zfs js
38. mkdir /root/logs
--OR--
ln -s /usr/local/jail/bin/js_geom js


rehash
39. edit sshd_config for security
vi /etc/ssh/sshd_config
ListenAddress 69.55.228.101
ListenAddress 10.1.4.118


10. edit root's path and login script:
kill -1 `cat /var/run/sshd.pid`
vi /root/.cshrc
add to path:
/usr/local/jail/bin
(if adaptec card installed, also add /compat/linux/usr/sbin)


Change alias entries (add G):
40. add crontab entries
alias la        ls -aG
crontab -e
alias lf        ls -FAG
5 0 * * * /usr/local/jail/bin/backup
alias ll        ls -lAG
1 0 1 * * /usr/local/jail/bin/ipfwreset
alias ls        ls -AG
0 18 * * * /usr/local/jail/bin/ipfwbackup
alias mbm      mb mount
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /usr/local/jail/bin/trafstats
alias mbu      mb umount
55 10,23 * * * /usr/local/jail/bin/trafficwatch.pl


and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "


at the bottom of the file add:
41. Reboot notify script
set sshtty=`who am i|awk '{print $2}'`
ln -s /usr/local/jail/bin/notify.sh /usr/local/etc/rc.d/notify.sh
/usr/sbin/rtprio 3 -`psj | grep $sshtty | awk '{print $2}'`


set shortty=`who am i | awk '{print $2}' | sed -E 's/.*(..)$/\1/'`
42. copy jailmake from prev system
foreach x (`psj | grep sh | grep $shortty | awk '{print $2}'`)
scp user@10.1.4.119:/usr/local/jail/bin/jailmake /usr/local/jail/bin
/usr/sbin/rtprio 2 -$x
rehash
end
NOTE: remove df altering code from jailmake since we put the correct df in the template, and make sure path to template file is right


To load the new file:
43. add to management db (on mail and devweb) jc.ref_machines and jc.ref_templates
source /root/.cshrc


11. install cvsup
uname -r
cd /usr/ports/net/cvsup-without-gui
5.4-RELEASE-p2-jc2
make install clean; rehash; mail -s 'cvs installed' dave.boodman@vtext.com < /dev/null


(stay close for gettext options, 2450: 27mins, supermicro: 17mins, 2950: 22mins)
insert into ref_machines values (null,'jail19','jail19.johncompanies.com',0,'l');
select machine_id from ref_machines where host='jail19';
+------------+
| machine_id |
+------------+
|        35 |
+------------+
insert into ref_templates values ('',' 6.2-RELEASE-jc1',5,'FreeBSD 6.2',0);


12. get latest sources for this release:
44. add to server/cabinet map. On mail:
cd /usr/src
vi /usr/local/www/mgmt/html/cabinetmap.html
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_7\
*default delete use-rel-suffix\
*default compress\
src-all" > sup


45. add an outside blocking rule to the firewall, so this machine can only be reached from inside the firewall. Follow example already in firewall jail17 is:
00119 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.228.200
00119 deny ip from any to 69.55.228.200


-OR-
jail19 would be 00119...
ipfw add 00119 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.228.200
ipfw add 00119 deny ip from any to 69.55.228.200


echo "*default host=cvsup4.freebsd.org\
46. select customers for probe map
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_7_1\
*default delete use-rel-suffix\
*default compress\
src-all" > sup


cvsup sup ; mail -s 'cvs sup done' dave.boodman@vtext.com < /dev/null
47. install raid monitor


(2450, ~12mins, supermicro, 27mins, 2950: 7mins)
cd /usr/ports/sysutils/asr-utils
make install clean


48. make gv start on boot


13. configure new kernel.
for i386:
cd /usr/src/sys/i386/conf
scp backup2:/mnt/data4/build/freebsd/kern_config-7.0-PAE ./mx2


-or-
scp backup2:/mnt/data4/build/freebsd/gvinum /etc/rc.d/gvinum
for amd64:
cd /usr/src/sys/amd64/conf
scp backup2:/mnt/data4/build/freebsd/kern_config-7.0-amd64 ./jail2


------


edit the kernel config and change ident to be the name of the jail:
vi jail2
ident          jail2


edit /sys/conf/newvers.sh to add –jc2 to the end of the BRANCH string (RELEASE-jc2)
vi /sys/conf/newvers.sh


notes: http://www.FreeBSD.org/doc/en_US.ISO8859-1/books/handbook/kernelconfig-config.html
gconcat label -v somelabel /dev/gvinum/a /dev/gvinum/b
bsdlabel -r -w /dev/concat/somelabel
newfs /dev/concat/somelabela
mount /dev/concat/somelabel /mount/point


14. bring over patches from backup2
umount /dev/concat/somelabel
gconcat stop somelabel
gconcat label -v /dev/gvinum/a /dev/gvinum/b /dev/gvinum/c /dev/gvinum/d
growfs /dev/concat/somelabel
mount /dev/concat/somelabel /mount/point


The various patches are in /mnt/data4/build/freebsd/patches on backup2. There are dirs for each version. Not all dirs are populated, but patches for later versions work on older ones unless there is a new patch in the older dir.
volume f
        plex org concat
        sd length 30449m drive data1


cd /usr/src
where f is
scp backup2:"/mnt/data4/build/freebsd/patches/7.x/*" .


Apply patches:
D data1                State: up      /dev/aacd0s1g  A: 30449/64241 MB (47%)
patch -l < jls-patch


15. build, install kernel and world
gvinum rm -r f
cd /usr/src
make buildworld buildkernel installkernel; mail -s 'kernel build done' dave.boodman@vtext.com < /dev/null
(2450: 1:56min, supermicro: 59mins, 2950: 38mins)
make installworld
(2450: 3min, supermicro: 1min, 2950: :34)
mergemaster -i
delete /var/tmp/temproot
delete bsnmpd
delete temporary ./etc/hosts
delete temporary ./etc/motd
delete /var/tmp/temproot


cd /usr/src/sys/modules/zfs
make
make install


16. populate devfs ruleset
so i setup 2 machines with 6.1. A had 2 logical drives and gv's created across both. B had 1 logical drive and gv's across the 1 drive. 
scp backup2:/mnt/data4/build/freebsd/devfs.rules /etc
the labeling for the gv's was v1-vN (till i ran out of space)
when i moved aac1 from A to B, the gv's on A's aac1, took precedence over the similarly labeled gv's on B's aac0. in other words. B's aac0 used to have v1-v30. A's aac1 had vn6-vn30. when A's aac1 was moved to B, v1-v5 were linked to B's aac0, and v6-v30 linked to A's aac1
i relabeled B's aac0 gv's to something different (not v1-vN) and tried again and they all showed up. 
i tried the experiment again this time moving A's aac0 to B. at that point nothing worked. i moved A's aac0 back to A and renamed the device (data1) to something else, then tried the test again. still no beans. 
it's hazy what i did after that cause the machine's weren't cooperating and i was trying to rename the device but basically i think i need to repeat the test and see if i can re-define the device, and probably also give it a unique name and i bet it would have worked. 
also interesting- A's aac1 contained data about A's aac0 which showed up when i put aac1 into B
skeeter: as for the gv stuff, it definitely sounds like using some sort of serial numbering scheme would be the way to go if you want to be able to move disks around....
that overlap is still an issue  (aac1's device was 'data2')  when i had A's aac0 in B nothing worked both disks gv devices were called data1
skeeter: I suppose you could serialize those names as well...</pre>


17. populate /etc/rc.conf with IPs and NFS settings
= FreeBSD 6.2 =
vi /etc/rc.conf


kern_securelevel_enable="NO"
Last updated 2007-10-15
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"


xntpd_enable="YES"
<pre>All time extimates below assume disks aren’t scrubbing. Setup instructions below are for LSI card:
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
sshd_enable="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.4.102"
devfs_system_ruleset="devfsrules_show_all"


hostname="jail2.johncompanies.com"
1. make sure bios is setup for bios console redirect
ifconfig_xl0="inet 10.1.4.102 netmask 255.255.255.0"
Supermicro:
ifconfig_fxp0="inet 69.55.228.53 netmask 255.255.255.0"
Console redirection:
defaultrouter="69.55.228.1"
Com port addr: on-board COM A
#ifconfig_fxp0_alias0="inet 69.55.2xx.xx netmask 255.255.255.0"
Baud: 38400
Console type: vt100
Flow control: none
Console connection: direct
Continue cr after post: on


static_routes="t1 office"
2450:
route_t1="-net 10.1.5 10.1.4.2"
Make sure running bios A09
route_office="-net 10.1.6 10.1.4.2"
Console Redirection: VT100/VT220
zfs_enable="YES"


18. make sure sysctls are set and preserved after reboot
2. assuming mirrors (or at least disks) created (if not, refer to this), boot to disk 1 of 5.4
echo "kern.consmute=0\
skip kernel config (enter)
kern.ipc.shm_use_phys=1\
custom install
kern.ipc.shmall=65535\
kern.ipc.shmmax=134217728\
net.inet.tcp.syncookies=0\
kern.maxfiles=32768\
kern.fallback_elf_brand=3\
kern.maxprocperuid=4000\
security.jail.sysvipc_allowed=1\
security.jail.allow_raw_sockets=1\
security.jail.socket_unixiproute_only=1\
security.jail.chflags_allowed=0\
compat.linux.osrelease=2.6.12\
kern.maxvnodes=400000" >> /etc/sysctl.conf


19. mount procfs
partition ->
echo "proc                    /proc          procfs  rw              0      0" >> /etc/fstab
move cursor to amrd0, hit space (takes you to partition map screen)
a for entire disk
q to quit and save
standard mbr (no boot manager)
space to unselect amrd0
cursor over amrd1
space
a for entire disk
q to quit and save
none (leave untouched)


For Dell 2950/2450:
cursor over amrd0
echo "linprocfs              /usr/compat/linux/proc linprocfs rw    0      0" >> /etc/fstab
space
(takes you into part. Screen again) q to exit
none
Make sure both are checked and tab to ok


For Dell 2950:
Label ->
echo "linsysfs                /usr/compat/linux/sys linsysfs rw      0      0" >> /etc/fstab
Make sure amrd0 is highlighted
a to start with defaults


20. reboot. Confirm new kernel is loaded
john likes:
/1g
/var 256m


uname –a
Glenn likes:
/128
/var 256
/usr 3g
/tmp 256


Check devfs rules:
We do:
devfs rule showsets
delete everything
devfs rule -s 3 show
/ 128M
swap 2G (for 2950 make it 4G)
/var 256M
/tmp 256M
/usr 3.5G (3584M)
/mnt/data1 remaining space (no need to newfs)


21. update ports:
Make sure to toggle S for soft updates on all (should look like UFS2+S Y under the Newfs column)
cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_7_0\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup


cvsup sup; mail -s 'cvs sup ports done' dave.boodman@vtext.com < /dev/null
move cursor to amrd1
swap 4G
/mnt/data2 remaining space (no need to newfs)


(2450: 18mins, supermicro: 19mins; 2950: 24mins)
q to save and exit


22. Install raid mgmt tool
distributions ->
developer
yes to install ports
exit


For LSI based cards:
media ->
cd


install linux_base:
commit ->
cd /usr/ports/emulators/linux_base-fc4
yes
make install clean
(2450: 14mins, supermicro: 12mins)
(2450: 7min, supermicro: 3mins, 2950: 14mins)


(for LSI)
yes to "visit general config" ->
cd /usr/ports/sysutils/linux-megamgr
Set root pwd
make install clean


cd /usr/ports/sysutils/megarc
Add user ‘user’ member group is wheel, set password
make install clean


(for Perc5/i)
Set tz
cd /usr/ports/sysutils/linux-megacli
make install clean


Test:
Networking->interfaces->Fxp0 ->
rehash; megacli ldinfo lall a0
No IPV6
dhcp=no
Set hostname & domain
Enable sshd


For adaptec:
exit...
On jail18:
exit install ->
scp /compat/linux/usr/sbin/aaccli user@10.1.4.102:~
yes


mv ~user/aaccli /compat/linux/usr/sbin/aaccli
take the cd out and let the machine reboot


test out;
3. double check the date/time
/compat/linux/usr/sbin/aaccli


For 3w9690:
4. edit /etc/make.conf (only add the console speed line if this is a supermicro capable of outputting BIOS redirect at that speed – 2450’s can only do 9600)
scp backup2:/d4/build/3ware/tw_cli-freebsd-x86_64-9.5.0.1.tgz /usr/local/sbin
echo "WITHOUT_X11=yes \
cd /usr/local/sbin xzvf tw_cli-freebsd-x86_64-9.5.0.1.tgz
KERNCONF=jail7 \
rm tw_cli-freebsd-x86_64-9.5.0.1.tgz
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf


test out;
5. add console="comconsole" to /boot/loader.conf
./tw_cli /c0 show allstatus
echo "console=""comconsole""" >> /boot/loader.conf


23. install rsync from ports
6. turn off all ttyv's except 0 and 1 in /etc/ttys
cd /usr/ports/net/rsync
also turn on ttyd0, change type to vt100:
make install clean
vi /etc/ttys
ttyd0  "/usr/libexec/getty std.9600"  vt100  on secure
 
kill -1 1
 
on console server:
vi /etc/remote
(rename port to jail18 depending on where and which digi plugged into)
test serial console


choose default options
7. populate hosts
echo "10.1.4.3 backup2" >> /etc/hosts


24. install perl from ports
8. put key in authorized_keys on backup2
Not necessary if linux_base is installed
cd
ssh-keygen -t dsa -b 1024
(default location, leave password blank)


cd /usr/ports/lang/perl5.8/
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
make install clean; rehash
(supermicro: 5min)


25. configure inetd to respond to mrtg load queries
confirm that you can ssh to backup2 without getting a login prompt
echo "load    stream  tcp    nowait  user    /usr/local/jail/bin/load.pl  load.pl" >> /etc/inetd.conf


echo "load            12384/tcp" >> /etc/services
ssh backup2


26. install bb client
9. create & populate binaries/scripts dirs
adduser
mkdir -p /usr/local/jail/bin
Username: bb
mkdir -p /usr/local/jail/rc.d
Full name: bb
mkdir -p /usr/local/jail/template/
Uid (Leave empty for default): 1984
mkdir /mnt/data1
Login group [bb]:
mkdir /mnt/data2
Login group is bb. Invite bb into other groups? []:
scp backup2:"/mnt/data4/bin/freebsd6.x/*" /usr/local/jail/bin
Login class [default]:
cd /usr/local/jail/rc.d/
Shell (sh csh tcsh nologin) [sh]:
touch quad1
Home directory [/home/bb]:
touch quad2
Use password-based authentication? [yes]:
touch quad3
Use an empty password? (yes/no) [no]:
touch quad4
Use a random password? (yes/no) [no]: yes
touch safe1
Lock out the account after creation? [no]:
touch safe2
Username  : bb
touch safe3
Password  : <random>
touch safe4
Full Name  : bb
chmod +x *
Uid        : 1984
cd /usr/local/jail/bin
Class      :
ln -s /usr/local/jail/rc.d/quad1 quad1
Groups    : bb
ln -s /usr/local/jail/rc.d/quad2 quad2
Home      : /home/bb
ln -s /usr/local/jail/rc.d/quad3 quad3
Shell      : /bin/sh
ln -s /usr/local/jail/rc.d/quad4 quad4
Locked    : no
ln -s /usr/local/jail/rc.d/safe1 safe1
OK? (yes/no): yes
ln -s /usr/local/jail/rc.d/safe2 safe2
ln -s /usr/local/jail/rc.d/safe3 safe3
ln -s /usr/local/jail/rc.d/safe4 safe4


cd /usr/home/bb
rehash
scp backup2:/mnt/data4/build/bb/bb-freebsd.tar .
tar xf bb-freebsd.tar


edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
10. edit root's path and login script:
echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
vi /root/.cshrc
10.1.4.108 jail2.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
add to path:
/usr/local/jail/bin
(if adaptec card installed, also add /compat/linux/usr/sbin)


vi /home/bb/bbc1.9e-btf/ext/openfiles
Change alias entries (add G):
MACHINE="jail2,johncompanies,com"      # HAS TO BE IN A,B,C FORM
alias la        ls -aG
alias lf        ls -FAG
alias ll        ls -lAG
alias ls        ls -AG
alias mbm      mb mount
alias mbu      mb umount


cd /usr/home/bb/bbc1.9e-btf/etc
and alter the prompt, set the following:
./bbchkcfg.sh
set prompt = "`/bin/hostname -s` %/# "
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src
make; make install
cd ..
./runbb.sh start
more BBOUT
(look for errors)
exit


echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
at the bottom of the file add:
chmod +x /usr/local/etc/rc.d/bb.sh
set sshtty=`who am i|awk '{print $2}'`
/usr/sbin/rtprio 3 -`psj | grep $sshtty | awk '{print $2}'`


set shortty=`who am i | awk '{print $2}' | sed -E 's/.*(..)$/\1/'`
foreach x (`psj | grep sh | grep $shortty | awk '{print $2}'`)
/usr/sbin/rtprio 2 -$x
end


NOTE: to get bb working on amd, had to copy over bin dir from linux dist
To load the new file:
source /root/.cshrc


27. configure load mrtg, on mail
11. install cvsup
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg
cd /usr/ports/net/cvsup-without-gui
(add new entry to file following existing format)
make install clean; rehash; mail -s 'cvs installed' dave.boodman@vtext.com < /dev/null


28. configure bb on mail:
(stay close for gettext options, 2450: 27mins, supermicro: 17mins, 2950: 22mins)
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
10.1.4.108 jail2.johncompanies.com # ssh


su bb
12. get latest sources for this release:
cd
cd /usr/src
bbsrc/bb/runbb.sh restart ; exit
echo "*default host=cvsup4.freebsd.org\
 
*default base=/usr\
29. remove reserve space, enable softupdates (probably already set, so not necessary)
*default prefix=/usr\
NOT APPLICABLE IF USING GVINUM
*default release=cvs tag=RELENG_6_2\
cd
*default delete use-rel-suffix\
umount /mnt/data1
*default compress\
umount /mnt/data2
src-all" > sup
tunefs -m 0 /mnt/data1
 
tunefs -m 0 /mnt/data2
cvsup sup ; mail -s 'cvs sup done' dave.boodman@vtext.com < /dev/null
mount -a


30. configure ntp
(2450, ~12mins, supermicro, 27mins, 2950: 7mins)
echo "server 10.1.4.5" > /etc/ntp.conf


/usr/sbin/ntpd -p /var/run/ntpd.pid
sleep 2; ntpq -p
(confirm it’s able to reach our time server)


31. mrtg switch graphs
13. configure new kernel. Get config from similar machine or there may be a master copy somewhere under /mnt/data4/build/freebsd (name the kernel config the same as the jail, ex jail18):
cd /usr/src/sys/i386/conf
scp backup2:/mnt/data4/build/freebsd/kern_config-6.2 ./jail7


32. fwd and reverse lookups on ns1c
or for PAE
vr johncompanies.com
vi internal.johncompanies.com
rndc reload johncompanies.com IN private
(edit the PTR too)


scp backup2:/mnt/data4/build/freebsd/kern_config-6.2-PAE ./jail7


33. create zpools
edit the kernel config and change ident to be the name of the jail:
vi jail7
ident          jail7


Make a g partition:
edit /sys/conf/newvers.sh to add –jc1 to the end of the BRANCH string (RELEASE-jc1)
vi /sys/conf/newvers.sh


bsdlabel -e /dev/mfid0s1
14. bring over patches from backup2


given:
The various patches are in /mnt/data4/build/freebsd/patches on backup2. There are dirs for each version. Not all dirs are populated, but patches for later versions work on older ones unless there is a new patch in the older dir.
# /dev/aacd0s1:
8 partitions:
#        size  offset    fstype  [fsize bsize bps/cpg]
  a:  262144        0    4.2BSD    2048 16384 16392
  b:  4194304  262144      swap
  c: 143363997        0    unused        0    0        # "raw" part, don't edit
  d:  524288  4456448    4.2BSD    2048 16384 32776
  e:  524288  4980736    4.2BSD    2048 16384 32776
  f:  6291456  5505024    4.2BSD    2048 16384 28552


new offset = 6291456 + 5505024 = 11796480
cd /usr/src
new size is size for 'c' partition minus the new start from above
scp backup2:"/mnt/data4/build/freebsd/patches/6.x/*" .
143363997 - 11796480 = 131567517
So:
g: 131567517 11796480 unused 0 0


bsdlabel -e /dev/mfid1s1
Apply patches:
change d to g
patch -l < jls-patch


Apply these only to 2950 with PAE:
patch -p0 < mfi-patch
patch -p0 < gvinum-staticcompile-patch
patch -p0 < gvinum-bin-patch


zpool create pool1 mfid0s1g
15. build, install kernel and world
zpool create pool2 mfid1s1g
cd /usr/src
zfs set atime=off pool1
make buildworld buildkernel installkernel; mail -s 'kernel build done' dave.boodman@vtext.com < /dev/null
zfs set atime=off pool2
(2450: 1:56min, supermicro: 59mins, 2950: 38mins)
make installworld
(2450: 3min, supermicro: 1min, 2950: :34)
mergemaster -i
delete /var/tmp/temproot
delete bsnmpd
delete temporary ./etc/hosts
delete temporary ./etc/motd
delete /var/tmp/temproot


echo ' vfs.zfs.prefetch_disable="1" \
16. populate devfs ruleset
vfs.zfs.arc_min=16777216 \
scp backup2:/mnt/data4/build/freebsd/devfs.rules /etc
vfs.zfs.arc_max=33554432 \
vm.kmem_size_max="1G" # for i386\
vm.kmem_size="1G" # for i386\
vm.kmem_size_max="1.5G" # for amd64\
vm.kmem_size="1.5G" # for amd64\
vfs.zfs.zil_disable="1" ' >> /boot/loader.conf


17. populate /etc/rc.conf with IPs and NFS settings
vi /etc/rc.conf


35. create the jail template
kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"


xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
sshd_enable="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.4.119"
devfs_system_ruleset="devfsrules_show_all"


zfs create -o mountpoint=/mnt/data1/jail -o quota=4G pool1/jail
ifconfig_xl0="inet 10.1.4.118 netmask 255.255.255.0"
 
ifconfig_fxp0="inet 69.55.228.101 netmask 255.255.255.0"
cd /usr/ports/sysutils/jailutils
defaultrouter="69.55.228.1"
make install clean
ifconfig_fxp0_alias0="inet 69.55.2xx.xx netmask 255.255.255.0"


cd /usr/src
static_routes="t1 office"
make world DESTDIR=/mnt/data1/jail
route_t1="-net 10.1.5 10.1.4.2"
(2450: 2:28mins, supermicro: 55mins, 2950: 1h )
route_office="-net 10.1.6 10.1.4.2"
cd etc
gvinum_enable="YES"
make distribution DESTDIR=/mnt/data1/jail
fsck_y_enable="YES"
mount -t devfs devfs /mnt/data1/jail/dev
background_fsck="NO"
devfs -m /mnt/data1/jail/dev rule -s 3 applyset
cd /mnt/data1/jail
ln -sf dev/null kernel
cp /usr/local/sbin/jkill /mnt/data1/jail/sbin


jail /mnt/data1/jail testhostname 192.168.11.100 /bin/sh
18. make sure sysctls are set and preserved after reboot
csh
echo "kern.consmute=0\
touch /etc/fstab
kern.ipc.shm_use_phys=1\
echo 'network_interfaces=""\
kern.ipc.shmall=65535\
hostname="newsystem"\
kern.ipc.shmmax=134217728\
kern_securelevel_enable="NO"\
net.inet.tcp.syncookies=0\
sendmail_enable="YES"\
kern.maxfiles=32768\
sshd_enable="YES"' > /etc/rc.conf
kern.fallback_elf_brand=3\
kern.maxprocperuid=4000\
security.jail.sysvipc_allowed=1\
security.jail.max_procs_per_jail: 1026\
security.jail.allow_raw_sockets=1\
security.jail.socket_unixiproute_only=1\
security.jail.chflags_allowed=0\
dev.amr.0.allow_volume_configure=1\
compat.linux.osrelease=2.6.12" >> /etc/sysctl.conf


echo "nameserver 69.55.225.225\
19. mount procfs
nameserver 69.55.230.3" >> /etc/resolv.conf
echo "proc                    /proc          procfs  rw              0      0" >> /etc/fstab


vi /etc/crontab
For Dell 2950:
remove the adjkerntz lines
echo "linprocfs              /usr/compat/linux/proc linprocfs rw    0      0" >> /etc/fstab
comment out periodic’s and put this line above them:
echo "linsysfs                /usr/compat/linux/sys linsysfs rw      0      0" >> /etc/fstab
# IF YOU UNCOMMENT THESE, PLEASE ADJUST THEIR RUN TIME


rm -rf /etc/periodic/daily/400.status-disks
20. reboot. Confirm new kernel is loaded


check /tmp for crap
uname –a


vi /etc/periodic/security/100.chksetuid
Check devfs rules:
replace: MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`
devfs rule showsets
with: MP='/' (use single quotes)
devfs rule -s 3 show


mkdir -p /usr/compat/linux/dev
21. update ports:
cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_2\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup


adduser (Add account for user, make sure in wheel group)
cvsup sup; mail -s 'cvs sup ports done' dave.boodman@vtext.com < /dev/null


user/root passwd: 8ico2987
(2450: 18mins, supermicro: 19mins; 2950: 24mins)


Set root password
22. Install raid mgmt tool
install linux_base:
cd /usr/ports/emulators/linux_base-fc4
make install clean
(2450: 7min, supermicro: 3mins, 2950: 14mins)


vi /usr/home/user/.profile (and add to the file):
(for LSI)
TERM=vt100;    export TERM
cd /usr/ports/sysutils/linux-megamgr
make install clean


tzsetup
cd /usr/ports/sysutils/megarc
make install clean


newaliases
(for Perc5/i)
cd /usr/ports/sysutils/linux-megacli
make install clean


rm /sbin/halt /sbin/reboot
Test:
ln /sbin/jkill /sbin/halt
rehash; megacli ldinfo lall a0
ln /sbin/jkill /sbin/reboot


#cd /dev
(for adaptec)
#rm console
This didn’t work: reibuild rpmdb cause we had probs installing aacapps
#ln -s null console
cd /compat/linux/bin
./rpm --initdb
./rpm --rebuilddb


vi /etc/syslog.conf (comment out console and move to /var/log/messages):
Install of linux-base lead to broken rpm on 6.2 so:
#*.err;kern.warning;auth.notice;mail.crit              /dev/console *.err;kern.warning;auth.notice;mail.crit                /var/log/messages
install aacapps-4.1-0.i386.rpm
scp backup2:/mnt/data4/build/freebsd/aacapps-4.1-0.i386.rpm /tmp/.
/compat/linux/bin/rpm -ivh --excludepath=/dev /tmp/aacapps-4.1-0.i386.rpm
scp user@10.1.4.107:/compat/linux/usr/sbin/aaccli /compat/linux/usr/sbin/aaccli


cd /libexec
test out;
mv ld-elf32.so.1 ld-elf32.so.1-orig
/compat/linux/usr/sbin/aaccli
ln ld-elf.so.1 ld-elf32.so.1


exit
rpm didn’t even install on latest so just scp’d over aaccli and it worked
exit


cd /usr/ports
23. install rsync from ports
make -DNOCLEANDEPENDS clean
cd /usr/ports/net/rsync
(2450: 15mins , supermicro: 29mins, 2950: 24mins)
make install clean
rm -fr /usr/ports/distfiles/*
cp -r /usr/ports /mnt/data1/jail/usr (2450: 2:00 mins , supermicro: 15mins, 2950: 3mins)


rm /mnt/data1/jail/root/.history
choose default options


cd
24. install perl from ports
zfs snapshot pool1/jail@now
Not necessary if linux_base is installed
zfs send pool1/jail@now | cat > /usr/local/jail/template/70template
zfs destroy pool1/jail@now


zfs destroy pool1/jail
cd /usr/ports/lang/perl5.8/
make install clean; rehash
(supermicro: 5min)


rmdir /mnt/data1/jail
25. configure inetd to respond to mrtg load queries
f
echo "load    stream  tcp    nowait  user    /usr/local/jail/bin/load.pl  load.pl" >> /etc/inetd.conf


36. setup backups
echo "load            12384/tcp" >> /etc/services
echo '#\!/bin/sh\
backupdir=/data/jail2\
server=backup1\
\
## ENTRY /etc\
## ENTRY /usr/local/etc\
## ENTRY /usr/local/jail\
## ENTRY /root/logs' > /usr/local/jail/bin/backup.config


on backup1:
26. install bb client
setup backup dirs:
adduser
mkdir -p /data/jail2/0
Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username  : bb
Password  : <random>
Full Name  : bb
Uid        : 1984
Class      :
Groups    : bb
Home      : /home/bb
Shell      : /bin/sh
Locked    : no
OK? (yes/no): yes


add the system to
cd /usr/home/bb
vi /usr/local/sbin/snapshot_rotate
scp backup2:/mnt/data4/build/bb/bb-freebsd.tar .
tar xvf bb-freebsd.tar


on mail:
edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
vi /usr/local/www/mgmt/cgi/backupgraph.pl
echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
(add hostname)
10.1.4.107 jail7.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts


Edit /usr/local/jail/bin/backup to use the right drives
vi /home/bb/bbc1.9e-btf/ext/openfiles
MACHINE="jail19,johncompanies,com"      # HAS TO BE IN A,B,C FORM


37. mkdir /root/logs
cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src
make; make install
cd ..
./runbb.sh start
more BBOUT
(look for errors)
exit


38. edit sshd_config for security
echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
vi /etc/ssh/sshd_config
chmod +x /usr/local/etc/rc.d/bb.sh
ListenAddress 69.55.238.233
ListenAddress 10.1.4.108


kill -1 `cat /var/run/sshd.pid`
27. configure load mrtg, on mail
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg
(add new entry to file following existing format)


39. add crontab entries
28. configure bb on mail:
crontab -e
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
5 0 * * * /usr/local/jail/bin/backup.zfs
10.1.4.119 jail19.johncompanies.com # ssh
1 0 1 * * /usr/local/jail/bin/ipfwreset
0 18 * * * /usr/local/jail/bin/ipfwbackup
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /usr/local/jail/bin/trafstats


On 2950 with Perc5/i also add:
su bb
*/5 * * * * /usr/local/jail/bin/perc5iraidchk
cd
bbsrc/bb/runbb.sh restart ; exit


40. Reboot notify script
29. remove reserve space, enable softupdates (probably already set, so not necessary)
ln -s /usr/local/jail/bin/notify.sh /usr/local/etc/rc.d/notify.sh
NOT APPLICABLE IF USING GVINUM
 
cd
41. add to management db (on mail and devweb) jc.ref_machines and jc.ref_templates
umount /mnt/data1
umount /mnt/data2
tunefs -m 0 /mnt/data1
tunefs -m 0 /mnt/data2
mount -a


uname -r
30. configure ntp
6.2-RELEASE-jc1
echo "server 10.1.4.5" > /etc/ntp.conf


insert into ref_machines values (null,'mx2','mx2.johncompanies.com',0,'m');
/usr/sbin/ntpd -p /var/run/ntpd.pid
select machine_id from ref_machines where host='mx2';
sleep 2; ntpq -p
+------------+
(confirm it’s able to reach our time server)
| machine_id |
+------------+
|        35 |
+------------+
insert into ref_templates values ('',' 7.1-RELEASE-jc2',44,'FreeBSD 7.1',0);


42. add to server/cabinet map. On mail:
31. mrtg switch graphs
vi /usr/local/www/mgmt/html/cabinetmap.html


43. add an outside blocking rule to the firewall, so this machine can only be reached from inside the firewall. Follow example already in firewall jail17 is:
32. fwd and reverse lookups on ns1c
vr johncompanies.com
00117 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.228.200
vi internal.johncompanies.com
00117 deny ip from any to 69.55.228.200
rndc reload johncompanies.com IN private
(edit the PTR too)


jail19 would be 00119...
ipfw add 00107 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.230.108
ipfw add 00107 deny ip from any to 69.55.220.108


44. select customers for probe map
33. enable noatime option
NOT APPLICABLE IF RUNNING GVINUM
data1 and data2 should look something like:
/dev/amrd0s1g          /mnt/data1      ufs    rw,noatime      2      2


47. patch jail against starting jails with rtprio


mv /usr/sbin/jail /usr/sbin/jail_
reboot
echo '#\!/bin/sh\
/usr/sbin/rtprio -t /usr/sbin/jail_ $*' > /usr/sbin/jail
chmod +x /usr/sbin/jail


48. make sure mail works
34. create gvinum volumes
If there are map errors:
cd /etc/mail; make maps


49. move and pare down generic kernel
Make a g partition:
mv kernel.old/ kernel.GENERIC


/ too full, remove old kernel modules except for:
bsdlabel -e /dev/amrd0s1


Id Refs Address            Size     Name
given:
  1   14 0xffffffff80100000 ac6c08   kernel
# /dev/aacd0s1:
2   1 0xffffffff80bc8000 1128     mfi_linux.ko
8 partitions:
  3   4 0xffffffff80bca000 39138    linux.ko
#        size  offset    fstype  [fsize bsize bps/cpg]
  4   1 0xffffffffb48f8000 358c     linprocfs.ko
  a:  262144        0    4.2BSD     2048 16384 16392
  5    1 0xffffffffb48fc000 9d3      linsysfs.ko
  b: 4194304   262144      swap
  6   1 0xffffffffb490c000 80ee8    zfs.ko
   c: 143363997        0   unused        0     0        # "raw" part, don't edit
  d:  524288 4456448   4.2BSD    2048 16384 32776
  e:  524288 4980736    4.2BSD     2048 16384 32776
  f: 6291456 5505024   4.2BSD    2048 16384 28552


new offset = 6291456 + 5505024 = 11796480
new size is size for 'c' partition minus the new start from above
143363997 - 11796480 = 131567517
So:
g: 131567517 11796480 unused 0 0


50. ntp doesn’t seem to start from rc so,
bsdlabel -e /dev/amrd1s1
change d to g
 
For a 73G drive (after OS), we can fit 31 2G volumes so:


echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh
echo 'drive data1 device /dev/aacd0s1g' > /tmp/cgv
chmod 0700 /usr/local/etc/rc.d/ntp.sh
</pre>


= 7.1 =
or


Last updated 2009-02-16
echo '#\!/bin/sh\
i="1"\
while [ $i -le 31 ]\
do\
  echo "volume v$i" >> /tmp/cgv;\
  echo 'plex org concat' >> /tmp/cgv;\
  echo 'sd length 2g drive data1' >> /tmp/cgv;\
i=`expr $i + 1`\
done' > /tmp/mkcgv
sh /tmp/mkcgv
gvinum create /tmp/cgv


<pre>All time extimates below assume disks aren’t scrubbing. Setup instructions below are for LSI card:


1. make sure bios is setup for bios console redirect
For a 146G drive (-4G for swap), we can fit 66 2G volumes so:
Supermicro:
 
Console redirection:
echo 'drive data2 device /dev/amrd1s1g' > /tmp/cgv
Com port addr: on-board COM A
sh
Baud: 38400
for f in 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97;\
Console type: vt100
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
Flow control: none
echo 'sd length 2g drive data2' >> /tmp/cgv; done; exit
Console connection: direct
Continue cr after post: on


2450:
gvinum create /tmp/cgv
Make sure running bios A09
Console Redirection: VT100/VT220


2. assuming mirrors (or at least disks) created (if not, refer to this), boot to disk 1 of 5.4
For 3rd 73G drive (after 2G swap), we can fit 33 2G volumes so:
skip kernel config (enter)
Label should be:
custom install
# /dev/aacd2s1:
8 partitions:
#        size  offset    fstype  [fsize bsize bps/cpg]
  c: 143299737        0    unused        0    0        # "raw" part, don't edit
  g: 143299721        16    unused        0    0


partition ->
/dev/aacd1s1d
move cursor to amrd0, hit space (takes you to partition map screen)
echo 'drive data3 device /dev/aacd2s1g' > /tmp/cgv
a for entire disk
sh
q to quit and save
for f in 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131;\
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 2g drive data3' >> /tmp/cgv; done


NOTE
gvinum create /tmp/cgv
For gvin probs made 2 slices, 1st 8960M
Freebsd boot mgr


standard mbr (no boot manager)
For a 2nd 73G drive (after 2G swap), we can fit 33 2G volumes so:
space to unselect amrd0
cursor over amrd1
space
a for entire disk
q to quit and save
none (leave untouched)


cursor over amrd0
echo 'drive data2 device /dev/aacd1s1g' > /tmp/cgv
space
sh
(takes you into part. Screen again) q to exit
for f in 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64;\
none
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
Make sure both are checked and tab to ok
echo 'sd length 2g drive data2' >> /tmp/cgv; done; exit


Label ->
gvinum create /tmp/cgv
Make sure mfid0 is highlighted


/ 256M
For a 2nd 73G drive (after 4G swap), we can fit 32 2G volumes so:
swap 2G (for 2950 make it 4G)
/var 256M (4.6G)
/tmp 256M
/usr 4G
/mnt/data1 remaining space (no need to newfs)


Make sure to toggle S for soft updates on all (should look like UFS2+S Y under the Newfs column)
echo 'drive data2 device /dev/aacd1s1g' > /tmp/cgv
sh
for f in 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63;\
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 2g drive data2' >> /tmp/cgv; done; exit


move cursor to mfid1
gvinum create /tmp/cgv
swap 8G
/mnt/data2 remaining space (no need to newfs)


q to save and exit
For a 3nd 73G drive, we can fit 34 2G volumes so:


distributions ->
echo 'drive data3 device /dev/mfid2s1g' > /tmp/cgv
developer
sh
yes to install ports
for f in 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163;\
exit
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 2g drive data3' >> /tmp/cgv; done; exit


media ->
gvinum create /tmp/cgv
cd


commit ->
2950:
yes
bsdlabel -e /dev/mfid0s1
(2450: 14mins, supermicro: 12mins)
bsdlabel -e /dev/mfid1s1


yes to "visit general config" ->
Set root pwd


Add user ‘user’ member group is wheel, set password


Set tz
For 1st drive (146G)
echo 'drive data1 device /dev/mfid0s1g' > /tmp/cgv
sh
for f in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63;\
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 2g drive data1' >> /tmp/cgv; done; exit


Networking->interfaces->Fxp0 ->
Usually there’s 2040m leftover so create one more smaller vol:
No IPV6
echo 'volume v64\
dhcp=no
plex org concat\
Set hostname & domain
sd length 2040m drive data1' >> /tmp/cgv
Enable sshd


exit...
gvinum create /tmp/cgv
exit install ->
yes


take the cd out and let the machine reboot
For 2nd drive (146G)
echo 'drive data2 device /dev/mfid1s1g' > /tmp/cgv
sh
for f in 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130;\
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 2g drive data2' >> /tmp/cgv; done; exit
gvinum create /tmp/cgv


3. double check the date/time


4. edit /etc/make.conf (only add the console speed line if this is a supermicro capable of outputting BIOS redirect at that speed – 2450’s can only do 9600)
35. create the jail template
echo "WITHOUT_X11=yes \
KERNCONF=jail2 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf


mkdir /mnt/jail
newfs /dev/gvinum/v1
mount /dev/gvinum/v1 /mnt/jail


cd /usr/src
make clean
rm -fr /usr/obj/
make buildworld
(2450: 2:28mins, supermicro: 53mins, 2950: 36min)
make world DESTDIR=/mnt/jail
(2450: 2:28mins, supermicro: 55mins, 2950: )
cd etc
make distribution DESTDIR=/mnt/jail
mount_devfs devfs /mnt/jail/dev
devfs -m /mnt/jail/dev rule -s 3 applyset
cd /mnt/jail
ln -sf dev/null kernel


5. add settings to /boot/loader.conf and /boot.config
jail /mnt/jail testhostname 192.168.11.100 /bin/sh
csh
touch /etc/fstab
echo 'network_interfaces=""\
kern_securelevel_enable="NO"\
sendmail_enable="YES"\
sshd_enable="YES"' > /etc/rc.conf


echo "-Dh" >> /boot.config
echo "nameserver 69.55.225.225\
nameserver 69.55.230.3" >> /etc/resolv.conf


echo 'console="comconsole,vidconsole" \
vi /etc/crontab
boot_multicons="YES" \
remove the adjkerntz lines
boot_serial="YES" \
comment out periodic’s and put this line above them:
mfi_linux_load="YES" \
# DO NOT UNCOMMENT THESE
comconsole_speed="115200"' >> /boot/loader.conf


(leave out the speed and mfi lines for 2450s)
rm -rf /etc/periodic/daily/400.status-disks


6. turn off all ttyv's except 0 and 1 in /etc/ttys
check /tmp for crap
also turn on ttyd0, change type to vt100:
vi /etc/ttys


ttyv2  "/usr/libexec/getty Pc"        cons25  off secure
vi /etc/periodic/security/100.chksetuid
ttyv3  "/usr/libexec/getty Pc"        cons25  off secure
replace: MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`
ttyv4  "/usr/libexec/getty Pc"        cons25  off secure
with: MP='/' (use single quotes)
ttyv5  "/usr/libexec/getty Pc"        cons25  off secure
ttyv6  "/usr/libexec/getty Pc"         cons25  off secure
ttyv7  "/usr/libexec/getty Pc"        cons25  off secure


ttyd0  "/usr/libexec/getty std.9600"  vt100  on secure
mkdir -p /usr/compat/linux/dev


kill -1 1
adduser (Add account for user, make sure in wheel group)


on console server:
user/root passwd: 8ico2987
vi /etc/remote
(rename port to jail18 depending on where and which digi plugged into)
test serial console


7. populate hosts
Set root password
echo "10.1.4.3 backup2" >> /etc/hosts
echo "10.1.4.8 backup1" >> /etc/hosts


vi /usr/home/user/.profile (and add to the file):
TERM=vt100;    export TERM


8. put key in authorized_keys on backup2
tzsetup
cd
ssh-keygen -t dsa -b 1024
(default location, leave password blank)


cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
newaliases
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'


confirm that you can ssh to backup2 and backup1 without getting a login prompt
#cd /dev
#rm console
#ln -s null console


ssh backup2 hostname
vi /etc/syslog.conf (comment out console and move to /var/log/messages):
#*.err;kern.warning;auth.notice;mail.crit              /dev/console *.err;kern.warning;auth.notice;mail.crit                /var/log/messages


ssh backup1 hostname


9. create & populate binaries/scripts dirs
exit
mkdir -p /usr/local/jail/bin
exit
mkdir -p /usr/local/jail/rc.d
mkdir -p /usr/local/jail/template/
mkdir /mnt/data1
mkdir /mnt/data2
scp backup2:"/mnt/data4/bin/freebsd7.x/*" /usr/local/jail/bin
cd /usr/local/jail/rc.d/
touch quad1
touch quad2
touch quad3
touch quad4
touch safe1
touch safe2
touch safe3
touch safe4
chmod +x *
cd /usr/local/jail/bin
ln -s /usr/local/jail/rc.d/quad1 quad1
ln -s /usr/local/jail/rc.d/quad2 quad2
ln -s /usr/local/jail/rc.d/quad3 quad3
ln -s /usr/local/jail/rc.d/quad4 quad4
ln -s /usr/local/jail/rc.d/safe1 safe1
ln -s /usr/local/jail/rc.d/safe2 safe2
ln -s /usr/local/jail/rc.d/safe3 safe3
ln -s /usr/local/jail/rc.d/safe4 safe4
ln -s /usr/local/jail/bin/jailmake_geom jailmake
ln -s /usr/local/jail/bin/js_geom js
ln -s /usr/local/jail/bin/canceljail_geom canceljail
ln -s /usr/local/jail/bin/jailmakeempty_geom jailmakeempty


ln -s /usr/local/jail/bin/jailmake_md jailmake
cd /usr/ports
ln -s /usr/local/jail/bin/js_md js
make -DNOCLEANDEPENDS clean
ln -s /usr/local/jail/bin/canceljail_md canceljail
(2450: 15mins , supermicro: 29mins, 2950: 24mins)
ln -s /usr/local/jail/bin/jailmakeempty_md jailmakeempty
rm -fr /usr/ports/distfiles/*
ln -s /usr/local/jail/bin/postboot_md postboot
cp -r /usr/ports /mnt/jail/usr (2450: 2:00 mins , supermicro: 15mins, 2950: 3mins)


rehash
rm /mnt/jail/root/.history


10. edit root's path and login script:
cd
vi /root/.cshrc
dump -0a -f /usr/local/jail/template/61template /dev/gvinum/v1


Change alias entries (add G):
umount /mnt/jail/dev
alias la        ls -aG
umount /dev/gvinum/v1
alias lf        ls -FAG
alias ll        ls -lAG
alias ls        ls -AG
alias mbm      mb mount
alias mbu      mb umount


rm -fr /mnt/jail


add to path:
/usr/local/jail/bin
(if adaptec card installed, also add /compat/linux/usr/sbin)


and alter the prompt, set the following:
36. setup backups
set prompt = "`/bin/hostname -s` %/# "
echo '#\!/bin/sh\
backupdir=/mnt/data4/jail7\
\
## ENTRY /etc\
## ENTRY /usr/local/etc\
## ENTRY /usr/local/jai7\
## ENTRY /root/logs' > /usr/local/jail/bin/backup.config


at the bottom of the file add:
on backup2:
set sshtty=`who am i|awk '{print $2}'`
setup backup dirs:
/usr/sbin/rtprio 3 -`psj | grep $sshtty | awk '{print $2}'`
mkdir -p /mnt/data2/jail7/0


set shortty=`who am i | awk '{print $2}' | sed -E 's/.*(..)$/\1/'`
add the system to
foreach x (`psj | grep sh | grep $shortty | awk '{print $2}'`)
vi /mnt/data4/bin/snapshot_rotate
/usr/sbin/rtprio 2 -$x
end


To load the new file:
on mail:
source /root/.cshrc
vi /usr/local/www/mgmt/cgi/backupgraph.pl
(add hostname)


11. install cvsup
Edit /usr/local/jail/bin/backup to use the right drives
cd /usr/ports/net/cvsup-without-gui
make install clean; rehash; mail -s 'cvs installed' dave.boodman@vtext.com < /dev/null


(stay close for gettext options, 2450: 27mins, supermicro: 17mins, 2950: 22mins)
37. mkdir /root/logs


12. get latest sources for this release:
38. edit sshd_config for security
cd /usr/src
vi /etc/ssh/sshd_config
echo "*default host=cvsup4.freebsd.org\
ListenAddress 69.55.226.128
*default base=/usr\
ListenAddress 10.1.4.114
*default prefix=/usr\
*default release=cvs tag=RELENG_7_1\
*default delete use-rel-suffix\
*default compress\
src-all" > sup


-OR-
kill -1 `cat /var/run/sshd.pid`


echo "*default host=cvsup4.freebsd.org\
39. add crontab entries
*default base=/usr\
crontab -e
*default prefix=/usr\
5 0 * * * /usr/local/jail/bin/backup
*default release=cvs tag=RELENG_7\
1 0 1 * * /usr/local/jail/bin/ipfwreset
*default delete use-rel-suffix\
0 18 * * * /usr/local/jail/bin/ipfwbackup
*default compress\
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /usr/local/jail/bin/trafstats
src-all" > sup
55 10,23 * * * /usr/local/jail/bin/trafficwatch.pl


(stable)
On 2950 with Perc5/i also add:
*/5 * * * * /usr/local/jail/bin/perc5iraidchk


40. Reboot notify script
ln -s /usr/local/jail/bin/notify.sh /usr/local/etc/rc.d/notify.sh


cvsup sup ; mail -s 'cvs sup done' dave.boodman@vtext.com < /dev/null
41. add to management db (on mail and devweb) jc.ref_machines and jc.ref_templates


(2450, ~12mins, supermicro, 27mins, 2950: 7mins)
uname -r
6.2-RELEASE-jc1


insert into ref_machines values (null,'jail19','jail19.johncompanies.com',0,'l');
select machine_id from ref_machines where host='jail19';
+------------+
| machine_id |
+------------+
|        35 |
+------------+
insert into ref_templates values ('',' 6.2-RELEASE-jc1',15,'FreeBSD 6.2',0);


13. configure new kernel.  
42. add to server/cabinet map. On mail:
for i386:
vi /usr/local/www/mgmt/html/cabinetmap.html
cd /usr/src/sys/i386/conf
scp backup2:/mnt/data4/build/freebsd/kern_config-7.0-PAE ./mx2
 
-or-


for amd64:
43. add an outside blocking rule to the firewall, so this machine can only be reached from inside the firewall. Follow example already in firewall jail17 is:
cd /usr/src/sys/amd64/conf
scp backup2:/mnt/data4/build/freebsd/kern_config-7.1-amd64 ./jail2
00117 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.228.200
00117 deny ip from any to 69.55.228.200


-------
jail19 would be 00119...
ipfw add 00107 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.230.108
ipfw add 00107 deny ip from any to 69.55.220.108


edit the kernel config and change ident to be the name of the jail:
44. select customers for probe map
vi jail2
ident          jail2


edit /sys/conf/newvers.sh to add –jc2 to the end of the BRANCH string (RELEASE-jc2)
45. install raid monitor (LSI 320 only):
vi /sys/conf/newvers.sh


notes: http://www.FreeBSD.org/doc/en_US.ISO8859-1/books/handbook/kernelconfig-config.html
scp backup2:/d4/build/megaraid/MegaMonitor1.02.tgz /tmp
pkg_add MegaMonitor1.02.tgz


14. bring over patches from backup2
edit line in /usr/local/etc/rc.d/megamonitor.sh to look like:
/usr/sbin/MegaCtrl -start -log /var/log/messages  -disChkCon -SMART9999 > /megamonitor.out


The various patches are in /mnt/data4/build/freebsd/patches on backup2. There are dirs for each version. Not all dirs are populated, but patches for later versions work on older ones unless there is a new patch in the older dir.
comment out:
localhost: /var/log/messages : : : **Monitor** :
In /usr/home/bb/bbc1.9e-btf/etc/bb-msgstab


cd /usr/src
cd /usr/ports/sysutils/megarc
scp backup2:"/mnt/data4/build/freebsd/patches/7.x/*" .
make install clean


Apply patches:
patch -l < jls-patch


46. make gv start on boot


15. build, install kernel and world
scp backup2:/mnt/data4/build/freebsd/gvinum /etc/rc.d/gvinum


cd /boot
If they start stale:


mv kernel kernel.GENERIC
echo '#\!/bin/sh\
cd kernel.GENERIC
i="1"\
mkdir hold
while [ $i -le 64 ]\
mv mfi_linux.ko hold/
do\
mv linux.ko hold/
        gvinum start v$i\
mv linprocfs.ko hold/
        i=`expr $i + 1`\
mv linsysfs.ko hold/
done' > /tmp/startgv
mv geom_vinum.ko hold/
mv geom_concat.ko hold/
rm *.ko
rm *.symbols
mv hold/* .
rmdir hold/


cd /usr/src
sh /tmp/startgv
make buildkernel installkernel


make buildworld ; mail -s 'kernel build done' dave.boodman@vtext.com < /dev/null
47. patch jail against starting jails with rtprio
(2450: 1:56min, supermicro: 59mins, 2950: 38mins)
make installworld
(2450: 3min, supermicro: 1min, 2950: :34)
mergemaster -i
delete /var/tmp/temproot
delete bsnmpd
delete temporary ./etc/hosts
delete temporary ./etc/motd
delete /var/tmp/temproot


cd /sys/modules/geom/geom_vinum
mv /usr/sbin/jail /usr/sbin/jail_
make
echo '#\!/bin/sh\
make install
/usr/sbin/rtprio -t /usr/sbin/jail_ $*' > /usr/sbin/jail
chmod +x /usr/sbin/jail


16. populate devfs ruleset
48. make sure mail works
scp backup2:/mnt/data4/build/freebsd/devfs.rules /etc
If there are map errors:
cd /etc/mail; make maps</pre>


17. populate /etc/rc.conf with IPs and NFS settings
= FreeBSD 6.2 -> 6.3 =
vi /etc/rc.conf


kern_securelevel_enable="NO"
Last updated 2008-08-07
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"


xntpd_enable="YES"
<pre>
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.4.102"
devfs_system_ruleset="devfsrules_show_all"


#ifconfig_xl0="inet 10.1.4.102 netmask 255.255.255.0"
1. remove old src
#ifconfig_fxp0="inet 69.55.228.53 netmask 255.255.255.0"
cd /usr
#ifconfig_fxp0_alias0="inet 69.55.2xx.xx netmask 255.255.255.0"
rm –fr src/*


static_routes="t1 office"
2. get latest sources for this release:
route_t1="-net 10.1.5 10.1.4.2"
route_office="-net 10.1.6 10.1.4.2"
gvinum_enable="YES"
fsck_y_enable="YES"
background_fsck="NO"
rc_mfi_raid_tty_log="YES"


18. make sure sysctls are set and preserved after reboot
cd /usr/src
echo "kern.consmute=0\
echo "*default host=cvsup4.freebsd.org\
kern.ipc.shm_use_phys=1\
*default base=/usr\
kern.ipc.shmall=131070\
*default prefix=/usr\
kern.ipc.shmmax=134217728\
*default release=cvs tag=RELENG_6_3\
net.inet.tcp.syncookies=0\
*default delete use-rel-suffix\
kern.maxfiles=32768\
*default compress\
kern.fallback_elf_brand=3\
src-all" > sup
kern.maxprocperuid=4000\
security.jail.sysvipc_allowed=1\
security.jail.allow_raw_sockets=1\
security.jail.socket_unixiproute_only=1\
security.jail.chflags_allowed=0\
dev.amr.0.allow_volume_configure=1\
compat.linux.osrelease=2.6.12" >> /etc/sysctl.conf


19. mount procfs
cvsup sup ; pagedave
echo "proc                    /proc          procfs  rw              0      0" >> /etc/fstab
(20min)


For Dell 2950/2450:
3. configure new kernel. Get config from similar machine or there may be a master copy somewhere under /mnt/data4/build/freebsd (name the kernel config the same as the jail, ex jail18):
echo "linprocfs              /usr/compat/linux/proc linprocfs rw    0      0" >> /etc/fstab
cd /usr/src/sys/i386/conf
scp backup2:/mnt/data4/build/freebsd/kern_config-6.2 ./jail7


For Dell 2950:
or for PAE
echo "linsysfs                /usr/compat/linux/sys linsysfs rw      0      0" >> /etc/fstab


mkdir -p /usr/compat/linux/proc
scp backup2:/mnt/data4/build/freebsd/kern_config-6.2-PAE ./jail7
mkdir -p /usr/compat/linux/sys


20. reboot. Confirm new kernel is loaded
edit the kernel config and change ident to be the name of the jail:
vi jail7
ident          jail7


uname -a
edit /sys/conf/newvers.sh to add –jc1 to the end of the BRANCH string (RELEASE-jc1)
vi /sys/conf/newvers.sh


Check devfs rules:
3. bring over patches from backup2
devfs rule showsets
devfs rule -s 3 show


21. update ports:
The various patches are in /mnt/data4/build/freebsd/patches on backup2. There are dirs for each version. Not all dirs are populated, but patches for later versions work on older ones unless there is a new patch in the older dir.
cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_7_1\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup


cvsup sup; mail -s 'cvs sup ports done' dave.boodman@vtext.com < /dev/null
cd /usr/src
scp backup2:"/mnt/data4/build/freebsd/patches/6.x/*" .


(2450: 18mins, supermicro: 19mins; 2950: 24mins)
Apply patches:
patch -l < jls-patch


22. Install raid mgmt tool
Apply these only to 2950 with PAE:
patch -p0 < gvinum-staticcompile-patch
patch -p0 < gvinum-bin-patch


For LSI based cards:
4. build, install kernel
cd /usr/src
make buildkernel;  pagedave
(2min)


install linux_base:
cd /boot
cd /usr/ports/emulators/linux_base-fc4
mv kernel.old kernel.old.old
make install clean
(optional move old kernel out of the way)
(2450: 7min, supermicro: 3mins, 2950: 14mins)


(for LSI)
cd /usr/src
cd /usr/ports/sysutils/linux-megamgr
make installkernel
make install clean


cd /usr/ports/sysutils/megarc
cd /boot
make install clean
mv kernel.old kernel.6.2


(for Perc5/i, 6/i)
cd /usr/ports/sysutils/linux-megacli2
make install clean


Test:
5. take ½ of mirror and test (boot up) in new hardware
rehash; megacli ldinfo lall a0


For adaptec:
6. improved loader configs:
On jail18:
scp /compat/linux/usr/sbin/aaccli user@10.1.4.102:~


mv ~user/aaccli /compat/linux/usr/sbin/aaccli
/boot/loader.conf:
console="comconsole,vidconsole"
boot_serial="YES"
boot_multicons="YES"


test out;
7. build userland
/compat/linux/usr/sbin/aaccli
cd /usr/src
make buildworld; pagedave
(33mins)
make installworld


For 3w9690:
mergemaster –i
scp backup2:/d4/build/3ware/tw_cli-freebsd-x86_64-9.5.0.1.tgz /usr/local/sbin
reboot
cd /usr/local/sbin xzvf tw_cli-freebsd-x86_64-9.5.0.1.tgz
8. patch jail against starting jails with rtprio
rm tw_cli-freebsd-x86_64-9.5.0.1.tgz


test out;
mv /usr/sbin/jail /usr/sbin/jail_
./tw_cli /c0 show allstatus
echo '#\!/bin/sh\
/usr/sbin/rtprio -t /usr/sbin/jail_ $*' > /usr/sbin/jail
chmod +x /usr/sbin/jail
</pre>


23. install rsync from ports
= FreeBSD 7.0 =
cd /usr/ports/net/rsync
make install clean


choose default options
Last updated 2008-04-30


25. configure inetd to respond to mrtg load queries
<pre>
echo "load    stream  tcp    nowait  user    /usr/local/jail/bin/load.pl  load.pl" >> /etc/inetd.conf


echo "load            12384/tcp" >> /etc/services
All time extimates below assume disks aren’t scrubbing. Setup instructions below are for LSI card:


26. install bb client
1. make sure bios is setup for bios console redirect
adduser
Supermicro:
Username: bb
Console redirection:
Full name: bb
Com port addr: on-board COM A
Uid (Leave empty for default): 1984
Baud: 38400
Login group [bb]:
Console type: vt100
Login group is bb. Invite bb into other groups? []:
Flow control: none
Login class [default]:
Console connection: direct
Shell (sh csh tcsh nologin) [sh]:  
Continue cr after post: on
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username  : bb
Password  : <random>
Full Name  : bb
Uid        : 1984
Class      :
Groups    : bb
Home      : /home/bb
Shell      : /bin/sh
Locked    : no
OK? (yes/no): yes


cd /usr/home/bb
2450:
scp backup2:/mnt/data4/build/bb/bb-freebsd_linuxcompat.tgz .
Make sure running bios A09
tar xzf bb-freebsd_linuxcompat.tgz
Console Redirection: VT100/VT220


edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
2. assuming mirrors (or at least disks) created (if not, refer to this), boot to disk 1 of 5.4
echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
skip kernel config (enter)
10.1.4.102 jail2.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
custom install


vi /home/bb/bbc1.9e-btf/ext/openfiles
partition ->
MACHINE="jail2,johncompanies,com"      # HAS TO BE IN A,B,C FORM
move cursor to amrd0, hit space (takes you to partition map screen)
a for entire disk
q to quit and save
standard mbr (no boot manager)
space to unselect amrd0
cursor over amrd1
space
a for entire disk
q to quit and save
none (leave untouched)
 
cursor over amrd0
space
(takes you into part. Screen again) q to exit
none
Make sure both are checked and tab to ok


cd /usr/home/bb/bbc1.9e-btf/etc
Label ->
./bbchkcfg.sh
Make sure mfid0 is highlighted
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf
./runbb.sh start
more BBOUT
(look for errors)
exit


echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
/ 256M
chmod +x /usr/local/etc/rc.d/bb.sh
swap 2G (for 2950 make it 4G)
/var 256M (4.6G)
/tmp 256M
/usr 4G
/mnt/data1 remaining space (no need to newfs)


Make sure to toggle S for soft updates on all (should look like UFS2+S Y under the Newfs column)


NOTE: to get bb working on amd, had to copy over bin dir from linux dist
move cursor to mfid1
swap 8G
/mnt/data2 remaining space (no need to newfs)


27. configure load mrtg, on mail
q to save and exit
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg
(add new entry to file following existing format)


28. configure bb on mail:
distributions ->
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
developer
10.1.4.102 jail2.johncompanies.com # ssh
yes to install ports
exit


su bb
media ->
cd
cd
bbsrc/bb/runbb.sh restart ; exit


29. remove reserve space, enable softupdates (probably already set, so not necessary)
commit ->
NOT APPLICABLE IF USING GVINUM
yes
cd
(2450: 14mins, supermicro: 12mins)
umount /mnt/data1
umount /mnt/data2
tunefs -m 0 /mnt/data1
tunefs -m 0 /mnt/data2
mount -a


30. configure ntp
yes to "visit general config" ->
echo "server 10.1.4.5" > /etc/ntp.conf
Set root pwd


/usr/sbin/ntpd -p /var/run/ntpd.pid
Add user ‘user’ member group is wheel, set password
sleep 2; ntpq -p
(confirm it’s able to reach our time server)


32. fwd and reverse lookups on ns1c
Set tz
vr johncompanies.com
vi internal.johncompanies.com
rndc reload johncompanies.com IN private
(edit the PTR too)


Networking->interfaces->Fxp0 ->
No IPV6
dhcp=no
Set hostname & domain
Enable sshd


33. create gvinum volumes
exit...
exit install ->
yes


Make a g partition:
take the cd out and let the machine reboot


2950:
3. double check the date/time
bsdlabel -e /dev/mfid0s1
 
bsdlabel -e /dev/mfid1s1
4. edit /etc/make.conf (only add the console speed line if this is a supermicro capable of outputting BIOS redirect at that speed – 2450’s can only do 9600)
bsdlabel -e /dev/mfid2s1
echo "WITHOUT_X11=yes \
KERNCONF=jail2 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf


given:
# /dev/aacd0s1:
8 partitions:
#        size  offset    fstype  [fsize bsize bps/cpg]
  a:  262144        0    4.2BSD    2048 16384 16392
  b:  4194304  262144      swap
  c: 143363997        0    unused        0    0        # "raw" part, don't edit
  d:  524288  4456448    4.2BSD    2048 16384 32776
  e:  524288  4980736    4.2BSD    2048 16384 32776
  f:  6291456  5505024    4.2BSD    2048 16384 28552


new offset = 6291456 + 5505024 = 11796480
new size is size for 'c' partition minus the new start from above
143363997 - 11796480 = 131567517
So:
g: 131567517 11796480 unused 0 0


bsdlabel -e /dev/amrd1s1
5. add settings to /boot/loader.conf and /boot.config
change d to g


For a 146G drive (after OS and 4G swap), we can fit 127 1G volumes so:
echo "-Dh" >> /boot.config


echo 'drive data1 device /dev/mfid0s1g' > /tmp/cgv
echo 'console="comconsole,vidconsole" \
boot_multicons="YES" \
boot_serial="YES" \
mfi_linux_load="YES" \
comconsole_speed="115200"' >> /boot/loader.conf


echo '#\!/bin/sh\
(leave out the speed and mfi lines for 2450s)
i="1"\
while [ $i -le 127 ]\
do\
  echo "volume v$i" >> /tmp/cgv;\
  echo 'plex org concat' >> /tmp/cgv;\
  echo 'sd length 1g drive data1' >> /tmp/cgv;\
i=`expr $i + 1`\
done' > /tmp/mkcgv
sh /tmp/mkcgv
gvinum create /tmp/cgv


volume 1
6. turn off all ttyv's except 0 and 1 in /etc/ttys
plex org concat
also turn on ttyd0, change type to vt100:
sd length 3g drive data1
vi /etc/ttys
volume 2
ttyd0  "/usr/libexec/getty std.9600"  vt100  on secure
plex org concat
 
sd length 5g drive data1
kill -1 1
volume 3
plex org concat
sd length 8g drive data1
volume 4
plex org concat
sd length 6g drive data1
volume 5
plex org concat
sd length 10g drive data1


for f in 1 2 3 4 5; do bsdlabel -rw /dev/gvinum/$f; done
on console server:
for f in 1 2 3 4 5; do newfs /dev/gvinum/${f}a; done
vi /etc/remote
for f in 1 2 3 4 5; do mkdir /mnt/data1/$f; mount /dev/gvinum/${f}a /mnt/data1/$f;done
(rename port to jail18 depending on where and which digi plugged into)
for f in 1 2 3 4 5; do dd if=/dev/zero of=/mnt/data1/$f/this_is_$f bs=1m count=$f; done
test serial console
for f in 1 2 3 4 5; do ll -h /mnt/data1/$f/; done


7. populate hosts
echo "10.1.4.3 backup2" >> /etc/hosts
echo "10.1.4.8 backup1" >> /etc/hosts


For a 300G drive (after OS and 4G swap), we can fit 273 1G volumes so:


echo 'drive data2 device /dev/mfid1s1g' > /tmp/cgv
8. put key in authorized_keys on backup2
cd
ssh-keygen -t dsa -b 1024
(default location, leave password blank)


echo '#\!/bin/sh\
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
i="128"\
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'
while [ $i -le 401 ]\
do\
  echo "volume v$i" >> /tmp/cgv;\
  echo 'plex org concat' >> /tmp/cgv;\
  echo 'sd length 1g drive data2' >> /tmp/cgv;\
i=`expr $i + 1`\
done' > /tmp/mkcgv
sh /tmp/mkcgv
gvinum create /tmp/cgv


confirm that you can ssh to backup2 and backup1 without getting a login prompt


For a 300G drive (after OS and 4G swap), we can fit 273 1G volumes so:
ssh backup2 hostname


echo 'drive data3 device /dev/mfid2s1g' > /tmp/cgv
ssh backup1 hostname


echo '#\!/bin/sh\
9. create & populate binaries/scripts dirs
i="402"\
mkdir -p /usr/local/jail/bin
while [ $i -le 675 ]\
mkdir -p /usr/local/jail/rc.d
do\
mkdir -p /usr/local/jail/template/
  echo "volume v$i" >> /tmp/cgv;\
mkdir /mnt/data1
  echo 'plex org concat' >> /tmp/cgv;\
mkdir /mnt/data2
  echo 'sd length 1g drive data3' >> /tmp/cgv;\
scp backup2:"/mnt/data4/bin/freebsd7.x/*" /usr/local/jail/bin
i=`expr $i + 1`\
cd /usr/local/jail/rc.d/
done' > /tmp/mkcgv
touch quad1
sh /tmp/mkcgv
touch quad2
gvinum create /tmp/cgv
touch quad3
touch quad4
touch safe1
touch safe2
touch safe3
touch safe4
chmod +x *
cd /usr/local/jail/bin
ln -s /usr/local/jail/rc.d/quad1 quad1
ln -s /usr/local/jail/rc.d/quad2 quad2
ln -s /usr/local/jail/rc.d/quad3 quad3
ln -s /usr/local/jail/rc.d/quad4 quad4
ln -s /usr/local/jail/rc.d/safe1 safe1
ln -s /usr/local/jail/rc.d/safe2 safe2
ln -s /usr/local/jail/rc.d/safe3 safe3
ln -s /usr/local/jail/rc.d/safe4 safe4
ln -s /usr/local/jail/bin/jailmake_zfs jailmake
--OR--
ln -s /usr/local/jail/bin/jailmake_geom jailmake


Sometimes there’s 2040m leftover so create one more smaller vol:
ln -s /usr/local/jail/bin/js_zfs js
echo 'volume v64\
--OR--
plex org concat\
ln -s /usr/local/jail/bin/js_geom js
sd length 2040m drive data1' >> /tmp/cgv


To delete:
rehash


echo '#\!/bin/sh\
10. edit root's path and login script:
i="1"\
vi /root/.cshrc
while [ $i -le 127 ]\
add to path:
do\
/usr/local/jail/bin
  echo "gvinum rm -r v$i" >> /tmp/dgv;\
(if adaptec card installed, also add /compat/linux/usr/sbin)
i=`expr $i + 1`\
done' > /tmp/mkdgv
sh /tmp/mkdgv
sh /tmp/dgv


dd if=/dev/zero of=/dev/sdb seek=285474816 bs=512
Change alias entries (add G):
dd if=/dev/zero of=/dev/mfid0s2 bs=1m count=1000 (do this to all drives)
alias la        ls -aG
alias lf        ls -FAG
alias ll        ls -lAG
alias ls        ls -AG
alias mbm      mb mount
alias mbu      mb umount


35. create the jail template
and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "


at the bottom of the file add:
set sshtty=`who am i|awk '{print $2}'`
/usr/sbin/rtprio 3 -`psj | grep $sshtty | awk '{print $2}'`


touch /mnt/data1/jail
set shortty=`who am i | awk '{print $2}' | sed -E 's/.*(..)$/\1/'`
mdconfig -a -t vnode -s 1g -f /mnt/data1/jail -u 0
foreach x (`psj | grep sh | grep $shortty | awk '{print $2}'`)
bsdlabel -rw md0
/usr/sbin/rtprio 2 -$x
newfs4x /dev/md0c
end
mkdir /mnt/data1/jail-DIR
mount /dev/md0c /mnt/data1/jail-DIR


cd /usr/ports/sysutils/jailutils
To load the new file:
make install clean
source /root/.cshrc


cd /usr/src
11. install cvsup
make world DESTDIR=/mnt/data1/jail-DIR
cd /usr/ports/net/cvsup-without-gui
(2450: 2:28mins, supermicro: 55mins, 2950: 1h )
make install clean; rehash; mail -s 'cvs installed' dave.boodman@vtext.com < /dev/null
cd etc
make distribution DESTDIR=/mnt/data1/jail-DIR
mount -t devfs devfs /mnt/data1/jail-DIR/dev
devfs -m /mnt/data1/jail-DIR/dev rule -s 3 applyset
cd /mnt/data1/jail-DIR
ln -sf dev/null kernel
cp /usr/local/sbin/jkill /mnt/data1/jail-DIR/sbin


jail /mnt/data1/jail-DIR testhostname 192.168.11.100 /bin/sh
(stay close for gettext options, 2450: 27mins, supermicro: 17mins, 2950: 22mins)
csh
touch /etc/fstab
echo 'network_interfaces=""\
hostname="newsystem"\
kern_securelevel_enable="NO"\
sendmail_enable="YES"\
sshd_enable="YES"' > /etc/rc.conf


echo "nameserver 69.55.225.225\
12. get latest sources for this release:
nameserver 69.55.230.3" >> /etc/resolv.conf
cd /usr/src
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_7\
*default delete use-rel-suffix\
*default compress\
src-all" > sup


vi /etc/crontab
remove the adjkerntz lines
comment out periodic’s and put this line above them:
# IF YOU UNCOMMENT THESE, PLEASE ADJUST THEIR RUN TIME


rm -rf /etc/periodic/daily/400.status-disks
-OR-


check /tmp for crap
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_7_1\
*default delete use-rel-suffix\
*default compress\
src-all" > sup


vi /etc/periodic/security/100.chksetuid
cvsup sup ; mail -s 'cvs sup done' dave.boodman@vtext.com < /dev/null
replace: MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`
with: MP='/' (use single quotes)


mkdir -p /usr/compat/linux/dev
(2450, ~12mins, supermicro, 27mins, 2950: 7mins)


adduser (Add account for user, make sure in wheel group)


user/root passwd: 8ico2987
13. configure new kernel.
for i386:
cd /usr/src/sys/i386/conf
scp backup2:/mnt/data4/build/freebsd/kern_config-7.0-PAE ./mx2


Set root password
-or-
for amd64:
cd /usr/src/sys/amd64/conf
scp backup2:/mnt/data4/build/freebsd/kern_config-7.0-amd64 ./jail2


vi /usr/home/user/.profile (and add to the file):
------
TERM=vt100;    export TERM


tzsetup
edit the kernel config and change ident to be the name of the jail:
vi jail2
ident          jail2


newaliases
edit /sys/conf/newvers.sh to add –jc2 to the end of the BRANCH string (RELEASE-jc2)
vi /sys/conf/newvers.sh


rm /sbin/halt /sbin/reboot
notes: http://www.FreeBSD.org/doc/en_US.ISO8859-1/books/handbook/kernelconfig-config.html
ln /sbin/jkill /sbin/halt
ln /sbin/jkill /sbin/reboot


#cd /dev
14. bring over patches from backup2
#rm console
#ln -s null console


vi /etc/syslog.conf (comment out console and move to /var/log/messages):
The various patches are in /mnt/data4/build/freebsd/patches on backup2. There are dirs for each version. Not all dirs are populated, but patches for later versions work on older ones unless there is a new patch in the older dir.
#*.err;kern.warning;auth.notice;mail.crit              /dev/console *.err;kern.warning;auth.notice;mail.crit                /var/log/messages


exit
cd /usr/src
exit
scp backup2:"/mnt/data4/build/freebsd/patches/7.x/*" .


cd libexec
Apply patches:
mv ld-elf32.so.1 ld-elf32.so.1-orig
patch -l < jls-patch
ln ld-elf.so.1 ld-elf32.so.1


may have to run:
15. build, install kernel and world
chflags noschg ld-elf32.so.1
cd /usr/src
chflags noschg ld-elf.so.1
make buildworld buildkernel installkernel; mail -s 'kernel build done' dave.boodman@vtext.com < /dev/null
then
(2450: 1:56min, supermicro: 59mins, 2950: 38mins)
chflags schg ld-elf.so.1
make installworld
chflags schg ld-elf32.so.1
(2450: 3min, supermicro: 1min, 2950: :34)
mergemaster -i
delete /var/tmp/temproot
delete bsnmpd
delete temporary ./etc/hosts
delete temporary ./etc/motd
delete /var/tmp/temproot


cd /usr/src/sys/modules/zfs
make
make install


cd /usr/ports
16. populate devfs ruleset
make -DNOCLEANDEPENDS clean
scp backup2:/mnt/data4/build/freebsd/devfs.rules /etc
(2450: 15mins , supermicro: 29mins, 2950: 18mins)
rm -fr /usr/ports/distfiles/*
cp -r /usr/ports /mnt/data1/jail-DIR/usr (2450: 2:00 mins , supermicro: 15mins, 2950: 3mins)


rm /mnt/data1/jail-DIR/root/.history
17. populate /etc/rc.conf with IPs and NFS settings
vi /etc/rc.conf


cd
kern_securelevel_enable="NO"
dump -0a -f /usr/local/jail/template/71template /dev/md0c
portmap_enable="NO"
umount /mnt/data1/jail-DIR/dev
sendmail_enable="NO"
umount /dev/md0c
usbd_enable="YES"
rmdir /mnt/data1/jail-DIR


xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
sshd_enable="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.4.102"
devfs_system_ruleset="devfsrules_show_all"


36. setup backups
hostname="jail2.johncompanies.com"
echo '#\!/bin/sh\
ifconfig_xl0="inet 10.1.4.102 netmask 255.255.255.0"
backupdir=/data/jail2\
ifconfig_fxp0="inet 69.55.228.53 netmask 255.255.255.0"
server=backup1\
defaultrouter="69.55.228.1"
\
#ifconfig_fxp0_alias0="inet 69.55.2xx.xx netmask 255.255.255.0"
## ENTRY /etc\
## ENTRY /usr/local/etc\
## ENTRY /usr/local/jail\
## ENTRY /root/logs' > /usr/local/jail/bin/backup.config


on backup1:
static_routes="t1 office"
setup backup dirs:
route_t1="-net 10.1.5 10.1.4.2"
ssh backup1 mkdir -p /data/jail2/0
route_office="-net 10.1.6 10.1.4.2"
zfs_enable="YES"


add the system to
18. make sure sysctls are set and preserved after reboot
vi /usr/local/sbin/snapshot_rotate
echo "kern.consmute=0\
kern.ipc.shm_use_phys=1\
kern.ipc.shmall=65535\
kern.ipc.shmmax=134217728\
net.inet.tcp.syncookies=0\
kern.maxfiles=32768\
kern.fallback_elf_brand=3\
kern.maxprocperuid=4000\
security.jail.sysvipc_allowed=1\
security.jail.allow_raw_sockets=1\
security.jail.socket_unixiproute_only=1\
security.jail.chflags_allowed=0\
compat.linux.osrelease=2.6.12\
kern.maxvnodes=400000" >> /etc/sysctl.conf


on mail:
19. mount procfs
vi /usr/local/www/mgmt/cgi/backupgraph.pl
echo "proc                    /proc          procfs  rw              0      0" >> /etc/fstab
(add hostname)


Edit /usr/local/jail/bin/backup.xxx to use the right drives and copy to /usr/local/jail/bin/backup
For Dell 2950/2450:
echo "linprocfs              /usr/compat/linux/proc linprocfs rw    0      0" >> /etc/fstab


37. mkdir /root/logs
For Dell 2950:
echo "linsysfs                /usr/compat/linux/sys linsysfs rw      0      0" >> /etc/fstab


38. edit sshd_config for security
20. reboot. Confirm new kernel is loaded
vi /etc/ssh/sshd_config
ListenAddress 69.55.228.53
ListenAddress 10.1.4.102


kill -1 `cat /var/run/sshd.pid`
uname –a


39. add crontab entries
Check devfs rules:
crontab -e
devfs rule showsets
5 0 * * * /usr/local/jail/bin/backup.md
devfs rule -s 3 show
1 0 1 * * /usr/local/jail/bin/ipfwreset
0 18 * * * /usr/local/jail/bin/ipfwbackup
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /usr/local/jail/bin/trafstats


On 2950 with Perc5/i also add:
21. update ports:
*/5 * * * * /usr/local/jail/bin/perc5iraidchk
cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_7_0\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup


40. Reboot notify script
cvsup sup; mail -s 'cvs sup ports done' dave.boodman@vtext.com < /dev/null
ln -s /usr/local/jail/bin/notify.sh /usr/local/etc/rc.d/notify.sh


41. add to management db (on mail and devweb) jc.ref_machines and jc.ref_templates
(2450: 18mins, supermicro: 19mins; 2950: 24mins)


uname -r
22. Install raid mgmt tool
6.2-RELEASE-jc1


insert into ref_machines values (null,'mx2','mx2.johncompanies.com',0,'m');
For LSI based cards:
select machine_id from ref_machines where host='mx2';
+------------+
| machine_id |
+------------+
|        35 |
+------------+
insert into ref_templates values ('',' 7.1-RELEASE-jc2',44,'FreeBSD 7.1',0);


42. add to server/cabinet map. On mail:
install linux_base:
vi /usr/local/www/mgmt/html/cabinetmap.html
cd /usr/ports/emulators/linux_base-fc4
make install clean
(2450: 7min, supermicro: 3mins, 2950: 14mins)


43. add an outside blocking rule to the firewall, so this machine can only be reached from inside the firewall. Follow example already in firewall jail17 is:
(for LSI)
cd /usr/ports/sysutils/linux-megamgr
00117 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.228.200
make install clean
00117 deny ip from any to 69.55.228.200


jail19 would be 00119...
cd /usr/ports/sysutils/megarc
ipfw add 00107 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.230.108
make install clean
ipfw add 00107 deny ip from any to 69.55.220.108


44. select customers for probe map
(for Perc5/i)
cd /usr/ports/sysutils/linux-megacli
make install clean


46. make gv start on boot
Test:
rehash; megacli ldinfo lall a0


scp backup2:/mnt/data4/build/freebsd/gvinum /etc/rc.d/gvinum
For adaptec:
On jail18:
scp /compat/linux/usr/sbin/aaccli user@10.1.4.102:~


If they start stale:
mv ~user/aaccli /compat/linux/usr/sbin/aaccli


echo '#\!/bin/sh\
test out;
i="1"\
/compat/linux/usr/sbin/aaccli
while [ $i -le 127 ]\
do\
        gvinum start v$i\
        i=`expr $i + 1`\
done' > /tmp/startgv


sh /tmp/startgv
For 3w9690:
scp backup2:/d4/build/3ware/tw_cli-freebsd-x86_64-9.5.0.1.tgz /usr/local/sbin
cd /usr/local/sbin xzvf tw_cli-freebsd-x86_64-9.5.0.1.tgz
rm tw_cli-freebsd-x86_64-9.5.0.1.tgz


test out;
./tw_cli /c0 show allstatus


47. patch jail against starting jails with rtprio
23. install rsync from ports
cd /usr/ports/net/rsync
make install clean


mv /usr/sbin/jail /usr/sbin/jail_
choose default options
echo '#\!/bin/sh\
/usr/sbin/rtprio -t /usr/sbin/jail_ $*' > /usr/sbin/jail
chmod +x /usr/sbin/jail


48. make sure mail works
24. install perl from ports
If there are map errors:
Not necessary if linux_base is installed
cd /etc/mail; make maps


49. move and pare down generic kernel
cd /usr/ports/lang/perl5.8/
mv kernel.old/ kernel.GENERIC
make install clean; rehash
(supermicro: 5min)


/  too full, remove old kernel modules except for:
25. configure inetd to respond to mrtg load queries
echo "load    stream  tcp    nowait  user    /usr/local/jail/bin/load.pl load.pl" >> /etc/inetd.conf


Id Refs Address           Size    Name
echo "load           12384/tcp" >> /etc/services
1  14 0xffffffff80100000 ac6c08  kernel
2    1 0xffffffff80bc8000 1128    mfi_linux.ko
3    4 0xffffffff80bca000 39138    linux.ko
4    1 0xffffffffb48f8000 358c    linprocfs.ko
5    1 0xffffffffb48fc000 9d3      linsysfs.ko
6    1 0xffffffffb490c000 80ee8    zfs.ko


26. install bb client
adduser
Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username  : bb
Password  : <random>
Full Name  : bb
Uid        : 1984
Class      :
Groups    : bb
Home      : /home/bb
Shell      : /bin/sh
Locked    : no
OK? (yes/no): yes


50. ntp doesn’t seem to start from rc so,
cd /usr/home/bb
 
scp backup2:/mnt/data4/build/bb/bb-freebsd.tar .
echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh
tar xf bb-freebsd.tar
chmod 0700 /usr/local/etc/rc.d/ntp.sh</pre>
 


= 7.2 =
edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.4.108 jail2.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts


Last updated 2009-06-18
vi /home/bb/bbc1.9e-btf/ext/openfiles
MACHINE="jail2,johncompanies,com"      # HAS TO BE IN A,B,C FORM


All time extimates below assume disks aren’t scrubbing. Setup instructions below are for LSI card:
cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src
make; make install
cd ..
./runbb.sh start
more BBOUT
(look for errors)
exit


1. make sure bios is setup for bios console redirect
echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
2950:
chmod +x /usr/local/etc/rc.d/bb.sh
Console redirection:
LCD string..
Date to GMT




2. assuming mirrors (or at least disks) created (if not, refer to this), boot to disk 1 of 7.2
NOTE: to get bb working on amd, had to copy over bin dir from linux dist
skip kernel config (enter)
 
custom install
27. configure load mrtg, on mail
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg
(add new entry to file following existing format)


partition ->
28. configure bb on mail:
move cursor to mfid0, hit space (takes you to partition map screen)
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
a for entire disk
10.1.4.108 jail2.johncompanies.com # ssh
q to quit and save


su bb
cd
bbsrc/bb/runbb.sh restart ; exit


standard mbr (no boot manager)
29. remove reserve space, enable softupdates (probably already set, so not necessary)
space to unselect mfid0
NOT APPLICABLE IF USING GVINUM
cursor over mfid1
cd
space
umount /mnt/data1
a for entire disk
umount /mnt/data2
q to quit and save
tunefs -m 0 /mnt/data1
none (leave untouched)
tunefs -m 0 /mnt/data2
mount -a


cursor over mfid0
30. configure ntp
space
echo "server 10.1.4.5" > /etc/ntp.conf
(takes you into part. Screen again) q to exit
none
Make sure both are checked and tab to ok


Label ->
/usr/sbin/ntpd -p /var/run/ntpd.pid
Make sure mfid0 is highlighted
sleep 2; ntpq -p
(confirm it’s able to reach our time server)


/ 256M
31. mrtg switch graphs
swap 2G (for 2950 make it 4G)
/var 256M (4.6G)
/tmp 256M
/usr 4G
/mnt/data1 remaining space (no need to newfs)


Make sure to toggle S for soft updates on all (should look like UFS2+S Y under the Newfs column)
32. fwd and reverse lookups on ns1c
vr johncompanies.com
vi internal.johncompanies.com
rndc reload johncompanies.com IN private
(edit the PTR too)


move cursor to mfid1
swap 8G (or 4G if there’s a 3rd drive)
/mnt/data2 remaining space (no need to newfs)


q to save and exit
33. create zpools


distributions ->
Make a g partition:
developer
yes to install ports
exit


media ->
bsdlabel -e /dev/mfid0s1
cd


commit ->
given:
yes
# /dev/aacd0s1:
(2450: 14mins, supermicro: 12mins)
8 partitions:
#        size  offset    fstype  [fsize bsize bps/cpg]
  a:  262144        0    4.2BSD    2048 16384 16392
  b:  4194304  262144      swap
  c: 143363997        0    unused        0    0        # "raw" part, don't edit
  d:  524288  4456448    4.2BSD    2048 16384 32776
  e:  524288  4980736    4.2BSD    2048 16384 32776
  f: 6291456  5505024    4.2BSD    2048 16384 28552


yes to "visit general config" ->
new offset = 6291456 + 5505024 = 11796480
Set root pwd
new size is size for 'c' partition minus the new start from above
143363997 - 11796480 = 131567517
So:
g: 131567517 11796480 unused 0 0


Add user ‘user’ member group is wheel, set password
bsdlabel -e /dev/mfid1s1
change d to g


Set tz


Networking->interfaces->Fxp0 ->
zpool create pool1 mfid0s1g
No IPV6
zpool create pool2 mfid1s1g
dhcp=no
zfs set atime=off pool1
Set hostname & domain
zfs set atime=off pool2
Enable sshd


exit...
echo ' vfs.zfs.prefetch_disable="1" \
exit install ->
vfs.zfs.arc_min=16777216 \
yes
vfs.zfs.arc_max=33554432 \
vm.kmem_size_max="1G" # for i386\
vm.kmem_size="1G" # for i386\
vm.kmem_size_max="1.5G" # for amd64\
vm.kmem_size="1.5G" # for amd64\
vfs.zfs.zil_disable="1" ' >> /boot/loader.conf


take the cd out and let the machine reboot


3. double check the date/time
35. create the jail template


4. edit /etc/make.conf
echo "WITHOUT_X11=yes \
KERNCONF=jail8 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf


zfs create -o mountpoint=/mnt/data1/jail -o quota=4G pool1/jail


5. add settings to /boot/loader.conf and /boot.config
cd /usr/ports/sysutils/jailutils
make install clean


echo "-Dh" >> /boot.config
cd /usr/src
make world DESTDIR=/mnt/data1/jail
(2450: 2:28mins, supermicro: 55mins, 2950: 1h )
cd etc
make distribution DESTDIR=/mnt/data1/jail
mount -t devfs devfs /mnt/data1/jail/dev
devfs -m /mnt/data1/jail/dev rule -s 3 applyset
cd /mnt/data1/jail
ln -sf dev/null kernel
cp /usr/local/sbin/jkill /mnt/data1/jail/sbin


echo 'console="comconsole,vidconsole" \
jail /mnt/data1/jail testhostname 192.168.11.100 /bin/sh
boot_multicons="YES" \
csh
boot_serial="YES" \
touch /etc/fstab
mfi_linux_load="YES" \
echo 'network_interfaces=""\
comconsole_speed="115200"' >> /boot/loader.conf
hostname="newsystem"\
kern_securelevel_enable="NO"\
sendmail_enable="YES"\
sshd_enable="YES"' > /etc/rc.conf


echo "nameserver 69.55.225.225\
nameserver 69.55.230.3" >> /etc/resolv.conf


6. turn off all ttyv's except 0 and 1 in /etc/ttys
vi /etc/crontab
also turn on ttyd0, change type to vt100:
remove the adjkerntz lines
vi /etc/ttys
comment out periodic’s and put this line above them:
# IF YOU UNCOMMENT THESE, PLEASE ADJUST THEIR RUN TIME


ttyv2  "/usr/libexec/getty Pc"        cons25  off secure
rm -rf /etc/periodic/daily/400.status-disks
ttyv3  "/usr/libexec/getty Pc"        cons25  off secure
ttyv4  "/usr/libexec/getty Pc"        cons25  off secure
ttyv5  "/usr/libexec/getty Pc"        cons25  off secure
ttyv6  "/usr/libexec/getty Pc"        cons25  off secure
ttyv7  "/usr/libexec/getty Pc"        cons25  off secure


ttyd0  "/usr/libexec/getty std.9600"  vt100  on secure
check /tmp for crap


kill -1 1
vi /etc/periodic/security/100.chksetuid
replace: MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`
with: MP='/' (use single quotes)


on console server:
mkdir -p /usr/compat/linux/dev
vi /etc/remote
(rename port to jail18 depending on where and which digi plugged into)
test serial console


7. populate hosts
adduser (Add account for user, make sure in wheel group)
echo "10.1.4.3 backup2" >> /etc/hosts
echo "10.1.4.8 backup1" >> /etc/hosts


user/root passwd: 8ico2987


8. put key in authorized_keys on backup2
Set root password
cd
ssh-keygen -t dsa -b 1024
(default location, leave password blank)


cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
vi /usr/home/user/.profile (and add to the file):
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'
TERM=vt100;    export TERM


confirm that you can ssh to backup2 and backup1 without getting a login prompt
tzsetup


ssh backup2 hostname
newaliases


ssh backup1 hostname
rm /sbin/halt /sbin/reboot
ln /sbin/jkill /sbin/halt
ln /sbin/jkill /sbin/reboot


9. create & populate binaries/scripts dirs
#cd /dev
mkdir -p /usr/local/jail/bin
#rm console
mkdir -p /usr/local/jail/rc.d
#ln -s null console
mkdir -p /usr/local/jail/template/
mkdir /mnt/data1
mkdir /mnt/data2
scp backup2:"/mnt/data4/bin/freebsd7.x/*" /usr/local/jail/bin
cd /usr/local/jail/rc.d/
touch quad1
touch deprecated
chmod +x *
cd /usr/local/jail/bin
ln -s /usr/local/jail/rc.d/quad1 quad1
ln -s /usr/local/jail/bin/jailmake_md jailmake
ln -s /usr/local/jail/bin/js_md js
ln -s /usr/local/jail/bin/canceljail_md canceljail
ln -s /usr/local/jail/bin/jailmakeempty_md jailmakeempty
ln -s /usr/local/jail/bin/postboot_md postboot


rehash
vi /etc/syslog.conf (comment out console and move to /var/log/messages):
#*.err;kern.warning;auth.notice;mail.crit              /dev/console *.err;kern.warning;auth.notice;mail.crit                /var/log/messages


10. edit root's path and login script:
cd /libexec
vi /root/.cshrc
mv ld-elf32.so.1 ld-elf32.so.1-orig
ln ld-elf.so.1 ld-elf32.so.1


Change alias entries (add G):
exit
alias la        ls -aG
exit
alias lf        ls -FAG
alias ll        ls -lAG
alias ls        ls -AG
alias mbm      mb mount
alias mbu      mb umount


cd /usr/ports
make -DNOCLEANDEPENDS clean
(2450: 15mins , supermicro: 29mins, 2950: 24mins)
rm -fr /usr/ports/distfiles/*
cp -r /usr/ports /mnt/data1/jail/usr (2450: 2:00 mins , supermicro: 15mins, 2950: 3mins)


add to path:
rm /mnt/data1/jail/root/.history
/usr/local/jail/bin
(if adaptec card installed, also add /compat/linux/usr/sbin)


and alter the prompt, set the following:
cd
set prompt = "`/bin/hostname -s` %/# "
zfs snapshot pool1/jail@now
zfs send pool1/jail@now | cat > /usr/local/jail/template/70template
zfs destroy pool1/jail@now


at the bottom of the file add:
zfs destroy pool1/jail
set sshtty=`who am i|awk '{print $2}'`
 
/usr/sbin/rtprio 3 -`psj | grep $sshtty | awk '{print $2}'`
rmdir /mnt/data1/jail
f


set shortty=`who am i | awk '{print $2}' | sed -E 's/.*(..)$/\1/'`
36. setup backups
foreach x (`psj | grep sh | grep $shortty | awk '{print $2}'`)
echo '#\!/bin/sh\
/usr/sbin/rtprio 2 -$x
backupdir=/data/jail2\
end
server=backup1\
\
## ENTRY /etc\
## ENTRY /usr/local/etc\
## ENTRY /usr/local/jail\
## ENTRY /root/logs' > /usr/local/jail/bin/backup.config


To load the new file:
on backup1:
source /root/.cshrc
setup backup dirs:
mkdir -p /data/jail2/0


11. install cvsup
add the system to
cd /usr/ports/net/cvsup-without-gui
vi /usr/local/sbin/snapshot_rotate
make install clean; rehash; mail -s 'cvs installed' dave.boodman@vtext.com < /dev/null


(stay close for gettext options, 2450: 27mins, supermicro: 17mins, 2950: 22mins)
on mail:
vi /usr/local/www/mgmt/cgi/backupgraph.pl
(add hostname)


12. get latest sources for this release:
Edit /usr/local/jail/bin/backup to use the right drives
cd /usr/src
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_7_2\
*default delete use-rel-suffix\
*default compress\
src-all" > sup


-OR-
37. mkdir /root/logs


echo "*default host=cvsup4.freebsd.org\
38. edit sshd_config for security
*default base=/usr\
vi /etc/ssh/sshd_config
*default prefix=/usr\
ListenAddress 69.55.238.233
*default release=cvs tag=RELENG_7\
ListenAddress 10.1.4.108
*default delete use-rel-suffix\
*default compress\
src-all" > sup


(stable)
kill -1 `cat /var/run/sshd.pid`


39. add crontab entries
crontab -e
5 0 * * * /usr/local/jail/bin/backup.zfs
1 0 1 * * /usr/local/jail/bin/ipfwreset
0 18 * * * /usr/local/jail/bin/ipfwbackup
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /usr/local/jail/bin/trafstats


cvsup sup ; mail -s 'cvs sup done' dave.boodman@vtext.com < /dev/null
On 2950 with Perc5/i also add:
*/5 * * * * /usr/local/jail/bin/perc5iraidchk


(2450, ~12mins, supermicro, 27mins, 2950: 7mins)
40. Reboot notify script
ln -s /usr/local/jail/bin/notify.sh /usr/local/etc/rc.d/notify.sh


41. add to management db (on mail and devweb) jc.ref_machines and jc.ref_templates


13. configure new kernel.  
uname -r
6.2-RELEASE-jc1


cd /usr/src/sys/amd64/conf
insert into ref_machines values (null,'mx2','mx2.johncompanies.com',0,'m');
scp backup2:/mnt/data4/build/freebsd/kern_config-7.1-amd64 ./jail8
select machine_id from ref_machines where host='mx2';
+------------+
| machine_id |
+------------+
|        35 |
+------------+
insert into ref_templates values ('',' 7.1-RELEASE-jc2',44,'FreeBSD 7.1',0);


edit the kernel config and change ident to be the name of the jail:
42. add to server/cabinet map. On mail:
vi jail8
vi /usr/local/www/mgmt/html/cabinetmap.html
ident          jail8


edit /sys/conf/newvers.sh to add –jc2 to the end of the BRANCH string (RELEASE-jc2)
43. add an outside blocking rule to the firewall, so this machine can only be reached from inside the firewall. Follow example already in firewall jail17 is:
vi /sys/conf/newvers.sh
00117 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.228.200
00117 deny ip from any to 69.55.228.200


notes: http://www.FreeBSD.org/doc/en_US.ISO8859-1/books/handbook/kernelconfig-config.html
jail19 would be 00119...
ipfw add 00107 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.230.108
ipfw add 00107 deny ip from any to 69.55.220.108


14. bring over patches from backup2
44. select customers for probe map


The various patches are in /mnt/data4/build/freebsd/patches on backup2. There are dirs for each version. Not all dirs are populated, but patches for later versions work on older ones unless there is a new patch in the older dir.
47. patch jail against starting jails with rtprio
 
mv /usr/sbin/jail /usr/sbin/jail_
echo '#\!/bin/sh\
/usr/sbin/rtprio -t /usr/sbin/jail_ $*' > /usr/sbin/jail
chmod +x /usr/sbin/jail
 
48. make sure mail works
If there are map errors:
cd /etc/mail; make maps


cd /usr/src
49. move and pare down generic kernel
scp backup2:"/mnt/data4/build/freebsd/patches/7.2/*" .
mv kernel.old/ kernel.GENERIC


Apply patches:
/  too full, remove old kernel modules except for:
patch -l < jls-patch


Id Refs Address            Size    Name
1  14 0xffffffff80100000 ac6c08  kernel
2    1 0xffffffff80bc8000 1128    mfi_linux.ko
3    4 0xffffffff80bca000 39138    linux.ko
4    1 0xffffffffb48f8000 358c    linprocfs.ko
5    1 0xffffffffb48fc000 9d3      linsysfs.ko
6    1 0xffffffffb490c000 80ee8    zfs.ko


15. build, install kernel and world


cd /boot
50. ntp doesn’t seem to start from rc so,


mv kernel kernel.GENERIC
echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh
cd kernel.GENERIC
chmod 0700 /usr/local/etc/rc.d/ntp.sh
mkdir hold
</pre>
mv mfi_linux.ko hold/
mv linux.ko hold/
mv linprocfs.ko hold/
mv linsysfs.ko hold/
mv geom_vinum.ko hold/
mv geom_concat.ko hold/
rm *.ko
rm *.symbols
mv hold/* .
rmdir hold/


cd /usr/src
= FreeBSD 7.1 =
make buildkernel installkernel


make buildworld ; mail -s 'buildworld done' dave.boodman@vtext.com < /dev/null
Last updated 2009-02-16
(2450: 1:56min, supermicro: 59mins, 2950: 38mins)
make installworld
(2450: 3min, supermicro: 1min, 2950: :34)
mergemaster -i
delete /var/tmp/temproot
delete bsnmpd
delete temporary ./etc/hosts
delete temporary ./etc/motd
delete /var/tmp/temproot


cd /sys/modules/geom/geom_vinum
<pre>All time extimates below assume disks aren’t scrubbing. Setup instructions below are for LSI card:
make
make install


16. populate devfs ruleset
1. make sure bios is setup for bios console redirect
scp backup2:/mnt/data4/build/freebsd/devfs.rules /etc
Supermicro:
Console redirection:
Com port addr: on-board COM A
Baud: 38400
Console type: vt100
Flow control: none
Console connection: direct
Continue cr after post: on


17. populate /etc/rc.conf with IPs and NFS settings
2450:
vi /etc/rc.conf
Make sure running bios A09
Console Redirection: VT100/VT220


kern_securelevel_enable="NO"
2. assuming mirrors (or at least disks) created (if not, refer to this), boot to disk 1 of 5.4
portmap_enable="NO"
skip kernel config (enter)
sendmail_enable="NO"
custom install
usbd_enable="YES"


xntpd_enable="YES"
partition ->
nfs_client_enable="YES"
move cursor to amrd0, hit space (takes you to partition map screen)
nfs_reserved_port_only="YES"
a for entire disk
inetd_enable="YES"
q to quit and save
inetd_flags="-wW -a 10.1.4.102"
devfs_system_ruleset="devfsrules_show_all"


#ifconfig_xl0="inet 10.1.4.102 netmask 255.255.255.0"
NOTE
#ifconfig_fxp0="inet 69.55.228.53 netmask 255.255.255.0"
For gvin probs made 2 slices, 1st 8960M
#ifconfig_fxp0_alias0="inet 69.55.2xx.xx netmask 255.255.255.0"
Freebsd boot mgr


static_routes="t1 office"
standard mbr (no boot manager)
route_t1="-net 10.1.5 10.1.4.2"
space to unselect amrd0
route_office="-net 10.1.6 10.1.4.2"
cursor over amrd1
gvinum_enable="YES"
space
fsck_y_enable="YES"
a for entire disk
background_fsck="NO"
q to quit and save
rc_mfi_raid_tty_log="YES"
none (leave untouched)


18. make sure sysctls are set and preserved after reboot
cursor over amrd0
echo "kern.consmute=0\
space
kern.ipc.shm_use_phys=1\
(takes you into part. Screen again) q to exit
kern.ipc.shmall=131070\
none
kern.ipc.shmmax=134217728\
Make sure both are checked and tab to ok
net.inet.tcp.syncookies=0\
kern.maxfiles=32768\
kern.fallback_elf_brand=3\
kern.maxprocperuid=4000\
security.jail.sysvipc_allowed=1\
security.jail.allow_raw_sockets=1\
security.jail.socket_unixiproute_only=1\
security.jail.chflags_allowed=0\
dev.amr.0.allow_volume_configure=1\
compat.linux.osrelease=2.6.12\
vm.pmap.shpgperproc=300" >> /etc/sysctl.conf


19. mount procfs
Label ->
echo "proc                    /proc          procfs  rw              0      0" >> /etc/fstab
Make sure mfid0 is highlighted


For Dell 2950/2450:
/ 256M
echo "linprocfs              /usr/compat/linux/proc linprocfs rw    0      0" >> /etc/fstab
swap 2G (for 2950 make it 4G)
/var 256M (4.6G)
/tmp 256M
/usr 4G
/mnt/data1 remaining space (no need to newfs)


For Dell 2950:
Make sure to toggle S for soft updates on all (should look like UFS2+S Y under the Newfs column)
echo "linsysfs                /usr/compat/linux/sys linsysfs rw      0      0" >> /etc/fstab


mkdir -p /usr/compat/linux/proc
move cursor to mfid1
mkdir -p /usr/compat/linux/sys
swap 8G
/mnt/data2 remaining space (no need to newfs)


19. enable noatime option
q to save and exit
NOT APPLICABLE IF RUNNING GVINUM
data1 and data2 should look something like:
/dev/amrd0s1g          /mnt/data1      ufs    rw,noatime      2      2


20. reboot. Confirm new kernel is loaded
distributions ->
developer
yes to install ports
exit


uname -a
media ->
cd


Check devfs rules:
commit ->
devfs rule showsets
yes
devfs rule -s 3 show
(2450: 14mins, supermicro: 12mins)


21. update ports:
yes to "visit general config" ->
cd /usr/ports
Set root pwd
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_7_2\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup


cvsup sup; mail -s 'cvs sup ports done' dave.boodman@vtext.com < /dev/null
Add user ‘user’ member group is wheel, set password


(2450: 18mins, supermicro: 19mins; 2950: 24mins)
Set tz


22. Install raid mgmt tool
Networking->interfaces->Fxp0 ->
No IPV6
dhcp=no
Set hostname & domain
Enable sshd


For LSI based cards:
exit...
exit install ->
yes


install linux_base:
take the cd out and let the machine reboot
cd /usr/ports/emulators/linux_base-fc4
make install clean
(2450: 7min, supermicro: 3mins, 2950: 14mins)


(for LSI)
3. double check the date/time
cd /usr/ports/sysutils/linux-megamgr
make install clean


cd /usr/ports/sysutils/megarc
4. edit /etc/make.conf (only add the console speed line if this is a supermicro capable of outputting BIOS redirect at that speed – 2450’s can only do 9600)
make install clean
echo "WITHOUT_X11=yes \
KERNCONF=jail2 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf


(for Perc5/i, 6/i)
cd /usr/ports/sysutils/linux-megacli2
make install clean


Test:
rehash; megacli ldinfo lall a0


For adaptec:
5. add settings to /boot/loader.conf and /boot.config
On jail18:
scp /compat/linux/usr/sbin/aaccli user@10.1.4.102:~


mv ~user/aaccli /compat/linux/usr/sbin/aaccli
echo "-Dh" >> /boot.config


test out;
echo 'console="comconsole,vidconsole" \
/compat/linux/usr/sbin/aaccli
boot_multicons="YES" \
boot_serial="YES" \
mfi_linux_load="YES" \
comconsole_speed="115200"' >> /boot/loader.conf


For 3w9690:
(leave out the speed and mfi lines for 2450s)
scp backup2:/d4/build/3ware/tw_cli-freebsd-x86_64-9.5.0.1.tgz /usr/local/sbin
cd /usr/local/sbin xzvf tw_cli-freebsd-x86_64-9.5.0.1.tgz
rm tw_cli-freebsd-x86_64-9.5.0.1.tgz


test out;
6. turn off all ttyv's except 0 and 1 in /etc/ttys
./tw_cli /c0 show allstatus
also turn on ttyd0, change type to vt100:
vi /etc/ttys


23. install rsync from ports
ttyv2  "/usr/libexec/getty Pc"        cons25  off secure
cd /usr/ports/net/rsync
ttyv3  "/usr/libexec/getty Pc"        cons25  off secure
make install clean
ttyv4  "/usr/libexec/getty Pc"        cons25  off secure
ttyv5  "/usr/libexec/getty Pc"        cons25  off secure
ttyv6  "/usr/libexec/getty Pc"        cons25  off secure
ttyv7  "/usr/libexec/getty Pc"        cons25  off secure


choose default options
ttyd0  "/usr/libexec/getty std.9600"  vt100  on secure


25. configure inetd to respond to mrtg load queries
kill -1 1
echo "load    stream  tcp    nowait  user    /usr/local/jail/bin/load.pl  load.pl" >> /etc/inetd.conf
 
on console server:
vi /etc/remote
(rename port to jail18 depending on where and which digi plugged into)
test serial console


echo "load            12384/tcp" >> /etc/services
7. populate hosts
echo "10.1.4.3 backup2" >> /etc/hosts
echo "10.1.4.8 backup1" >> /etc/hosts


26. install bb client
adduser
Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username  : bb
Password  : <random>
Full Name  : bb
Uid        : 1984
Class      :
Groups    : bb
Home      : /home/bb
Shell      : /bin/sh
Locked    : no
OK? (yes/no): yes


cd /usr/home/bb
8. put key in authorized_keys on backup2
scp backup2:/mnt/data4/build/bb/bb-freebsd_linuxcompat.tgz .
cd
tar xzf bb-freebsd_linuxcompat.tgz
ssh-keygen -t dsa -b 1024
(default location, leave password blank)


edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'
10.1.4.108 jail8.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts


vi /home/bb/bbc1.9e-btf/ext/openfiles
confirm that you can ssh to backup2 and backup1 without getting a login prompt
MACHINE="jail8,johncompanies,com"      # HAS TO BE IN A,B,C FORM


cd /usr/home/bb/bbc1.9e-btf/etc
ssh backup2 hostname
./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf
./runbb.sh start
more BBOUT
(look for errors)
exit


echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
ssh backup1 hostname
chmod +x /usr/local/etc/rc.d/bb.sh


9. create & populate binaries/scripts dirs
mkdir -p /usr/local/jail/bin
mkdir -p /usr/local/jail/rc.d
mkdir -p /usr/local/jail/template/
mkdir /mnt/data1
mkdir /mnt/data2
scp backup2:"/mnt/data4/bin/freebsd7.x/*" /usr/local/jail/bin
cd /usr/local/jail/rc.d/
touch quad1
touch quad2
touch quad3
touch quad4
touch safe1
touch safe2
touch safe3
touch safe4
chmod +x *
cd /usr/local/jail/bin
ln -s /usr/local/jail/rc.d/quad1 quad1
ln -s /usr/local/jail/rc.d/quad2 quad2
ln -s /usr/local/jail/rc.d/quad3 quad3
ln -s /usr/local/jail/rc.d/quad4 quad4
ln -s /usr/local/jail/rc.d/safe1 safe1
ln -s /usr/local/jail/rc.d/safe2 safe2
ln -s /usr/local/jail/rc.d/safe3 safe3
ln -s /usr/local/jail/rc.d/safe4 safe4
ln -s /usr/local/jail/bin/jailmake_geom jailmake
ln -s /usr/local/jail/bin/js_geom js
ln -s /usr/local/jail/bin/canceljail_geom canceljail
ln -s /usr/local/jail/bin/jailmakeempty_geom jailmakeempty


NOTE: to get bb working on amd, had to copy over bin dir from linux dist
ln -s /usr/local/jail/bin/jailmake_md jailmake
 
ln -s /usr/local/jail/bin/js_md js
27. configure load mrtg, on mail
ln -s /usr/local/jail/bin/canceljail_md canceljail
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg
ln -s /usr/local/jail/bin/jailmakeempty_md jailmakeempty
(add new entry to file following existing format)
ln -s /usr/local/jail/bin/postboot_md postboot


28. configure bb on mail:
rehash
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
10.1.4.102 jail2.johncompanies.com # ssh


su bb
10. edit root's path and login script:
cd
vi /root/.cshrc
bbsrc/bb/runbb.sh restart ; exit


29. remove reserve space, enable softupdates (probably already set, so not necessary)
Change alias entries (add G):
NOT APPLICABLE IF USING GVINUM
alias la        ls -aG
cd
alias lf        ls -FAG
umount /mnt/data1
alias ll        ls -lAG
umount /mnt/data2
alias ls        ls -AG
tunefs -m 0 /mnt/data1
alias mbm      mb mount
tunefs -m 0 /mnt/data2
alias mbu      mb umount
mount -a


30. configure ntp
echo "server 10.1.4.5" > /etc/ntp.conf


/usr/sbin/ntpd -p /var/run/ntpd.pid
add to path:
sleep 2; ntpq -p
/usr/local/jail/bin
(confirm it’s able to reach our time server)
(if adaptec card installed, also add /compat/linux/usr/sbin)


32. fwd and reverse lookups on ns1c
and alter the prompt, set the following:
vr johncompanies.com
set prompt = "`/bin/hostname -s` %/# "
vi internal.johncompanies.com
rndc reload johncompanies.com IN private
(edit the PTR too)


at the bottom of the file add:
set sshtty=`who am i|awk '{print $2}'`
/usr/sbin/rtprio 3 -`psj | grep $sshtty | awk '{print $2}'`


set shortty=`who am i | awk '{print $2}' | sed -E 's/.*(..)$/\1/'`
foreach x (`psj | grep sh | grep $shortty | awk '{print $2}'`)
/usr/sbin/rtprio 2 -$x
end


35. create the jail template
To load the new file:
source /root/.cshrc


11. install cvsup
cd /usr/ports/net/cvsup-without-gui
make install clean; rehash; mail -s 'cvs installed' dave.boodman@vtext.com < /dev/null


touch /mnt/data1/jail
(stay close for gettext options, 2450: 27mins, supermicro: 17mins, 2950: 22mins)
mdconfig -a -t vnode -s 1g -f /mnt/data1/jail -u 0
bsdlabel -rw md0
newfs4x -i 4096 /dev/md0c
mkdir /mnt/data1/jail-DIR
mount /dev/md0c /mnt/data1/jail-DIR


cd /usr/ports/sysutils/jailutils
12. get latest sources for this release:
make install clean
cd /usr/src
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_7_1\
*default delete use-rel-suffix\
*default compress\
src-all" > sup


cd /usr/src
-OR-
make world DESTDIR=/mnt/data1/jail-DIR; pagedave
(2450: 2:28mins, supermicro: 55mins, 2950: 1h )
cd etc
make distribution DESTDIR=/mnt/data1/jail-DIR
mount -t devfs devfs /mnt/data1/jail-DIR/dev
devfs -m /mnt/data1/jail-DIR/dev rule -s 3 applyset
cd /mnt/data1/jail-DIR
ln -sf dev/null kernel
cp /usr/local/sbin/jkill /mnt/data1/jail-DIR/sbin


jail /mnt/data1/jail-DIR testhostname 192.168.11.100 /bin/sh
echo "*default host=cvsup4.freebsd.org\
csh
*default base=/usr\
touch /etc/fstab
*default prefix=/usr\
echo 'network_interfaces=""\
*default release=cvs tag=RELENG_7\
hostname="newsystem"\
*default delete use-rel-suffix\
kern_securelevel_enable="NO"\
*default compress\
sendmail_enable="YES"\
src-all" > sup
sshd_enable="YES"' > /etc/rc.conf


echo "nameserver 69.55.225.225\
(stable)
nameserver 69.55.230.3" >> /etc/resolv.conf


vi /etc/crontab
remove the adjkerntz lines
comment out periodic’s and put this line above them:
# IF YOU UNCOMMENT THESE, PLEASE ADJUST THEIR RUN TIME


rm -rf /etc/periodic/daily/400.status-disks
cvsup sup ; mail -s 'cvs sup done' dave.boodman@vtext.com < /dev/null


check /tmp for crap
(2450, ~12mins, supermicro, 27mins, 2950: 7mins)


vi /etc/periodic/security/100.chksetuid
replace: MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`
with: MP='/' (use single quotes)


mkdir -p /usr/compat/linux/dev
13. configure new kernel.
for i386:
cd /usr/src/sys/i386/conf
scp backup2:/mnt/data4/build/freebsd/kern_config-7.0-PAE ./mx2


adduser (Add account for user, make sure in wheel group)
-or-


Username: user
for amd64:
Full name: user
cd /usr/src/sys/amd64/conf
Uid (Leave empty for default):
scp backup2:/mnt/data4/build/freebsd/kern_config-7.1-amd64 ./jail2
Login group [user]:
Login group is user. Invite user into other groups? []: wheel
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/user]:
Home directory permissions (Leave empty for default):
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: y
Lock out the account after creation? [no]:
Username  : user
Password  : <random>
Full Name  : user
Uid        : 1001
Class      :
Groups    : user
Home      : /home/user
Home Mode  :
Shell      : /bin/sh
Locked    : no
OK? (yes/no): y
adduser: INFO: Successfully added (user) to the user database.
adduser: INFO: Password for (user) is: 901gmYjO
Add another user? (yes/no): n
Goodbye!


vi /usr/home/user/.profile (and add to the file):
-------
TERM=vt100;    export TERM


tzsetup
edit the kernel config and change ident to be the name of the jail:
vi jail2
ident          jail2


newaliases
edit /sys/conf/newvers.sh to add –jc2 to the end of the BRANCH string (RELEASE-jc2)
vi /sys/conf/newvers.sh


rm /sbin/halt /sbin/reboot
notes: http://www.FreeBSD.org/doc/en_US.ISO8859-1/books/handbook/kernelconfig-config.html
ln /sbin/jkill /sbin/halt
ln /sbin/jkill /sbin/reboot


#cd /dev
14. bring over patches from backup2
#rm console
#ln -s null console


vi /etc/syslog.conf (comment out console and move to /var/log/messages):
The various patches are in /mnt/data4/build/freebsd/patches on backup2. There are dirs for each version. Not all dirs are populated, but patches for later versions work on older ones unless there is a new patch in the older dir.
#*.err;kern.warning;auth.notice;mail.crit              /dev/console *.err;kern.warning;auth.notice;mail.crit                /var/log/messages


exit
cd /usr/src
exit
scp backup2:"/mnt/data4/build/freebsd/patches/7.x/*" .


cd libexec
Apply patches:
chflags noschg ld-elf32.so.1
patch -l < jls-patch
chflags noschg ld-elf.so.1


mv ld-elf32.so.1 ld-elf32.so.1-orig
ln ld-elf.so.1 ld-elf32.so.1


chflags schg ld-elf.so.1
15. build, install kernel and world
chflags schg ld-elf32.so.1


cd /boot


cd /usr/ports
mv kernel kernel.GENERIC
make -DNOCLEANDEPENDS clean
cd kernel.GENERIC
(2450: 15mins , supermicro: 29mins, 2950: 18mins)
mkdir hold
rm -fr /usr/ports/distfiles/*
mv mfi_linux.ko hold/
cp -r /usr/ports /mnt/data1/jail-DIR/usr (2450: 2:00 mins , supermicro: 15mins, 2950: 3mins)
mv linux.ko hold/
 
mv linprocfs.ko hold/
rm /mnt/data1/jail-DIR/root/.history
mv linsysfs.ko hold/
mv geom_vinum.ko hold/
mv geom_concat.ko hold/
rm *.ko
rm *.symbols
mv hold/* .
rmdir hold/


cd  
cd /usr/src
umount /mnt/data1/jail-DIR/dev
make buildkernel installkernel
dump -0a -f /usr/local/jail/template/72template /dev/md0c
umount /dev/md0c
rmdir /mnt/data1/jail-DIR


make buildworld ; mail -s 'kernel build done' dave.boodman@vtext.com < /dev/null
(2450: 1:56min, supermicro: 59mins, 2950: 38mins)
make installworld
(2450: 3min, supermicro: 1min, 2950: :34)
mergemaster -i
delete /var/tmp/temproot
delete bsnmpd
delete temporary ./etc/hosts
delete temporary ./etc/motd
delete /var/tmp/temproot


36. setup backups
cd /sys/modules/geom/geom_vinum
echo '#\!/bin/sh\
make
backupdir=/data/jail2\
make install
server=backup1\
\
## ENTRY /etc\
## ENTRY /usr/local/etc\
## ENTRY /usr/local/jail\
## ENTRY /root/logs' > /usr/local/jail/bin/backup.config


on backup1:
16. populate devfs ruleset
setup backup dirs:
scp backup2:/mnt/data4/build/freebsd/devfs.rules /etc
ssh backup1 mkdir -p /data/jail2/0


add the system to
17. populate /etc/rc.conf with IPs and NFS settings
vi /usr/local/sbin/snapshot_rotate
vi /etc/rc.conf


on mail:
kern_securelevel_enable="NO"
vi /usr/local/www/mgmt/cgi/backupgraph.pl
portmap_enable="NO"
(add hostname)
sendmail_enable="NO"
usbd_enable="YES"


Edit /usr/local/jail/bin/backup.xxx to use the right drives and copy to /usr/local/jail/bin/backup
xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.4.102"
devfs_system_ruleset="devfsrules_show_all"


37. mkdir /root/logs
#ifconfig_xl0="inet 10.1.4.102 netmask 255.255.255.0"
#ifconfig_fxp0="inet 69.55.228.53 netmask 255.255.255.0"
#ifconfig_fxp0_alias0="inet 69.55.2xx.xx netmask 255.255.255.0"


38. edit sshd_config for security
static_routes="t1 office"
vi /etc/ssh/sshd_config
route_t1="-net 10.1.5 10.1.4.2"
ListenAddress 69.55.228.53
route_office="-net 10.1.6 10.1.4.2"
ListenAddress 10.1.4.102
gvinum_enable="YES"
fsck_y_enable="YES"
background_fsck="NO"
rc_mfi_raid_tty_log="YES"


kill -1 `cat /var/run/sshd.pid`
18. make sure sysctls are set and preserved after reboot
echo "kern.consmute=0\
kern.ipc.shm_use_phys=1\
kern.ipc.shmall=131070\
kern.ipc.shmmax=134217728\
net.inet.tcp.syncookies=0\
kern.maxfiles=32768\
kern.fallback_elf_brand=3\
kern.maxprocperuid=4000\
security.jail.sysvipc_allowed=1\
security.jail.allow_raw_sockets=1\
security.jail.socket_unixiproute_only=1\
security.jail.chflags_allowed=0\
dev.amr.0.allow_volume_configure=1\
compat.linux.osrelease=2.6.12" >> /etc/sysctl.conf


39. add crontab entries
19. mount procfs
crontab -e
echo "proc                    /proc          procfs  rw              0       0" >> /etc/fstab
5 0 * * * /usr/local/jail/bin/backup.md
1 0 1 * * /usr/local/jail/bin/ipfwreset
0 18 * * * /usr/local/jail/bin/ipfwbackup
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /usr/local/jail/bin/trafstats


On 2950 with Perc5/i also add:
For Dell 2950/2450:
*/5 * * * * /usr/local/jail/bin/perc5iraidchk
echo "linprocfs              /usr/compat/linux/proc linprocfs rw    0      0" >> /etc/fstab


40. Reboot notify script
For Dell 2950:
ln -s /usr/local/jail/bin/notify.sh /usr/local/etc/rc.d/notify.sh
echo "linsysfs                /usr/compat/linux/sys linsysfs rw      0      0" >> /etc/fstab


41. add to management db (on mail and devweb) jc.ref_machines and jc.ref_templates
mkdir -p /usr/compat/linux/proc
mkdir -p /usr/compat/linux/sys


uname -r
20. reboot. Confirm new kernel is loaded
6.2-RELEASE-jc1


insert into ref_machines values (null,'mx2','mx2.johncompanies.com',0,'m');
uname -a
select machine_id from ref_machines where host='mx2';
+------------+
| machine_id |
+------------+
|        35 |
+------------+
insert into ref_templates values ('',' 7.1-RELEASE-jc2',44,'FreeBSD 7.1',0);


42. add to server/cabinet map. On mail:
Check devfs rules:
vi /usr/local/www/mgmt/html/cabinetmap.html
devfs rule showsets
devfs rule -s 3 show


43. add an outside blocking rule to the firewall, so this machine can only be reached from inside the firewall. Follow example already in firewall jail17 is:
21. update ports:
cd /usr/ports
00117 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.228.200
echo "*default host=cvsup4.FreeBSD.org\
00117 deny ip from any to 69.55.228.200
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_7_1\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup


jail19 would be 00119...
cvsup sup; mail -s 'cvs sup ports done' dave.boodman@vtext.com < /dev/null
ipfw add 00107 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.230.108
ipfw add 00107 deny ip from any to 69.55.220.108


44. select customers for probe map
(2450: 18mins, supermicro: 19mins; 2950: 24mins)


47. patch jail against starting jails with rtprio
22. Install raid mgmt tool


mv /usr/sbin/jail /usr/sbin/jail_
For LSI based cards:
echo '#\!/bin/sh\
/usr/sbin/rtprio -t /usr/sbin/jail_ $*' > /usr/sbin/jail
chmod +x /usr/sbin/jail


48. make sure mail works
install linux_base:
If there are map errors:
cd /usr/ports/emulators/linux_base-fc4
cd /etc/mail; make maps
make install clean
(2450: 7min, supermicro: 3mins, 2950: 14mins)
 
(for LSI)
cd /usr/ports/sysutils/linux-megamgr
make install clean
 
cd /usr/ports/sysutils/megarc
make install clean
 
(for Perc5/i, 6/i)
cd /usr/ports/sysutils/linux-megacli2
make install clean


Test:
rehash; megacli ldinfo lall a0


50. ntp doesn’t seem to start from rc so,
For adaptec:
On jail18:
scp /compat/linux/usr/sbin/aaccli user@10.1.4.102:~


echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh
mv ~user/aaccli /compat/linux/usr/sbin/aaccli
chmod 0700 /usr/local/etc/rc.d/ntp.sh


51. recover space on /usr
test out;
/compat/linux/usr/sbin/aaccli


rm -fr /usr/obj
For 3w9690:
scp backup2:/d4/build/3ware/tw_cli-freebsd-x86_64-9.5.0.1.tgz /usr/local/sbin
cd /usr/local/sbin xzvf tw_cli-freebsd-x86_64-9.5.0.1.tgz
rm tw_cli-freebsd-x86_64-9.5.0.1.tgz


52. setup fuse
test out;
./tw_cli /c0 show allstatus


cd /usr/ports/sysutils/fusefs-kmod/
23. install rsync from ports
make install
cd /usr/ports/net/rsync
make install clean


vi /etc/rc.conf
choose default options
fusefs_enable="YES"


sysctl vfs.usermount=1
25. configure inetd to respond to mrtg load queries
echo "load    stream  tcp    nowait  user    /usr/local/jail/bin/load.pl  load.pl" >> /etc/inetd.conf


cd /usr/ports/sysutils/fusefs-sshfs
echo "load            12384/tcp" >> /etc/services
make install


sshfs 1005@usw-s009.rsync.net: /mnt/data1/69.55.234.68-col00001-DIR/mnt
26. install bb client
adduser
Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username  : bb
Password  : <random>
Full Name  : bb
Uid        : 1984
Class      :
Groups    : bb
Home      : /home/bb
Shell      : /bin/sh
Locked    : no
OK? (yes/no): yes
 
cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd_linuxcompat.tgz .
tar xzf bb-freebsd_linuxcompat.tgz
 
edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.4.102 jail2.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
 
vi /home/bb/bbc1.9e-btf/ext/openfiles
MACHINE="jail2,johncompanies,com"      # HAS TO BE IN A,B,C FORM
 
cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf
./runbb.sh start
more BBOUT
(look for errors)
exit
 
echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh
 
 
NOTE: to get bb working on amd, had to copy over bin dir from linux dist
 
27. configure load mrtg, on mail
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg
(add new entry to file following existing format)
 
28. configure bb on mail:
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
10.1.4.102 jail2.johncompanies.com # ssh
 
su bb
cd
bbsrc/bb/runbb.sh restart ; exit
 
29. remove reserve space, enable softupdates (probably already set, so not necessary)
NOT APPLICABLE IF USING GVINUM
cd
umount /mnt/data1
umount /mnt/data2
tunefs -m 0 /mnt/data1
tunefs -m 0 /mnt/data2
mount -a
 
30. configure ntp
echo "server 10.1.4.5" > /etc/ntp.conf
 
/usr/sbin/ntpd -p /var/run/ntpd.pid
sleep 2; ntpq -p
(confirm it’s able to reach our time server)
 
32. fwd and reverse lookups on ns1c
vr johncompanies.com
vi internal.johncompanies.com
rndc reload johncompanies.com IN private
(edit the PTR too)
 
 
33. create gvinum volumes
 
Make a g partition:
 
2950:
bsdlabel -e /dev/mfid0s1
bsdlabel -e /dev/mfid1s1
bsdlabel -e /dev/mfid2s1
 
given:
# /dev/aacd0s1:
8 partitions:
#        size  offset    fstype  [fsize bsize bps/cpg]
  a:  262144        0    4.2BSD    2048 16384 16392
  b:  4194304  262144      swap
  c: 143363997        0    unused        0    0        # "raw" part, don't edit
  d:  524288  4456448    4.2BSD    2048 16384 32776
  e:  524288  4980736    4.2BSD    2048 16384 32776
  f:  6291456  5505024    4.2BSD    2048 16384 28552
 
new offset = 6291456 + 5505024 = 11796480
new size is size for 'c' partition minus the new start from above
143363997 - 11796480 = 131567517
So:
g: 131567517 11796480 unused 0 0
 
bsdlabel -e /dev/amrd1s1
change d to g
 
For a 146G drive (after OS and 4G swap), we can fit 127 1G volumes so:
 
echo 'drive data1 device /dev/mfid0s1g' > /tmp/cgv
 
echo '#\!/bin/sh\
i="1"\
while [ $i -le 127 ]\
do\
  echo "volume v$i" >> /tmp/cgv;\
  echo 'plex org concat' >> /tmp/cgv;\
  echo 'sd length 1g drive data1' >> /tmp/cgv;\
i=`expr $i + 1`\
done' > /tmp/mkcgv
sh /tmp/mkcgv
gvinum create /tmp/cgv
 
volume 1
plex org concat
sd length 3g drive data1
volume 2
plex org concat
sd length 5g drive data1
volume 3
plex org concat
sd length 8g drive data1
volume 4
plex org concat
sd length 6g drive data1
volume 5
plex org concat
sd length 10g drive data1
 
for f in 1 2 3 4 5; do bsdlabel -rw /dev/gvinum/$f; done
for f in 1 2 3 4 5; do newfs /dev/gvinum/${f}a; done
for f in 1 2 3 4 5; do mkdir /mnt/data1/$f; mount /dev/gvinum/${f}a /mnt/data1/$f;done
for f in 1 2 3 4 5; do dd if=/dev/zero of=/mnt/data1/$f/this_is_$f bs=1m count=$f; done
for f in 1 2 3 4 5; do ll -h /mnt/data1/$f/; done
 
 
For a 300G drive (after OS and 4G swap), we can fit 273 1G volumes so:
 
echo 'drive data2 device /dev/mfid1s1g' > /tmp/cgv
 
echo '#\!/bin/sh\
i="128"\
while [ $i -le 401 ]\
do\
  echo "volume v$i" >> /tmp/cgv;\
  echo 'plex org concat' >> /tmp/cgv;\
  echo 'sd length 1g drive data2' >> /tmp/cgv;\
i=`expr $i + 1`\
done' > /tmp/mkcgv
sh /tmp/mkcgv
gvinum create /tmp/cgv
 
 
For a 300G drive (after OS and 4G swap), we can fit 273 1G volumes so:
 
echo 'drive data3 device /dev/mfid2s1g' > /tmp/cgv
 
echo '#\!/bin/sh\
i="402"\
while [ $i -le 675 ]\
do\
  echo "volume v$i" >> /tmp/cgv;\
  echo 'plex org concat' >> /tmp/cgv;\
  echo 'sd length 1g drive data3' >> /tmp/cgv;\
i=`expr $i + 1`\
done' > /tmp/mkcgv
sh /tmp/mkcgv
gvinum create /tmp/cgv
 
Sometimes there’s 2040m leftover so create one more smaller vol:
echo 'volume v64\
plex org concat\
sd length 2040m drive data1' >> /tmp/cgv
 
To delete:
 
echo '#\!/bin/sh\
i="1"\
while [ $i -le 127 ]\
do\
  echo "gvinum rm -r v$i" >> /tmp/dgv;\
i=`expr $i + 1`\
done' > /tmp/mkdgv
sh /tmp/mkdgv
sh /tmp/dgv
 
dd if=/dev/zero of=/dev/sdb seek=285474816 bs=512
dd if=/dev/zero of=/dev/mfid0s2 bs=1m count=1000 (do this to all drives)
 
35. create the jail template
 
 
touch /mnt/data1/jail
mdconfig -a -t vnode -s 1g -f /mnt/data1/jail -u 0
bsdlabel -rw md0
newfs4x /dev/md0c
mkdir /mnt/data1/jail-DIR
mount /dev/md0c /mnt/data1/jail-DIR
 
cd /usr/ports/sysutils/jailutils
make install clean
 
cd /usr/src
make world DESTDIR=/mnt/data1/jail-DIR
(2450: 2:28mins, supermicro: 55mins, 2950: 1h )
cd etc
make distribution DESTDIR=/mnt/data1/jail-DIR
mount -t devfs devfs /mnt/data1/jail-DIR/dev
devfs -m /mnt/data1/jail-DIR/dev rule -s 3 applyset
cd /mnt/data1/jail-DIR
ln -sf dev/null kernel
cp /usr/local/sbin/jkill /mnt/data1/jail-DIR/sbin
 
jail /mnt/data1/jail-DIR testhostname 192.168.11.100 /bin/sh
csh
touch /etc/fstab
echo 'network_interfaces=""\
hostname="newsystem"\
kern_securelevel_enable="NO"\
sendmail_enable="YES"\
sshd_enable="YES"' > /etc/rc.conf
 
echo "nameserver 69.55.225.225\
nameserver 69.55.230.3" >> /etc/resolv.conf
 
vi /etc/crontab
remove the adjkerntz lines
comment out periodic’s and put this line above them:
# IF YOU UNCOMMENT THESE, PLEASE ADJUST THEIR RUN TIME
 
rm -rf /etc/periodic/daily/400.status-disks
 
check /tmp for crap
 
vi /etc/periodic/security/100.chksetuid
replace: MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`
with: MP='/' (use single quotes)
 
mkdir -p /usr/compat/linux/dev
 
adduser (Add account for user, make sure in wheel group)
 
user/root passwd: 8ico2987
 
Set root password
 
vi /usr/home/user/.profile (and add to the file):
TERM=vt100;    export TERM
 
tzsetup
 
newaliases
 
rm /sbin/halt /sbin/reboot
ln /sbin/jkill /sbin/halt
ln /sbin/jkill /sbin/reboot
 
#cd /dev
#rm console
#ln -s null console
 
vi /etc/syslog.conf (comment out console and move to /var/log/messages):
#*.err;kern.warning;auth.notice;mail.crit              /dev/console *.err;kern.warning;auth.notice;mail.crit                /var/log/messages
 
exit
exit
 
cd libexec
mv ld-elf32.so.1 ld-elf32.so.1-orig
ln ld-elf.so.1 ld-elf32.so.1
 
may have to run:
chflags noschg ld-elf32.so.1
chflags noschg ld-elf.so.1
then
chflags schg ld-elf.so.1
chflags schg ld-elf32.so.1
 
 
cd /usr/ports
make -DNOCLEANDEPENDS clean
(2450: 15mins , supermicro: 29mins, 2950: 18mins)
rm -fr /usr/ports/distfiles/*
cp -r /usr/ports /mnt/data1/jail-DIR/usr (2450: 2:00 mins , supermicro: 15mins, 2950: 3mins)
 
rm /mnt/data1/jail-DIR/root/.history
 
cd
dump -0a -f /usr/local/jail/template/71template /dev/md0c
umount /mnt/data1/jail-DIR/dev
umount /dev/md0c
rmdir /mnt/data1/jail-DIR
 
 
36. setup backups
echo '#\!/bin/sh\
backupdir=/data/jail2\
server=backup1\
\
## ENTRY /etc\
## ENTRY /usr/local/etc\
## ENTRY /usr/local/jail\
## ENTRY /root/logs' > /usr/local/jail/bin/backup.config
 
on backup1:
setup backup dirs:
ssh backup1 mkdir -p /data/jail2/0
 
add the system to
vi /usr/local/sbin/snapshot_rotate
 
on mail:
vi /usr/local/www/mgmt/cgi/backupgraph.pl
(add hostname)
 
Edit /usr/local/jail/bin/backup.xxx to use the right drives and copy to /usr/local/jail/bin/backup
 
37. mkdir /root/logs
 
38. edit sshd_config for security
vi /etc/ssh/sshd_config
ListenAddress 69.55.228.53
ListenAddress 10.1.4.102
 
kill -1 `cat /var/run/sshd.pid`
 
39. add crontab entries
crontab -e
5 0 * * * /usr/local/jail/bin/backup.md
1 0 1 * * /usr/local/jail/bin/ipfwreset
0 18 * * * /usr/local/jail/bin/ipfwbackup
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /usr/local/jail/bin/trafstats
 
On 2950 with Perc5/i also add:
*/5 * * * * /usr/local/jail/bin/perc5iraidchk
 
40. Reboot notify script
ln -s /usr/local/jail/bin/notify.sh /usr/local/etc/rc.d/notify.sh
 
41. add to management db (on mail and devweb) jc.ref_machines and jc.ref_templates
 
uname -r
6.2-RELEASE-jc1
 
insert into ref_machines values (null,'mx2','mx2.johncompanies.com',0,'m');
select machine_id from ref_machines where host='mx2';
+------------+
| machine_id |
+------------+
|        35 |
+------------+
insert into ref_templates values ('',' 7.1-RELEASE-jc2',44,'FreeBSD 7.1',0);
 
42. add to server/cabinet map. On mail:
vi /usr/local/www/mgmt/html/cabinetmap.html
 
43. add an outside blocking rule to the firewall, so this machine can only be reached from inside the firewall. Follow example already in firewall jail17 is:
00117 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.228.200
00117 deny ip from any to 69.55.228.200
 
jail19 would be 00119...
ipfw add 00107 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.230.108
ipfw add 00107 deny ip from any to 69.55.220.108
 
44. select customers for probe map
 
46. make gv start on boot
 
scp backup2:/mnt/data4/build/freebsd/gvinum /etc/rc.d/gvinum
 
If they start stale:
 
echo '#\!/bin/sh\
i="1"\
while [ $i -le 127 ]\
do\
        gvinum start v$i\
        i=`expr $i + 1`\
done' > /tmp/startgv
 
sh /tmp/startgv
 
 
47. patch jail against starting jails with rtprio
 
mv /usr/sbin/jail /usr/sbin/jail_
echo '#\!/bin/sh\
/usr/sbin/rtprio -t /usr/sbin/jail_ $*' > /usr/sbin/jail
chmod +x /usr/sbin/jail
 
48. make sure mail works
If there are map errors:
cd /etc/mail; make maps
 
49. move and pare down generic kernel
mv kernel.old/ kernel.GENERIC
 
/  too full, remove old kernel modules except for:
 
Id Refs Address            Size    Name
1  14 0xffffffff80100000 ac6c08  kernel
2    1 0xffffffff80bc8000 1128    mfi_linux.ko
3    4 0xffffffff80bca000 39138    linux.ko
4    1 0xffffffffb48f8000 358c    linprocfs.ko
5    1 0xffffffffb48fc000 9d3      linsysfs.ko
6    1 0xffffffffb490c000 80ee8    zfs.ko
 
 
50. ntp doesn’t seem to start from rc so,
 
echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh
chmod 0700 /usr/local/etc/rc.d/ntp.sh</pre>
 
= FreeBSD 7.2 =
 
Last updated 2009-06-18
 
<pre>All time extimates below assume disks aren’t scrubbing. Setup instructions below are for LSI card:
 
1. make sure bios is setup for bios console redirect
2950:
Console redirection:
LCD string..
Date to GMT
 
 
2. assuming mirrors (or at least disks) created (if not, refer to this), boot to disk 1 of 7.2
skip kernel config (enter)
custom install
 
partition ->
move cursor to mfid0, hit space (takes you to partition map screen)
a for entire disk
q to quit and save
 
 
standard mbr (no boot manager)
space to unselect mfid0
cursor over mfid1
space
a for entire disk
q to quit and save
none (leave untouched)
 
cursor over mfid0
space
(takes you into part. Screen again) q to exit
none
Make sure both are checked and tab to ok
 
Label ->
Make sure mfid0 is highlighted
 
/ 256M
swap 2G (for 2950 make it 4G)
/var 256M (4.6G)
/tmp 256M
/usr 4G
/mnt/data1 remaining space (no need to newfs)
 
Make sure to toggle S for soft updates on all (should look like UFS2+S Y under the Newfs column)
 
move cursor to mfid1
swap 8G (or 4G if there’s a 3rd drive)
/mnt/data2 remaining space (no need to newfs)
 
q to save and exit
 
distributions ->
developer
yes to install ports
exit
 
media ->
cd
 
commit ->
yes
(2450: 14mins, supermicro: 12mins)
 
yes to "visit general config" ->
Set root pwd
 
Add user ‘user’ member group is wheel, set password
 
Set tz
 
Networking->interfaces->Fxp0 ->
No IPV6
dhcp=no
Set hostname & domain
Enable sshd
 
exit...
exit install ->
yes
 
take the cd out and let the machine reboot
 
3. double check the date/time
 
4. edit /etc/make.conf
echo "WITHOUT_X11=yes \
KERNCONF=jail8 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf
 
 
5. add settings to /boot/loader.conf and /boot.config
 
echo "-Dh" >> /boot.config
 
echo 'console="comconsole,vidconsole" \
boot_multicons="YES" \
boot_serial="YES" \
mfi_linux_load="YES" \
comconsole_speed="115200"' >> /boot/loader.conf
 
 
6. turn off all ttyv's except 0 and 1 in /etc/ttys
also turn on ttyd0, change type to vt100:
vi /etc/ttys
 
ttyv2  "/usr/libexec/getty Pc"        cons25  off secure
ttyv3  "/usr/libexec/getty Pc"        cons25  off secure
ttyv4  "/usr/libexec/getty Pc"        cons25  off secure
ttyv5  "/usr/libexec/getty Pc"        cons25  off secure
ttyv6  "/usr/libexec/getty Pc"        cons25  off secure
ttyv7  "/usr/libexec/getty Pc"        cons25  off secure
 
ttyd0  "/usr/libexec/getty std.9600"  vt100  on secure
 
kill -1 1
 
on console server:
vi /etc/remote
(rename port to jail18 depending on where and which digi plugged into)
test serial console
 
7. populate hosts
echo "10.1.4.3 backup2" >> /etc/hosts
echo "10.1.4.8 backup1" >> /etc/hosts
 
 
8. put key in authorized_keys on backup2
cd
ssh-keygen -t dsa -b 1024
(default location, leave password blank)
 
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'
 
confirm that you can ssh to backup2 and backup1 without getting a login prompt
 
ssh backup2 hostname
 
ssh backup1 hostname
 
9. create & populate binaries/scripts dirs
mkdir -p /usr/local/jail/bin
mkdir -p /usr/local/jail/rc.d
mkdir -p /usr/local/jail/template/
mkdir /mnt/data1
mkdir /mnt/data2
scp backup2:"/mnt/data4/bin/freebsd7.x/*" /usr/local/jail/bin
cd /usr/local/jail/rc.d/
touch quad1
touch deprecated
chmod +x *
cd /usr/local/jail/bin
ln -s /usr/local/jail/rc.d/quad1 quad1
ln -s /usr/local/jail/bin/jailmake_md jailmake
ln -s /usr/local/jail/bin/js_md js
ln -s /usr/local/jail/bin/canceljail_md canceljail
ln -s /usr/local/jail/bin/jailmakeempty_md jailmakeempty
ln -s /usr/local/jail/bin/postboot_md postboot
 
rehash
 
10. edit root's path and login script:
vi /root/.cshrc
 
Change alias entries (add G):
alias la        ls -aG
alias lf        ls -FAG
alias ll        ls -lAG
alias ls        ls -AG
alias mbm      mb mount
alias mbu      mb umount
 
 
add to path:
/usr/local/jail/bin
(if adaptec card installed, also add /compat/linux/usr/sbin)
 
and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "
 
at the bottom of the file add:
set sshtty=`who am i|awk '{print $2}'`
/usr/sbin/rtprio 3 -`psj | grep $sshtty | awk '{print $2}'`
 
set shortty=`who am i | awk '{print $2}' | sed -E 's/.*(..)$/\1/'`
foreach x (`psj | grep sh | grep $shortty | awk '{print $2}'`)
/usr/sbin/rtprio 2 -$x
end
 
To load the new file:
source /root/.cshrc
 
11. install cvsup
cd /usr/ports/net/cvsup-without-gui
make install clean; rehash; mail -s 'cvs installed' dave.boodman@vtext.com < /dev/null
 
(stay close for gettext options, 2450: 27mins, supermicro: 17mins, 2950: 22mins)
 
12. get latest sources for this release:
cd /usr/src
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_7_2\
*default delete use-rel-suffix\
*default compress\
src-all" > sup
 
-OR-
 
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_7\
*default delete use-rel-suffix\
*default compress\
src-all" > sup
 
(stable)
 
 
cvsup sup ; mail -s 'cvs sup done' dave.boodman@vtext.com < /dev/null
 
(2450, ~12mins, supermicro, 27mins, 2950: 7mins)
 
 
13. configure new kernel.
 
cd /usr/src/sys/amd64/conf
scp backup2:/mnt/data4/build/freebsd/kern_config-7.1-amd64 ./jail8
 
edit the kernel config and change ident to be the name of the jail:
vi jail8
ident          jail8
 
edit /sys/conf/newvers.sh to add –jc2 to the end of the BRANCH string (RELEASE-jc2)
vi /sys/conf/newvers.sh
 
notes: http://www.FreeBSD.org/doc/en_US.ISO8859-1/books/handbook/kernelconfig-config.html
 
14. bring over patches from backup2
 
The various patches are in /mnt/data4/build/freebsd/patches on backup2. There are dirs for each version. Not all dirs are populated, but patches for later versions work on older ones unless there is a new patch in the older dir.
 
cd /usr/src
scp backup2:"/mnt/data4/build/freebsd/patches/7.2/*" .
 
Apply patches:
patch -l < jls-patch
 
 
15. build, install kernel and world
 
cd /boot
 
mv kernel kernel.GENERIC
cd kernel.GENERIC
mkdir hold
mv mfi_linux.ko hold/
mv linux.ko hold/
mv linprocfs.ko hold/
mv linsysfs.ko hold/
mv geom_vinum.ko hold/
mv geom_concat.ko hold/
rm *.ko
rm *.symbols
mv hold/* .
rmdir hold/
 
cd /usr/src
make buildkernel installkernel
 
make buildworld ; mail -s 'buildworld done' dave.boodman@vtext.com < /dev/null
(2450: 1:56min, supermicro: 59mins, 2950: 38mins)
make installworld
(2450: 3min, supermicro: 1min, 2950: :34)
mergemaster -i
delete /var/tmp/temproot
delete bsnmpd
delete temporary ./etc/hosts
delete temporary ./etc/motd
delete /var/tmp/temproot
 
cd /sys/modules/geom/geom_vinum
make
make install
 
16. populate devfs ruleset
scp backup2:/mnt/data4/build/freebsd/devfs.rules /etc
 
17. populate /etc/rc.conf with IPs and NFS settings
vi /etc/rc.conf
 
kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"
 
xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.4.102"
devfs_system_ruleset="devfsrules_show_all"
 
#ifconfig_xl0="inet 10.1.4.102 netmask 255.255.255.0"
#ifconfig_fxp0="inet 69.55.228.53 netmask 255.255.255.0"
#ifconfig_fxp0_alias0="inet 69.55.2xx.xx netmask 255.255.255.0"
 
static_routes="t1 office"
route_t1="-net 10.1.5 10.1.4.2"
route_office="-net 10.1.6 10.1.4.2"
gvinum_enable="YES"
fsck_y_enable="YES"
background_fsck="NO"
rc_mfi_raid_tty_log="YES"
 
18. make sure sysctls are set and preserved after reboot
echo "kern.consmute=0\
kern.ipc.shm_use_phys=1\
kern.ipc.shmall=131070\
kern.ipc.shmmax=134217728\
net.inet.tcp.syncookies=0\
kern.maxfiles=32768\
kern.fallback_elf_brand=3\
kern.maxprocperuid=4000\
security.jail.sysvipc_allowed=1\
security.jail.allow_raw_sockets=1\
security.jail.socket_unixiproute_only=1\
security.jail.chflags_allowed=0\
dev.amr.0.allow_volume_configure=1\
compat.linux.osrelease=2.6.12\
vm.pmap.shpgperproc=300" >> /etc/sysctl.conf
 
19. mount procfs
echo "proc                    /proc          procfs  rw              0      0" >> /etc/fstab
 
For Dell 2950/2450:
echo "linprocfs              /usr/compat/linux/proc linprocfs rw    0      0" >> /etc/fstab
 
For Dell 2950:
echo "linsysfs                /usr/compat/linux/sys linsysfs rw      0      0" >> /etc/fstab
 
mkdir -p /usr/compat/linux/proc
mkdir -p /usr/compat/linux/sys
 
19. enable noatime option
NOT APPLICABLE IF RUNNING GVINUM
data1 and data2 should look something like:
/dev/amrd0s1g          /mnt/data1      ufs    rw,noatime      2      2
 
20. reboot. Confirm new kernel is loaded
 
uname -a
 
Check devfs rules:
devfs rule showsets
devfs rule -s 3 show
 
21. update ports:
cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_7_2\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup
 
cvsup sup; mail -s 'cvs sup ports done' dave.boodman@vtext.com < /dev/null
 
(2450: 18mins, supermicro: 19mins; 2950: 24mins)
 
22. Install raid mgmt tool
 
For LSI based cards:
 
install linux_base:
cd /usr/ports/emulators/linux_base-fc4
make install clean
(2450: 7min, supermicro: 3mins, 2950: 14mins)
 
(for LSI)
cd /usr/ports/sysutils/linux-megamgr
make install clean
 
cd /usr/ports/sysutils/megarc
make install clean
 
(for Perc5/i, 6/i)
cd /usr/ports/sysutils/linux-megacli2
make install clean
 
Test:
rehash; megacli ldinfo lall a0
 
For adaptec:
On jail18:
scp /compat/linux/usr/sbin/aaccli user@10.1.4.102:~
 
mv ~user/aaccli /compat/linux/usr/sbin/aaccli
 
test out;
/compat/linux/usr/sbin/aaccli
 
For 3w9690:
scp backup2:/d4/build/3ware/tw_cli-freebsd-x86_64-9.5.0.1.tgz /usr/local/sbin
cd /usr/local/sbin xzvf tw_cli-freebsd-x86_64-9.5.0.1.tgz
rm tw_cli-freebsd-x86_64-9.5.0.1.tgz
 
test out;
./tw_cli /c0 show allstatus
 
23. install rsync from ports
cd /usr/ports/net/rsync
make install clean
 
choose default options
 
25. configure inetd to respond to mrtg load queries
echo "load    stream  tcp    nowait  user    /usr/local/jail/bin/load.pl  load.pl" >> /etc/inetd.conf
 
echo "load            12384/tcp" >> /etc/services
 
26. install bb client
adduser
Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username  : bb
Password  : <random>
Full Name  : bb
Uid        : 1984
Class      :
Groups    : bb
Home      : /home/bb
Shell      : /bin/sh
Locked    : no
OK? (yes/no): yes
 
cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd_linuxcompat.tgz .
tar xzf bb-freebsd_linuxcompat.tgz
 
edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.4.108 jail8.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
 
vi /home/bb/bbc1.9e-btf/ext/openfiles
MACHINE="jail8,johncompanies,com"      # HAS TO BE IN A,B,C FORM
 
cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf
./runbb.sh start
more BBOUT
(look for errors)
exit
 
echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh
 
 
NOTE: to get bb working on amd, had to copy over bin dir from linux dist
 
27. configure load mrtg, on mail
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg
(add new entry to file following existing format)
 
28. configure bb on mail:
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
10.1.4.102 jail2.johncompanies.com # ssh
 
su bb
cd
bbsrc/bb/runbb.sh restart ; exit
 
29. remove reserve space, enable softupdates (probably already set, so not necessary)
NOT APPLICABLE IF USING GVINUM
cd
umount /mnt/data1
umount /mnt/data2
tunefs -m 0 /mnt/data1
tunefs -m 0 /mnt/data2
mount -a
 
30. configure ntp
echo "server 10.1.4.5" > /etc/ntp.conf
 
/usr/sbin/ntpd -p /var/run/ntpd.pid
sleep 2; ntpq -p
(confirm it’s able to reach our time server)
 
32. fwd and reverse lookups on ns1c
vr johncompanies.com
vi internal.johncompanies.com
rndc reload johncompanies.com IN private
(edit the PTR too)
 
 
 
35. create the jail template
 
 
touch /mnt/data1/jail
mdconfig -a -t vnode -s 1g -f /mnt/data1/jail -u 0
bsdlabel -rw md0
newfs4x -i 4096 /dev/md0c
mkdir /mnt/data1/jail-DIR
mount /dev/md0c /mnt/data1/jail-DIR
 
cd /usr/ports/sysutils/jailutils
make install clean
 
cd /usr/src
make world DESTDIR=/mnt/data1/jail-DIR; pagedave
(2450: 2:28mins, supermicro: 55mins, 2950: 1h )
cd etc
make distribution DESTDIR=/mnt/data1/jail-DIR
mount -t devfs devfs /mnt/data1/jail-DIR/dev
devfs -m /mnt/data1/jail-DIR/dev rule -s 3 applyset
cd /mnt/data1/jail-DIR
ln -sf dev/null kernel
cp /usr/local/sbin/jkill /mnt/data1/jail-DIR/sbin
 
jail /mnt/data1/jail-DIR testhostname 192.168.11.100 /bin/sh
csh
touch /etc/fstab
echo 'network_interfaces=""\
hostname="newsystem"\
kern_securelevel_enable="NO"\
sendmail_enable="YES"\
sshd_enable="YES"' > /etc/rc.conf
 
echo "nameserver 69.55.225.225\
nameserver 69.55.230.3" >> /etc/resolv.conf
 
vi /etc/crontab
remove the adjkerntz lines
comment out periodic’s and put this line above them:
# IF YOU UNCOMMENT THESE, PLEASE ADJUST THEIR RUN TIME
 
rm -rf /etc/periodic/daily/400.status-disks
 
check /tmp for crap
 
vi /etc/periodic/security/100.chksetuid
replace: MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`
with: MP='/' (use single quotes)
 
mkdir -p /usr/compat/linux/dev
 
adduser (Add account for user, make sure in wheel group)
 
Username: user
Full name: user
Uid (Leave empty for default):
Login group [user]:
Login group is user. Invite user into other groups? []: wheel
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/user]:
Home directory permissions (Leave empty for default):
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: y
Lock out the account after creation? [no]:
Username  : user
Password  : <random>
Full Name  : user
Uid        : 1001
Class      :
Groups    : user
Home      : /home/user
Home Mode  :
Shell      : /bin/sh
Locked    : no
OK? (yes/no): y
adduser: INFO: Successfully added (user) to the user database.
adduser: INFO: Password for (user) is: 901gmYjO
Add another user? (yes/no): n
Goodbye!
 
vi /usr/home/user/.profile (and add to the file):
TERM=vt100;    export TERM
 
tzsetup
 
newaliases
 
rm /sbin/halt /sbin/reboot
ln /sbin/jkill /sbin/halt
ln /sbin/jkill /sbin/reboot
 
#cd /dev
#rm console
#ln -s null console
 
vi /etc/syslog.conf (comment out console and move to /var/log/messages):
#*.err;kern.warning;auth.notice;mail.crit              /dev/console *.err;kern.warning;auth.notice;mail.crit                /var/log/messages
 
exit
exit
 
cd libexec
chflags noschg ld-elf32.so.1
chflags noschg ld-elf.so.1
 
mv ld-elf32.so.1 ld-elf32.so.1-orig
ln ld-elf.so.1 ld-elf32.so.1
 
chflags schg ld-elf.so.1
chflags schg ld-elf32.so.1
 
 
cd /usr/ports
make -DNOCLEANDEPENDS clean
(2450: 15mins , supermicro: 29mins, 2950: 18mins)
rm -fr /usr/ports/distfiles/*
cp -r /usr/ports /mnt/data1/jail-DIR/usr (2450: 2:00 mins , supermicro: 15mins, 2950: 3mins)
 
rm /mnt/data1/jail-DIR/root/.history
 
cd
umount /mnt/data1/jail-DIR/dev
dump -0a -f /usr/local/jail/template/72template /dev/md0c
umount /dev/md0c
rmdir /mnt/data1/jail-DIR
 
 
36. setup backups
echo '#\!/bin/sh\
backupdir=/data/jail2\
server=backup1\
\
## ENTRY /etc\
## ENTRY /usr/local/etc\
## ENTRY /usr/local/jail\
## ENTRY /root/logs' > /usr/local/jail/bin/backup.config
 
on backup1:
setup backup dirs:
ssh backup1 mkdir -p /data/jail2/0
 
add the system to
vi /usr/local/sbin/snapshot_rotate
 
on mail:
vi /usr/local/www/mgmt/cgi/backupgraph.pl
(add hostname)
 
Edit /usr/local/jail/bin/backup.xxx to use the right drives and copy to /usr/local/jail/bin/backup
 
37. mkdir /root/logs
 
38. edit sshd_config for security
vi /etc/ssh/sshd_config
ListenAddress 69.55.228.53
ListenAddress 10.1.4.102
 
kill -1 `cat /var/run/sshd.pid`
 
39. add crontab entries
crontab -e
5 0 * * * /usr/local/jail/bin/backup.md
1 0 1 * * /usr/local/jail/bin/ipfwreset
0 18 * * * /usr/local/jail/bin/ipfwbackup
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /usr/local/jail/bin/trafstats
 
On 2950 with Perc5/i also add:
*/5 * * * * /usr/local/jail/bin/perc5iraidchk
 
40. Reboot notify script
ln -s /usr/local/jail/bin/notify.sh /usr/local/etc/rc.d/notify.sh
 
41. add to management db (on mail and devweb) jc.ref_machines and jc.ref_templates
 
uname -r
6.2-RELEASE-jc1
 
insert into ref_machines values (null,'mx2','mx2.johncompanies.com',0,'m');
select machine_id from ref_machines where host='mx2';
+------------+
| machine_id |
+------------+
|        35 |
+------------+
insert into ref_templates values ('',' 7.1-RELEASE-jc2',44,'FreeBSD 7.1',0);
 
42. add to server/cabinet map. On mail:
vi /usr/local/www/mgmt/html/cabinetmap.html
 
43. add an outside blocking rule to the firewall, so this machine can only be reached from inside the firewall. Follow example already in firewall jail17 is:
00117 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.228.200
00117 deny ip from any to 69.55.228.200
 
jail19 would be 00119...
ipfw add 00107 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.230.108
ipfw add 00107 deny ip from any to 69.55.220.108
 
44. select customers for probe map
 
47. patch jail against starting jails with rtprio
 
mv /usr/sbin/jail /usr/sbin/jail_
echo '#\!/bin/sh\
/usr/sbin/rtprio -t /usr/sbin/jail_ $*' > /usr/sbin/jail
chmod +x /usr/sbin/jail
 
48. make sure mail works
If there are map errors:
cd /etc/mail; make maps
 
 
50. ntp doesn’t seem to start from rc so,
 
echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh
chmod 0700 /usr/local/etc/rc.d/ntp.sh
 
51. recover space on /usr
 
rm -fr /usr/obj
 
52. setup fuse
 
cd /usr/ports/sysutils/fusefs-kmod/
make install
 
vi /etc/rc.conf
fusefs_enable="YES"
 
sysctl vfs.usermount=1
 
cd /usr/ports/sysutils/fusefs-sshfs
make install
 
sshfs 1005@usw-s009.rsync.net: /mnt/data1/69.55.234.68-col00001-DIR/mnt</pre>
 
= Stress testing a 6.x jail =
 
 
<pre>mkdir /mnt/jail
newfs /dev/gvinum/v1
mount /dev/gvinum/v1 /mnt/jail
cd /mnt/jail
restore -r -f /path/to/61stress .
cd ..
umount /mnt/jail
 
sh
for f in 1 2 3 4 5 6 7 8 9 10; do mkdir /mnt/data1/$f; done
 
gconcat label v3-v5 /dev/gvinum/v3 /dev/gvinum/v4 /dev/gvinum/v5
bsdlabel -r -w /dev/concat/v3-v5
newfs /dev/concat/v3-v5a
mount /dev/concat/v3-v5a /mnt/data1/1
bsdlabel -r -w /dev/gvinum/v10
newfs /dev/gvinum/v10a
mount /dev/gvinum/v10a /mnt/data1/2
gconcat label v23-v25 /dev/gvinum/v23 /dev/gvinum/v24 /dev/gvinum/v25
bsdlabel -r -w /dev/concat/v23-v25
newfs /dev/concat/v23-v25a
mount /dev/concat/v23-v25a /mnt/data1/3
bsdlabel -r -w /dev/gvinum/v50
newfs /dev/gvinum/v50a
mount /dev/gvinum/v50a /mnt/data1/4
gconcat label v63-v65 /dev/gvinum/v63 /dev/gvinum/v64 /dev/gvinum/v65
bsdlabel -r -w /dev/concat/v63-v65
newfs /dev/concat/v63-v65a
mount /dev/concat/v63-v65a /mnt/data1/5
bsdlabel -r -w /dev/gvinum/v70
newfs /dev/gvinum/v70a
mount /dev/gvinum/v70a /mnt/data1/6
gconcat label v83-v85 /dev/gvinum/v83 /dev/gvinum/v84 /dev/gvinum/v85
bsdlabel -r -w /dev/concat/v83-v85
newfs /dev/concat/v83-v85a
mount /dev/concat/v83-v85a /mnt/data1/7
bsdlabel -r -w /dev/gvinum/v100
newfs /dev/gvinum/v100a
mount /dev/gvinum//v100a /mnt/data1/8
gconcat label v123-v125 /dev/gvinum/v123 /dev/gvinum/v124 /dev/gvinum/v125
bsdlabel -r -w /dev/concat/v123-v125
mount /dev/concat/v123-v125a /mnt/data1/9
newfs /dev/concat/v123-v125a
mount /dev/concat/v123-v125a /mnt/data1/9
bsdlabel -r -w /dev/gvinum/v130
newfs /dev/gvinum/v130a
mount /dev/gvinum//v130a /mnt/data1/10
 
sh
for f in 1 2 3 4 5 6 7 8 9 10; do cd /mnt/data1/$f; dump -0a -f - /dev/gvinum/v1| restore -r -f - ; done
 
sh
for f in 1 2 3 4 5 6 7 8 9 10; do mount_devfs devfs /mnt/data1/$f/dev; devfs -m /mnt/data1/$f/dev rule -s 3 applyset; done
 
sh
for f in 15 16 17 18 19 20 21 22 23 24; do ifconfig bce0 alias 10.1.6.$f netmask 255.255.255.255; done
 
jail /mnt/data1/1 testhostname1 10.1.6.15 /bin/sh /etc/rc 2> /dev/null
jail /mnt/data1/2 testhostname2 10.1.6.16 /bin/sh /etc/rc 2> /dev/null
jail /mnt/data1/3 testhostname3 10.1.6.17 /bin/sh /etc/rc 2> /dev/null
jail /mnt/data1/4 testhostname4 10.1.6.18 /bin/sh /etc/rc 2> /dev/null
jail /mnt/data1/5 testhostname5 10.1.6.19 /bin/sh /etc/rc 2> /dev/null
jail /mnt/data1/6 testhostname6 10.1.6.20 /bin/sh /etc/rc 2> /dev/null
jail /mnt/data1/7 testhostname7 10.1.6.21 /bin/sh /etc/rc 2> /dev/null
jail /mnt/data1/8 testhostname8 10.1.6.22 /bin/sh /etc/rc 2> /dev/null
jail /mnt/data1/9 testhostname9 10.1.6.23 /bin/sh /etc/rc 2> /dev/null
jail /mnt/data1/10 testhostname10 10.1.6.24 /bin/sh /etc/rc 2> /dev/null
 
systat -vmstat</pre>
 
 
= Stress testing zfs =
 
<pre>zfs receive pool1/jail2 < /usr/local/jail/template/template
zfs set mountpoint=/mnt/data1/jail2 pool1/jail2
zfs set quota=4G pool1/jail2
zfs destroy pool1/jail2@now</pre>
install apps that will run when jail started
 
<pre>cp -r /usr/src /mnt/data1/jail2/usr
rm -fr /mnt/data1/jail2/usr/obj
 
mount -t devfs devfs /mnt/data1/jail2/dev; devfs -m /mnt/data1/jail2/dev rule -s 3 applyset;
 
jail /mnt/data1/jail2 stress-test 69.55.234.86 /bin/sh
csh
cd /usr/ports/benchmarks/bonnie
make install clean
cd /usr/ports/sysutils/stress
make install clean
cd /usr/ports/net/rsync
make install clean
cd /usr/ports/lang/perl5.8
make install clean
 
cd /usr/local/etc/rc.d/
 
cat >> boot.sh
sleep 30 && sh /usr/local/etc/rc.d/buildworld &
sleep 30 && sh /usr/local/etc/rc.d/portindex &
sleep 30 && sh /usr/local/etc/rc.d/stress &
sleep 30 && sh /usr/local/etc/rc.d/bonnie &
 
cat > buildworld
#!/bin/sh
while (true); do cd /usr/src; make buildworld; done
 
cat > portindex
while (true); do cd /usr/ports/; make index; done
 
cat > stress
/usr/local/bin/stress -c 1 -i 1 -m 1 -d 1 --vm-bytes 768M --hdd-bytes 128M
 
cat > bonnie
#!/bin/sh
while (true); do /usr/local/bin/bonnie -s 2g; done
 
 
chmod +x *
 
exit
exit
 
cd
zfs snapshot pool1/jail2@now
 
 
 
sh
for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105; do zfs send pool1/jail2@now | zfs receive pool1/jail$f;
zfs set quota=10G pool1/jail$f;
zfs set mountpoint=/mnt/data1/jail$f pool1/jail$f;
done
 
for f in 106 107 108 114 115 116 117 118 119 120 121 122 123 125 126 127 128 129 130 131; do zfs send pool1/jail2@now | zfs receive pool2/jail$f
zfs set quota=10G pool2/jail$f;
zfs set mountpoint=/mnt/data2/jail$f pool2/jail$f;
done
 
for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101; do chmod 0 /mnt/data1/jail$f/usr/local/etc/rc.d/bonnie; done
for f in 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105; do chmod 0 /mnt/data1/jail$f/usr/local/etc/rc.d/stress; done
 
for f in 115 116 117 118 119 120 121 122 123 125 126 127; do chmod 0 /mnt/data2/jail$f/usr/local/etc/rc.d/bonnie; done
for f in 119 120 121 122 123 125 126 127 128 129 130 131; do chmod 0 /mnt/data2/jail$f/usr/local/etc/rc.d/stress; done
 
for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105; do mount -t devfs devfs /mnt/data1/jail$f/dev; devfs -m /mnt/data1/jail$f/dev rule -s 3 applyset; done
 
for f in 106 107 108 114 115 116 117 118 119 120 121 122 123 125 126 127 128 129 130 131; do mount -t devfs devfs /mnt/data2/jail$f/dev; devfs -m /mnt/data2/jail$f/dev rule -s 3 applyset; done
 
for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105;
do jail /mnt/data1/jail$f testhostname$f 69.55.234.$f /bin/sh /etc/rc 2> /dev/null; done
 
for f in 106 107 108 114 115 116 117 118 119 120 121 122 123 125 126 127 128 129 130 131;
do jail /mnt/data2/jail$f testhostname$f 69.55.234.$f /bin/sh /etc/rc 2> /dev/null; done
 
 
systat -vmstat
sysctl vfs.numvnodes
 
 
for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105; do umount /mnt/data1/jail$f/dev; zfs destroy -r pool1/jail$f; done
 
for f in 106 107 108 114 115 116 117 118 119 120 121 122 123 125 126 127 128 129 130 131; do umount /mnt/data2/jail$f/dev; zfs destroy -r pool2/jail$f; done
 
 
for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105; do touch /mnt/data1/jail$f;
mdconfig -a -t vnode -s 10g -f /mnt/data1/jail$f -u $f;
bsdlabel -r -w md$f auto;
newfs -O 1 /dev/md${f}a;
mkdir /mnt/data1/jail$f-DIR;
mount /dev/md${f}a /mnt/data1/jail$f-DIR;
rsync -aSH /mnt/data2/jail2/ /mnt/data1/jail$f-DIR/;
done
 
for f in 106 107 108 114 115 116 117 118 119 120 121 122 123 125 126 127 128 129 130 131; do touch /mnt/data2/jail$f;
mdconfig -a -t vnode -s 10g -f /mnt/data2/jail$f -u $f;
bsdlabel -r -w md$f auto;
newfs -O 1 /dev/md${f}a;
mkdir /mnt/data2/jail$f-DIR;
mount /dev/md${f}a /mnt/data2/jail$f-DIR;
rsync -aSH /mnt/data1/jail86-DIR/ /mnt/data2/jail$f-DIR/;
done
 
for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105;
do chmod 0700 /mnt/data1/jail$f-DIR/usr/local/etc/rc.d/bonnie; done
for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105;
do chmod 0700 /mnt/data1/jail$f-DIR/usr/local/etc/rc.d/stress; done
 
for f in 106 107 108 114 115 116 117 118 119 120 121 122 123 125 126 127 128 129 130 131;
do chmod 0700 /mnt/data2/jail$f-DIR/usr/local/etc/rc.d/bonnie; done
for f in 106 107 108 114 115 116 117 118 119 120 121 122 123 125 126 127 128 129 130 131;
do chmod 0700 /mnt/data2/jail$f-DIR/usr/local/etc/rc.d/stress; done
 
for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101; do chmod 0 /mnt/data1/jail$f-DIR/usr/local/etc/rc.d/bonnie; done
for f in 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105; do chmod 0 /mnt/data1/jail$f-DIR/usr/local/etc/rc.d/stress; done
 
for f in 115 116 117 118 119 120 121 122 123 125 126 127; do chmod 0 /mnt/data2/jail$f-DIR/usr/local/etc/rc.d/bonnie; done
for f in 119 120 121 122 123 125 126 127 128 129 130 131; do chmod 0 /mnt/data2/jail$f-DIR/usr/local/etc/rc.d/stress; done
 
for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105; do mount -t devfs devfs /mnt/data1/jail$f-DIR/dev; devfs -m /mnt/data1/jail$f-DIR/dev rule -s 3 applyset; done
 
for f in 106 107 108 114 115 116 117 118 119 120 121 122 123 125 126 127 128 129 130 131; do mount -t devfs devfs /mnt/data2/jail$f-DIR/dev; devfs -m /mnt/data2/jail$f-DIR/dev rule -s 3 applyset; done
 
for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105;
do jail /mnt/data1/jail$f-DIR testhostname$f 69.55.234.$f /bin/sh /etc/rc 2> /dev/null; done
 
for f in 106 107 108 114 115 116 117 118 119 120 121 122 123 125 126 127 128 129 130 131;
do jail /mnt/data2/jail$f-DIR testhostname$f 69.55.234.$f /bin/sh /etc/rc 2> /dev/null; done
 
 
for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105;
do cp /mnt/data1/boot.sh /mnt/data1/jail$f-DIR/usr/local/etc/rc.d/;
cp /mnt/data1/stress /mnt/data1/jail$f-DIR/usr/local/etc/rc.d/;
cp /mnt/data1/bonnie /mnt/data1/jail$f-DIR/usr/local/etc/rc.d/;
done
 
for f in 106 107 108 114 115 116 117 118 119 120 121 122 123 125 126 127 128 129 130 131;
do cp /mnt/data1/boot.sh /mnt/data2/jail$f-DIR/usr/local/etc/rc.d/;
cp /mnt/data1/stress /mnt/data2/jail$f-DIR/usr/local/etc/rc.d/;
cp /mnt/data1/bonnie /mnt/data2/jail$f-DIR/usr/local/etc/rc.d/;
done
 
for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105; do umount /mnt/data1/jail$f-DIR/dev; umount /mnt/data1/jail$f-DIR; mdconfig -d -u $f; done
 
for f in 106 107 108 114 115 116 117 118 119 120 121 122 123 125 126 127 128 129 130 131; do umount /mnt/data2/jail$f-DIR/dev; umount /mnt/data2/jail$f-DIR; mdconfig -d -u $f;  done
 
for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105; do
mkdir /mnt/data1/jail$f-DIR/;
cd /mnt/data1/jail$f-DIR/;
dump -0a -f - /dev/md0a | restore -r -f -;
done
 
for f in 106 107 108 114 115 116 117 118 119 120 121 122 123 125 126 127 128 129 130 131; do mkdir /mnt/data2/jail$f-DIR/;
cd /mnt/data2/jail$f-DIR/;
dump -0a -f - /dev/md0a | restore -r -f -;
done
 
for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105; do chmod 0700 /mnt/data1/jail$f-DIR/usr/local/etc/rc.d/*; done
 
for f in 106 107 108 114 115 116 117 118 119 120 121 122 123 125 126 127 128 129 130 131; do chmod 0700 /mnt/data2/jail$f-DIR/usr/local/etc/rc.d/*;  done
 
 
 
--vm-bytes 300M</pre>
 
= Reloading FreeBSD (while preserving customer data) =
 
<pre>2. assuming 2 mirrors, boot to disk 1 of 4.11
skip kernel config (enter)
custom install
(skip partition)
label
move cursor to aacd0, hit space (takes you to partition map screen)
given this as the pre-existing partition map:
Filesystem    1K-blocks  Used    Avail Capacity  Mounted on
/dev/aacd0s1a    128990  36888    81784    31%    /
/dev/aacd0s1h  30499106      4 28059174    0%    /mnt/data1
/dev/aacd1s1e  69526550      4 63964422    0%    /mnt/data2
/dev/aacd0s1f    257998    10  237350    0%    /tmp
/dev/aacd0s1g  3096462 852082  1996664    30%    /usr
/dev/aacd0s1e    257998  1198  236162    1%    /var
procfs                4      4        0  100%    /proc
 
relabel and all mount points according to prev settings
 
or, delete all partitions except for data parts, re-create as necessary
 
re-toggle newfs on all mount points except for /mnt/data1 - make sure under the Newfs column the data1 (and data2) partiotions look like: UFS+S Y
 
make sure under the Newfs column the data1 (and data2) partiotions look like
UFS+S N
 
space to unselect aacd0
cursor over aacd1
space
 
Space to select both drives and tab to ok
 
Continue installation as per normal
distributions
developer
yes to install ports
exit
media
cd
commit
yes
(2450: 16mins, supermicro: 11mins)
yes to "visit general config"
Set root pwd
Add user ‘user’ member group is wheel, set password
Set tz
Networking->interfaces->Fxp0
No IPV6
dhcp=yes
Set hostname & domain
 
exit...
exit install
yes
take the cd out and let the machine reboot</pre>

Latest revision as of 10:48, 25 September 2013

FreeBSD 9.x[edit]

Assumptions[edit]

Setup instructions below assume this is DELL 2950 with an LSI-based SAS RAID card.

Server is at castle, connected to pub, private, serial and DRAC

Assuming OS loading done via IPKVM with ISO mounted via USB

Assumes at 4 drives, 2 mirrors

Configure server BIOS[edit]

setup console redirect, speed 115200

set LCD string to name of server "jail8"

set date to GMT

go into RAID bios and setup mirrors


Setup DRAC[edit]

DRAC setup

Install OS (sysinstall)[edit]

boot to bootonly disk for AMD version of FreeBSD, i.e. FreeBSD-8.3-RELEASE-amd64-bootonly.iso

when the install menu appears, choose custom install

partition menu[edit]

move cursor to mfid0, hit space (takes you to partition map screen). If there is only 1 mirror, there will be no option to select a specifit drive: mfid0 will be selected.

type 'a' to use entire disk
type 'q' to quit and save
choose 'freebsd standard mbr'

space to unselect mfid0
cursor down to mfid1
hit space

type 'a' to use entire disk
type 'q' to quit and save
choose 'none' for boot mgr (leave untouched)

cursor over mfid0
space
(takes you into part. Screen again) 'q' to exit
none for boot mgr

Make sure both drives (mfid0 and mfid1) are checked and tab to ok

Label menu[edit]

Make sure mfid0 is highlighted at the top of the screen, setup the following partitions

/ 512M
swap 6G
/var 1G
/tmp 256M
/usr 8G
/mnt/data1 remaining space

All partitions except / should be setup for soft updates. If not, type 's' to enable for soft updates on all except for / (should look like UFS2+S Y under the Newfs column)

move cursor to mfid1 at the top of the screen
swap 8G (or 4G if there’s a 3rd drive)
/mnt/data2 remaining space
'q' to save and exit

distributions[edit]

Choose the following distribudions

  • developer (ok to install ports)
  • custom -> lib32

exit

media[edit]

if you are installing via a cd, no need to enter this menu or change anything. Otherwise, choose ftp to install via ftp. You will be prompted to setup networking. You will need to choose a nic (typically bce0 or bce1). Say no to DHCP and IPv6. Hopefully the public nic cable was installed in bce0 so start with that nic and provide the hostname, (public) IP, netmask, gateway and DNS. When configured, it should start pinging. If it doesn't, have the NOC swap cables. Select any FTP server, usually Main or ftp4.

commit[edit]

this usually takes 12mins
during the process you may need to select a new ftp mirror, this is not a problem.
at the conclution of the install you will be prompted to enter the root password (2x) and returned to the configuration menu.

add user[edit]

Add user 'user'. Defaults for everything is fine, just remember to enter 'wheel' in the member group field. Do set the password.

Setup timezone[edit]

PT

Networking[edit]

page down to the bottom and enable '[X]' sshd

If you installed via cd, you will need to visit:
interfaces->bce0->
No IPV6
dhcp=no
Set hostname, IP, DNS, gateway
(i.e. setup the nic as indicated above)


Exit the install and if you installed via CD, take it out and let the machine reboot

Configure OS, kernel, userland, jail[edit]

double check the date/time[edit]

 date

populate /etc/resolv.conf[edit]

echo "search johncompanies.com \
nameserver 69.55.225.225 \
nameserver 69.55.230.3  \
nameserver 69.55.229.3" > /etc/resolv.conf

edit /etc/make.conf[edit]

echo "WITHOUT_X11=yes \
KERNCONF=jail4 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf

setup bootloader for console, etc[edit]

add settings to /boot/loader.conf and /boot.config: 

echo "-Dh" >> /boot.config

echo 'console="comconsole,vidconsole" \
boot_multicons="YES" \
boot_serial="YES" \
mfi_linux_load="YES" \
comconsole_speed="115200"' >> /boot/loader.conf


enable login via serial console[edit]

turn off all ttyv's except 0 and 1 in /etc/ttys and turn on ttyd0, change type to vt100: 

vi /etc/ttys

The changed lines should look like: 

ttyv2   "/usr/libexec/getty Pc"         cons25  off secure
ttyv3   "/usr/libexec/getty Pc"         cons25  off secure
ttyv4   "/usr/libexec/getty Pc"         cons25  off secure
ttyv5   "/usr/libexec/getty Pc"         cons25  off secure
ttyv6   "/usr/libexec/getty Pc"         cons25  off secure
ttyv7   "/usr/libexec/getty Pc"         cons25  off secure
# Serial terminals
# The 'dialup' keyword identifies dialin lines to login, fingerd etc.
ttyu0   "/usr/libexec/getty std.9600"  vt100   on secure

Restart init 

kill -1 1

At this point you should have a login on console.

To configure serial console access, login to the console server as root and run: 

# vi /etc/remote

following examples there, rename port to server's hostname, depending on where and which digi box this server is plugged into. Make sure to get speed right too: 115200


populate hosts[edit]

If server is at castle: 

echo "10.1.4.3 backup2" >> /etc/hosts
echo "10.1.4.8 backup1" >> /etc/hosts

If server is at i2b: 

echo "69.55.230.10 backup2" >> /etc/hosts
echo "10.1.2.3 backup3" >> /etc/hosts
echo "69.55.230.11 backup1" >> /etc/hosts


create ssh key, upload to backup servers[edit]

cd
ssh-keygen -t dsa -b 1024

(default location, leave password blank)

If server is at castle: 

cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys' 
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'

If server is at i2b: 

cat /root/.ssh/id_dsa.pub | ssh backup3 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys' 
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'

confirm that you can ssh to backup2 and backup1 without getting a login prompt 

ssh backup2 hostname
ssh backup1 hostname
ssh backup3 hostname


create & populate binaries/scripts dirs[edit]

mkdir -p /usr/local/jail/bin
mkdir -p /usr/local/jail/rc.d
mkdir -p /usr/local/jail/template/
mkdir /mnt/data1
mkdir /mnt/data2
scp backup2:"/mnt/data4/bin/freebsd9.x/*" /usr/local/jail/bin
cd /usr/local/jail/rc.d/
touch quad1
touch deprecated
chmod +x *
cd /usr/local/jail/bin
ln -s /usr/local/jail/rc.d/quad1 quad1
ln -s /usr/local/jail/bin/jailmake_md jailmake
ln -s /usr/local/jail/bin/js_md js
ln -s /usr/local/jail/bin/canceljail_md canceljail
ln -s /usr/local/jail/bin/jailmakeempty_md jailmakeempty
ln -s /usr/local/jail/bin/postboot_md postboot
ln -s /usr/local/jail/bin/preboot_md preboot
ln -s /usr/local/jail/bin/startjail_md startjail
ln -s /usr/local/jail/bin/stopjail_md stopjail

rehash

edit root's path and login script[edit]

vi /root/.cshrc

Change alias entries (add G): 

alias la        ls -aG
alias lf        ls -FAG
alias ll        ls -lAG
alias ls        ls -AG
alias mbm       mb mount
alias mbu       mb umount
alias cjb       cd /usr/local/jail/bin
alias cd1       cd /mnt/data1
alias cd2       cd /mnt/data2
alias cd3       cd /mnt/data3
alias jtop      jtop lj
alias j         jobs

add to path be careful to leave a space after bin and make sure the wrapping isn't broken: 

/usr/local/jail/bin

alter the prompt, set the following: 

set prompt = "$user@`/bin/hostname -s` %/# "

at the bottom of the file add: 

set sshtty=`who am i|awk '{print $2}'`
/usr/sbin/rtprio 3 -`psj | grep $sshtty | awk '{print $2}'`

set shortty=`who am i | awk '{print $2}' | sed -E 's/.*(..)$/\1/'`
foreach x (`psj | grep sh | grep $shortty | awk '{print $2}'`)
/usr/sbin/rtprio 2 -$x
end

Make the new settings active in current shell: 

source /root/.cshrc

update ports[edit]

portsnap fetch
portsnap extract

To update later on: 

portsnap fetch
portsnap update


install svn[edit]

setenv PACKAGESITE "ftp://ftp4.freebsd.org/pub/FreeBSD/ports/amd64/packages-9-current/Latest/"
pkg_add subversion

get latest sources for this release[edit]

cd /usr

# mv src/ src.orig
tar cvzf src.orig.tgz src 
rm -fr src/*
svn checkout svn://svn.FreeBSD.org/base/stable/9 /usr/src

To update: 

make update SVN_UPDATE=yes

configure new kernel[edit]

Pull down the kernel config we are using for this distribution. In this case we use an 8.2 kernel config on 8.3, which is valid. The local file should be the same name as host- jail4 in this example 

cd /usr/src/sys/amd64/conf 
scp backup2:/mnt/data4/build/freebsd/kern_config-9.1-amd64 ./jail4

edit the kernel config and change ident to be the name of the jail: 

vi jail4
ident           jail4

Optional, edit /sys/conf/newvers.sh to add –jc2 to the end of the BRANCH string (RELEASE-jc2) 

vi /sys/conf/newvers.sh

notes on kernel configuring: http://www.FreeBSD.org/doc/en_US.ISO8859-1/books/handbook/kernelconfig-config.html

install patches[edit]

We don't have any patches right now. Refer to older FreeBSD version build docs on how that is/was done.

build, install kernel and world[edit]

Rename current generic kernel so it will always be available to boot from. Save room by removing non-needed kernel modules: 

cd /boot
mv kernel kernel.GENERIC
cd kernel.GENERIC
mkdir hold
mv mfi_linux.ko hold/
mv linux.ko hold/
mv linprocfs.ko hold/
mv linsysfs.ko hold/
mv geom_vinum.ko hold/
mv geom_concat.ko hold/
mv zfs.* hold/
mv opensolaris* hold/

rm *.ko
rm *.symbols
mv hold/* .
rmdir hold/

Note on -DWITHOUT_CLANG: try to do it without including that directive, it may work for you.

cd /usr/src
make buildworld KERNCONF=jail4; mail -s 'buildworld done' dave.boodman@vtext.com < /dev/null


make -DWITHOUT_CLANG buildworld KERNCONF=jail4; mail -s 'buildworld done' dave.boodman@vtext.com < /dev/null

~4.5hr 

cd /usr/src
make buildkernel installkernel



mergemaster -p

You will be prompted to merge, replace or ignore files changed by the src update. In most cases you can delete the temp (new) files. 

make -DWITHOUT_CLANG installworld

~34min


ONLY if this will be a zfs system (not currently used in 8.x): 

cd /sys/modules/zfs
make 
make install
cd /sys/modules/opensolaris
make 
make install

populate devfs ruleset[edit]

scp backup2:/mnt/data4/build/freebsd/devfs.rules.8x /etc/devfs.rules


populate /etc/rc.conf with IPs and service settings[edit]

vi /etc/rc.conf

kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"

nfs_client_enable="YES"
nfs_reserved_port_only="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.4.XXX"
devfs_system_ruleset="devfsrules_show_all"

ifconfig_bce1="inet 10.1.4.XXX netmask 255.255.255.0"
ifconfig_bce0="inet 69.55.2XX.XXX netmask 255.255.255.0"
#ifconfig_bce0_alias0="inet 69.55.2XX.XXX netmask 255.255.255.0"

fsck_y_enable="YES"
background_fsck="NO"
#rc_mfi_raid_tty_log="YES"
#zfs_enable="YES"

Modify IPs, hostname, gateway for this box.

make sure sysctls are set and preserved upon boot[edit]

echo "kern.consmute=0\
kern.ipc.shm_use_phys=1\
kern.ipc.shmall=131070\
kern.ipc.shmmax=134217728\
net.inet.tcp.syncookies=0\
kern.maxfiles=32768\
kern.fallback_elf_brand=3\
kern.maxprocperuid=4000\
security.jail.sysvipc_allowed=1\
security.jail.allow_raw_sockets=1\
security.jail.socket_unixiproute_only=1\
security.jail.chflags_allowed=0\
dev.amr.0.allow_volume_configure=1\
compat.linux.osrelease=2.6.12\
vm.pmap.shpgperproc=500\
security.bsd.unprivileged_read_msgbuf=0\
kern.maxvnodes=400000" >> /etc/sysctl.conf

Tuning note: watch vfs.numvnodes while the server is live to get guidance on where to set maxvnodes


mount procfs[edit]

echo "proc                    /proc           procfs  rw              0       0" >> /etc/fstab
echo "linprocfs               /usr/compat/linux/proc linprocfs rw     0       0" >> /etc/fstab
echo "linsysfs                /usr/compat/linux/sys linsysfs rw       0       0" >> /etc/fstab
mkdir -p /usr/compat/linux/proc
mkdir -p /usr/compat/linux/sys


enable noatime option[edit]

grep data /etc/fstab

data1 and data2 should look something like (add ',noatime' after 'rw'): 

/dev/mfid0s1g           /mnt/data1      ufs     rw,noatime      2       2
/dev/mfid1s1d           /mnt/data2      ufs     rw,noatime      2       2

reboot. Confirm new kernel is loaded, devfs in place[edit]

uname -a

Check devfs rules 

devfs rule showsets
devfs rule -s 3 show

Should see: 

#  devfs rule showsets
1
2
3
4

#  devfs rule -s 3 show
100 include 1
207 path pts* unhide
217 path fd unhide
218 path fd/* unhide
300 path ttyp* unhide
301 path ttyq* unhide
302 path ttyr* unhide
303 path ttys* unhide
304 path ttyP* unhide
305 path ttyQ* unhide
306 path ttyR* unhide
307 path ttyS* unhide
400 path null unhide
500 path zero unhide
600 path random unhide
610 path urandom unhide
700 path mem unhide
710 path kmem unhide
810 path mdctl unhide
900 path stdin unhide
910 path stdout unhide
920 path stderr unhide


Install raid mgmt tool[edit]

Perc5/i, 6/i[edit]

Pull over cli from previous system (jail9) 

scp root@10.1.4.109:"/usr/local/sbin/mega*" /usr/local/sbin/
scp root@10.1.4.109:/usr/local/libexec/MegaCli /usr/local/libexec/MegaCli

These are linux-based tools. This will require linux base...which you might install via: 

pkg_add -r linux_base

Test: 

rehash; megacli ldinfo lall a0

or 

megarc -ldInfo -a0 -Lall

(2850)

however, linux does seem to be installed already so we don't need to pkg_add or port install or rsync anything over from a current system.

2850 PERC 4e/Di- no linux[edit]

cd /usr/ports/distfiles/
fetch http://backup01.best-hosting.ru/pub/FreeBSD/ports/distfiles/dr_freebsd_1.51.zip
cd /usr/ports/sysutils/megarc
make install clean
megarc -dispCfg -a0


install rsync from ports[edit]

cd /usr/ports/net/rsync
make install clean

choose default options


configure inetd to respond to mrtg load queries[edit]

echo "load    stream  tcp     nowait  user    /usr/local/jail/bin/load.pl  load.pl" >> /etc/inetd.conf
echo "load            12384/tcp" >> /etc/services


install perl[edit]

cd /usr/ports/lang/perl5.12
make install clean

choose defaults


install bb client[edit]

Compiling from source on AMD64 will not work. So, we use a linux-compiled version and rely on linux compat. 

adduser

Output/response: 

Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]: 
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username   : bb
Password   : <random>
Full Name  : bb
Uid        : 1984
Class      :
Groups     : bb
Home       : /home/bb
Shell      : /bin/sh
Locked     : no
OK? (yes/no): yes
cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd_linuxcompat.tgz .
tar xzf bb-freebsd_linuxcompat.tgz

edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like: 

echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.4.104 jail4.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

Edit for machine name and private IP.

if this machine is at i2b: 

echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
69.55.228.104 jail4.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

vi /home/bb/bbc1.9e-btf/ext/openfiles 

MACHINE="jail4,johncompanies,com"      # HAS TO BE IN A,B,C FORM

Edit for machine name. 

cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh 
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf
./runbb.sh start
more BBOUT 
(look for errors)
exit

Put in script to start bb @ boot: 

echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh

remove reserve space[edit]

cd
umount /mnt/data1
umount /mnt/data2
tunefs -m 0 /mnt/data1
tunefs -m 0 /mnt/data2
mount -a


setup rdate[edit]

cd /usr/ports/sysutils/rdate
make install clean

crontab -e
0 0 * * * /usr/local/sbin/rdate -s utcnist.colorado.edu

/usr/local/sbin/rdate -s utcnist.colorado.edu

We used to use ntpd, however it listens on jail IPs which is a security risk. So we stopped. Here's the old instructions: install new ntp from ports 

/usr/ports/net/ntp
make install clean

echo "server 10.1.4.5" > /etc/ntp.conf
/usr/sbin/ntpd -p /var/run/ntpd.pid
sleep 2; ntpq -p

(confirm it’s able to reach our time server)


make a data partition[edit]

IF you didn't format the data partition during sysinstall:

Create a g partition on 2nd mirror – bsdlabel no longer works (below shows d partition made with sysinstall): 

jail8 /usr/home/bb# gpart show
=>       63  285474735  mfid0  MBR  (136G)
         63  285458922      1  freebsd  [active]  (136G)
  285458985      15813         - free -  (7.7M)

=>        0  285458922  mfid0s1  BSD  (136G)
          0     524288        1  freebsd-ufs  (256M)
     524288   12582912        2  freebsd-swap  (6.0G)
   13107200     524288        4  freebsd-ufs  (256M)
   13631488     524288        5  freebsd-ufs  (256M)
   14155776    8388608        6  freebsd-ufs  (4.0G)
   22544384  262914538        7  freebsd-ufs  (125G)

=>       63  584843175  mfid1  MBR  (279G)
         63  584830197      1  freebsd  [active]  (279G)
  584830260      12978         - free -  (6.3M)

=>        0  584830197  mfid1s1  BSD  (279G)
          0   16777216        2  freebsd-swap  (8.0G)
   16777216  568052981        4  freebsd-ufs  (271G)

jail8 /usr/home/bb# gpart show mfid1s1
=>        0  584830197  mfid1s1  BSD  (279G)
          0   16777216        2  freebsd-swap  (8.0G)
   16777216  568052981        4  freebsd-ufs  (271G)

# gpart list mfid1s1

Geom name: mfid1s1
fwheads: 255
fwsectors: 63
last: 584830196
first: 0
entries: 8
scheme: BSD
Providers:
1. Name: mfid1s1b
   Mediasize: 8589934592 (8.0G)
   Sectorsize: 512
   Mode: r1w1e0
   rawtype: 1
   length: 8589934592
   offset: 0
   type: freebsd-swap
   index: 2
   end: 16777215
   start: 0
2. Name: mfid1s1d
   Mediasize: 290843126272 (271G)
   Sectorsize: 512
   Mode: r0w0e0
   rawtype: 7
   length: 290843126272
   offset: 8589934592
   type: freebsd-ufs
   index: 4
   end: 584830196
   start: 16777216
Consumers:
1. Name: mfid1s1
   Mediasize: 299433060864 (279G)
   Sectorsize: 512
   Mode: r1w1e1

# gpart delete -i 4 mfid1s1
mfid1s1d deleted
jail8 /usr/home/bb# gpart list mfid1s1
Geom name: mfid1s1
fwheads: 255
fwsectors: 63
last: 584830196
first: 0
entries: 8
scheme: BSD
Providers:
1. Name: mfid1s1b
   Mediasize: 8589934592 (8.0G)
   Sectorsize: 512
   Mode: r1w1e0
   rawtype: 1
   length: 8589934592
   offset: 0
   type: freebsd-swap
   index: 2
   end: 16777215
   start: 0
Consumers:
1. Name: mfid1s1
   Mediasize: 299433060864 (279G)
   Sectorsize: 512
   Mode: r1w1e1

# gpart show mfid1s1
=>        0  584830197  mfid1s1  BSD  (279G)
          0   16777216        2  freebsd-swap  (8.0G)
   16777216  568052981           - free -  (271G)

# gpart add  -t freebsd-ufs -i 7 mfid1s1
mfid1s1g added

# gpart show mfid1s1
=>        0  584830197  mfid1s1  BSD  (279G)
          0   16777216        2  freebsd-swap  (8.0G)
   16777216  568052981        7  freebsd-ufs  (271G)

Here's how we USED to do it with bsdlabel: 

bsdlabel -e /dev/mfid0s1

given: 

# /dev/aacd0s1:
8 partitions:
#        size   offset    fstype   [fsize bsize bps/cpg]
  a:   262144        0    4.2BSD     2048 16384 16392
  b:  4194304   262144      swap
  c: 143363997        0    unused        0     0         # "raw" part, don't edit
  d:   524288  4456448    4.2BSD     2048 16384 32776
  e:   524288  4980736    4.2BSD     2048 16384 32776
  f:  6291456  5505024    4.2BSD     2048 16384 28552

new offset = 6291456 + 5505024 = 11796480
new size is size for 'c' partition minus the new start from above
143363997 - 11796480 = 131567517

So:

g: 131567517 11796480 unused 0 0


create the jail template[edit]

cd /usr/ports/sysutils/jailutils
make install clean

Create an md device to hold the jail: 

touch /mnt/data1/jail-template20g
mdconfig -a -t vnode -s 20g -f /mnt/data1/jail-template20g -u 0
newfs -O 1 /dev/md0
mkdir /mnt/data1/jail-DIR
mount /dev/md0 /mnt/data1/jail-DIR

Build world into the jail: 

cd /usr/src
make world DESTDIR=/mnt/data1/jail-DIR; taskdone

~4.5 hr

Make /etc into the jail, mount dev, copy in jkill: 

cd etc
make distribution DESTDIR=/mnt/data1/jail-DIR
mount -t devfs devfs /mnt/data1/jail-DIR/dev
devfs -m /mnt/data1/jail-DIR/dev rule -s 3 applyset 
cd /mnt/data1/jail-DIR
ln -sf dev/null kernel
cp /usr/local/sbin/jkill /mnt/data1/jail-DIR/sbin

Enter into jail to do configuration: 

jail /mnt/data1/jail-DIR testhostname 192.168.11.100 /bin/sh
csh

Create fstab: 

touch /etc/fstab
echo 'network_interfaces=""\
hostname="newsystem"\
kern_securelevel_enable="NO"\
sendmail_enable="YES"\
sshd_enable="YES"' > /etc/rc.conf

echo "nameserver 69.55.225.225\
nameserver 69.55.230.3" >> /etc/resolv.conf

Edit crontab: 

vi /etc/crontab

remove the adjkerntz lines comment out periodic’s and put this line above them: 

# IF YOU UNCOMMENT THESE, PLEASE ADJUST THEIR RUN TIME

rm -rf /etc/periodic/daily/400.status-disks

check and remove any crap in /tmp 

vi /etc/periodic/security/100.chksetuid

replace: MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`
with: MP='/' (use single quotes) 

mkdir -p /usr/compat/linux/dev

Add account for user. Output/response: 

adduser

Username: user
Full name: user
Uid (Leave empty for default):
Login group [user]:
Login group is user. Invite user into other groups? []: wheel
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/user]:
Home directory permissions (Leave empty for default):
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: y
Lock out the account after creation? [no]:
Username   : user
Password   : <random>
Full Name  : user
Uid        : 1001
Class      :
Groups     : user
Home       : /home/user
Home Mode  :
Shell      : /bin/sh
Locked     : no
OK? (yes/no): y
adduser: INFO: Successfully added (user) to the user database.
adduser: INFO: Password for (user) is: 901gmYjO
Add another user? (yes/no): n
Goodbye!

set TERM: 

vi /usr/home/user/.profile 
TERM=vt100;     export TERM

Set time zone to PT: 

tzsetup

Reload aliases: 

newaliases

Replace reboot/halt: 

rm /sbin/halt /sbin/reboot
ln /sbin/jkill /sbin/halt
ln /sbin/jkill /sbin/reboot

Redirect console output, comment out console and move to /var/log/messages: 

vi /etc/syslog.conf
#*.err;kern.warning;auth.notice;mail.crit               /dev/console
*.err;kern.warning;auth.notice;mail.crit                /var/log/messages 

exit
exit

Lib32 compat library: 

cd libexec
chflags noschg ld-elf32.so.1
chflags noschg ld-elf.so.1
mv ld-elf32.so.1 ld-elf32.so.1-orig
ln ld-elf.so.1 ld-elf32.so.1
chflags schg ld-elf.so.1
chflags schg ld-elf32.so.1

Replace traceroute: 

mv /mnt/data1/jail-DIR/usr/sbin/traceroute /mnt/data1/jail-DIR/usr/sbin/_traceroute
echo '#\!/bin/sh\
/usr/sbin/_traceroute -i bce0 $1' >> /mnt/data1/jail-DIR/usr/sbin/traceroute 
chmod +x /mnt/data1/jail-DIR/usr/sbin/traceroute

Modify 'bce0' to reflect whichever nic is public on this hardware.

Clean out ports before copying into the jail (~30mins): 

cd /usr/ports
make -DNOCLEANDEPENDS clean 
rm -fr /usr/ports/distfiles/*
cp -r /usr/ports /mnt/data1/jail-DIR/usr 

rm /mnt/data1/jail-DIR/root/.history

Umount the jail and dump it: 

cd /
umount /mnt/data1/jail-DIR/dev
dump -0a -f /usr/local/jail/template/template /dev/md0
umount /dev/md0
rmdir /mnt/data1/jail-DIR
mdconfig -d -u 0

setup backups[edit]

echo '#\!/bin/sh\
backupdir=/data/jail3\
server=backup1\
\
## ENTRY /etc\
## ENTRY /usr/local/etc\
## ENTRY /usr/local/jail\
## ENTRY /root/logs' > /usr/local/jail/bin/backup.config

Edit to reflect backup server and jail hostname

On backup server, setup backup dirs: 

ssh backup1 mkdir -p /data/jail3/0

On backup server, setup backup dirs: 

backup1# vi /usr/local/sbin/snapshot_rotate

Edit /usr/local/jail/bin/backup.xxx to use the right drives and copy to /usr/local/jail/bin/backup 

vi /usr/local/jail/bin/backup.md

adjust df so it includes all relevant drives, currently df > /etc/df.bak is fine. Also, make sure the binary/script source is pulling from the right dir on backup2: 

/usr/local/bin/rsync -a backup2:/mnt/data4/bin/freebsd8.x/ /usr/local/jail/bin/

cp /usr/local/jail/bin/backup.md /usr/local/jail/bin/backup


create /root/logs[edit]

mkdir /root/logs


edit sshd_config[edit]

vi /etc/ssh/sshd_config
ListenAddress 69.55.229.7
ListenAddress 10.1.4.103

Adjust to pub/private IP for jail.

Restart sshd: 

kill -1 `cat /var/run/sshd.pid`

add crontab entries[edit]

crontab -e
5 0 * * * /usr/local/jail/bin/backup
1 0 1 * * /usr/local/jail/bin/ipfwreset
0 18 * * * /usr/local/jail/bin/ipfwbackup
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /usr/local/jail/bin/trafstats
0 0,6,12,18 * * * /usr/local/jail/bin/sync_jail_names

On 2950: 

*/5 * * * * /usr/local/jail/bin/perc5iraidchk

On 3ware-based servers: 

0 0 * * * /usr/local/jail/bin/3wraidchk


Reboot notify script[edit]

ln -s /usr/local/jail/bin/notify.sh /usr/local/etc/rc.d/notify.sh


patch jail against starting jails with rtprio[edit]

mv /usr/sbin/jail /usr/sbin/jail_
echo '#\!/bin/sh\
/usr/sbin/rtprio -t /usr/sbin/jail_ $*' > /usr/sbin/jail
chmod +x /usr/sbin/jail


make sure mail works[edit]

If there are map errors: 

cd /etc/mail; make maps


recover space on /usr (optional)[edit]

rm -fr /usr/obj


wrapper for jps[edit]

mv /usr/local/sbin/jps /usr/local/sbin/jps_


wrapper for jls[edit]

mv /usr/sbin/jls /usr/sbin/jls_


wrapper for jexec[edit]

mv /usr/sbin/jexec /usr/sbin/jexec_


install jtop[edit]

cd /usr/ports/sysutils/jtop
make install clean


block jails from reaching private net[edit]

echo 'ipfw add 1 deny ip from 69.55.224.0/20 to 10.1.4.0/24' > /usr/local/etc/rc.d/ipfw.sh
chmod 0700 /usr/local/etc/rc.d/ipfw.sh

add to management infrastructure[edit]

mail[edit]

add to management db[edit]

tables: jc.ref_machines and jc.ref_templates

on jail run: 

uname -r

Which shows something like: 

8.3-RELEASE-p2

Insert into db: 

insert into ref_machines values (null,'jail3','mx3.johncompanies.com',0,'f8');
select machine_id from ref_machines where host='jail3';
+------------+
| machine_id |
+------------+
|         35 |
+------------+
insert into ref_templates values ('','8.3-RELEASE-jc2',35,'FreeBSD 8.3',1);

add to bb server[edit]

vi /usr/home/bb/bbsrc/bb1.9i-btf/etc/bb-hosts
10.1.4.109 jail9.johncompanies.com # ssh

In the case of an i2b server, use real ip: 

69.55.229.7 jail3.johncompanies.com # ssh

su bb
cd
bbsrc/bb/runbb.sh restart ; exit

Update backupgraph[edit]

vi /usr/local/www/mgmt/cgi/backupgraph.pl

(add hostname)

Update load mrtg[edit]

vi /usr/local/www/mgmt/mrtg/mrtg1.cfg

(add new entry to file following existing format)

ns1c[edit]

fwd and reverse lookups: 

vr johncompanies.com

vi internal.johncompanies.com
rndc reload johncompanies.com IN private

ptr 69.55.227.x

wiki[edit]

add to server/cabinet map

firewall[edit]

add an outside blocking rule to the firewall, so this machine can only be reached from inside the firewall. Follow example already in firewall jail17 is: 

00117 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.228.200
00117 deny ip from any to 69.55.228.200

jail19 would be 00119... 

ipfw add 00109 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 } to 69.55.232.3
ipfw add 00109 deny ip from any to 69.55.232.3

select some customers for castle probe map[edit]

FreeBSD 8.x[edit]

Assumptions[edit]

Setup instructions below assume this is DELL 2950 with an LSI-based SAS RAID card.

Server is at castle, connected to pub, private, serial and DRAC

Assuming OS loading done via IPKVM with ISO mounted via USB

Assumes at 4 drives, 2 mirrors

Configure server BIOS[edit]

setup console redirect, speed 115200

set LCD string to name of server "jail8"

set date to GMT

go into RAID bios and setup mirrors

configure DRAC: TODO

Install OS (sysinstall)[edit]

boot to bootonly disk for AMD version of FreeBSD, i.e. FreeBSD-8.3-RELEASE-amd64-bootonly.iso

when the install menu appears, choose custom install

partition menu[edit]

move cursor to mfid0, hit space (takes you to partition map screen). If there is only 1 mirror, there will be no option to select a specifit drive: mfid0 will be selected.

type 'a' to use entire disk
type 'q' to quit and save
choose 'freebsd standard mbr'

space to unselect mfid0
cursor down to mfid1
hit space

type 'a' to use entire disk
type 'q' to quit and save
choose 'none' for boot mgr (leave untouched)

cursor over mfid0
space
(takes you into part. Screen again) 'q' to exit
none for boot mgr

Make sure both drives (mfid0 and mfid1) are checked and tab to ok

Label menu[edit]

Make sure mfid0 is highlighted at the top of the screen, setup the following partitions

/ 512M
swap 6G
/var 256M
/tmp 256M
/usr 5G
/mnt/data1 remaining space

All partitions except / should be setup for soft updates. If not, type 's' to enable for soft updates on all except for / (should look like UFS2+S Y under the Newfs column)

move cursor to mfid1 at the top of the screen
swap 8G (or 4G if there’s a 3rd drive)
/mnt/data2 remaining space
'q' to save and exit

distributions[edit]

Choose the following distribudions

  • developer (ok to install ports)
  • custom -> lib32

exit

media[edit]

if you are installing via a cd, no need to enter this menu or change anything. Otherwise, choose ftp to install via ftp. You will be prompted to setup networking. You will need to choose a nic (typically bce0 or bce1). Say no to DHCP and IPv6. Hopefully the public nic cable was installed in bce0 so start with that nic and provide the hostname, (public) IP, netmask, gateway and DNS. When configured, it should start pinging. If it doesn't, have the NOC swap cables. Select any FTP server, usually Main or ftp4.

commit[edit]

this usually takes 12mins
during the process you may need to select a new ftp mirror, this is not a problem.
at the conclution of the install you will be prompted to enter the root password (2x) and returned to the configuration menu.

add user[edit]

Add user 'user'. Defaults for everything is fine, just remember to enter 'wheel' in the member group field. Do set the password.

Setup timezone[edit]

PT

Networking[edit]

page down to the bottom and enable '[X]' sshd

If you installed via cd, you will need to visit:
interfaces->bce0->
No IPV6
dhcp=no
Set hostname, IP, DNS, gateway
(i.e. setup the nic as indicated above)


Exit the install and if you installed via CD, take it out and let the machine reboot

Configure OS, kernel, userland, jail[edit]

double check the date/time[edit]

populate /etc/resolv.conf[edit]

echo "nameserver 69.55.225.225\
nameserver 69.55.230.3
nameserver 69.55.229.3" > /etc/resolv.conf


edit /etc/make.conf[edit]

echo "WITHOUT_X11=yes \
KERNCONF=jail3 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf


setup bootloader for console, etc[edit]

add settings to /boot/loader.conf and /boot.config: 

echo "-Dh" >> /boot.config

echo 'console="comconsole,vidconsole" \
boot_multicons="YES" \
boot_serial="YES" \
mfi_linux_load="YES" \
comconsole_speed="115200"' >> /boot/loader.conf


enable login via serial console[edit]

turn off all ttyv's except 0 and 1 in /etc/ttys and turn on ttyd0, change type to vt100: 

vi /etc/ttys

The changed lines should look like: 

ttyv2   "/usr/libexec/getty Pc"         cons25  off secure
ttyv3   "/usr/libexec/getty Pc"         cons25  off secure
ttyv4   "/usr/libexec/getty Pc"         cons25  off secure
ttyv5   "/usr/libexec/getty Pc"         cons25  off secure
ttyv6   "/usr/libexec/getty Pc"         cons25  off secure
ttyv7   "/usr/libexec/getty Pc"         cons25  off secure
# Serial terminals
# The 'dialup' keyword identifies dialin lines to login, fingerd etc.
ttyu0   "/usr/libexec/getty std.9600"  vt100   on secure

Restart init 

kill -1 1

At this point you should have a login on console.

To configure serial console access, login to the console server as root and run: 

# vi /etc/remote

following examples there, rename port to server's hostname, depending on where and which digi box this server is plugged into. Make sure to get speed right too: 115200


populate hosts[edit]

If server is at castle: 

echo "10.1.4.3 backup2" >> /etc/hosts
echo "10.1.4.8 backup1" >> /etc/hosts

If server is at i2b: 

echo "69.55.230.10 backup2" >> /etc/hosts
echo "10.1.2.3 backup3" >> /etc/hosts
echo "69.55.230.11 backup1" >> /etc/hosts


create ssh key, upload to backup servers[edit]

cd
ssh-keygen -t dsa -b 1024

(default location, leave password blank)

If server is at castle: 

cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys' 
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'

If server is at i2b: 

cat /root/.ssh/id_dsa.pub | ssh backup3 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys' 
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'

confirm that you can ssh to backup2 and backup1 without getting a login prompt 

ssh backup2 hostname
ssh backup1 hostname
ssh backup3 hostname


create & populate binaries/scripts dirs[edit]

mkdir -p /usr/local/jail/bin
mkdir -p /usr/local/jail/rc.d
mkdir -p /usr/local/jail/template/
mkdir /mnt/data1
mkdir /mnt/data2
scp backup2:"/mnt/data4/bin/freebsd8.x/*" /usr/local/jail/bin
cd /usr/local/jail/rc.d/
touch quad1
touch deprecated
chmod +x *
cd /usr/local/jail/bin
ln -s /usr/local/jail/rc.d/quad1 quad1
ln -s /usr/local/jail/bin/jailmake_md jailmake
ln -s /usr/local/jail/bin/js_md js
ln -s /usr/local/jail/bin/canceljail_md canceljail
ln -s /usr/local/jail/bin/jailmakeempty_md jailmakeempty
ln -s /usr/local/jail/bin/postboot_md postboot
ln -s /usr/local/jail/bin/preboot_md preboot
ln -s /usr/local/jail/bin/startjail_md startjail
ln -s /usr/local/jail/bin/stopjail_md stopjail

rehash


edit root's path and login script[edit]

vi /root/.cshrc

Change alias entries (add G): 

alias la        ls -aG
alias lf        ls -FAG
alias ll        ls -lAG
alias ls        ls -AG
alias mbm       mb mount
alias mbu       mb umount
alias cjb       cd /usr/local/jail/bin
alias cd1       cd /mnt/data1
alias cd2       cd /mnt/data2
alias cd3       cd /mnt/data3
alias jtop      jtop lj
alias j         jobs

add to path be careful to leave a space after bin and make sure the wrapping isn't broken: 

/usr/local/jail/bin

alter the prompt, set the following: 

set prompt = "`/bin/hostname -s` %/# "

at the bottom of the file add: 

set sshtty=`who am i|awk '{print $2}'`
/usr/sbin/rtprio 3 -`psj | grep $sshtty | awk '{print $2}'`

set shortty=`who am i | awk '{print $2}' | sed -E 's/.*(..)$/\1/'`
foreach x (`psj | grep sh | grep $shortty | awk '{print $2}'`)
/usr/sbin/rtprio 2 -$x
end

Make the new settings active in current shell: 

source /root/.cshrc


install cvsup[edit]

cd /usr/ports/net/cvsup-without-gui 
make install clean; rehash; mail -s 'cvs installed' support@johncompanies.com < /dev/null

stand by for gettext options (use defaults). this process takes approx 22mins- hence the email/page notice above.


get latest sources for this release[edit]

cd /usr/src 
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_8_3\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

If you need to run stable (cause release is broken or some other reason) make the sup file look like: 

echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_8\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

cvsup sup ; mail -s 'cvs sup done' support@johncompanies.com < /dev/null

time varies, 10-20mins


configure new kernel[edit]

Pull down the kernel config we are using for this distribution. In this case we use an 8.2 kernel config on 8.3, which is valid. The local file should be the same name as host- jail3 in this example 

cd /usr/src/sys/amd64/conf 
scp backup2:/mnt/data4/build/freebsd/kern_config-8.2-amd64 ./jail3

edit the kernel config and change ident to be the name of the jail: 

vi jail3
ident           jail3

Optional, edit /sys/conf/newvers.sh to add –jc2 to the end of the BRANCH string (RELEASE-jc2) 

vi /sys/conf/newvers.sh

notes on kernel configuring: http://www.FreeBSD.org/doc/en_US.ISO8859-1/books/handbook/kernelconfig-config.html


install patches[edit]

The various patches are in /mnt/data4/build/freebsd/patches on backup2. There are dirs for each version. Not all dirs are populated, but patches for later versions work on older ones unless there is a new patch in the older dir.

there are no patches we use for 8.x, but here would be the commands: 

cd /usr/src
scp backup2:"/mnt/data4/build/freebsd/patches/8.0/*" .

Apply patches, i.e. the jls-patch: 

patch -l < jls-patch


build, install kernel and world[edit]

Rename current generic kernel so it will always be available to boot from. Save room by removing non-needed kernel modules: 

cd /boot
mv kernel kernel.GENERIC
cd kernel.GENERIC
mkdir hold
mv mfi_linux.ko hold/
mv linux.ko hold/
mv linprocfs.ko hold/
mv linsysfs.ko hold/
mv geom_vinum.ko hold/
mv geom_concat.ko hold/
mv zfs.* hold/
mv opensolaris* hold/

rm *.ko
rm *.symbols
mv hold/* .
rmdir hold/

cd /usr/src
make buildkernel installkernel

make buildworld ; mail -s 'buildworld done' dave.boodman@vtext.com < /dev/null

~38mins 

make installworld

~34min 

mergemaster -i

You will be prompted to merge, replace or ignore files changed by the src update. In most cases you can delete the temp (new) files.

ONLY if this will be a zfs system (not currently used in 8.x): 

cd /sys/modules/zfs
make 
make install
cd /sys/modules/opensolaris
make 
make install


populate devfs ruleset[edit]

scp backup2:/mnt/data4/build/freebsd/devfs.rules.8x /etc/devfs.rules


populate /etc/rc.conf with IPs and service settings[edit]

vi /etc/rc.conf

kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"

nfs_client_enable="YES"
nfs_reserved_port_only="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.4.103"
devfs_system_ruleset="devfsrules_show_all"

ifconfig_bce1="inet 10.1.4.103 netmask 255.255.255.0"
ifconfig_bce0="inet 69.55.229.7 netmask 255.255.255.0"
#ifconfig_bce0_alias0="inet 69.55.2xx.xx netmask 255.255.255.0"

fsck_y_enable="YES"
background_fsck="NO"
#rc_mfi_raid_tty_log="YES"
#zfs_enable="YES"

Modify IPs, hostname, gateway for this box.

make sure sysctls are set and preserved upon boot[edit]

echo "kern.consmute=0\
kern.ipc.shm_use_phys=1\
kern.ipc.shmall=131070\
kern.ipc.shmmax=134217728\
net.inet.tcp.syncookies=0\
kern.maxfiles=32768\
kern.fallback_elf_brand=3\
kern.maxprocperuid=4000\
security.jail.sysvipc_allowed=1\
security.jail.allow_raw_sockets=1\
security.jail.socket_unixiproute_only=1\
security.jail.chflags_allowed=0\
dev.amr.0.allow_volume_configure=1\
compat.linux.osrelease=2.6.12\
vm.pmap.shpgperproc=500\
security.bsd.unprivileged_read_msgbuf=0\
kern.maxvnodes=400000" >> /etc/sysctl.conf

Tuning note: watch vfs.numvnodes while the server is live to get guidance on where to set maxvnodes


mount procfs[edit]

echo "proc                    /proc           procfs  rw              0       0" >> /etc/fstab
echo "linprocfs               /usr/compat/linux/proc linprocfs rw     0       0" >> /etc/fstab
echo "linsysfs                /usr/compat/linux/sys linsysfs rw       0       0" >> /etc/fstab
mkdir -p /usr/compat/linux/proc
mkdir -p /usr/compat/linux/sys


enable noatime option[edit]

data1 and data2 should look something like (add ',noatime' after 'rw'): 

/dev/mfid0s1g           /mnt/data1      ufs     rw,noatime      2       2
/dev/mfid1s1d           /mnt/data2      ufs     rw,noatime      2       2


reboot. Confirm new kernel is loaded, devfs in place[edit]

uname -a

Check devfs rules 

devfs rule showsets
devfs rule -s 3 show

Should see: 

#  devfs rule showsets
1
2
3
4

#  devfs rule -s 3 show
100 include 1
207 path pts* unhide
217 path fd unhide
218 path fd/* unhide
300 path ttyp* unhide
301 path ttyq* unhide
302 path ttyr* unhide
303 path ttys* unhide
304 path ttyP* unhide
305 path ttyQ* unhide
306 path ttyR* unhide
307 path ttyS* unhide
400 path null unhide
500 path zero unhide
600 path random unhide
610 path urandom unhide
700 path mem unhide
710 path kmem unhide
810 path mdctl unhide
900 path stdin unhide
910 path stdout unhide
920 path stderr unhide


update ports[edit]

cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_8_3\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup

cvsup sup; mail -s 'cvs sup ports done' support@johncompanies.com < /dev/null

~24mins


Install raid mgmt tool[edit]

Perc5/i, 6/i[edit]

Pull over cli from previous system (jail9) 

scp /usr/local/sbin/mega* root@10.1.4.109:/usr/local/sbin/
scp /usr/local/libexec/MegaCli root@10.1.4.109:/usr/local/libexec/MegaCli

These are linux-based tools. This will require linux base which we would normally install from ports, but since it's failing lately, we just pull in the compat libraries from another system: 

rsync -aSHv --exclude=proc --exclude=sys 10.1.4.109:/usr/compat/linux/ /usr/compat/linux/

Test: 

rehash; megacli ldinfo lall a0

DEPRECATED: Assuming it worked, here's how we used to install linux_base: 

cd /usr/ports/emulators/linux_base-fc4
make install clean

(didnt succeed due to libtool requirement) 

cd /usr/ports/distfiles
fetch http://www.lsi.com/DistributionSystem/AssetDocument/support/downloads/megaraid/miscellaneous/linux/2.00.15_Linux_MegaCLI.zip
cd /usr/ports/sysutils/linux-megacli
make install clean

(also failed due to libtool)


2850 PERC 4e/Di- no linux[edit]

cd /usr/ports/distfiles/
fetch http://backup01.best-hosting.ru/pub/FreeBSD/ports/distfiles/dr_freebsd_1.51.zip
cd /usr/ports/sysutils/megarc
make install clean
megarc -dispCfg -a0


install rsync from ports[edit]

cd /usr/ports/net/rsync
make install clean

choose default options


configure inetd to respond to mrtg load queries[edit]

echo "load    stream  tcp     nowait  user    /usr/local/jail/bin/load.pl  load.pl" >> /etc/inetd.conf
echo "load            12384/tcp" >> /etc/services


install perl[edit]

cd /usr/ports/lang/perl5.12
make install clean

choose defaults


install bb client[edit]

Compiling from source on AMD64 will not work. So, we use a linux-compiled version and rely on linux compat. Per above, linux compat won't install on 8.x - libtool 2.4 need. So, instead we copy(ed) over linux: 

rsync -aSHv --exclude=proc --exclude=sys 10.1.4.108:/usr/compat/linux/ /usr/compat/linux/

adduser

Output/response: 

Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]: 
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username   : bb
Password   : <random>
Full Name  : bb
Uid        : 1984
Class      :
Groups     : bb
Home       : /home/bb
Shell      : /bin/sh
Locked     : no
OK? (yes/no): yes
cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd_linuxcompat.tgz .
tar xzf bb-freebsd_linuxcompat.tgz

edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like: 

echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.4.103 jail3.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

Edit for machine name and private IP.

if this machine is at i2b: 

echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
69.55.229.7 jail3.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

vi /home/bb/bbc1.9e-btf/ext/openfiles 

MACHINE="jail3,johncompanies,com"      # HAS TO BE IN A,B,C FORM

Edit for machine name. 

cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh 
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf
./runbb.sh start
more BBOUT 
(look for errors)
exit

Put in script to start bb @ boot: 

echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh

remove reserve space[edit]

cd
umount /mnt/data1
umount /mnt/data2
tunefs -m 0 /mnt/data1
tunefs -m 0 /mnt/data2
mount -a


setup rdate[edit]

cd /usr/ports/sysutils/rdate
make install clean

crontab -e
0 0 * * * /usr/local/sbin/rdate -s utcnist.colorado.edu

/usr/local/sbin/rdate -s utcnist.colorado.edu

We used to use ntpd, however it listens on jail IPs which is a security risk. So we stopped. Here's the old instructions: install new ntp from ports 

/usr/ports/net/ntp
make install clean

echo "server 10.1.4.5" > /etc/ntp.conf
/usr/sbin/ntpd -p /var/run/ntpd.pid
sleep 2; ntpq -p

(confirm it’s able to reach our time server)


make a data partition[edit]

IF you didn't format the data partition during sysinstall:

Create a g partition on 2nd mirror – bsdlabel no longer works (below shows d partition made with sysinstall): 

jail8 /usr/home/bb# gpart show
=>       63  285474735  mfid0  MBR  (136G)
         63  285458922      1  freebsd  [active]  (136G)
  285458985      15813         - free -  (7.7M)

=>        0  285458922  mfid0s1  BSD  (136G)
          0     524288        1  freebsd-ufs  (256M)
     524288   12582912        2  freebsd-swap  (6.0G)
   13107200     524288        4  freebsd-ufs  (256M)
   13631488     524288        5  freebsd-ufs  (256M)
   14155776    8388608        6  freebsd-ufs  (4.0G)
   22544384  262914538        7  freebsd-ufs  (125G)

=>       63  584843175  mfid1  MBR  (279G)
         63  584830197      1  freebsd  [active]  (279G)
  584830260      12978         - free -  (6.3M)

=>        0  584830197  mfid1s1  BSD  (279G)
          0   16777216        2  freebsd-swap  (8.0G)
   16777216  568052981        4  freebsd-ufs  (271G)

jail8 /usr/home/bb# gpart show mfid1s1
=>        0  584830197  mfid1s1  BSD  (279G)
          0   16777216        2  freebsd-swap  (8.0G)
   16777216  568052981        4  freebsd-ufs  (271G)

# gpart list mfid1s1

Geom name: mfid1s1
fwheads: 255
fwsectors: 63
last: 584830196
first: 0
entries: 8
scheme: BSD
Providers:
1. Name: mfid1s1b
   Mediasize: 8589934592 (8.0G)
   Sectorsize: 512
   Mode: r1w1e0
   rawtype: 1
   length: 8589934592
   offset: 0
   type: freebsd-swap
   index: 2
   end: 16777215
   start: 0
2. Name: mfid1s1d
   Mediasize: 290843126272 (271G)
   Sectorsize: 512
   Mode: r0w0e0
   rawtype: 7
   length: 290843126272
   offset: 8589934592
   type: freebsd-ufs
   index: 4
   end: 584830196
   start: 16777216
Consumers:
1. Name: mfid1s1
   Mediasize: 299433060864 (279G)
   Sectorsize: 512
   Mode: r1w1e1

# gpart delete -i 4 mfid1s1
mfid1s1d deleted
jail8 /usr/home/bb# gpart list mfid1s1
Geom name: mfid1s1
fwheads: 255
fwsectors: 63
last: 584830196
first: 0
entries: 8
scheme: BSD
Providers:
1. Name: mfid1s1b
   Mediasize: 8589934592 (8.0G)
   Sectorsize: 512
   Mode: r1w1e0
   rawtype: 1
   length: 8589934592
   offset: 0
   type: freebsd-swap
   index: 2
   end: 16777215
   start: 0
Consumers:
1. Name: mfid1s1
   Mediasize: 299433060864 (279G)
   Sectorsize: 512
   Mode: r1w1e1

# gpart show mfid1s1
=>        0  584830197  mfid1s1  BSD  (279G)
          0   16777216        2  freebsd-swap  (8.0G)
   16777216  568052981           - free -  (271G)

# gpart add  -t freebsd-ufs -i 7 mfid1s1
mfid1s1g added

# gpart show mfid1s1
=>        0  584830197  mfid1s1  BSD  (279G)
          0   16777216        2  freebsd-swap  (8.0G)
   16777216  568052981        7  freebsd-ufs  (271G)

Here's how we USED to do it with bsdlabel: 

bsdlabel -e /dev/mfid0s1

given: 

# /dev/aacd0s1:
8 partitions:
#        size   offset    fstype   [fsize bsize bps/cpg]
  a:   262144        0    4.2BSD     2048 16384 16392
  b:  4194304   262144      swap
  c: 143363997        0    unused        0     0         # "raw" part, don't edit
  d:   524288  4456448    4.2BSD     2048 16384 32776
  e:   524288  4980736    4.2BSD     2048 16384 32776
  f:  6291456  5505024    4.2BSD     2048 16384 28552

new offset = 6291456 + 5505024 = 11796480
new size is size for 'c' partition minus the new start from above
143363997 - 11796480 = 131567517

So:

g: 131567517 11796480 unused 0 0


create the jail template[edit]

cd /usr/ports/sysutils/jailutils make install clean

Create an md device to hold the jail: 

touch /mnt/data1/jail-template20g
mdconfig -a -t vnode -s 20g -f /mnt/data1/jail-template20g -u 0
newfs -O 1 /dev/md0
mkdir /mnt/data1/jail-DIR
mount /dev/md0 /mnt/data1/jail-DIR

Build world into the jail: 

cd /usr/src
make world DESTDIR=/mnt/data1/jail-DIR; taskdone

~1hr

Make /etc into the jail, mount dev, copy in jkill: 

cd etc
make distribution DESTDIR=/mnt/data1/jail-DIR
mount -t devfs devfs /mnt/data1/jail-DIR/dev
devfs -m /mnt/data1/jail-DIR/dev rule -s 3 applyset 
cd /mnt/data1/jail-DIR
ln -sf dev/null kernel
cp /usr/local/sbin/jkill /mnt/data1/jail-DIR/sbin

Enter into jail to do configuration: 

jail /mnt/data1/jail-DIR testhostname 192.168.11.100 /bin/sh
csh

Create fstab: 

touch /etc/fstab
echo 'network_interfaces=""\
hostname="newsystem"\
kern_securelevel_enable="NO"\
sendmail_enable="YES"\
sshd_enable="YES"' > /etc/rc.conf

echo "nameserver 69.55.225.225\
nameserver 69.55.230.3" >> /etc/resolv.conf

Edit crontab: 

vi /etc/crontab

remove the adjkerntz lines comment out periodic’s and put this line above them: 

# IF YOU UNCOMMENT THESE, PLEASE ADJUST THEIR RUN TIME

rm -rf /etc/periodic/daily/400.status-disks

check and remove any crap in /tmp 

vi /etc/periodic/security/100.chksetuid

replace: MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`
with: MP='/' (use single quotes) 

mkdir -p /usr/compat/linux/dev

Add account for user. Output/response: 

adduser

Username: user
Full name: user
Uid (Leave empty for default):
Login group [user]:
Login group is user. Invite user into other groups? []: wheel
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/user]:
Home directory permissions (Leave empty for default):
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: y
Lock out the account after creation? [no]:
Username   : user
Password   : <random>
Full Name  : user
Uid        : 1001
Class      :
Groups     : user
Home       : /home/user
Home Mode  :
Shell      : /bin/sh
Locked     : no
OK? (yes/no): y
adduser: INFO: Successfully added (user) to the user database.
adduser: INFO: Password for (user) is: 901gmYjO
Add another user? (yes/no): n
Goodbye!

set TERM: 

vi /usr/home/user/.profile 
TERM=vt100;     export TERM

Set time zone to PT: 

tzsetup

Reload aliases: 

newaliases

Replace reboot/halt: 

rm /sbin/halt /sbin/reboot
ln /sbin/jkill /sbin/halt
ln /sbin/jkill /sbin/reboot

Redirect console output, comment out console and move to /var/log/messages: 

vi /etc/syslog.conf
#*.err;kern.warning;auth.notice;mail.crit               /dev/console
*.err;kern.warning;auth.notice;mail.crit                /var/log/messages 

exit
exit

Lib32 compat library: 

cd libexec
chflags noschg ld-elf32.so.1
chflags noschg ld-elf.so.1
mv ld-elf32.so.1 ld-elf32.so.1-orig
ln ld-elf.so.1 ld-elf32.so.1
chflags schg ld-elf.so.1
chflags schg ld-elf32.so.1

Replace traceroute: 

mv /mnt/data1/jail-DIR/usr/sbin/traceroute /mnt/data1/jail-DIR/usr/sbin/_traceroute
echo '#\!/bin/sh\
/usr/sbin/_traceroute -i bce0 $1' >> /mnt/data1/jail-DIR/usr/sbin/traceroute 
chmod +x /mnt/data1/jail-DIR/usr/sbin/traceroute

Modify 'bce0' to reflect whichever nic is public on this hardware.

Clean out ports before copying into the jail (~30mins): 

cd /usr/ports
make -DNOCLEANDEPENDS clean 
rm -fr /usr/ports/distfiles/*
cp -r /usr/ports /mnt/data1/jail-DIR/usr 

rm /mnt/data1/jail-DIR/root/.history

Umount the jail and dump it: 

cd /
umount /mnt/data1/jail-DIR/dev
dump -0a -f /usr/local/jail/template/template /dev/md0
umount /dev/md0
rmdir /mnt/data1/jail-DIR
mdconfig -d -u 0


setup backups[edit]

echo '#\!/bin/sh\
backupdir=/data/jail3\
server=backup1\
\
## ENTRY /etc\
## ENTRY /usr/local/etc\
## ENTRY /usr/local/jail\
## ENTRY /root/logs' > /usr/local/jail/bin/backup.config

Edit to reflect backup server and jail hostname

On backup server, setup backup dirs: 

ssh backup1 mkdir -p /data/jail3/0

On backup server, setup backup dirs: 

backup1# vi /usr/local/sbin/snapshot_rotate

Edit /usr/local/jail/bin/backup.xxx to use the right drives and copy to /usr/local/jail/bin/backup 

vi /usr/local/jail/bin/backup.md

adjust df so it includes all relevant drives, currently df > /etc/df.bak is fine. Also, make sure the binary/script source is pulling from the right dir on backup2: 

/usr/local/bin/rsync -a backup2:/mnt/data4/bin/freebsd8.x/ /usr/local/jail/bin/

cp /usr/local/jail/bin/backup.md /usr/local/jail/bin/backup


create /root/logs[edit]

mkdir /root/logs


edit sshd_config[edit]

vi /etc/ssh/sshd_config
ListenAddress 69.55.229.7
ListenAddress 10.1.4.103

Adjust to pub/private IP for jail. Restart sshd: 

kill -1 `cat /var/run/sshd.pid`

add crontab entries[edit]

crontab -e
5 0 * * * /usr/local/jail/bin/backup
1 0 1 * * /usr/local/jail/bin/ipfwreset
0 18 * * * /usr/local/jail/bin/ipfwbackup
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /usr/local/jail/bin/trafstats
0 0,6,12,18 * * * /usr/local/jail/bin/sync_jail_names

On 2950: 

*/5 * * * * /usr/local/jail/bin/perc5iraidchk

On 3ware-based servers: 

0 0 * * * /usr/local/jail/bin/3wraidchk


Reboot notify script[edit]

ln -s /usr/local/jail/bin/notify.sh /usr/local/etc/rc.d/notify.sh


patch jail against starting jails with rtprio[edit]

mv /usr/sbin/jail /usr/sbin/jail_
echo '#\!/bin/sh\
/usr/sbin/rtprio -t /usr/sbin/jail_ $*' > /usr/sbin/jail
chmod +x /usr/sbin/jail


make sure mail works[edit]

If there are map errors: 

cd /etc/mail; make maps


recover space on /usr (optional)[edit]

rm -fr /usr/obj


wrapper for jps[edit]

mv /usr/local/sbin/jps /usr/local/sbin/jps_


wrapper for jls[edit]

mv /usr/sbin/jls /usr/sbin/jls_


wrapper for jexec[edit]

mv /usr/sbin/jexec /usr/sbin/jexec_


install jtop[edit]

cd /usr/ports/sysutils/jtop
make install clean


block jails from reaching private net[edit]

echo 'ipfw add 1 deny ip from 69.55.224.0/20 to 10.1.4.0/24' > /usr/local/etc/rc.d/ipfw.sh
chmod 0700 /usr/local/etc/rc.d/ipfw.sh

add to management infrastructure[edit]

mail[edit]

add to management db[edit]

tables: jc.ref_machines and jc.ref_templates

on jail run: 

uname -r

Which shows something like: 

8.3-RELEASE-p2

Insert into db: 

insert into ref_machines values (null,'jail3','mx3.johncompanies.com',0,'f8');
select machine_id from ref_machines where host='jail3';
+------------+
| machine_id |
+------------+
|         35 |
+------------+
insert into ref_templates values ('','8.3-RELEASE-jc2',35,'FreeBSD 8.3',1);

add to bb server[edit]

vi /usr/home/bb/bbsrc/bb1.9i-btf/etc/bb-hosts
10.1.4.109 jail9.johncompanies.com # ssh

In the case of an i2b server, use real ip: 

69.55.229.7 jail3.johncompanies.com # ssh

su bb
cd
bbsrc/bb/runbb.sh restart ; exit

Update backupgraph[edit]

vi /usr/local/www/mgmt/cgi/backupgraph.pl

(add hostname)

Update load mrtg[edit]

vi /usr/local/www/mgmt/mrtg/mrtg1.cfg

(add new entry to file following existing format)

ns1c[edit]

fwd and reverse lookups: 

vr johncompanies.com

vi internal.johncompanies.com
rndc reload johncompanies.com IN private

ptr 69.55.227.x

wiki[edit]

add to server/cabinet map

firewall[edit]

add an outside blocking rule to the firewall, so this machine can only be reached from inside the firewall. Follow example already in firewall jail17 is: 

00117 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.228.200
00117 deny ip from any to 69.55.228.200

jail19 would be 00119... 

ipfw add 00109 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 } to 69.55.232.3
ipfw add 00109 deny ip from any to 69.55.232.3

select some customers for castle probe map[edit]

FreeBSD 4.11[edit]

4.11
Last updated 2006-1-26

All time extimates below assume disks aren’t scrubbing

1. make sure bios is setup for bios console redirect
Supermicro:
Console redirection:
Com port addr: on-board COM A
Baud: 38400
Console type: vt100
Flow control: none
Console connection: direct
Continue cr after post: off

2450:
Make sure running bios A09
Console Redirection: VT100/VT220
	
2. assuming mirrors (or at least disks) created (if not, refer to this), boot to disk 1 of 4.11 
skip kernel config (enter)
custom install
partition
move cursor to aacd0, hit space (takes you to partition map screen)
a for entire disk
q to quit and save
standard mbr (no boot manager)
space to unselect aacd0
cursor over aacd1
space
a for entire disk
q to quit and save
none (leave untouched)

cursor over aacd0
space
(takes you into part. Screen again) q to exit
none
Make sure both are checked and tab to ok

label
Make sure aacd0 is highlighted
a to start with defaults

john likes:
/1g
/var 256m

Glenn likes:
/128
/var 256
/usr 3g
/tmp 256

We do:
delete everything
/ 128M
swap 4G
/var 256M
/tmp 256M
/usr 3G
/mnt/data1 remaining space

Make sure to toggle S for soft updates on all (should look like UFS+S Y under the Newfs column)
Set the noatime option

move cursor to aacd1
swap 4G
/mnt/data2 remaining space
Set the noatime option

q to save and exit

distributions
developer
yes to install ports
exit
media
cd
commit
yes
(2450: 16mins, supermicro: 11mins)
yes to "visit general config"
Set root pwd
Add user ‘user’ member group is wheel, set password
Set tz
Networking->interfaces->Fxp0
No IPV6
dhcp=yes
Set hostname & domain

Startup services:
Disble usbd

exit...
exit install
yes
take the cd out and let the machine reboot

3. put some temp settings in /etc/rc.conf:
usbd_enable="NO"
sendmail_enable="NO"

4. reboot

5. double check the date/time

6. edit /etc/make.conf (only add the console speed line if this is a supermicro capable of outputting BIOS redirect at that speed – 2450’s can only do 9600)
cat >> /etc/make.conf
WITHOUT_X11=yes
KERNCONF=jail18
BOOT_COMCONSOLE_SPEED=38400

7. install cvsup
cd /usr/ports/net/cvsup-without-gui 
make install clean (stay close for gettext options, 2450: 21mins, supermicro: 14mins)
rehash

8. get latest sources for this release:
cd /usr/src 
cat > sup
*default host=cvsup4.freebsd.org
*default base=/usr
*default prefix=/usr
*default release=cvs tag=RELENG_4_11
*default delete use-rel-suffix
*default compress
src-all

cvsup sup (2450, 4.10: 13mins, supermicro, 4.11: 11mins)

9. populate hosts
cat >> /etc/hosts
10.1.4.3	backup2

10. put key in authorized_keys on backup2
ssh-keygen -t dsa -b 1024 (default location, leave password blank)
scp /root/.ssh/id_dsa.pub user@backup2:/tmp/jail18pub
on backup2:
cat /tmp/jail18pub >> /root/.ssh/authorized_keys

confirm that you can ssh to backup2 without getting a login prompt

11. configure new kernel. Get config from similar machine or there may be a master copy somewhere under /mnt/data4/build (name the kernel config the same as the jail, ex jail4):
cd /usr/src/sys/i386/conf 
scp backup2:/mnt/data4/build/freebsd/kern_config-4.11 ./jail18

edit the kernel config and change ident to be the name of the jail:
ident           jail4

IMPORTANT CUSTOMIZATION:
for machines with >4G RAM, add to the config:
options         PAE

for supermicro mobo’s with broadcom nics, add to the config:
device          bge             # Broadcom BCM570x (``Tigon III'')

for machines where lots of postgres might be running, change SHMMAXPGS:
options         SHMMAXPGS=40960

edit /sys/conf/newvers.sh to add –jc2 to the end of the BRANCH string (RELEASE-p9-jc2)

12. bring over patches from backup2
The various patches are in /mnt/data4/build/freebsd/patches on backup2. There are dirs for each version. Not all dirs are populated, but patches for later versions work on older ones unless there is a new patch in the older dir.
cd /usr/src
scp backup2:"/mnt/data4/build/freebsd/patches/4.11/*" .
jail_proc_patch      100% |*************************************************|  2593    00:00
restore-patch        100% |*************************************************|  3295    00:00
411ps-jail-patch     100% |*************************************************|  2602    00:00
jail_rtprio_patch    100% |*************************************************|   301    00:00
udp-patch            100% |*************************************************|   594    00:00

Apply patches:
patch < 411ps-jail-patch
patch < jail_proc_patch
patch < restore-patch
patch < jail_rtprio_patch
patch < udp-patch

13. build, install kernel and world
make buildworld buildkernel installkernel (2450: 48min, supermicro: 20mins)
(Any compile errors can be looked up in /usr/include/sys/signal.h, other errors, do a rm -R /usr/obj/*)
make installworld (2450: 2min, supermicro: 1mins)
mergemaster -i
(answer no to most of it)

14. reboot. Confirm new kernel is loaded (uname -a)

15. update ports:
cd /usr/ports
cat > sup
*default host=cvsup4.FreeBSD.org
*default base=/usr
*default prefix=/usr
*default release=cvs tag=RELENG_4
*default delete use-rel-suffix
*default compress
ports-all tag=.

cvsup sup (2450: 26mins, supermicro: 26mins)

16. add console="comconsole" to /boot/loader.conf
cat >> /boot/loader.conf
console="comconsole"

17. edit /etc/ttys and turn off all ttyv's except 0 and 1
also turn on ttyd0, change type to vt100:
ttyd0   "/usr/libexec/getty std.9600"  vt100   on secure

kill -1 1

on console:
vi /etc/remote
(rename port to jail4 depending on where and which digi plugged into)
test serial console

18. install linux_base:
cd /usr/ports/emulators/linux_base
make install clean (2450: 7min, supermicro: 2mins)

reibuild rpmdb cause we had probs installing aacapps
cd /compat/linux/bin
./rpm --initdb
./rpm --rebuilddb

19. install aacapps-4.1-0.i386.rpm
scp backup2:/mnt/data4/build/freebsd/aacapps-4.1-0.i386.rpm /tmp/.
/compat/linux/bin/rpm -ivh /tmp/aacapps-4.1-0.i386.rpm
cd /dev
sh MAKEDEV aac0

test out /compat/linux/usr/sbin/aaccli

20. edit root's path and login script:
vi /root/.cshrc
add to path: /usr/local/jail/bin /compat/linux/usr/sbin

Change alias entries (add G):
alias la        ls -aG
alias lf        ls -FAG
alias ll        ls -lAG
alias ls        ls -AG
alias mbm       mb mount
alias mbu       mb umount

and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "

at the bottom of the file add:
set sshtty=`who am i|awk '{print $2}'`
/usr/sbin/rtprio 3 -`ps auxwJ | grep $sshtty | awk '{print $2}'`

set shortty=`who am i | awk '{print $2}' | sed -E 's/.*(..)$/\1/'`
foreach x (`ps cauxJ | grep sh | grep $shortty | awk '{print $2}'`)
/usr/sbin/rtprio 2 -$x
end
 
21. install rsync from ports
cd /usr/ports/net/rsync
make install clean

22. create & populate binaries/scripts dirs
mkdir -p /usr/local/jail/bin
mkdir -p /usr/local/jail/rc.d
scp backup2:"/mnt/data4/bin/freebsd/*" /usr/local/jail/bin
cd /usr/local/jail/rc.d/
touch quad1
touch quad2
touch quad3
touch quad4
touch safe1
touch safe2
touch safe3
touch safe4
chmod +x *
cd /usr/local/jail/bin
ln -s /usr/local/jail/rc.d/quad1 quad1
ln -s /usr/local/jail/rc.d/quad2 quad2
ln -s /usr/local/jail/rc.d/quad3 quad3
ln -s /usr/local/jail/rc.d/quad4 quad4
ln -s /usr/local/jail/rc.d/safe1 safe1
ln -s /usr/local/jail/rc.d/safe2 safe2
ln -s /usr/local/jail/rc.d/safe3 safe3
ln -s /usr/local/jail/rc.d/safe4 safe4
rehash

23. configure inetd to respond to mrtg load queries
cat >> /etc/inetd.conf
load    stream  tcp     nowait  user    /usr/local/jail/bin/load.pl  load.pl

cat >> /etc/services
load            12384/tcp

kill -HUP `cat /var/run/inetd.pid`

24. configure load mrtg, on mail
vi /usr/local/etc/mrtg/mrtg1.cfg
(add new entry to file following existing format)

25. configure bb on mail:
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
10.1.4.104 jail4.johncompanies.com # ssh

su bb
cd /usr/home/bb/bbsrc/bb1.9e-btf
./runbb.sh stop
./runbb.sh start
exit

26. install bb client
adduser -group 1984 -shell /bin/csh -uid 1984 bb
cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd.tar .
tar xvf bb-freebsd.tar

cat > /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
10.1.4.105 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh
10.1.4.118 jail18.johncompanies.com # ssh

vi /home/bb/bbc1.9e-btf/ext/openfiles   and change:
MACHINE="jail18,johncompanies,com"      # HAS TO BE IN A,B,C FORM

cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
./bbchkhosts.sh (ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd bbc1.9e-btf/
./runbb.sh start
more BBOUT (look for errors)
exit

cat > /usr/local/etc/rc.d/bb.sh
su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"

chmod +x /usr/local/etc/rc.d/bb.sh

27. remove reserve space, enable softupdates (probably already set, so not necessary)
cd
umount /mnt/data1
umount /mnt/data2
tunefs -m 0 /mnt/data1
tunefs -m 0 /mnt/data2

mount -a

28. configure ntp
cat > /etc/ntp.conf
server 10.1.4.105

/usr/sbin/ntpd -p /var/run/ntpd.pid
ntpq -p
(confirm it’s able to reach our time server)

29. mrtg switch graphs

31. fwd and reverse lookups on ns1c
vi johncompanies.com
rr johncompanies.com
vi internal.johncompanies.com
rndc reload johncompanies.com IN private
(edit the PTR too)

32. create all /dev/vn and /dev/pty files in /dev
cat > /tmp/runme.sh
#!/bin/sh
cd /dev
for i in 1 2 3 4 5 6 7 ; do sh MAKEDEV pty$i ; done
for i in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 ; do sh MAKEDEV vn$i ; done

NOTE: there are only 4 lines in the above output - however the lines may be wrapped on your terminal - make sure that there are only four lines in the script you make.

chmod +x /tmp/runme.sh
/tmp/runme.sh

rm /tmp/runme.sh

ls /dev/vn*|wc -l (make sure the output is a high number – larger than 128)
ls /dev/pty*|wc -l (make sure the output is exactly 256)

33. populate /etc/rc.conf with IPs and NFS settings
vi /etc/rc.conf

kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="NO"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
sshd_enable="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.4.104"

ifconfig_xl0="inet 10.1.4.104 netmask 255.255.255.0"
ifconfig_fxp0="inet 69.55.228.101 netmask 255.255.255.0"
defaultrouter="69.55.228.1"
ifconfig_fxp0_alias0="inet 69.55.2xx.xx netmask 255.255.255.0"

static_routes="t1 office"
route_t1="-net 10.1.5 10.1.4.2"
route_office="-net 10.1.6 10.1.4.2"

34. make sure sysctls are set and preserved after reboot
cat >> /etc/sysctl.conf
kern.consmute=0
jail.sysvipc_allowed=1
kern.ipc.shm_use_phys=1
kern.ipc.shmall=65535
kern.ipc.shmmax=134217728
net.inet.tcp.syncookies=0
kern.maxfiles=32768
kern.fallback_elf_brand=3
kern.maxprocperuid=4000
jail.max_procs_per_jail: 1026

35. reboot

36. create the jail template
vnconfig -T -S 1g -Z -s labels -c /dev/vn1 /mnt/data1/jail
disklabel -r -w vn1 auto
newfs /dev/vn1c
cd /usr/src
mkdir -p /mnt/data1/jail-DIR
mount /dev/vn1c /mnt/data1/jail-DIR
make world DESTDIR=/mnt/data1/jail-DIR (2450: 45mins, supermicro: 19mins)
cd etc
make distribution DESTDIR=/mnt/data1/jail-DIR -DNO_MAKEDEV_RUN
cd /mnt/data1/jail-DIR/dev
sh MAKEDEV jail
cd /mnt/data1/jail-DIR
ln -sf dev/null kernel

jail /mnt/data1/jail-DIR testhostname 192.168.11.100 /bin/sh
csh
touch /etc/fstab
cat > /etc/rc.conf
portmap_enable="NO"
network_interfaces=""
hostname="newsystem"
kern_securelevel_enable="NO"
sendmail_enable="YES"
sshd_enable="YES"

cat >> /etc/resolv.conf
nameserver 69.55.225.225
nameserver 69.55.230.3

vi /etc/crontab
remove the adjkerntz lines
comment out periodic’s and put this line above them:
# DO NOT UNCOMMENT THESE

rm -rf /etc/periodic/daily/400.status-disks

mv /bin/df /bin/df_

cat > /bin/df
#!/bin/sh
/bin/df_ $* .

chmod +x /bin/df

cat > /sbin/mount
echo `df | tail -1 | awk '{print $1 " on " $6 " (ufs, local)"}'`

check /tmp for crap

vi /etc/periodic/security/100.chksetuid
replace: MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`
with: MP='/'

mkdir -p /usr/compat/linux/dev

adduser (Add account for user)

put user in wheel group
vi /etc/group

vi /usr/home/user/.profile (and add to the file):
TERM=vt100;     export TERM

cd /etc
vipw -d .
root:$1$krszPxhk$xkCepSnz3mIikT3vCtJCt0:0:0::0:0:Charlie &:/root:/bin/csh
user:$1$Mx9p5Npk$QdMU6c8YQqp2FW2M3irEh/:1001:1001::0:0:User &:/home/user:/bin/sh

tzsetup

newaliases 

chflags schg /dev/*mem

cd /dev
rm console
ln -s null console

exit
exit

cd /usr/ports
make -DNOCLEANDEPENDS clean (2450: 47mins , supermicro: 17mins)
rm -fr /usr/ports/distfiles/*
cp -r /usr/ports /mnt/data1/jail-DIR/usr

cd /mnt/data1/jail-DIR/dev
rm kmem
mknod kmem c 2 1 root:kmem
chmod 640 kmem
rm mem
mknod mem c 2 0 root:kmem
chmod 640 mem

rm /mnt/data1/jail-DIR/root/.history

sh
for i in 1 2 3 4 5 6 7 ; do sh MAKEDEV pty$i ; done
exit

cd /mnt/data1/jail-DIR/usr/compat/linux/dev
mknod null c 2 2
mknod random c 2 3

cd 
mkdir /usr/local/jail/template/
dump -0a -f /usr/local/jail/template/411template /dev/vn1

umount /dev/vn1c
vnconfig -u /dev/vn1

rm /mnt/data1/jail
rm -fr /mnt/data1/jail-DIR

37. setup backups
cat > /usr/local/jail/bin/backup.config
#!/bin/sh
backupdir=/mnt/data2/jail18_rsync

## ENTRY /etc
## ENTRY /usr/local/etc
## ENTRY /usr/local/jail
## ENTRY /root/logs

on backup2:
setup backup dirs:
mkdir -p /mnt/data2/jail18_rsync/0_today
mkdir -p /mnt/data2/jail18_rsync/1_yesterday
mkdir -p /mnt/data2/jail18_rsync/2_two_day

add the system to /mnt/data1/bin/rsync_houseclean 

on mail:
vi /usr/local/www/mgmt/cgi/backupgraph.pl
(add hostname)

38. mkdir /root/logs

39. edit sshd_config for security
vi /etc/ssh/sshd_config
ListenAddress 69.55.228.101
ListenAddress 10.1.4.104

kill -1 `cat /var/run/sshd.pid`

40. add crontab entries
crontab -e
5 0 * * * /usr/local/jail/bin/backup
1 0 1 * * /usr/local/jail/bin/ipfwreset
0 18 * * * /usr/local/jail/bin/ipfwbackup
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /usr/local/jail/bin/trafstats
55 10,23 * * * /usr/local/jail/bin/trafficwatch.pl


41. Reboot notify script
cat > /usr/local/etc/rc.d/notify.sh
echo "`/bin/hostname` rebooted" | /usr/bin/mail reboot@johncompanies.com

chmod +x /usr/local/etc/rc.d/notify.sh

42. copy jailmake from prev system
scp user@10.1.4.118:/usr/local/jail/bin/jailmake /usr/local/jail/bin
rehash
NOTE: remove df altering code from jailmake since we put the correct df in the template, and make sure path to template file is right

43. add to templates via mgmt system

44. add to server/cabinet map. On mail:
vi /usr/local/www/mgmt/html/cabinetmap.html

45. add an outside blocking rule to the firewall, so this machine can only be reached from inside the firewall. Follow example already in firewall jail17 is:
 
00117 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.228.2
00117 deny ip from any to 69.55.228.2

jail4 would be 00104...
ipfw add 00118 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.228.140
ipfw add 00118 deny ip from any to 69.55.228.140

46. select customers for probe map

FreeBSD 6.1[edit]

Last updated 2006-05-09 

All time extimates below assume disks aren’t scrubbing

1. make sure bios is setup for bios console redirect
Supermicro:
Console redirection:
Com port addr: on-board COM A
Baud: 38400
Console type: vt100
Flow control: none
Console connection: direct
Continue cr after post: off

2450:
Make sure running bios A09
Console Redirection: VT100/VT220

2. assuming mirrors (or at least disks) created (if not, refer to this), boot to disk 1 of 5.4
skip kernel config (enter)
custom install

partition ->
move cursor to amrd0, hit space (takes you to partition map screen)
a for entire disk
q to quit and save
standard mbr (no boot manager)
space to unselect aacd0
cursor over aacd1
space
a for entire disk
q to quit and save
none (leave untouched)

cursor over aacd0
space
(takes you into part. Screen again) q to exit
none
Make sure both are checked and tab to ok

Label ->
Make sure aacd0 is highlighted
a to start with defaults

john likes:
/1g
/var 256m

Glenn likes:
/128
/var 256
/usr 3g
/tmp 256

We do:
delete everything
/ 128M
swap 2G
/var 256M
/tmp 256M
/usr 3G
/mnt/data1 remaining space

Make sure to toggle S for soft updates on all (should look like UFS2+S Y under the Newfs column)

move cursor to aacd1
swap 2G
/mnt/data2 remaining space

q to save and exit

distributions ->
developer
yes to install ports
exit

media ->
cd

commit ->
yes
(2450: 14mins, supermicro: 12mins)

yes to "visit general config" ->
Set root pwd

Add user ‘user’ member group is wheel, set password

Set tz

Networking->interfaces->Fxp0 ->
No IPV6
dhcp=no
Set hostname & domain
Enable sshd

exit...
exit install ->
yes

take the cd out and let the machine reboot
when it comes back up, enter junk when it asks for key seed


3. double check the date/time

4. edit /etc/make.conf (only add the console speed line if this is a supermicro capable of outputting BIOS redirect at that speed – 2450’s can only do 9600)
echo "WITHOUT_X11=yes \
KERNCONF=jail19 \
BOOT_COMCONSOLE_SPEED=38400" >> /etc/make.conf

5. add console="comconsole" to /boot/loader.conf
echo "console=""comconsole""" >> /boot/loader.conf

6. turn off all ttyv's except 0 and 1 in /etc/ttys
also turn on ttyd0, change type to vt100:
vi /etc/ttys
ttyd0   "/usr/libexec/getty std.9600"  vt100   on secure

kill -1 1

on console server:
vi /etc/remote
(rename port to jail18 depending on where and which digi plugged into)
test serial console

7. install cvsup
cd /usr/ports/net/cvsup-without-gui 
make install clean; rehash
(stay close for gettext options, 2450: 27mins, supermicro: 17mins)

8. get latest sources for this release:
cd /usr/src 
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_1\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

cvsup sup 
(2450, ~12mins, supermicro, 27mins)

9. populate hosts
echo "10.1.4.3 backup2" >> /etc/hosts

10. put key in authorized_keys on backup2
cd
ssh-keygen -t dsa -b 1024 
(default location, leave password blank)
scp /root/.ssh/id_dsa.pub user@backup2:/tmp/jailkey

cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'

confirm that you can ssh to backup2 without getting a login prompt

ssh backup2

11. configure new kernel. Get config from similar machine or there may be a master copy somewhere under /mnt/data4/build/freebsd (name the kernel config the same as the jail, ex jail18):
cd /usr/src/sys/i386/conf 
scp backup2:/mnt/data4/build/freebsd/kern_config-6.1 ./jail19

edit the kernel config and change ident to be the name of the jail:
vi jail14
ident           jail14

edit /sys/conf/newvers.sh to add –jc1 to the end of the BRANCH string (RELEASE-jc1)
vi /sys/conf/newvers.sh

12. bring over patches from backup2
The various patches are in /mnt/data4/build/freebsd/patches on backup2. There are dirs for each version. Not all dirs are populated, but patches for later versions work on older ones unless there is a new patch in the older dir.
cd /usr/src
scp backup2:"/mnt/data4/build/freebsd/patches/6.1/*" .
jail_proc_patch      100% |*************************************************|  2593    00:00
restore-patch        100% |*************************************************|  3295    00:00
54ps-jail-patch      100% |*************************************************|  2602    00:00
jail_rtprio_patch    100% |*************************************************|   301    00:00
udp-patch            100% |*************************************************|   594    00:00

Apply patches:
patch < 54ps-jail-patch
patch < jail_proc_patch
patch < restore-patch
patch < jail_rtprio_patch
patch < udp-patch

13. build, install kernel and world
cd /usr/src
make buildworld buildkernel installkernel 
(2450: 1:56min, supermicro::59mins)
make installworld 
(2450: 3min, supermicro: 1min)
mergemaster -i
delete /var/tmp/temproot
delete bsnmpd
delete temporary ./etc/hosts
delete temporary ./etc/motd
delete /var/tmp/temproot

14. reboot. Confirm new kernel is loaded

uname –a

15. update ports:
cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_1\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup

cvsup sup 
(2450: 18mins, supermicro: 19mins)

18. (only applies if adaptec card installed)
install linux_base:
cd /usr/ports/emulators/linux_base
make install clean 
(2450: 7min, supermicro: 3mins)

reibuild rpmdb cause we had probs installing aacapps
cd /compat/linux/bin
./rpm --initdb
./rpm --rebuilddb

install aacapps-4.1-0.i386.rpm
scp backup2:/mnt/data4/build/freebsd/aacapps-4.1-0.i386.rpm /tmp/.
/compat/linux/bin/rpm -ivh --excludepath=/dev /tmp/aacapps-4.1-0.i386.rpm

test out;
/compat/linux/usr/sbin/aaccli


20. edit root's path and login script:
vi /root/.cshrc
add to path: 
/usr/local/jail/bin 
(if adaptec card installed, also add /compat/linux/usr/sbin)

Change alias entries (add G):
alias la        ls -aG
alias lf        ls -FAG
alias ll        ls -lAG
alias ls        ls -AG
alias mbm       mb mount
alias mbu       mb umount

and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "

at the bottom of the file add:
set sshtty=`who am i|awk '{print $2}'`
/usr/sbin/rtprio 3 -`psj | grep $sshtty | awk '{print $2}'`

set shortty=`who am i | awk '{print $2}' | sed -E 's/.*(..)$/\1/'`
foreach x (`psj | grep sh | grep $shortty | awk '{print $2}'`)
/usr/sbin/rtprio 2 -$x
end
 
21. install rsync from ports
cd /usr/ports/net/rsync
make install clean

choose default options

21. install perl from ports
PROB NOT NECESSARY – INSTALLED WITH LINUX I THINK
cd /usr/ports/lang/perl5.8/
make install clean; rehash
(supermicro: 5min)

22. create & populate binaries/scripts dirs
mkdir -p /usr/local/jail/bin
mkdir -p /usr/local/jail/rc.d
mkdir /mnt/data1
mkdir /mnt/data2
scp backup2:"/mnt/data4/bin/freebsd6.x/*" /usr/local/jail/bin
cd /usr/local/jail/rc.d/
touch quad1
touch quad2
touch quad3
touch quad4
touch safe1
touch safe2
touch safe3
touch safe4
chmod +x *
cd /usr/local/jail/bin
ln -s /usr/local/jail/rc.d/quad1 quad1
ln -s /usr/local/jail/rc.d/quad2 quad2
ln -s /usr/local/jail/rc.d/quad3 quad3
ln -s /usr/local/jail/rc.d/quad4 quad4
ln -s /usr/local/jail/rc.d/safe1 safe1
ln -s /usr/local/jail/rc.d/safe2 safe2
ln -s /usr/local/jail/rc.d/safe3 safe3
ln -s /usr/local/jail/rc.d/safe4 safe4

rehash

23. configure inetd to respond to mrtg load queries
echo "load    stream  tcp     nowait  user    /usr/local/jail/bin/load.pl  load.pl" >> /etc/inetd.conf

echo "load            12384/tcp" >> /etc/services

26. install bb client
adduser
cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd.tar .
tar xvf bb-freebsd.tar

edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
echo "10.1.4.105 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.4.103 jail3.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

vi /home/bb/bbc1.9e-btf/ext/openfiles 
MACHINE="jail19,johncompanies,com"      # HAS TO BE IN A,B,C FORM

cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh (y to questions)
./bbchkhosts.sh (ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src
make; make install
cd ..
./runbb.sh start
more BBOUT 
(look for errors)
exit

echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh

modify bb-msgtab to look for raid failures

24. configure load mrtg, on mail
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg 
(add new entry to file following existing format)

25. configure bb on mail:
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
10.1.4.119 jail19.johncompanies.com # ssh

su bb
cd
bbsrc/bb/runbb.sh restart ; exit

27. remove reserve space, enable softupdates (probably already set, so not necessary)
NOT APPLICABLE IF USING GVINUM
cd
umount /mnt/data1
umount /mnt/data2
tunefs -m 0 /mnt/data1
tunefs -m 0 /mnt/data2
mount -a

28. configure ntp
echo "server 10.1.4.105" > /etc/ntp.conf

/usr/sbin/ntpd -p /var/run/ntpd.pid
sleep 2; ntpq -p
(confirm it’s able to reach our time server)

29. mrtg switch graphs

31. fwd and reverse lookups on ns1c
vr johncompanies.com
vi internal.johncompanies.com
rndc reload johncompanies.com IN private
(edit the PTR too)

33. populate /etc/rc.conf with IPs and NFS settings
vi /etc/rc.conf

kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
sshd_enable="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.4.119"
devfs_system_ruleset="devfsrules_show_all"

ifconfig_xl0="inet 10.1.4.118 netmask 255.255.255.0"
ifconfig_fxp0="inet 69.55.228.101 netmask 255.255.255.0"
defaultrouter="69.55.228.1"
ifconfig_fxp0_alias0="inet 69.55.2xx.xx netmask 255.255.255.0"

static_routes="t1 office"
route_t1="-net 10.1.5 10.1.4.2"
route_office="-net 10.1.6 10.1.4.2"
gvinum_enable="YES"
fsck_y_enable="YES"
background_fsck="NO"

34. make sure sysctls are set and preserved after reboot
echo "kern.consmute=0\
kern.ipc.shm_use_phys=1\
kern.ipc.shmall=65535\
kern.ipc.shmmax=134217728\
net.inet.tcp.syncookies=0\
kern.maxfiles=32768\
kern.fallback_elf_brand=3\
kern.maxprocperuid=4000\
security.jail.sysvipc_allowed=1\
security.jail.max_procs_per_jail: 1026\
security.jail.allow_raw_sockets=1\
security.jail.socket_unixiproute_only=1\
security.jail.chflags_allowed=0" >> /etc/sysctl.conf

35. mount procfs
echo "proc                    /proc           procfs  rw              0       0" >> /etc/fstab

36. enable noatime option
NOT APPLICABLE IF RUNNING GVINUM
data1 and data2 should look something like:
/dev/amrd0s1g           /mnt/data1      ufs     rw,noatime      2       2

36. populate devfs ruleset
scp backup2:/mnt/data4/build/freebsd/devfs.rules /etc

35. reboot
Check rules:
devfs rule showsets
devfs rule -s 3 show

36. create gvinum volumes

Make a g partition:

bsdlabel -e /dev/aacd0s1

given:
# /dev/aacd0s1:
8 partitions:
#        size   offset    fstype   [fsize bsize bps/cpg]
  a:   262144        0    4.2BSD     2048 16384 16392
  b:  4194304   262144      swap
  c: 143363997        0    unused        0     0         # "raw" part, don't edit
  d:   524288  4456448    4.2BSD     2048 16384 32776
  e:   524288  4980736    4.2BSD     2048 16384 32776
  f:  6291456  5505024    4.2BSD     2048 16384 28552

new offset = 6291456 + 5505024 = 11796480
new size is size for 'c' partition minus the new start from above
143363997 - 11796480 = 131567517
So:
g: 131567517 11796480 unused 0 0

For a 73G drive (after OS), we can fit 31 2G volumes so:

echo 'drive data1 device /dev/aacd0s1g' > /tmp/cgv
sh
for f in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31;\
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 2g drive data1' >> /tmp/cgv; done; exit

gvinum create /tmp/cgv

For a 146G drive (-4G for swap), we can fit 66 2G volumes so:

echo 'drive data2 device /dev/aacd1s1g' > /tmp/cgv
sh
for f in 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97;\
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 2g drive data2' >> /tmp/cgv; done; exit

gvinum create /tmp/cgv

For 3rd 73G drive (after 2G swap), we can fit 33 2G volumes so:
Label should be:
# /dev/aacd2s1:
8 partitions:
#        size   offset    fstype   [fsize bsize bps/cpg]
  c: 143299737        0    unused        0     0         # "raw" part, don't edit
  g: 143299721        16    unused        0     0

echo 'drive data3 device /dev/aacd2s1g' > /tmp/cgv
sh
for f in 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131;\
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 2g drive data3' >> /tmp/cgv; done

gvinum create /tmp/cgv

For a 2nd 73G drive (after 2G swap), we can fit 33 2G volumes so:

echo 'drive data2 device /dev/aacd1s1g' > /tmp/cgv
sh
for f in 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64;\
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 2g drive data2' >> /tmp/cgv; done; exit

gvinum create /tmp/cgv


36. create the jail template

mkdir /mnt/jail
newfs /dev/gvinum/v1
mount /dev/gvinum/v1 /mnt/jail

cd /usr/src
make clean
rm -fr /usr/obj/
make world DESTDIR=/mnt/jail
(2450: 2:28mins, supermicro: 55mins)
cd etc
make distribution DESTDIR=/mnt/jail
mount_devfs devfs /mnt/jail/dev
devfs -m /mnt/jail/dev rule -s 3 applyset 
cd /mnt/jail
ln -sf dev/null kernel

jail /mnt/jail testhostname 192.168.11.100 /bin/sh
csh
touch /etc/fstab
echo 'network_interfaces=""\
hostname="newsystem"\
kern_securelevel_enable="NO"\
sendmail_enable="YES"\
sshd_enable="YES"' > /etc/rc.conf

echo "nameserver 69.55.225.225\
nameserver 69.55.230.3" >> /etc/resolv.conf

vi /etc/crontab
remove the adjkerntz lines
comment out periodic’s and put this line above them:
# DO NOT UNCOMMENT THESE

rm -rf /etc/periodic/daily/400.status-disks

check /tmp for crap

vi /etc/periodic/security/100.chksetuid
replace: MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`
with: MP='/' (use single quotes)

mkdir -p /usr/compat/linux/dev

adduser (Add account for user, make sure in wheel group)

user/root passwd: 8ico2987

Set root password

vi /usr/home/user/.profile (and add to the file):
TERM=vt100;     export TERM

tzsetup

newaliases 

#cd /dev
#rm console
#ln -s null console

exit
exit

cd /usr/ports
make -DNOCLEANDEPENDS clean 
(2450: 15mins , supermicro: 29mins)
rm -fr /usr/ports/distfiles/*
cp -r /usr/ports /mnt/jail/usr (2450: 2:00 mins , supermicro: 15mins)

rm /mnt/jail/root/.history

cd 
mkdir /usr/local/jail/template/
dump -0a -f /usr/local/jail/template/61template /dev/gvinum/v1

umount /mnt/jail/dev
umount /dev/gvinum/v1

rm -fr /mnt/jail

37. setup backups
echo '#\!/bin/sh\
backupdir=/mnt/data3jail3_rsync\
\
## ENTRY /etc\
## ENTRY /usr/local/etc\
## ENTRY /usr/local/jail\
## ENTRY /root/logs' > /usr/local/jail/bin/backup.config

on backup2:
setup backup dirs:
mkdir -p /mnt/data3/jail3/0

add the system to 
vi /mnt/data4/bin/snapshot_rotate

on mail:
vi /usr/local/www/mgmt/cgi/backupgraph.pl
(add hostname)

38. mkdir /root/logs

39. edit sshd_config for security
vi /etc/ssh/sshd_config
ListenAddress 69.55.228.101
ListenAddress 10.1.4.118

kill -1 `cat /var/run/sshd.pid`

40. add crontab entries
crontab -e
5 0 * * * /usr/local/jail/bin/backup
1 0 1 * * /usr/local/jail/bin/ipfwreset
0 18 * * * /usr/local/jail/bin/ipfwbackup
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /usr/local/jail/bin/trafstats
55 10,23 * * * /usr/local/jail/bin/trafficwatch.pl


41. Reboot notify script
ln -s /usr/local/jail/bin/notify.sh /usr/local/etc/rc.d/notify.sh 

42. copy jailmake from prev system
scp user@10.1.4.119:/usr/local/jail/bin/jailmake /usr/local/jail/bin
rehash
NOTE: remove df altering code from jailmake since we put the correct df in the template, and make sure path to template file is right

43. add to management db (on mail and devweb) jc.ref_machines and jc.ref_templates

uname -r
5.4-RELEASE-p2-jc2

insert into ref_machines values (null,'jail19','jail19.johncompanies.com',0,'l');
select machine_id from ref_machines where host='jail19';
+------------+
| machine_id |
+------------+
|         35 |
+------------+
insert into ref_templates values ('',' 6.2-RELEASE-jc1',5,'FreeBSD 6.2',0);

44. add to server/cabinet map. On mail:
vi /usr/local/www/mgmt/html/cabinetmap.html

45. add an outside blocking rule to the firewall, so this machine can only be reached from inside the firewall. Follow example already in firewall jail17 is:
 
00119 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.228.200
00119 deny ip from any to 69.55.228.200

jail19 would be 00119...
ipfw add 00119 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.228.200
ipfw add 00119 deny ip from any to 69.55.228.200

46. select customers for probe map

47. install raid monitor

cd /usr/ports/sysutils/asr-utils
make install clean

48. make gv start on boot


scp backup2:/mnt/data4/build/freebsd/gvinum /etc/rc.d/gvinum




gconcat label -v somelabel /dev/gvinum/a /dev/gvinum/b
bsdlabel -r -w /dev/concat/somelabel
newfs /dev/concat/somelabela
mount /dev/concat/somelabel /mount/point

umount /dev/concat/somelabel
gconcat stop somelabel
gconcat label -v /dev/gvinum/a /dev/gvinum/b /dev/gvinum/c /dev/gvinum/d
growfs /dev/concat/somelabel
mount /dev/concat/somelabel /mount/point

volume f
        plex org concat
        sd length 30449m drive data1

where f is 

D data1                 State: up       /dev/aacd0s1g   A: 30449/64241 MB (47%)

gvinum rm -r f


so i setup 2 machines with 6.1. A had 2 logical drives and gv's created across both. B had 1 logical drive and gv's across the 1 drive.  
the labeling for the gv's was v1-vN (till i ran out of space) 
when i moved aac1 from A to B, the gv's on A's aac1, took precedence over the similarly labeled gv's on B's aac0. in other words. B's aac0 used to have v1-v30. A's aac1 had vn6-vn30. when A's aac1 was moved to B, v1-v5 were linked to B's aac0, and v6-v30 linked to A's aac1 
i relabeled B's aac0 gv's to something different (not v1-vN) and tried again and they all showed up.  
i tried the experiment again this time moving A's aac0 to B. at that point nothing worked. i moved A's aac0 back to A and renamed the device (data1) to something else, then tried the test again. still no beans.  
it's hazy what i did after that cause the machine's weren't cooperating and i was trying to rename the device but basically i think i need to repeat the test and see if i can re-define the device, and probably also give it a unique name and i bet it would have worked.  
also interesting- A's aac1 contained data about A's aac0 which showed up when i put aac1 into B 
skeeter:	as for the gv stuff, it definitely sounds like using some sort of serial numbering scheme would be the way to go if you want to be able to move disks around.... 
that overlap is still an issue  (aac1's device was 'data2')  when i had A's aac0 in B nothing worked both disks gv devices were called data1 
skeeter:	I suppose you could serialize those names as well...

FreeBSD 6.2[edit]

Last updated 2007-10-15 

All time extimates below assume disks aren’t scrubbing. Setup instructions below are for LSI card:

1. make sure bios is setup for bios console redirect
Supermicro:
Console redirection:
Com port addr: on-board COM A
Baud: 38400
Console type: vt100
Flow control: none
Console connection: direct
Continue cr after post: on

2450:
Make sure running bios A09
Console Redirection: VT100/VT220

2. assuming mirrors (or at least disks) created (if not, refer to this), boot to disk 1 of 5.4
skip kernel config (enter)
custom install

partition ->
move cursor to amrd0, hit space (takes you to partition map screen)
a for entire disk
q to quit and save
standard mbr (no boot manager)
space to unselect amrd0
cursor over amrd1
space
a for entire disk
q to quit and save
none (leave untouched)

cursor over amrd0
space
(takes you into part. Screen again) q to exit
none
Make sure both are checked and tab to ok

Label ->
Make sure amrd0 is highlighted
a to start with defaults

john likes:
/1g
/var 256m

Glenn likes:
/128
/var 256
/usr 3g
/tmp 256

We do:
delete everything
/ 128M
swap 2G (for 2950 make it 4G)
/var 256M
/tmp 256M
/usr 3.5G (3584M)
/mnt/data1 remaining space (no need to newfs)

Make sure to toggle S for soft updates on all (should look like UFS2+S Y under the Newfs column)

move cursor to amrd1
swap 4G
/mnt/data2 remaining space (no need to newfs)

q to save and exit

distributions ->
developer
yes to install ports
exit

media ->
cd

commit ->
yes
(2450: 14mins, supermicro: 12mins)

yes to "visit general config" ->
Set root pwd

Add user ‘user’ member group is wheel, set password

Set tz

Networking->interfaces->Fxp0 ->
No IPV6
dhcp=no
Set hostname & domain
Enable sshd

exit...
exit install ->
yes

take the cd out and let the machine reboot

3. double check the date/time

4. edit /etc/make.conf (only add the console speed line if this is a supermicro capable of outputting BIOS redirect at that speed – 2450’s can only do 9600)
echo "WITHOUT_X11=yes \
KERNCONF=jail7 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf

5. add console="comconsole" to /boot/loader.conf
echo "console=""comconsole""" >> /boot/loader.conf

6. turn off all ttyv's except 0 and 1 in /etc/ttys
also turn on ttyd0, change type to vt100:
vi /etc/ttys
ttyd0   "/usr/libexec/getty std.9600"  vt100   on secure

kill -1 1

on console server:
vi /etc/remote
(rename port to jail18 depending on where and which digi plugged into)
test serial console

7. populate hosts
echo "10.1.4.3 backup2" >> /etc/hosts

8. put key in authorized_keys on backup2
cd
ssh-keygen -t dsa -b 1024 
(default location, leave password blank)

cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'

confirm that you can ssh to backup2 without getting a login prompt

ssh backup2

9. create & populate binaries/scripts dirs
mkdir -p /usr/local/jail/bin
mkdir -p /usr/local/jail/rc.d
mkdir -p /usr/local/jail/template/
mkdir /mnt/data1
mkdir /mnt/data2
scp backup2:"/mnt/data4/bin/freebsd6.x/*" /usr/local/jail/bin
cd /usr/local/jail/rc.d/
touch quad1
touch quad2
touch quad3
touch quad4
touch safe1
touch safe2
touch safe3
touch safe4
chmod +x *
cd /usr/local/jail/bin
ln -s /usr/local/jail/rc.d/quad1 quad1
ln -s /usr/local/jail/rc.d/quad2 quad2
ln -s /usr/local/jail/rc.d/quad3 quad3
ln -s /usr/local/jail/rc.d/quad4 quad4
ln -s /usr/local/jail/rc.d/safe1 safe1
ln -s /usr/local/jail/rc.d/safe2 safe2
ln -s /usr/local/jail/rc.d/safe3 safe3
ln -s /usr/local/jail/rc.d/safe4 safe4

rehash

10. edit root's path and login script:
vi /root/.cshrc
add to path: 
/usr/local/jail/bin 
(if adaptec card installed, also add /compat/linux/usr/sbin)

Change alias entries (add G):
alias la        ls -aG
alias lf        ls -FAG
alias ll        ls -lAG
alias ls        ls -AG
alias mbm       mb mount
alias mbu       mb umount

and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "

at the bottom of the file add:
set sshtty=`who am i|awk '{print $2}'`
/usr/sbin/rtprio 3 -`psj | grep $sshtty | awk '{print $2}'`

set shortty=`who am i | awk '{print $2}' | sed -E 's/.*(..)$/\1/'`
foreach x (`psj | grep sh | grep $shortty | awk '{print $2}'`)
/usr/sbin/rtprio 2 -$x
end

To load the new file:
source /root/.cshrc

11. install cvsup
cd /usr/ports/net/cvsup-without-gui 
make install clean; rehash; mail -s 'cvs installed' dave.boodman@vtext.com < /dev/null

(stay close for gettext options, 2450: 27mins, supermicro: 17mins, 2950: 22mins)

12. get latest sources for this release:
cd /usr/src 
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_2\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

cvsup sup ; mail -s 'cvs sup done' dave.boodman@vtext.com < /dev/null

(2450, ~12mins, supermicro, 27mins, 2950: 7mins)


13. configure new kernel. Get config from similar machine or there may be a master copy somewhere under /mnt/data4/build/freebsd (name the kernel config the same as the jail, ex jail18):
cd /usr/src/sys/i386/conf 
scp backup2:/mnt/data4/build/freebsd/kern_config-6.2 ./jail7

or for PAE

scp backup2:/mnt/data4/build/freebsd/kern_config-6.2-PAE ./jail7

edit the kernel config and change ident to be the name of the jail:
vi jail7
ident           jail7

edit /sys/conf/newvers.sh to add –jc1 to the end of the BRANCH string (RELEASE-jc1)
vi /sys/conf/newvers.sh

14. bring over patches from backup2

The various patches are in /mnt/data4/build/freebsd/patches on backup2. There are dirs for each version. Not all dirs are populated, but patches for later versions work on older ones unless there is a new patch in the older dir.

cd /usr/src
scp backup2:"/mnt/data4/build/freebsd/patches/6.x/*" .

Apply patches:
patch -l < jls-patch

Apply these only to 2950 with PAE:
patch -p0 < mfi-patch
patch -p0 < gvinum-staticcompile-patch
patch -p0 < gvinum-bin-patch

15. build, install kernel and world
cd /usr/src
make buildworld buildkernel installkernel; mail -s 'kernel build done' dave.boodman@vtext.com < /dev/null
(2450: 1:56min, supermicro: 59mins, 2950: 38mins)
make installworld 
(2450: 3min, supermicro: 1min, 2950: :34)
mergemaster -i
delete /var/tmp/temproot
delete bsnmpd
delete temporary ./etc/hosts
delete temporary ./etc/motd
delete /var/tmp/temproot

16. populate devfs ruleset
scp backup2:/mnt/data4/build/freebsd/devfs.rules /etc

17. populate /etc/rc.conf with IPs and NFS settings
vi /etc/rc.conf

kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
sshd_enable="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.4.119"
devfs_system_ruleset="devfsrules_show_all"

ifconfig_xl0="inet 10.1.4.118 netmask 255.255.255.0"
ifconfig_fxp0="inet 69.55.228.101 netmask 255.255.255.0"
defaultrouter="69.55.228.1"
ifconfig_fxp0_alias0="inet 69.55.2xx.xx netmask 255.255.255.0"

static_routes="t1 office"
route_t1="-net 10.1.5 10.1.4.2"
route_office="-net 10.1.6 10.1.4.2"
gvinum_enable="YES"
fsck_y_enable="YES"
background_fsck="NO"

18. make sure sysctls are set and preserved after reboot
echo "kern.consmute=0\
kern.ipc.shm_use_phys=1\
kern.ipc.shmall=65535\
kern.ipc.shmmax=134217728\
net.inet.tcp.syncookies=0\
kern.maxfiles=32768\
kern.fallback_elf_brand=3\
kern.maxprocperuid=4000\
security.jail.sysvipc_allowed=1\
security.jail.max_procs_per_jail: 1026\
security.jail.allow_raw_sockets=1\
security.jail.socket_unixiproute_only=1\
security.jail.chflags_allowed=0\
dev.amr.0.allow_volume_configure=1\
compat.linux.osrelease=2.6.12" >> /etc/sysctl.conf

19. mount procfs
echo "proc                    /proc           procfs  rw              0       0" >> /etc/fstab

For Dell 2950:
echo "linprocfs               /usr/compat/linux/proc linprocfs rw     0       0" >> /etc/fstab
echo "linsysfs                /usr/compat/linux/sys linsysfs rw       0       0" >> /etc/fstab

20. reboot. Confirm new kernel is loaded

uname –a

Check devfs rules:
devfs rule showsets
devfs rule -s 3 show

21. update ports:
cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_2\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup

cvsup sup; mail -s 'cvs sup ports done' dave.boodman@vtext.com < /dev/null

(2450: 18mins, supermicro: 19mins; 2950: 24mins)

22. Install raid mgmt tool
install linux_base:
cd /usr/ports/emulators/linux_base-fc4
make install clean 
(2450: 7min, supermicro: 3mins, 2950: 14mins)

(for LSI)
cd /usr/ports/sysutils/linux-megamgr
make install clean

cd /usr/ports/sysutils/megarc
make install clean

(for Perc5/i)
cd /usr/ports/sysutils/linux-megacli
make install clean

Test:
rehash; megacli ldinfo lall a0

 (for adaptec)
This didn’t work: reibuild rpmdb cause we had probs installing aacapps
cd /compat/linux/bin
./rpm --initdb
./rpm --rebuilddb

Install of linux-base lead to broken rpm on 6.2 so:
install aacapps-4.1-0.i386.rpm
scp backup2:/mnt/data4/build/freebsd/aacapps-4.1-0.i386.rpm /tmp/.
/compat/linux/bin/rpm -ivh --excludepath=/dev /tmp/aacapps-4.1-0.i386.rpm
scp user@10.1.4.107:/compat/linux/usr/sbin/aaccli /compat/linux/usr/sbin/aaccli

test out;
/compat/linux/usr/sbin/aaccli

rpm didn’t even install on latest so just scp’d over aaccli and it worked

23. install rsync from ports
cd /usr/ports/net/rsync
make install clean

choose default options

24. install perl from ports
Not necessary if linux_base is installed

cd /usr/ports/lang/perl5.8/
make install clean; rehash
(supermicro: 5min)

25. configure inetd to respond to mrtg load queries
echo "load    stream  tcp     nowait  user    /usr/local/jail/bin/load.pl  load.pl" >> /etc/inetd.conf

echo "load            12384/tcp" >> /etc/services

26. install bb client
adduser
Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username   : bb
Password   : <random>
Full Name  : bb
Uid        : 1984
Class      :
Groups     : bb
Home       : /home/bb
Shell      : /bin/sh
Locked     : no
OK? (yes/no): yes

cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd.tar .
tar xvf bb-freebsd.tar

edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.4.107 jail7.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

vi /home/bb/bbc1.9e-btf/ext/openfiles 
MACHINE="jail19,johncompanies,com"      # HAS TO BE IN A,B,C FORM

cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh 
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src
make; make install
cd ..
./runbb.sh start
more BBOUT 
(look for errors)
exit

echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh

27. configure load mrtg, on mail
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg 
(add new entry to file following existing format)

28. configure bb on mail:
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
10.1.4.119 jail19.johncompanies.com # ssh

su bb
cd
bbsrc/bb/runbb.sh restart ; exit

29. remove reserve space, enable softupdates (probably already set, so not necessary)
NOT APPLICABLE IF USING GVINUM
cd
umount /mnt/data1
umount /mnt/data2
tunefs -m 0 /mnt/data1
tunefs -m 0 /mnt/data2
mount -a

30. configure ntp
echo "server 10.1.4.5" > /etc/ntp.conf

/usr/sbin/ntpd -p /var/run/ntpd.pid
sleep 2; ntpq -p
(confirm it’s able to reach our time server)

31. mrtg switch graphs

32. fwd and reverse lookups on ns1c
vr johncompanies.com
vi internal.johncompanies.com
rndc reload johncompanies.com IN private
(edit the PTR too)


33. enable noatime option
NOT APPLICABLE IF RUNNING GVINUM
data1 and data2 should look something like:
/dev/amrd0s1g           /mnt/data1      ufs     rw,noatime      2       2


reboot

34. create gvinum volumes

Make a g partition:

bsdlabel -e /dev/amrd0s1

given:
# /dev/aacd0s1:
8 partitions:
#        size   offset    fstype   [fsize bsize bps/cpg]
  a:   262144        0    4.2BSD     2048 16384 16392
  b:  4194304   262144      swap
  c: 143363997        0    unused        0     0         # "raw" part, don't edit
  d:   524288  4456448    4.2BSD     2048 16384 32776
  e:   524288  4980736    4.2BSD     2048 16384 32776
  f:  6291456  5505024    4.2BSD     2048 16384 28552

new offset = 6291456 + 5505024 = 11796480
new size is size for 'c' partition minus the new start from above
143363997 - 11796480 = 131567517
So:
g: 131567517 11796480 unused 0 0

bsdlabel -e /dev/amrd1s1
change d to g

For a 73G drive (after OS), we can fit 31 2G volumes so:

echo 'drive data1 device /dev/aacd0s1g' > /tmp/cgv

or 

echo '#\!/bin/sh\
i="1"\
while [ $i -le 31 ]\
do\
  echo "volume v$i" >> /tmp/cgv;\
  echo 'plex org concat' >> /tmp/cgv;\
  echo 'sd length 2g drive data1' >> /tmp/cgv;\
i=`expr $i + 1`\
done' > /tmp/mkcgv
sh /tmp/mkcgv
gvinum create /tmp/cgv


For a 146G drive (-4G for swap), we can fit 66 2G volumes so:

echo 'drive data2 device /dev/amrd1s1g' > /tmp/cgv
sh
for f in 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97;\
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 2g drive data2' >> /tmp/cgv; done; exit

gvinum create /tmp/cgv

For 3rd 73G drive (after 2G swap), we can fit 33 2G volumes so:
Label should be:
# /dev/aacd2s1:
8 partitions:
#        size   offset    fstype   [fsize bsize bps/cpg]
  c: 143299737        0    unused        0     0         # "raw" part, don't edit
  g: 143299721        16    unused        0     0

/dev/aacd1s1d
echo 'drive data3 device /dev/aacd2s1g' > /tmp/cgv
sh
for f in 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131;\
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 2g drive data3' >> /tmp/cgv; done

gvinum create /tmp/cgv

For a 2nd 73G drive (after 2G swap), we can fit 33 2G volumes so:

echo 'drive data2 device /dev/aacd1s1g' > /tmp/cgv
sh
for f in 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64;\
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 2g drive data2' >> /tmp/cgv; done; exit

gvinum create /tmp/cgv

For a 2nd 73G drive (after 4G swap), we can fit 32 2G volumes so:

echo 'drive data2 device /dev/aacd1s1g' > /tmp/cgv
sh
for f in 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63;\
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 2g drive data2' >> /tmp/cgv; done; exit

gvinum create /tmp/cgv

For a 3nd 73G drive, we can fit 34 2G volumes so:

echo 'drive data3 device /dev/mfid2s1g' > /tmp/cgv
sh
for f in 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163;\
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 2g drive data3' >> /tmp/cgv; done; exit

gvinum create /tmp/cgv

2950:
bsdlabel -e /dev/mfid0s1
bsdlabel -e /dev/mfid1s1



For 1st drive (146G)
echo 'drive data1 device /dev/mfid0s1g' > /tmp/cgv
sh
for f in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63;\
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 2g drive data1' >> /tmp/cgv; done; exit

Usually there’s 2040m leftover so create one more smaller vol:
echo 'volume v64\
plex org concat\
sd length 2040m drive data1' >> /tmp/cgv

gvinum create /tmp/cgv

For 2nd drive (146G)
echo 'drive data2 device /dev/mfid1s1g' > /tmp/cgv
sh
for f in 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130;\
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 2g drive data2' >> /tmp/cgv; done; exit
gvinum create /tmp/cgv


35. create the jail template

mkdir /mnt/jail
newfs /dev/gvinum/v1
mount /dev/gvinum/v1 /mnt/jail

cd /usr/src
make clean
rm -fr /usr/obj/
make buildworld 
(2450: 2:28mins, supermicro: 53mins, 2950: 36min)
make world DESTDIR=/mnt/jail
(2450: 2:28mins, supermicro: 55mins, 2950: )
cd etc
make distribution DESTDIR=/mnt/jail
mount_devfs devfs /mnt/jail/dev
devfs -m /mnt/jail/dev rule -s 3 applyset 
cd /mnt/jail
ln -sf dev/null kernel

jail /mnt/jail testhostname 192.168.11.100 /bin/sh
csh
touch /etc/fstab
echo 'network_interfaces=""\
kern_securelevel_enable="NO"\
sendmail_enable="YES"\
sshd_enable="YES"' > /etc/rc.conf

echo "nameserver 69.55.225.225\
nameserver 69.55.230.3" >> /etc/resolv.conf

vi /etc/crontab
remove the adjkerntz lines
comment out periodic’s and put this line above them:
# DO NOT UNCOMMENT THESE

rm -rf /etc/periodic/daily/400.status-disks

check /tmp for crap

vi /etc/periodic/security/100.chksetuid
replace: MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`
with: MP='/' (use single quotes)

mkdir -p /usr/compat/linux/dev

adduser (Add account for user, make sure in wheel group)

user/root passwd: 8ico2987

Set root password

vi /usr/home/user/.profile (and add to the file):
TERM=vt100;     export TERM

tzsetup

newaliases 

#cd /dev
#rm console
#ln -s null console

vi /etc/syslog.conf (comment out console and move to /var/log/messages):
#*.err;kern.warning;auth.notice;mail.crit               /dev/console *.err;kern.warning;auth.notice;mail.crit                /var/log/messages 


exit
exit

cd /usr/ports
make -DNOCLEANDEPENDS clean 
(2450: 15mins , supermicro: 29mins, 2950: 24mins)
rm -fr /usr/ports/distfiles/*
cp -r /usr/ports /mnt/jail/usr (2450: 2:00 mins , supermicro: 15mins, 2950: 3mins)

rm /mnt/jail/root/.history

cd 
dump -0a -f /usr/local/jail/template/61template /dev/gvinum/v1

umount /mnt/jail/dev
umount /dev/gvinum/v1

rm -fr /mnt/jail


36. setup backups
echo '#\!/bin/sh\
backupdir=/mnt/data4/jail7\
\
## ENTRY /etc\
## ENTRY /usr/local/etc\
## ENTRY /usr/local/jai7\
## ENTRY /root/logs' > /usr/local/jail/bin/backup.config

on backup2:
setup backup dirs:
mkdir -p /mnt/data2/jail7/0

add the system to 
vi /mnt/data4/bin/snapshot_rotate

on mail:
vi /usr/local/www/mgmt/cgi/backupgraph.pl
(add hostname)

Edit /usr/local/jail/bin/backup to use the right drives

37. mkdir /root/logs

38. edit sshd_config for security
vi /etc/ssh/sshd_config
ListenAddress 69.55.226.128
ListenAddress 10.1.4.114

kill -1 `cat /var/run/sshd.pid`

39. add crontab entries
crontab -e
5 0 * * * /usr/local/jail/bin/backup
1 0 1 * * /usr/local/jail/bin/ipfwreset
0 18 * * * /usr/local/jail/bin/ipfwbackup
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /usr/local/jail/bin/trafstats
55 10,23 * * * /usr/local/jail/bin/trafficwatch.pl

On 2950 with Perc5/i also add:
*/5 * * * * /usr/local/jail/bin/perc5iraidchk

40. Reboot notify script
ln -s /usr/local/jail/bin/notify.sh /usr/local/etc/rc.d/notify.sh 

41. add to management db (on mail and devweb) jc.ref_machines and jc.ref_templates

uname -r
6.2-RELEASE-jc1

insert into ref_machines values (null,'jail19','jail19.johncompanies.com',0,'l');
select machine_id from ref_machines where host='jail19';
+------------+
| machine_id |
+------------+
|         35 |
+------------+
insert into ref_templates values ('',' 6.2-RELEASE-jc1',15,'FreeBSD 6.2',0);

42. add to server/cabinet map. On mail:
vi /usr/local/www/mgmt/html/cabinetmap.html

43. add an outside blocking rule to the firewall, so this machine can only be reached from inside the firewall. Follow example already in firewall jail17 is:
 
00117 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.228.200
00117 deny ip from any to 69.55.228.200

jail19 would be 00119...
ipfw add 00107 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.230.108
ipfw add 00107 deny ip from any to 69.55.220.108

44. select customers for probe map

45. install raid monitor (LSI 320 only):

scp backup2:/d4/build/megaraid/MegaMonitor1.02.tgz /tmp
pkg_add MegaMonitor1.02.tgz

edit line in /usr/local/etc/rc.d/megamonitor.sh to look like:
/usr/sbin/MegaCtrl -start -log /var/log/messages  -disChkCon -SMART9999 > /megamonitor.out

comment out:
localhost: /var/log/messages : : : **Monitor** : 
In /usr/home/bb/bbc1.9e-btf/etc/bb-msgstab

cd /usr/ports/sysutils/megarc
make install clean


46. make gv start on boot

scp backup2:/mnt/data4/build/freebsd/gvinum /etc/rc.d/gvinum

If they start stale:

echo '#\!/bin/sh\
i="1"\
while [ $i -le 64 ]\
do\
        gvinum start v$i\
        i=`expr $i + 1`\
done' > /tmp/startgv

sh /tmp/startgv

47. patch jail against starting jails with rtprio

mv /usr/sbin/jail /usr/sbin/jail_
echo '#\!/bin/sh\
/usr/sbin/rtprio -t /usr/sbin/jail_ $*' > /usr/sbin/jail
chmod +x /usr/sbin/jail

48. make sure mail works
If there are map errors:
cd /etc/mail; make maps

FreeBSD 6.2 -> 6.3[edit]

Last updated 2008-08-07 


1. remove old src
cd /usr
rm –fr src/*

2.  get latest sources for this release:

cd /usr/src 
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_3\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

cvsup sup ; pagedave
(20min)

3. configure new kernel. Get config from similar machine or there may be a master copy somewhere under /mnt/data4/build/freebsd (name the kernel config the same as the jail, ex jail18):
cd /usr/src/sys/i386/conf 
scp backup2:/mnt/data4/build/freebsd/kern_config-6.2 ./jail7

or for PAE

scp backup2:/mnt/data4/build/freebsd/kern_config-6.2-PAE ./jail7

edit the kernel config and change ident to be the name of the jail:
vi jail7
ident           jail7

edit /sys/conf/newvers.sh to add –jc1 to the end of the BRANCH string (RELEASE-jc1)
vi /sys/conf/newvers.sh

3. bring over patches from backup2

The various patches are in /mnt/data4/build/freebsd/patches on backup2. There are dirs for each version. Not all dirs are populated, but patches for later versions work on older ones unless there is a new patch in the older dir.

cd /usr/src
scp backup2:"/mnt/data4/build/freebsd/patches/6.x/*" .

Apply patches:
patch -l < jls-patch

Apply these only to 2950 with PAE:
patch -p0 < gvinum-staticcompile-patch
patch -p0 < gvinum-bin-patch

4. build, install kernel
cd /usr/src
make buildkernel;  pagedave
(2min)

cd /boot
mv kernel.old kernel.old.old
(optional move old kernel out of the way)

cd /usr/src
make installkernel

cd /boot
mv kernel.old kernel.6.2


5. take ½ of mirror and test (boot up) in new hardware

6. improved loader configs:

/boot/loader.conf:
console="comconsole,vidconsole"
boot_serial="YES"
boot_multicons="YES"

7. build userland
cd /usr/src
make buildworld; pagedave
(33mins)
make installworld 

mergemaster –i
reboot
8. patch jail against starting jails with rtprio

mv /usr/sbin/jail /usr/sbin/jail_
echo '#\!/bin/sh\
/usr/sbin/rtprio -t /usr/sbin/jail_ $*' > /usr/sbin/jail
chmod +x /usr/sbin/jail

FreeBSD 7.0[edit]

Last updated 2008-04-30 


All time extimates below assume disks aren’t scrubbing. Setup instructions below are for LSI card:

1. make sure bios is setup for bios console redirect
Supermicro:
Console redirection:
Com port addr: on-board COM A
Baud: 38400
Console type: vt100
Flow control: none
Console connection: direct
Continue cr after post: on

2450:
Make sure running bios A09
Console Redirection: VT100/VT220

2. assuming mirrors (or at least disks) created (if not, refer to this), boot to disk 1 of 5.4
skip kernel config (enter)
custom install

partition ->
move cursor to amrd0, hit space (takes you to partition map screen)
a for entire disk
q to quit and save
standard mbr (no boot manager)
space to unselect amrd0
cursor over amrd1
space
a for entire disk
q to quit and save
none (leave untouched)

cursor over amrd0
space
(takes you into part. Screen again) q to exit
none
Make sure both are checked and tab to ok

Label ->
Make sure mfid0 is highlighted

/ 256M
swap 2G (for 2950 make it 4G)
/var 256M (4.6G)
/tmp 256M
/usr 4G
/mnt/data1 remaining space (no need to newfs)

Make sure to toggle S for soft updates on all (should look like UFS2+S Y under the Newfs column)

move cursor to mfid1
swap 8G
/mnt/data2 remaining space (no need to newfs)

q to save and exit

distributions ->
developer
yes to install ports
exit

media ->
cd

commit ->
yes
(2450: 14mins, supermicro: 12mins)

yes to "visit general config" ->
Set root pwd

Add user ‘user’ member group is wheel, set password

Set tz

Networking->interfaces->Fxp0 ->
No IPV6
dhcp=no
Set hostname & domain
Enable sshd

exit...
exit install ->
yes

take the cd out and let the machine reboot

3. double check the date/time

4. edit /etc/make.conf (only add the console speed line if this is a supermicro capable of outputting BIOS redirect at that speed – 2450’s can only do 9600)
echo "WITHOUT_X11=yes \
KERNCONF=jail2 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf



5. add settings to /boot/loader.conf and /boot.config

echo "-Dh" >> /boot.config

echo 'console="comconsole,vidconsole" \
boot_multicons="YES" \
boot_serial="YES" \
mfi_linux_load="YES" \
comconsole_speed="115200"' >> /boot/loader.conf

(leave out the speed and mfi lines for 2450s)

6. turn off all ttyv's except 0 and 1 in /etc/ttys
also turn on ttyd0, change type to vt100:
vi /etc/ttys
ttyd0   "/usr/libexec/getty std.9600"  vt100   on secure

kill -1 1

on console server:
vi /etc/remote
(rename port to jail18 depending on where and which digi plugged into)
test serial console

7. populate hosts
echo "10.1.4.3 backup2" >> /etc/hosts
echo "10.1.4.8 backup1" >> /etc/hosts


8. put key in authorized_keys on backup2
cd
ssh-keygen -t dsa -b 1024 
(default location, leave password blank)

cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'

confirm that you can ssh to backup2 and backup1 without getting a login prompt

ssh backup2 hostname

ssh backup1 hostname

9. create & populate binaries/scripts dirs
mkdir -p /usr/local/jail/bin
mkdir -p /usr/local/jail/rc.d
mkdir -p /usr/local/jail/template/
mkdir /mnt/data1
mkdir /mnt/data2
scp backup2:"/mnt/data4/bin/freebsd7.x/*" /usr/local/jail/bin
cd /usr/local/jail/rc.d/
touch quad1
touch quad2
touch quad3
touch quad4
touch safe1
touch safe2
touch safe3
touch safe4
chmod +x *
cd /usr/local/jail/bin
ln -s /usr/local/jail/rc.d/quad1 quad1
ln -s /usr/local/jail/rc.d/quad2 quad2
ln -s /usr/local/jail/rc.d/quad3 quad3
ln -s /usr/local/jail/rc.d/quad4 quad4
ln -s /usr/local/jail/rc.d/safe1 safe1
ln -s /usr/local/jail/rc.d/safe2 safe2
ln -s /usr/local/jail/rc.d/safe3 safe3
ln -s /usr/local/jail/rc.d/safe4 safe4
ln -s /usr/local/jail/bin/jailmake_zfs jailmake
--OR--
ln -s /usr/local/jail/bin/jailmake_geom jailmake

ln -s /usr/local/jail/bin/js_zfs js
--OR--
ln -s /usr/local/jail/bin/js_geom js

rehash

10. edit root's path and login script:
vi /root/.cshrc
add to path: 
/usr/local/jail/bin 
(if adaptec card installed, also add /compat/linux/usr/sbin)

Change alias entries (add G):
alias la        ls -aG
alias lf        ls -FAG
alias ll        ls -lAG
alias ls        ls -AG
alias mbm       mb mount
alias mbu       mb umount

and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "

at the bottom of the file add:
set sshtty=`who am i|awk '{print $2}'`
/usr/sbin/rtprio 3 -`psj | grep $sshtty | awk '{print $2}'`

set shortty=`who am i | awk '{print $2}' | sed -E 's/.*(..)$/\1/'`
foreach x (`psj | grep sh | grep $shortty | awk '{print $2}'`)
/usr/sbin/rtprio 2 -$x
end

To load the new file:
source /root/.cshrc

11. install cvsup
cd /usr/ports/net/cvsup-without-gui 
make install clean; rehash; mail -s 'cvs installed' dave.boodman@vtext.com < /dev/null

(stay close for gettext options, 2450: 27mins, supermicro: 17mins, 2950: 22mins)

12. get latest sources for this release:
cd /usr/src 
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_7\
*default delete use-rel-suffix\
*default compress\
src-all" > sup


-OR-

echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_7_1\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

cvsup sup ; mail -s 'cvs sup done' dave.boodman@vtext.com < /dev/null

(2450, ~12mins, supermicro, 27mins, 2950: 7mins)


13. configure new kernel. 
for i386:
cd /usr/src/sys/i386/conf 
scp backup2:/mnt/data4/build/freebsd/kern_config-7.0-PAE ./mx2

-or-
for amd64:
cd /usr/src/sys/amd64/conf 
scp backup2:/mnt/data4/build/freebsd/kern_config-7.0-amd64 ./jail2

------

edit the kernel config and change ident to be the name of the jail:
vi jail2
ident           jail2

edit /sys/conf/newvers.sh to add –jc2 to the end of the BRANCH string (RELEASE-jc2)
vi /sys/conf/newvers.sh

notes: http://www.FreeBSD.org/doc/en_US.ISO8859-1/books/handbook/kernelconfig-config.html

14. bring over patches from backup2

The various patches are in /mnt/data4/build/freebsd/patches on backup2. There are dirs for each version. Not all dirs are populated, but patches for later versions work on older ones unless there is a new patch in the older dir.

cd /usr/src
scp backup2:"/mnt/data4/build/freebsd/patches/7.x/*" .

Apply patches:
patch -l < jls-patch

15. build, install kernel and world
cd /usr/src
make buildworld buildkernel installkernel; mail -s 'kernel build done' dave.boodman@vtext.com < /dev/null
(2450: 1:56min, supermicro: 59mins, 2950: 38mins)
make installworld 
(2450: 3min, supermicro: 1min, 2950: :34)
mergemaster -i
delete /var/tmp/temproot
delete bsnmpd
delete temporary ./etc/hosts
delete temporary ./etc/motd
delete /var/tmp/temproot

cd /usr/src/sys/modules/zfs
make 
make install

16. populate devfs ruleset
scp backup2:/mnt/data4/build/freebsd/devfs.rules /etc

17. populate /etc/rc.conf with IPs and NFS settings
vi /etc/rc.conf

kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
sshd_enable="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.4.102"
devfs_system_ruleset="devfsrules_show_all"

hostname="jail2.johncompanies.com"
ifconfig_xl0="inet 10.1.4.102 netmask 255.255.255.0"
ifconfig_fxp0="inet 69.55.228.53 netmask 255.255.255.0"
defaultrouter="69.55.228.1"
#ifconfig_fxp0_alias0="inet 69.55.2xx.xx netmask 255.255.255.0"

static_routes="t1 office"
route_t1="-net 10.1.5 10.1.4.2"
route_office="-net 10.1.6 10.1.4.2"
zfs_enable="YES"

18. make sure sysctls are set and preserved after reboot
echo "kern.consmute=0\
kern.ipc.shm_use_phys=1\
kern.ipc.shmall=65535\
kern.ipc.shmmax=134217728\
net.inet.tcp.syncookies=0\
kern.maxfiles=32768\
kern.fallback_elf_brand=3\
kern.maxprocperuid=4000\
security.jail.sysvipc_allowed=1\
security.jail.allow_raw_sockets=1\
security.jail.socket_unixiproute_only=1\
security.jail.chflags_allowed=0\
compat.linux.osrelease=2.6.12\
kern.maxvnodes=400000" >> /etc/sysctl.conf

19. mount procfs
echo "proc                    /proc           procfs  rw              0       0" >> /etc/fstab

For Dell 2950/2450:
echo "linprocfs               /usr/compat/linux/proc linprocfs rw     0       0" >> /etc/fstab

For Dell 2950:
echo "linsysfs                /usr/compat/linux/sys linsysfs rw       0       0" >> /etc/fstab

20. reboot. Confirm new kernel is loaded

uname –a

Check devfs rules:
devfs rule showsets
devfs rule -s 3 show

21. update ports:
cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_7_0\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup

cvsup sup; mail -s 'cvs sup ports done' dave.boodman@vtext.com < /dev/null

(2450: 18mins, supermicro: 19mins; 2950: 24mins)

22. Install raid mgmt tool

For LSI based cards:

install linux_base:
cd /usr/ports/emulators/linux_base-fc4
make install clean 
(2450: 7min, supermicro: 3mins, 2950: 14mins)

(for LSI)
cd /usr/ports/sysutils/linux-megamgr
make install clean

cd /usr/ports/sysutils/megarc
make install clean

(for Perc5/i)
cd /usr/ports/sysutils/linux-megacli
make install clean

Test:
rehash; megacli ldinfo lall a0

For adaptec:
On jail18: 
scp /compat/linux/usr/sbin/aaccli user@10.1.4.102:~

mv ~user/aaccli /compat/linux/usr/sbin/aaccli

test out;
/compat/linux/usr/sbin/aaccli

For 3w9690:
scp backup2:/d4/build/3ware/tw_cli-freebsd-x86_64-9.5.0.1.tgz /usr/local/sbin
cd /usr/local/sbin xzvf tw_cli-freebsd-x86_64-9.5.0.1.tgz
rm tw_cli-freebsd-x86_64-9.5.0.1.tgz

test out;
./tw_cli /c0 show allstatus

23. install rsync from ports
cd /usr/ports/net/rsync
make install clean

choose default options

24. install perl from ports
Not necessary if linux_base is installed

cd /usr/ports/lang/perl5.8/
make install clean; rehash
(supermicro: 5min)

25. configure inetd to respond to mrtg load queries
echo "load    stream  tcp     nowait  user    /usr/local/jail/bin/load.pl  load.pl" >> /etc/inetd.conf

echo "load            12384/tcp" >> /etc/services

26. install bb client
adduser
Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username   : bb
Password   : <random>
Full Name  : bb
Uid        : 1984
Class      :
Groups     : bb
Home       : /home/bb
Shell      : /bin/sh
Locked     : no
OK? (yes/no): yes

cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd.tar .
tar xf bb-freebsd.tar

edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.4.108 jail2.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

vi /home/bb/bbc1.9e-btf/ext/openfiles 
MACHINE="jail2,johncompanies,com"      # HAS TO BE IN A,B,C FORM

cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh 
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src
make; make install
cd ..
./runbb.sh start
more BBOUT 
(look for errors)
exit

echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh


NOTE: to get bb working on amd, had to copy over bin dir from linux dist 

27. configure load mrtg, on mail
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg 
(add new entry to file following existing format)

28. configure bb on mail:
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
10.1.4.108 jail2.johncompanies.com # ssh

su bb
cd
bbsrc/bb/runbb.sh restart ; exit

29. remove reserve space, enable softupdates (probably already set, so not necessary)
NOT APPLICABLE IF USING GVINUM
cd
umount /mnt/data1
umount /mnt/data2
tunefs -m 0 /mnt/data1
tunefs -m 0 /mnt/data2
mount -a

30. configure ntp
echo "server 10.1.4.5" > /etc/ntp.conf

/usr/sbin/ntpd -p /var/run/ntpd.pid
sleep 2; ntpq -p
(confirm it’s able to reach our time server)

31. mrtg switch graphs

32. fwd and reverse lookups on ns1c
vr johncompanies.com
vi internal.johncompanies.com
rndc reload johncompanies.com IN private
(edit the PTR too)


33. create zpools

Make a g partition:

bsdlabel -e /dev/mfid0s1

given:
# /dev/aacd0s1:
8 partitions:
#        size   offset    fstype   [fsize bsize bps/cpg]
  a:   262144        0    4.2BSD     2048 16384 16392
  b:  4194304   262144      swap
  c: 143363997        0    unused        0     0         # "raw" part, don't edit
  d:   524288  4456448    4.2BSD     2048 16384 32776
  e:   524288  4980736    4.2BSD     2048 16384 32776
  f:  6291456  5505024    4.2BSD     2048 16384 28552

new offset = 6291456 + 5505024 = 11796480
new size is size for 'c' partition minus the new start from above
143363997 - 11796480 = 131567517
So:
g: 131567517 11796480 unused 0 0

bsdlabel -e /dev/mfid1s1
change d to g


zpool create pool1 mfid0s1g
zpool create pool2 mfid1s1g
zfs set atime=off pool1
zfs set atime=off pool2

echo ' vfs.zfs.prefetch_disable="1" \
vfs.zfs.arc_min=16777216 \
vfs.zfs.arc_max=33554432 \
vm.kmem_size_max="1G" # for i386\
vm.kmem_size="1G" # for i386\
vm.kmem_size_max="1.5G" # for amd64\
vm.kmem_size="1.5G" # for amd64\
vfs.zfs.zil_disable="1" ' >> /boot/loader.conf


35. create the jail template


zfs create -o mountpoint=/mnt/data1/jail -o quota=4G pool1/jail

cd /usr/ports/sysutils/jailutils
make install clean

cd /usr/src
make world DESTDIR=/mnt/data1/jail
(2450: 2:28mins, supermicro: 55mins, 2950: 1h )
cd etc
make distribution DESTDIR=/mnt/data1/jail
mount -t devfs devfs /mnt/data1/jail/dev
devfs -m /mnt/data1/jail/dev rule -s 3 applyset 
cd /mnt/data1/jail
ln -sf dev/null kernel
cp /usr/local/sbin/jkill /mnt/data1/jail/sbin

jail /mnt/data1/jail testhostname 192.168.11.100 /bin/sh
csh
touch /etc/fstab
echo 'network_interfaces=""\
hostname="newsystem"\
kern_securelevel_enable="NO"\
sendmail_enable="YES"\
sshd_enable="YES"' > /etc/rc.conf

echo "nameserver 69.55.225.225\
nameserver 69.55.230.3" >> /etc/resolv.conf

vi /etc/crontab
remove the adjkerntz lines
comment out periodic’s and put this line above them:
# IF YOU UNCOMMENT THESE, PLEASE ADJUST THEIR RUN TIME

rm -rf /etc/periodic/daily/400.status-disks

check /tmp for crap

vi /etc/periodic/security/100.chksetuid
replace: MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`
with: MP='/' (use single quotes)

mkdir -p /usr/compat/linux/dev

adduser (Add account for user, make sure in wheel group)

user/root passwd: 8ico2987

Set root password

vi /usr/home/user/.profile (and add to the file):
TERM=vt100;     export TERM

tzsetup

newaliases 

rm /sbin/halt /sbin/reboot
ln /sbin/jkill /sbin/halt
ln /sbin/jkill /sbin/reboot

#cd /dev
#rm console
#ln -s null console

vi /etc/syslog.conf (comment out console and move to /var/log/messages):
#*.err;kern.warning;auth.notice;mail.crit               /dev/console *.err;kern.warning;auth.notice;mail.crit                /var/log/messages 

cd /libexec
mv ld-elf32.so.1 ld-elf32.so.1-orig
ln ld-elf.so.1 ld-elf32.so.1

exit
exit

cd /usr/ports
make -DNOCLEANDEPENDS clean 
(2450: 15mins , supermicro: 29mins, 2950: 24mins)
rm -fr /usr/ports/distfiles/*
cp -r /usr/ports /mnt/data1/jail/usr (2450: 2:00 mins , supermicro: 15mins, 2950: 3mins)

rm /mnt/data1/jail/root/.history

cd 
zfs snapshot pool1/jail@now
zfs send pool1/jail@now | cat > /usr/local/jail/template/70template 
zfs destroy pool1/jail@now

zfs destroy pool1/jail

rmdir /mnt/data1/jail
f

36. setup backups
echo '#\!/bin/sh\
backupdir=/data/jail2\
server=backup1\
\
## ENTRY /etc\
## ENTRY /usr/local/etc\
## ENTRY /usr/local/jail\
## ENTRY /root/logs' > /usr/local/jail/bin/backup.config

on backup1:
setup backup dirs:
mkdir -p /data/jail2/0

add the system to 
vi /usr/local/sbin/snapshot_rotate

on mail:
vi /usr/local/www/mgmt/cgi/backupgraph.pl
(add hostname)

Edit /usr/local/jail/bin/backup to use the right drives

37. mkdir /root/logs

38. edit sshd_config for security
vi /etc/ssh/sshd_config
ListenAddress 69.55.238.233
ListenAddress 10.1.4.108

kill -1 `cat /var/run/sshd.pid`

39. add crontab entries
crontab -e
5 0 * * * /usr/local/jail/bin/backup.zfs
1 0 1 * * /usr/local/jail/bin/ipfwreset
0 18 * * * /usr/local/jail/bin/ipfwbackup
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /usr/local/jail/bin/trafstats

On 2950 with Perc5/i also add:
*/5 * * * * /usr/local/jail/bin/perc5iraidchk

40. Reboot notify script
ln -s /usr/local/jail/bin/notify.sh /usr/local/etc/rc.d/notify.sh 

41. add to management db (on mail and devweb) jc.ref_machines and jc.ref_templates

uname -r
6.2-RELEASE-jc1

insert into ref_machines values (null,'mx2','mx2.johncompanies.com',0,'m');
select machine_id from ref_machines where host='mx2';
+------------+
| machine_id |
+------------+
|         35 |
+------------+
insert into ref_templates values ('',' 7.1-RELEASE-jc2',44,'FreeBSD 7.1',0);

42. add to server/cabinet map. On mail:
vi /usr/local/www/mgmt/html/cabinetmap.html

43. add an outside blocking rule to the firewall, so this machine can only be reached from inside the firewall. Follow example already in firewall jail17 is:
 
00117 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.228.200
00117 deny ip from any to 69.55.228.200

jail19 would be 00119...
ipfw add 00107 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.230.108
ipfw add 00107 deny ip from any to 69.55.220.108

44. select customers for probe map

47. patch jail against starting jails with rtprio

mv /usr/sbin/jail /usr/sbin/jail_
echo '#\!/bin/sh\
/usr/sbin/rtprio -t /usr/sbin/jail_ $*' > /usr/sbin/jail
chmod +x /usr/sbin/jail

48. make sure mail works
If there are map errors:
cd /etc/mail; make maps

49. move and pare down generic kernel
mv kernel.old/ kernel.GENERIC

/  too full, remove old kernel modules except for:

Id Refs Address            Size     Name
 1   14 0xffffffff80100000 ac6c08   kernel
 2    1 0xffffffff80bc8000 1128     mfi_linux.ko
 3    4 0xffffffff80bca000 39138    linux.ko
 4    1 0xffffffffb48f8000 358c     linprocfs.ko
 5    1 0xffffffffb48fc000 9d3      linsysfs.ko
 6    1 0xffffffffb490c000 80ee8    zfs.ko


50. ntp doesn’t seem to start from rc so,

echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh
chmod 0700 /usr/local/etc/rc.d/ntp.sh

FreeBSD 7.1[edit]

Last updated 2009-02-16 

All time extimates below assume disks aren’t scrubbing. Setup instructions below are for LSI card:

1. make sure bios is setup for bios console redirect
Supermicro:
Console redirection:
Com port addr: on-board COM A
Baud: 38400
Console type: vt100
Flow control: none
Console connection: direct
Continue cr after post: on

2450:
Make sure running bios A09
Console Redirection: VT100/VT220

2. assuming mirrors (or at least disks) created (if not, refer to this), boot to disk 1 of 5.4
skip kernel config (enter)
custom install

partition ->
move cursor to amrd0, hit space (takes you to partition map screen)
a for entire disk
q to quit and save

NOTE
For gvin probs made 2 slices, 1st 8960M
Freebsd boot mgr

standard mbr (no boot manager)
space to unselect amrd0
cursor over amrd1
space
a for entire disk
q to quit and save
none (leave untouched)

cursor over amrd0
space
(takes you into part. Screen again) q to exit
none
Make sure both are checked and tab to ok

Label ->
Make sure mfid0 is highlighted

/ 256M
swap 2G (for 2950 make it 4G)
/var 256M (4.6G)
/tmp 256M
/usr 4G
/mnt/data1 remaining space (no need to newfs)

Make sure to toggle S for soft updates on all (should look like UFS2+S Y under the Newfs column)

move cursor to mfid1
swap 8G
/mnt/data2 remaining space (no need to newfs)

q to save and exit

distributions ->
developer
yes to install ports
exit

media ->
cd

commit ->
yes
(2450: 14mins, supermicro: 12mins)

yes to "visit general config" ->
Set root pwd

Add user ‘user’ member group is wheel, set password

Set tz

Networking->interfaces->Fxp0 ->
No IPV6
dhcp=no
Set hostname & domain
Enable sshd

exit...
exit install ->
yes

take the cd out and let the machine reboot

3. double check the date/time

4. edit /etc/make.conf (only add the console speed line if this is a supermicro capable of outputting BIOS redirect at that speed – 2450’s can only do 9600)
echo "WITHOUT_X11=yes \
KERNCONF=jail2 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf



5. add settings to /boot/loader.conf and /boot.config

echo "-Dh" >> /boot.config

echo 'console="comconsole,vidconsole" \
boot_multicons="YES" \
boot_serial="YES" \
mfi_linux_load="YES" \
comconsole_speed="115200"' >> /boot/loader.conf

(leave out the speed and mfi lines for 2450s)

6. turn off all ttyv's except 0 and 1 in /etc/ttys
also turn on ttyd0, change type to vt100:
vi /etc/ttys

ttyv2   "/usr/libexec/getty Pc"         cons25  off secure
ttyv3   "/usr/libexec/getty Pc"         cons25  off secure
ttyv4   "/usr/libexec/getty Pc"         cons25  off secure
ttyv5   "/usr/libexec/getty Pc"         cons25  off secure
ttyv6   "/usr/libexec/getty Pc"         cons25  off secure
ttyv7   "/usr/libexec/getty Pc"         cons25  off secure

ttyd0   "/usr/libexec/getty std.9600"  vt100   on secure

kill -1 1

on console server:
vi /etc/remote
(rename port to jail18 depending on where and which digi plugged into)
test serial console

7. populate hosts
echo "10.1.4.3 backup2" >> /etc/hosts
echo "10.1.4.8 backup1" >> /etc/hosts


8. put key in authorized_keys on backup2
cd
ssh-keygen -t dsa -b 1024 
(default location, leave password blank)

cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys' 
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'

confirm that you can ssh to backup2 and backup1 without getting a login prompt

ssh backup2 hostname

ssh backup1 hostname

9. create & populate binaries/scripts dirs
mkdir -p /usr/local/jail/bin
mkdir -p /usr/local/jail/rc.d
mkdir -p /usr/local/jail/template/
mkdir /mnt/data1
mkdir /mnt/data2
scp backup2:"/mnt/data4/bin/freebsd7.x/*" /usr/local/jail/bin
cd /usr/local/jail/rc.d/
touch quad1
touch quad2
touch quad3
touch quad4
touch safe1
touch safe2
touch safe3
touch safe4
chmod +x *
cd /usr/local/jail/bin
ln -s /usr/local/jail/rc.d/quad1 quad1
ln -s /usr/local/jail/rc.d/quad2 quad2
ln -s /usr/local/jail/rc.d/quad3 quad3
ln -s /usr/local/jail/rc.d/quad4 quad4
ln -s /usr/local/jail/rc.d/safe1 safe1
ln -s /usr/local/jail/rc.d/safe2 safe2
ln -s /usr/local/jail/rc.d/safe3 safe3
ln -s /usr/local/jail/rc.d/safe4 safe4
ln -s /usr/local/jail/bin/jailmake_geom jailmake
ln -s /usr/local/jail/bin/js_geom js
ln -s /usr/local/jail/bin/canceljail_geom canceljail
ln -s /usr/local/jail/bin/jailmakeempty_geom jailmakeempty

ln -s /usr/local/jail/bin/jailmake_md jailmake
ln -s /usr/local/jail/bin/js_md js
ln -s /usr/local/jail/bin/canceljail_md canceljail
ln -s /usr/local/jail/bin/jailmakeempty_md jailmakeempty
ln -s /usr/local/jail/bin/postboot_md postboot

rehash

10. edit root's path and login script:
vi /root/.cshrc

Change alias entries (add G):
alias la        ls -aG
alias lf        ls -FAG
alias ll        ls -lAG
alias ls        ls -AG
alias mbm       mb mount
alias mbu       mb umount


add to path: 
/usr/local/jail/bin 
(if adaptec card installed, also add /compat/linux/usr/sbin)

and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "

at the bottom of the file add:
set sshtty=`who am i|awk '{print $2}'`
/usr/sbin/rtprio 3 -`psj | grep $sshtty | awk '{print $2}'`

set shortty=`who am i | awk '{print $2}' | sed -E 's/.*(..)$/\1/'`
foreach x (`psj | grep sh | grep $shortty | awk '{print $2}'`)
/usr/sbin/rtprio 2 -$x
end

To load the new file:
source /root/.cshrc

11. install cvsup
cd /usr/ports/net/cvsup-without-gui 
make install clean; rehash; mail -s 'cvs installed' dave.boodman@vtext.com < /dev/null

(stay close for gettext options, 2450: 27mins, supermicro: 17mins, 2950: 22mins)

12. get latest sources for this release:
cd /usr/src 
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_7_1\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

-OR-

echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_7\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

(stable)


cvsup sup ; mail -s 'cvs sup done' dave.boodman@vtext.com < /dev/null

(2450, ~12mins, supermicro, 27mins, 2950: 7mins)


13. configure new kernel. 
for i386:
cd /usr/src/sys/i386/conf 
scp backup2:/mnt/data4/build/freebsd/kern_config-7.0-PAE ./mx2

-or-

for amd64:
cd /usr/src/sys/amd64/conf 
scp backup2:/mnt/data4/build/freebsd/kern_config-7.1-amd64 ./jail2

-------

edit the kernel config and change ident to be the name of the jail:
vi jail2
ident           jail2

edit /sys/conf/newvers.sh to add –jc2 to the end of the BRANCH string (RELEASE-jc2)
vi /sys/conf/newvers.sh

notes: http://www.FreeBSD.org/doc/en_US.ISO8859-1/books/handbook/kernelconfig-config.html

14. bring over patches from backup2

The various patches are in /mnt/data4/build/freebsd/patches on backup2. There are dirs for each version. Not all dirs are populated, but patches for later versions work on older ones unless there is a new patch in the older dir.

cd /usr/src
scp backup2:"/mnt/data4/build/freebsd/patches/7.x/*" .

Apply patches:
patch -l < jls-patch


15. build, install kernel and world

cd /boot

mv kernel kernel.GENERIC
cd kernel.GENERIC
mkdir hold
mv mfi_linux.ko hold/
mv linux.ko hold/
mv linprocfs.ko hold/
mv linsysfs.ko hold/
mv geom_vinum.ko hold/
mv geom_concat.ko hold/
rm *.ko
rm *.symbols
mv hold/* .
rmdir hold/

cd /usr/src
make buildkernel installkernel

make buildworld ; mail -s 'kernel build done' dave.boodman@vtext.com < /dev/null
(2450: 1:56min, supermicro: 59mins, 2950: 38mins)
make installworld 
(2450: 3min, supermicro: 1min, 2950: :34)
mergemaster -i
delete /var/tmp/temproot
delete bsnmpd
delete temporary ./etc/hosts
delete temporary ./etc/motd
delete /var/tmp/temproot

cd /sys/modules/geom/geom_vinum
make 
make install

16. populate devfs ruleset
scp backup2:/mnt/data4/build/freebsd/devfs.rules /etc

17. populate /etc/rc.conf with IPs and NFS settings
vi /etc/rc.conf

kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.4.102"
devfs_system_ruleset="devfsrules_show_all"

#ifconfig_xl0="inet 10.1.4.102 netmask 255.255.255.0"
#ifconfig_fxp0="inet 69.55.228.53 netmask 255.255.255.0"
#ifconfig_fxp0_alias0="inet 69.55.2xx.xx netmask 255.255.255.0"

static_routes="t1 office"
route_t1="-net 10.1.5 10.1.4.2"
route_office="-net 10.1.6 10.1.4.2"
gvinum_enable="YES"
fsck_y_enable="YES"
background_fsck="NO"
rc_mfi_raid_tty_log="YES"

18. make sure sysctls are set and preserved after reboot
echo "kern.consmute=0\
kern.ipc.shm_use_phys=1\
kern.ipc.shmall=131070\
kern.ipc.shmmax=134217728\
net.inet.tcp.syncookies=0\
kern.maxfiles=32768\
kern.fallback_elf_brand=3\
kern.maxprocperuid=4000\
security.jail.sysvipc_allowed=1\
security.jail.allow_raw_sockets=1\
security.jail.socket_unixiproute_only=1\
security.jail.chflags_allowed=0\
dev.amr.0.allow_volume_configure=1\
compat.linux.osrelease=2.6.12" >> /etc/sysctl.conf

19. mount procfs
echo "proc                    /proc           procfs  rw              0       0" >> /etc/fstab

For Dell 2950/2450:
echo "linprocfs               /usr/compat/linux/proc linprocfs rw     0       0" >> /etc/fstab

For Dell 2950:
echo "linsysfs                /usr/compat/linux/sys linsysfs rw       0       0" >> /etc/fstab

mkdir -p /usr/compat/linux/proc
mkdir -p /usr/compat/linux/sys

20. reboot. Confirm new kernel is loaded

uname -a

Check devfs rules:
devfs rule showsets
devfs rule -s 3 show

21. update ports:
cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_7_1\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup

cvsup sup; mail -s 'cvs sup ports done' dave.boodman@vtext.com < /dev/null

(2450: 18mins, supermicro: 19mins; 2950: 24mins)

22. Install raid mgmt tool

For LSI based cards:

install linux_base:
cd /usr/ports/emulators/linux_base-fc4
make install clean 
(2450: 7min, supermicro: 3mins, 2950: 14mins)

(for LSI)
cd /usr/ports/sysutils/linux-megamgr
make install clean

cd /usr/ports/sysutils/megarc
make install clean

(for Perc5/i, 6/i)
cd /usr/ports/sysutils/linux-megacli2
make install clean

Test:
rehash; megacli ldinfo lall a0

For adaptec:
On jail18: 
scp /compat/linux/usr/sbin/aaccli user@10.1.4.102:~

mv ~user/aaccli /compat/linux/usr/sbin/aaccli

test out;
/compat/linux/usr/sbin/aaccli

For 3w9690:
scp backup2:/d4/build/3ware/tw_cli-freebsd-x86_64-9.5.0.1.tgz /usr/local/sbin
cd /usr/local/sbin xzvf tw_cli-freebsd-x86_64-9.5.0.1.tgz
rm tw_cli-freebsd-x86_64-9.5.0.1.tgz

test out;
./tw_cli /c0 show allstatus

23. install rsync from ports
cd /usr/ports/net/rsync
make install clean

choose default options

25. configure inetd to respond to mrtg load queries
echo "load    stream  tcp     nowait  user    /usr/local/jail/bin/load.pl  load.pl" >> /etc/inetd.conf

echo "load            12384/tcp" >> /etc/services

26. install bb client
adduser
Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]: 
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username   : bb
Password   : <random>
Full Name  : bb
Uid        : 1984
Class      :
Groups     : bb
Home       : /home/bb
Shell      : /bin/sh
Locked     : no
OK? (yes/no): yes

cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd_linuxcompat.tgz .
tar xzf bb-freebsd_linuxcompat.tgz

edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.4.102 jail2.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

vi /home/bb/bbc1.9e-btf/ext/openfiles 
MACHINE="jail2,johncompanies,com"      # HAS TO BE IN A,B,C FORM

cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh 
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf
./runbb.sh start
more BBOUT 
(look for errors)
exit

echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh


NOTE: to get bb working on amd, had to copy over bin dir from linux dist 

27. configure load mrtg, on mail
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg 
(add new entry to file following existing format)

28. configure bb on mail:
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
10.1.4.102 jail2.johncompanies.com # ssh

su bb
cd
bbsrc/bb/runbb.sh restart ; exit

29. remove reserve space, enable softupdates (probably already set, so not necessary)
NOT APPLICABLE IF USING GVINUM
cd
umount /mnt/data1
umount /mnt/data2
tunefs -m 0 /mnt/data1
tunefs -m 0 /mnt/data2
mount -a

30. configure ntp
echo "server 10.1.4.5" > /etc/ntp.conf

/usr/sbin/ntpd -p /var/run/ntpd.pid
sleep 2; ntpq -p
(confirm it’s able to reach our time server)

32. fwd and reverse lookups on ns1c
vr johncompanies.com
vi internal.johncompanies.com
rndc reload johncompanies.com IN private
(edit the PTR too)


33. create gvinum volumes

Make a g partition:

2950:
bsdlabel -e /dev/mfid0s1
bsdlabel -e /dev/mfid1s1
bsdlabel -e /dev/mfid2s1

given:
# /dev/aacd0s1:
8 partitions:
#        size   offset    fstype   [fsize bsize bps/cpg]
  a:   262144        0    4.2BSD     2048 16384 16392
  b:  4194304   262144      swap
  c: 143363997        0    unused        0     0         # "raw" part, don't edit
  d:   524288  4456448    4.2BSD     2048 16384 32776
  e:   524288  4980736    4.2BSD     2048 16384 32776
  f:  6291456  5505024    4.2BSD     2048 16384 28552

new offset = 6291456 + 5505024 = 11796480
new size is size for 'c' partition minus the new start from above
143363997 - 11796480 = 131567517
So:
g: 131567517 11796480 unused 0 0

bsdlabel -e /dev/amrd1s1
change d to g

For a 146G drive (after OS and 4G swap), we can fit 127 1G volumes so:

echo 'drive data1 device /dev/mfid0s1g' > /tmp/cgv

echo '#\!/bin/sh\
i="1"\
while [ $i -le 127 ]\
do\
  echo "volume v$i" >> /tmp/cgv;\
  echo 'plex org concat' >> /tmp/cgv;\
  echo 'sd length 1g drive data1' >> /tmp/cgv;\
i=`expr $i + 1`\
done' > /tmp/mkcgv
sh /tmp/mkcgv
gvinum create /tmp/cgv

volume 1
plex org concat
sd length 3g drive data1
volume 2
plex org concat
sd length 5g drive data1
volume 3
plex org concat
sd length 8g drive data1
volume 4
plex org concat
sd length 6g drive data1
volume 5
plex org concat
sd length 10g drive data1

for f in 1 2 3 4 5; do bsdlabel -rw /dev/gvinum/$f; done
for f in 1 2 3 4 5; do newfs /dev/gvinum/${f}a; done
for f in 1 2 3 4 5; do mkdir /mnt/data1/$f; mount /dev/gvinum/${f}a /mnt/data1/$f;done
for f in 1 2 3 4 5; do dd if=/dev/zero of=/mnt/data1/$f/this_is_$f bs=1m count=$f; done
for f in 1 2 3 4 5; do ll -h /mnt/data1/$f/; done


For a 300G drive (after OS and 4G swap), we can fit 273 1G volumes so:

echo 'drive data2 device /dev/mfid1s1g' > /tmp/cgv

echo '#\!/bin/sh\
i="128"\
while [ $i -le 401 ]\
do\
  echo "volume v$i" >> /tmp/cgv;\
  echo 'plex org concat' >> /tmp/cgv;\
  echo 'sd length 1g drive data2' >> /tmp/cgv;\
i=`expr $i + 1`\
done' > /tmp/mkcgv
sh /tmp/mkcgv
gvinum create /tmp/cgv


For a 300G drive (after OS and 4G swap), we can fit 273 1G volumes so:

echo 'drive data3 device /dev/mfid2s1g' > /tmp/cgv

echo '#\!/bin/sh\
i="402"\
while [ $i -le 675 ]\
do\
  echo "volume v$i" >> /tmp/cgv;\
  echo 'plex org concat' >> /tmp/cgv;\
  echo 'sd length 1g drive data3' >> /tmp/cgv;\
i=`expr $i + 1`\
done' > /tmp/mkcgv
sh /tmp/mkcgv
gvinum create /tmp/cgv

Sometimes there’s 2040m leftover so create one more smaller vol:
echo 'volume v64\
plex org concat\
sd length 2040m drive data1' >> /tmp/cgv

To delete:

echo '#\!/bin/sh\
i="1"\
while [ $i -le 127 ]\
do\
  echo "gvinum rm -r v$i" >> /tmp/dgv;\
i=`expr $i + 1`\
done' > /tmp/mkdgv
sh /tmp/mkdgv
sh /tmp/dgv

dd if=/dev/zero of=/dev/sdb seek=285474816 bs=512
dd if=/dev/zero of=/dev/mfid0s2 bs=1m count=1000 (do this to all drives)

35. create the jail template


touch /mnt/data1/jail 
mdconfig -a -t vnode -s 1g -f /mnt/data1/jail -u 0
bsdlabel -rw md0 
newfs4x /dev/md0c
mkdir /mnt/data1/jail-DIR
mount /dev/md0c /mnt/data1/jail-DIR

cd /usr/ports/sysutils/jailutils
make install clean

cd /usr/src
make world DESTDIR=/mnt/data1/jail-DIR
(2450: 2:28mins, supermicro: 55mins, 2950: 1h )
cd etc
make distribution DESTDIR=/mnt/data1/jail-DIR
mount -t devfs devfs /mnt/data1/jail-DIR/dev
devfs -m /mnt/data1/jail-DIR/dev rule -s 3 applyset 
cd /mnt/data1/jail-DIR
ln -sf dev/null kernel
cp /usr/local/sbin/jkill /mnt/data1/jail-DIR/sbin

jail /mnt/data1/jail-DIR testhostname 192.168.11.100 /bin/sh
csh
touch /etc/fstab
echo 'network_interfaces=""\
hostname="newsystem"\
kern_securelevel_enable="NO"\
sendmail_enable="YES"\
sshd_enable="YES"' > /etc/rc.conf

echo "nameserver 69.55.225.225\
nameserver 69.55.230.3" >> /etc/resolv.conf

vi /etc/crontab
remove the adjkerntz lines
comment out periodic’s and put this line above them:
# IF YOU UNCOMMENT THESE, PLEASE ADJUST THEIR RUN TIME

rm -rf /etc/periodic/daily/400.status-disks

check /tmp for crap

vi /etc/periodic/security/100.chksetuid
replace: MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`
with: MP='/' (use single quotes)

mkdir -p /usr/compat/linux/dev

adduser (Add account for user, make sure in wheel group)

user/root passwd: 8ico2987

Set root password

vi /usr/home/user/.profile (and add to the file):
TERM=vt100;     export TERM

tzsetup

newaliases 

rm /sbin/halt /sbin/reboot
ln /sbin/jkill /sbin/halt
ln /sbin/jkill /sbin/reboot

#cd /dev
#rm console
#ln -s null console

vi /etc/syslog.conf (comment out console and move to /var/log/messages):
#*.err;kern.warning;auth.notice;mail.crit               /dev/console *.err;kern.warning;auth.notice;mail.crit                /var/log/messages 

exit
exit

cd libexec
mv ld-elf32.so.1 ld-elf32.so.1-orig
ln ld-elf.so.1 ld-elf32.so.1

may have to run:
chflags noschg ld-elf32.so.1
chflags noschg ld-elf.so.1
then
chflags schg ld-elf.so.1
chflags schg ld-elf32.so.1


cd /usr/ports
make -DNOCLEANDEPENDS clean 
(2450: 15mins , supermicro: 29mins, 2950: 18mins)
rm -fr /usr/ports/distfiles/*
cp -r /usr/ports /mnt/data1/jail-DIR/usr (2450: 2:00 mins , supermicro: 15mins, 2950: 3mins)

rm /mnt/data1/jail-DIR/root/.history

cd 
dump -0a -f /usr/local/jail/template/71template /dev/md0c
umount /mnt/data1/jail-DIR/dev
umount /dev/md0c
rmdir /mnt/data1/jail-DIR


36. setup backups
echo '#\!/bin/sh\
backupdir=/data/jail2\
server=backup1\
\
## ENTRY /etc\
## ENTRY /usr/local/etc\
## ENTRY /usr/local/jail\
## ENTRY /root/logs' > /usr/local/jail/bin/backup.config

on backup1:
setup backup dirs:
ssh backup1 mkdir -p /data/jail2/0

add the system to 
vi /usr/local/sbin/snapshot_rotate

on mail:
vi /usr/local/www/mgmt/cgi/backupgraph.pl
(add hostname)

Edit /usr/local/jail/bin/backup.xxx to use the right drives and copy to /usr/local/jail/bin/backup 

37. mkdir /root/logs

38. edit sshd_config for security
vi /etc/ssh/sshd_config
ListenAddress 69.55.228.53
ListenAddress 10.1.4.102

kill -1 `cat /var/run/sshd.pid`

39. add crontab entries
crontab -e
5 0 * * * /usr/local/jail/bin/backup.md
1 0 1 * * /usr/local/jail/bin/ipfwreset
0 18 * * * /usr/local/jail/bin/ipfwbackup
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /usr/local/jail/bin/trafstats

On 2950 with Perc5/i also add:
*/5 * * * * /usr/local/jail/bin/perc5iraidchk

40. Reboot notify script
ln -s /usr/local/jail/bin/notify.sh /usr/local/etc/rc.d/notify.sh 

41. add to management db (on mail and devweb) jc.ref_machines and jc.ref_templates

uname -r
6.2-RELEASE-jc1

insert into ref_machines values (null,'mx2','mx2.johncompanies.com',0,'m');
select machine_id from ref_machines where host='mx2';
+------------+
| machine_id |
+------------+
|         35 |
+------------+
insert into ref_templates values ('',' 7.1-RELEASE-jc2',44,'FreeBSD 7.1',0);

42. add to server/cabinet map. On mail:
vi /usr/local/www/mgmt/html/cabinetmap.html

43. add an outside blocking rule to the firewall, so this machine can only be reached from inside the firewall. Follow example already in firewall jail17 is:
 
00117 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.228.200
00117 deny ip from any to 69.55.228.200

jail19 would be 00119...
ipfw add 00107 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.230.108
ipfw add 00107 deny ip from any to 69.55.220.108

44. select customers for probe map

46. make gv start on boot

scp backup2:/mnt/data4/build/freebsd/gvinum /etc/rc.d/gvinum

If they start stale:

echo '#\!/bin/sh\
i="1"\
while [ $i -le 127 ]\
do\
        gvinum start v$i\
        i=`expr $i + 1`\
done' > /tmp/startgv

sh /tmp/startgv


47. patch jail against starting jails with rtprio

mv /usr/sbin/jail /usr/sbin/jail_
echo '#\!/bin/sh\
/usr/sbin/rtprio -t /usr/sbin/jail_ $*' > /usr/sbin/jail
chmod +x /usr/sbin/jail

48. make sure mail works
If there are map errors:
cd /etc/mail; make maps

49. move and pare down generic kernel
mv kernel.old/ kernel.GENERIC

/  too full, remove old kernel modules except for:

Id Refs Address            Size     Name
 1   14 0xffffffff80100000 ac6c08   kernel
 2    1 0xffffffff80bc8000 1128     mfi_linux.ko
 3    4 0xffffffff80bca000 39138    linux.ko
 4    1 0xffffffffb48f8000 358c     linprocfs.ko
 5    1 0xffffffffb48fc000 9d3      linsysfs.ko
 6    1 0xffffffffb490c000 80ee8    zfs.ko


50. ntp doesn’t seem to start from rc so,

echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh
chmod 0700 /usr/local/etc/rc.d/ntp.sh

FreeBSD 7.2[edit]

Last updated 2009-06-18 

All time extimates below assume disks aren’t scrubbing. Setup instructions below are for LSI card:

1. make sure bios is setup for bios console redirect
2950:
Console redirection:
LCD string..
Date to GMT


2. assuming mirrors (or at least disks) created (if not, refer to this), boot to disk 1 of 7.2
skip kernel config (enter)
custom install

partition ->
move cursor to mfid0, hit space (takes you to partition map screen)
a for entire disk
q to quit and save


standard mbr (no boot manager)
space to unselect mfid0
cursor over mfid1
space
a for entire disk
q to quit and save
none (leave untouched)

cursor over mfid0
space
(takes you into part. Screen again) q to exit
none
Make sure both are checked and tab to ok

Label ->
Make sure mfid0 is highlighted

/ 256M
swap 2G (for 2950 make it 4G)
/var 256M (4.6G)
/tmp 256M
/usr 4G
/mnt/data1 remaining space (no need to newfs)

Make sure to toggle S for soft updates on all (should look like UFS2+S Y under the Newfs column)

move cursor to mfid1
swap 8G (or 4G if there’s a 3rd drive)
/mnt/data2 remaining space (no need to newfs)

q to save and exit

distributions ->
developer
yes to install ports
exit

media ->
cd

commit ->
yes
(2450: 14mins, supermicro: 12mins)

yes to "visit general config" ->
Set root pwd

Add user ‘user’ member group is wheel, set password

Set tz

Networking->interfaces->Fxp0 ->
No IPV6
dhcp=no
Set hostname & domain
Enable sshd

exit...
exit install ->
yes

take the cd out and let the machine reboot

3. double check the date/time

4. edit /etc/make.conf 
echo "WITHOUT_X11=yes \
KERNCONF=jail8 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf


5. add settings to /boot/loader.conf and /boot.config

echo "-Dh" >> /boot.config

echo 'console="comconsole,vidconsole" \
boot_multicons="YES" \
boot_serial="YES" \
mfi_linux_load="YES" \
comconsole_speed="115200"' >> /boot/loader.conf


6. turn off all ttyv's except 0 and 1 in /etc/ttys
also turn on ttyd0, change type to vt100:
vi /etc/ttys

ttyv2   "/usr/libexec/getty Pc"         cons25  off secure
ttyv3   "/usr/libexec/getty Pc"         cons25  off secure
ttyv4   "/usr/libexec/getty Pc"         cons25  off secure
ttyv5   "/usr/libexec/getty Pc"         cons25  off secure
ttyv6   "/usr/libexec/getty Pc"         cons25  off secure
ttyv7   "/usr/libexec/getty Pc"         cons25  off secure

ttyd0   "/usr/libexec/getty std.9600"  vt100   on secure

kill -1 1

on console server:
vi /etc/remote
(rename port to jail18 depending on where and which digi plugged into)
test serial console

7. populate hosts
echo "10.1.4.3 backup2" >> /etc/hosts
echo "10.1.4.8 backup1" >> /etc/hosts


8. put key in authorized_keys on backup2
cd
ssh-keygen -t dsa -b 1024 
(default location, leave password blank)

cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys' 
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'

confirm that you can ssh to backup2 and backup1 without getting a login prompt

ssh backup2 hostname

ssh backup1 hostname

9. create & populate binaries/scripts dirs
mkdir -p /usr/local/jail/bin
mkdir -p /usr/local/jail/rc.d
mkdir -p /usr/local/jail/template/
mkdir /mnt/data1
mkdir /mnt/data2
scp backup2:"/mnt/data4/bin/freebsd7.x/*" /usr/local/jail/bin
cd /usr/local/jail/rc.d/
touch quad1
touch deprecated
chmod +x *
cd /usr/local/jail/bin
ln -s /usr/local/jail/rc.d/quad1 quad1
ln -s /usr/local/jail/bin/jailmake_md jailmake
ln -s /usr/local/jail/bin/js_md js
ln -s /usr/local/jail/bin/canceljail_md canceljail
ln -s /usr/local/jail/bin/jailmakeempty_md jailmakeempty
ln -s /usr/local/jail/bin/postboot_md postboot

rehash

10. edit root's path and login script:
vi /root/.cshrc

Change alias entries (add G):
alias la        ls -aG
alias lf        ls -FAG
alias ll        ls -lAG
alias ls        ls -AG
alias mbm       mb mount
alias mbu       mb umount


add to path: 
/usr/local/jail/bin 
(if adaptec card installed, also add /compat/linux/usr/sbin)

and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "

at the bottom of the file add:
set sshtty=`who am i|awk '{print $2}'`
/usr/sbin/rtprio 3 -`psj | grep $sshtty | awk '{print $2}'`

set shortty=`who am i | awk '{print $2}' | sed -E 's/.*(..)$/\1/'`
foreach x (`psj | grep sh | grep $shortty | awk '{print $2}'`)
/usr/sbin/rtprio 2 -$x
end

To load the new file:
source /root/.cshrc

11. install cvsup
cd /usr/ports/net/cvsup-without-gui 
make install clean; rehash; mail -s 'cvs installed' dave.boodman@vtext.com < /dev/null

(stay close for gettext options, 2450: 27mins, supermicro: 17mins, 2950: 22mins)

12. get latest sources for this release:
cd /usr/src 
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_7_2\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

-OR-

echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_7\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

(stable)


cvsup sup ; mail -s 'cvs sup done' dave.boodman@vtext.com < /dev/null

(2450, ~12mins, supermicro, 27mins, 2950: 7mins)


13. configure new kernel. 

cd /usr/src/sys/amd64/conf 
scp backup2:/mnt/data4/build/freebsd/kern_config-7.1-amd64 ./jail8

edit the kernel config and change ident to be the name of the jail:
vi jail8
ident           jail8

edit /sys/conf/newvers.sh to add –jc2 to the end of the BRANCH string (RELEASE-jc2)
vi /sys/conf/newvers.sh

notes: http://www.FreeBSD.org/doc/en_US.ISO8859-1/books/handbook/kernelconfig-config.html

14. bring over patches from backup2

The various patches are in /mnt/data4/build/freebsd/patches on backup2. There are dirs for each version. Not all dirs are populated, but patches for later versions work on older ones unless there is a new patch in the older dir.

cd /usr/src
scp backup2:"/mnt/data4/build/freebsd/patches/7.2/*" .

Apply patches:
patch -l < jls-patch


15. build, install kernel and world

cd /boot

mv kernel kernel.GENERIC
cd kernel.GENERIC
mkdir hold
mv mfi_linux.ko hold/
mv linux.ko hold/
mv linprocfs.ko hold/
mv linsysfs.ko hold/
mv geom_vinum.ko hold/
mv geom_concat.ko hold/
rm *.ko
rm *.symbols
mv hold/* .
rmdir hold/

cd /usr/src
make buildkernel installkernel

make buildworld ; mail -s 'buildworld done' dave.boodman@vtext.com < /dev/null
(2450: 1:56min, supermicro: 59mins, 2950: 38mins)
make installworld 
(2450: 3min, supermicro: 1min, 2950: :34)
mergemaster -i
delete /var/tmp/temproot
delete bsnmpd
delete temporary ./etc/hosts
delete temporary ./etc/motd
delete /var/tmp/temproot

cd /sys/modules/geom/geom_vinum
make 
make install

16. populate devfs ruleset
scp backup2:/mnt/data4/build/freebsd/devfs.rules /etc

17. populate /etc/rc.conf with IPs and NFS settings
vi /etc/rc.conf

kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.4.102"
devfs_system_ruleset="devfsrules_show_all"

#ifconfig_xl0="inet 10.1.4.102 netmask 255.255.255.0"
#ifconfig_fxp0="inet 69.55.228.53 netmask 255.255.255.0"
#ifconfig_fxp0_alias0="inet 69.55.2xx.xx netmask 255.255.255.0"

static_routes="t1 office"
route_t1="-net 10.1.5 10.1.4.2"
route_office="-net 10.1.6 10.1.4.2"
gvinum_enable="YES"
fsck_y_enable="YES"
background_fsck="NO"
rc_mfi_raid_tty_log="YES"

18. make sure sysctls are set and preserved after reboot
echo "kern.consmute=0\
kern.ipc.shm_use_phys=1\
kern.ipc.shmall=131070\
kern.ipc.shmmax=134217728\
net.inet.tcp.syncookies=0\
kern.maxfiles=32768\
kern.fallback_elf_brand=3\
kern.maxprocperuid=4000\
security.jail.sysvipc_allowed=1\
security.jail.allow_raw_sockets=1\
security.jail.socket_unixiproute_only=1\
security.jail.chflags_allowed=0\
dev.amr.0.allow_volume_configure=1\
compat.linux.osrelease=2.6.12\
vm.pmap.shpgperproc=300" >> /etc/sysctl.conf

19. mount procfs
echo "proc                    /proc           procfs  rw              0       0" >> /etc/fstab

For Dell 2950/2450:
echo "linprocfs               /usr/compat/linux/proc linprocfs rw     0       0" >> /etc/fstab

For Dell 2950:
echo "linsysfs                /usr/compat/linux/sys linsysfs rw       0       0" >> /etc/fstab

mkdir -p /usr/compat/linux/proc
mkdir -p /usr/compat/linux/sys

19. enable noatime option
NOT APPLICABLE IF RUNNING GVINUM
data1 and data2 should look something like:
/dev/amrd0s1g           /mnt/data1      ufs     rw,noatime      2       2

20. reboot. Confirm new kernel is loaded

uname -a

Check devfs rules:
devfs rule showsets
devfs rule -s 3 show

21. update ports:
cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_7_2\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup

cvsup sup; mail -s 'cvs sup ports done' dave.boodman@vtext.com < /dev/null

(2450: 18mins, supermicro: 19mins; 2950: 24mins)

22. Install raid mgmt tool

For LSI based cards:

install linux_base:
cd /usr/ports/emulators/linux_base-fc4
make install clean 
(2450: 7min, supermicro: 3mins, 2950: 14mins)

(for LSI)
cd /usr/ports/sysutils/linux-megamgr
make install clean

cd /usr/ports/sysutils/megarc
make install clean

(for Perc5/i, 6/i)
cd /usr/ports/sysutils/linux-megacli2
make install clean

Test:
rehash; megacli ldinfo lall a0

For adaptec:
On jail18: 
scp /compat/linux/usr/sbin/aaccli user@10.1.4.102:~

mv ~user/aaccli /compat/linux/usr/sbin/aaccli

test out;
/compat/linux/usr/sbin/aaccli

For 3w9690:
scp backup2:/d4/build/3ware/tw_cli-freebsd-x86_64-9.5.0.1.tgz /usr/local/sbin
cd /usr/local/sbin xzvf tw_cli-freebsd-x86_64-9.5.0.1.tgz
rm tw_cli-freebsd-x86_64-9.5.0.1.tgz

test out;
./tw_cli /c0 show allstatus

23. install rsync from ports
cd /usr/ports/net/rsync
make install clean

choose default options

25. configure inetd to respond to mrtg load queries
echo "load    stream  tcp     nowait  user    /usr/local/jail/bin/load.pl  load.pl" >> /etc/inetd.conf

echo "load            12384/tcp" >> /etc/services

26. install bb client
adduser
Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]: 
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username   : bb
Password   : <random>
Full Name  : bb
Uid        : 1984
Class      :
Groups     : bb
Home       : /home/bb
Shell      : /bin/sh
Locked     : no
OK? (yes/no): yes

cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd_linuxcompat.tgz .
tar xzf bb-freebsd_linuxcompat.tgz

edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.4.108 jail8.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

vi /home/bb/bbc1.9e-btf/ext/openfiles 
MACHINE="jail8,johncompanies,com"      # HAS TO BE IN A,B,C FORM

cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh 
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf
./runbb.sh start
more BBOUT 
(look for errors)
exit

echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh


NOTE: to get bb working on amd, had to copy over bin dir from linux dist 

27. configure load mrtg, on mail
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg 
(add new entry to file following existing format)

28. configure bb on mail:
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
10.1.4.102 jail2.johncompanies.com # ssh

su bb
cd
bbsrc/bb/runbb.sh restart ; exit

29. remove reserve space, enable softupdates (probably already set, so not necessary)
NOT APPLICABLE IF USING GVINUM
cd
umount /mnt/data1
umount /mnt/data2
tunefs -m 0 /mnt/data1
tunefs -m 0 /mnt/data2
mount -a

30. configure ntp
echo "server 10.1.4.5" > /etc/ntp.conf

/usr/sbin/ntpd -p /var/run/ntpd.pid
sleep 2; ntpq -p
(confirm it’s able to reach our time server)

32. fwd and reverse lookups on ns1c
vr johncompanies.com
vi internal.johncompanies.com
rndc reload johncompanies.com IN private
(edit the PTR too)



35. create the jail template


touch /mnt/data1/jail 
mdconfig -a -t vnode -s 1g -f /mnt/data1/jail -u 0
bsdlabel -rw md0 
newfs4x -i 4096 /dev/md0c
mkdir /mnt/data1/jail-DIR
mount /dev/md0c /mnt/data1/jail-DIR

cd /usr/ports/sysutils/jailutils
make install clean

cd /usr/src
make world DESTDIR=/mnt/data1/jail-DIR; pagedave
(2450: 2:28mins, supermicro: 55mins, 2950: 1h )
cd etc
make distribution DESTDIR=/mnt/data1/jail-DIR
mount -t devfs devfs /mnt/data1/jail-DIR/dev
devfs -m /mnt/data1/jail-DIR/dev rule -s 3 applyset 
cd /mnt/data1/jail-DIR
ln -sf dev/null kernel
cp /usr/local/sbin/jkill /mnt/data1/jail-DIR/sbin

jail /mnt/data1/jail-DIR testhostname 192.168.11.100 /bin/sh
csh
touch /etc/fstab
echo 'network_interfaces=""\
hostname="newsystem"\
kern_securelevel_enable="NO"\
sendmail_enable="YES"\
sshd_enable="YES"' > /etc/rc.conf

echo "nameserver 69.55.225.225\
nameserver 69.55.230.3" >> /etc/resolv.conf

vi /etc/crontab
remove the adjkerntz lines
comment out periodic’s and put this line above them:
# IF YOU UNCOMMENT THESE, PLEASE ADJUST THEIR RUN TIME

rm -rf /etc/periodic/daily/400.status-disks

check /tmp for crap

vi /etc/periodic/security/100.chksetuid
replace: MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`
with: MP='/' (use single quotes)

mkdir -p /usr/compat/linux/dev

adduser (Add account for user, make sure in wheel group)

Username: user
Full name: user
Uid (Leave empty for default):
Login group [user]:
Login group is user. Invite user into other groups? []: wheel
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/user]:
Home directory permissions (Leave empty for default):
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: y
Lock out the account after creation? [no]:
Username   : user
Password   : <random>
Full Name  : user
Uid        : 1001
Class      :
Groups     : user
Home       : /home/user
Home Mode  :
Shell      : /bin/sh
Locked     : no
OK? (yes/no): y
adduser: INFO: Successfully added (user) to the user database.
adduser: INFO: Password for (user) is: 901gmYjO
Add another user? (yes/no): n
Goodbye!

vi /usr/home/user/.profile (and add to the file):
TERM=vt100;     export TERM

tzsetup

newaliases 

rm /sbin/halt /sbin/reboot
ln /sbin/jkill /sbin/halt
ln /sbin/jkill /sbin/reboot

#cd /dev
#rm console
#ln -s null console

vi /etc/syslog.conf (comment out console and move to /var/log/messages):
#*.err;kern.warning;auth.notice;mail.crit               /dev/console *.err;kern.warning;auth.notice;mail.crit                /var/log/messages 

exit
exit

cd libexec
chflags noschg ld-elf32.so.1
chflags noschg ld-elf.so.1

mv ld-elf32.so.1 ld-elf32.so.1-orig
ln ld-elf.so.1 ld-elf32.so.1

chflags schg ld-elf.so.1
chflags schg ld-elf32.so.1


cd /usr/ports
make -DNOCLEANDEPENDS clean 
(2450: 15mins , supermicro: 29mins, 2950: 18mins)
rm -fr /usr/ports/distfiles/*
cp -r /usr/ports /mnt/data1/jail-DIR/usr (2450: 2:00 mins , supermicro: 15mins, 2950: 3mins)

rm /mnt/data1/jail-DIR/root/.history

cd 
umount /mnt/data1/jail-DIR/dev
dump -0a -f /usr/local/jail/template/72template /dev/md0c
umount /dev/md0c
rmdir /mnt/data1/jail-DIR


36. setup backups
echo '#\!/bin/sh\
backupdir=/data/jail2\
server=backup1\
\
## ENTRY /etc\
## ENTRY /usr/local/etc\
## ENTRY /usr/local/jail\
## ENTRY /root/logs' > /usr/local/jail/bin/backup.config

on backup1:
setup backup dirs:
ssh backup1 mkdir -p /data/jail2/0

add the system to 
vi /usr/local/sbin/snapshot_rotate

on mail:
vi /usr/local/www/mgmt/cgi/backupgraph.pl
(add hostname)

Edit /usr/local/jail/bin/backup.xxx to use the right drives and copy to /usr/local/jail/bin/backup 

37. mkdir /root/logs

38. edit sshd_config for security
vi /etc/ssh/sshd_config
ListenAddress 69.55.228.53
ListenAddress 10.1.4.102

kill -1 `cat /var/run/sshd.pid`

39. add crontab entries
crontab -e
5 0 * * * /usr/local/jail/bin/backup.md
1 0 1 * * /usr/local/jail/bin/ipfwreset
0 18 * * * /usr/local/jail/bin/ipfwbackup
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /usr/local/jail/bin/trafstats

On 2950 with Perc5/i also add:
*/5 * * * * /usr/local/jail/bin/perc5iraidchk

40. Reboot notify script
ln -s /usr/local/jail/bin/notify.sh /usr/local/etc/rc.d/notify.sh 

41. add to management db (on mail and devweb) jc.ref_machines and jc.ref_templates

uname -r
6.2-RELEASE-jc1

insert into ref_machines values (null,'mx2','mx2.johncompanies.com',0,'m');
select machine_id from ref_machines where host='mx2';
+------------+
| machine_id |
+------------+
|         35 |
+------------+
insert into ref_templates values ('',' 7.1-RELEASE-jc2',44,'FreeBSD 7.1',0);

42. add to server/cabinet map. On mail:
vi /usr/local/www/mgmt/html/cabinetmap.html

43. add an outside blocking rule to the firewall, so this machine can only be reached from inside the firewall. Follow example already in firewall jail17 is:
 
00117 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.228.200
00117 deny ip from any to 69.55.228.200

jail19 would be 00119...
ipfw add 00107 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.230.108
ipfw add 00107 deny ip from any to 69.55.220.108

44. select customers for probe map

47. patch jail against starting jails with rtprio

mv /usr/sbin/jail /usr/sbin/jail_
echo '#\!/bin/sh\
/usr/sbin/rtprio -t /usr/sbin/jail_ $*' > /usr/sbin/jail
chmod +x /usr/sbin/jail

48. make sure mail works
If there are map errors:
cd /etc/mail; make maps


50. ntp doesn’t seem to start from rc so,

echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh
chmod 0700 /usr/local/etc/rc.d/ntp.sh

51. recover space on /usr

rm -fr /usr/obj

52. setup fuse

cd /usr/ports/sysutils/fusefs-kmod/
make install

vi /etc/rc.conf
fusefs_enable="YES"

sysctl vfs.usermount=1

cd /usr/ports/sysutils/fusefs-sshfs
make install

sshfs 1005@usw-s009.rsync.net: /mnt/data1/69.55.234.68-col00001-DIR/mnt

Stress testing a 6.x jail[edit]

mkdir /mnt/jail
newfs /dev/gvinum/v1
mount /dev/gvinum/v1 /mnt/jail
cd /mnt/jail
restore -r -f /path/to/61stress .
cd ..
umount /mnt/jail

sh
for f in 1 2 3 4 5 6 7 8 9 10; do mkdir /mnt/data1/$f; done

gconcat label v3-v5 /dev/gvinum/v3 /dev/gvinum/v4 /dev/gvinum/v5
bsdlabel -r -w /dev/concat/v3-v5
newfs /dev/concat/v3-v5a
mount /dev/concat/v3-v5a /mnt/data1/1
bsdlabel -r -w /dev/gvinum/v10
newfs /dev/gvinum/v10a
mount /dev/gvinum/v10a /mnt/data1/2
gconcat label v23-v25 /dev/gvinum/v23 /dev/gvinum/v24 /dev/gvinum/v25
bsdlabel -r -w /dev/concat/v23-v25
newfs /dev/concat/v23-v25a
mount /dev/concat/v23-v25a /mnt/data1/3
bsdlabel -r -w /dev/gvinum/v50
newfs /dev/gvinum/v50a
mount /dev/gvinum/v50a /mnt/data1/4
gconcat label v63-v65 /dev/gvinum/v63 /dev/gvinum/v64 /dev/gvinum/v65
bsdlabel -r -w /dev/concat/v63-v65
newfs /dev/concat/v63-v65a
mount /dev/concat/v63-v65a /mnt/data1/5
bsdlabel -r -w /dev/gvinum/v70
newfs /dev/gvinum/v70a
mount /dev/gvinum/v70a /mnt/data1/6
gconcat label v83-v85 /dev/gvinum/v83 /dev/gvinum/v84 /dev/gvinum/v85
bsdlabel -r -w /dev/concat/v83-v85
newfs /dev/concat/v83-v85a
mount /dev/concat/v83-v85a /mnt/data1/7
bsdlabel -r -w /dev/gvinum/v100
newfs /dev/gvinum/v100a
mount /dev/gvinum//v100a /mnt/data1/8
gconcat label v123-v125 /dev/gvinum/v123 /dev/gvinum/v124 /dev/gvinum/v125
bsdlabel -r -w /dev/concat/v123-v125
mount /dev/concat/v123-v125a /mnt/data1/9
newfs /dev/concat/v123-v125a
mount /dev/concat/v123-v125a /mnt/data1/9
bsdlabel -r -w /dev/gvinum/v130
newfs /dev/gvinum/v130a
mount /dev/gvinum//v130a /mnt/data1/10

sh
for f in 1 2 3 4 5 6 7 8 9 10; do cd /mnt/data1/$f; dump -0a -f - /dev/gvinum/v1| restore -r -f - ; done

sh
for f in 1 2 3 4 5 6 7 8 9 10; do mount_devfs devfs /mnt/data1/$f/dev; devfs -m /mnt/data1/$f/dev rule -s 3 applyset; done

sh
for f in 15 16 17 18 19 20 21 22 23 24; do ifconfig bce0 alias 10.1.6.$f netmask 255.255.255.255; done

jail /mnt/data1/1 testhostname1 10.1.6.15 /bin/sh /etc/rc 2> /dev/null
jail /mnt/data1/2 testhostname2 10.1.6.16 /bin/sh /etc/rc 2> /dev/null
jail /mnt/data1/3 testhostname3 10.1.6.17 /bin/sh /etc/rc 2> /dev/null
jail /mnt/data1/4 testhostname4 10.1.6.18 /bin/sh /etc/rc 2> /dev/null
jail /mnt/data1/5 testhostname5 10.1.6.19 /bin/sh /etc/rc 2> /dev/null
jail /mnt/data1/6 testhostname6 10.1.6.20 /bin/sh /etc/rc 2> /dev/null
jail /mnt/data1/7 testhostname7 10.1.6.21 /bin/sh /etc/rc 2> /dev/null
jail /mnt/data1/8 testhostname8 10.1.6.22 /bin/sh /etc/rc 2> /dev/null
jail /mnt/data1/9 testhostname9 10.1.6.23 /bin/sh /etc/rc 2> /dev/null
jail /mnt/data1/10 testhostname10 10.1.6.24 /bin/sh /etc/rc 2> /dev/null

systat -vmstat


Stress testing zfs[edit]

zfs receive pool1/jail2 < /usr/local/jail/template/template
zfs set mountpoint=/mnt/data1/jail2 pool1/jail2
zfs set quota=4G pool1/jail2
zfs destroy pool1/jail2@now

install apps that will run when jail started 

cp -r /usr/src /mnt/data1/jail2/usr
rm -fr /mnt/data1/jail2/usr/obj

mount -t devfs devfs /mnt/data1/jail2/dev; devfs -m /mnt/data1/jail2/dev rule -s 3 applyset;

jail /mnt/data1/jail2 stress-test 69.55.234.86 /bin/sh
csh
cd /usr/ports/benchmarks/bonnie
make install clean
cd /usr/ports/sysutils/stress
make install clean
cd /usr/ports/net/rsync
make install clean
cd /usr/ports/lang/perl5.8
make install clean

cd /usr/local/etc/rc.d/

cat >> boot.sh
sleep 30 && sh /usr/local/etc/rc.d/buildworld &
sleep 30 && sh /usr/local/etc/rc.d/portindex &
sleep 30 && sh /usr/local/etc/rc.d/stress &
sleep 30 && sh /usr/local/etc/rc.d/bonnie &

cat > buildworld
#!/bin/sh
while (true); do cd /usr/src; make buildworld; done

cat > portindex
while (true); do cd /usr/ports/; make index; done

cat > stress
/usr/local/bin/stress -c 1 -i 1 -m 1 -d 1 --vm-bytes 768M --hdd-bytes 128M

cat > bonnie
#!/bin/sh
while (true); do /usr/local/bin/bonnie -s 2g; done


chmod +x *

exit
exit

cd 
zfs snapshot pool1/jail2@now



sh
for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105; do zfs send pool1/jail2@now | zfs receive pool1/jail$f;
zfs set quota=10G pool1/jail$f; 
zfs set mountpoint=/mnt/data1/jail$f pool1/jail$f;
done

for f in 106 107 108 114 115 116 117 118 119 120 121 122 123 125 126 127 128 129 130 131; do zfs send pool1/jail2@now | zfs receive pool2/jail$f
zfs set quota=10G pool2/jail$f;
zfs set mountpoint=/mnt/data2/jail$f pool2/jail$f;
done

for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101; do chmod 0 /mnt/data1/jail$f/usr/local/etc/rc.d/bonnie; done
for f in 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105; do chmod 0 /mnt/data1/jail$f/usr/local/etc/rc.d/stress; done

for f in 115 116 117 118 119 120 121 122 123 125 126 127; do chmod 0 /mnt/data2/jail$f/usr/local/etc/rc.d/bonnie; done
for f in 119 120 121 122 123 125 126 127 128 129 130 131; do chmod 0 /mnt/data2/jail$f/usr/local/etc/rc.d/stress; done

for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105; do mount -t devfs devfs /mnt/data1/jail$f/dev; devfs -m /mnt/data1/jail$f/dev rule -s 3 applyset; done

for f in 106 107 108 114 115 116 117 118 119 120 121 122 123 125 126 127 128 129 130 131; do mount -t devfs devfs /mnt/data2/jail$f/dev; devfs -m /mnt/data2/jail$f/dev rule -s 3 applyset; done

for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105; 
do jail /mnt/data1/jail$f testhostname$f 69.55.234.$f /bin/sh /etc/rc 2> /dev/null; done

for f in 106 107 108 114 115 116 117 118 119 120 121 122 123 125 126 127 128 129 130 131; 
do jail /mnt/data2/jail$f testhostname$f 69.55.234.$f /bin/sh /etc/rc 2> /dev/null; done


systat -vmstat
sysctl vfs.numvnodes


for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105; do umount /mnt/data1/jail$f/dev; zfs destroy -r pool1/jail$f; done

for f in 106 107 108 114 115 116 117 118 119 120 121 122 123 125 126 127 128 129 130 131; do umount /mnt/data2/jail$f/dev; zfs destroy -r pool2/jail$f; done


for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105; do touch /mnt/data1/jail$f; 
mdconfig -a -t vnode -s 10g -f /mnt/data1/jail$f -u $f;
bsdlabel -r -w md$f auto;
newfs -O 1 /dev/md${f}a;
mkdir /mnt/data1/jail$f-DIR;
mount /dev/md${f}a /mnt/data1/jail$f-DIR;
rsync -aSH /mnt/data2/jail2/ /mnt/data1/jail$f-DIR/;
done

for f in 106 107 108 114 115 116 117 118 119 120 121 122 123 125 126 127 128 129 130 131; do touch /mnt/data2/jail$f; 
mdconfig -a -t vnode -s 10g -f /mnt/data2/jail$f -u $f;
bsdlabel -r -w md$f auto;
newfs -O 1 /dev/md${f}a;
mkdir /mnt/data2/jail$f-DIR;
mount /dev/md${f}a /mnt/data2/jail$f-DIR;
rsync -aSH /mnt/data1/jail86-DIR/ /mnt/data2/jail$f-DIR/;
done

for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105; 
do chmod 0700 /mnt/data1/jail$f-DIR/usr/local/etc/rc.d/bonnie; done
for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105; 
do chmod 0700 /mnt/data1/jail$f-DIR/usr/local/etc/rc.d/stress; done

for f in 106 107 108 114 115 116 117 118 119 120 121 122 123 125 126 127 128 129 130 131; 
do chmod 0700 /mnt/data2/jail$f-DIR/usr/local/etc/rc.d/bonnie; done
for f in 106 107 108 114 115 116 117 118 119 120 121 122 123 125 126 127 128 129 130 131; 
do chmod 0700 /mnt/data2/jail$f-DIR/usr/local/etc/rc.d/stress; done

for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101; do chmod 0 /mnt/data1/jail$f-DIR/usr/local/etc/rc.d/bonnie; done
for f in 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105; do chmod 0 /mnt/data1/jail$f-DIR/usr/local/etc/rc.d/stress; done

for f in 115 116 117 118 119 120 121 122 123 125 126 127; do chmod 0 /mnt/data2/jail$f-DIR/usr/local/etc/rc.d/bonnie; done
for f in 119 120 121 122 123 125 126 127 128 129 130 131; do chmod 0 /mnt/data2/jail$f-DIR/usr/local/etc/rc.d/stress; done

for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105; do mount -t devfs devfs /mnt/data1/jail$f-DIR/dev; devfs -m /mnt/data1/jail$f-DIR/dev rule -s 3 applyset; done

for f in 106 107 108 114 115 116 117 118 119 120 121 122 123 125 126 127 128 129 130 131; do mount -t devfs devfs /mnt/data2/jail$f-DIR/dev; devfs -m /mnt/data2/jail$f-DIR/dev rule -s 3 applyset; done

for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105; 
do jail /mnt/data1/jail$f-DIR testhostname$f 69.55.234.$f /bin/sh /etc/rc 2> /dev/null; done

for f in 106 107 108 114 115 116 117 118 119 120 121 122 123 125 126 127 128 129 130 131; 
do jail /mnt/data2/jail$f-DIR testhostname$f 69.55.234.$f /bin/sh /etc/rc 2> /dev/null; done


for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105; 
do cp /mnt/data1/boot.sh /mnt/data1/jail$f-DIR/usr/local/etc/rc.d/;
cp /mnt/data1/stress /mnt/data1/jail$f-DIR/usr/local/etc/rc.d/;
cp /mnt/data1/bonnie /mnt/data1/jail$f-DIR/usr/local/etc/rc.d/;
done

for f in 106 107 108 114 115 116 117 118 119 120 121 122 123 125 126 127 128 129 130 131; 
do cp /mnt/data1/boot.sh /mnt/data2/jail$f-DIR/usr/local/etc/rc.d/; 
cp /mnt/data1/stress /mnt/data2/jail$f-DIR/usr/local/etc/rc.d/;
cp /mnt/data1/bonnie /mnt/data2/jail$f-DIR/usr/local/etc/rc.d/;
done

for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105; do umount /mnt/data1/jail$f-DIR/dev; umount /mnt/data1/jail$f-DIR; mdconfig -d -u $f; done

for f in 106 107 108 114 115 116 117 118 119 120 121 122 123 125 126 127 128 129 130 131; do umount /mnt/data2/jail$f-DIR/dev; umount /mnt/data2/jail$f-DIR; mdconfig -d -u $f;  done

for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105; do 
mkdir /mnt/data1/jail$f-DIR/;
cd /mnt/data1/jail$f-DIR/;
dump -0a -f - /dev/md0a | restore -r -f -;
done

for f in 106 107 108 114 115 116 117 118 119 120 121 122 123 125 126 127 128 129 130 131; do mkdir /mnt/data2/jail$f-DIR/;
cd /mnt/data2/jail$f-DIR/;
dump -0a -f - /dev/md0a | restore -r -f -;
done

for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105; do chmod 0700 /mnt/data1/jail$f-DIR/usr/local/etc/rc.d/*; done

for f in 106 107 108 114 115 116 117 118 119 120 121 122 123 125 126 127 128 129 130 131; do chmod 0700 /mnt/data2/jail$f-DIR/usr/local/etc/rc.d/*;  done



--vm-bytes 300M

Reloading FreeBSD (while preserving customer data)[edit]

2. assuming 2 mirrors, boot to disk 1 of 4.11 
skip kernel config (enter)
custom install
(skip partition)
label
move cursor to aacd0, hit space (takes you to partition map screen)
given this as the pre-existing partition map:
Filesystem    1K-blocks   Used    Avail Capacity  Mounted on
/dev/aacd0s1a    128990  36888    81784    31%    /
/dev/aacd0s1h  30499106      4 28059174     0%    /mnt/data1
/dev/aacd1s1e  69526550      4 63964422     0%    /mnt/data2
/dev/aacd0s1f    257998     10   237350     0%    /tmp
/dev/aacd0s1g   3096462 852082  1996664    30%    /usr
/dev/aacd0s1e    257998   1198   236162     1%    /var
procfs                4      4        0   100%    /proc

relabel and all mount points according to prev settings

or, delete all partitions except for data parts, re-create as necessary

re-toggle newfs on all mount points except for /mnt/data1 - make sure under the Newfs column the data1 (and data2) partiotions look like: UFS+S Y

make sure under the Newfs column the data1 (and data2) partiotions look like
UFS+S N

space to unselect aacd0
cursor over aacd1
space

Space to select both drives and tab to ok

Continue installation as per normal
distributions
developer
yes to install ports
exit
media
cd
commit
yes
(2450: 16mins, supermicro: 11mins)
yes to "visit general config"
Set root pwd
Add user ‘user’ member group is wheel, set password
Set tz
Networking->interfaces->Fxp0
No IPV6
dhcp=yes
Set hostname & domain

exit...
exit install
yes
take the cd out and let the machine reboot
Retrieved from "https://wiki.jcihosting.com/index.php?title=Jail_Server_Install&oldid=1333"