Clear space on boxes

2012-11-19T03:03:28Z

99.139.51.90:

'''bwdb: we move old traffic flow files to backup2'''
 
<ol start="1">
<li>go to bwdb (10.1.4.203 - a7p1o4) /usr/home/archive#</li>
<li>scp the oldest month to any available disk with space on backup2:</li>
<ol style="list-style-type: upper-alpha">
<li>since there are many files, you can't do a whole month at a time like this:</li>
*scp ft-v05.2012-06-* 10.1.4.3:/d2/bwdb/archive/ && rm -f ft-v05.2012-06-*
<li>you have to start with the single digit days first:</li>
*scp ft-v05.2012-07-0* 10.1.4.3:/d2/bwdb/archive/ && rm -f ft-v05.2012-07-0*
<li>then you can proceed with the full month:</li>
*scp ft-v05.2012-06-* 10.1.4.3:/d2/bwdb/archive/ && rm -f ft-v05.2012-06-*
</ol>
<li>mostly the same on bwdb2 (a7p1o5):</li>
<ol style="list-style-type: upper-alpha">
<li>bwdb2 /home/archive# scp ft-v05.2011-05* backup2:/mnt/data2/bwdb/archive/bwdb2/</li>
<li>bwdb2 /home/archive# rm ft-v05.2011-05*</li>
</ol>
</ol>

'''jail8'''
 
<ol start="1">
<li>rm /var/mail/root</li>
</ol>
 
'''jail9'''
<ol start="1">
<li>jail9 /mnt/data1# '''rm -fr old-col02106-mdfile-cxld-20120618'''</li>
<li>jail9 /mnt/data1#''' df .'''</li>
</ol>
Filesystem 1K-blocks Used Avail Capacity Mounted on
/dev/mfid0s1g 126049816 118309094 7740722 94% /mnt/data1
<ol start="3">
<li>jail9 /mnt/data1#''' gzip old-col02124-mdfile-cxld-20120817'''</li>
</ol>

'''backup2'''
 
nothing yet...
 
to find candidate to move off host machines

'''jails''' 
for f in `ls |grep -v DIR`; do du -sh $f; df -h $f-DIR 
 
look for bang for the buck 
 

'''virts''' 
 
vdf = df -h in container 
vdff = df everyone in cur dir (e.g. /vz or /vz1 or /vz2)

Network Time (ntp)

2012-11-19T02:22:47Z

99.139.51.90:

# see if its connected:

::[root@quar1 /vz/private]# '''ntpq -pn'''
::ntpq: read: Connection refused

:<big>not running</big>

<ol start="2">
<li>start it:</li>

[root@quar1 /vz/private]# '''service ntpd start''' 
Starting ntpd: [ OK ]
</ol>

<ol start="3">
<li>Query the NTP Server to see if it is paired</li>
[root@quar1 /vz/private]#''' ntpq -pn'''
remote refid st t when poll reach delay offset jitter 10.1.4.5 66.187.233.4 2 u 4 64 1 0.317 876558. 0.000
 
</ol>
<ol start="4">
<li>[root@quar1 /vz/private]#''' date'''</li>
Thu Oct 11 10:58:05 PDT 2012
</ol>
:<big>time is 11:13</big>
that output from ntpq is deceiving- we're syncing connected. you want to see a * or + next to it (ex on mail):

<ol start="5">
<li>mail /usr/local/www/scripts#''' ntpq -pn'''</li>
</ol>
remote refid st t when poll reach delay offset jitter
==============================================================================
*66.187.233.4 .CDMA. 1 u 550 1024 377 75.159 -0.613 0.100
217.204.76.170 0.0.0.0 16 u - 1024 0 0.000 0.000 4000.00
64.112.189.11 0.0.0.0 16 u - 1024 0 0.000 0.000 4000.00
66.69.112.130 0.0.0.0 16 u - 1024 0 0.000 0.000 4000.00
+80.85.129.25 193.79.237.14 2 u 624 1024 377 147.067 -0.291 0.149

(btw, if at least one is not + or * our ENTIRE network is off since all boxes sync to mail.
edit /etc/ntp.conf to add servers)

see: http://doc.ntp.org/4.1.1/ntpq.htm

so its not connecting to our ntpd on mail (10.1.4.5)

and this is likely cause of the large time gap (offset 876558.) so we fix it:

<ol start="6">
<li>[root@quar1 /vz/private]# '''date -s "11:17:00"''' 
Thu Oct 11 11:17:00 PDT 2012</li>
</ol>

<ol start="7">
<li>[root@quar1 /vz/private]#''' ntpq -pn'''</li>
</ol>
remote refid st t when poll reach delay offset jitter
==============================================================================
10.1.4.5 66.187.233.4 2 u 1029 64 0 0.000 0.000 4000.00

no offset but not connected. give it some time...

<ol start="8">
<li>[root@quar1 /vz/private]#''' sleep 300; ntpq -pn'''</li>
</ol>
remote refid st t when poll reach delay offset jitter
==============================================================================
*10.1.4.5 66.187.233.4 2 u 44 64 37 0.304 19234.8 0.649

<big>Yay, done.</big>

Jail Server Install

2012-11-11T23:07:05Z

99.139.51.90: /* 8.x */

= 8.x =

All time estimates below assume disks aren’t scrubbing. Setup instructions below are for LSI card:

1. make sure bios is setup for bios console redirect
2950:
Console redirection:
LCD string..
Date to GMT

2. assuming mirrors (or at least disks) created (if not, refer to this), boot to disk 1 of 7.2
skip kernel config (enter)
custom install

partition ->
move cursor to mfid0, hit space (takes you to partition map screen)
a for entire disk
q to quit and save

standard mbr (no boot manager)
space to unselect mfid0
cursor over mfid1
space
a for entire disk
q to quit and save
none (leave untouched)

cursor over mfid0
space
(takes you into part. Screen again) q to exit
none
Make sure both are checked and tab to ok

Label ->
Make sure mfid0 is highlighted

/ 512M
swap 2G (for 2950 make it 6G)
/var 256M
/tmp 256M
/usr 5G
/mnt/data1 remaining space

Make sure to toggle S for soft updates on all (should look like UFS2+S Y under the Newfs column)

move cursor to mfid1
swap 8G (or 4G if there’s a 3rd drive)
/mnt/data2 remaining space (no need to newfs)

q to save and exit

distributions ->
developer
custom -> lib32
yes to install ports
exit

media ->
cd (or ftp in case of no cd)

commit ->
yes
(2450: 14mins, supermicro: 12mins)

yes to "visit general config" ->
Set root pwd

Add user ‘user’ member group is wheel, set password

Set tz

Networking->interfaces->bce0 ->
No IPV6
dhcp=no
Set hostname & domain
Enable sshd

exit...
exit install ->
yes

take the cd out and let the machine reboot

3. double check the date/time

4. edit /etc/make.conf
echo "WITHOUT_X11=yes \
KERNCONF=jail3 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf

5. add settings to /boot/loader.conf and /boot.config

echo "-Dh" >> /boot.config

echo 'console="comconsole,vidconsole" \
boot_multicons="YES" \
boot_serial="YES" \
mfi_linux_load="YES" \
comconsole_speed="115200"' >> /boot/loader.conf

6. turn off all ttyv's except 0 and 1 in /etc/ttys
also turn on ttyd0, change type to vt100:
vi /etc/ttys

ttyv2 "/usr/libexec/getty Pc" cons25 off secure
ttyv3 "/usr/libexec/getty Pc" cons25 off secure
ttyv4 "/usr/libexec/getty Pc" cons25 off secure
ttyv5 "/usr/libexec/getty Pc" cons25 off secure
ttyv6 "/usr/libexec/getty Pc" cons25 off secure
ttyv7 "/usr/libexec/getty Pc" cons25 off secure
# Serial terminals
# The 'dialup' keyword identifies dialin lines to login, fingerd etc.
ttyu0 "/usr/libexec/getty std.9600" vt100 on secure

kill -1 1

on console server:
vi /etc/remote
(rename port to jail8 depending on where and which digi plugged into)
test serial console

7. populate hosts
echo "10.1.4.3 backup2" >> /etc/hosts
echo "10.1.4.8 backup1" >> /etc/hosts
echo "10.1.2.3 backup3" >> /etc/hosts

8. put key in authorized_keys on backup2
cd
ssh-keygen -t dsa -b 1024
(default location, leave password blank)

cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup3 'cat - >> /root/.ssh/authorized_keys'

confirm that you can ssh to backup2 and backup1 without getting a login prompt

ssh backup2 hostname

ssh backup1 hostname

ssh backup3 hostname

9. create & populate binaries/scripts dirs
mkdir -p /usr/local/jail/bin
mkdir -p /usr/local/jail/rc.d
mkdir -p /usr/local/jail/template/
mkdir /mnt/data1
mkdir /mnt/data2
scp backup2:"/mnt/data4/bin/freebsd8.x/*" /usr/local/jail/bin
cd /usr/local/jail/rc.d/
touch quad1
touch deprecated
chmod +x *
cd /usr/local/jail/bin
ln -s /usr/local/jail/rc.d/quad1 quad1
ln -s /usr/local/jail/bin/jailmake_md jailmake
ln -s /usr/local/jail/bin/js_md js
ln -s /usr/local/jail/bin/canceljail_md canceljail
ln -s /usr/local/jail/bin/jailmakeempty_md jailmakeempty
ln -s /usr/local/jail/bin/postboot_md postboot
ln -s /usr/local/jail/bin/preboot_md preboot
ln -s /usr/local/jail/bin/startjail_md startjail
ln -s /usr/local/jail/bin/stopjail_md stopjail

rehash

10. edit root's path and login script:
vi /root/.cshrc

Change alias entries (add G):
alias la ls -aG
alias lf ls -FAG
alias ll ls -lAG
alias ls ls -AG
alias mbm mb mount
alias mbu mb umount
alias cjb cd /usr/local/jail/bin
alias cd1 cd /mnt/data1
alias cd2 cd /mnt/data2
alias cd3 cd /mnt/data3
alias jtop jtop lj
alias j jobs

add to path:
/usr/local/jail/bin
(if adaptec card installed, also add /compat/linux/usr/sbin)

and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "

at the bottom of the file add:
set sshtty=`who am i|awk '{print $2}'`
/usr/sbin/rtprio 3 -`psj | grep $sshtty | awk '{print $2}'`

set shortty=`who am i | awk '{print $2}' | sed -E 's/.*(..)$/\1/'`
foreach x (`psj | grep sh | grep $shortty | awk '{print $2}'`)
/usr/sbin/rtprio 2 -$x
end

To load the new file:
source /root/.cshrc

11. install cvsup
cd /usr/ports/net/cvsup-without-gui
make install clean; rehash; mail -s 'cvs installed' dave.boodman@vtext.com < /dev/null

(stay close for gettext options, 2450: 27mins, supermicro: 17mins, 2950: 22mins)

12. get latest sources for this release:
cd /usr/src
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_8_3\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

-OR-

echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_8\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

(stable)

cvsup sup ; mail -s 'cvs sup done' dave.boodman@vtext.com < /dev/null

(2450, ~12mins, supermicro, 27mins, 2950: 7mins)

13. configure new kernel.

cd /usr/src/sys/amd64/conf
scp backup2:/mnt/data4/build/freebsd/kern_config-8.2-amd64 ./jail3

edit the kernel config and change ident to be the name of the jail:
vi jail3
ident jail3

edit /sys/conf/newvers.sh to add –jc2 to the end of the BRANCH string (RELEASE-jc2)
vi /sys/conf/newvers.sh

notes: http://www.FreeBSD.org/doc/en_US.ISO8859-1/books/handbook/kernelconfig-config.html

14. bring over patches from backup2

The various patches are in /mnt/data4/build/freebsd/patches on backup2. There are dirs for each version. Not all dirs are populated, but patches for later versions work on older ones unless there is a new patch in the older dir.

cd /usr/src
scp backup2:"/mnt/data4/build/freebsd/patches/8.0/*" .

Apply patches:
patch -l < jls-patch

15. build, install kernel and world

cd /boot

mv kernel kernel.GENERIC
cd kernel.GENERIC
mkdir hold
mv mfi_linux.ko hold/
mv linux.ko hold/
mv linprocfs.ko hold/
mv linsysfs.ko hold/
mv geom_vinum.ko hold/
mv geom_concat.ko hold/
mv zfs.* hold/
mv opensolaris* hold/

rm *.ko
rm *.symbols
mv hold/* .
rmdir hold/

cd /usr/src
make buildkernel installkernel

make buildworld ; mail -s 'buildworld done' dave.boodman@vtext.com < /dev/null
(2450: 1:56min, supermicro: 59mins, 2950: 38mins)
make installworld
(2450: 3min, supermicro: 1min, 2950: :34)
mergemaster -i

cd /sys/modules/zfs
make
make install

cd /sys/modules/opensolaris
make
make install

16. populate devfs ruleset
scp backup2:/mnt/data4/build/freebsd/devfs.rules.8x /etc/devfs.rules

17. populate /etc/rc.conf with IPs and NFS settings
vi /etc/rc.conf

kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"

nfs_client_enable="YES"
nfs_reserved_port_only="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.2.103"
devfs_system_ruleset="devfsrules_show_all"

ifconfig_em1="inet 10.1.2.103 netmask 255.255.255.0"
ifconfig_em0="inet 69.55.229.7 netmask 255.255.255.0"
#ifconfig_fxp0_alias0="inet 69.55.2xx.xx netmask 255.255.255.0"

fsck_y_enable="YES"
background_fsck="NO"
#rc_mfi_raid_tty_log="YES"
#zfs_enable="YES"

18. make sure sysctls are set and preserved after reboot
echo "kern.consmute=0\
kern.ipc.shm_use_phys=1\
kern.ipc.shmall=131070\
kern.ipc.shmmax=134217728\
net.inet.tcp.syncookies=0\
kern.maxfiles=32768\
kern.fallback_elf_brand=3\
kern.maxprocperuid=4000\
security.jail.sysvipc_allowed=1\
security.jail.allow_raw_sockets=1\
security.jail.socket_unixiproute_only=1\
security.jail.chflags_allowed=0\
dev.amr.0.allow_volume_configure=1\
compat.linux.osrelease=2.6.12\
vm.pmap.shpgperproc=500\
security.bsd.unprivileged_read_msgbuf=0\
kern.maxvnodes=400000" >> /etc/sysctl.conf

NOTE: watch vfs.numvnodes to see where to set maxvnodes

19. mount procfs
echo "proc /proc procfs rw 0 0" >> /etc/fstab

For Dell 2950/2450:
echo "linprocfs /usr/compat/linux/proc linprocfs rw 0 0" >> /etc/fstab

For Dell 2950:
echo "linsysfs /usr/compat/linux/sys linsysfs rw 0 0" >> /etc/fstab

mkdir -p /usr/compat/linux/proc
mkdir -p /usr/compat/linux/sys

19. enable noatime option
NOT APPLICABLE IF RUNNING GVINUM or zfs
data1 and data2 should look something like:
/dev/amrd0s1g /mnt/data1 ufs rw,noatime 2 2

20. reboot. Confirm new kernel is loaded

uname -a

Check devfs rules:
devfs rule showsets
devfs rule -s 3 show

21. update ports:
cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_8_3\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup

cvsup sup; mail -s 'cvs sup ports done' dave.boodman@vtext.com < /dev/null

(2450: 18mins, supermicro: 19mins; 2950: 24mins)

22. Install raid mgmt tool

(for 2850 PERC 4e/Di- no linux)
cd /usr/ports/distfiles/
fetch http://backup01.best-hosting.ru/pub/FreeBSD/ports/distfiles/dr_freebsd_1.51.zip
cd /usr/ports/sysutils/megarc
make install clean
megarc -dispCfg -a0

need to install perl since linux base won't grab it:
cd /usr/ports/lang/perl5.8
make install clean

---------------

(for Perc5/i, 6/i)
install linux_base:
cd /usr/ports/emulators/linux_base-fc4
make install clean
(2450: 7min, supermicro: 3mins, 2950: 14mins)
Note: didnt succeed due to libtool requirement

cd /usr/ports/distfiles
fetch http://www.lsi.com/DistributionSystem/AssetDocument/support/downloads/megaraid/miscellaneous/linux/2.00.15_Linux_MegaCLI.zip
cd /usr/ports/sysutils/linux-megacli
make install clean
also failed due to libtool, so did
scp /usr/local/sbin/mega* root@10.1.4.110:/usr/local/sbin/
scp /usr/local/libexec/MegaCli root@10.1.4.110:/usr/local/libexec/MegaCli

Test:
rehash; megacli ldinfo lall a0

23. install rsync from ports
cd /usr/ports/net/rsync
make install clean

choose default options

24. configure inetd to respond to mrtg load queries
echo "load stream tcp nowait user /usr/local/jail/bin/load.pl load.pl" >> /etc/inetd.conf

echo "load 12384/tcp" >> /etc/services

25. install bb client
(need linux compat for this, won't install on 8.2 - libtool 2.4 need. So, instead copied over linux: rsync -aSHv --exclude=proc --exclude=sys 10.1.4.108:/usr/compat/linux/ /usr/compat/linux/)

adduser
Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username : bb
Password : <random>
Full Name : bb
Uid : 1984
Class :
Groups : bb
Home : /home/bb
Shell : /bin/sh
Locked : no
OK? (yes/no): yes

cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd_linuxcompat.tgz .
tar xzf bb-freebsd_linuxcompat.tgz

edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
69.55.229.7 jail3.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
or
echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.4.103 jail3.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

vi /home/bb/bbc1.9e-btf/ext/openfiles
MACHINE="jail3,johncompanies,com" # HAS TO BE IN A,B,C FORM

cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf
./runbb.sh start
more BBOUT
(look for errors)
exit

echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh

NOTE: to get bb working on amd, had to copy over bin dir from linux dist

26. configure load mrtg, on mail
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg
(add new entry to file following existing format)

27. configure bb on mail:
vi /usr/home/bb/bbsrc/bb1.9i-btf/etc/bb-hosts
10.1.4.109 jail9.johncompanies.com # ssh
or
69.55.229.7 jail3.johncompanies.com # ssh

su bb
cd
bbsrc/bb/runbb.sh restart ; exit

28. remove reserve space, enable softupdates (probably already set, so not necessary)
cd
umount /mnt/data1
umount /mnt/data2
tunefs -m 0 /mnt/data1
tunefs -m 0 /mnt/data2
mount -a

29. DEPRECATED - ntpd listens on jail IPs- security risk
echo "server 10.1.4.5" > /etc/ntp.conf

/usr/sbin/ntpd -p /var/run/ntpd.pid
sleep 2; ntpq -p
(confirm it’s able to reach our time server)

But there's a bug so install new ntp from ports
/usr/ports/net/ntp

30. fwd and reverse lookups on ns1c
vr johncompanies.com
vi internal.johncompanies.com
rndc reload johncompanies.com IN private
(edit the PTR too)

31. if needed, make a g partition

bsdlabel -e /dev/mfid0s1

given:
# /dev/aacd0s1:
8 partitions:
# size offset fstype [fsize bsize bps/cpg]
a: 262144 0 4.2BSD 2048 16384 16392
b: 4194304 262144 swap
c: 143363997 0 unused 0 0 # "raw" part, don't edit
d: 524288 4456448 4.2BSD 2048 16384 32776
e: 524288 4980736 4.2BSD 2048 16384 32776
f: 6291456 5505024 4.2BSD 2048 16384 28552

new offset = 6291456 + 5505024 = 11796480
new size is size for 'c' partition minus the new start from above
143363997 - 11796480 = 131567517
So:
g: 131567517 11796480 unused 0 0

Create a g partition on 2nd mirror – bsdlabel no longer works (below shows d partition made with sysinstall):

jail8 /usr/home/bb# gpart show
=> 63 285474735 mfid0 MBR (136G)
63 285458922 1 freebsd [active] (136G)
285458985 15813 - free - (7.7M)

=> 0 285458922 mfid0s1 BSD (136G)
0 524288 1 freebsd-ufs (256M)
524288 12582912 2 freebsd-swap (6.0G)
13107200 524288 4 freebsd-ufs (256M)
13631488 524288 5 freebsd-ufs (256M)
14155776 8388608 6 freebsd-ufs (4.0G)
22544384 262914538 7 freebsd-ufs (125G)

=> 63 584843175 mfid1 MBR (279G)
63 584830197 1 freebsd [active] (279G)
584830260 12978 - free - (6.3M)

=> 0 584830197 mfid1s1 BSD (279G)
0 16777216 2 freebsd-swap (8.0G)
16777216 568052981 4 freebsd-ufs (271G)

jail8 /usr/home/bb# gpart show mfid1s1
=> 0 584830197 mfid1s1 BSD (279G)
0 16777216 2 freebsd-swap (8.0G)
16777216 568052981 4 freebsd-ufs (271G)

# gpart list mfid1s1

Geom name: mfid1s1
fwheads: 255
fwsectors: 63
last: 584830196
first: 0
entries: 8
scheme: BSD
Providers:
1. Name: mfid1s1b
Mediasize: 8589934592 (8.0G)
Sectorsize: 512
Mode: r1w1e0
rawtype: 1
length: 8589934592
offset: 0
type: freebsd-swap
index: 2
end: 16777215
start: 0
2. Name: mfid1s1d
Mediasize: 290843126272 (271G)
Sectorsize: 512
Mode: r0w0e0
rawtype: 7
length: 290843126272
offset: 8589934592
type: freebsd-ufs
index: 4
end: 584830196
start: 16777216
Consumers:
1. Name: mfid1s1
Mediasize: 299433060864 (279G)
Sectorsize: 512
Mode: r1w1e1

# gpart delete -i 4 mfid1s1
mfid1s1d deleted
jail8 /usr/home/bb# gpart list mfid1s1
Geom name: mfid1s1
fwheads: 255
fwsectors: 63
last: 584830196
first: 0
entries: 8
scheme: BSD
Providers:
1. Name: mfid1s1b
Mediasize: 8589934592 (8.0G)
Sectorsize: 512
Mode: r1w1e0
rawtype: 1
length: 8589934592
offset: 0
type: freebsd-swap
index: 2
end: 16777215
start: 0
Consumers:
1. Name: mfid1s1
Mediasize: 299433060864 (279G)
Sectorsize: 512
Mode: r1w1e1

# gpart show mfid1s1
=> 0 584830197 mfid1s1 BSD (279G)
0 16777216 2 freebsd-swap (8.0G)
16777216 568052981 - free - (271G)

# gpart add -t freebsd-ufs -i 7 mfid1s1
mfid1s1g added

# gpart show mfid1s1
=> 0 584830197 mfid1s1 BSD (279G)
0 16777216 2 freebsd-swap (8.0G)
16777216 568052981 7 freebsd-ufs (271G)

32. create the jail template

touch /mnt/data1/jail-template20g
mdconfig -a -t vnode -s 20g -f /mnt/data1/jail-template20g -u 0
newfs -O 1 /dev/md0
mkdir /mnt/data1/jail-DIR
mount /dev/md0 /mnt/data1/jail-DIR

cd /usr/ports/sysutils/jailutils
make install clean

cd /usr/src
make world DESTDIR=/mnt/data1/jail-DIR; pagedave
(2450: 2:28mins, supermicro: 55mins, 2950: 1h )
cd etc
make distribution DESTDIR=/mnt/data1/jail-DIR
mount -t devfs devfs /mnt/data1/jail-DIR/dev
devfs -m /mnt/data1/jail-DIR/dev rule -s 3 applyset
cd /mnt/data1/jail-DIR
ln -sf dev/null kernel
cp /usr/local/sbin/jkill /mnt/data1/jail-DIR/sbin

jail /mnt/data1/jail-DIR testhostname 192.168.11.100 /bin/sh
csh
touch /etc/fstab
echo 'network_interfaces=""\
hostname="newsystem"\
kern_securelevel_enable="NO"\
sendmail_enable="YES"\
sshd_enable="YES"' > /etc/rc.conf

echo "nameserver 69.55.225.225\
nameserver 69.55.230.3" >> /etc/resolv.conf

vi /etc/crontab
remove the adjkerntz lines
comment out periodic’s and put this line above them:
# IF YOU UNCOMMENT THESE, PLEASE ADJUST THEIR RUN TIME

rm -rf /etc/periodic/daily/400.status-disks

check /tmp for crap

vi /etc/periodic/security/100.chksetuid
replace: MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`
with: MP='/' (use single quotes)

mkdir -p /usr/compat/linux/dev

adduser (Add account for user, make sure in wheel group)

Username: user
Full name: user
Uid (Leave empty for default):
Login group [user]:
Login group is user. Invite user into other groups? []: wheel
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/user]:
Home directory permissions (Leave empty for default):
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: y
Lock out the account after creation? [no]:
Username : user
Password : <random>
Full Name : user
Uid : 1001
Class :
Groups : user
Home : /home/user
Home Mode :
Shell : /bin/sh
Locked : no
OK? (yes/no): y
adduser: INFO: Successfully added (user) to the user database.
adduser: INFO: Password for (user) is: 901gmYjO
Add another user? (yes/no): n
Goodbye!

vi /usr/home/user/.profile (and add to the file):
TERM=vt100; export TERM

tzsetup

newaliases

rm /sbin/halt /sbin/reboot
ln /sbin/jkill /sbin/halt
ln /sbin/jkill /sbin/reboot

#cd /dev
#rm console
#ln -s null console

vi /etc/syslog.conf (comment out console and move to /var/log/messages):
#*.err;kern.warning;auth.notice;mail.crit /dev/console *.err;kern.warning;auth.notice;mail.crit /var/log/messages

exit
exit

cd libexec
chflags noschg ld-elf32.so.1
chflags noschg ld-elf.so.1

mv ld-elf32.so.1 ld-elf32.so.1-orig
ln ld-elf.so.1 ld-elf32.so.1

chflags schg ld-elf.so.1
chflags schg ld-elf32.so.1

mv /mnt/data1/jail-DIR/usr/sbin/traceroute /mnt/data1/jail-DIR/usr/sbin/_traceroute

echo '#\!/bin/sh\
/usr/sbin/_traceroute -i bce0 $1' >> /mnt/data1/jail-DIR/usr/sbin/traceroute
chmod +x /mnt/data1/jail-DIR/usr/sbin/traceroute

cd /usr/ports
make -DNOCLEANDEPENDS clean
(2450: 15mins , supermicro: 29mins, 2950: 18mins)
rm -fr /usr/ports/distfiles/*
cp -r /usr/ports /mnt/data1/jail-DIR/usr (2450: 2:00 mins , supermicro: 15mins, 2950: 3mins)

rm /mnt/data1/jail-DIR/root/.history

cd
umount /mnt/data1/jail-DIR/dev
dump -0a -f /usr/local/jail/template/template /dev/md0
umount /dev/md0
rmdir /mnt/data1/jail-DIR
mdconfig -d -u 0

33. setup backups
echo '#\!/bin/sh\
backupdir=/data/jail3\
server=backup1\
\
## ENTRY /etc\
## ENTRY /usr/local/etc\
## ENTRY /usr/local/jail\
## ENTRY /root/logs' > /usr/local/jail/bin/backup.config

on backup1:
setup backup dirs:
ssh backup1 mkdir -p /data/jail3/0

on backup1, add the system to
vi /usr/local/sbin/snapshot_rotate

on mail:
vi /usr/local/www/mgmt/cgi/backupgraph.pl
(add hostname)

Edit /usr/local/jail/bin/backup.xxx to use the right drives and copy to /usr/local/jail/bin/backup

34. mkdir /root/logs

35. edit sshd_config for security
vi /etc/ssh/sshd_config
ListenAddress 69.55.229.7
ListenAddress 10.1.2.103

kill -1 `cat /var/run/sshd.pid`

36. add crontab entries
crontab -e
5 0 * * * /usr/local/jail/bin/backup.md
1 0 1 * * /usr/local/jail/bin/ipfwreset
0 18 * * * /usr/local/jail/bin/ipfwbackup
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /usr/local/jail/bin/trafstats
0 0,6,12,18 * * * /usr/local/jail/bin/sync_jail_names
*/5 * * * * /usr/local/jail/bin/perc5iraidchk
*/5 * * * * /usr/local/jail/bin/perc4eraidchk

37. Reboot notify script
ln -s /usr/local/jail/bin/notify.sh /usr/local/etc/rc.d/notify.sh

38. add to management db (on mail and devweb) jc.ref_machines and jc.ref_templates

uname -r
8.0-RELEASE-p2

insert into ref_machines values (null,'mx2','mx2.johncompanies.com',0,'m');
select machine_id from ref_machines where host='mx2';
+------------+
| machine_id |
+------------+
| 35 |
+------------+
insert into ref_templates values ('',' 8.3-RELEASE-jc2',10,'FreeBSD 8.3',0);

39. add to server/cabinet map. On mail:
vi /usr/local/www/mgmt/html/cabinetmap.html

40. add an outside blocking rule to the firewall, so this machine can only be reached from inside the firewall. Follow example already in firewall jail17 is:

00117 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.228.200
00117 deny ip from any to 69.55.228.200

jail19 would be 00119...
ipfw add 00109 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 } to 69.55.232.3
ipfw add 00109 deny ip from any to 69.55.232.3

41. select customers for probe map

42. patch jail against starting jails with rtprio

mv /usr/sbin/jail /usr/sbin/jail_
echo '#\!/bin/sh\
/usr/sbin/rtprio -t /usr/sbin/jail_ $*' > /usr/sbin/jail
chmod +x /usr/sbin/jail

43. make sure mail works
If there are map errors:
cd /etc/mail; make maps

44. rdate

cd /usr/ports/sysutils/rdate
make install clean

crontab -e
0 0 * * * /usr/local/sbin/rdate -s utcnist.colorado.edu

/usr/local/sbin/rdate -s utcnist.colorado.edu

45. recover space on /usr (optional)

rm -fr /usr/obj

46. wrapper jps

mv /usr/local/sbin/jps /usr/local/sbin/jps_

47. wrapper jls

mv /usr/sbin/jls /usr/sbin/jls_

48. wrapper jexec

mv /usr/sbin/jexec /usr/sbin/jexec_

49. install jtop

cd /usr/ports/sysutils/jtop
make install clean

50. block jails from reaching private net
echo 'ipfw add 1 deny ip from 69.55.224.0/20 to 10.1.4.0/24' > /usr/local/etc/rc.d/ipfw.sh
chmod 0700 /usr/local/etc/rc.d/ipfw.sh

xx. setup fuse

cd /usr/ports/sysutils/fusefs-kmod/
make install

vi /etc/rc.conf
fusefs_enable="YES"

sysctl vfs.usermount=1

cd /usr/ports/sysutils/fusefs-sshfs
make install

sshfs 1005@usw-s009.rsync.net: /mnt/data1/69.55.234.68-col00001-DIR/mnt

JCWiki - User contributions [en]

Clear space on boxes

Network Time (ntp)

Jail Server Install