FreeBSD Reference - Revision history

70.230.212.110: /* FreeBSD drops to boot: prompt */

2013-03-15T22:40:00Z

FreeBSD drops to boot: prompt

Show changes

70.230.212.110: /* Creating a FreeBSD partition by hand on 5.x+ */

2013-03-15T21:09:41Z

Creating a FreeBSD partition by hand on 5.x+

Show changes

70.230.212.110: /* Change the serial console speed on FreeBSD */

2013-03-15T20:57:35Z

Change the serial console speed on FreeBSD

← Older revision		Revision as of 13:57, 15 March 2013
Line 1,447:		Line 1,447:


	== Change the serial console speed on FreeBSD ==		= Change the serial console speed on FreeBSD =

	http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/serialconsole-setup.html		http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/serialconsole-setup.html

	http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/serialconsole-setup.html#SERIALCONSOLE-COM2		http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/serialconsole-setup.html#SERIALCONSOLE-COM2

	Edit the speed in /etc/make.conf		Edit the speed in /etc/make.conf
	Then,		Then,

			[[#Rebuild_boot_blocks\|Rebuild boot blocks]]

			= Rebuild boot blocks =

	<pre>cd /sys/boot		<pre>cd /sys/boot
Line 1,460:		Line 1,465:
	make install		make install
	reboot</pre>		reboot</pre>


	= Notes on resizing gconcats (with growfs) =		= Notes on resizing gconcats (with growfs) =

70.230.212.110: /* Notes on resizing gconcats (with growfs) */

2013-03-15T20:56:07Z

Notes on resizing gconcats (with growfs)

← Older revision		Revision as of 13:56, 15 March 2013
Line 1,344:		Line 1,344:

	that matches up the rules with the directories, and every five minutes overwrites the users' /jc_traffic_dump with the latest traffic stats.		that matches up the rules with the directories, and every five minutes overwrites the users' /jc_traffic_dump with the latest traffic stats.

			= Recovering customers from a dead jail =

			These are old instructions for 4.x on a Dell 2450. This will differ a bit (no need to create devices) in modern OS's

			Start with 4.10 cd1, have cd2 on hand. (or use kern, mfs, fixit floppies from john)
			<pre>Skip kernel config
			Enter fixit

			cd /dev
			mknod aacd0s1a c 151 0x00020000
			(use major and minor from a similar system)

			mkdir /mnt

			mount /dev/aacd0s1a /mnt</pre>

			/boot/loader.conf is where serial console redirect output lives, so comment that if you want to catch output on kvm

			to switch to old kernel:
			chflags noschg kernel
			mv kernel kernel.new

			do this to new kernel:
			chflags schg kernel

			to get on network:
			ifconfig fxp0 10.1.4.? netmask 255.255.255.0
			route add default 10.1.4.1

			To switch kernel from debug before kernel boot,
			unload kernel
			load <kernel file name>
			ex: kernel.GENERIC


			= Creating a FreeBSD partition by hand on 5.x+ =

			Make sure nothing is mounted on /dev/aacd1s1e

			<pre>df
			Filesystem 1K-blocks Used Avail Capacity Mounted on
			/dev/aacd0s1a 126702 53952 62614 46% /
			devfs 1 1 0 100% /dev
			/dev/aacd0s1h 64723478 3701892 61021586 6% /mnt/data1
			/dev/aacd1s1e 139035548 15496136 123539412 11% /mnt/data2
			/dev/aacd0s1f 253678 462 232922 0% /tmp
			/dev/aacd0s1g 3045006 1771920 1029486 63% /usr
			/dev/aacd0s1e 253678 10162 223222 4% /var
			procfs 4 4 0 100% /proc

			umount /mnt/data2

			bsdlabel -e /dev/aacd1s1

			# /dev/aacd1s1:
			8 partitions:
			# size offset fstype [fsize bsize bps/cpg]
			b: 4194304 0 swap
			c: 286679862 0 unused 0 0 # "raw" part, don't edit
			e: 282485558 4194304 4.2BSD 0 0 0</pre>

			going to create a new e, 2G, and the rest will be f, so replace e with:

			<pre>e: 4194304 4194304 4.2BSD 0 0 0
			f: 278291254 8388608 4.2BSD 0 0 0</pre>

			make sure it took:

			<pre>bsdlabel /dev/aacd1s1
			# /dev/aacd1s1:
			8 partitions:
			# size offset fstype [fsize bsize bps/cpg]
			b: 4194304 0 swap
			c: 286679862 0 unused 0 0 # "raw" part, don't edit
			e: 4194304 4194304 4.2BSD 0 0 0
			f: 278291254 8388608 4.2BSD 0 0 0</pre>

			create new fs:

			newfs /dev/aacd1s1e
			(output)
			newfs /dev/aacd1s1f
			(output)

			mkdir /mnt/data3

			mount /dev/aacd1s1e /mnt/data2
			mount /dev/aacd1s1f /mnt/data3

			<pre>df
			Filesystem 1K-blocks Used Avail Capacity Mounted on
			/dev/aacd0s1a 126702 53954 62612 46% /
			devfs 1 1 0 100% /dev
			/dev/aacd0s1h 64723478 3701892 61021586 6% /mnt/data1
			/dev/aacd0s1f 253678 462 232922 0% /tmp
			/dev/aacd0s1g 3045006 1771920 1029486 63% /usr
			/dev/aacd0s1e 253678 10162 223222 4% /var
			procfs 4 4 0 100% /proc
			/dev/aacd1s1e 2026030 4 1863944 0% /mnt/data2
			/dev/aacd1s1f 134764734 4 123983552 0% /mnt/data3</pre>


			== Change the serial console speed on FreeBSD ==

			http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/serialconsole-setup.html
			http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/serialconsole-setup.html#SERIALCONSOLE-COM2

			Edit the speed in /etc/make.conf
			Then,

			<pre>cd /sys/boot
			make clean
			make
			make install
			reboot</pre>


	= Notes on resizing gconcats (with growfs) =		= Notes on resizing gconcats (with growfs) =

70.230.212.110: /* Setting up bandwidth caps on firewall */

2013-03-14T23:31:26Z

Setting up bandwidth caps on firewall

← Older revision		Revision as of 16:31, 14 March 2013
Line 1,245:		Line 1,245:

	more than one rule can feed into a pipe, so the speed of everything that matches will get lumped together in the same pipe. this is useful when a customer has more than on IP or system, and you want to limit his total combined speed.		more than one rule can feed into a pipe, so the speed of everything that matches will get lumped together in the same pipe. this is useful when a customer has more than on IP or system, and you want to limit his total combined speed.

			== Setting up bandwidth caps on a jail ==

			If there's a situation where a VPS (or VPSs) on a jail are spiking bandwidth, but it's hard to tell who's the culprit, you can put in a global ipfw rule that will limit the connect speed for the entire server and/or each IP on the server.

			The commands to limit the overall outbound connect speed are:
			/sbin/ipfw pipe 1 config bw 40Mbit/s
			/sbin/ipfw add 1 pipe 1 ip from me to any

			The commands to limit the individual outbound connect speed are:
			/sbin/ipfw pipe 2 config bw 12Mbit/s mask src-ip 0xffffffff
			/sbin/ipfw add 2 pipe 2 ip from me to any

	== ipfw on jail machines ==		== ipfw on jail machines ==

70.230.212.110: /* Setting up bandwidth caps */

2013-03-14T23:25:51Z

Setting up bandwidth caps

← Older revision		Revision as of 16:25, 14 March 2013
Line 1,188:		Line 1,188:
	tcpdump –vvv –n –i em1		tcpdump –vvv –n –i em1

	== Setting up bandwidth caps ==		== Setting up bandwidth caps on firewall ==

	Creating a new pipe to limit someone's outbound speed:		Creating a new pipe to limit someone's outbound speed:

70.230.212.110: /* BSD Help */

2013-03-02T00:34:33Z

BSD Help

← Older revision		Revision as of 17:34, 1 March 2013
Line 1,355:		Line 1,355:
	or for 1G volumes: 2G: 4194302 - 16 = 4194286 2G: 2097152 * 2 - 2 = 4194302</pre>		or for 1G volumes: 2G: 4194302 - 16 = 4194286 2G: 2097152 * 2 - 2 = 4194302</pre>

	== BSD Help ==		= BSD Help =

	you can send any question you want to freebsd-questions@freebsd.org – you do not have to be subscribed to this mailing list to send it an email. If anything is stumping you, you should send email there and ask. Further, you can get on EFNET and join #freebsdhelp to ask questions as well.		you can send any question you want to freebsd-questions@freebsd.org – you do not have to be subscribed to this mailing list to send it an email. If anything is stumping you, you should send email there and ask. Further, you can get on EFNET and join #freebsdhelp to ask questions as well.

	if not subscribed might not see answers, so sub before asking		if not subscribed might not see answers, so sub before asking

			= FreeBSD drops to boot: prompt =

			At the prompt help it find the loader:

			<pre>Default: 0:ad(0,a)
			boot: /boot/loader</pre>

70.230.212.110: /* Notes on resizing gconcats (with growfs) */

2013-03-02T00:33:34Z

Notes on resizing gconcats (with growfs)

← Older revision		Revision as of 17:33, 1 March 2013
Line 1,354:		Line 1,354:
	4G: 4194304 * 2 - 2 = 8388606		4G: 4194304 * 2 - 2 = 8388606
	or for 1G volumes: 2G: 4194302 - 16 = 4194286 2G: 2097152 * 2 - 2 = 4194302</pre>		or for 1G volumes: 2G: 4194302 - 16 = 4194286 2G: 2097152 * 2 - 2 = 4194302</pre>

			== BSD Help ==

			you can send any question you want to freebsd-questions@freebsd.org – you do not have to be subscribed to this mailing list to send it an email. If anything is stumping you, you should send email there and ask. Further, you can get on EFNET and join #freebsdhelp to ask questions as well.

			if not subscribed might not see answers, so sub before asking

70.230.212.110: /* Handling a DoS attack */

2013-03-02T00:21:03Z

Handling a DoS attack

← Older revision		Revision as of 17:21, 1 March 2013
Line 1,332:		Line 1,332:

	that matches up the rules with the directories, and every five minutes overwrites the users' /jc_traffic_dump with the latest traffic stats.		that matches up the rules with the directories, and every five minutes overwrites the users' /jc_traffic_dump with the latest traffic stats.

	~~== Handling a DoS attack ==~~

	~~There are two types of DoS attacks - inbound and outbound. An inbound attack targets a johncompanies IP, on the johncompanies network.~~

	Our [[Infrastructure_Machines#DOS_attacks\|doswatch]] script will generally notice this right away - within 5 seconds, and it will almost always stop a UDP flood. Everything else (TCP) will be noticed by Castle. They have the ability to insert a null-route for the target.

	Inbound DoS attacks are generally not serious and they get stopped by castle (usually) before you would even have a chance to do anything about it. When a customer is DoS’d, our agreement with Castle is that the customer is moved to the bad boy block. There is a(n email) paste for this notification. If the IP is nulled then they’re dead anyway and giving them a new IP is the only option anyway.

	~~The other type of DoS attack is much more serious. This is an outbound attack from one of our systems (always a linux system, it seems) to the outside world.~~

	The problem with this type of attack is that the data path between the jail and virt systems and our firewall is so fast, that any DoS application that is run from a customer system will almost certainly use up all the CPU on the firewall, and stop all network traffic – basically we are completely down. And since the app is running on the inside, it will never stop - if someone doesn't do something, the attack will go on indefinitely.

	~~It should be noted that castle also null-routes outbound attacks, so their headache is over - the problem is, our public network is still completely down.~~

	We’ve mostly solved this problem with iptables rules in place on all virts which restrict the speed of outbound UDP packets. It doesn’t restrict DNS traffic so we’re still succeptible to UDP attacks targeted at a remote DNS server.

	When an outbound attack occurs Castle will usually have the offending IP so all you have to do is find the VPS and stop it. If you’re unable to use the mgmt site to lookup the customer machine, you can do a database search from the shell on mail:

	~~<pre>mail# m 69.55.238.98~~
	~~status 1~~
	~~cid col00214~~
	~~first_name Shawn~~
	~~last_name Yeager~~
	~~email shawn@shawnyeager.com~~
	~~system jail15.johncompanies.com~~
	~~dir /mnt/data1/69.55.238.98-col00214-DIR~~
	~~veid~~
	~~ip 69.55.238.98;~~
	~~notes canc'ld PP S-8SK44779NL028624B on 8/6/03 Now paying bulk</pre>~~

	~~Once you find the machine, ssh to it’s private IP, then do a ps to see what was/is running at the time of attack (<tt>vp <veid></tt>). To further see where the process is run:~~
	~~ps wwp <pid>~~
	~~If the machine is barely responsive you might just have to stop their ve the first thing you do.~~

	~~If you can’t reach the offending machine, tell castle to unplug the yellow cable from the back of that machine~~

	If this were a jail machine, you would simply log onto that jail machine, run `top` and see the obvious problem process, run <tt>jpid</tt> and <tt>ps wwp <pid></tt> to see who it belongs to, <tt>jailkill</tt> that server, and so on. If you can note what username it is running under, all the better.

	If an attack is ongoing and castle has not null routed the IP, you need to try to figure out the IP that's sending/receiving the attack. In the case of the former, it's usually easy to find the VPS/server (once the IP is identified) and cut them off by shutting off their port, dropping their traffic at the firewall, stopping the dos'ing proc on the VPS, or stopping the server.

	When the attack is coming in, our attacked IP needs to be null routed to stop the attacker(s). However, when castle takes their packet capture they will almost always get a false positive when looking for the top talker. This is because the top talkers are rsync.net (69.43.165.0/27) and col01372 (69.55.234.230, 69.55.234.246). So any search should exclude those IPs and ranges. Any other IP they find should be closely inspected to make sure it's not one of large customers, like col01372.

	Sometimes you’ll have no idea an attack has happened until you get an email from Castle that there was a DoS attack. When you do get notification that there was a DoS attack, and if a null route has been placed, it’s important to contact the customer ASAP to 1) Notify him of what’s happened and, 2) Give him a new IP

	~~Tasks:~~

	# Create an entry in the Mgmt -> Reference -> DoSLog, using info fed to us by Castle (add the duration of the attack to the time of the attack to determine the time the attack ended). If he’s receiving a new IP (see below), add a note about what IP he was moved to: “moved to 69.55.239.XX”
	# Any customer who’s the target of an attack will lose his attacked IP and be given a new IP from the “bad boy” block – 69.55.239.0/24. Any customer who’s machine was used to DoS attack someone else does not need to receive a new IP.

	~~Incoming DOS:~~

	# Send an “incoming dos” email to the customer to explain what happened. Take care to cc his alt contacts esp if his email looks like it’s hosted on our server. Optional- you may want to preemptively update his DNS so that any domains pointed at the old IP will point at the new badboy IP. The ipswap script is useful for this purpose if there are many domains.
	# Switch out the attacked IP for the bad boy IP: for VPS’s, remove the old IP and add the new one. For dedicateds, tell them they will need to assign the new IP and remove the old one from their server. Offer an IPKVM if they’re nervous. Remind them that they may need to change the gateway IP if the attacked IP was the primary IP on their server.
	# Once the attacked IP has been removed, respond to the DoS email from Castle telling them it’s ok to remove the null and what bad boy IP they were moved to (Castle needs confirmation on the move). Save the email from castle to the “dos” folder.

	~~Outgoing DOS:~~

	# Do a ps and top on the customer’s VPS to see what process is doing the DoS attack. It’s almost always obvious and usually includes an IP address as part of the process name. Stop his VPS. Send an “outgoing dos” email to the customer to explain what happened, showing them their ps, telling them we took them offline till we could (in the case of a jail) coordinate a time with them to bring it back online when we know they’ll be waiting to login and take action to patch/fix (in the case of a linux VPS remind them they can fire it up via control panel- give link…unless the hacker has root access in which case assume he can reach the cpl as well and disable it or block all traffic to all ports). It’s good in these cases to offer to block off all ssh access (in our firewall) except from an IP they give us until the compromise has been identified and removed, ESPECIALLY if the dos process is a real user (not apache, or some other service) cause the hacker could log back in the moment the server comes back up. Take care to cc his alt contacts esp if his email looks like it’s hosted on our server.
	~~# Save the email to the “dos” folder.~~
	# Work with the customer to definitively identify how the hacker got in in the first place (and to prevent a future occurrence). Possibly offer the customer a new VPS to move into since the old one is “tainted”. Encourage them to look at passwords and patch all software, esp web-based.

	= Notes on resizing gconcats (with growfs) =		= Notes on resizing gconcats (with growfs) =

70.230.212.110: /* Handling a DoS attack */

2013-03-02T00:20:43Z

Handling a DoS attack

← Older revision		Revision as of 17:20, 1 March 2013
Line 1,363:		Line 1,363:
	notes canc'ld PP S-8SK44779NL028624B on 8/6/03 Now paying bulk</pre>		notes canc'ld PP S-8SK44779NL028624B on 8/6/03 Now paying bulk</pre>

	Once you find the machine, ssh to it’s private IP, then do a ps to see what was/is running at the time of attack (vp <veid>). To further see where the process is run:		Once you find the machine, ssh to it’s private IP, then do a ps to see what was/is running at the time of attack (<tt>vp <veid></tt>). To further see where the process is run:
	ps wwp <pid>		ps wwp <pid>
	If the machine is barely responsive you might just have to stop their ve the first thing you do.		If the machine is barely responsive you might just have to stop their ve the first thing you do.

	If you can’t reach the offending machine, tell castle to unplug the yellow cable from the back of that machine		If you can’t reach the offending machine, tell castle to unplug the yellow cable from the back of that machine

	If this were a jail machine, you would simply log onto that jail machine, run `top` and see the obvious problem process, run jpid and ps wwp <pid> to see who it belongs to, jailkill that server, and so on. If you can note what username it is running under, all the better.		If this were a jail machine, you would simply log onto that jail machine, run `top` and see the obvious problem process, run <tt>jpid</tt> and <tt>ps wwp <pid></tt> to see who it belongs to, <tt>jailkill</tt> that server, and so on. If you can note what username it is running under, all the better.

	If an attack is ongoing and castle has not null routed the IP, you need to try to figure out the IP that's sending/receiving the attack. In the case of the former, it's usually easy to find the VPS/server (once the IP is identified) and cut them off by shutting off their port, dropping their traffic at the firewall, stopping the dos'ing proc on the VPS, or stopping the server.		If an attack is ongoing and castle has not null routed the IP, you need to try to figure out the IP that's sending/receiving the attack. In the case of the former, it's usually easy to find the VPS/server (once the IP is identified) and cut them off by shutting off their port, dropping their traffic at the firewall, stopping the dos'ing proc on the VPS, or stopping the server.