Editing DoS Attacks



There are two types of DoS attacks - inbound and outbound.  An inbound attack targets a johncompanies IP, on the johncompanies network.

Our [[Infrastructure_Machines#DOS_attacks|doswatch]] script will generally notice this right away - within 5 seconds, and it will almost always stop a UDP flood. Everything else (TCP) will be noticed by Castle. They have the ability to insert a null-route for the  target.

Inbound DoS attacks are generally not serious and they get stopped by castle (usually) before you would even have a chance to do anything about it. When a customer is DoS’d, our agreement with Castle is that the customer is moved to the bad boy block. There is a(n email) paste for this notification. If the IP is nulled then they’re dead anyway and giving them a new IP is the only option anyway.

The other type of DoS attack is much more serious.  This is an outbound attack from one of our systems (always a linux system, it seems) to the outside world.

The problem with this type of attack is that the data path between the jail and virt systems and our firewall is so fast, that any DoS application that is run from a customer system will almost certainly use up all the CPU on the firewall, and stop all network traffic – basically we are completely down.  And since the app is running on the inside, it will never stop - if someone doesn't do something, the attack will go on indefinitely.

It should be noted that castle also null-routes outbound attacks, so their headache is over - the problem is, our public network is still completely down.

We’ve mostly solved this problem with iptables rules in place on all virts which restrict the speed of outbound UDP packets. It doesn’t restrict DNS traffic so we’re still succeptible to UDP attacks targeted at a remote DNS server.

When an outbound attack occurs Castle will usually have the offending IP so all you have to do is find the VPS and stop it. If you’re unable to use the mgmt site to lookup the customer machine, you can do a database search from the shell on mail:

<pre>mail# m 69.55.238.98
status      1
cid         col00214
first_name  Shawn
last_name   Yeager
email       shawn@shawnyeager.com
system      jail15.johncompanies.com
dir         /mnt/data1/69.55.238.98-col00214-DIR
veid
ip          69.55.238.98;
notes       canc'ld PP S-8SK44779NL028624B on 8/6/03  Now paying bulk</pre>

Once you find the machine, ssh to it’s private IP, then do a ps to see what was/is running at the time of attack (<tt>vp <veid></tt>). To further see where the process is run:
 ps wwp <pid> 
If the machine is barely responsive you might just have to stop their ve the first thing you do.

If you can’t reach the offending machine, tell castle to unplug the yellow cable from the back of that machine

If this were a jail machine, you would simply log onto that jail machine, run `top` and see the obvious problem process, run <tt>jpid</tt> and <tt>ps wwp <pid></tt> to see who it belongs to, <tt>jailkill</tt> that server, and so on.  If you can note what username it is running under, all the better.

If an attack is ongoing and castle has not null routed the IP, you need to try to figure out the IP that's sending/receiving the attack. In the case of the former, it's usually easy to find the VPS/server (once the IP is identified) and cut them off by shutting off their port, dropping their traffic at the firewall, stopping the dos'ing proc on the VPS, or stopping the server.

When the attack is coming in, our attacked IP needs to be null routed to stop the attacker(s). However, when castle takes their packet capture they will almost always get a false positive when looking for the top talker. This is because the top talkers are rsync.net (69.43.165.0/27) and col01372 (69.55.234.230, 69.55.234.246). So any search should exclude those IPs and ranges. Any other IP they find should be closely inspected to make sure it's not one of large customers, like col01372.

Sometimes you’ll have no idea an attack has happened until you get an email from Castle that there was a DoS attack. When you do get notification that there was a DoS attack, and if a null route has been placed, it’s important to contact the customer ASAP to 1) Notify him of what’s happened and, 2) Give him a new IP

Tasks:

# Create an entry in the Mgmt -> Reference -> DoSLog, using info fed to us by Castle (add the duration of the attack to the time of the attack to determine the time the attack ended). If he’s receiving a new IP (see below), add a note about what IP he was moved to: “moved to 69.55.239.XX”
# Any customer who’s the target of an attack will lose his attacked IP and be given a new IP from the “bad boy” block – 69.55.239.0/24. Any customer who’s machine was used to DoS attack someone else does not need to receive a new IP.

Incoming DOS:

# Send an “incoming dos” email to the customer to explain what happened. Take care to cc his alt contacts esp if his email looks like it’s hosted on our server. Optional- you may want to preemptively update his DNS so that any domains pointed at the old IP will point at the new badboy IP. The ipswap script is useful for this purpose if there are many domains.
# Switch out the attacked IP for the bad boy IP: for VPS’s, remove the old IP and add the new one. For dedicateds, tell them they will need to assign the new IP and remove the old one from their server. Offer an IPKVM if they’re nervous. Remind them that they may need to change the gateway IP if the attacked IP was the primary IP on their server.
# Once the attacked IP has been removed, respond to the DoS email from Castle telling them it’s ok to remove the null and what bad boy IP they were moved to (Castle needs confirmation on the move). Save the email from castle to the “dos” folder.

Outgoing DOS:

# Do a ps and top on the customer’s VPS to see what process is doing the DoS attack. It’s almost always obvious and usually includes an IP address as part of the process name. Stop his VPS. Send an “outgoing dos” email to the customer to explain what happened,  showing them their ps, telling them we took them offline till we could (in the case of a jail) coordinate a time with them to bring it back online when we know they’ll be waiting to login and take action to patch/fix (in the case of a linux VPS remind them they can fire it up via control panel- give link…unless the hacker has root access in which case assume he can reach the cpl as well and disable it or block all traffic to all ports). It’s good in these cases to offer to block off all ssh access (in our firewall) except from an IP they give us until the compromise has been identified and removed, ESPECIALLY if the dos process is a real user (not apache, or some other service) cause the hacker could log back in the moment the server comes back up. Take care to cc his alt contacts esp if his email looks like it’s hosted on our server. 
# Save the email to the “dos” folder.
# Work with the customer to definitively identify how the hacker got in in the first place (and to prevent a future occurrence). Possibly offer the customer a new VPS to move into since the old one is “tainted”. Encourage them to look at passwords and patch all software, esp web-based.