[SOLVED] IPTABLES rate limit to block DDOS!

AsadMoeen · 02-29-2012, 08:14 AM

Well, I've been a victim of Ddos attacks and I really can't figure out how to avoid it. On some machines it works, while on some it doesn't or probably I'm doing it wrong in some way,

Attackers are using multiple IPs to attack my game server ports that make my game-server output 1mb/s to each IP Address, these are UDP reflective attacks technically. Here is how I see to protect them,

Code:

iptables -A INPUT -p udp -m state --state NEW -m recent --set --name DDOS --rsource 
iptables -A INPUT -p udp -m state --state NEW -m recent --update --seconds 1 --hitcount 5 --name DDOS --rsource -j DROP

Technically this would block every attacker on UDP ports. I check if the attacker is blocked using "iftop" or "tcpdump" when I see that the output to attacker's IP becomes 0, this confirms me that the protection is working.

On my CentOS 6 machine running iptables 1.4.7, I am using similar method to block it and its really weird that it works sometimes and the next hour it doesn't. After a lot of tries, I restarted my system and iptables, protection worked fine and attacker was blocked. I logged into my system today and he is attacking me again and although I restarted iptables, hes not getting blocked. I dont want to restart my system again and again to make it work so what could be the issue?

Also, some people say UDP is state-less but whatever it technically is - I've always used this command and it has worked before for me, why not now or why partially now?

lisle2011 · 02-29-2012, 09:23 AM

A malcontent or two is causing you grief.

What I see is that your line to DROP contains no port numbers, it seems that perhaps you have configured this in a way that is oversimplified and you need an if then fi.

Perhaps this example from my firewall will help.
if [ "$CONNECTION_TRACKING" = "1" ]; then
$IPTABLES -A OUTPUT -o $INTERNET -p udp -m multiport --destination-port $NFS_PORT,$LOCKD_PORT -m state --state NEW -j REJECT
$IPTABLES -A INPUT -i $INTERNET -p udp -m multiport --destination-port $NFS_PORT,$LOCKD_PORT -m state --state NEW -j DROP
else
$IPTABLES -A OUTPUT -o $INTERNET -p udp -m multiport --destination-port $NFS_PORT -j REJECT
$IPTABLES -A INPUT -i $INTERNET -p udp -m multiport --destination-port $NFS_PORT -j DROP
fi

I realize that this is specific to NFS but perhaps you could remodel what you have to do something similar and solve this issue.

We can never get away from bandits, they steal, break, obscure and generally are outlaws. Tracking their IP may be futile as they could be using techniques that obscure their real IP address. If you use REJECT they will know because they get that info so DROP should reveal nothing. As far as they are concerned they are unable to connect.

I can check a little deeper if you like.

AsadMoeen · 02-29-2012, 10:09 AM

I was using ports before but that didn't work too so I then started simplifying the commands to no use:

Here was the command :

Code:

iptables -A INPUT -p udp -m udp --dport 20100:20500 -m state --state NEW -m recent --set --name DEFAULT --rsource 
iptables -A INPUT -p udp -m udp --dport 20100:20500 -m state --state NEW -m recent --update --seconds 1 --hitcount 10 --name DEFAULT --rsource -j REJECT

AsadMoeen · 02-29-2012, 10:59 AM

Okay I tried CSF and that also doesn't block the attacker so now I'm sure problem is not with my commands.

Setting rate-limit in iptables manually or through CSF ( and restarting system ) blocks the attacker very fine but after 10 hours ( might be less, I just woke up after that much ), attacker is attacking again and iptables are doing nothing. After doing a lot of commands even then iptables wont work but on restarting the system, it works again by itself.

Why does this happen?

I have another VPS with InterServer.net running CentOS OpenVZ Kernel 2.6.18 ( like the machine above ), same thing is happening on that machine. At this moment, I just restarted my VPS and iptables is working fine and blocking the attacker but I'm sure when i wake up tomorrow, I'll see the attacker not being blocked anymore.

So Is there any issue with this kernel?

This is weird isn't it?

At least programming or mathematics can't explain that shit.

AsadMoeen · 02-29-2012, 11:07 AM

Okay this is what it appears to be.

If someone is attacking the system right at the current time, and if you restart your system - It will block the IP Addresses.

If now another attacker comes, it will not block that attacker or even if the previous attacker takes a break and attacks again, he will get through once again. Same case with the CentOS machine and its really weird.

Uptil now I was using rate limit to block it but I observed, attackers are sending constant packet of UDP length 14 and even blocking the length alone doesn't block the packets. But leave that aside, even directly blocking new attacker's IP won't work while the previous attacker who was attacking while server was restarting is blocked ( with rate-limit ). But maybe if I'm using CSF, I can't manually do length or manual block until I flush the chain?

I'm literally all messed up in the head now!

unSpawn · 02-29-2012, 05:22 PM

Your threads, over the past year, have displayed a wide range of trouble ranging from crashes to (perceived) root level compromises. Reading back it seems your DDoS problem started early last year (according to this, this and this thread) and apparently you found the issue urgent enough to start this thread (duplicate). What is bothering me in all of this is that these threads, over the period nearly of a year, have not lead to anything conclusive. Part of it seems due to a lack of follow-ups and part of it is due to you failing to communicate OS, service, attack and measure effectiveness details properly. It is important to realize that in terms of Netfilter usage, and other aspects like the ability for a hosting company to oversell, OpenVZ is not equal to a dedicated host. Unless you manage dom0 yourself, then until your dom0 owner cooperates, you will not be able to achieve what you need to.

I strongly suggest that you start from scratch in your next reply and post detailed OS information: kernel version, game version and ports used, iptables version, full iptables rule set as in 'iptables-save > outputfile' and enough packet captures that can be used to analyze traffic.

AsadMoeen · 03-04-2012, 02:36 AM

Well, thank you for your reply.

Iptables rules are somehow not working even on XEN machines with latest kernel and not even on 2.6.18 Kernel OpenVZ. I tried contacting my providers but they couldn't help me about it. I don't know why on restart, the rules start working and after a few hours - the attacker is able to break in.

The above rules are the only thing I have in my iptables list.

I am getting over 20mb/s DDOS attack!

AsadMoeen · 03-04-2012, 02:37 AM

It could also be possible that rules are working and tcpdump/iftop is showing some wrong information?

Can it be?

AsadMoeen · 03-04-2012, 05:14 AM

In my /var/log/messages:

I see this,

Mar 4 15:15:22 gsh kernel: device eth0 entered promiscuous mode
Mar 4 15:15:24 gsh kernel: device eth0 left promiscuous mode
Mar 4 15:18:51 gsh kernel: device eth0 entered promiscuous mode
Mar 4 15:18:53 gsh kernel: device eth0 left promiscuous mode
Mar 4 15:21:16 gsh kernel: device eth0 entered promiscuous mode
Mar 4 15:21:18 gsh kernel: device eth0 left promiscuous mode
Mar 4 16:32:17 gsh kernel: device eth0 entered promiscuous mode
Mar 4 16:32:19 gsh kernel: device eth0 left promiscuous mode
Mar 4 17:40:21 gsh kernel: device eth0 entered promiscuous mode
Mar 4 17:40:24 gsh kernel: device eth0 left promiscuous mode

lisle2011 · 03-04-2012, 10:59 AM

It would be more useful to have you respond to unSpawn in the manner requested. The information you impart is not very useful.

Quote:

strongly suggest that you start from scratch in your next reply and post detailed OS information: kernel version, game version and ports used, iptables version, full iptables rule set as in 'iptables-save > outputfile' and enough packet captures that can be used to analyze traffic.

Your firewall seems to be the problem and perhaps it is configured incorrectly. Responding positively to a request for information will net you more help than you can get get by offering conversational type dialogue.

Please respond.

AsadMoeen · 03-05-2012, 04:19 AM

Here you go:

I am using 2 machines,

Debian 6 OpenVZ Kernel:

Code:

vps:~# uname -a
Linux vps 2.6.18-238.5.1.el5.028stab085.5 #1 SMP Thu Apr 14 15:42:34 MSD 2011 i686 GNU/Linux

Debian 6 Xen HVM Custom Kernel:

Code:

root@vps:~# uname -a
Linux vps 3.2.6-rt13 #2 SMP PREEMPT Sun Feb 26 19:03:15 CET 2012 x86_64 GNU/Linux

Another CentOS 6 machine on Kernel 2.6.18 was also tested not to work but I would exclude that.

Game Server/ Application Information:

Soldier of Fortune 2 v1.00
Default Port : 20100

Its a reflective UDP attack so attacker's IP is spoofed ( Multiple IPs that change everyday ).

Iptables and tcpdump output:

Code:

# Generated by iptables-save v1.4.2 on Wed Feb  1 17:28:45 2012
*mangle
:PREROUTING ACCEPT [70783678:5700316975]
:INPUT ACCEPT [70783678:5700316975]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [70282328:14612200657]
:POSTROUTING ACCEPT [70282328:14612200657]
COMMIT
# Completed on Wed Feb  1 17:28:45 2012
# Generated by iptables-save v1.4.2 on Wed Feb  1 17:28:45 2012
*filter
:INPUT ACCEPT [70783676:5700316893]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [70282328:14612200657]
-A INPUT -p udp -m udp --dport 20100 -m state --state NEW -m recent --set --name DEFAULT --rsource 
-A INPUT -p udp -m udp --dport 20100 -m state --state NEW -m recent --update --seconds 1 --hitcount 10 --name DEFAULT --rsource -j DROP 
-A INPUT -p tcp -m tcp --dport 20100 -m state --state NEW -m recent --set --name SSH --rsource 
-A INPUT -p tcp -m tcp --dport 20100 -m state --state NEW -m recent --update --seconds 1 --hitcount 10 --rttl --name SSH --rsource -j DROP 

COMMIT
# Completed on Wed Feb  1 17:28:45 2012
# Generated by iptables-save v1.4.2 on Wed Feb  1 17:28:45 2012
*nat
:PREROUTING ACCEPT [517867:23092482]
:POSTROUTING ACCEPT [78245:4356869]
:OUTPUT ACCEPT [78245:4356869]
COMMIT
# Completed on Wed Feb  1 17:28:45 2012

Already tried length and string blocking commands but that also didn't work.

Code:

13:55:53.751149 IP v-66-150-121-181.unman-vds.internap-nyc.nfoservers.com.27015 > MyServerIP.20100: UDP, length 15
13:55:53.751324 IP MyServerIP.20100 > v-66-150-121-181.unman-vds.internap-nyc.nfoservers.com.27015: UDP, length 648
13:55:53.752255 IP v-66-150-121-181.unman-vds.internap-nyc.nfoservers.com.27015 > MyServerIP.20100: UDP, length 15
13:55:53.752507 IP MyServerIP.20100 > v-66-150-121-181.unman-vds.internap-nyc.nfoservers.com.27015: UDP, length 648

Just a partial output.

Anything else required?

AsadMoeen · 03-05-2012, 08:38 AM

It appears anything specified with -A INPUT -m does not work ( the -m option ).

unSpawn · 03-05-2012, 10:56 AM

Quote:

Originally Posted by AsadMoeen

It appears anything specified with -A INPUT -m does not work ( the -m option ).

"Does not work" is not proper diagnostics. If the iptables recent module was available then it should show in the list of available target matches and module listing and if it was in use then catting your target bucket should show hits. Checking that makes no sense unless the iptables recent module was configured in the OpenVZ hardware node and your OpenVZ container was restarted. Without these two conditions being met you will not be able to use the module. Maybe it didn't sink in but that's basically what I wrote in post #6.

Quote:

Originally Posted by AsadMoeen

Soldier of Fortune 2 v1.00

It seems you run an old version. I don't know if there's any server updates but if you look at the CVE entries you see SOF2 isn't without problems: http://cve.mitre.org/cgi-bin/cvekey....ier+of+Fortune. Check the vendor for patches. Also tell me if you have Punkbuster enabled on the server.

Quote:

Originally Posted by AsadMoeen

Anything else required?

Yes. I haven't got a clue about your traffic. What I would like is output along the lines of you running (as root) something like 'tcpdump -C 50 -W 3 -p -n -nn -s 0 -i eth -w /path/to/eth.pcap' where "eth" is your 'net-facing Ethernet device name (like "eth0") and "/path/to/eth.pcap" is the path and file name on a partition with enough space to support storing 3 packet captures of 50 megs each. Once you have got these send me an email to discuss where I can download the files for analysis.

AsadMoeen · 03-05-2012, 11:53 AM

Well if you have Msn/ Skype or stuff, maybe I could show you.

V1.00 is a separate game and all other updates are separate so no problem with the game.

I also run a Cod4 server and on testing, it can also be flooded.

AsadMoeen · 03-05-2012, 01:10 PM

I know I appear to be a fool by saying this.

When I restart my machine, it works for 2 hours.

Linux has made me made.

I can pay a bit to someone who can help out, he may access my machine.