Iptables is not blocking data

v4lk3r · 06-16-2015, 10:58 AM

I have blocked an attacker IP that was sending me lots of UDP packets.

iptables -I INPUT 1 -s IP_OF_ATTACKER -j DROP

This rule was working all fine.

iptables -nvL --line-numbers

22G traffic was blocked for 2-3 days:

num pkts bytes target prot opt in out source destination
1 3203K 22G DROP all -- * * ATTACKER_IP 0.0.0.0/0

However, from last 2-3 days, this rule isn't working anymore. Attacker is sending UDP packets and iptables are not blocking them.

num pkts bytes target prot opt in out source destination
1 707K 3553M DROP all -- * * ATTACKER_IP 0.0.0.0/0

What could be the reason?

PS: Please do not suggest about contacting hosting provider if they were any help I wouldn't be here

Edit

I used wireshark/tcpdump to analyze/capture packets. It shows all packets are UDP. I use iptables command (as mentioned above) to see how much data iptables rule has blocked. Above is the output of iptables blocked data. When iptable was blocking all the data, our server was working all fine.

**IP Table rules**
# Generated by iptables-save v1.4.21 on Mon Jun 15 23:26:40 2015
*raw
:PREROUTING ACCEPT [8393:667810]
:OUTPUT ACCEPT [7043:795032]
COMMIT
# Completed on Mon Jun 15 23:26:40 2015
# Generated by iptables-save v1.4.21 on Mon Jun 15 23:26:40 2015
*nat
:PREROUTING ACCEPT [2517:112725]
:INPUT ACCEPT [2517:112725]
:OUTPUT ACCEPT [1018:179752]
:POSTROUTING ACCEPT [1018:179752]
COMMIT
# Completed on Mon Jun 15 23:26:40 2015
# Generated by iptables-save v1.4.21 on Mon Jun 15 23:26:40 2015
*mangle
:PREROUTING ACCEPT [8393:667810]
:INPUT ACCEPT [8393:667810]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [7043:795032]
:POSTROUTING ACCEPT [7043:795032]
COMMIT
# Completed on Mon Jun 15 23:26:40 2015
# Generated by iptables-save v1.4.21 on Mon Jun 15 23:26:40 2015
*filter
:INPUT ACCEPT [23:1500]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [18:2068]
:fail2ban-ssh - [0:0]
-A INPUT -s xx.xxx.xx.xx/32 -j DROP
COMMIT
# Completed on Mon Jun 15 23:26:40 2015

smallpond · 06-16-2015, 12:11 PM

The problem with a UDP flood attack is that you don't know the source. They aren't expecting a response, so the attacker can make up any source IP and blast away. It is not unusual to send packets with random source IP to defeat your rules.

v4lk3r · 06-16-2015, 03:15 PM

Quote:

Originally Posted by smallpond

The problem with a UDP flood attack is that you don't know the source. They aren't expecting a response, so the attacker can make up any source IP and blast away. It is not unusual to send packets with random source IP to defeat your rules.

Thanks for your reply. I checked all the packets carefully. Attacker is not faking his IP and he is using the same packet size, same IP, and same attack. I can't comprehend how he is able to bypass this time. Any idea?

Sefyir · 06-16-2015, 04:02 PM

I'm not sure why the ip isn't working but...
why are you accepting the udp packets in the first place? Do you have any specific programs that are listening for udp packets?

Why not drop all packets except related or established ones and NEW packets for specific ports for programs listening? Then rate limiting those ports to acceptable levels.

Dropping all packets from ip address X is a very easy thing to get around and definitely not scalable unless you use a external program.

Edit: For example

Code:

iptables -P INPUT DROP
iptables -P FORWARD DROP
iptables -P OUTPUT ACCEPT

iptables -A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
iptables -A INPUT -i lo -j ACCEPT # Allow loopback

# Services
iptables -A INPUT -p tcp --dport 22 -m conntrack --ctstate NEW -m limit --limit 5/m -j ACCEPT # SSH

will drop everything but a local loopback, related or established connections, or NEW connections to ssh (dropping if more then 5 a minute - per ip)

v4lk3r · 06-16-2015, 04:36 PM

Quote:

Originally Posted by Sefyir

I'm not sure why the ip isn't working but...
why are you accepting the udp packets in the first place? Do you have any specific programs that are listening for udp packets?

Why not drop all packets except related or established ones and NEW packets for specific ports for programs listening? Then rate limiting those ports to acceptable levels.

Dropping all packets from ip address X is a very easy thing to get around and definitely not scalable unless you use a external program.

Edit: For example

Code:

iptables -P INPUT DROP
iptables -P FORWARD DROP
iptables -P OUTPUT ACCEPT

iptables -A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
iptables -A INPUT -i lo -j ACCEPT # Allow loopback

# Services
iptables -A INPUT -p tcp --dport 22 -m conntrack --ctstate NEW -m limit --limit 5/m -j ACCEPT # SSH

will drop everything but a local loopback, related or established connections, or NEW connections to ssh (dropping if more then 5 a minute - per ip)

Thanks. UDP is used for DNS resolution. My question is why iptables aren't working to block ip? If there is some issue with blocking ip using iptables then it may not work for limiting packets as well

Sefyir · 06-16-2015, 04:51 PM

I'm a little confused what rule in your iptable rules drops the attacker
Is it the one bolded?

Code:

# Generated by iptables-save v1.4.21 on Mon Jun 15 23:26:40 2015
*filter
:INPUT ACCEPT [23:1500]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [18:2068]
:fail2ban-ssh - [0:0]
-A INPUT -s xx.xxx.xx.xx/32 -j DROP
COMMIT
# Completed on Mon Jun 15 23:26:40 2015

Other then that, your entire ruleset is accepting everything. it also looks like a range then a particular ip address (trying to be clear)

chrism01 · 06-17-2015, 06:20 AM

Actually, /32 is one address - there's only 32 bits in an IPv4 address.
IPv6 is different kettle of fish

v4lk3r · 06-17-2015, 01:40 PM

Quote:

Originally Posted by Sefyir

I'm a little confused what rule in your iptable rules drops the attacker
Is it the one bolded?

Code:

# Generated by iptables-save v1.4.21 on Mon Jun 15 23:26:40 2015
*filter
:INPUT ACCEPT [23:1500]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [18:2068]
:fail2ban-ssh - [0:0]
-A INPUT -s xx.xxx.xx.xx/32 -j DROP
COMMIT
# Completed on Mon Jun 15 23:26:40 2015

Other then that, your entire ruleset is accepting everything. it also looks like a range then a particular ip address (trying to be clear)

Yes the bold rule is blocking the traffic.

Sefyir · 06-17-2015, 02:34 PM

Ok. /32 is one address (something new every day)

iptables doesn't do a lot of thinking in this rule
If the packet source is xx.xxx.xx.xx and is sent to the machine (INPUT), it is dropped.

Quote:

I checked all the packets carefully.
...
However, from last 2-3 days, this rule isn't working anymore. Attacker is sending UDP packets and iptables are not blocking them.

num pkts bytes target prot opt in out source destination
1 707K 3553M DROP all -- * * ATTACKER_IP 0.0.0.0/0

My impression is that the attacker is using different ips, maybe only slightly different. Your ip rule says it has blocked 3.5GB of data from the user so it looks like it is working. What if the other 19G is spread out to 5 (or more) ip addresses? How are you sure that there aren't other ip addresses sending a equal amount of data? You say you used wireshark to monitor the traffic, but if the user is still sending 22G of data over a period of days, that's a lot of traffic to manually check and be certain the attacker is at no point changing ip addresses.

Why not see who sends a abuse-level of data and log them?

v4lk3r · 06-17-2015, 02:37 PM

Quote:

Originally Posted by Sefyir

Ok. /32 is one address (something new every day)

iptables doesn't do a lot of thinking in this rule
If the packet source is xx.xxx.xx.xx and is sent to the machine (INPUT), it is dropped.

My impression is that the attacker is using different ips, maybe only slightly different. Your ip rule says it has blocked 3.5GB of data from the user so it looks like it is working. What if the other 19G is spread out to 5 (or more) ip addresses? How are you sure that there aren't other ip addresses sending a equal amount of data? You say you used wireshark to monitor the traffic, but if the user is still sending 22G of data over a period of days, that's a lot of traffic to manually check and be certain the attacker is at no point changing ip addresses.

Why not see who sends a abuse-level of data and log them?

Thanks for your prompt reply. I logged the data and analyzed using Wireshark. I also kept any eye on "iftop" stats to see which IPs are sending me lots of traffic and there was only one.

Habitual · 06-17-2015, 04:06 PM

I'd use http://www.tcpiputils.com/ to find and block the the entire "IP-range/subnet" CIDR shown there. and I never deny any /32s.
It's a waste of time. Reasoning is if there's one infected host on a network segment, there are others.
</opinion>

adrninistration · 06-21-2015, 07:17 PM

if you could make an backup of your existing setup, then create an seperate segment for the iptables setup using fail2ban.

cd /etc/fail2ban
sudo cp jail.conf jail.local
sudo nano jail.local

---------- Post added 06-21-15 at 02:18 PM ----------

then try and apply the same rules you currently have.

This should overcome the issue with iptables running on the same host if.

v4lk3r · 06-22-2015, 10:39 AM

Quote:

Originally Posted by adrninistration

if you could make an backup of your existing setup, then create an seperate segment for the iptables setup using fail2ban.

cd /etc/fail2ban
sudo cp jail.conf jail.local
sudo nano jail.local

---------- Post added 06-21-15 at 02:18 PM ----------

then try and apply the same rules you currently have.

This should overcome the issue with iptables running on the same host if.

It didn't work. Would it be possible that he can use ipv6 to DoS my server? Maybe I am blocking his ipv4 address not ipv6? In iftop command it only shows ipv4 though.