Iptables is dropping accepted packages

Jinkzer · 03-11-2008, 01:09 PM

We have run into some strange problem with one of our servers. The server software is listening on TCP port 4100 for connections from other servers sending data. Everything works fine for some time (minutes to hours) but suddenly iptables start blocking packages destinated for port 4100. This generates a great loss of data and corrupt db. Strange thing here is that while one server getting drops another server succesfully connects to the port. Is there something we missing in the configuration?

Server is running RedHat ES 4 (Update 5)
/etc/sysconfig/iptables

Code:

Generated by iptables-save v1.2.11 on Wed Feb  6 10:19:58 2008
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [470:56680]
:RH-Firewall-1-INPUT - [0:0]
-A INPUT -j RH-Firewall-1-INPUT
-A FORWARD -j RH-Firewall-1-INPUT
-A RH-Firewall-1-INPUT -i lo -j ACCEPT
-A RH-Firewall-1-INPUT -p icmp -m icmp --icmp-type any -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 4100 -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 4200 -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 4400 -j ACCEPT
-A RH-Firewall-1-INPUT -m limit --limit 1/sec -j LOG --log-prefix "Dropped by firewall: "
-A RH-Firewall-1-INPUT -j REJECT --reject-with icmp-host-prohibited
COMMIT
# Completed on Wed Feb  6 10:19:58 2008

Code:

/var/log/messages
Mar 11 15:00:42 DBSERV kernel: Dropped by firewall: IN=bond0 OUT= MAC=00:1c:c4:59:09:da:00:30:18:4a:36:5a:08:00 SRC=172.21.221.98 DST=172.21.221.65 LEN=64 TOS=0x00 PREC=0x00 TTL=62 ID=43380 DF PROTO=TCP SPT=32884 DPT=4100 WINDOW=8420 RES=0x00 ACK URGP=0
Mar 11 15:00:43 DBSERV kernel: Dropped by firewall: IN=bond0 OUT= MAC=00:1c:c4:59:09:da:00:30:18:4a:36:5a:08:00 SRC=172.21.221.98 DST=172.21.221.65 LEN=64 TOS=0x00 PREC=0x00 TTL=62 ID=43382 DF PROTO=TCP SPT=32884 DPT=4100 WINDOW=8420 RES=0x00 ACK URGP=0
Mar 11 15:00:44 DBSERV kernel: Dropped by firewall: IN=bond0 OUT= MAC=00:1c:c4:59:09:da:00:30:18:4a:36:5a:08:00 SRC=172.21.221.98 DST=172.21.221.65 LEN=64 TOS=0x00 PREC=0x00 TTL=62 ID=43384 DF PROTO=TCP SPT=32884 DPT=4100 WINDOW=8420 RES=0x00 ACK URGP=0
Mar 11 15:00:45 DBSERV kernel: Dropped by firewall: IN=bond0 OUT= MAC=00:1c:c4:59:09:da:00:30:18:4a:36:5a:08:00 SRC=172.21.221.98 DST=172.21.221.65 LEN=191 TOS=0x00 PREC=0x00 TTL=62 ID=43386 DF PROTO=TCP SPT=32884 DPT=4100 WINDOW=8420 RES=0x00 ACK PSH URGP=0
Mar 11 15:00:45 DBSERV kernel: Dropped by firewall: IN=bond0 OUT= MAC=00:1c:c4:59:09:da:00:30:18:4a:36:5a:08:00 SRC=172.21.221.98 DST=172.21.221.65 LEN=191 TOS=0x00 PREC=0x00 TTL=62 ID=43388 DF PROTO=TCP SPT=32884 DPT=4100 WINDOW=8420 RES=0x00 ACK PSH URGP=0
Mar 11 15:00:45 DBSERV kernel: Dropped by firewall: IN=bond0 OUT= MAC=00:1c:c4:59:09:da:00:30:18:4a:36:5a:08:00 SRC=172.21.221.98 DST=172.21.221.65 LEN=64 TOS=0x00 PREC=0x00 TTL=62 ID=43390 DF PROTO=TCP SPT=32884 DPT=4100 WINDOW=8420 RES=0x00 ACK URGP=0
Mar 11 15:00:46 DBSERV kernel: Dropped by firewall: IN=bond0 OUT= MAC=00:1c:c4:59:09:da:00:30:18:4a:36:5a:08:00 SRC=172.21.221.98 DST=172.21.221.65 LEN=191 TOS=0x00 PREC=0x00 TTL=62 ID=43392 DF PROTO=TCP SPT=32884 DPT=4100 WINDOW=8420 RES=0x00 ACK PSH URGP=0
Mar 11 15:00:47 DBSERV kernel: Dropped by firewall: IN=bond0 OUT= MAC=00:1c:c4:59:09:da:00:30:18:4a:36:5a:08:00 SRC=172.21.221.98 DST=172.21.221.65 LEN=191 TOS=0x00 PREC=0x00 TTL=62 ID=43394 DF PROTO=TCP SPT=32884 DPT=4100 WINDOW=8420 RES=0x00 ACK PSH URGP=0

And so on...

Any ideas are much appriciated.

x_terminat_or_3 · 03-11-2008, 02:38 PM

Try this

Code:

-A RH-Firewall-1-INPUT -p tcp -m state --state NEW,INVALID -m tcp --dport 4100 -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m state --state NEW,INVALID -m tcp --dport 4200 -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m state --state NEW,INVALID -m tcp --dport 4400 -j ACCEPT

But that shouldn't have dropped packets from an existing connection state.

Can you try removing the limit for a bit and see if the problem persists

Jinkzer · 03-11-2008, 02:53 PM

Hi x_terminat_or_3,

I'll give it a shot tomorrow when I have access to the server, Im not sure what state the connection is when we get the drops, from the connecting (or connected) application I get a "connection error" in the log. Not always but the error might come say once or twice a day whith the drops above. In between the errors everything seems to work fine.

In total we have 3 network cards in the server, 2 of them are in a bonding (master/slave) configuration, not sure if this could have something to do with anything, we don't see anything in logs that interfaces have any errors.

Get back tomorrow with the results.

Deleriux · 03-13-2008, 09:15 AM

You might want to consider checking sar around about the same time to see what kind of memory was being used on the server and/or what kind of load the server was using.

Im banking on some packets getting put into INVALID as a state.