I've been troubleshooting some intermittent connectivity issue between two systems in the same VLAN. The client is a RHEL 5.5 system and the server is a load balancer appliance.
The RHEL system is sending HTTP requests to the load balancer and about every couple hundred requests, one will hang for a few minutes. I ran a netstat on the RHEL while it was hanging and I saw a connection to the LB in a "SYN SENT" state. It stayed in this state for about a minute or two.
I ran tcpdumps on both systems and it looks like RHEL is intermittently rejecting the SYN-ACK from the LB. RHEL sends a SYN, LB sends a SYN-ACK, then RHEL responds back to the LB with "ICMP Destination unreachable (Host administratively prohibited)." Both tcpdumps show the same info. If I disable iptables on the RHEL, the issue goes away.
I looked at iptables and it looked okay to me. I'm not sure why it would only block some of the SYN-ACKs from the LB. Below is what the iptables looks like. Both the RHEL and LB are on 192.168.10.0/24. We have another RHEL system that has the same exact iptables and it doesn't have this problem. Any insight into this issue would be appreciated.
Code:
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [49012328:34994280488]
:RH-Firewall-1-INPUT - [0:0]
-A INPUT -j RH-Firewall-1-INPUT
-A FORWARD -j RH-Firewall-1-INPUT
-A RH-Firewall-1-INPUT -i lo -j ACCEPT
-A RH-Firewall-1-INPUT -p icmp -m icmp --icmp-type any -j ACCEPT
-A RH-Firewall-1-INPUT -p esp -j ACCEPT
-A RH-Firewall-1-INPUT -p ah -j ACCEPT
-A RH-Firewall-1-INPUT -d 224.0.0.251 -p udp -m udp --dport 5353 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 80 -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 443 -j ACCEPT
-A RH-Firewall-1-INPUT -s 192.168.10.0/255.255.255.0 -p tcp -m state --state NEW -m tcp -j ACCEPT
-A RH-Firewall-1-INPUT -s 192.168.10.0/255.255.255.0 -p udp -m state --state NEW -m udp -j ACCEPT
-A RH-Firewall-1-INPUT -m limit --limit 60/minute -j LOG --log-level 7 --log-prefix "Dropped by firewall: "
-A RH-Firewall-1-INPUT -j REJECT --reject-with icmp-host-prohibited
COMMIT
Additionally, I added some logging for iptables and it confirms that it is dropping the packets (RHEL is 192.168.10.100, LB is 192.168.10.200:
Code:
Jun 24 10:22:42 RHEL kernel: Dropped by firewall: IN=eth0 OUT= MAC=12:34:56:ab:cd:ef:00:11:22:33:44:55:66:77 SRC=192.168.10.200 DST=192.168.10.100 LEN=64 TOS=0x00 PREC=0x00 TTL=255 ID=25603 DF PROTO=TCP SPT=9080 DPT=33982 WINDOW=4380 RES=0x00 ACK SYN URGP=0
