LinuxQuestions.org
Visit the LQ Articles and Editorials section
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Networking
User Name
Password
Linux - Networking This forum is for any issue related to networks or networking.
Routing, network cards, OSI, etc. Anything is fair game.

Notices

Reply
 
Search this Thread
Old 04-22-2008, 09:22 PM   #1
xnomad
Member
 
Registered: Jun 2005
Posts: 53

Rep: Reputation: 15
TCP handshake fails, SYN/ACK ignored by system.


Hi,

We are experiencing an intermittent TCP handshake problem between two of our servers. 10.2.2.21 is running apache on CENTOS 5 and is a proxy for 10.2.2.30 which is running IIS on Windows 2003. Below is a tcpdump showing a normal TCP handshake, and below that the one where it fails. We notice long outages during a request when this happens, and it is always exactly a 45 second wait.

All port 80 traffic from the internet is forwarded by DNAT rule on our border router (CENTOS 5 running shorewall/iptables firewall) to 10.2.2.21 and then the apache proxy hands it over to the IIS on 10.2.2.30


The TCP dump is identical on both machines so the SYN/ACK is actually received by the network interface on the Linux server, but for some reason the TCP stack doesn't respond with an ACK for quite some time, and keeps sending SYN's as if the SYN/ACK never arrived.



Normal:

No. Time Source Destination Protocol Info
1298 1.995355 10.2.2.21 10.2.2.30 TCP 60447 > http [SYN] Seq=0 Len=0 MSS=1460 TSV=3170731083 TSER=0 WS=7
1299 1.995383 10.2.2.30 10.2.2.21 TCP http > 60447 [SYN, ACK] Seq=0 Ack=1 Win=16384 Len=0 MSS=1460 WS=0 TSV=0 TSER=0
1300 1.995500 10.2.2.21 10.2.2.30 TCP 60447 > http [ACK] Seq=1 Ack=1 Win=5888 Len=0 TSV=3170731083 TSER=0
1301 1.995755 10.2.2.21 10.2.2.30 HTTP GET /home.asp HTTP/1.1


Problem:


No. Time Source Destination Protocol Info
2244 7.564111 10.2.2.21 10.2.2.30 TCP 60527 > http [SYN] Seq=0 Len=0 MSS=1460 TSV=3170736652 TSER=0 WS=7
2245 7.564145 10.2.2.30 10.2.2.21 TCP http > 60527 [SYN, ACK] Seq=0 Ack=1 Win=16384 Len=0 MSS=1460 WS=0 TSV=0 TSER=0
2246 10.564549 10.2.2.21 10.2.2.30 TCP 60527 > http [SYN] Seq=0 Len=0 MSS=1460 TSV=3170739652 TSER=0 WS=7
2247 10.732139 10.2.2.30 10.2.2.21 TCP http > 60527 [SYN, ACK] Seq=0 Ack=1 Win=16384 Len=0 MSS=1460 WS=0 TSV=0 TSER=0
2248 16.564328 10.2.2.21 10.2.2.30 TCP 60527 > http [SYN] Seq=0 Len=0 MSS=1460 TSV=3170745652 TSER=0 WS=7
2249 17.294295 10.2.2.30 10.2.2.21 TCP http > 60527 [SYN, ACK] Seq=0 Ack=1 Win=16384 Len=0 MSS=1460 WS=0 TSV=0 TSER=0
2250 28.563889 10.2.2.21 10.2.2.30 TCP 60527 > http [SYN] Seq=0 Len=0 MSS=1460 TSV=3170757652 TSER=0 WS=7
2251 52.563006 10.2.2.21 10.2.2.30 TCP 60527 > http [SYN] Seq=0 Len=0 MSS=1460 TSV=3170781652 TSER=0 WS=7
2252 52.563040 10.2.2.30 10.2.2.21 TCP [TCP Previous segment lost] http > 60527 [SYN, ACK] Seq=11717498 Ack=1 Win=16384

Len=0 MSS=1460 WS=0 TSV=0 TSER=0
2253 52.563150 10.2.2.21 10.2.2.30 TCP 60527 > http [ACK] Seq=1 Ack=11717499 Win=5888 Len=0 TSV=3170781652 TSER=0
2254 52.563473 10.2.2.21 10.2.2.30 HTTP GET /userecncount.asp?id=585 HTTP/1.1




Both servers are Dell Poweredge 2950, quad core servers with 3GB of Memory. They aren't overloaded at all by the processes they are running. The Linux server has 2 interfaces bonded in a failover configuration, and the Windows machine has 2 interfaces in a failover teaming configuration. The teaming and bonding is set up so that only one interface is live, there is no load balancing. As the TCP dump is identical on both machines, I doubt the interfaces are the culprit.

The netowrk bandwith on the Linux machine is nowhere overloaded either.

Something that might be related is that doing a 'netstat -antp' on the linux machine shows that there are always aroud 2500 open sockets with status TIME_WAIT. This is because we have a lot of database connections going from this machine to a Postgresql server.

As far as I know /proc/sys/fs/file-max value is 295328, so isn't that the maximum amount of sockets the system can open? So surely 2500 open sockets shouldn't congest the TCP stack?

Surely a configuration option for apache wouldn't influence a low level TCP handshake as this is at layer 4?

Thanks,

Last edited by xnomad; 04-23-2008 at 01:16 AM.
 
Old 09-28-2011, 12:10 PM   #2
lavicenne
LQ Newbie
 
Registered: Sep 2011
Posts: 1

Rep: Reputation: Disabled
Same issue using Solaris 10

Same issue occured on Solaris 10.
We thought first that the tcp_conn_req_max_q and tcp_conn_req_max_q0 were too low (now set @ 16384 and tcpListenDrop staying at 0) but the Load Balancers are still showing 1% of packets not SYN-ACKed.

Same probe on the same rythm ....

If someone has a clue, please share your knowledge

Thanks
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
TCP - ACK frequency meir129 Linux - Networking 2 04-08-2014 02:01 PM
iptables; ACK/SYN/etc; understand the bits, and potential firewall entries TheLinuxDuck Linux - Security 2 10-18-2011 10:17 PM
Linux change TCP kernel Parameter for TCP DELAY ACK TICKS linux_mando Linux - Networking 5 08-22-2006 09:20 AM
is it possible to ignore TCP three way handshake? Thinking Programming 7 12-02-2005 05:20 AM
TCP packet flags (SYN, FIN, ACK, etc) and firewall rules TheLinuxDuck Linux - Security 12 04-29-2005 12:30 AM


All times are GMT -5. The time now is 09:19 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration