LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Networking
User Name
Password
Linux - Networking This forum is for any issue related to networks or networking.
Routing, network cards, OSI, etc. Anything is fair game.

Notices


Reply
  Search this Thread
Old 09-11-2007, 02:37 AM   #1
filippo.crea
LQ Newbie
 
Registered: Nov 2004
Posts: 9

Rep: Reputation: 0
TCP: Out of sequence


Hi to all,

I have a client/server application. The client is windows xp and the server is linux red hat 4.0. The network is IPSec. If the server has to send an application message bigger than mtu size (1400), for example 2000 bytes, the tcp split the message in two segment, the first with size 1360 and the second with size 680.

The problem is that everytime the server send a message bigger then mtu size, the client receive the segment in out of sequence, and send duplicate ack. Sometimes, when the server receive a lot of duplicate acks, retransmit the segment.

With a sniffer application I saw that the first message (1360) had the push flag set to 0, instead the second one (680) had the push flag set to 1.

I have already tried to use the TCP_NODELAY and TCP_CORK option, but the scenario didn't change.

Can anyone help me with this?

Thanks,
Filippo.
 
Old 09-11-2007, 06:36 AM   #2
XavierP
Moderator
 
Registered: Nov 2002
Location: Kent, England
Distribution: Debian Testing
Posts: 19,192
Blog Entries: 4

Rep: Reputation: 475Reputation: 475Reputation: 475Reputation: 475Reputation: 475
Moved: This thread is more suitable in Linux-Networking and has been moved accordingly to help your thread/question get the exposure it deserves.
 
Old 09-11-2007, 07:07 AM   #3
stress_junkie
Senior Member
 
Registered: Dec 2005
Location: Massachusetts, USA
Distribution: Ubuntu 10.04 and CentOS 5.5
Posts: 3,873

Rep: Reputation: 335Reputation: 335Reputation: 335Reputation: 335
I have found that having a lot of duplicate acks and out of sequence packets can be caused by faulty hardware. It could be a NIC or it could be some other hardware in the network such as a switch/hub. This would require that you perform the ancient and venerable tasks which, taken together, are known as device level troubleshooting. Basically that means that you have to swap one suspect component at a time out of the configuration and replace it with a known good replacement. When you swap out the bad component the problem will go away. Then you know what component caused the problem. Don't forget that cables are components too. A bad cable could cause this kind of problem.

Last edited by stress_junkie; 09-11-2007 at 07:11 AM.
 
Old 09-11-2007, 07:21 AM   #4
nx5000
Senior Member
 
Registered: Sep 2005
Location: Out
Posts: 3,307

Rep: Reputation: 57
Do you mean DF bit? Because PUSH has nothing to do with MTU handling.

I suggest that you read
http://www.faqs.org/rfcs/rfc2401.html
Especially
6. ICMP Processing (relevant to IPsec)
Appendix B -- Analysis/Discussion of PMTU/DF/Fragmentation Issues

Do you run the sniffer on the server?

On a LAN, getting duplicate acks or retransmission could mean bad hardware or your ethernet segment is too long (when CMCA/CD listens to noise on the wire, it does not have the 64bit time to hear somebody else talking then there is a conflict with two machines talking on the same wire). It's called late collisions and can be seen on TCP levels. The normal 5% collisions are not seen on tcp level.

Last edited by nx5000; 09-11-2007 at 07:24 AM.
 
Old 09-11-2007, 07:21 AM   #5
filippo.crea
LQ Newbie
 
Registered: Nov 2004
Posts: 9

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by stress_junkie View Post
I have found that having a lot of duplicate acks and out of sequence packets can be caused by faulty hardware. It could be a NIC or it could be some other hardware in the network such as a switch/hub. This would require that you perform the ancient and venerable tasks which, taken together, are known as device level troubleshooting. Basically that means that you have to swap one suspect component at a time out of the configuration and replace it with a known good replacement. When you swap out the bad component the problem will go away. Then you know what component caused the problem. Don't forget that cables are components too. A bad cable could cause this kind of problem.
Thanks a lot for your reply, but ...

I have the same problem on 12 different server, so I don't think it can be an hw problem of the server.

The network is shared with a lot of other applications that doesnt't have the same problem, so I don't think it can be a network hw problem.

If I send message smaller than the mtu size, everything work correctly (no duplicate ack, no retransmit).

Do you know a way to set the push flag for very tcp segment, may be a kernel parameter?

Thanks again

Filippo.
 
Old 09-11-2007, 07:37 AM   #6
filippo.crea
LQ Newbie
 
Registered: Nov 2004
Posts: 9

Original Poster
Rep: Reputation: 0
No, I mean the PSH flag in the TCP Datagram. The field where the TCP can set URG, ACK, RST, SYN and FIN bit flag.
 
Old 09-11-2007, 07:50 AM   #7
nx5000
Senior Member
 
Registered: Sep 2005
Location: Out
Posts: 3,307

Rep: Reputation: 57
Yes, AFAIK it has nothing to do with your problem. Usually an application will set the last packet with a PSH flag so that the receiver TCP stack PuSHes it to its application.
MTU operates on *IP level*, not an end-to-end algorithm.
TCP windows are defined as end-to-end, meaning the two workstations agree on a sliding window.
As some networks have a more limited packet size, your tcp segment can be fragmented to be able to pass through this network. This is done by the outer router before entering this network.
Path MTU discovery will probe the complete path and your workstation gets the lowest MTU of any links so that no fragmentation will be needed on the complete path.
Unfortunatly this creates security problems because some routers will have to send ICMP message too big back to the emitting router. These can be filtered and in tunnels, probably people do not want you to see the IP address space.

A solution can be to set the MTU of the workstation to a low value and disable the pmtu discovery (sysctl switch in linux)

Last edited by nx5000; 09-13-2007 at 05:39 AM.
 
Old 09-11-2007, 08:01 AM   #8
filippo.crea
LQ Newbie
 
Registered: Nov 2004
Posts: 9

Original Poster
Rep: Reputation: 0
No, on server side I receive only the duplicate ack message, because the client receive first the small segment with the push flag set to 1 then the other segment with the push flag set to 0.
 
Old 09-13-2007, 05:39 AM   #9
nx5000
Senior Member
 
Registered: Sep 2005
Location: Out
Posts: 3,307

Rep: Reputation: 57
Hi, I've updated my post which contained some wrong information. This subject interest me because I might have to play with VPNs in the next days where people have performance problems or sometimes complete cut of network. I added a radical solution that I've never tested but that theorically could work (from memory)
More information:
http://www.ietf.org/rfc/rfc2923.txt
http://www.ietf.org/rfc/rfc1191.txt
http://www.ietf.org/rfc/rfc1435.txt

A question bothers me. Is it only one application on the server that fails to communicate? Does ssh work well at least?

Also I still don't understand why you would receive duplicate ack if MTU is really your problem. I expect no communication at all during maybe 10 minutes and then traffic again failing.

Good luck

Last edited by nx5000; 09-13-2007 at 05:41 AM.
 
Old 09-13-2007, 09:20 AM   #10
filippo.crea
LQ Newbie
 
Registered: Nov 2004
Posts: 9

Original Poster
Rep: Reputation: 0
Hi, I have already tested my application without using the VPN, but the behaviour is always the same (The first message doesn't have the psh flag setted).

If you are going to use VPN, remember to reduce the MTU on your server and in your client system (if you can). This is useful for the network because so it has enough space to put the ipsec header in the frame without re-fragment the message.

1th answer: The server has the same behavior with different server and client application.

2th answer: I didn't try my application with ssh protocol, but I don't think it can change something.

3th answer: The server receive duplicate acks because the client receive the messages in out of sequence. The communication never stops, but if the server receive 3* duplicate acks in a short period think that the message is lost from the client, so the server retrasmit it. The retrasmission introduce a significant latency (That's my real problem).

*3 is the default for the parameter of the Kernel called (net.ipv4.tcp_reordering).

If I don't find a way to force the psh flag, I will try to increase the value of the parameter to 10, so I hope to avoid the retrasmission.

Thanks a lot for your interesting,

Filippo.
 
Old 09-15-2007, 06:36 AM   #11
au_hank
LQ Newbie
 
Registered: Sep 2007
Posts: 17

Rep: Reputation: 0
I had the similiar problem before.

but it's because that our network used ip-ip tunnel, that introduced overhead so the MTU of the tunnel was smaller than the physical one.

don't know why it's not permitted to do fragment on the tunnel, so
all connections(not only vpn, but http was the same) were dropped when there was big ip package.

Last edited by au_hank; 09-15-2007 at 06:37 AM.
 
Old 09-15-2007, 07:11 AM   #12
filippo.crea
LQ Newbie
 
Registered: Nov 2004
Posts: 9

Original Poster
Rep: Reputation: 0
Thanks for the information, but I don't understand how did you solve the problem. Did you decrease the MTU size?

Thanks,
Filippo.
 
Old 09-15-2007, 07:14 AM   #13
au_hank
LQ Newbie
 
Registered: Sep 2007
Posts: 17

Rep: Reputation: 0
Quote:
Originally Posted by filippo.crea View Post
Thanks for the information, but I don't understand how did you solve the problem. Did you decrease the MTU size?

Thanks,
Filippo.
yes, change windows's MTU by running regedit. can't remember which key, you might google it
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
TCP Sequence Prediction introuble Linux - General 4 12-16-2006 02:54 PM
Accessing TCP flags in TCP packets on Linux using C !! vishamr2000 Programming 2 10-16-2006 09:46 AM
Linux change TCP kernel Parameter for TCP DELAY ACK TICKS linux_mando Linux - Networking 5 08-22-2006 08:20 AM
nmap - TCP Sequence Prediction ddaas Linux - Security 2 02-09-2005 02:16 AM
Woody 3.0 Open Ports 1470/tcp/uaiact 1518/tcp/vpvd What for?How can I remove them? alexxxis Debian 5 07-05-2004 05:18 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Networking

All times are GMT -5. The time now is 01:27 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration