TCP: Out of sequence
 

Hi to all,

I have a client/server application. The client is windows xp and the server is linux red hat 4.0. The network is IPSec. If the server has to send an application message bigger than mtu size (1400), for example 2000 bytes, the tcp split the message in two segment, the first with size 1360 and the second with size 680.

The problem is that everytime the server send a message bigger then mtu size, the client receive the segment in out of sequence, and send duplicate ack. Sometimes, when the server receive a lot of duplicate acks, retransmit the segment.

With a sniffer application I saw that the first message (1360) had the push flag set to 0, instead the second one (680) had the push flag set to 1.

I have already tried to use the TCP_NODELAY and TCP_CORK option, but the scenario didn't change.

Can anyone help me with this?

Thanks,
Filippo.

Moved: This thread is more suitable in Linux-Networking and has been moved accordingly to help your thread/question get the exposure it deserves.

I have found that having a lot of duplicate acks and out of sequence packets can be caused by faulty hardware. It could be a NIC or it could be some other hardware in the network such as a switch/hub. This would require that you perform the ancient and venerable tasks which, taken together, are known as device level troubleshooting. :) Basically that means that you have to swap one suspect component at a time out of the configuration and replace it with a known good replacement. When you swap out the bad component the problem will go away. Then you know what component caused the problem. Don't forget that cables are components too. A bad cable could cause this kind of problem.

Do you mean DF bit? Because PUSH has nothing to do with MTU handling.

I suggest that you read
http://www.faqs.org/rfcs/rfc2401.html
Especially
6. ICMP Processing (relevant to IPsec)
Appendix B -- Analysis/Discussion of PMTU/DF/Fragmentation Issues

Do you run the sniffer on the server?

On a LAN, getting duplicate acks or retransmission could mean bad hardware or your ethernet segment is too long (when CMCA/CD listens to noise on the wire, it does not have the 64bit time to hear somebody else talking then there is a conflict with two machines talking on the same wire). It's called late collisions and can be seen on TCP levels. The normal 5% collisions are not seen on tcp level.

Thanks a lot for your reply, but ...

I have the same problem on 12 different server, so I don't think it can be an hw problem of the server.

The network is shared with a lot of other applications that doesnt't have the same problem, so I don't think it can be a network hw problem.

If I send message smaller than the mtu size, everything work correctly (no duplicate ack, no retransmit).

Do you know a way to set the push flag for very tcp segment, may be a kernel parameter?

Thanks again

Filippo.

No, I mean the PSH flag in the TCP Datagram. The field where the TCP can set URG, ACK, RST, SYN and FIN bit flag.

Yes, AFAIK it has nothing to do with your problem. Usually an application will set the last packet with a PSH flag so that the receiver TCP stack PuSHes it to its application.
MTU operates on *IP level*, not an end-to-end algorithm.
TCP windows are defined as end-to-end, meaning the two workstations agree on a sliding window.
As some networks have a more limited packet size, your tcp segment can be fragmented to be able to pass through this network. This is done by the outer router before entering this network.
Path MTU discovery will probe the complete path and your workstation gets the lowest MTU of any links so that no fragmentation will be needed on the complete path.
Unfortunatly this creates security problems because some routers will have to send ICMP message too big back to the emitting router. These can be filtered and in tunnels, probably people do not want you to see the IP address space.

A solution can be to set the MTU of the workstation to a low value and disable the pmtu discovery (sysctl switch in linux)

No, on server side I receive only the duplicate ack message, because the client receive first the small segment with the push flag set to 1 then the other segment with the push flag set to 0.

Hi, I've updated my post which contained some wrong information. This subject interest me because I might have to play with VPNs in the next days where people have performance problems or sometimes complete cut of network. I added a radical solution that I've never tested but that theorically could work (from memory)
More information:
http://www.ietf.org/rfc/rfc2923.txt
http://www.ietf.org/rfc/rfc1191.txt
http://www.ietf.org/rfc/rfc1435.txt

A question bothers me. Is it only one application on the server that fails to communicate? Does ssh work well at least?

Also I still don't understand why you would receive duplicate ack if MTU is really your problem. I expect no communication at all during maybe 10 minutes and then traffic again failing.

Good luck

Hi, I have already tested my application without using the VPN, but the behaviour is always the same (The first message doesn't have the psh flag setted).

If you are going to use VPN, remember to reduce the MTU on your server and in your client system (if you can). This is useful for the network because so it has enough space to put the ipsec header in the frame without re-fragment the message.

1th answer: The server has the same behavior with different server and client application.

2th answer: I didn't try my application with ssh protocol, but I don't think it can change something.

3th answer: The server receive duplicate acks because the client receive the messages in out of sequence. The communication never stops, but if the server receive 3* duplicate acks in a short period think that the message is lost from the client, so the server retrasmit it. The retrasmission introduce a significant latency (That's my real problem).

*3 is the default for the parameter of the Kernel called (net.ipv4.tcp_reordering).

If I don't find a way to force the psh flag, I will try to increase the value of the parameter to 10, so I hope to avoid the retrasmission.

Thanks a lot for your interesting,

Filippo.

I had the similiar problem before.

but it's because that our network used ip-ip tunnel, that introduced overhead so the MTU of the tunnel was smaller than the physical one.

don't know why it's not permitted to do fragment on the tunnel, so
all connections(not only vpn, but http was the same) were dropped when there was big ip package.

Thanks for the information, but I don't understand how did you solve the problem. Did you decrease the MTU size?

Thanks,
Filippo.

yes, change windows's MTU by running regedit. can't remember which key, you might google it