Client Server process always drops circa 10% of replies irrespective of load
 

Hi
We have a problem that has been puzzling us for two days solid. I'm just throwing it out in case someone has come across the issue before.

Platform: SuSE 8.1

A pthreaded program using worker threads has been written. It experiences the following problems on Linux:

When multiple clients connect simultaneously (as in hundreds or thousands) circa 90% of the clients send a query to the server and get a successful reply.

Whether it is a dozen or 100 or 1000 clients, the remaining (circa) 10% successfully send a query to the server, has the server process it and the server successfully replies - but the client never sees the reply.

This is irrespective of the number of threads run, irrespective of whether the number of clients is more or less than the number of threads run and irrespective of whether the clients are separate processes or threads of a separate client. The result is always the same.

Configuration:
SO_LINGER has been set and the size of the SO_RCVBUF and SO_SNDBUF have been increased. The effect is the same with Nagel on or off and there are sufficient file descriptors.

Already checked:
On the server all writes are checked for length and send with multiple writes when necessary.
All writes, flushes and closes are checked for errors and none have been reported.

Other information:
SSL encryption and decryption is involved.
It should also be noted that it is not a question of thread safety as that has been carefully checked.

We're baffled so any help or pointers people might have would be greatly appreciated,

Thanks
Graham

What protocol: TCP or UDP packets?
When you say even the 'exceptions' replies do get sent from the server, are you checking inside the seerver or checking tcpdump/wireshark to see if the replies actually leave the server.
Best to run tcpdump from another system to be sure.

Hi Chris

Thanks for your reply.

The protocol is TCP. Unfortunately we can't run tcpdump/wireshark because this is happening on the loopback interface on the limited test environment we're working in.

By "exceptions" we mean that when the server detects an error it sends the error back to the client ....

Thanks
Graham

So are you saying you're running the 'server' and all the 'clients' on the same machine???
That's what you seem to be implying, if this is happening on the loopback interface..
By 'exceptions', I meant the 10% of 'clients' who don't get a reply.

Hi Chris,

Thanks for your replies and offers to help. Our developer discovered a couple of things in his code but he also discovered that fragmented packets are seemingly not handled as well on SUSE Linux as they were on FreeBSD and Solaris. He rewrote his program code to combine all replies in to single writes.

Thanks
Graham