LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices


Reply
  Search this Thread
Old 09-22-2005, 03:17 PM   #1
JCipriani
Member
 
Registered: Aug 2005
Location: Pittsburgh, PA, USA
Distribution: Redhat 9, OS X 10.4.x, Win2K
Posts: 85

Rep: Reputation: 15
Sockets: multiple send() calls throttle server framerate.


I am having an annoying problem that I've sort of narrowed down but I still can't figure it out. Consider the following test code:

Code:
#define TWO_BYTES_AT_ONCE  0

void do_server_simple (void) {

  int s, c;
  unsigned char data;
  unsigned char data2[2];
  fps_info fps;

#if (TWO_BYTES_AT_ONCE)
  printf("server sending two bytes per request, 1 send\n");
#else
  printf("server sending two bytes per request, 2 sends\n");
#endif

  s = net_listen(10101);
  c = accept(s, NULL, NULL);
  net_close(s);

  reset_fps(&fps);

  while (1) {

    /* receive client bytes */
    recv(c, &data, 1, 0);

    /* send it back twice */
#if (TWO_BYTES_AT_ONCE)
    data2[0] = data;
    data2[1] = data;
    send(c, data2, 2, 0);
#else
    send(c, &data, 1, 0);
    send(c, &data, 1, 0);
#endif

    tick_fps(c, &fps);

  }

}


void do_client_simple (const char *addr) {

  int c;
  unsigned char data = 'x', data1, data2;
  fps_info fps;

  if ((c = net_connect(addr, 10101)) == -1) {
    fprintf(stderr, "what are you doing?\n");
    return;
  }

  reset_fps(&fps);

  while (1) {

    /* send byte to server */
    send(c, &data, 1, 0);

    /* receive the two byte response */
    data1 = data2 = 0;
    recv(c, &data1, 1, 0);
    recv(c, &data2, 1, 0);

    /* this will handle all the errors i didn't look for above */
    if (data1 != 'x' || data2 != 'x') {
      fprintf(stderr, "unexpected reply\n");
      break;
    }

    tick_fps(c, &fps);

  }

  net_close(c);

}
This is code for a simple client and server, where the server responds with two copies of every byte that the client sends it. The problem is, when I send both bytes back from the server in one call to send(), everything works fine, and fast. When I send both bytes back from the server one byte at a time with two calls to send(), the client/server frame rate is throttled down to exactly 25 FPS. When I say "exactly" I mean +/- 0.001 FPS.

For clarity, in the above code, I've removed most of my error checking, and some of the minor functions as well. During my tests, however, I do know that none of the above calls failed.

I can't figure out why this is happening but it's causing me a lot of problems in my real application. I have tried this on multiple Redhat 9 (kernel 2.4.20-31.9) machines, but I have not tried it on a machine with a different distribution/kernel. The following things have no effect:
  • Using select() to wait for bytes before the recv() calls.
  • Handling clients on separate threads in the server.
  • Receiving both bytes with a single call to recv() on the client side (doesn't really use a single call anyway when server is using two sends).
  • Does not matter if I'm running the server and client on the same machine, or on different machines on the same LAN, or on different machines connected with a patch cable, or on different machines not on the same LAN.

I am about to do more detailed timing tests to figure out the delay and I will post my results. However, testing in my real application shows that exactly one call to recv() (not necessarily the same call) on both the client and server side is taking exactly 0.04 seconds. Whether the delay is coming from the client or the server, I have not been able to determine yet.

My code for creating the sockets is nothing special. I am using connected, blocking, SOCK_STREAM sockets, and the only "extra" thing I do is set SO_REUSEADDR on the server's listening socket.

In addition to this, one of my coworkers has been using some client/server code that he wrote for quite some time now, and revealed to me today that his application was mysteriously limited to 25.000 FPS as well.

Has anybody had this problem... or does anybody know enough to shed some light on what could be causing these delays?

Thanks,
Jason

PS: Source code for the whole test program is here. It's ugly, sorry. The #defines are somewhere in the middle of the code if you want to mess with them.

Last edited by JCipriani; 09-22-2005 at 03:40 PM.
 
Old 09-22-2005, 04:04 PM   #2
JCipriani
Member
 
Registered: Aug 2005
Location: Pittsburgh, PA, USA
Distribution: Redhat 9, OS X 10.4.x, Win2K
Posts: 85

Original Poster
Rep: Reputation: 15
Here are some results for a very simple set of timing tests that I did. First, for the first 5 requests handled by the server, I timed the recv() and two send() calls using gettimeofday. The values in this table are in seconds, the first column is the recv(), the second and third columns are the two sends(), respectively:
Code:
server sending two bytes per request, 2 sends
server:   0.000004   0.000020   0.005487 total=  0.005511
server:   0.000008   0.000043   0.000003 total=  0.000054
server:   0.030322   0.000010   0.000002 total=  0.030334
server:   0.039941   0.000010   0.000001 total=  0.039952
server:   0.039959   0.000010   0.000001 total=  0.039970
The rest of the times are similar to the last one. This is the corresponding times on the client side, for the send() and then the two recvs():
Code:
client:   0.000016   0.000076   0.000016 total=  0.000108
client:   0.000006   0.005569   0.030283 total=  0.035858
client:   0.000070   0.000004   0.039856 total=  0.039930
client:   0.000071   0.000003   0.039867 total=  0.039941
client:   0.000070   0.000003   0.039799 total=  0.039872
As you can see the very first frame was OK. The results for the second frame indicate that most of the time was lost during the client's second recv() call. I can not determine whether the client took 0.3 seconds to receive the data, or whether the server took 0.3 seconds to send it after it's second send() actually returned.

Here are the results when the server uses just a single send() call, same deal as above except there's only two columns in the server timings:
Code:
server sending two bytes per request, 1 send
server:   0.000004   0.005457 total=  0.005461
server:   0.000008   0.000027 total=  0.000035
server:   0.000068   0.000007 total=  0.000075
server:   0.000049   0.000007 total=  0.000056
server:   0.000048   0.000007 total=  0.000055
Client:
Code:
client:   0.000015   0.000075   0.000003 total=  0.000093
client:   0.000007   0.005569   0.000004 total=  0.005580
client:   0.000054   0.000003   0.000001 total=  0.000058
client:   0.000051   0.000002   0.000002 total=  0.000055
client:   0.000051   0.000003   0.000001 total=  0.000055
With the stray 5ms delays presumably being because the computer was just catching up with itself immediately after the client started and the connection was accepted.

I don't know how to interpret any of that, really. I'd love to be able to find out exactly where the delay is taking place... I think I have some profiling tools on this machine somewhere, I'll post any more info I can come up with.
 
Old 09-22-2005, 06:17 PM   #3
JCipriani
Member
 
Registered: Aug 2005
Location: Pittsburgh, PA, USA
Distribution: Redhat 9, OS X 10.4.x, Win2K
Posts: 85

Original Poster
Rep: Reputation: 15
I've been playing with it more. It seems that this problem does not happen if I use UDP instead of TCP to send the data back and forth. Also, when using TCP, tcpdump says:

Code:
000048 mixalot.10101 > mixalot.33079: P 27:28(1) ack 13 win 32767 <nop,nop,timestamp 145586 145586> (DF)
000012 mixalot.33079 > mixalot.10101: . ack 28 win 32767 <nop,nop,timestamp 145586 145586> (DF)
000012 mixalot.33079 > mixalot.10101: P 13:14(1) ack 28 win 32767 <nop,nop,timestamp 145586 145586> (DF)
000011 mixalot.10101 > mixalot.33079: P 28:29(1) ack 14 win 32767 <nop,nop,timestamp 145586 145586> (DF)
039917 mixalot.33079 > mixalot.10101: . ack 29 win 32767 <nop,nop,timestamp 145590 145586> (DF)
000049 mixalot.10101 > mixalot.33079: P 29:30(1) ack 14 win 32767 <nop,nop,timestamp 145590 145590> (DF)
000011 mixalot.33079 > mixalot.10101: . ack 30 win 32767 <nop,nop,timestamp 145590 145590> (DF)
000012 mixalot.33079 > mixalot.10101: P 14:15(1) ack 30 win 32767 <nop,nop,timestamp 145590 145590> (DF)
000011 mixalot.10101 > mixalot.33079: P 30:31(1) ack 15 win 32767 <nop,nop,timestamp 145590 145590> (DF)
039916 mixalot.33079 > mixalot.10101: . ack 31 win 32767 <nop,nop,timestamp 145594 145590> (DF)
I used the -ttt option so those timestamps are deltas. 10101 is the server, 33079 is the client.I don't know much about that output but I do see the 0.04 second delays in the ack packets. I don't understand why this would be the case, though. TCP has been around for a while... and I've seen it work very fast; this suggests that I am doing something wrong but I'm not sure what it is. I mean, it seems like I should be able to send two bytes with separate calls to send() and not have it slow down. And the fact that it chose 0.04 seconds is mysterious to me as well.

So... I still haven't figured it out yet...
 
Old 09-22-2005, 07:06 PM   #4
JCipriani
Member
 
Registered: Aug 2005
Location: Pittsburgh, PA, USA
Distribution: Redhat 9, OS X 10.4.x, Win2K
Posts: 85

Original Poster
Rep: Reputation: 15
Figured it out. The solution:
Code:
int one = 1;
setsockopt(sock, IPPROTO_TCP, TCP_NODELAY, &one, sizeof(int);
The problem had something to do with "delayed ACK" packets. Apparently ACK packets are delayed a few ms in hopes that more packets will arrive with in that time to "coalesce" into a single received packet. The above option turns that behavior off. Unfortunately, the better way to do it is to leave that algorithm on and just change my application to send entire blocks of data with one send() call. Oh well.

Simple, I guess. It took me a while to find that though. I did learn a lot about TCP in the mean time.

Last edited by JCipriani; 09-22-2005 at 07:11 PM.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
send/recv binary files with C++ sockets dafatdude Programming 3 02-12-2006 08:02 AM
how can i send a file using linux sockets?? crapodino Programming 2 10-09-2005 11:17 PM
sockets dont ever send data alaios Programming 1 09-21-2005 06:59 PM
cant send a whole struct with sockets (c code) alaios Programming 6 09-20-2005 06:23 AM
Server does'nt send mail until shutdown and LAN can't send or receive. Wolfy Linux - Networking 0 08-02-2004 07:31 PM

LinuxQuestions.org > Forums > Non-*NIX Forums > Programming

All times are GMT -5. The time now is 08:27 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration