Application to listen at multiple ports

taabish · 12-23-2010, 03:56 AM

Hello Friends:
This is my first post at the forum. I am using a C application which connects to various other C applications at different ports through TCP,sends requests to all of them and then waits for the responses. The problem is that when the requests are fewer it gives a good response but with the increase in load the port at which the master application is listening get hogs and is unable to receive the responses.
Can you please suggest the best way to tackle this issue? Is it possible for the master application to listen at multiple ports so that the load gets distributed and how?

eSelix · 12-23-2010, 04:20 AM

Distribution network load on different ports will not give you anything. "Port" is only virtual term. All ports are bound to exactly one input on your network card. For this purpose you need many network cards. But first you should optimize your application. About how many requests you say, that it blocks your connection? You said that your application sends requests on different ports - you known, you have only 65536 ports available on your system?

taabish · 12-23-2010, 04:40 AM

Let me explain in detail:
I have a master application(say for example which listens on port 9090) and which connects to 15 other applications as for now. Now when all the 15 applications sends response simultaneously to my master application the port of master application gets hog and I need to take restart for the same. After restarting my master application it works fine.
Please note that all the application are running on the same Linux Server. A maximum of 5-6 lacks request is received per day.

eSelix · 12-23-2010, 03:18 PM

Only 15 applications? It is very small load. And if you use TCP then all packets should be delivered, your system take care of this. Look at web servers, they can receive thousands connections simultaneously with one network card on one port. You definitely have some bug in your application. The best will be to paste your code for server and clients, if you can.

theNbomr · 12-23-2010, 03:43 PM

Either your application is poorly coded, or the volume of data &/or processing is high, relative to the CPU speed. 15 simultaneous connections is not a very high number. Web servers, mail servers, and other such application routinely handle far more connections than that.

--- rod.

taabish · 12-23-2010, 10:37 PM

This is the thread where master application waits for the responses:
void pp_mc_server_thread(void *dummy)
{

fd_set fd_readset;
int result;
pp_req_resp_t *p_rc_resp;

debug_print("pp_mc_server_thread: Started");

while(g_app_status)
{

FD_ZERO(&fd_readset);

FD_SET((unsigned int) g_mc_server_sockfd, &fd_readset);

debug_print("pp_mc_server_thread: Waiting For Connection Request From Apps");

result = select(FD_SETSIZE, &fd_readset, NULL, NULL, NULL);

if(result <= 0)
{
err_print("pp_mc_server_thread: Select failure in Server Socket, Exiting!!!");
g_app_status = 0;
break;
}

debug_print("pp_mc_server_thread: Connection Request From Apps");

//Check for data on Master Client Server Socket Fd
if (FD_ISSET (g_mc_server_sockfd, &fd_readset))
{
g_mc_conn_sockfd = pp_accept (g_mc_server_sockfd);
debug_print("pp_mc_server_thread: Received [%s] with pp_accept",strerror(errno));
if (g_mc_conn_sockfd < 0)
{
err_print("pp_mc_server_thread: Accept Failed");
continue;
}

debug_print("pp_mc_server_thread: Connected To Apps");
p_rc_resp = (pp_req_resp_t *)malloc(sizeof(pp_req_resp_t));
memset(p_rc_resp,0,sizeof(pp_req_resp_t));
sleep(1);
if (pp_read(g_mc_conn_sockfd, (char *) p_rc_resp, sizeof(pp_req_resp_t)) <= 0)
{

err_print("pp_mc_server_thread: Error in reading Req/Resp From Connection for %s", p_rc_resp->req_id);
pp_disconnect(g_mc_conn_sockfd);
free(p_rc_resp);
}
else
{
//Process Apps Response
pp_proc_rc_resp(p_rc_resp);
}
pp_disconnect(g_mc_conn_sockfd);
}

}
debug_print("pp_mc_server_thread: Exited");

return;

}

And this his how all the clients send the response to the master client:
int pp_send_resp_to_mc(pp_req_resp_t *req)
{
int result;
pp_req_resp_t resp_ack;
struct timeval timeout;
fd_set fd_readset;

//Create connection with Master Client Server
g_mc_conn_sockfd = pp_connect(g_mc_server_port, g_mc_ip_addr);

if (g_mc_conn_sockfd < 0)
{
err_print("pp_send_resp_to_mc: Unable to Open Connection with Server");
if(req!=NULL)
{
debug_print("\n pp_send_resp_to_mc: 3 req = %d",req);
free(req);
req=NULL;
}
return PP_FALSE;
}
debug_print("pp_send_resp_to_mc: Connected To Server");

//Send response to Server
errno = 0;
usleep(1000);
result = pp_write (g_mc_conn_sockfd, (char *) req, sizeof(pp_req_resp_t));
debug_print("Received [%s] with pp_write for SOID %s and the result is %d",strerror(errno),req->req_id,result);
if (result < sizeof(pp_req_resp_t) )
{
err_print("pp_send_resp_to_mc: Failed to Send Response to Server for %s", req->req_id);
pp_disconnect(g_mc_conn_sockfd);
g_mc_conn_sockfd = -1;
if(req!=NULL)
{
debug_print("\n pp_send_resp_to_mc: 4 req = %d",req);
free(req);
}
return PP_FALSE;
}
debug_print("pp_send_resp_to_mc: Send Response for %s", req->req_id);

//Wait for response from Server
FD_ZERO(&fd_readset);

memset(&timeout, 0, sizeof(struct timeval));
timeout.tv_sec = PP_MAX_ACK_TIME;
FD_SET((unsigned int) g_mc_conn_sockfd, &fd_readset);
errno = 0;
result = select(FD_SETSIZE, &fd_readset, NULL, NULL, &timeout);
if (result <= 0 || (FD_ISSET(g_mc_conn_sockfd, &fd_readset) == 0))
{
debug_print("Select result is less than zero %d for SOID %s",result,req->req_id);
//Timer Expired
pp_disconnect(g_mc_conn_sockfd);
g_mc_conn_sockfd = -1;
if(req!=NULL)
{
free(req);
req=NULL;
}

return PP_FALSE;
}

//Read Ack for the request
memset(&resp_ack, 0, sizeof(pp_req_resp_t));
usleep(10000);
if (pp_read(g_mc_conn_sockfd, (char *) &resp_ack, sizeof(pp_req_resp_t)) <= 0)
{
usleep(10000);
if (pp_read(g_mc_conn_sockfd, (char *) &resp_ack, sizeof(pp_req_resp_t)) <= 0)
{
debug_print("Received [%s] with pp_read for SOID %s",strerror(errno),req->req_id);

}
err_print("pp_send_resp_to_mc: Error in Reading Ack from Connection for %s",req->req_id);
pp_disconnect(g_mc_conn_sockfd);
g_mc_conn_sockfd = -1;
if(req!=NULL)
{
free(req);
req=NULL;
}
return PP_FALSE;
}
err_print("pp_send_resp_to_mc: Ack Received for %s and the result is %d",req->req_id,result);
pp_disconnect(g_mc_conn_sockfd);
g_mc_conn_sockfd = -1;
if(req!=NULL)
{
free(req);
req=NULL;
}
return PP_TRUE;
}

Please suggest.

theNbomr · 12-24-2010, 08:51 AM

What is the purpose of all the sleep() and usleep() call? These could explain why the throughput is being limited.

Your code is very difficult to follow. It would be easier if it was posted in [code ][/code] tags to preserve formatting (did you not notice that everyone else does that?), and if the pieces were separated according to client vs. server components. You do understand the distinction between a client and a server, in the context of socket oriented programming, right? Your comments in the code suggest that you may not, and are confusing:

Quote:

clients send the response to the master client

There are servers and there are clients. Servers listen & accept connections, and clients connect. Either side can send or receive on a stream oriented (TCP) connection. Adding language like 'master client' may make sense to you in the context of your application, but most readers will not understand that language.

--- rod.

dwhitney67 · 12-24-2010, 11:37 AM

It seems that the server is only servicing one client at a time. Depending on the depth of the listen-socket backlog, certain clients may not be successfully connecting to the server. The server is held up awaiting data from each and every client; this is perhaps the wrong approach. The server should only service it's listen socket; allow another process or thread to service the client(s) that connect.

As for the code, I agree with theNbomr that it is very hard to read. One thing that bugs me is the lack of modularization of the code.

Consider something similar to the following (which btw, contains some pseudo-code):
Server:

Code:

int main(...)
{
   int listen_sd = socket(...);
   ...
   bind(listen_sd, ...);

   listen(listen_sd, 5);

   while (!done)
   {
      int client_sd = accept(listen_sd, 0, 0);

      if (client_sd >= 0)
      {
         pid_t pid = fork();

         if (pid == 0)
         {
            handleClient(client_sd);
            exit(0);
         }
      }
   }

   close(listen_sd);
   return 0;
}


void handleClient(int sd)
{
   fd_set save_sd;
   FD_ZERO(&save_sd);
   FD_SET(sd, &save_sd);

   while (1)
   {
      fd_set read_sd = save_sd;
      int    sel     = select(sd + 1, &read_sd, 0, 0, 0);

      if (sel > 0)
      {
         int bytes_read = recv(sd, ...);

         if (bytes_read < 0)
         {
            /* error detected */

            /* is it non-fatal? */
            if (errno == EAGAIN)
              continue;

            /* fatal; get out of while-loop */
            break;
         }
         else if (bytes_read == 0)
         {
            /* client disconnected */
            break;
         }
         else
         {
            /* got data */
         }
      }
   }

   close(sd);
}