Select system call usage

itrilok · 01-27-2013, 01:48 AM

Hi all,

I am trying to write application pgm using select system call. I was creating three server sockets in a single process and each server socket keeps listening on the corresponding port then later added these three sockfd's to the FD_SET for the use of select call such that which sever/client fd is unblocked it will serve the fd .

Ex:
//create three server fd's
// add to FD set
FD_SET (server1, sockfds);
FD_SET (server2, sockfds);
FD_SET (server3, sockfds);

testfds = sockfds;
while (select (FD_SETSIZE, &testfds, 0))
{
loop for all fd's
{
if it is server fd then
{
// accept client
// add clientfd to sockfds
}
else {
// read message from client
}
}
testfds = sockfds;
}

But when I try to connect to the corresponding servers using client pgm, server process is throwing "Bus error". what could be the pbm?

Also, did anyone tried scenario as given below

Say three servers s1,s2, s3 are created in a single process and listening on ports x1,x2,x3. Now the clients should be connected to the servers s1,s2 and s3 on the corresponding ports and the server should serve both accepting client connections and also should serve clients by reading their messages. And the code should be written using select call.

jpollard · 01-27-2013, 07:09 AM

Without the real code, the only thing I see is that the clientfd is never removed from sockfds - which would leave it with fds being closed but still in the list of valid fds. That in turn would cause errors when trying to read from them.

itrilok · 01-27-2013, 11:19 PM

#include <netinet/in.h>
#include <sys/time.h>
#include <sys/ioctl.h>
#include <unistd.h>
#include <stdlib.h>
int main()
{
int server_sockfd, client_sockfd;
int server_len, client_len;
struct sockaddr_in server_address;
struct sockaddr_in client_address;
int result, i = 0, j, maxclients=3;
int clientfd[10] = {0};

fd_set serverfds, clientfds, sfds, cfds;

server_sockfd = socket(AF_INET, SOCK_STREAM, 0);
server_address.sin_family = AF_INET;
server_address.sin_addr.s_addr = inet_addr("10.0.2.15");
server_address.sin_port = htons(9734);
server_len = sizeof(server_address);
bind(server_sockfd, (struct sockaddr *)&server_address, server_len);
if (listen(server_sockfd, 3) < 0)
{
perror("listen");
exit(EXIT_FAILURE);
}

FD_ZERO(&serverfds);
FD_ZERO(&clientfds);
FD_ZERO(&sfds);
FD_ZERO(&cfds);

FD_SET(server_sockfd, &serverfds);

while(1) {
char ch;
int fd;
int nread;

sfds = serverfds;
cfds = clientfds;
sleep (5);
printf("server waiting\n");
result = select(FD_SETSIZE, &sfds, &cfds, NULL, NULL);
if(result < 1) {
perror("select:");
exit(1);
}

for(fd = 0; fd < FD_SETSIZE; fd++) {
if(FD_ISSET(fd,&sfds)) {
if(fd == server_sockfd) {
client_len = sizeof(client_address);
client_sockfd = accept(server_sockfd,
(struct sockaddr *)&client_address, &client_len);
clientfd[i++] = client_sockfd;
FD_SET(client_sockfd, &clientfds);
printf("adding client on fd %d\n", client_sockfd);

}
else if (FD_ISSET(fd, &cfds)) {
for (j=0; j< maxclients; j++)
if (fd == clientfd[j]) {
recv(fd, &ch, 1, 0);
printf("serving client on fd %d\n", clientfd[j]);
ch++;
printf ("%c\n",ch);
}
}
}
}
}
}

Hi here is the complete code ... Thanks in Advance.

jschiwal · 01-27-2013, 11:30 PM

Moved: This thread is more suitable in Programming and has been moved accordingly to help your question get the exposure it deserves.

Also please place your code inside of [ code ] ... [ / code ] brackets to preserve the indentation. Spaces in particular can be difficult to notice.

jpollard · 01-28-2013, 03:48 AM

First a short note on the use of select

-- making the number of fds waiting on FD_SETSIZE works... but is slow.

The usual way of doing this is to use the largest fd you are waiting for + 1.

The usual limit on FD_SETSIZ is 1024, so for every loop (if it only has 1 fd) will be 1024. Where using the actual number would be closer to 4 or 5... It does mean that the actual usage of the fds is a bit more work.

Now some apparent bugs:

-- use of clientfd[i++] is problematical - Each connection increments i, but closing an fd does not decrement. There may only be 3 connections at one time, but repeated connections will overflow the array.

--- It looks like it should work... until a connection closes. The closed socket is not removed from the fd_set so it will alway be presented to the select, even though it is invalid. You should be getting a EBADF error from select.

--- The use of the readfd set and write fdset. Your code does a recv on a client fd... but the client fd is only ready for writing. It isn't in the readfdset to determine if data is available for the recv. Most of the time, the writefdset will always be set when a connection exists.

Right now, I'm not seeing any bus errors or segfaults (program itself doesn't work, but still looking into it - my symptom is that select returns something ready, but is neither in the read set or write set).

What system are you running this on?

jpollard · 01-28-2013, 03:50 AM

I added a few things to the program:

Code:

#include <netinet/in.h>
#include <sys/time.h>
#include <sys/ioctl.h>
#include <unistd.h>
#include <stdlib.h>
#include <stdio.h>
int main()
{
    int server_sockfd, client_sockfd;
    int server_len, client_len;
    struct sockaddr_in server_address;
    struct sockaddr_in client_address;
    int result, i = 0, j, maxclients=3;
    int clientfd[10] = {0};
	    int fdmax = 0;

    fd_set serverfds, clientfds, sfds, cfds;

    server_sockfd = socket(AF_INET, SOCK_STREAM, 0);
    server_address.sin_family = AF_INET;
    server_address.sin_addr.s_addr = inet_addr("127.0.0.1");
    server_address.sin_port = htons(9734);
    server_len = sizeof(server_address);
    bind(server_sockfd, (struct sockaddr *)&server_address, server_len);
    if (listen(server_sockfd, 3) < 0)
    {
	perror("listen");
	exit(EXIT_FAILURE);
    }


    FD_ZERO(&serverfds);
    FD_ZERO(&clientfds);
    FD_ZERO(&sfds);
    FD_ZERO(&cfds);

    FD_SET(server_sockfd, &serverfds);
    fdmax = server_sockfd;	/* the only one so far */

    while(1) {
	char ch;
	int fd;
	int nread;

	sfds = serverfds;
	cfds = clientfds;
	sleep (5);
	printf("server waiting\n");
	result = select(fdmax+1, &sfds, &cfds, NULL, NULL);
	printf("select result=%d\n",result);
	if(result < 1) {
	    perror("select:");
	    exit(1);
	}

	for(fd = 0; fd < fdmax+1; fd++) {
	    if(FD_ISSET(fd,&sfds)) {
		if(fd == server_sockfd) {
printf("server getting client\n");
		    client_len = sizeof(client_address);
		    client_sockfd = accept(server_sockfd,
		    (struct sockaddr *)&client_address, &client_len);
		    clientfd[i++] = client_sockfd;
		    FD_SET(client_sockfd, &clientfds);
		    printf("adding client on fd %d\n", client_sockfd);
		    fdmax = client_sockfd > fdmax ?client_sockfd:fdmax;

		}
	    else if (FD_ISSET(fd, &cfds)) {
printf("client ready\n");
		for (j=0; j< maxclients; j++)
		    if (fd == clientfd[j]) {
printf("processing client\n");
			recv(fd, &ch, 1, 0);
			printf("serving client on fd %d\n", clientfd[j]);
			ch++;
			printf ("%c\n",ch);
		    }
		}
	    }
	}
    }

Just for my testing. The include <stdio.h> was to eliminate a warning about printf not being defined. The other printfs are there to identify some goings on.

jpollard · 01-28-2013, 04:18 AM

Got it.

There seems a bit of an issue with how nesting worked.

Note the bugs are still there. I added braces for all if statements which seems to have allowed the handling to go through.

I was using telent utility for testing, so it used line mode (meaning you have to hit enter before any data is sent. You can set character mode if you want).

The server hangs on the recv while processing a client connection. The write fdset is always ready... and the server isn't waiting for data, so once it enters client handling, it hangs until data is provided.

Things get confused when there are two clients because of this. The first client will hang the server, and it won't matter what the second client does (in fact, the server won't even see the connection until the first client is given data...)

The final code I used has the localhost IP number, and a few additional prints, and a primitive fdmax to reduce the number of iterations:

Code:

#include <netinet/in.h>
#include <sys/time.h>
#include <sys/ioctl.h>
#include <unistd.h>
#include <stdlib.h>
#include <stdio.h>
int main()
{
    int server_sockfd, client_sockfd;
    int server_len, client_len;
    struct sockaddr_in server_address;
    struct sockaddr_in client_address;
    int result, i = 0, j, maxclients=3;
    int clientfd[10] = {0};
            int fdmax = 0;

    fd_set serverfds, clientfds, sfds, cfds;

    server_sockfd = socket(AF_INET, SOCK_STREAM, 0);
    server_address.sin_family = AF_INET;
    server_address.sin_addr.s_addr = inet_addr("127.0.0.1");
    server_address.sin_port = htons(9734);
    server_len = sizeof(server_address);
    bind(server_sockfd, (struct sockaddr *)&server_address, server_len);
    if (listen(server_sockfd, 3) < 0)
    {
        perror("listen");
        exit(EXIT_FAILURE);
    }


    FD_ZERO(&serverfds);
    FD_ZERO(&clientfds);
    FD_ZERO(&sfds);
    FD_ZERO(&cfds);

    FD_SET(server_sockfd, &serverfds);
    fdmax = server_sockfd;      /* the only one so far */
    while(1) {
        char ch;
        int fd;
        int nread;

        sfds = serverfds;
        cfds = clientfds;
        sleep (5);
        printf("server waiting\n");
        result = select(fdmax+1, &sfds, &cfds, NULL, NULL);
        printf("select result=%d\n",result);
        if(result < 1) {
            perror("select:");
            exit(1);
        }

        for(fd = 0; fd < fdmax+1; fd++) {
            if(FD_ISSET(fd,&sfds)) {
                if(fd == server_sockfd) {
printf("server getting client\n");
                    client_len = sizeof(client_address);
                    client_sockfd = accept(server_sockfd,
                    (struct sockaddr *)&client_address, &client_len);
                    clientfd[i++] = client_sockfd;
                    FD_SET(client_sockfd, &clientfds);
                    printf("adding client on fd %d\n", client_sockfd);
                    fdmax = client_sockfd > fdmax ?client_sockfd:fdmax;
                }
            } else if (FD_ISSET(fd, &cfds)) {
printf("client ready\n");
                for (j=0; j< maxclients; j++) {
                    if (fd == clientfd[j]) {
printf("processing client\n");
                        recv(fd, &ch, 1, 0);
                        printf("serving client on fd %d\n", clientfd[j]);
                        ch++;
                        printf ("%c\n",ch);
                    }
                }
            }
        }
    }
}

To get a better example you need to put the client connections in the read fdset so that you know when data is available for reading. For this program, you can assume that you can always write (it is not always true, but things get much more complicated trying to handle multiplexing within a single threaded process).

jpollard · 01-28-2013, 07:36 AM

As a last note, I think the problem was being caused an ambiguous/dangling else...

specifically the part around:

Code:

           if(FD_ISSET(fd,&sfds)) {
                if(fd == server_sockfd) {
...
                }
            } else if (FD_ISSET(fd, &cfds)) {

In your original, you didn't have the braces on the else. This then associated the else part with the if(fd == server_sockfd), which is the wrong association. This meant that the only FD_ISSET tested was on the read socket set, which only had the server_sockfd set... and the client test never happens. Since the client set was always writable, the server just looped testing the listening socket.

itrilok · 01-29-2013, 07:51 PM

Hi pollard,

Thanks a lot for all your inputs.

itrilok · 01-29-2013, 11:33 PM

Hi pollard,

If socket descriptor can be made to non-blocking fd then hang can be prevented I think so.

jpollard · 01-30-2013, 06:22 AM

Yes, it can. And easily too - just use the MSG_DONTWAIT for the flags parameter to recv. If no data is available you will get the EAGAIN or EWOULDBLOCK messages (different systems provide different errors.. either is valid).

But that makes it harder to identify when data is present. You can use the FIONREAD ioctl to determine how many bytes are present too. The FIONREAD is nice because you know how many bytes you have to process, before actually trying to process any - otherwise you have to depend on the return status from the read to say how many actually occurred. I tend to use read rather than recv for stream sockets; there is no other data to be obtained with a stream, as the accept gets the source address. Either one works so it may just be up to personal preference as to which to use.

This is also why network protocols are a pain - you always end up with a state diagram for a connection where the new data + current state determine what the application is going to do. It is also why many servers use threads - creating a thread to process a connection allows the thread to represent the state - and that eliminates having to track it separately.

The state/thread is needed to prevent a single threaded server from starving some connections - usually (for simplicity) you have a processing loop:

Code:

while more input
   read input
   generate buffer
   output buffer

The problem with it is that if the client connection provides data fast enough, no other connection gets processed.
With this simple loop can be subsumed into the select loop, it presents problems when the "more input" is only partial - a packet in transit may get split, so not all of it arrives... With non-blocking I/O you don't know if the entire data is available (based on size of the expected packet). And that means you have to also include repacking operations. With blocking reads, you will get the entire data... but at the expense of not handling other connections that are ready to be processed.

So the simple case is more like (without states..., and only using the read fdset on select):

Code:

loop
   wait for input via select
      or handle errors...
   for each possible stream
      if stream ready
         if server_fd,
            accept connection
            add connection to input fdset
         else
            get input
            if end of file,
               close stream
               remove from input fdset
            else
               generate data buffer for
               output data buffers to stream
               if end of file
                  close stream
                  remove from input fdset
end loop

This is close to what you have...

Now the changes for non-blocking I/O:

Code:

loop
   wait for input via select
      or handle errors...
   for each possible stream
      if stream ready
         if server_fd,
            accept connection
            add connection to input fdset
         else
            get input
            if error, 
               if end of file,
                  close stream
                  remove from input fdset
               else assume EWOULDBLOCK or EAGAIN
            else
               generate data buffer
               output data buffer to stream
               if error,
                  if end of file
                     close stream
                     remove from input fdset
                  else assume EWOULDBLOCK or EAGAIN
end loop

And that is only for a simple processing loop.

NOTE: The above will loose output data if the output cannot be sent (the error handling on the output to the stream).

To prevent this, you end up using the write fdset, and a buffer associated with the stream (don't forget to handle what to do when/if the buffer is full...). And this is part of the state information needed for a stream.

At least no stream will get starved for attention.

Because of the very simple processing (read a byte, increment it, write a byte) I don't expect any data losses. Not impossible, just hard to cause without writing a special client program that deliberately doesn't read the data.