programming a linux server daemon

tilman1 · 09-11-2011, 07:46 AM

Hello

I am writing a daemon that is controlling a power controller via a serial link. The power controller basically turns on and off additional hardware, and measures the consumed current.

Up to 5 clients shall be able to connect to that daemon via TCP sockets. The daemon shall process their messages and then assemble control commands for the power controller.

Since the power controller is an exclusive resource, its data needs to be available to all clients. I hence used a non-blocking accept in a while loop. Not surprisingly, this drives up the load -- which makes me think that there are better approaches.

Here some pseudo code to illustrate it:

Code:

...
fork
...
list_fd = listen
while (1)

 if accept(NON_BLOCKING)
   add client to clients_list
 end if

 for all client in clients_list
   data = recv(client,NON_BLOCKING)
   if (exist(data))
    process(client,data)
   end if
 end for

end while

A better approach would probably be to work with blocking system calls and fork a process for every connected tcp socket, i.e. replace the for loop above by fork.

Code:

...
fork
...
list_fd = listen
while (1)
 if accept(BLOCKING)
   (add client to clients_list)
   fork()
   if child
     while(connection_established(client))
       data = recv(client, BLOCKING)
       process(client,data)
   end if
end while

Questions:
a) How do I design the access to the power controller in the process function ? It is an exclusive resource, and the access from client connections to be pipelined and controlled to avoid race conditions and contradicting commands from diffent socket connections.

Is an additional process by forking the right way to go ? How can I ensure that the processes work on global data available to the whole process group rather than accessing local copies of data (I understand, fork fully copies the parent process ) ?

b) Once completed, I would like to run the daemon on an embedded box (arm9) with the usual constraints in resources. Is a bunch of "forks" computationally to expensive to this ? Is there a more light weight approach ? What about pthreads for example ?

c) Is there sample code available for my scenario that I could use as example ?

Thanks
Tilman

ta0kira · 09-11-2011, 08:26 PM

Quote:

Originally Posted by tilman1

Here some pseudo code to illustrate it:

Adding a nanosleep of even .1s to the loop would greatly reduce the load, but I have another solution below.

Quote:

Originally Posted by tilman1

A better approach would probably be to work with blocking system calls and fork a process for every connected tcp socket, i.e. replace the for loop above by fork.

If this is an acceptable solution, you might just write a program that communicates via standard input and output and set up xinetd to listen to the socket for you. That doesn't sound feasible, though, given that the different connections can't function independently.

Quote:

Originally Posted by tilman1

a) How do I design the access to the power controller in the process function ? It is an exclusive resource, and the access from client connections to be pipelined and controlled to avoid race conditions and contradicting commands from different socket connections.

You can set the bound socket and new connections to non-blocking mode, then perform a "read" select on all of them at once. If the bound socket is "read ready" you can accept a new connection. If client sockets are "read ready" you can read a single command from one of them, process that command to completion, then read a command from another client. If you're using stream input that will be a little more complicated.

Here's an outline of what you might do:

Create a socket, bind it, and listen.
Set that socket to O_NONBLOCK with fcntl.
Create a list of int, either in an array or in a linked list. This is to store socket descriptors.
Add the bound socket to the list.
while (1)
1. Fill an fd_set with all descriptors from the list. To start with that will only be the bound socket.
2. select using the fd_set as the "read" set.
3. If the return is -1 and errno==EINTR then continue. If the return is -1 and errno is something else then break.
4. Iterate the list of descriptors, checking the fd_set with FD_ISSET to figure out if the given descriptor is ready.
  1. If the bound descriptor is ready, accept the connection, set it to O_NONBLOCK, and add the new descriptor to the list. If there's an accept error that indicates no more connections can be made then break both loops.
  2. For other descriptors that are ready, read a command, process it, and wait for it to complete, then move onto checking the next descriptor. If read returns 0, or -1 and errno!=EAGAIN, remove the current file descriptor from the list.

If you're using fgets and not read you'll have to keep iterating through step 5.4.2 until you get an empty or error return from each "read ready" file descriptor, and you'd remove the descriptor if feof returns non-zero for the stream.

I'd normally write up an example, but I don't have time right now to write it an test it.
Kevin Barry

tilman1 · 09-17-2011, 03:51 AM

Hi

I wrote myself a prototype for the daemon. Using telnet, one can connect to it, and it will echo everything until disc(onnect) is keyed in. It mostly seems to work.

Surprisingly, it however terminates itself, if telnet session is killed or interrupted.

I am aiming at writing a server and a client that establish a robust connection, i.e. if the connection is interrupted, the server shall wait for an incoming connection while the client shall try to reconnect.
Prerequist of course is a server that does not terminate itself if the connection breaks....

Thanks

Tilman

Code:

#include <sys/types.h>
#include <sys/stat.h>
#include <sys/socket.h>

#include <stdio.h>
#include <stdlib.h>
#include <fcntl.h>
#include <errno.h>
#include <unistd.h>
#include <syslog.h>
#include <string.h>

#include <signal.h>

#include <netdb.h>          /* hostent struct, gethostbyname() */
#include <arpa/inet.h>      /* inet_ntoa() to format IP address */
#include <netinet/in.h>     /* in_addr structure */

#define SUCCESS 0
#define GENERAL_FAILURE 1
#define BACKLOG 2
#define MAX_LINE 40

#define CONNECTED 1
#define NOT_CONNECTED 2
#define TRUE 1
#define FALSE 0

struct sock_desc_t {
	int connected;
	int fd;
	struct sockaddr_in addr;
	socklen_t addrlen;
};

struct sock_desc_t clients[BACKLOG];
void serve_socket(struct sock_desc_t *client);
int open_listen_socket(int port);

int exit_flag = 0;
int daemon_flag = 1;

void signal_handler(sig)
int sig;
{
	switch(sig)
	{
		case SIGHUP:
			syslog(LOG_INFO,"hangup signal catched");
			break;
    	case SIGTERM:
    		exit_flag = 1;
    		syslog(LOG_INFO,"terminate signal catched");
    		break;
    }
}


static int daemonize(void)
{
    pid_t pid, sid;

    /* already a daemon */
    if ( getppid() == 1 ) return(SUCCESS);

    /* Fork off the parent process */
    pid = fork();
    if (pid < 0) {
        return(GENERAL_FAILURE);
    }
    /* If we got a good PID, then we can exit the parent process. */
    if (pid > 0) {
        exit(SUCCESS);
    }

    /* At this point we are executing as the child process */

    /* Change the file mode mask */
    umask(0);

    /* Create a new SID for the child process */
    sid = setsid();
    if (sid < 0) {
        return(GENERAL_FAILURE);
    }

    /* Change the current working directory.  This prevents the current
       directory from being locked; hence not being able to remove it. */
    if ((chdir("/")) < 0) {
        return(GENERAL_FAILURE);
    }

    /* Redirect standard files to /dev/null */
    freopen( "/dev/null", "r", stdin);
    freopen( "/dev/null", "w", stdout);
    freopen( "/dev/null", "w", stderr);
    return(SUCCESS);
}

int open_listen_socket(int port)
{
	int sd;
	struct sockaddr_in addr;    /* internet address */

	// open socket that listens for incoming requests
   sd = socket(PF_INET, SOCK_STREAM, 0);
   bzero(&addr, sizeof(addr));
   addr.sin_family = AF_INET;
   addr.sin_port = htons(port);
   addr.sin_addr.s_addr = INADDR_ANY;

   int nFlags;
   nFlags = fcntl(sd, F_GETFL, 0);
   nFlags |= O_NONBLOCK;
   fcntl(sd,F_SETFD , nFlags);


   if ( bind(sd, (struct sockaddr*)&addr, sizeof(addr)) != 0 )
   {
   	syslog(LOG_ERR,"Could not bind to socket on port %d\n",addr.sin_port);
   	return(-1);
   }
   else if ( listen(sd, BACKLOG) != 0 )
   {
   	syslog(LOG_ERR,"listen");
   	return(-1);
   }
   // set socket to non blocking
   //else if (fcntl(sd, F_SETFL, O_NDELAY) < 0) {
   else if (fcntl(sd, F_SETFL, O_NONBLOCK ) < 0) {
	 perror("Can't set socket to non-blocking");
   	 return(-1);
   }
   return(sd);
}
/*  Read a line from a socket  */

ssize_t Readline(int sockd, void *vptr, size_t maxlen) {
    ssize_t n, rc;
    char    c, *buffer;

    buffer = vptr;

    for ( n = 1; n < maxlen; n++ ) {

	if ( (rc = read(sockd, &c, 1)) == 1 ) {
	    *buffer++ = c;
	    if ( c == '\n' )
		break;
	}
	else if ( rc == 0 ) {
	    if ( n == 1 )
		return 0;
	    else
		break;
	}
	else {
	    if ( errno == EINTR )
		continue;
	    return -1;
	}
    }

    *buffer = 0;
    return n;
}


/*  Write a line to a socket  */

ssize_t Writeline(int sockd, const void *vptr, size_t n) {
    size_t      nleft;
    ssize_t     nwritten;
    const char *buffer;

    buffer = vptr;
    nleft  = n;

    while ( nleft > 0 ) {
	if ( (nwritten = write(sockd, buffer, nleft)) <= 0 ) {
	    if ( errno == EINTR )
		nwritten = 0;
	    else
		return -1;
	}
	nleft  -= nwritten;
	buffer += nwritten;
    }

    return n;
}



void serve_socket(struct sock_desc_t *client)
{
	char  buffer[MAX_LINE];
	/*  Retrieve an input line from the connected socket
		    then simply write it back to the same socket.     */

		Readline(client->fd, buffer, MAX_LINE-1);
		Writeline(client->fd, buffer, strlen(buffer));


		/*  Close the connected socket  */
		if (strstr(buffer,"disc") != NULL)
		{
			client->connected = NOT_CONNECTED;
			if (daemon_flag == 0)
				fprintf(stdout, "disconnecting %d  flag = %d\n",client->fd, client->connected );
			else
				syslog (LOG_DEBUG,"disconnecting %d  flag = %d",client->fd, client->connected );


			if ( close(client->fd) < 0 ) {
				fprintf(stderr, "Error calling close()\n");
				exit(EXIT_FAILURE);
			}

		}

}


int main( int argc, char *argv[] ) {
	char *dev_name = NULL;
	int i,len;
	int port,sd;
	int client;
	int rc;



   // parse command line arguments
	for (i = 0; i < argc; i++)
	{
		// get port number from command line arguments
		if (strcmp(argv[i],"-p") == 0)
		{
			i++;
			if (i<argc)
				port = atoi(argv[i]);

		}
		// daemon mode/foreground
		if (strcmp(argv[i],"-f") == 0)
		{
			daemon_flag = 0;
		}
	}

	// no port number giveb
	if (port == 0)
		exit(1);

    /* Now we are a daemon */
	if (daemon_flag == 1) {
		if (daemonize()!=SUCCESS)
		{
			return(GENERAL_FAILURE);
		}
	}

    // open log facility
    openlog("Sample daemon",LOG_PID, LOG_DAEMON);
    syslog(LOG_INFO,"Daemon started\n");

    // install signal handler
    signal(SIGCHLD,SIG_IGN); /* ignore child */
    signal(SIGTSTP,SIG_IGN); /* ignore tty signals */
    signal(SIGTTOU,SIG_IGN);
    signal(SIGTTIN,SIG_IGN);
    signal(SIGHUP,signal_handler); /* catch hangup signal */
    signal(SIGTERM,signal_handler); /* catch kill signal */


    // init socket structs
    for (client=0;client<BACKLOG;client++)
    {
    	clients[client].fd = -1;
    	clients[client].connected = NOT_CONNECTED;
    }


    struct sockaddr *sockaddr;

    if ((sd = open_listen_socket(port)) > -1)
    { // socket waiting for incoming connection request
    	fd_set rfds;
    	struct timeval tv;

    	while(exit_flag == 0)
    	{
    		// server communication on opened sockets
    		FD_ZERO(&rfds);
    		FD_SET(sd, &rfds);
    		int max_fd = sd;
    		for (client=0;client<BACKLOG;client++)
    		{
    			if(clients[client].connected == CONNECTED)
    			{
    				FD_SET(clients[client].fd, &rfds);
    				if (max_fd < clients[client].fd)
    					max_fd = clients[client].fd;
    			}
    		}
    		max_fd++;


        	/* Wait up to one seconds. */
        	tv.tv_sec = 1;
        	tv.tv_usec = 0;

    		int retval = select(max_fd, &rfds, NULL, NULL, &tv);

    		if (retval)
    		{
    			if (FD_ISSET(sd, &rfds) == TRUE)
    			{
    				if (daemon_flag == 1)
    					syslog (LOG_DEBUG, "connection request ");
    				else
    					fprintf(stdout, "connection request\n ");

    				// check for incomming connection requests
					for (client=0;client<BACKLOG;client++)
					{
						if (clients[client].connected == NOT_CONNECTED)
						{
							sockaddr = (struct sockaddr *)&clients[client].addr;
							socklen_t addrlen = sizeof(clients[client].addr);
							if ((clients[client].fd = accept(sd, sockaddr, &addrlen)) == -1)
							{
								if (!((errno == EAGAIN) | (errno == EWOULDBLOCK )))
								{ 	// non-fatal error masked by if statement above.
									// log fatl error in syslog  and prepare termination
									if (daemon_flag == 1)
										syslog (LOG_ERR,"Could not connect to socket: %s", strerror(errno));
									else
										fprintf(stderr,"Could not connect to socket: %s\n", strerror(errno));
									// terminate
									exit_flag == TRUE;
								}
							}
							else
							{
								clients[client].connected = CONNECTED;
								if (daemon_flag == 1)
								{
									syslog(LOG_DEBUG, "connected (fd=%d, adr=%s, port=%d)\n",
										clients[client].fd,
										inet_ntoa(clients[client].addr.sin_addr),
										ntohs(clients[client].addr.sin_port));
								}
								else
								{
									fprintf(stdout, "connected (fd=%d, adr=%s, port=%d)\n",
											clients[client].fd,
											inet_ntoa(clients[client].addr.sin_addr),
											ntohs(clients[client].addr.sin_port));
								}
							}
						}
						else
						{
							if (daemon_flag == 1)
								syslog (LOG_ERR,"fd storage busy (fd=%d)",clients[client].fd);
							else
								fprintf(stderr,"fd storage busy (fd=%d)\n",clients[client].fd);
						}
					}
    			}


    			// socket to server
    			for (client=0;client<BACKLOG;client++)
    			{
    				if (clients[client].connected == CONNECTED)
    				{
    					if (FD_ISSET(clients[client].fd, &rfds) == TRUE)
    					{
    						if (daemon_flag == 1)
    							syslog (LOG_ERR,"Serving socket %d",clients[client].fd);
    						else
    							fprintf(stdout,"Serving socket %d\n",clients[client].fd);
    							serve_socket(&clients[client]);
    					}
    				}
   				}

    		}
    		else
    		{ // no socket to serve. Update status information
    		//	 syslog(LOG_DEBUG,"no socket to serve");
    		}
    	}
    }

    if (daemon_flag == 1)
    	syslog (LOG_DEBUG, "close sd ");
    else
    	fprintf(stdout, "close sd \n");
    close(sd);
    syslog(LOG_DEBUG,"server daemon terminated");
    closelog();
    return SUCCESS;
}

ta0kira · 09-17-2011, 09:22 AM

Quote:

Originally Posted by tilman1

Surprisingly, it however terminates itself, if telnet session is killed or interrupted.

You should block or handle SIGPIPE. You get that when you write or fflush a pipe or socket that's closed on the other end.

Quote:

Originally Posted by tilman1

Code:

/*  Read a line from a socket  */

ssize_t Readline(int sockd, void *vptr, size_t maxlen) {
    ssize_t n, rc;
    char    c, *buffer;

    buffer = vptr;

    for ( n = 1; n < maxlen; n++ ) {

	if ( (rc = read(sockd, &c, 1)) == 1 ) {
	    *buffer++ = c;
	    if ( c == '\n' )
		break;
	}
	else if ( rc == 0 ) {
	    if ( n == 1 )
		return 0;
	    else
		break;
	}
	else {
	    if ( errno == EINTR )
		continue;
	    return -1;
	}
    }

    *buffer = 0;
    return n;
}

You might also associate a buffer with each client socket and add an index argument to Readline in case there's an EAGAIN or EWOULDBLOCK in the middle of a line. This will let you read a line over multiple calls to Readline. You also have nothing to handle clients disconnecting. You should check for a read of length 0, which indicates EOF, or a non-EAGAIN/EWOULDBLOCK read error. In both cases you need to close the socket and deactivate it in clients.
Kevin Barry

ta0kira · 09-17-2011, 09:41 AM

Quote:

Originally Posted by tilman1

Code:

    			if (FD_ISSET(sd, &rfds) == TRUE)
    			{
    				if (daemon_flag == 1)
    					syslog (LOG_DEBUG, "connection request ");
    				else
    					fprintf(stdout, "connection request\n ");

    				// check for incomming connection requests
					for (client=0;client<BACKLOG;client++)
					{
						if (clients[client].connected == NOT_CONNECTED)
						{
							sockaddr = (struct sockaddr *)&clients[client].addr;
							socklen_t addrlen = sizeof(clients[client].addr);
							if ((clients[client].fd = accept(sd, sockaddr, &addrlen)) == -1)
							{
								if (!((errno == EAGAIN) | (errno == EWOULDBLOCK )))
								{ 	// non-fatal error masked by if statement above.
									// log fatl error in syslog  and prepare termination
									if (daemon_flag == 1)
										syslog (LOG_ERR,"Could not connect to socket: %s", strerror(errno));
									else
										fprintf(stderr,"Could not connect to socket: %s\n", strerror(errno));
									// terminate
									exit_flag == TRUE;
								}
							}
							else
							{
								clients[client].connected = CONNECTED;
								if (daemon_flag == 1)
								{
									syslog(LOG_DEBUG, "connected (fd=%d, adr=%s, port=%d)\n",
										clients[client].fd,
										inet_ntoa(clients[client].addr.sin_addr),
										ntohs(clients[client].addr.sin_port));
								}
								else
								{
									fprintf(stdout, "connected (fd=%d, adr=%s, port=%d)\n",
											clients[client].fd,
											inet_ntoa(clients[client].addr.sin_addr),
											ntohs(clients[client].addr.sin_port));
								}
							}
						}
						else
						{
							if (daemon_flag == 1)
								syslog (LOG_ERR,"fd storage busy (fd=%d)",clients[client].fd);
							else
								fprintf(stderr,"fd storage busy (fd=%d)\n",clients[client].fd);
						}
					}
    			}

This will cause you problems if clients is full and there's a waiting connection because select will no longer block. You have (at least) two options for handling such a situation:

accept the connection unconditionally and close it if there isn't a slot for it. This will cause (essentially) immediate connect failure for the connecting client.
When a new socket results in a full clients, set a global "full" flag. When that flag is set skip inserting sd into rfds. As soon as soon as a slot in clients opens up unset the "full" flag. This will force the new client to wait for an opening, but it might also cause a connect timeout. (If I recall correctly, though, if there's a "listen" slot open on the server then connect might succeed even before accept happens.)

Kevin Barry

tilman1 · 09-25-2011, 04:35 PM

Hi Kevin

First step taken:

I have worked on the server side:
1) I have made the accept blocking and added the file descriptor to the file descriptor list of the select statement.
2) If the list of file descriptors is full, I accept a new incoming connection and then close it right away. That terminates the connection on the client side realiably as well
3) I close a client connection not only if read is -1 but also if read returns 0 (EOF).

The server now no longer terminates, if I kill the client, but resets the file descriptor and waits for a reconnect.

Now I need to make the client side stabile before I can try running it on the arm9.

Thanks

Tilman

sundialsvcs · 09-25-2011, 10:48 PM

Part of the problem with projects like this is that you're starting from scratch as though you were the first person in the entire world to do this... when it is a far better strategy to take full advantage of the fact that you are not.

For example, Perl provides a Net::Server package in its vast CPAN library which can do all of what you're contemplating and much more. (And it's just one of dozens if not hundreds of other viable candidates.)

Basically, if your purpose is to get from point A to point B, you shouldn't be looking for trees to chop down so that you can make lumber to make a chariot to help you carve a new road through the forest. You should be taking the bus.

P.S. Please do not take offense at this comment, nor take it "personally." I am seeking to make a point, but n-o-t at your expense!! Should you for a moment think otherwise, I cordially apologize in advance.

theNbomr · 09-26-2011, 02:31 PM

What sundialsvcs said. General-purpose solutions to our problem already exist. Look into open source control system toolkits and SCADA systems. These provide all of the functionality you need, plus a whole lot that you haven't thought of yet.
--- rod.