LinuxQuestions.org
Latest LQ Deal: Latest LQ Deals
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Blogs > rtmistler
User Name
Password

Notices


Rate this Entry

Creating a daemon to launch and monitor your processes

Posted 05-30-2013 at 01:11 PM by rtmistler
Updated 12-09-2014 at 02:29 PM by rtmistler

Most of my Linux application architectures are really a collection of processes which are started at boot and need be maintained in case something unexpected occurs. Long ago I started with daemons, but also learned and developed a few additional concepts.

The links have changed over the years, but if you search for "linux daemon" and go down some of the search results, you'll find similar entries as:

http://www.netzmafia.de/skripten/uni...mon-howto.html

Many of these tell you common things to do:

1.) Fork a child
2.) Close the parent
3.) Set up your desired umask
4.) Change your SID
5.) Change the working directory
6.) Close or redirect file handles for stdin, stdout, stderr
7.) Run forever in your loop, because you're the daemon

These guides are great, but what do you next do with this, and why?

You start something and it runs forever; however starting one thing, running it forever eventually ends up being a not very useful program. You will find that you'll need to create some child processes to move your data around and make your application do something. Such things to me are defined as:
- Data ingress/process/egress. Take data from one location, process it either routing it, switching it, or converting it, then sending data out to another location. Complicate that with several possible input or output points and you may have a router, or other type of intelligent network switch.
- Data ingress/process/display. Take real-time data from a sensor, process it in some form, and present that data to a display of some type. Example: A GPS device where you take in GPS data from satellites, process it to determine where you are located, and then draw a representation of your location on a map for display.

What these applications have in common are that they all are better architectures if they are multiple processes or threads, versus a single run-forever process.

So what do we do once we've created our daemon to make an environment more suitable to these applications?

These are the following concepts which I'm going to cover:
- File Limits and Core Dumps
- PIPES
- Creating child processes
- Monitoring child processes
- Restarting child processes, if needed

File Limits and Core Dumps
If you check file limits via the ulimit command, you will see that the shell environment you have, has a variety of file limits. The one that matters most to me is the coredump limits. Typically I find that my daemons end up having zeros for the coredump limit, when started; and therefore I need to change this. If you look at setrlimit(2) you'll see that this and getrlimit(2) are functions which you can use to read and set your file limits. Below is how I set the coredump limits to be "unlimited". Note that if your application crashes and even if it does so in a manner where a core file would be created; if this limit is set to zero, there will be no core file created. Therefore it is useful to set this limit now, before you create any child processes, which will inherit this property from your daemon.

Code:
    struct rlimit rlim = { RLIM_INFINITY, RLIM_INFINITY };

    /* Set core file limits */
    if(setrlimit(RLIMIT_CORE, &rlim) < 0) {
        dae_debug("Error setting core file ulimit: %s\n", strerror(errno));
    }
PIPES
These are one of my most useful tools. They are a form of interprocess communications. Establish one here and it is available to all child processes. Pipes are unidirectional. I create them before launching the child processes using pipe(2) to accomplish this. All pipes established at this point will be visible to my child processes; however you'll see later when I create the child processes and manage them, I limit the exposure for that process by coding convention. For now, let's say I establish 6 pipes, and do this for 3 processes. My example application will have 3 processes; GUI, MotionSensor, LightSensor. The GUI gets data from the motion and light sensors and can also send commands, or request status from the sensors. Say this is part of a gesture detection system, which has motion and light sensors as well as a main controller. The main controller needs to control the sensors, read their versions, and process data from the sensors. Here is how we create the 6 pipes:

Code:
    int fl;
    int pipe2ui[2], pipe2motion[2], pipe2light[2];

    /* The pipe2ui is used for communications from all the sensory */
    /* processes: MOTION and LIGHT to the GUI.                     */
    /* Create the PIPE */
    if(pipe(pipe2ui) == -1) {
        dae_debug("Pipe error\n");
    }
    /* Set the RECEIVE side of the PIPE [0] to be non-blocking */
    fcntl(pipe2ui[0], F_GETFL, &fl);
    fcntl(pipe2ui[0], F_SETFL, (fl | O_NONBLOCK));
    /* Set the TRANSMIT side of the PIPE [1] to be non-blocking */
    fcntl(pipe2ui[1], F_GETFL, &fl);
    fcntl(pipe2ui[1], F_SETFL, (fl | O_NONBLOCK));

    dae_log("pipe2ui[0]: %d\npipe2ui[1]: %d\n", pipe2ui[0], pipe2ui[1]);
Note: dae_debug() and dae_log() are my own macros which write to files.

See that the pipe(2) library function takes in a two element integer array and creates two uni-directional pipes. The [0] entry is the receiving side and the [1] entry is the sending side. As you create the remaining pipes, pipe2motion and pipe2light, the numerical handles will increase. Therefore the handles here may be "0", and "1", the next two handles will be "2", and "3", and finally "4", and "5". The end result is that the ODD numbers will here always be the transmit numbers. And it merely works out that way. If we were to do some form of system library call which took a handle such as a file open(2) handle, then that would take up one number from the system pool of handles. Note also that in the /proc/<your-process-number>/fd directory, you will see these handles appear. You can actually cat and echo to them to cause pipe communications.

A main point here is, if you write "Hi" to descriptor 1, it will be read-able at descriptor 0, it will not be read-able at descriptor 1, that is a write-only descriptor and likewise descriptor 0 is a read-only descriptor.

This is not beneficial until you fork() more child processes, but we'll see later how this is used. The reason why it's not beneficial now is that you are one process, one program. Writing data to anywhere is fine, but you are aware of that data, so using a message transfer mechanism is not really required. Only after you create child processes, will it be relevant because data developed within the scope of a different process will be visible only to that process.

Creating Child Processes
To create your child processes, fork(2) is used as is exec(3) or one of the various alternative execv functions. I create a "spawn_and_run_<name>" function for each intended child, later when I monitor and if I need to restart I recall that spawn_and_run function.

This "spawn_and_run" function is not related to a former system command, named spawn, I just recall that command and typically use that verbiage.

Since the GUI talks to the other sensor processes, it needs the send descriptor for MOTION and LIGHT and it also needs to know it's receive descriptor.

Here's the code and I'll explain inline as well as following it:

Code:
static int spawn_and_run_gui(pid_t *pid, int *rx_fd, int *tx_motion_fd, int *tx_light_fd)
{
    /* NULL terminated argument list for the exec() call */
    char *args[5] = {
        "mygui",
        NULL,
        NULL,
        NULL,
        NULL
    };
    char arg_1[8], arg_2[8], arg_3[8];

    /* Set up the argument list */
    sprintf(arg_1, "%d", rx_fd[0]);         // GUI receive descriptor
    sprintf(arg_2, "%d", tx_motion_fd[1]);  // Transmit descriptor for MOTION process
    sprintf(arg_3, "%d", tx_light_fd[1]);   // Transmit descriptor for LIGHT process

    /* Copy into the local array */
    args[1] = arg_1;
    args[2] = arg_2;
    args[3] = arg_3;

    dae_debug("Launching graphical UI with arguments: %s %s %s %s\n",
              args[0], args[1], args[2], args[3]);

    /* fork() makes an exact copy of "this" process, all resources and file handles */
    if((*pid = fork()) == -1) {
        dae_debug("Error performing fork() of Graphical User Interface child process: %s\n", strerror(errno));
        return 0;
    }

    /* This is the child, the Graphical User Interface application thread and does not normally return */
    if(*pid == 0) {
        /* Run the graphical UI */
        /* Use a fully qualified path name */
        if(execvp("/home/mylogin/mygui", args) == -1) {
            dae_log("Error invoking execv %d:%s\n", errno, strerror(errno));
            return -1;
        }
        return -1;
    }

    /* Else case we are the parent, so continue return normally */
    dae_log("Graphical User Interface Running: %d\n", *pid);
    return 1;
}
As mentioned before, this spawn and run function is a function so it can be re-used. The subtleties of this process are:
- fork() returns -1 in error, which means nothing has happened.
- fork() returns a positive number, which means you are still the parent; hence we record into our passing argument's value, the pid of the created child and return.
- fork() returns 0 when you are the child. Which in my case means I set up the arguments to perform an exec() call which only returns if there's an error; otherwise it will never return.
- In truth, the child process is an exact copy of the parent (daemon) process at the point which the child was created, that's what fork() does. Moving forwards, the independent actions of each process are unique. What we do in the child is to take only the information intended for the child to use and pass that along via the exec() argument list. In the parent, we record the child process ID (pid) and monitor it for signals to see if it has exited.

Here follows the invocation of the spawn_and_run function:

Code:
    pid_t ui_pid;
    int ui_running;

    /* Spawn and run the Graphical User Interface application  */
    ui_running = spawn_and_run_gui(&ui_pid, pipe2ui, pipe2motion, pipe2light);
    if(ui_running < 0) {
        dae_log("Graphical User Interface application did not launch\n");
        return -1;
    }
Monitoring Child Processes
The point of the daemon recording the pid for the created child is so that it can monitor the continued existence of that process. Or a group of processes. I've narrowed my code examples to show mainly the GUI, however MOTION and LIGHT also get spawn_and_run functions and get monitored. After all the spawning is done, the daemon enters it's while(1) loop and iterates forever checking these statuses. I do use nanosleep() in that loop to sleep for 50 mS, which is my arbitrary choice, but seems to work well for me.

Within my forever loop, the daemon uses waitpid() to check, but not block on the status of each process it knows about. It then parses the returned status to determine if the process in question has halted for some reason. Later in my forever loop, the local "running" variables are checked and if they have been reset, the lost processes get re-spawned and run again.

Code:
   int w_status;

    while(1) {
        . . .

        /* GUI */
        /* Get status on the GUI, but don't wait around, just check it */
        if(waitpid(ui_pid, &w_status, WNOHANG) == ui_pid) {
            /* If successfully found, then the GUI terminated, check for why */
            if(WIFSIGNALED(w_status)) {
                dae_log("Graphical User Interface process %d was terminated by signal %d.  Restarting\n", ui_pid, WTERMSIG(w_status));
                if(WCOREDUMP(w_status)) {
                    dae_log("Graphical User Interface process %d generated a core dump.\n", ui_pid);
                    save_core_data();
                }
            }
            else if(WIFEXITED(w_status)) {
                dae_log("Graphical User Interface process %d has self terminated with exit code %d.  Restarting\n", ui_pid, WEXITSTATUS(w_status));
            }
            else {
                dae_log("Graphical User Interface process %d has failed for unknown reasons.  Restarting\n", ui_pid);
            }

            /* Set the flag to cause a later restart */
            ui_running = 0;
        }
        /* This error indicates that there was no signal, but the process in question is also no longer available */
        if(errno == ECHILD) {
            dae_log("Graphical User Interface process %d has terminated for unknown reasons.  Restarting\n", ui_pid);
            ui_running = 0;
        }
Restarting Child Processes
The child processes which have been detected to be failed are restarted by invoking the spawn_and_run function appropriate for the failed child. The PIPES and other resources, as known by the daemon, have not changed. Therefore the arguments to use for the spawn_and_run function calls are no different than before. If a nature of your architecture causes a process failure to corrupt a common system resource, then you would need to correct for that. In my case, this does not happen. Child processes can fail, exit without error, or crash fantastically; in the end it means that the child is gone, so we need to start another. As long as the daemon is intact, and if it's running, doing this monitoring, it is intact; then it can respawn the missing children.

Closing Remarks and What is Missing
To summarize this architecture, I'll reiterate a few points.

1.) A daemon is started, really it's nothing more than "a" user process, but it's made special by some actions performed once it starts.
2.) The daemon allocates and creates a bunch of resources for status and communications, all suitable for the children to use.
3.) The daemon creates all the children and gives each their unique handles for interconnection to each other by organizing a list of passing arguments via exec().
4.) The daemon monitors the children; having created them, it knows of their process IDs and therefore can check their status.

A missing item here is what happens if the daemon crashes? Or can a child fail in such a manner as to affect the operation of the daemon? A script could be written to create and monitor the daemon when the system starts. However it is possible that some highly improbable fault of the daemon could also cause it's controlling script to have a failure as well. The only real protection I've found for these circumstances are protective coding, testing, and understanding of the various states of failures which can occur. I don't use a script to run and monitor the daemon. As with all programming projects, as I encounter more cases of exceptions, I add more protective measures to monitor for them and recover from them, if possible. This architecture is a beginning which is intended to have a primary controller and monitor which helps to allow your application to continue to run properly.

Communications Between Child Processes
In reviewing my draft here; I did say earlier that I would discuss how the PIPES are used by the child processes for communications, therefore in closing I'll cover that concept.

You saw earlier in the GUI example how the exec() call argument list was set up specifically for the GUI:

[0] GUI program, [1] GUI receive handle, [2] MOTION transmit handle, [3] LIGHT transmit handle

The GUI child needs to know it's receive PIPE handle. Say that is "0", then the other side of that PIPE would be "1", because when the pipe is constructed using pipe(2), you get the two sides of it together. This does not mean that the GUI is supposed to then conclude "I can transmit to myself by taking my receive handle and adding one.", what the GUI is supposed to do is listen for data from that handle. The data can be any form, but as a programmer you ought to have decided upon a messaging structure and use that structure when you communicate from one process to another. A very common structure is to use ASCII control characters as start of message and end of message, for instance CR is 0x0d and could mean start of message. LF is 0x0a and could mean end of message. No other ASCII control characters should be used, instead all message contents should be printable ASCII. Otherwise if you desire to use a binary protocol, then framing, and checksums; or even CRC's should be employed.

The GUI also knows the transmit PIPE handles for both the MOTION and LIGHT child processes. Why? This is how the GUI sends data TO those processes. And likewise, the MOTION process has the GUI transmit PIPE handle, as the LIGHT process also has the GUI transmit PIPE handle.

Linux will not allow the two processes to interleave data unintentionally onto the PIPE. All sent by either MOTION or LIGHT will get to GUI intact. If fragments are sent by either, then fragments will be received. But swapping out of processes, interrupts, or other context swaps will not cause fragmentation of your data to the PIPE, provided all these processes are still the same priority. If you modify your scheduling, and do so differently for each of these three processes, then there will be problems. But if you do nothing to your scheduling, then they all will have the same policies for scheduling, because they all were created from the same daemon, and therefore they cannot interrupt each other in a destructive manner.

Further, the receive PIPE at the GUI is unique to the GUI process. This is seen in the /proc file system. There are other copies of the receive PIPE for the GUI, for instance within the MOTION and LIGHT processes; however the intentions here are that those processes do not look at that side of the PIPE. For instance, why would you transmit to a PIPE and then look at the receive side? Sure, you can validate what you just sent and that gets boring after you've tested that it works. Another thing you can do is have MOTION look at GUI's receiver and check to see if LIGHT has sent anything to GUI. Yes, that's technically allowed here, but against my conventions. My conventions are that when created, you are given the specific receive handles and specific transmit handles which you are to use, no more or less. Therefore if you needed to look at a receive handle which contained your outbound data as well as other processes outbound data, then you'd be given that descriptor at your start. That is messy because it forces you to parse and ignore your own outgoing data. Instead if I need to do that, I establish a different PIPE intended to foment communications between the other two processes. I.e. if MOTION and LIGHT need to talk directly, I make sure that they have each other's descriptor.

Example architecture 1:
GUI receives from MOTION and LIGHT
GUI occasionally sends commands to MOTION and LIGHT

Implemented via:
GUI launched with it's receive descriptor, and MOTION and LIGHT transmit descriptors
MOTION launched with it's receive descriptor, and GUI transmit descriptor
LIGHT launched with it's receive descriptor, and GUI transmit descriptor

Example architecture 2:
GUI receives from MOTION and LIGHT
GUI occasionally sends commands to MOTION and LIGHT
MOTION sends data to LOG (a fourth process)

Implemented via:
GUI launched with it's receive descriptor, and MOTION and LIGHT transmit descriptors
MOTION launched with it's receive descriptor, GUI transmit descriptor, and LOG transmit descriptor
LIGHT launched with it's receive descriptor, and GUI transmit descriptor
LOG launched with it's receive descriptor

LOG only does one thing, places all data given to it into a log file on disk. Since disk access can have delays in the system calls, this process is created to not block the operations of the MOTION process but still get the data to the disk file. In this architecture, nobody else talks to, or knows about LOG except MOTION, and the daemon.

For discussion about how to process received data in a fast manner, using select(), see my other blog entry on this subject:

http://www.linuxquestions.org/questi...-use-it-35530/
Views 4972 Comments 0
« Prev     Main     Next »
Total Comments 0

Comments

 

  



All times are GMT -5. The time now is 11:18 AM.

Main Menu
Advertisement
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration