[SOLVED] Some help needed with my init script

genderbender · 05-04-2012, 03:25 AM

I've got a script which continually pings two IP addresses, in the event of either failing, a piece of software is restarted and details are logged. This is meant to run as a background process and only allow single instances of the ping one liner'. Specifically this does not work - can someone help me troubleshoot - thanks.

Code:

ROUTER="10.161.1.1 10.161.1.2"
#RESTART_COMMAND=`/etc/init.d/opennhrp restart`
LOG=/var/log/messages
PING_STATE=0
start() {
                PING_ON=`pgrep -lf /bin/ping | wc -l`
                if [ $PING_ON -eq 1 ]; then
                        echo "Already running, run restart to restart the process."
                        exit 1
                fi
                while [ $PING_ON -eq 0 ]
                do
                        for IP in $ROUTER;
                                do
                                        nohup /bin/ping -c 10 $IP &> /dev/null &
                                        if [ $? -ne 0 ]; then
                                                let PING_STATE=PING_STATE+1
                                                FAILED_SERVER=$IP
                                        fi
                                done
                                        if [ $PING_STATE -ge 2 ]; then
                                                echo `date "+%b  %e %T"` `hostname -s` "Tunnel to $FAILED_SERVER went down, opennhrp restarting." >> $LOG
                                                /etc/init.d/opennhrp restart
                                                PING_STATE=0
                                        fi
                        pkill -f /bin/ping
                done
}

stop () {
        STOP=0
        for i in `pgrep -lf /bin/ping | awk '{print $1}'`
                do
                        kill -9 $i > /dev/null 2>&1
                        pkill -f /bin/ping > /dev/null 2>&1
                        stop
                        let STOP=STOP+1
                        if [ $STOP -eq 1 ]; then
                                echo Stoppping ping requests...
                        fi
                done
        return 0
}

case "$1" in
        start)
        start
        ;;

        stop)
        stop
        ;;

        restart|reload|force-reload)
        stop
        start
        ;;

        *)
        echo "Usage: /etc/init.d/pingcheck {start|stop|reload|force-reload|restart}"
        exit 1
esac

exit 0

catkin · 05-04-2012, 06:12 AM

What are the symptoms of "does not work"?

genderbender · 05-04-2012, 06:16 AM

When I exit the script running from ssh, the ping stops. The behaviour is not the same when run via init; not sure if this is a suitable way to run it...

Nominal Animal · 05-04-2012, 07:37 AM

There are two issues I can see:

You do the pinging on the background.

Why on earth would you do that? The proper way to do heartbeat pinging is by doing only one, with a timeout:
Code:
```
ping -c 1 -w seconds host
```
The output from the command is not important, as it will return nonzero/failure exit status if the host does not respond within five seconds. Typically, you'd simply run that command in a loop, sleeping a few seconds between checks. A simple if ! ping -c 1 -w seconds host &>/dev/null ; then ... fi is enough to catch the case when the host does not answer to a ping. (You might wish to have your script first escalate to "alert" mode, perhaps doing more pings, before concluding it is not accessible.)
You may not be detaching the daemon from your session.

Whenever a login session (be it console or SSH or X11) ends, a TERM signal is sent to the entire session, killing all processes still alive in the session.

To properly daemonize a command, I recommend using
Code:
```
( setsid command args.. </dev/null >/dev/null 2>/dev/null & )
```
Because the command is in parentheses, it is run in a subshell. Because of the ampersand, the command is run in the background of the subshell. That alone is enough to keep it running, as the backgrounded command is reparented to init when the subshell exits. Certain signals are delivered via the tty; redirecting standard input, output and error to /dev/null severs that connection. Finally, setsid starts a new session for the command to run in. In all, this completely severs the connection between the starting shell and the command itself.

genderbender · 05-04-2012, 08:00 AM

The ping actually keeps the link up. I'll give your suggestions a try though and tell you if it still works, thanks

Nominal Animal · 05-04-2012, 08:09 AM

Quote:

Originally Posted by genderbender

The ping actually keeps the link up.

That does not matter at all. There is nothing wrong in pinging at a regular interval; I only mean that running the ping command on the background is the wrong way to do it.

Like I wrote above, your script should simply regularly call ping -c 1 -w seconds host in a loop, not keep pinging on the background. Whenever the command fails, restart opennhrp, then re-start the pinging loop.

The optimal delay (sleep in the loop) depends on what you consider a failure -- a single ping might get lost now and then, so maybe three or four successive failures constitute a downed link? --, and the maximum interval between pings when keeping the connection alive. Remember that in your script, the true maximum ping interval is the ping timeout PLUS the sleep duration.

Do you want to see a skeleton Bash script I'd personally start with?

genderbender · 05-04-2012, 08:16 AM

OK, it seems to instantly terminate when I run this and then enter a loop; output is as follows (infinitely):

Code:

/etc/init.d/pingcheck: line 5: 16133 Terminated              setsid nohup /bin/ping -c 1 -w 5 $IP < /dev/null > /dev/null 2> /dev/null
/etc/init.d/pingcheck: line 5: 16136 Terminated              setsid nohup /bin/ping -c 1 -w 5 $IP < /dev/null > /dev/null 2> /dev/null
etc

Obviously if I run this in parentheses no output is displayed...

genderbender · 05-04-2012, 08:55 AM

Found potentially another problem, if the init script is started while opennhrp is being restarted (no pings running) then it will start an additional instance. I'm interested in your skeleton script, although don't want to have to rework my entire script... The basics are very similar, ping one host, then ping another if both fail restart something and try again after a short delay.

Nominal Animal · 05-04-2012, 09:35 AM

First, you need to separate your init script and the actual daemon.

Assume you have /etc/pingcheck.conf with

Code:

# This is pingcheck configuration file,
# sourced by Bash.

# Hosts pinged.
ENDPOINTS="10.161.1.1 10.161.1.2"

# Command to run when ALL endpoints become unreachable.
COMMAND="/sbin/service opennhrp restart"

# Minimum interval between pings
INTERVAL=30

# Ping timeout (maximum ping interval = INTERVAL+TIMEOUT)
TIMEOUT=5

# Syslog facility and tag.
FACILITY=user
LOGTAG=pingcheck

# PID file location.
PIDFILE=/var/run/pingcheck.pid

The daemon could then be (untested!) say /usr/local/bin/pingcheck:

Code:

#!/bin/bash

# This script can only be run by root.
if [ "$(/usr/bin/id -u)" != "0" ]; then
    echo "This script can only be run by root." >&2
    exit 1
fi

# Source the configuration file.
. /etc/pingcheck.conf || exit $?

# Some of the commands yield unused output; discard it all.
exec </dev/null >/dev/null 2>/dev/null

while [ 1 ]; do

    ok=0
    count=0
    for IP in $ENDPOINTS ; do
        ping -c 1 -w $TIMEOUT $IP &>/dev/null && ok=$[ok + 1]
        count=$[count + 1]
    done

    # All pings successful?
    if [ $ok -eq $count ]; then
        sleep $INTERVAL
        continue
    fi

    # All pings failed?
    if [ $ok -eq 0 ]; then
        if $COMMAND ; then
            logger -i -p $FACILITY.notice -t $LOGTAG "$[count-ok] of $count endpoints unreachable; restart successful"
        else
            logger -i -p $FACILITY.notice -t $LOGTAG "$[count-ok] of $count endpoints unreachable; restart failed"
        fi
    fi

    # Sleep until next round.
    sleep $INTERVAL
done

This version uses logger to log to syslog, but you can obviously do whatever you prefer.

The "all pings successful" if clause is actually not needed. I included it to make it easier for you to edit or add conditions to suit your needs; this way the success case is explicitly handled correctly.

In the init script, use ps -o pid= -C pingcheck to get the PID of the running daemon. It will return nothing if the daemon is not running. To terminate the script, just send it a TERM signal; i.e.

Code:

pid=$(ps -o pid= -C pingcheck)
[ -n "$pid" ] && kill -TERM $pid

To test the script, use for example

Code:

sudo sh -c '( setsid /usr/local/bin/pingcheck </dev/null >/dev/null 2>/dev/null & )'

In the init script, you can start the daemon using

Code:

( setsid /usr/local/bin/pingcheck </dev/null >/dev/null 2>/dev/null & )

genderbender · 05-04-2012, 11:13 AM

Wow - thanks for your help

I'll give it a shot.

Nominal Animal · 05-04-2012, 11:38 AM

Quote:

Originally Posted by genderbender

I'll give it a shot.

Okay, just let us know your results. (I should be able to fix any obvious problems in the script, if you find any.)

The init script needed to start, restart, stop, or query the status of the daemon is very simple, very similar to your original init script. If the daemon script works for you, then we can easily edit your init script to control the daemon script instead.

genderbender · 05-08-2012, 04:17 AM

This worked a treat, big thanks for your help.