LinuxQuestions.org - [SOLVED] Some help needed with my init script

- Programming (https://www.linuxquestions.org/questions/programming-9/)

- - Some help needed with my init script (https://www.linuxquestions.org/questions/programming-9/some-help-needed-with-my-init-script-943220/)

Some help needed with my init script

I've got a script which continually pings two IP addresses, in the event of either failing, a piece of software is restarted and details are logged. This is meant to run as a background process and only allow single instances of the ping one liner'. Specifically this does not work - can someone help me troubleshoot - thanks.

Code:

ROUTER="10.161.1.1 10.161.1.2"

#RESTART_COMMAND=`/etc/init.d/opennhrp restart`

LOG=/var/log/messages

PING_STATE=0

start() {

                PING_ON=`pgrep -lf /bin/ping | wc -l`

                if [ $PING_ON -eq 1 ]; then

                        echo "Already running, run restart to restart the process."

                        exit 1

                fi

                while [ $PING_ON -eq 0 ]

                do

                        for IP in $ROUTER;

                                do

                                        nohup /bin/ping -c 10 $IP &> /dev/null &

                                        if [ $? -ne 0 ]; then

                                                let PING_STATE=PING_STATE+1

                                                FAILED_SERVER=$IP

                                        fi

                                done

                                        if [ $PING_STATE -ge 2 ]; then

                                                echo `date "+%b  %e %T"` `hostname -s` "Tunnel to $FAILED_SERVER went down, opennhrp restarting." >> $LOG

                                                /etc/init.d/opennhrp restart

                                                PING_STATE=0

                                        fi

                        pkill -f /bin/ping

                done

}



stop () {

        STOP=0

        for i in `pgrep -lf /bin/ping | awk '{print $1}'`

                do

                        kill -9 $i > /dev/null 2>&1

                        pkill -f /bin/ping > /dev/null 2>&1

                        stop

                        let STOP=STOP+1

                        if [ $STOP -eq 1 ]; then

                                echo Stoppping ping requests...

                        fi

                done

        return 0

}



case "$1" in

        start)

        start

        ;;



        stop)

        stop

        ;;



        restart|reload|force-reload)

        stop

        start

        ;;



        *)

        echo "Usage: /etc/init.d/pingcheck {start|stop|reload|force-reload|restart}"

        exit 1

esac



exit 0

What are the symptoms of "does not work"?

When I exit the script running from ssh, the ping stops. The behaviour is not the same when run via init; not sure if this is a suitable way to run it...

There are two issues I can see:

You do the pinging on the background.

Why on earth would you do that? The proper way to do heartbeat pinging is by doing only one, with a timeout:

Code:

ping -c 1 -w seconds host

The output from the command is not important, as it will return nonzero/failure exit status if the host does not respond within five seconds. Typically, you'd simply run that command in a loop, sleeping a few seconds between checks. A simple if ! ping -c 1 -w seconds host &>/dev/null ; then ... fi is enough to catch the case when the host does not answer to a ping. (You might wish to have your script first escalate to "alert" mode, perhaps doing more pings, before concluding it is not accessible.)
You may not be detaching the daemon from your session.

Whenever a login session (be it console or SSH or X11) ends, a TERM signal is sent to the entire session, killing all processes still alive in the session.

To properly daemonize a command, I recommend using

Code:

( setsid command args.. </dev/null >/dev/null 2>/dev/null & )

Because the command is in parentheses, it is run in a subshell. Because of the ampersand, the command is run in the background of the subshell. That alone is enough to keep it running, as the backgrounded command is reparented to init when the subshell exits. Certain signals are delivered via the tty; redirecting standard input, output and error to /dev/null severs that connection. Finally, setsid starts a new session for the command to run in. In all, this completely severs the connection between the starting shell and the command itself.

The ping actually keeps the link up. I'll give your suggestions a try though and tell you if it still works, thanks :)

Quote:

Originally Posted by genderbender (Post 4670274)

The ping actually keeps the link up.

That does not matter at all. There is nothing wrong in pinging at a regular interval; I only mean that running the ping command on the background is the wrong way to do it.

Like I wrote above, your script should simply regularly call ping -c 1 -w seconds host in a loop, not keep pinging on the background. Whenever the command fails, restart opennhrp, then re-start the pinging loop.

The optimal delay (sleep in the loop) depends on what you consider a failure -- a single ping might get lost now and then, so maybe three or four successive failures constitute a downed link? --, and the maximum interval between pings when keeping the connection alive. Remember that in your script, the true maximum ping interval is the ping timeout PLUS the sleep duration.

Do you want to see a skeleton Bash script I'd personally start with?

OK, it seems to instantly terminate when I run this and then enter a loop; output is as follows (infinitely):

Code:

/etc/init.d/pingcheck: line 5: 16133 Terminated              setsid nohup /bin/ping -c 1 -w 5 $IP < /dev/null > /dev/null 2> /dev/null

/etc/init.d/pingcheck: line 5: 16136 Terminated              setsid nohup /bin/ping -c 1 -w 5 $IP < /dev/null > /dev/null 2> /dev/null

etc

Obviously if I run this in parentheses no output is displayed...

Found potentially another problem, if the init script is started while opennhrp is being restarted (no pings running) then it will start an additional instance. I'm interested in your skeleton script, although don't want to have to rework my entire script... The basics are very similar, ping one host, then ping another if both fail restart something and try again after a short delay.

First, you need to separate your init script and the actual daemon.

Assume you have /etc/pingcheck.conf with

Code:

# This is pingcheck configuration file,

# sourced by Bash.



# Hosts pinged.

ENDPOINTS="10.161.1.1 10.161.1.2"



# Command to run when ALL endpoints become unreachable.

COMMAND="/sbin/service opennhrp restart"



# Minimum interval between pings

INTERVAL=30



# Ping timeout (maximum ping interval = INTERVAL+TIMEOUT)

TIMEOUT=5



# Syslog facility and tag.

FACILITY=user

LOGTAG=pingcheck



# PID file location.

PIDFILE=/var/run/pingcheck.pid

The daemon could then be (untested!) say /usr/local/bin/pingcheck:

Code:

#!/bin/bash



# This script can only be run by root.

if [ "$(/usr/bin/id -u)" != "0" ]; then

    echo "This script can only be run by root." >&2

    exit 1

fi



# Source the configuration file.

. /etc/pingcheck.conf || exit $?



# Some of the commands yield unused output; discard it all.

exec </dev/null >/dev/null 2>/dev/null



while [ 1 ]; do



    ok=0

    count=0

    for IP in $ENDPOINTS ; do

        ping -c 1 -w $TIMEOUT $IP &>/dev/null && ok=$[ok + 1]

        count=$[count + 1]

    done



    # All pings successful?

    if [ $ok -eq $count ]; then

        sleep $INTERVAL

        continue

    fi



    # All pings failed?

    if [ $ok -eq 0 ]; then

        if $COMMAND ; then

            logger -i -p $FACILITY.notice -t $LOGTAG "$[count-ok] of $count endpoints unreachable; restart successful"

        else

            logger -i -p $FACILITY.notice -t $LOGTAG "$[count-ok] of $count endpoints unreachable; restart failed"

        fi

    fi



    # Sleep until next round.

    sleep $INTERVAL

done

This version uses logger to log to syslog, but you can obviously do whatever you prefer.

The "all pings successful" if clause is actually not needed. I included it to make it easier for you to edit or add conditions to suit your needs; this way the success case is explicitly handled correctly.

In the init script, use ps -o pid= -C pingcheck to get the PID of the running daemon. It will return nothing if the daemon is not running. To terminate the script, just send it a TERM signal; i.e.

Code:

pid=$(ps -o pid= -C pingcheck)

[ -n "$pid" ] && kill -TERM $pid

To test the script, use for example

Code:

sudo sh -c '( setsid /usr/local/bin/pingcheck </dev/null >/dev/null 2>/dev/null & )'

In the init script, you can start the daemon using

Code:

( setsid /usr/local/bin/pingcheck </dev/null >/dev/null 2>/dev/null & )

Wow - thanks for your help :) I'll give it a shot.

Quote:

Originally Posted by genderbender (Post 4670428)

I'll give it a shot.

Okay, just let us know your results. (I should be able to fix any obvious problems in the script, if you find any.)

The init script needed to start, restart, stop, or query the status of the daemon is very simple, very similar to your original init script. If the daemon script works for you, then we can easily edit your init script to control the daemon script instead.

This worked a treat, big thanks for your help.