check and restart (if dead) a daemonized process
 

I have been using Unison for a year or two to replicate our file server between remote locations (both running CentOS) in near-real time, and it works extremely well.


I have also set up a cron job to make sure that a daemonized version of Unison is running and to restart it, if it's not.

Unfortunately, my script doesn't seem to work consistently.

I have placed the following script in /etc/init.d, so that Unison starts upon machine startup:
unisond:

#! /bin/bash
#
# unisond Start/Stop Unison.
#
# Script created by: Alden Levy, Engine No. 9, Inc.
#
/usr/local/sbin/daemonize -a -e /var/log/unison/error.log -o /var/log/unison/unison.log -p /var/run/unison.pid -v /usr/bin/unison


And in /etc/cron.hourly, I created
check_unison:
#! /bin/bash
#
# check_unison Check to see if Unison is running, and restart it, if it's not.
#
# Script created by: Alden Levy, Engine No. 9, Inc.
#
export upid=$(pidof unison)
export ustorepid=$(cat /var/run/unison.pid)
if [ "$upid" != "$ustorepid" ] || [ -z "$upid" ] #if the PID of unison != stored PID of unison then
#unison is not running, or was started incorrectly
if [ -n "$upid" ]; then
kill "$upid" #if unison is running, it wasn't running correctly, so kill it
fi
if [ -f /var/run/unison.pid ]
then
rm -f /var/run/unison.pid #remove stored PID of unison
fi
/etc/init.d/unison & #start unison
pidof unison > /var/run/unison.pid #store PID of unison
exit 1 #exit with code 1 to show that unison was restarted
else
echo "unison is running" #otherwise, we're cool, so tell us and exit
exit 0
fi
exit 0


My scripts run and create the pid file. Permissions are 755 and owned by root:root. If I run check_unison, it works okay, but every few months, I see that Unison has stopped, and it won't restart without my intervention.

Any help you can provide would be greatly appreciated.

Thanks!

Personally I'd put a short sleep (10 seconds) between these 2 lines:

/etc/init.d/unison & #start unison
pidof unison > /var/run/unison.pid #store PID of unison

because it may not start successfully, ie it may start, but then fail almost immediately. Because you're not waiting at all, you may catch that initial start, so think its ok, but it could fail a few seconds later.
Also, some progs take a few seconds to startup, especially if the system is busy.
If it doesn't already log everything to eg /var/log/messages, at startup
/etc/init.d/unison
the I'd add

/etc/init.d/unison >/var/log/unison.log 2>&1 &

or similar.

Thanks. I'll give it a try and report back.

Not to marginalize your own work at this endeavor, but check out daemontools if you'd like to try an alternative to your homegrown solution.

http://cr.yp.to/daemontools.html

It's funny, but I remember looking at this a year or two ago, but I can't remember why I didn't use it.

I'll give it a try. Thanks for the idea.