Debian 8 - Restart closed/stopped Python script process

matrixebiz · 09-01-2017, 08:33 AM

Hello, I am running Debian 8 Jessie and I also run a Python script in a loop but sometimes it crashes and Python stops so I am wondering what I can use to monitor the running Python script and if stopped and not running anymore, the system will restart it. Thank you

rtmistler · 09-01-2017, 01:27 PM

Haven't really used them, I'm more familiar with services which I think are older, but this sounds exactly like what a cron job would do for you. It would run your script and if your script ended or failed, the structure of how you created the cron job would allow you to tell it to auto-restart, and in either case, for instance if it terminated normally or in error, you may wish to have that job rescheduled perpetually. This is what I understand cron to be for.

matrixebiz · 09-01-2017, 02:32 PM

Hello, thanks for responding.
I can create a cron job but I'm only familiar with creating them to run a job every so many minutes, how do I set it to monitor it upon failure as I have the Python script looping indefinitely? Thanks

rtmistler · 09-01-2017, 02:37 PM

That's what I don't know. Services you configure it for auto-restart. I've not done cron. Perhaps someone monitoring may have something to add.

Just me googling, I found this:

https://stackoverflow.com/questions/...ron-12am-daily

One of the top answers gives some direct guidance, but unsure if this works for you if you modify for your case.

matrixebiz · 09-01-2017, 02:49 PM

Actually, I was just thinking and my script just loops every 90 seconds so technically I can remove the loop from the Python script and have the cron job run the script every 90 seconds. What cron settings would I use to run it every 90 seconds? can I do this? : 1.5 * * * * /usr/bin/python script.py

EDIT: looks like I can't do 90 seconds so I would have to run it twice, every 3 minutes but have one of them sleep for 90 seconds;

*/3 * * * * /usr/bin/python script.py
*/3 * * * * sleep 90;/usr/bin/python script.py

matrixebiz · 09-01-2017, 06:16 PM

Can I do this?
* * * * * sleep 100;/usr/bin/python script.py

So basically it will be told to run every minute but it will sleep 100 seconds then once the sleep is over, the minute will be over-do so will run immediately again then sleep again for 100 seconds and so on...

Sefyir · 09-02-2017, 09:09 AM

I believe cron runs once per minute.
If that's not frequent enough, I wouldn't suggest doing tricks like sleeping as that seems likely to break.
I'm pretty sure systemd is able to handle this but I've yet to find any clear documentation on how to do this (maybe others can provide this?)

Quote:

I also run a Python script in a loop but sometimes it crashes and Python stops

It might be wiser to address why it is crashing in the first place. Any errors? memory issues?

I will sometimes do this however:

Code:

while true;
do
    /my/script.py
done

But that makes it hard (impossible) to stop it for good until removing that from cron and rebooting.

So I will also do the below. That way if I do touch /tmp/delete_if_i_exist and kill the python script, it won't restart

Code:

while [ ! -f /tmp/delete_if_i_exist ];
do
    /my/script.py
done

This can easily be done in crontab

Code:

# /etc/cron
@reboot while true; do /my/script.py; done
@reboot while [ ! -f /tmp/delete_if_i_exist ]; do /my/script.py; done

matrixebiz · 09-06-2017, 07:18 AM

Thanks for the information, I'll try and figure out what that means

Do I only do this in Crontab? ;
@reboot while true; do /my/script.py; done
@reboot while [ ! -f /tmp/delete_if_i_exist ]; do /my/script.py; done

I don't want the computer to reboot though

jlinkels · 09-06-2017, 07:39 AM

Basically there are 2 options if the script stops:

The Pyhton process is killed and has disappeared
The Python process is still running but it doesn't do anything ('hung')

In the first case your detection is to run a "ps ax | grep name-of-your-script".
The second case is more difficult. The way I do it is to have my script to touch a file somewhere in /var/run and check the timestamp. In other applications I write logging data to a database. So I include the time stamp when I write and check the time stamp.

The next step is to create a start/stop script, like those used in /etc/init.d. In this start/stop script you do the check as described above and you avoid starting the script twice. So it is safe to start this script as many times as you want. It will however only create a new instance of the script if it is not running yet. It is an option to include you script in the scripts started for the runlevel at boot.

The you can call this script every 1-2 minutes from cron. I have set up these kind of mechanisms for years for real-time processes.

Real-time does not necessarily mean fast, but it means that there is interaction with the real world and the time is relevant. It is nice to try and find why the script stops, but that is not alway possible. If external events play a role, like network connections, remote database access, subprocesses which may not end it is possible that a script or programs gets stuck or crashes. For example if you pull the plug of a mounted drive, or this drive had a power failure. It might be possible to make your script bomb-proof. But it is easier to do the reasonable effort, and restart your script when stuck or terminated.

Oh, and don't forget to send yourself an e-mail when the script restarted. You want to know this. Or at the very least, log the event.

jlinkels