LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices

Reply
 
Search this Thread
Old 11-08-2006, 09:43 AM   #1
Dave Lerner
Member
 
Registered: May 2005
Location: Virginia, USA
Distribution: MEPIS, Ubuntu
Posts: 39

Rep: Reputation: 16
Preventing multiple instances of a shell script from running concurrently


This is a method, for use within a shell script, that prevents multiple instances of a task from running concurrently by using a lockfile containing the process ID. I based it on equivalent code from the package rsnapshot, which is a Perl script.

Is there a simpler way of doing this, using an existing Linux command or
package?

Code:
#!/bin/sh
LOCKFILE='/var/run/example.pid'

# check for existing lockfile
if [ -e "$LOCKFILE" ]; then
# lockfile exists
   if [ ! -r "$LOCKFILE" ]; then
      echo error: lockfile is not readable
      exit 1
   fi
   PID=`cat "$LOCKFILE"`
   kill -0 "$PID" 2>/dev/null
   if [ $? == 0 ]; then
      echo error: existing instance of this task is already running
      exit 1
   fi
# process that created lockfile is no longer running - delete lockfile
   rm -f "$LOCKFILE"
   if [ $? != 0 ]; then
# error: failed to delete lockfile
      exit 1
   fi
fi

# create lockfile
echo $$ >"$LOCKFILE"
if [ $? != 0 ]; then
# error: failed to create lockfile
   exit 1
fi

# -----
echo 'Simulated processing.'
# -----

# delete lockfile
rm -f "$LOCKFILE"
if [ $? != 0 ]; then
# error: failed to delete lockfile
   exit 1
fi

Last edited by Dave Lerner; 11-08-2006 at 09:44 AM.
 
Old 11-08-2006, 10:43 AM   #2
bigearsbilly
Senior Member
 
Registered: Mar 2004
Location: england
Distribution: FreeBSD, Debian, Mint, Puppy
Posts: 3,284

Rep: Reputation: 172Reputation: 172
There really isn't a simple way to do it.
I've been trying on and off for years.

here's my latest version

mind you it's designed for ksh not bash

Code:
#!/no/such/shell
# you must 'dot' this file into a script you want to lock
# as in '. lockme.ksh'
# and it must be a korn shell script
# ==================================

LOCKME_NAME=${0##*/}
export LOCK_FILE=/tmp/$LOCKME_NAME.lock


if [ -f "$LOCK_FILE" -a -z "${LOCKME_ON=}"  ]
then
    >&2 cat <<EOF

    cannot run $0

    lockfile $LOCK_FILE found

EOF
    exit 99
fi


if [ -z "${LOCKME_ON=}" ]
then
 export LOCKME_ON=yes
 # echo LOCKME_ON=$LOCKME_ON
 date > $LOCK_FILE
 chmod 666 $LOCK_FILE
 # echo doing run LOCKME=$LOCKME_ON
 set +u
 $0 $@ 
 exec rm $LOCK_FILE
fi
 
Old 11-08-2006, 04:18 PM   #3
Dave Lerner
Member
 
Registered: May 2005
Location: Virginia, USA
Distribution: MEPIS, Ubuntu
Posts: 39

Original Poster
Rep: Reputation: 16
Thanks. I'll have to spend some time on that to understand it.

Meanwhile, I've simplified my version a bit by using logical operators instead of if-then structures, and moved the code into functions:

Code:
#!/bin/sh
LOCKFILE='example.pid'

source 'lockfile.lib'
add_lockfile "$LOCKFILE"

# -----
echo 'Simulated processing.'
# -----

remove_lockfile "$LOCKFILE"
lockfile.lib

Code:
#**
# Add lockfile.
#
# If the lockfile exists, try to read it (and stop if we can't) to get a PID,
# then see if that PID exists.  If it does, stop; otherwise assume it's
# a stale lock and remove it.
#
# Then create a new lockfile.
#
# @param string $1  Path to lockfile
#**
function add_lockfile {

# check for existing lockfile
	if [ -e "$1" ]; then
# lockfile exists - ensure that it's readable
		[ -r "$1" ] || { echo error: lockfile is not readable; exit 1; }
# ensure that process that created lockfile is no longer running
		kill -0 "`cat "$1"`" 2>/dev/null && { echo error: an instance of this script is already running; exit 1; }
# delete lockfile
		rm -f "$1" || { echo error: failed to delete lockfile; exit 1; }
	fi

# create lockfile
	echo $$ >"$1" || { echo error: failed to create lockfile; exit 1; }
}

#**
# Remove lockfile.
#
# @param string $1  Path to lockfile
#**
function remove_lockfile {
	rm -f "$1" || { echo warning: failed to delete lockfile; exit 1; }
}
 
Old 11-08-2006, 04:37 PM   #4
unSpawn
Moderator
 
Registered: May 2001
Posts: 27,019
Blog Entries: 54

Rep: Reputation: 2766Reputation: 2766Reputation: 2766Reputation: 2766Reputation: 2766Reputation: 2766Reputation: 2766Reputation: 2766Reputation: 2766Reputation: 2766Reputation: 2766
Code:
#!/bin/sh
IAM=(`pgrep -d " " -f ${0//*\//}`)
[ ${#IAM[@]} -gt 1 ] && { echo I AM running; exit 127; } \
|| echo "I AM not running. Running now ;-p"
exit 0
This is shorter if "prevent multiple instances running" is the only criterium.
Note it doesn't check if $$ equals one of ${IAM[@]} (pgrep -nvf).
Using a PID file may be "better" and more compatible. At least it's "standard".
IMHO using a lock file (file w/o PID contents) should be avoided.

Last edited by unSpawn; 11-08-2006 at 04:40 PM.
 
Old 11-08-2006, 04:54 PM   #5
Dave Lerner
Member
 
Registered: May 2005
Location: Virginia, USA
Distribution: MEPIS, Ubuntu
Posts: 39

Original Poster
Rep: Reputation: 16
Thanks, that's an interesting approach. I didn't know about the pgrep command.
 
Old 11-08-2006, 06:21 PM   #6
taylor_venable
Member
 
Registered: Jun 2005
Location: Indiana, USA
Distribution: OpenBSD, Ubuntu
Posts: 892

Rep: Reputation: 40
This is interesting. Just my here: it strikes me that using a lock file would be a bad idea, because the process that created it could be killed before it gets a chance to delete the file. Even if you put the creator's PID inside it, and use that as protection (i.e. new processes can spawn if lock file is in place, but no process with the creator PID is running), it could still fail if a new process happened to get that same PID. Best to use the full invocation path, like unSpawn suggests.
 
Old 11-08-2006, 07:16 PM   #7
Dave Lerner
Member
 
Registered: May 2005
Location: Virginia, USA
Distribution: MEPIS, Ubuntu
Posts: 39

Original Poster
Rep: Reputation: 16
"it could still fail if a new process happened to get that same PID"

Is that possible? My impression was that process IDs don't get reused (until rebooting), but I could be wrong.

Another factor is whether the /var/run directory gets cleared on boot. I could check the behavior by experimenting, but I don't know what the standard behavior is.

Last edited by Dave Lerner; 11-08-2006 at 07:19 PM.
 
Old 11-09-2006, 05:25 AM   #8
bigearsbilly
Senior Member
 
Registered: Mar 2004
Location: england
Distribution: FreeBSD, Debian, Mint, Puppy
Posts: 3,284

Rep: Reputation: 172Reputation: 172
the trouble with pgrep is (on solaris at least)
it has a limit to the searchable command line length,
i.e. long script names aren't revealed fully. so there is
a potential for weird bugs.
(tried that route myself decided against it)

Quote:
it strikes me that using a lock file would be a bad idea, because the process that created it could be killed before it gets a chance to delete the file.
as they say: "sh*t happens"
you can't guard against everything and shell programming has limitations -
you just have to work round them as best you can.

My posted method works but is not perfect, however it is simple.
Though you get apparently multiple process in your 'ps' as the script calls itself - but I live with it.
 
Old 11-09-2006, 07:12 AM   #9
unSpawn
Moderator
 
Registered: May 2001
Posts: 27,019
Blog Entries: 54

Rep: Reputation: 2766Reputation: 2766Reputation: 2766Reputation: 2766Reputation: 2766Reputation: 2766Reputation: 2766Reputation: 2766Reputation: 2766Reputation: 2766Reputation: 2766
Best to use the full invocation path, like unSpawn suggests.
No, I actually chopped off the path. GNU/Linux pgrep does allow "-f" with wildcards though.


Is that possible? My impression was that process IDs don't get reused (until rebooting), but I could be wrong.
PID numbers increment and I am pretty sure PID space is exhaustable (but large, say 32K). So the theoretical chance the script gets the same PID gets greater as the process consumes more of the PID space. Concurrent runs (fork bomb) could help speed up exhaustion though before you reach the limit you most likely run into the OOM killer :-] (I know GRSecurity has features that add/enhance randomness in a few area's and PIDs are one of them.)


as they say: "sh*t happens"
you can't guard against everything and shell programming has limitations -
you just have to work round them as best you can.

With all due respect but there's a difference between saying "shit happens" and taking precautions like using PIDs, and saying "shit happens" and resorting to a lock file. There is no reasoning against using PIDs being a standard: daemons use it, SysV init scripts and process checkers depend on it. Anyone who remembers crashing say Netscape also remembers the "stale lock" messages on restart and experienced it was an inaccurate method.
 
Old 11-09-2006, 07:27 AM   #10
bigearsbilly
Senior Member
 
Registered: Mar 2004
Location: england
Distribution: FreeBSD, Debian, Mint, Puppy
Posts: 3,284

Rep: Reputation: 172Reputation: 172
The simple fact is there are limits to what you can do with shell scripts.

I've been trying for years to find a neat solution to this
and I don't think there is one. A solution where I don't have to
continually cut and paste in to each script. Where I can keep the script clean.
Which is what I am saying.

Also a solution that works well with the calling script when it has
user traps and things like set -o nounset

You can either use a simple lockfile solution or a convoluted
process solution.

I opt for the former as I am lazy and it's simple
and effective.
So occasionally you get a dead lock file, no big deal if it tells you about it.
Just do a manual check with ps and delete it if neccessary.
 
Old 11-09-2006, 09:18 AM   #11
Dave Lerner
Member
 
Registered: May 2005
Location: Virginia, USA
Distribution: MEPIS, Ubuntu
Posts: 39

Original Poster
Rep: Reputation: 16
It sounds like the method I posted is reasonably fault-tolerant.

The only case in which it would fail is if (1) the script exits abnormally without deleting the lockfile and (2) there happens to be an unrelated process matching the PID saved in the PID file. (I'll call it a PID file instead of a lock file to avoid confusion.)

My guess is that the probability of both of those conditions occurring is very small.

And even if both conditions occur, the result is the same as the "dead lock file" situation.

I suppose I could refine the method further by using the PID in the PID file to get the path for that process from /proc/PID/cmdline, and seeing if that matches the script's path.

I don't completely understand the code you guys have posted here, so I could be missing something.

---

In answer to a comment I posted earlier, the article http://www.pathname.com/fhs/pub/fhs-...MEVARIABLEDATA states that the files in /var/run get cleared on boot.

I did some googling on process IDs, and found some articles talking about extending them from 16 bits to 32 bits. But the articles were old, and I couldn't tell if that change had been effected in recent kernels, or even if such a change were planned.

Since Linux is designed to be run for extended periods without rebooting, and new processes get created frequently, I'm surprised that a 16-bit PID limit doesn't get hit a lot.

Last edited by Dave Lerner; 11-09-2006 at 09:31 AM.
 
Old 11-09-2006, 03:21 PM   #12
taylor_venable
Member
 
Registered: Jun 2005
Location: Indiana, USA
Distribution: OpenBSD, Ubuntu
Posts: 892

Rep: Reputation: 40
Quote:
Originally Posted by unSpawn
No, I actually chopped off the path. GNU/Linux pgrep does allow "-f" with wildcards though.
Oops, sorry about that. I hadn't seen pgrep either until you showed it. I was still in awe of its neatness.

Quote:
Originally Posted by unSpawn
PID numbers increment and I am pretty sure PID space is exhaustable (but large, say 32K). So the theoretical chance the script gets the same PID gets greater as the process consumes more of the PID space. Concurrent runs (fork bomb) could help speed up exhaustion <...>
I've kept my workstation up for about a month, and when building software packages from source, I know I've exhausted my PIDs probably through several cycles.

I'm not sure random PIDs would help the distance to probable repetition; in fact they'd probably make it worse (since incremental PIDs ensure that you won't repeat for the longest possible time). OpenBSD does have them turned on by default for security reasons, though; I suppose it can be pretty useful. The other BSDs have such a feature, but its usually disabled in the provided builds. As for Linux, I'm not sure, but I don't see why it wouldn't have such a feature.

Bah, now I've wandered off topic. But I guess the important thing is that it works for you, Dave.

Also, I'd think that 2^16 (65536) currently running processes would be plenty, unless you a absolutely vast amount of resources (maybe in a huge cluster?) Especially when you consider that concurrent programs will probably present themselves as a single process to the operating system (with user-level threads like in Ada [Tasks], Java [Threads], Erlang [Processes], etc). Or maybe I'm completely wrong here; I'm no expert, to be sure.
 
Old 01-03-2007, 09:27 PM   #13
rnturn
Member
 
Registered: Jan 2003
Location: Illinois (Chicago area)
Distribution: Red Hat (8.0), SuSE (10.x, 11.x, 12.2), Solaris (8-10), Tru64
Posts: 914

Rep: Reputation: 45
Quote:
Originally Posted by taylor_venable
This is interesting. Just my here: it strikes me that using a lock file would be a bad idea, because the process that created it could be killed before it gets a chance to delete the file.
A way around that would be to code a trap handler function in the script to catch the common signals that could be used to kill it and perform cleanup of any scratch files, etc., as well as the lock file before finally exiting. The script itself can signal itself (instead of merely issuing an "exit") at the end to ensure that all the cleanup is done.

Quote:
Even if you put the creator's PID inside it, and use that as protection (i.e. new processes can spawn if lock file is in place, but no process with the creator PID is running), it could still fail if a new process happened to get that same PID. Best to use the full invocation path, like unSpawn suggests.
Whew! I would never have considered that use of the lock file. I saw it's use as "If this exists, someone's already running this script. Go away until later."
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
shell script - while loop with multiple conditions ronsha Programming 13 12-10-2005 04:08 PM
Preventing multiple instances of MPlayer (GUI) MrPotato Linux - Software 1 08-22-2005 04:02 PM
Why are multiple instances of mysqld running? philpq Linux - Newbie 2 02-17-2005 03:01 PM
Adding multiple user shell script plexus Programming 2 06-19-2004 08:36 PM
preventing multiple process instances the_dayi Programming 2 07-14-2003 05:57 AM


All times are GMT -5. The time now is 09:51 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration