Semaphore-like shell code

omnio · 03-10-2007, 03:43 AM

Hi there,

I'm trying to write a very simple semaphore in a shell script which allows only one instance of a function to be executed at a time. I tried using just a lock file, like this:

Code:

LOCK="somefile"

my_function() {
	while [ -e "${LOCK}" ]
	do
		continue
	done

	# announce my_function started.
	touch "${LOCK}"

	[... do everything that needs to be done ...]

	# announce my_function ended.
	rm "${LOCK}"
}

Theoretically, when an instance of my_function detects another instance of itself running, it waits for it to finish and continue only after that.

This works most of the time, but in some cases (which are rare but still exist), two instances of my_function can check the lock file IN THE SAME TIME, see it doesn't exist, create it and continue executing the rest of the lines. Any idea how to prevent this from happening (in a shell script)?

So far I found this: http://www.unixreview.com/documents/s=9303/sam0408f/ but it looks like a way too complex solution.

Thanks.

unSpawn · 03-10-2007, 09:37 AM

I think you should reorder and check the lock first. Then maybe you could wedge in a small "sleep" or else, how about setting a lock *before* you even reach your function?

Code:

my_function() {
 sleep 2s
 [ -e "${LOCK}" ] && return 127 || touch "${LOCK}"
 while [ -e "${LOCK}" ]; do something; done
 rm "${LOCK}"
}

omnio · 03-10-2007, 03:32 PM

Thank you, it looks like you are always there to help me with your suggestions.

Quote:

Originally Posted by unSpawn

I think you should reorder and check the lock first.

Well this was pretty much what I was doing:

Code:

my_function() {
	while [ -e "${LOCK}" ]
	do
		continue
	done

The above code is quite at the beginning and the loop shouldn't be left as long as there is a (previous) lock.

Quote:

Originally Posted by unSpawn

how about setting a lock *before* you even reach your function?

This seems to be a very interesting approach, I think it's worth taking chances with it, thanks.

Quote:

Originally Posted by unSpawn

Then maybe you could wedge in a small "sleep" or else,

Well I think this doesn't solve this theoretical question, of more instances of my_function being called at the same time, checking for the lock (and detecting none so it's ok to go on) and step on one another's toes. Unfortunatelly this is a practical problem too. I know that at the processor level there is no such thing as "exactly the same time", but it seems that when it comes about the filesystem & shell, there is (maybe due to their latency).

Thanks again; I'll try to mess with locks before reaching the function and see what happens. Best wishes.

tread · 03-10-2007, 07:43 PM

You should use the command lockfile - it's part of the procmail package.

jlinkels · 03-10-2007, 08:08 PM

Interesting approach. However, I think it is not solvable this way.

As long as the touch ($file) function is not atomic, there is always a chance that the two instances create the file at the same time. Having an atomic function is very essential for the concepts of semaphores. Most processors provide a Test And Set instruction which is atomic and cannot be interrupted. Trying to achieve this without atomic operations in a multitasking environment is very difficult.

The author of the link you provide seems to know where he is talking about, at least his explanation of semaphores is correct. (Funny that the original meaning of P and V got lost: I am Dutch, and I have seen so many different explanations for P and V)Without studying his program in-depth, I would not be surprised if such a complicated approach is necessary. Getting something *very* simple to work (like an atomic operation) might be complicated if the elementary building blocks lack.

jlinkels

omnio · 03-11-2007, 01:58 AM

Quote:

Originally Posted by tread

You should use the command lockfile - it's part of the procmail package.

Thanks, I knew about it but I refrained from using it since this would have forced users to have procmail installed on their systems.

omnio · 03-11-2007, 03:19 AM

Quote:

Originally Posted by jlinkels

As long as the touch ($file) function is not atomic, there is always a chance that the two instances create the file at the same time.

Yes, you are perfectly right. Your emphasis on the need of being *atomic* reminded me of an article about using bash's noclobber mode for lock files, something like this:

Code:

if ( set -o noclobber; echo "$$" > "$lockfile") 2> /dev/null; 
then
   [ ... do everything that needs to be done ...] 
   
   rm -f "$lockfile"

else
   echo "Failed to acquire lockfile: $lockfile." 
   echo "Held by $(cat $lockfile)"
fi

The above code seems to be atomic.

The original article is here : http://www.davidpashley.com/articles...l-scripts.html

Thank you for your time ( & let me know if you think that it will fail too). Best wishes.

kshkid · 03-11-2007, 03:23 AM

Quote:

Originally Posted by omnio

Hi there,

I'm trying to write a very simple semaphore in a shell script which allows only one instance of a function to be executed at a time. I tried using just a lock file, like this:

Code:

LOCK="somefile"

my_function() {
	while [ -e "${LOCK}" ]
	do
		continue
	done

	# announce my_function started.
	touch "${LOCK}"

	[... do everything that needs to be done ...]

	# announce my_function ended.
	rm "${LOCK}"
}

This is quite interesting!

And I think I have got a valid concern here!

Code:

	while [ -e "${LOCK}" ]
	do
		continue
	done

What if after the check -e "${LOCK}"
the file is deleted forcefully and in that instant of time another function executes the block assuming that there is NO valid $LOCK, it would create its own $LOCK file , while the former process when returning to the looping part again as

-e "${LOCK}

it would be in a state of nothing had happened, and it would just continue

since we dont bind atomicity to the operations of a while code,

the process wouldnt be aware of such a rapid change of deleting and creating a file instataneously.

What should be concentrated is, notification of the process when a file or status of the file changes ?

Am I right?

Please correct me if am wrong!

omnio · 03-11-2007, 04:01 AM

Quote:

Originally Posted by kshkid

What if after the check -e "${LOCK}"
the file is deleted forcefully and in that instant of time another function executes the block assuming that there is NO valid $LOCK, it would create its own $LOCK file , while the former process when returning to the looping part again as

-e "${LOCK}

it would be in a state of nothing had happened, and it would just continue

Of course, you're right and that went to my mind too, and as workaround I made sure to have another lock: if a "waiting and checking" my_function is detected, the forthcoming functions will just exit instead of waiting too. Also, I had to make sure nothing is passed to my_function (no VAR, for example), so that it won't matter which instance, newer or older, is executed. But obviously - if the noclobber mode will work fine (see the above post) many of these concerns will go away.

Quote:

Originally Posted by kshkid

since we dont bind atomicity to the operations of a while code,

the process wouldnt be aware of such a rapid change of deleting and creating a file instataneously.

What should be concentrated is, notification of the process when a file or status of the file changes ?

Also right, you were probably thinking of something like:

Code:

inotifywait ${LOCK}

But the above also has 2 drawbacks:
1. it will work only with linux kernels >= 2.6.13 (no Solaris or BSD)
2. inotify-tools needs to be installed.

Thanks for your reply.
Cheers.

tread · 03-11-2007, 06:07 AM

Quote:

Originally Posted by jlinkels

Interesting approach. However, I think it is not solvable this way.

As long as the touch ($file) function is not atomic, there is always a chance that the two instances create the file at the same time. Having an atomic function is very essential for the concepts of semaphores. Most processors provide a Test And Set instruction which is atomic and cannot be interrupted. Trying to achieve this without atomic operations in a multitasking environment is very difficult.

The author of the link you provide seems to know where he is talking about, at least his explanation of semaphores is correct. (Funny that the original meaning of P and V got lost: I am Dutch, and I have seen so many different explanations for P and V)Without studying his program in-depth, I would not be surprised if such a complicated approach is necessary. Getting something *very* simple to work (like an atomic operation) might be complicated if the elementary building blocks lack.

jlinkels

I believe symlink creation is atomic. So you could touch a file, then try creating a symbolic link to it - if this succeeds only then do you have the lock. I don't remember where I read this though.

omnio · 03-11-2007, 06:38 AM

Quote:

Originally Posted by tread

I believe symlink creation is atomic. So you could touch a file, then try creating a symbolic link to it - if this succeeds only then do you have the lock. I don't remember where I read this though.

Yes I think it will work too. Just like in the noclobber mode, if destination doesn't exist it will be created, and if does exist "ln" will exit 1. Excellent, thanks.

kshkid · 03-11-2007, 11:12 PM

Is there any reason that symlink creation should be atomic?

What forces the operation to be an atomic one ?

omnio · 03-12-2007, 03:20 AM

Quote:

Originally Posted by kshkid

Is there any reason that symlink creation should be atomic?

What forces the operation to be an atomic one ?

I'm not quite in a position to explain this in depth; I think that it would be ideal if both the check for the lock file and its creation should happen with a single I/O operation on the disk, but I'm not sure this is the case here (or even possible). However, in both cases (noclobber & symlink) they are issued by a single command and it seems quite fast so I'm happy with them both; so far I changed my code to use the noclobber mode and I get no more errors because of lock overlapping, so I'm moving on. Maybe someone else can explain this better.

bigearsbilly · 03-12-2007, 03:36 AM

It's not going to be possible using shell scripts and
not trivial using anything else.

I've been shell scripting every working day for years.
There is no foolproof way around this problem, that I know of.

which is why, happily, they still employ people like me to type rm lockfile
and kill -9 occasionally