LinuxQuestions.org
Help answer threads with 0 replies.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices

Reply
 
Search this Thread
Old 03-10-2007, 03:43 AM   #1
omnio
Member
 
Registered: Feb 2007
Location: $HOME
Distribution: Hardened Gentoo
Posts: 66
Blog Entries: 1

Rep: Reputation: 16
Semaphore-like shell code


Hi there,

I'm trying to write a very simple semaphore in a shell script which allows only one instance of a function to be executed at a time. I tried using just a lock file, like this:

Code:
LOCK="somefile"

my_function() {
	while [ -e "${LOCK}" ]
	do
		continue
	done

	# announce my_function started.
	touch "${LOCK}"

	[... do everything that needs to be done ...]

	# announce my_function ended.
	rm "${LOCK}"
}
Theoretically, when an instance of my_function detects another instance of itself running, it waits for it to finish and continue only after that.

This works most of the time, but in some cases (which are rare but still exist), two instances of my_function can check the lock file IN THE SAME TIME, see it doesn't exist, create it and continue executing the rest of the lines. Any idea how to prevent this from happening (in a shell script)?

So far I found this: http://www.unixreview.com/documents/s=9303/sam0408f/ but it looks like a way too complex solution.

Thanks.
 
Old 03-10-2007, 09:37 AM   #2
unSpawn
Moderator
 
Registered: May 2001
Posts: 27,470
Blog Entries: 54

Rep: Reputation: 2901Reputation: 2901Reputation: 2901Reputation: 2901Reputation: 2901Reputation: 2901Reputation: 2901Reputation: 2901Reputation: 2901Reputation: 2901Reputation: 2901
I think you should reorder and check the lock first. Then maybe you could wedge in a small "sleep" or else, how about setting a lock *before* you even reach your function?
Code:
my_function() {
 sleep 2s
 [ -e "${LOCK}" ] && return 127 || touch "${LOCK}"
 while [ -e "${LOCK}" ]; do something; done
 rm "${LOCK}"
}
 
Old 03-10-2007, 03:32 PM   #3
omnio
Member
 
Registered: Feb 2007
Location: $HOME
Distribution: Hardened Gentoo
Posts: 66
Blog Entries: 1

Original Poster
Rep: Reputation: 16
Thank you, it looks like you are always there to help me with your suggestions.

Quote:
Originally Posted by unSpawn
I think you should reorder and check the lock first.
Well this was pretty much what I was doing:
Code:
my_function() {
	while [ -e "${LOCK}" ]
	do
		continue
	done
The above code is quite at the beginning and the loop shouldn't be left as long as there is a (previous) lock.

Quote:
Originally Posted by unSpawn
how about setting a lock *before* you even reach your function?
This seems to be a very interesting approach, I think it's worth taking chances with it, thanks.

Quote:
Originally Posted by unSpawn
Then maybe you could wedge in a small "sleep" or else,
Well I think this doesn't solve this theoretical question, of more instances of my_function being called at the same time, checking for the lock (and detecting none so it's ok to go on) and step on one another's toes. Unfortunatelly this is a practical problem too. I know that at the processor level there is no such thing as "exactly the same time", but it seems that when it comes about the filesystem & shell, there is (maybe due to their latency).

Thanks again; I'll try to mess with locks before reaching the function and see what happens. Best wishes.
 
Old 03-10-2007, 07:43 PM   #4
tread
LQ Newbie
 
Registered: Feb 2007
Posts: 29

Rep: Reputation: 15
You should use the command lockfile - it's part of the procmail package.
 
Old 03-10-2007, 08:08 PM   #5
jlinkels
Senior Member
 
Registered: Oct 2003
Location: Bonaire
Distribution: Debian Lenny/Squeeze/Wheezy/Sid
Posts: 4,166

Rep: Reputation: 506Reputation: 506Reputation: 506Reputation: 506Reputation: 506Reputation: 506
Interesting approach. However, I think it is not solvable this way.

As long as the touch ($file) function is not atomic, there is always a chance that the two instances create the file at the same time. Having an atomic function is very essential for the concepts of semaphores. Most processors provide a Test And Set instruction which is atomic and cannot be interrupted. Trying to achieve this without atomic operations in a multitasking environment is very difficult.

The author of the link you provide seems to know where he is talking about, at least his explanation of semaphores is correct. (Funny that the original meaning of P and V got lost: I am Dutch, and I have seen so many different explanations for P and V)Without studying his program in-depth, I would not be surprised if such a complicated approach is necessary. Getting something *very* simple to work (like an atomic operation) might be complicated if the elementary building blocks lack.

jlinkels
 
Old 03-11-2007, 01:58 AM   #6
omnio
Member
 
Registered: Feb 2007
Location: $HOME
Distribution: Hardened Gentoo
Posts: 66
Blog Entries: 1

Original Poster
Rep: Reputation: 16
Quote:
Originally Posted by tread
You should use the command lockfile - it's part of the procmail package.
Thanks, I knew about it but I refrained from using it since this would have forced users to have procmail installed on their systems.
 
Old 03-11-2007, 03:19 AM   #7
omnio
Member
 
Registered: Feb 2007
Location: $HOME
Distribution: Hardened Gentoo
Posts: 66
Blog Entries: 1

Original Poster
Rep: Reputation: 16
Quote:
Originally Posted by jlinkels
As long as the touch ($file) function is not atomic, there is always a chance that the two instances create the file at the same time.
Yes, you are perfectly right. Your emphasis on the need of being *atomic* reminded me of an article about using bash's noclobber mode for lock files, something like this:

Code:
if ( set -o noclobber; echo "$$" > "$lockfile") 2> /dev/null; 
then
   [ ... do everything that needs to be done ...] 
   
   rm -f "$lockfile"

else
   echo "Failed to acquire lockfile: $lockfile." 
   echo "Held by $(cat $lockfile)"
fi
The above code seems to be atomic.

The original article is here : http://www.davidpashley.com/articles...l-scripts.html

Thank you for your time ( & let me know if you think that it will fail too). Best wishes.

Last edited by omnio; 03-11-2007 at 05:04 AM.
 
Old 03-11-2007, 03:23 AM   #8
kshkid
Member
 
Registered: Dec 2005
Distribution: RHEL3, FC3
Posts: 383

Rep: Reputation: 30
Quote:
Originally Posted by omnio
Hi there,

I'm trying to write a very simple semaphore in a shell script which allows only one instance of a function to be executed at a time. I tried using just a lock file, like this:

Code:
LOCK="somefile"

my_function() {
	while [ -e "${LOCK}" ]
	do
		continue
	done

	# announce my_function started.
	touch "${LOCK}"

	[... do everything that needs to be done ...]

	# announce my_function ended.
	rm "${LOCK}"
}
This is quite interesting!

And I think I have got a valid concern here!

Code:
	while [ -e "${LOCK}" ]
	do
		continue
	done
What if after the check -e "${LOCK}"
the file is deleted forcefully and in that instant of time another function executes the block assuming that there is NO valid $LOCK, it would create its own $LOCK file , while the former process when returning to the looping part again as

-e "${LOCK}

it would be in a state of nothing had happened, and it would just continue

since we dont bind atomicity to the operations of a while code,

the process wouldnt be aware of such a rapid change of deleting and creating a file instataneously.

What should be concentrated is, notification of the process when a file or status of the file changes ?

Am I right?

Please correct me if am wrong!

 
Old 03-11-2007, 04:01 AM   #9
omnio
Member
 
Registered: Feb 2007
Location: $HOME
Distribution: Hardened Gentoo
Posts: 66
Blog Entries: 1

Original Poster
Rep: Reputation: 16
Quote:
Originally Posted by kshkid
What if after the check -e "${LOCK}"
the file is deleted forcefully and in that instant of time another function executes the block assuming that there is NO valid $LOCK, it would create its own $LOCK file , while the former process when returning to the looping part again as

-e "${LOCK}

it would be in a state of nothing had happened, and it would just continue
Of course, you're right and that went to my mind too, and as workaround I made sure to have another lock: if a "waiting and checking" my_function is detected, the forthcoming functions will just exit instead of waiting too. Also, I had to make sure nothing is passed to my_function (no VAR, for example), so that it won't matter which instance, newer or older, is executed. But obviously - if the noclobber mode will work fine (see the above post) many of these concerns will go away.

Quote:
Originally Posted by kshkid
since we dont bind atomicity to the operations of a while code,

the process wouldnt be aware of such a rapid change of deleting and creating a file instataneously.

What should be concentrated is, notification of the process when a file or status of the file changes ?
Also right, you were probably thinking of something like:
Code:
inotifywait ${LOCK}
But the above also has 2 drawbacks:
1. it will work only with linux kernels >= 2.6.13 (no Solaris or BSD)
2. inotify-tools needs to be installed.

Thanks for your reply.
Cheers.

Last edited by omnio; 03-12-2007 at 02:24 AM.
 
Old 03-11-2007, 06:07 AM   #10
tread
LQ Newbie
 
Registered: Feb 2007
Posts: 29

Rep: Reputation: 15
Quote:
Originally Posted by jlinkels
Interesting approach. However, I think it is not solvable this way.

As long as the touch ($file) function is not atomic, there is always a chance that the two instances create the file at the same time. Having an atomic function is very essential for the concepts of semaphores. Most processors provide a Test And Set instruction which is atomic and cannot be interrupted. Trying to achieve this without atomic operations in a multitasking environment is very difficult.

The author of the link you provide seems to know where he is talking about, at least his explanation of semaphores is correct. (Funny that the original meaning of P and V got lost: I am Dutch, and I have seen so many different explanations for P and V)Without studying his program in-depth, I would not be surprised if such a complicated approach is necessary. Getting something *very* simple to work (like an atomic operation) might be complicated if the elementary building blocks lack.

jlinkels
I believe symlink creation is atomic. So you could touch a file, then try creating a symbolic link to it - if this succeeds only then do you have the lock. I don't remember where I read this though.
 
Old 03-11-2007, 06:38 AM   #11
omnio
Member
 
Registered: Feb 2007
Location: $HOME
Distribution: Hardened Gentoo
Posts: 66
Blog Entries: 1

Original Poster
Rep: Reputation: 16
Quote:
Originally Posted by tread
I believe symlink creation is atomic. So you could touch a file, then try creating a symbolic link to it - if this succeeds only then do you have the lock. I don't remember where I read this though.
Yes I think it will work too. Just like in the noclobber mode, if destination doesn't exist it will be created, and if does exist "ln" will exit 1. Excellent, thanks.
 
Old 03-11-2007, 11:12 PM   #12
kshkid
Member
 
Registered: Dec 2005
Distribution: RHEL3, FC3
Posts: 383

Rep: Reputation: 30
Is there any reason that symlink creation should be atomic?

What forces the operation to be an atomic one ?

 
Old 03-12-2007, 03:20 AM   #13
omnio
Member
 
Registered: Feb 2007
Location: $HOME
Distribution: Hardened Gentoo
Posts: 66
Blog Entries: 1

Original Poster
Rep: Reputation: 16
Quote:
Originally Posted by kshkid
Is there any reason that symlink creation should be atomic?

What forces the operation to be an atomic one ?

I'm not quite in a position to explain this in depth; I think that it would be ideal if both the check for the lock file and its creation should happen with a single I/O operation on the disk, but I'm not sure this is the case here (or even possible). However, in both cases (noclobber & symlink) they are issued by a single command and it seems quite fast so I'm happy with them both; so far I changed my code to use the noclobber mode and I get no more errors because of lock overlapping, so I'm moving on. Maybe someone else can explain this better.
 
Old 03-12-2007, 03:36 AM   #14
bigearsbilly
Senior Member
 
Registered: Mar 2004
Location: england
Distribution: FreeBSD, Debian, Mint, Puppy
Posts: 3,290

Rep: Reputation: 174Reputation: 174
It's not going to be possible using shell scripts and
not trivial using anything else.

I've been shell scripting every working day for years.
There is no foolproof way around this problem, that I know of.

which is why, happily, they still employ people like me to type rm lockfile
and kill -9 occasionally
 
  


Reply

Tags
semaphores, shell


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Shell source code drimades Linux - General 6 10-24-2006 10:52 AM
convert shell script to c code 3saul Linux - Software 1 01-02-2006 09:40 AM
shell scource code??? akshay_jp Linux - General 3 11-05-2005 09:46 PM
c code vs shell script again? khucinx Programming 1 05-12-2004 09:44 PM
c code vs shell script? khucinx Programming 1 05-12-2004 06:06 PM


All times are GMT -5. The time now is 11:16 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration