LinuxQuestions.org
Help answer threads with 0 replies.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices


Reply
  Search this Thread
Old 01-03-2017, 03:10 PM   #1
scotttc
LQ Newbie
 
Registered: Nov 2016
Posts: 5

Rep: Reputation: Disabled
Possible problem running processes in parallel


I have this script that works fine when run sequentially. It calls several sub scripts that do the leg work. The sub scripts DO NOT have dependencies directly to one another. The are interested in a few of the same .a files it seems.

I keep running into an inconsistent (but frequent) problem where the .a file cannot be found by scons.. even though when I look it is there.

I am sending 8 scripts to the background and waiting for them all to complete. It seems different scripts return the error each time but they all run into the same issue with this file (and occasionally a different error), but it changes each time I run it.

Could this be related to running the scripts in parallel?

Any ideas on how I can diagnose this as well?


I am running CentOs7 in vsphere with 8 processors and 32GB memory available to it.

thanks in advance!

Last edited by scotttc; 01-03-2017 at 03:11 PM.
 
Old 01-03-2017, 05:18 PM   #2
jailbait
LQ Guru
 
Registered: Feb 2003
Location: Blue Ridge Mountain
Distribution: Linux Mint 17, Debian 8
Posts: 7,865

Rep: Reputation: 311Reputation: 311Reputation: 311Reputation: 311
Quote:
Originally Posted by scotttc View Post
Could this be related to running the scripts in parallel?
Yes, when you run processes in parallel you have to allow for the fact that any file that is written to by more than one process can only be accessed by one process at a time. Each process using such a file has to obtain exclusive control of the file before writing to the file and release control when it is finished with the file.

You can use either lock files or mutex locks but every process has to use the same method, i.e. you can't mix lock files and mutexes for the same item being controlled.

The following links will give you a grounding in the theory of how lock files and mutexes work. Note that mutexes can be used for a wide range of non-sharable items in addition to files.

https://en.wikipedia.org/wiki/File_locking

https://en.wikipedia.org/wiki/Lock_(computer_science)


------------------------
Steve Stites
 
Old 01-04-2017, 09:34 AM   #3
scotttc
LQ Newbie
 
Registered: Nov 2016
Posts: 5

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by jailbait View Post
Yes, when you run processes in parallel you have to allow for the fact that any file that is written to by more than one process can only be accessed by one process at a time. Each process using such a file has to obtain exclusive control of the file before writing to the file and release control when it is finished with the file.

You can use either lock files or mutex locks but every process has to use the same method, i.e. you can't mix lock files and mutexes for the same item being controlled.

The following links will give you a grounding in the theory of how lock files and mutexes work. Note that mutexes can be used for a wide range of non-sharable items in addition to files.

https://en.wikipedia.org/wiki/File_locking

https://en.wikipedia.org/wiki/Lock_(computer_science)


------------------------
Steve Stites

Thanks Steve. So after reading up on file locking some I (think) I see what my problem is. I do not need to write to the .a file, I only need to read from it so it surprises me that I cannot have multiple processes looking at the same file at the same time since all the data will stay the same.. unless the first process to access the file puts a lock on it, I do not know if this is the case or how to tell.

Besides diving (very) deep into the code and putting a checkFileIsFree function before it accesses it is there a way I can manage the files accessing the same file at the same time, like a queue or something?

Thanks,
Scott
 
Old 01-04-2017, 09:41 AM   #4
wpeckham
Senior Member
 
Registered: Apr 2010
Location: Continental USA
Distribution: Debian, Ubuntu, Fedora, RedHat, DSL, Puppy, CentOS, Knoppix, Mint-DE, Sparky, Vsido, tinycore, Q4OS
Posts: 3,199

Rep: Reputation: 1386Reputation: 1386Reputation: 1386Reputation: 1386Reputation: 1386Reputation: 1386Reputation: 1386Reputation: 1386Reputation: 1386Reputation: 1386
Quote:
Originally Posted by scotttc View Post
Thanks Steve. So after reading up on file locking some I (think) I see what my problem is. I do not need to write to the .a file, I only need to read from it so it surprises me that I cannot have multiple processes looking at the same file at the same time since all the data will stay the same.. unless the first process to access the file puts a lock on it, I do not know if this is the case or how to tell.

Besides diving (very) deep into the code and putting a checkFileIsFree function before it accesses it is there a way I can manage the files accessing the same file at the same time, like a queue or something?

Thanks,
Scott
How big is the file? Would it be reasonable to slurp the entire file into a variable: Perl style? If not, there are other techniques.
 
Old 01-04-2017, 10:22 AM   #5
sundialsvcs
LQ Guru
 
Registered: Feb 2004
Location: SE Tennessee, USA
Distribution: Gentoo, LFS
Posts: 9,078
Blog Entries: 4

Rep: Reputation: 3179Reputation: 3179Reputation: 3179Reputation: 3179Reputation: 3179Reputation: 3179Reputation: 3179Reputation: 3179Reputation: 3179Reputation: 3179Reputation: 3179
You should probably pipe the STDERR and STDOUT outputs of each process to its own file so that you can see what they might be reporting. Be sure to code them in such a way that they don't "assume" that something will work "merely because it always did."

It is possible to open a file for shared access, provided that you use locks or other means to synchronize access to it and you make sure to flush buffers to disk before releasing control.

If you want several processes to read a file at the same time, you may need to specify "shared access" when opening the file.
 
Old 01-04-2017, 11:52 AM   #6
jailbait
LQ Guru
 
Registered: Feb 2003
Location: Blue Ridge Mountain
Distribution: Linux Mint 17, Debian 8
Posts: 7,865

Rep: Reputation: 311Reputation: 311Reputation: 311Reputation: 311
Quote:
Originally Posted by scotttc View Post

Besides diving (very) deep into the code and putting a checkFileIsFree function before it accesses it is there a way I can manage the files accessing the same file at the same time, like a queue or something?
Yes, you can use mutexes. The advantage of a mutex is that it is all done in memory which is much faster than writing lock files to a hard drive and then later deleting them.

-----------------
Steve Stites
 
  


Reply

Tags
background, parallel, process, script


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
LXer: Nvidia and ARM: It's a parallel, parallel, parallel world LXer Syndicated Linux News 0 03-21-2013 07:10 PM
problem in running parallel program in c roshani Linux - Newbie 3 03-05-2013 09:15 AM
Parallel processes execution using fork() Thodoris21 Programming 3 05-15-2012 05:36 PM
Perl - How to do multiple child processes in parallel? DevonB Programming 4 09-03-2010 10:15 PM
parallel boot processes/threads? dbrazeau Linux - Kernel 3 02-22-2010 01:53 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 04:46 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration