LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 03-19-2015, 11:33 AM   #1
nouse
LQ Newbie
 
Registered: Sep 2013
Posts: 21

Rep: Reputation: Disabled
Querying up jobs in a multi-core environment, via a shell script


Hi!

I am running a script per nohup ./script.sh:
#!/bin/bash
Do.stuff -some_parameters &
Do.stuff -some_parameters

I have 64 cores at my disposal, so i start a script with 60 lines (ignoring 4 cores to be free for allday-jobs). I have to do this for ~800 jobs. A job is done within 20 hours, so each morning i come in, start a script with 60 lines, and come back the other morning.

Today i was thinking...would it be possible to query my jobs? So, whenever a core is free, start the next job from the list given in the script?




I know, i am expected to have already tried things out. But seriously, i cant. This is way beyond my skill level. I am just asking to avoid coming to the lab this weekend

Last edited by nouse; 03-19-2015 at 12:19 PM.
 
Old 03-19-2015, 12:36 PM   #2
suicidaleggroll
LQ Guru
 
Registered: Nov 2010
Location: Colorado
Distribution: OpenSUSE, CentOS
Posts: 5,259

Rep: Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947
I think you'd be much better off using a scheduling system like torque/maui/moab. All jobs get submitted to the scheduler, and the scheduler then automatically farms them out based on memory requirements, system load, etc.

At a bare minimum, you could check the load by parsing the output of "uptime". If it's less than 64 it means you have an available processor. Just make sure you give it time to update, since it's a load average it takes some time to react to load changes.

Last edited by suicidaleggroll; 03-19-2015 at 12:38 PM.
 
Old 03-19-2015, 12:59 PM   #3
nouse
LQ Newbie
 
Registered: Sep 2013
Posts: 21

Original Poster
Rep: Reputation: Disabled
Thanks! Ill ask my system admin if he can install such a thing. Seems like a must-have-tool anyway (especially when many coworkers are asking for cpu-time).
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Script to run XBMC in a multi-monitor environment refuses to work correctly Big-Blue Linux - Software 0 07-25-2013 11:45 PM
Shell script for rDNS with for loop querying each line of a text file wolfsden3 Programming 3 06-11-2012 04:20 PM
how to write a shell script for submitting multiple jobs sequentially cliffyao Programming 4 05-27-2010 03:44 PM
Multi Core Environment - process migration chamila1986 Programming 5 10-04-2009 08:19 AM
ps/jobs in shell script paraiso Linux - Newbie 3 04-22-2005 10:04 AM


All times are GMT -5. The time now is 02:10 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration