Newbie! How do I get background jobs to execute sequentially?

Suncoast · 05-04-2009, 12:15 PM

Hello everyone

I'm looking for a general pointer, or label for what I am trying to do. Not a specific line of code.

I've written a couple of small sh script that gets some user input, then calls several programs to run in the background with the &. My problem is, they all run at the same time.

Is there some way to get these jobs to run sequentially rather than consecutively? If yes, what is this process called? I'm thinking there should be a way to line up background jobs in a job queue, similar to how a print queue works, one job at a time. After searching for a couple of hours, I'm thinking there must be a name for this, but I don't know what it is.

Thanks,
Steve

Edit: Linux Slackware Kernel 2.6.x

MensaWater · 05-04-2009, 01:56 PM

If you need them sequential simply don't put the "&" behind all of them. Run the script that calls them with an "&" instead. That backgrounds the whole script which does everything sequentially within the backgrounded process.

PTrenholme · 05-04-2009, 02:15 PM

A simple alternative to jlightner's suggestion (and that suggestion is a good one) is to build a script in /tmp that just invokes the jobs with the answers as parameters, run the script with the "&," do a disown, do a rm /tmp/<script_name> (which you can do while it's running), and exit the "master" script.

ntubski · 05-04-2009, 04:33 PM

Wouldn't it be even simpler to use a subshell?

Code:

(program1 args
 program2 args
 etc...) &

Is disown needed either way though?

Suncoast · 05-05-2009, 02:11 PM

Thank you for your kind replies. I'm taking this to mean there is no job queue system in Gnu/Linux.

The /tmp suggestion is something new to me, I will work on that. Thank you.

MensaWater · 05-05-2009, 03:03 PM

There is a basic job queuing and scheduling mechanism called cron (and another called anacron for systems like laptops that aren't on all the time).

There are also full blown commercial tools like Tivoli Maestro that allow you to do jobs, schedules etc... across multiple UNIX/Linux machines and make things dependent on each other.

However, you don't have to "queue" things here though since you want them to run sequentially. You just run them in order. You are misunderstanding use of the "&". It doesn't simply background - it also says to run "asynchronously" i.e. the opposite of sequentially.

Rather than trying to impose whatever OS world view you're used to on Linux you should work out WHAT you want to do then ask that question which is what we thought you were doing. Making an assumption based on the question you did NOT ask is faulty logic.

bigearsbilly · 05-05-2009, 06:25 PM

Quote:

Originally Posted by Suncoast

I'm taking this to mean there is no job queue system in Gnu/Linux.

very very wrong.

as mentioned cron.
also at and batch

observe:

Code:

for f in date ls ps pwd
do 
echo $f | at now + 1 minute
done 
Job 3 will be executed using /bin/sh
Job 4 will be executed using /bin/sh
Job 5 will be executed using /bin/sh
Job 6 will be executed using /bin/sh
check the queue
$ atq
Date                            Owner           Queue   Job#
Wed May  6 00:24:00 BST 2009    billy           c       3
Wed May  6 00:24:00 BST 2009    billy           c       4
Wed May  6 00:24:00 BST 2009    billy           c       5
Wed May  6 00:24:00 BST 2009    billy           c       6
$

bigearsbilly · 05-05-2009, 06:28 PM

Quote:

Originally Posted by jlightner

Rather than trying to impose whatever OS world view you're used to on Linux you should work out WHAT you want to do then ask that question which is what we thought you were doing. Making an assumption based on the question you did NOT ask is faulty logic.

exactly,
tell us what you are trying to do, not how you want it done.

Suncoast · 05-05-2009, 07:19 PM

Quote:

Originally Posted by jlightner

Rather than trying to impose whatever OS world view you're used to on Linux you should work out WHAT you want to do then ask that question which is what we thought you were doing. Making an assumption based on the question you did NOT ask is faulty logic.

Well hold on. I'm in new territory here, and I'm trying to learn. I thought I summarized what I was looking for by saying;

Quote:

Originally Posted by Suncoast

I'm thinking there should be a way to line up background jobs in a job queue, similar to how a print queue works, one job at a time. After searching for a couple of hours, I'm thinking there must be a name for this, but I don't know what it is.

Apparently I failed.

I normally try any suggestions before responding, but seeing where this thread is headed I felt an immediate response was needed. I will now familiarize myself with at, batch, and look again at cron.

Thank you.

PTrenholme · 05-05-2009, 07:33 PM

bigearsbilly's demonstration suggests another possible solution:

Run the script to get your parameters, and write the parameters for each job to a file, either a per-job file or a shared one. (I'd use per-job files in /tmp a delete them when finished, but you might want to use the "last run" values as defaults for your "next run.")
Have the script start the first job with an at -f <script_name> now, where that script reads its parameters from the file.
When the script terminates, have it issue an at -f <next_script> now to start the next script.
Repeat step 3 for each job in the chain.

Note that you don't, in fact, need to use the at command for this. A simple sh <script_name> &;disown would also work.

You could also just run a script like this:

Code:

#!/bin/bash
#
# Ask questions to set parameters
....
#
# Run the jobs in sequence
for ((i=0;[ -e Job_$i ];++i))
do
  . Job_$i $Job_i_parameter_1 . . .
done

amysaraantony · 05-06-2009, 11:34 AM

cron is your friend Suncoast

Debian

MensaWater · 05-06-2009, 12:31 PM

You might want to look at "JOB CONTROL" section of bash man page (type man bash then do a locate for JOB CONTROL in that page).

However talking about the queuing a printer does is sort of the antithesis of the purpose of UNIX/Linux. That is to say UNIX was designed to be a "multiuser/multiprocess" system. Linux was written as a clone of UNIX. What you want to do could help insure resources are available for later processes as they're not tied up by earlier ones but except in extreme situations this is simply not necessary due to the built in process scheduling/clock ticks etc... done at a very low level.

If you have a specific set of jobs with no set time period easily defined in cron then what you might do is write each job so that IT executes the next one after it completes. That is easily doable with scripts and programs. Also you could make job end markers (e.g. touch /tmp/job1end or touch /var/tmp/billybob) then make the next job look for that file - if it isn't there for today (assuming you expect all jobs within a 24 hour period) then the subsequent job simply waits a period of time and checks again. This too is easily accomplished with a while loop in a script.