Making bash script WAIT after starting subshell

mintybadger · 07-13-2009, 11:22 AM

First post - great site. I hope you can help.

Below is a snippet of code from a bash shell script I'm developing to plot some charts (with gradsdap). I used to loop through my text file of locations plotting each chart one at a time, then discovered the power of being able to spawn multiple instances (subshells) of plotting by adding an & at the end of the line. This speeds up the plots by a factor of 5. Excellent! However I'm having trouble getting the script to WAIT for all the child processes to finish before carrying on. I've tried "wait" and also wait for the child processID, but whatever the script seems to run on - echoing "finished meteograms" straight away.

I thought just putting "wait" would have the desired effect, but I must be missing something more subtle - or obvious!

Any thoughts? Thanks

...
if [ $downloadready ] ;then

filein="/home/burn/grads_etc/locslatlong.txt"
grep '[a-zA-Z]' $filein | awk '{print $1" "$2" "$3}' | while read lat lon location
do
(gradsdap -bpcx "/home/burn/lib/meteogram_single $today $dt $lat $lon $location" | tail -c+16) &

# metgramppid=$!
done
# wait $metgramppid
# wait # really want the script to stop here until all the
#+ subshells started above are done.
echo finished meteograms? # this pops up right away - hmm

# next command....
fi
...

zQUEz · 07-13-2009, 11:31 AM

If I understand you correctly, you want to wait until the multiple "gradsdap" processes complete before continuing with the script. If so, why not do a `ps` and look to see if all the `gradsdap` commands have completed. Once they have (e.g. they are no longer in a `ps` listing) then you continue.

While running a loop to check if the `gradsdap` processes are alive, put in a sleep in the loop to check every 30secs or so.

allanf · 07-13-2009, 12:50 PM

Quote:

Originally Posted by mintybadger

First post - great site. I hope you can help.

Below is a snippet of code from a bash shell script I'm developing to plot some charts (with gradsdap). I used to loop through my text file of locations plotting each chart one at a time, then discovered the power of being able to spawn multiple instances (subshells) of plotting by adding an & at the end of the line. This speeds up the plots by a factor of 5. Excellent! However I'm having trouble getting the script to WAIT for all the child processes to finish before carrying on. I've tried "wait" and also wait for the child processID, but whatever the script seems to run on - echoing "finished meteograms" straight away.

I thought just putting "wait" would have the desired effect, but I must be missing something more subtle - or obvious!

Any thoughts? Thanks

...
if [ $downloadready ] ;then

filein="/home/burn/grads_etc/locslatlong.txt"
grep '[a-zA-Z]' $filein | awk '{print $1" "$2" "$3}' | while read lat lon location
do
(gradsdap -bpcx "/home/burn/lib/meteogram_single $today $dt $lat $lon $location" | tail -c+16) &

# metgramppid=$!
done
# wait $metgramppid
# wait # really want the script to stop here until all the
#+ subshells started above are done.
echo finished meteograms? # this pops up right away - hmm

# next command....
fi
...

You can do:

Code:

if [ $downloadready ] ;then

    set -a child_pid
    filein="/home/burn/grads_etc/locslatlong.txt"
    grep '[a-zA-Z]' $filein | awk '{print $1" "$2" "$3}' | while read lat lon location; do
        gradsdap -bpcx "/home/burn/lib/meteogram_single $today $dt $lat $lon $location" | tail -c+16 &
        child_pid[${child_pid[*]}]=$!
    done
    for pid_id in ${child_pid[*]}; do
        wait ${pid_id}
    done

fi

Of course the "wait" without a PID will wait on all children.

David1357 · 07-13-2009, 12:57 PM

Quote:

Originally Posted by mintybadger

Any thoughts?

I created a simple test script that did the equivalent of yours and it worked. Maybe you need to provide the full path to "gradsdap" or maybe you have some other error in your command.

Hobbletoe · 07-13-2009, 01:08 PM

I have the following alias on all of my servers for use which with very little modification could work for you ...

Code:

watch_me(){ while ps -ef | grep $1 | grep -v grep > /dev/null
do
  sleep 5
done;
echo "*** Watch for $1 completed at `date +%H:%M:%S`. ***"; }

Then, I do a watch_me for whatever I happen to be doing (compressing or uncompressing files normally), and it lets me know when it finished (so that I can time things). I think that you could modify this to

Code:

while ps -ef | grep gradsdap | grep -v grep > /dev/null
do
  sleep 5
done;

Put that in after you kick off all of your gradsdap processes, and your script should loop there until they are all finished.

This is assuming that you are the only person on this box. If someone else uses it, then if they run gradsdap, then your script would wait until any process that they are running finishes as well. If that is the case, then add in another grep for your username as well, and then it would watch for when all of your gradsdap processes have finished, and it wouldn't worry about processes by other users.

chrism01 · 07-14-2009, 01:56 AM

I think you'll find the 'while' loop creates a sub-shell that then creates backgrounded process.
'wait' only waits for procs backgrounded from the 'current' shell.
Replace 'while' with a 'for' loop and you should be good.

ntubski · 07-14-2009, 02:56 AM

Quote:

Originally Posted by David1357

I created a simple test script that did the equivalent of yours and it worked. Maybe you need to provide the full path to "gradsdap" or maybe you have some other error in your command.

I think the problem has to do with the subshell created by the | while read construct (which is why allanf's script won't work either).

Code:

#!/bin/bash
echo -e 'foo\nbar\nbaz' | while read word ; do
    (sleep 5 ; echo done $word ) &
done
wait
echo all finished

This script results in the following

Code:

~/tmp$ ./wait.sh 
all done
~/tmp$ done foo [5 seconds later]
done bar
done baz

Code:

#!/bin/bash
while read word ; do
    (sleep 5 ; echo done $word ) &
done < <(echo -e 'foo\nbar\nbaz')
wait
echo all finished

But this script works as expected:

Code:

~/tmp$ ./wait.sh 
done foo [5 seconds later]
done bar
done baz
all finished

Another possibility, if you don't want to use process substitution is

Code:

#!/bin/sh
echo -e 'foo\nbar\nbaz' | ( while read word ; do
    (sleep 5 ; echo done $word ) &
done ; wait )

echo all finished

mintybadger · 07-14-2009, 07:40 AM

ntubski and chrism01 - I think you've nailed my problem. When running from a pipe, the while loop was starting in a subshell itself, then the wait command wouldn't work for all the other sub-sub-shells.

Thanks to all though - you inspired me to work out the same conclusions myself as well as gaining some other tips as always.

http://mywiki.wooledge.org/BashFAQ/024 gives some more background about this.

Cheers

David1357 · 07-14-2009, 04:40 PM

Quote:

Originally Posted by David1357

I created a simple test script that did the equivalent of yours and it worked.

I finally found the line in the "man bash" page that explains the problem to me:

"Each command in a pipeline is executed as a separate process (i.e., in a subshell)."

When I created my "simple test script", I used a "for" loop instead of the "| while" syntax. So my script was not equivalent. I like ntubski's answer.

kailash19 · 08-04-2011, 04:49 AM

I encountered this post searching for solution to my problem similar to the one described here.

Thanks ntubski for the explanation, that pipe starts a new sub-shell.

Great site!!!!!!