LinuxQuestions.org - Shell scripting and background processes

- Programming (https://www.linuxquestions.org/questions/programming-9/)

- - Shell scripting and background processes - help needed. (https://www.linuxquestions.org/questions/programming-9/shell-scripting-and-background-processes-help-needed-64099/)

Shell scripting and background processes - help needed.

Hi all...

I have a shell script that goes to 4 web sites to download 4 price files.

curl http://www.site.com/pricelist1.zip
curl http://www.nextsite.com/pricelist2.zip
curl http://www.thirdsite.com/pricelist3.zip
curl http://www.fourthsite.com/pricelist4.zip

After all 4 files are downloaded, each file is unzipped and imported into my database.

gunzip pricelist1.zip
gunzip pricelist2.zip
gunzip pricelist3.zip
gunzip pricelist4.zip

Since the downloads take a while it would make more sense if I ran the downloads as background processes. However, I obviously CANT run the gunzip commands until all 4 files have been completed.

My question is:
How can I start 4 background processes up, and have it wait until all four are complete until it starts the next step.

Something like:
--------------
curl http://www.site.com/pricelist1.zip &
curl http://www.nextsite.com/pricelist2.zip &
curl http://www.thirdsite.com/pricelist3.zip &
curl http://www.fourthsite.com/pricelist4.zip &

# Wait until all 4 downloads are complete

Unzip all 4 files
--------------

Any ideas on how to handle this one?

David

You could try piping the output from curl to gunzip then into a file like so:

Code:

curl http://www.site.com/pricelist | gunzip -f > pricelist.txt &

That way you could do it all in one step.

Ian

You mention .zip files.
As far as I know gunzip doesn't handle .zip files, and unzip doesn't output to a pipe (stdout). So I think green_dragon's solution doesn't work in the case of .zip files.

But there is a way around this that doesn't wait until all files are downloaded, but start unzipping a file in the background as soon as it is downloaded, and then waits until all are done downloading and unzipping. It works by starting sub-shells (bash in this example) in the background, each of which downloads a file and then unzipping it. Try something like:

Code:

#!/bin/bash



URL1="http://www.site.com/pricelist1.zip"

URL2="http://www.nextsite.com/pricelist2.zip"

URL3="http://www.thirdsite.com/pricelist3.zip"

URL4="http://www.fourthsite.com/pricelist4.zip"



(curl -s -o list1.zip $URL1 ; echo 'got #1' ; unzip -qq list1.zip ; echo '#1 done')&

(curl -s -o list2.zip $URL2 ; echo 'got #2' ; unzip -qq list2.zip ; echo '#2 done')&

(curl -s -o list3.zip $URL3 ; echo 'got #3' ; unzip -qq list3.zip ; echo '#3 done')&

(curl -s -o list4.zip $URL4 ; echo 'got #4' ; unzip -qq list4.zip ; echo '#4 done')&



wait

echo DONE!

However if it is .gz files you meant, it's probably better to stick with green_dragon's solution.

Thanks guys. Both replies were very helpful

The chaining of commands with the ; is needed for sure.

As well, the use of the 'WAIT' command (I didn't know that one) helped wait until the processes were finished.

Brilliant!

Dave