Latest LQ Deal: Complete CCNA, CCNP & Red Hat Certification Training Bundle
Go Back > Forums > Non-*NIX Forums > Programming
User Name
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.


  Search this Thread
Old 06-06-2003, 08:05 PM   #1
LQ Newbie
Registered: Jun 2003
Posts: 2

Rep: Reputation: 0
Shell scripting and background processes - help needed.

Hi all...

I have a shell script that goes to 4 web sites to download 4 price files.


After all 4 files are downloaded, each file is unzipped and imported into my database.


Since the downloads take a while it would make more sense if I ran the downloads as background processes. However, I obviously CANT run the gunzip commands until all 4 files have been completed.

My question is:
How can I start 4 background processes up, and have it wait until all four are complete until it starts the next step.

Something like:
curl &
curl &
curl &
curl &

# Wait until all 4 downloads are complete

Unzip all 4 files

Any ideas on how to handle this one?

Old 06-06-2003, 08:14 PM   #2
Registered: Oct 2002
Location: Lower Alabama
Distribution: Slackware, OpenBSD 3.9
Posts: 344

Rep: Reputation: 31
You could try piping the output from curl to gunzip then into a file like so:
curl | gunzip -f > pricelist.txt &
That way you could do it all in one step.


Last edited by green_dragon37; 06-06-2003 at 08:16 PM.
Old 06-07-2003, 11:33 AM   #3
Senior Member
Registered: Aug 2002
Location: Groningen, The Netherlands
Distribution: ubuntu
Posts: 2,530

Rep: Reputation: 110Reputation: 110
You mention .zip files.
As far as I know gunzip doesn't handle .zip files, and unzip doesn't output to a pipe (stdout). So I think green_dragon's solution doesn't work in the case of .zip files.

But there is a way around this that doesn't wait until all files are downloaded, but start unzipping a file in the background as soon as it is downloaded, and then waits until all are done downloading and unzipping. It works by starting sub-shells (bash in this example) in the background, each of which downloads a file and then unzipping it. Try something like:


(curl -s -o $URL1 ; echo 'got #1' ; unzip -qq ; echo '#1 done')&
(curl -s -o $URL2 ; echo 'got #2' ; unzip -qq ; echo '#2 done')&
(curl -s -o $URL3 ; echo 'got #3' ; unzip -qq ; echo '#3 done')&
(curl -s -o $URL4 ; echo 'got #4' ; unzip -qq ; echo '#4 done')&

echo DONE!
However if it is .gz files you meant, it's probably better to stick with green_dragon's solution.

Last edited by Hko; 06-07-2003 at 11:37 AM.
Old 06-08-2003, 10:15 AM   #4
LQ Newbie
Registered: Jun 2003
Posts: 2

Original Poster
Rep: Reputation: 0
Thanks guys. Both replies were very helpful

The chaining of commands with the ; is needed for sure.

As well, the use of the 'WAIT' command (I didn't know that one) helped wait until the processes were finished.




Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off

Similar Threads
Thread Thread Starter Forum Replies Last Post
shell scripting - help needed jonhewer Linux - Newbie 54 09-09-2005 03:58 PM
background processes digit Linux - Software 3 07-27-2003 01:27 PM
processes in background zetsui Linux - General 8 06-14-2003 01:52 PM
bash: interaction with background processes ... cdex Programming 5 08-11-2002 03:42 PM
background processes hakum Programming 1 04-08-2002 11:29 PM > Forums > Non-*NIX Forums > Programming

All times are GMT -5. The time now is 06:22 PM.

Main Menu
Write for LQ is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration