LinuxQuestions.org
Latest LQ Deal: Latest LQ Deals
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices


Reply
  Search this Thread
Old 06-06-2003, 07:05 PM   #1
trafalgar
LQ Newbie
 
Registered: Jun 2003
Posts: 2

Rep: Reputation: 0
Shell scripting and background processes - help needed.


Hi all...

I have a shell script that goes to 4 web sites to download 4 price files.

curl http://www.site.com/pricelist1.zip
curl http://www.nextsite.com/pricelist2.zip
curl http://www.thirdsite.com/pricelist3.zip
curl http://www.fourthsite.com/pricelist4.zip


After all 4 files are downloaded, each file is unzipped and imported into my database.

gunzip pricelist1.zip
gunzip pricelist2.zip
gunzip pricelist3.zip
gunzip pricelist4.zip

Since the downloads take a while it would make more sense if I ran the downloads as background processes. However, I obviously CANT run the gunzip commands until all 4 files have been completed.

My question is:
How can I start 4 background processes up, and have it wait until all four are complete until it starts the next step.

Something like:
--------------
curl http://www.site.com/pricelist1.zip &
curl http://www.nextsite.com/pricelist2.zip &
curl http://www.thirdsite.com/pricelist3.zip &
curl http://www.fourthsite.com/pricelist4.zip &

# Wait until all 4 downloads are complete

Unzip all 4 files
--------------

Any ideas on how to handle this one?

David
 
Old 06-06-2003, 07:14 PM   #2
green_dragon37
Member
 
Registered: Oct 2002
Location: Lower Alabama
Distribution: Slackware, OpenBSD 3.9
Posts: 344

Rep: Reputation: 31
You could try piping the output from curl to gunzip then into a file like so:
Code:
curl http://www.site.com/pricelist | gunzip -f > pricelist.txt &
That way you could do it all in one step.

Ian

Last edited by green_dragon37; 06-06-2003 at 07:16 PM.
 
Old 06-07-2003, 10:33 AM   #3
Hko
Senior Member
 
Registered: Aug 2002
Location: Groningen, The Netherlands
Distribution: Debian
Posts: 2,536

Rep: Reputation: 111Reputation: 111
You mention .zip files.
As far as I know gunzip doesn't handle .zip files, and unzip doesn't output to a pipe (stdout). So I think green_dragon's solution doesn't work in the case of .zip files.

But there is a way around this that doesn't wait until all files are downloaded, but start unzipping a file in the background as soon as it is downloaded, and then waits until all are done downloading and unzipping. It works by starting sub-shells (bash in this example) in the background, each of which downloads a file and then unzipping it. Try something like:
Code:
#!/bin/bash

URL1="http://www.site.com/pricelist1.zip"
URL2="http://www.nextsite.com/pricelist2.zip"
URL3="http://www.thirdsite.com/pricelist3.zip"
URL4="http://www.fourthsite.com/pricelist4.zip"

(curl -s -o list1.zip $URL1 ; echo 'got #1' ; unzip -qq list1.zip ; echo '#1 done')&
(curl -s -o list2.zip $URL2 ; echo 'got #2' ; unzip -qq list2.zip ; echo '#2 done')&
(curl -s -o list3.zip $URL3 ; echo 'got #3' ; unzip -qq list3.zip ; echo '#3 done')&
(curl -s -o list4.zip $URL4 ; echo 'got #4' ; unzip -qq list4.zip ; echo '#4 done')&

wait
echo DONE!
However if it is .gz files you meant, it's probably better to stick with green_dragon's solution.

Last edited by Hko; 06-07-2003 at 10:37 AM.
 
Old 06-08-2003, 09:15 AM   #4
trafalgar
LQ Newbie
 
Registered: Jun 2003
Posts: 2

Original Poster
Rep: Reputation: 0
Thanks guys. Both replies were very helpful

The chaining of commands with the ; is needed for sure.

As well, the use of the 'WAIT' command (I didn't know that one) helped wait until the processes were finished.

Brilliant!

Dave
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
shell scripting - help needed jonhewer Linux - Newbie 54 09-09-2005 02:58 PM
background processes digit Linux - Software 3 07-27-2003 12:27 PM
processes in background zetsui Linux - General 8 06-14-2003 12:52 PM
bash: interaction with background processes ... cdex Programming 5 08-11-2002 02:42 PM
background processes hakum Programming 1 04-08-2002 10:29 PM

LinuxQuestions.org > Forums > Non-*NIX Forums > Programming

All times are GMT -5. The time now is 12:00 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration