LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices


Reply
  Search this Thread
Old 04-08-2004, 07:12 PM   #1
ashibaka
LQ Newbie
 
Registered: Mar 2003
Posts: 8

Rep: Reputation: 0
I'm terrible at writing shell scripts. A little help fetching files?


I'd like to fetch a bunch of files from a website, for mirroring in case it goes down. I tried to write a script like such...

Code:
#!/usr/bin/perl
$x = 0;
while ($x<10) {
        $x = $x + 1;
        system("wget //website/images/wc00" + $x + ".gif");
}
while ($x<100) {
$x = $x + 1;
system("wget //website/images/wc0" + $x + ".gif");


Yeah, I'm terrible at writing shell scripts. It's really badly written, and guess what, it didn't work. Can anyone help me a bit?
 
Old 04-08-2004, 07:17 PM   #2
bdp
Member
 
Registered: Apr 2002
Distribution: RH 9
Posts: 230

Rep: Reputation: 30
have you tried httrack, it's a command line utility that calls itself a 'website copier' and may accomplish the mirroring you require.
 
Old 04-08-2004, 07:19 PM   #3
ashibaka
LQ Newbie
 
Registered: Mar 2003
Posts: 8

Original Poster
Rep: Reputation: 0
I only need these particular numbered images, so I figure httrack would go a little overboard.
 
Old 04-09-2004, 03:24 AM   #4
urzumph
Member
 
Registered: Jan 2004
Location: Australia
Distribution: Debian
Posts: 168

Rep: Reputation: 30
I don't know perl, but that's not a shell script - shell scripts are only interpreted by command prompts (eg bash, ash etc)

[Edit]
Even thou I don't write perl, I noticed some things -
Your second while loop does not close the brackets.
x is not reset before running the second loop
you need the full web-path for wget, not just the extension.
What's with the // before the 'website'?

Re writen, that should give you (in I-am-guessing-perl) :
Code:
#!/usr/bin/perl
$x = 0;
$website="http://www.google.com"
while ($x<10) {
        $x = $x + 1;
        system("wget " + $website + "/website/images/wc00" + $x + ".gif");
}
$x=0;
while ($x<100) {
$x = $x + 1;
system("wget " + $website + "/website/images/wc0" + $x + ".gif");
}
Also, do you need to run a command to convert x (an int) into a string? or does the + operator do that?

Last edited by urzumph; 04-09-2004 at 03:30 AM.
 
Old 04-09-2004, 04:29 AM   #5
/bin/bash
Senior Member
 
Registered: Jul 2003
Location: Indiana
Distribution: Mandrake Slackware-current QNX4.25
Posts: 1,802

Rep: Reputation: 47
From man curl
Code:
SYNOPSIS
       curl [options] [URL...]



DESCRIPTION
       curl is a client to get documents/files from or send docu_
       ments  to  a  server, using any of the supported protocols
       (HTTP, HTTPS, FTP, GOPHER, DICT, TELNET,  LDAP  or  FILE).
       The  command  is designed to work without user interaction
       or any kind of interactivity.

       curl offers a busload of useful tricks like proxy support,
       user  authentication,  ftp upload, HTTP post, SSL (https:)
       connections, cookies, file transfer resume and more.



URL
       The URL  syntax  is  protocol  dependent.  You'll  find  a
       detailed description in RFC 2396.

       You  can specify multiple URLs or parts of URLs by writing
       part sets within braces as in:

        http://site.{one,two,three}.com

       or you can get sequences of alphanumeric series  by  using
       [] as in:

        ftp://ftp.numericals.com/file[1-100].txt
        ftp://ftp.numericals.com/file[001-100].txt    (with lead_
       ing zeros)
        ftp://ftp.letters.com/file[a-z].txt

       It is possible to specify up to 9 sets  or  series  for  a
       URL, but no nesting is supported at the moment:

        http://www.any.org/archive[1996-1999]/vol_
       ume[1-4]part{a,b,c,index}.html

       You can specify any amount of URLs on  the  command  line.
       They  will be fetched in a sequential manner in the speci_
       fied order.

       Curl will attempt to re-use connections for multiple  file
       transfers, so that getting many files from the same server
       will not do multiple connects / handshakes. This  improves
       speed. Of course this is only done on files specified on a
       single command line and cannot be  used  between  separate
       curl invokes.
 
Old 04-09-2004, 04:56 AM   #6
Gnuru
Member
 
Registered: Jan 2004
Posts: 53

Rep: Reputation: 15
Firstly, as someone else pointed out this is not a shell script, this is a perl script. Secondly, if you are going to use perl, then use all the tools that perl gives you, So there is really no need to fork a system call, rather use LWP::Simple. Also always use strict. Try something like this:

Code:
#!/usr/bin/perl
use LWP::Simple;
use warnings;
use strict;
my $x = 0;
my $url = "http://website/images/";
my $file =  'wc';
while (1) {
        $x++;
        my $file_name = $file . sprintf("%02d", $x) . '.gif';
        print "Getting $file_name....";
        my $return_value = getstore($url . $file_name, $file_name);
        print "Got it!\n" if $return_value == 200;
        print "Can't get it!" && last if $return_value =~ /^4/;
}
WARNING CODE IS UNTESTED!
 
Old 04-10-2004, 12:41 AM   #7
ashibaka
LQ Newbie
 
Registered: Mar 2003
Posts: 8

Original Poster
Rep: Reputation: 0
Thanks everyone. I'm sure at least one of these (probably all) will solve my problem.


edit:

Code:
curl http://website/[001-100].gif -o "#1.gif"
This did the job.

Last edited by ashibaka; 04-10-2004 at 12:45 AM.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Writing Shell Scripts in Redhat 6.1 KDE Ace Rimmer Linux - Newbie 5 08-16-2016 07:41 PM
Difficulties writing shell scripts lostinsyntax Programming 8 05-02-2005 07:42 AM
Difficulties writing shell scripts lostinsyntax Linux - Newbie 2 05-01-2005 03:48 PM
Writing C++ Shell Scripts MasterKin8T Programming 5 09-29-2003 10:43 AM
writing Linux shell scripts in Windows NightWolf_NZ Linux - Newbie 3 09-10-2003 09:28 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - General

All times are GMT -5. The time now is 12:39 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration