I'm terrible at writing shell scripts. A little help fetching files?

ashibaka · 04-08-2004, 07:12 PM

I'd like to fetch a bunch of files from a website, for mirroring in case it goes down. I tried to write a script like such...

Code:

#!/usr/bin/perl
$x = 0;
while ($x<10) {
        $x = $x + 1;
        system("wget //website/images/wc00" + $x + ".gif");
}
while ($x<100) {
$x = $x + 1;
system("wget //website/images/wc0" + $x + ".gif");

Yeah, I'm terrible at writing shell scripts. It's really badly written, and guess what, it didn't work. Can anyone help me a bit?

bdp · 04-08-2004, 07:17 PM

have you tried httrack, it's a command line utility that calls itself a 'website copier' and may accomplish the mirroring you require.

ashibaka · 04-08-2004, 07:19 PM

I only need these particular numbered images, so I figure httrack would go a little overboard.

urzumph · 04-09-2004, 03:24 AM

I don't know perl, but that's not a shell script - shell scripts are only interpreted by command prompts (eg bash, ash etc)

[Edit]
Even thou I don't write perl, I noticed some things -
Your second while loop does not close the brackets.
x is not reset before running the second loop
you need the full web-path for wget, not just the extension.
What's with the // before the 'website'?

Re writen, that should give you (in I-am-guessing-perl) :

Code:

#!/usr/bin/perl
$x = 0;
$website="http://www.google.com"
while ($x<10) {
        $x = $x + 1;
        system("wget " + $website + "/website/images/wc00" + $x + ".gif");
}
$x=0;
while ($x<100) {
$x = $x + 1;
system("wget " + $website + "/website/images/wc0" + $x + ".gif");
}

Also, do you need to run a command to convert x (an int) into a string? or does the + operator do that?

/bin/bash · 04-09-2004, 04:29 AM

From man curl

Code:

SYNOPSIS
       curl [options] [URL...]



DESCRIPTION
       curl is a client to get documents/files from or send docu_
       ments  to  a  server, using any of the supported protocols
       (HTTP, HTTPS, FTP, GOPHER, DICT, TELNET,  LDAP  or  FILE).
       The  command  is designed to work without user interaction
       or any kind of interactivity.

       curl offers a busload of useful tricks like proxy support,
       user  authentication,  ftp upload, HTTP post, SSL (https:)
       connections, cookies, file transfer resume and more.



URL
       The URL  syntax  is  protocol  dependent.  You'll  find  a
       detailed description in RFC 2396.

       You  can specify multiple URLs or parts of URLs by writing
       part sets within braces as in:

        http://site.{one,two,three}.com

       or you can get sequences of alphanumeric series  by  using
       [] as in:

        ftp://ftp.numericals.com/file[1-100].txt
        ftp://ftp.numericals.com/file[001-100].txt    (with lead_
       ing zeros)
        ftp://ftp.letters.com/file[a-z].txt

       It is possible to specify up to 9 sets  or  series  for  a
       URL, but no nesting is supported at the moment:

        http://www.any.org/archive[1996-1999]/vol_
       ume[1-4]part{a,b,c,index}.html

       You can specify any amount of URLs on  the  command  line.
       They  will be fetched in a sequential manner in the speci_
       fied order.

       Curl will attempt to re-use connections for multiple  file
       transfers, so that getting many files from the same server
       will not do multiple connects / handshakes. This  improves
       speed. Of course this is only done on files specified on a
       single command line and cannot be  used  between  separate
       curl invokes.

Gnuru · 04-09-2004, 04:56 AM

Firstly, as someone else pointed out this is not a shell script, this is a perl script. Secondly, if you are going to use perl, then use all the tools that perl gives you, So there is really no need to fork a system call, rather use LWP::Simple. Also always use strict. Try something like this:

Code:

#!/usr/bin/perl
use LWP::Simple;
use warnings;
use strict;
my $x = 0;
my $url = "http://website/images/";
my $file =  'wc';
while (1) {
        $x++;
        my $file_name = $file . sprintf("%02d", $x) . '.gif';
        print "Getting $file_name....";
        my $return_value = getstore($url . $file_name, $file_name);
        print "Got it!\n" if $return_value == 200;
        print "Can't get it!" && last if $return_value =~ /^4/;
}

WARNING CODE IS UNTESTED!

ashibaka · 04-10-2004, 12:41 AM

Thanks everyone. I'm sure at least one of these (probably all) will solve my problem.

edit:

Code:

curl http://website/[001-100].gif -o "#1.gif"

This did the job.