Use wget to download multiple files with wildcards

Anant Khaitan · 11-30-2007, 08:57 AM

I am trying to download all jpg files from a particular http site.. tell me the exact syntax ...
I have tried this :

Code:

$  wget -r -l1 --no-parent -A jpg  http://www.mikeswanson.com/wallpaper/images/

but it is not working..

pixellany · 11-30-2007, 09:12 AM

Quote:

tell me the exact syntax ...

Please??

From the wget man page:

Quote:

You want to download all the GIFs from a directory on an HTTP
server. You tried wget http://www.server.com/dir/*.gif, but that
didn't work because HTTP retrieval does not support globbing. In
that case, use:

wget -r -l1 --no-parent -A.gif http://www.server.com/dir/

More verbose, but the effect is the same. -r -l1 means to retrieve
recursively, with maximum depth of 1. --no-parent means that ref‐
erences to the parent directory are ignored, and -A.gif means to
download only the GIF files. -A "*.gif" would have worked too.

man pages are not exactly light reading, but they usually have the answer.

Anant Khaitan · 11-30-2007, 12:09 PM

^^
Brother I have already tried what u had mentioned...It works for certain sites...Indeed I use this for downloading entire site..
check my first post..
Here some permission problem.. "403 forbidden"...
and

Code:

-A gif

or

Code:

-A.gif

It doesn't make any difference here...

So can I expect a constructive reply ..
Ok here is the output of the command :

Code:

--22:24:40--  http://www.mikeswanson.com/wallpaper/images/
           => `www.mikeswanson.com/wallpaper/images/index.html'
Resolving www.mikeswanson.com... 209.132.227.101
Connecting to www.mikeswanson.com|209.132.227.101|:80... connected.
HTTP request sent, awaiting response... 403 Forbidden
22:24:41 ERROR 403: Forbidden.

Removing www.mikeswanson.com/wallpaper/images/index.html since it should be rejected.
unlink: No such file or directory

FINISHED --22:24:41--
Downloaded: 0 bytes in 0 files

Fluffy · 11-30-2007, 12:31 PM

Notice these lines:
HTTP request sent, awaiting response... 403 Forbidden
22:24:41 ERROR 403: Forbidden.

Even if you got the syntax right, you wouldn't be able to download all the images anyways.

Code:

wget -r -l1 --no-parent -A jpg  http://www.mikeswanson.com/wallpaper/images/

Should be

Code:

wget -r -|1 --no-parent -A.jpg http://www.mikeswanson.com/wallpaper/images/

Notice the difference between -l1 (dash L one) and -|1 (dash pipe one) also -A jpg and -A.jpg.

pixellany · 11-30-2007, 01:17 PM

First this:

Quote:

tell me the exact syntax ...

Then this:

Quote:

So can I expect a constructive reply ..

I would really encourage you to start using please and thank you in place of these commanding statements......

Anant Khaitan · 12-01-2007, 12:38 AM

^^^
Sorry pixellany if I was rude.. nyways thanx for mentioning...

@ Fluffy

Code:

wget -r -|1 --no-parent -A.jpg http://www.mikeswanson.com/wallpaper/images/

returns

Code:

bash: 1: command not found

it should be 'l' means 'level' not '|'
and AFAIK 'A.gif' or 'A gif' it doesn't matter at all

Fluffy · 12-01-2007, 09:29 AM

Oops. I guess we both screwed up a little then.
Damn font makers need to make a good default font that you can tell the difference between | and l and 1. -_-

But, again, note the 403 forbidden error message you got. With the way mikeswanson.com has it's folders setup you can't view/download from a directory unless you know the exact filename you want to download. (Sometimes not even then.)

mattington · 12-04-2009, 01:03 AM

I mean.. I realize this thread is 2 years old but there are other wildcard command options open to you. For instance, to get pictures that are numbered in order you can do:

wget -nd http://www.cracked.com/blog/wp-content/uploads/2009/12/zorklon{1,2,3,4,5}.jpg

This will repeat the command for each number enumerated in the {}

lag_rvp · 08-23-2013, 10:45 PM

Quote:

Originally Posted by Fluffy

Notice these lines:
HTTP request sent, awaiting response... 403 Forbidden
22:24:41 ERROR 403: Forbidden.

Even if you got the syntax right, you wouldn't be able to download all the images anyways.

Code:

wget -r -l1 --no-parent -A jpg  http://www.mikeswanson.com/wallpaper/images/

Should be

Code:

wget -r -|1 --no-parent -A.jpg http://www.mikeswanson.com/wallpaper/images/

Notice the difference between -l1 (dash L one) and -|1 (dash pipe one) also -A jpg and -A.jpg.

And further bumping an old thread on my first post no less.

This worked for me to DL all of the .mp3's in a directory, just used your same code minuse the -|1 for the website I ripped.

Code:

wget -r --no-parent -A.jpg http://www.mikeswanson.com/wallpaper/images/