wget searching and downloading images on a website ?

memo007 · 08-06-2008, 11:49 PM

I was needing images on a server which are on the server but i have to access them manually.
It would be for example www.xyz.com/collection/846_02_900_x.jpg
When i change to /collection/847_02_900_x.jpg (just one number higher) there is a picture as well, and i need all the pictures in the collection folder.. However, if i just go to www.xyz.com/collection there are no images, it just says the there was an error... So without me having to sit 3 hours and do it all manually, going from 846_02_900_x.jpg 847_02_900_x.jpg 848_02_900_x.jpg etc.. i was wondering if wget can download the pictures somehow?

When i try it with wget -m -p -k http://www.xyz.com/collection all i get is HTTP request sent, awaiting response... 404 Not Found
And of course when i do wget -m -p -k www.xyz.com/collection/846_02_900_x.jpg it download only that image. The question is if there is a possibility to set it to increase certain values automatically such as 846, and download accordingly, without changing any other numbers ?

Mr. C. · 08-07-2008, 12:08 AM

Wget doesn't give you the magic ability to scan a web sites directories. Wget follows links. If a page has links to those images, wget can be told to follow the links to download the images.

Alternatively, you can create a file that contains those links, and have wget iterate over those.

memo007 · 08-07-2008, 12:17 AM

I don't necessarily need to scan it but only specify what images to download from one number to the other..

like from 847_02_900_x.jpg to 10000_02_900_x.jpg download all images..
To simply specify a numeric sequence or a range ...

Quote:

Originally Posted by Mr. C.

Wget doesn't give you the magic ability to scan a web sites directories. Wget follows links. If a page has links to those images, wget can be told to follow the links to download the images.

Alternatively, you can create a file that contains those links, and have wget iterate over those.

Mr. C. · 08-07-2008, 12:34 AM

Right, so you need a script that can generate the file names. You can take it from here:

Code:

for i in $(seq 847 10000) ; do 
   echo ${i}_02_900_x.jpg
done

memo007 · 08-07-2008, 12:38 AM

Thanks, but after i paste this i get numbers?
How do i use this with wget?

Quote:

Originally Posted by Mr. C.

Right, so you need a script that can generate the file names. You can take it from here:

Code:

for i in $(seq 847 10000) ; do 
   echo ${i}_02_900_x.jpg
done

Mr. C. · 08-07-2008, 12:47 AM

You get file names...

847_02_900_x.jpg
848_02_900_x.jpg
849_02_900_x.jpg
850_02_900_x.jpg
...

Are these not the file names you want? Replace echo with wget, and add the missing part of the URI to the filename.

jiml8 · 08-07-2008, 11:19 AM

Why are you trying to harvest a site's images anyway? Do you have permission to suck up their bandwidth this way?

If I catch you doing that on my site, I'll blacklist you...and my site watches for exactly that, so it likely will catch you.

memo007 · 08-08-2008, 02:34 PM

Grow up man...

Quote:

Originally Posted by jiml8

Why are you trying to harvest a site's images anyway? Do you have permission to suck up their bandwidth this way?

If I catch you doing that on my site, I'll blacklist you...and my site watches for exactly that, so it likely will catch you.

ncsuapex · 08-08-2008, 02:37 PM

wget -r -l5 --no-parent -A.jpg www.xyz.com/collection/

the l5 is the number of levels it goes down. So l5 would go 5 levels down from collection

farslayer · 08-08-2008, 02:50 PM

come on.. scraping multiple images in this fashion is the fastest way to grow your Pr0n collection..

But seriously though, all kidding aside, have you looked at httrack ? it's another option you could try.

Quote:

httrack
Description: Copy websites to your computer (Offline browser)
HTTrack is an offline browser utility, allowing you to download a World Wide website from the Internet to a local directory, building
recursively all directories, getting html, images, and other files from the server to your computer.

HTTrack arranges the original site's relative link-structure.

taxtropel · 11-13-2008, 10:46 AM

If you know the range of URLs (image or otherwise) then using seq as above you get something like...

Code:

for i in $(seq 1 20)
  do
    wget http://mybuddies.site.org/blarg/filename_$i.blarg
done

and of course as seen on the command line

Code:

for i in $(seq 1 20); do wget http://mybuddies.site.org/blarg/filename_$i.blarg; done

And yeah. Seriously jiml8.
Your post was completely off topic.