is there a linux program that downloads an entire website?

lemuel · 01-04-2005, 10:34 PM

Elomis · 01-04-2005, 10:48 PM

If you are using firefox (and you should be) there is an extension called Scrapbook that lets you snake a website for offline viewing. Keep in mind the importance of respecting people's intellectual property rights when investigating this sort of thing.

:-)

E

uman · 01-04-2005, 10:57 PM

wget -r

lemuel · 01-04-2005, 11:00 PM

cool!
(although something that downloads an entire website in one click would be way cooler)

I'll go check that out.

thanks!

Shade · 01-05-2005, 01:51 AM

wget is great for this sort of thing. You can also use it to do things like grab all files with a certain extension in a directory on a website, or just mirror an entire site...

man wget

--Shade

lemuel · 01-05-2005, 03:00 AM

um...

what does it mean to mirror an entire site?

scuzzman · 01-05-2005, 03:47 AM

To grab an entire site, directory structure and all, and repost it for viewing to alleviate some of the stress from the original server.

lemuel · 01-05-2005, 04:39 AM

oh ok thanks for the info

I already tried wget using options -r -k -p on a website and found out there are still other directories it didn't download. What's wrong?

MasterC · 01-05-2005, 04:46 AM

wget -m is for mirroring

Cool

lemuel · 01-05-2005, 04:56 AM

I also tried wget -m. how come there are other directories within the website that werent downloaded?

MasterC · 01-06-2005, 02:10 AM

Could be a robot.txt file? Sometimes people will set it up to deliberately stop people from recursively downloading every directory on their website... .htaccess...

Cool