LinuxQuestions.org
Review your favorite Linux distribution.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices


Reply
  Search this Thread
Old 01-04-2005, 11:34 PM   #1
lemuel
Member
 
Registered: Nov 2003
Location: Cebu, Philippines
Distribution: Ubuntu
Posts: 73

Rep: Reputation: 15
is there a linux program that downloads an entire website?


is there a linux program that downloads an entire website?
 
Old 01-04-2005, 11:48 PM   #2
Elomis
Member
 
Registered: Dec 2004
Location: Sydney
Distribution: SUSE
Posts: 89

Rep: Reputation: 15
If you are using firefox (and you should be) there is an extension called Scrapbook that lets you snake a website for offline viewing. Keep in mind the importance of respecting people's intellectual property rights when investigating this sort of thing.

:-)

E
 
Old 01-04-2005, 11:57 PM   #3
uman
Member
 
Registered: Sep 2004
Distribution: Gentoo
Posts: 134

Rep: Reputation: 15
wget -r
 
Old 01-05-2005, 12:00 AM   #4
lemuel
Member
 
Registered: Nov 2003
Location: Cebu, Philippines
Distribution: Ubuntu
Posts: 73

Original Poster
Rep: Reputation: 15
cool!
(although something that downloads an entire website in one click would be way cooler)

I'll go check that out.

thanks!
 
Old 01-05-2005, 02:51 AM   #5
Shade
Senior Member
 
Registered: Mar 2003
Location: Burke, VA
Distribution: RHEL, Slackware, Ubuntu, Fedora
Posts: 1,418
Blog Entries: 1

Rep: Reputation: 46
wget is great for this sort of thing. You can also use it to do things like grab all files with a certain extension in a directory on a website, or just mirror an entire site...

man wget



--Shade
 
Old 01-05-2005, 04:00 AM   #6
lemuel
Member
 
Registered: Nov 2003
Location: Cebu, Philippines
Distribution: Ubuntu
Posts: 73

Original Poster
Rep: Reputation: 15
um...

what does it mean to mirror an entire site?
 
Old 01-05-2005, 04:47 AM   #7
scuzzman
Senior Member
 
Registered: May 2004
Location: Hilliard, Ohio, USA
Distribution: Slackware, Kubuntu
Posts: 1,851

Rep: Reputation: 47
To grab an entire site, directory structure and all, and repost it for viewing to alleviate some of the stress from the original server.
 
Old 01-05-2005, 05:39 AM   #8
lemuel
Member
 
Registered: Nov 2003
Location: Cebu, Philippines
Distribution: Ubuntu
Posts: 73

Original Poster
Rep: Reputation: 15
oh ok thanks for the info

I already tried wget using options -r -k -p on a website and found out there are still other directories it didn't download. What's wrong?
 
Old 01-05-2005, 05:46 AM   #9
MasterC
LQ Guru
 
Registered: Mar 2002
Location: Salt Lake City, UT - USA
Distribution: Gentoo ; LFS ; Kubuntu
Posts: 12,612

Rep: Reputation: 68
wget -m is for mirroring

Cool
 
Old 01-05-2005, 05:56 AM   #10
lemuel
Member
 
Registered: Nov 2003
Location: Cebu, Philippines
Distribution: Ubuntu
Posts: 73

Original Poster
Rep: Reputation: 15
I also tried wget -m. how come there are other directories within the website that werent downloaded?
 
Old 01-06-2005, 03:10 AM   #11
MasterC
LQ Guru
 
Registered: Mar 2002
Location: Salt Lake City, UT - USA
Distribution: Gentoo ; LFS ; Kubuntu
Posts: 12,612

Rep: Reputation: 68
Could be a robot.txt file? Sometimes people will set it up to deliberately stop people from recursively downloading every directory on their website... .htaccess...

Cool
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
how to copy entire website using linux ngan_yine Linux - Newbie 3 03-09-2005 05:04 PM
What program is needed for making own website js72 Linux - Software 7 04-09-2004 02:45 AM
music downloads program nick2003 Linux - Software 2 11-14-2003 11:16 AM
program for website killi Linux - Software 7 06-27-2003 01:49 PM
Start a program from website jeucken Linux - Networking 2 08-07-2002 07:35 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 05:15 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration