LinuxQuestions.org
LinuxAnswers - the LQ Linux tutorial section.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices



Reply
 
Search this Thread
Old 08-16-2009, 05:01 AM   #1
beckettisdogg
Member
 
Registered: Mar 2009
Distribution: Slackware 13.0
Posts: 60

Rep: Reputation: 16
Unhappy wget, web crawler, web spider and web archiving


is wget considered a web crawler? I thought web crawler was supposed to march all around the internet and collect all the information?

Are there differences among wget, web crawler, web spider and web archiving?

thank you world-class experts @ LQ
 
Old 08-16-2009, 08:27 AM   #2
carltm
Member
 
Registered: Jan 2007
Location: Canton, MI
Distribution: CentOS, SuSE, Red Hat, Debian, etc.
Posts: 700

Rep: Reputation: 96
I would not consider wget a web crawler. It is a tool that can be used to mirror one or more
websites, so it could be used to make your own web crawler.

Web crawler and web spider pretty much mean the same thing. They pull the content from one
or more websites. This could be part of an archiving process, but it could also be to check
that links are good or to index the data on the page.

A web archiver uses a web crawler to make a copy of one or more websites, and these pages are
then made available on a web server.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Benchmarking Web Spider Geneset Programming 2 04-28-2009 01:17 PM
web crawler/viewer microsoft/linux General 10 05-07-2006 03:31 AM
wget as web spider/crawler kpachopoulos Linux - Software 2 08-27-2005 01:58 PM
I need a web crawler and indexer for linux jrenzi Programming 2 10-28-2004 02:11 AM
linux web crawler demmylls Linux - Software 2 03-06-2004 09:56 AM


All times are GMT -5. The time now is 05:28 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration