Register a domain and help support LQ
Go Back > Forums > Linux Forums > Linux - Newbie
User Name
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!


  Search this Thread
Old 08-16-2009, 04:01 AM   #1
Registered: Mar 2009
Distribution: Slackware 13.0
Posts: 60

Rep: Reputation: 16
Unhappy wget, web crawler, web spider and web archiving

is wget considered a web crawler? I thought web crawler was supposed to march all around the internet and collect all the information?

Are there differences among wget, web crawler, web spider and web archiving?

thank you world-class experts @ LQ
Old 08-16-2009, 07:27 AM   #2
Registered: Jan 2007
Location: Canton, MI
Distribution: CentOS, SuSE, Red Hat, Debian, etc.
Posts: 703

Rep: Reputation: 97
I would not consider wget a web crawler. It is a tool that can be used to mirror one or more
websites, so it could be used to make your own web crawler.

Web crawler and web spider pretty much mean the same thing. They pull the content from one
or more websites. This could be part of an archiving process, but it could also be to check
that links are good or to index the data on the page.

A web archiver uses a web crawler to make a copy of one or more websites, and these pages are
then made available on a web server.


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off

Similar Threads
Thread Thread Starter Forum Replies Last Post
Benchmarking Web Spider Geneset Programming 2 04-28-2009 12:17 PM
web crawler/viewer microsoft/linux General 10 05-07-2006 02:31 AM
wget as web spider/crawler kpachopoulos Linux - Software 2 08-27-2005 12:58 PM
I need a web crawler and indexer for linux jrenzi Programming 2 10-28-2004 01:11 AM
linux web crawler demmylls Linux - Software 2 03-06-2004 08:56 AM

All times are GMT -5. The time now is 07:29 PM.

Main Menu
Write for LQ is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration