LinuxQuestions.org
Review your favorite Linux distribution.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices



Reply
 
Search this Thread
Old 11-11-2005, 01:36 PM   #1
paleogryph
Member
 
Registered: Mar 2003
Location: SLC, UT, US
Distribution: Fedora 12
Posts: 34

Rep: Reputation: 15
robots.txt


I don't use robots.txt on my web server because I have read it's actually better security-wise not to have one at all.

What do web masters think about controlling who crawls?

Does it really make a difference if spiders can see what dirs you are keeping them out of?
 
Old 11-11-2005, 03:32 PM   #2
fouldsy
Senior Member
 
Registered: Jan 2002
Location: St Louis, MO
Distribution: Ubuntu
Posts: 1,284

Rep: Reputation: 47
Never come across robots.txt being considered naughty naughty. If you think about, without them they're gonna crawl all your directories anyway. Giving them a list saying "don't look here" isn't really encouraging them to look there, it just stops them indexing those directories. Maybe the author of the article was just a little paranoid. Anything you didn't want publicly available would be password protected anyway, yes ?
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Linux Powered Robots. rvijay Linux - News 8 07-31-2005 10:01 AM
txt >> mysql LLS Programming 2 06-21-2005 12:15 PM
Stop the Google robots lothario Linux - Software 2 04-18-2005 06:47 AM
configuring robots.txt jc materi Linux - Security 1 04-09-2005 11:37 AM
The Robots movie ahz General 4 03-24-2005 05:31 PM


All times are GMT -5. The time now is 01:38 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration