Welcome to the most active Linux Forum on the web.
Go Back > Forums > Linux Forums > Linux - Software
User Name
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.


  Search this Thread
Old 11-11-2005, 12:36 PM   #1
Registered: Mar 2003
Location: SLC, UT, US
Distribution: Fedora 12
Posts: 34

Rep: Reputation: 15

I don't use robots.txt on my web server because I have read it's actually better security-wise not to have one at all.

What do web masters think about controlling who crawls?

Does it really make a difference if spiders can see what dirs you are keeping them out of?
Old 11-11-2005, 02:32 PM   #2
Senior Member
Registered: Jan 2002
Location: St Louis, MO
Distribution: Ubuntu
Posts: 1,284

Rep: Reputation: 47
Never come across robots.txt being considered naughty naughty. If you think about, without them they're gonna crawl all your directories anyway. Giving them a list saying "don't look here" isn't really encouraging them to look there, it just stops them indexing those directories. Maybe the author of the article was just a little paranoid. Anything you didn't want publicly available would be password protected anyway, yes ?


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off

Similar Threads
Thread Thread Starter Forum Replies Last Post
Linux Powered Robots. rvijay Linux - News 8 07-31-2005 09:01 AM
txt >> mysql LLS Programming 2 06-21-2005 11:15 AM
Stop the Google robots lothario Linux - Software 2 04-18-2005 05:47 AM
configuring robots.txt jc materi Linux - Security 1 04-09-2005 10:37 AM
The Robots movie ahz General 4 03-24-2005 04:31 PM > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 05:58 AM.

Main Menu
Write for LQ is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration