LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Linux - Software (http://www.linuxquestions.org/questions/linux-software-2/)
-   -   robots.txt (http://www.linuxquestions.org/questions/linux-software-2/robots-txt-382180/)

paleogryph 11-11-2005 12:36 PM

robots.txt
 
I don't use robots.txt on my web server because I have read it's actually better security-wise not to have one at all.

What do web masters think about controlling who crawls?

Does it really make a difference if spiders can see what dirs you are keeping them out of?

fouldsy 11-11-2005 02:32 PM

Never come across robots.txt being considered naughty naughty. If you think about, without them they're gonna crawl all your directories anyway. Giving them a list saying "don't look here" isn't really encouraging them to look there, it just stops them indexing those directories. Maybe the author of the article was just a little paranoid. Anything you didn't want publicly available would be password protected anyway, yes ;) ?


All times are GMT -5. The time now is 03:36 PM.