-   Linux - Software (
-   -   robots.txt (

paleogryph 11-11-2005 12:36 PM

I don't use robots.txt on my web server because I have read it's actually better security-wise not to have one at all.

What do web masters think about controlling who crawls?

Does it really make a difference if spiders can see what dirs you are keeping them out of?

fouldsy 11-11-2005 02:32 PM

Never come across robots.txt being considered naughty naughty. If you think about, without them they're gonna crawl all your directories anyway. Giving them a list saying "don't look here" isn't really encouraging them to look there, it just stops them indexing those directories. Maybe the author of the article was just a little paranoid. Anything you didn't want publicly available would be password protected anyway, yes ;) ?

All times are GMT -5. The time now is 05:18 AM.