I don't use robots.txt on my web server because I have read it's actually better security-wise not to have one at all.
What do web masters think about controlling who crawls?
Does it really make a difference if spiders can see what dirs you are keeping them out of?
Never come across robots.txt being considered naughty naughty. If you think about, without them they're gonna crawl all your directories anyway. Giving them a list saying "don't look here" isn't really encouraging them to look there, it just stops them indexing those directories. Maybe the author of the article was just a little paranoid. Anything you didn't want publicly available would be password protected anyway, yes ;) ?
|All times are GMT -5. The time now is 04:50 AM.|