configuring robots.txt

jc materi · 04-09-2005, 09:30 AM

Apache has a default file in the document root called robots.txt restricting the scan of search bots. I have kept all the defaults but the line "/all_our_e-mail_addresses" must be changed and I'm not sure what should go there. Any examples would be appreciated (without using actual email addresses, of course).

macemoneta · 04-09-2005, 10:37 AM

That's a sample directory or file that you would want to exclude (one that contained all your email addresses - a confusing choice for an example). Some better examples:

User-agent: *
Disallow: /some/secret/directory
Disallow: /tmp/
Disallow: /my_important_file

There's more information on the layout of this file here