I need to stop the viewing of robots.txt on my website. I get the contents of the file displayed in my browser when I issue the command:
http://www.inwoon.net/robots.txt
Help me stop this as it displays all the directories I don't want them to go to. I greatly appreciate the help as I am right at the level of Linux knowledge to be dangerous, capable but still dangerous.
Thanks much;

Welcome on LQ,

robots.txt list directories or files that should not be indexed by webcrawler. However the rules are not mandatory for those programs. Consequently the robots.txt must be available to the public to give the crawlers a chance to read your rules. Furthermore robots.txt-rules do never apply to web browsers.

Thus if you want not show this file, make it unreadable for all clients or for certain user agents. If the directories contain secret contents, why are they accessible from the internet? Perhaps you should require the user to authenticate before accessing this directories.

Thank you for the response. Are you saying I should popup an authentication dialog before letting anyone view the contents of a directory?
Is this something I have to do in apache. I am hosting my own web server with my own static ip in house. I have total access to the configuration. If you could direct me to relevant documentation on how to set this idea up I would be forever in your debt.
Once again thank you for the answer!!

Take a look at
http://forums.ukwebmasterworld.com/p...ed-robots.html

@repo: Thanks that's an awesome lead. I will implement and report back.
That still leaves the problem of the directories being available to list in a browser.
ie. http://www.inwoon.net/EnergyLibrary/
While I want this one to be displayed all of the other I don't want them to look at.
Any suggestions will be greatly appreciated.

To disable directory listing in apache:
http://www.ducea.com/2006/06/26/apac...ctory-indexes/
http://httpd.apache.org/docs/1.3/misc/FAQ.html