Quote:
Originally Posted by newbie14
Do you want me to go through all the files manually and extract .html logs?
|
I don't know what you mean by ".html logs" but if you mean Apaches access_log then if you're satisfied by what your own answer provided then by all means go do something else. If you don't understand or are curious as to what was going then you investigate further.
Quote:
Originally Posted by newbie14
should I put in /var/www/html or any other location?
|
Note spiders look for "/robots.txt" so, yes, in whatever place your web sites docroot (aka its "/") resides. That may be /var/www/html/, yes.