Isn't robots.txt a file you can put on your webserver to prevent a bot from actually indexing it? Remember this from an old war game
.
Anyway I guess you'd first have to check what exactly you are blocking in the robots.txt file you have on your webserver. You might get a clue on what to do after that. I'm lost on the "mutation file list" though.
Found this though on Google:
Code:
In the case of articles, they can be represented in more than one site.They also look for “content evolution.In general 65% of web content does not change on a weekly basis. 0.8% will change completely on a weekly basis, such as a news site. They are looking at the “average page mutation” of a URL and a website. If too high, , they will filter out the duplicate results.
So from this what I get is this has something to do with making sure that content on your website doesnt get listed elsewhere. The mutation file list is probably something that will inform you if your content is infact robbed by the "entry" in the robots.txt and put on another site.
I'm just doing some "logical" guesswork here. So if someone finds this illogical or incorrect or knows otherwise please correct me.
Hope I helped though
.
Cheers
Arvind