I'm sure it's the noscript jail. This is the content of the ban email that I receive.
The IP 18.104.22.168 has just been banned by Fail2Ban after
6 attempts against apache-noscript.
Here are more information about 22.214.171.124:
# Query terms are ambiguous. The query is assumed to be:
# "n 126.96.36.199"
# Use "?" to get help.
# The following results may also be obtained via:
NetRange: 188.8.131.52 - 184.108.40.206
NetType: Direct Allocation
OrgName: Google Inc.
Address: 1600 Amphitheatre Parkway
City: Mountain View
OrgAbuseName: Google Inc
OrgTechName: Google Inc
# available at: https://www.arin.net/whois_tou.html
I'd prefer not to add exceptions based on the user-agent alone because this information is easily spoofed. I would like to provide an exception to the noscript jail based on remote addresses that can be reliably attributed to Google's bots.
As for scanning for errors and adding files to a robots.txt, I understand how robots.txt work and I could easily formulate a PHP script to write more detail to the robots.txt file, but I'm concerned a) about how complex it would be to efficiently scan apache logs (a very large amount of data) and b) about my robots.txt file growing without bound due to varying query strings or unique-but-non-existent urls, etc.