I am looking for an easy way to find bots or wired user agents from my huge log file so i can block them but my access.log is very big and is not so easy to search each line
Any other way or any useful grep command?
At the moment i have this:
grep 'spider\|bot' access.log | sort -u -f >> bots.txt
Still trying to work out how to just print out the spider / bot name and remove the duplicates....
Looking also for some ideas for what else i can search for other than spider/bot as i don't know what else is bad or can cause huge load on my server....