Lots of nonsense requests for twiki server

Zilvermeeuw · 10-18-2006, 04:00 PM

I get lots of requests for my twiki server

Here is a part of the logfile:

Code:

74.6.68.18 - - [17/Oct/2006:06:17:53 +0200] "GET /twiki/bin/changes/Main HTTP/1.0" 404 399
74.6.68.125 - - [17/Oct/2006:06:18:19 +0200] "GET /twiki/bin/oops.pl/TWiki/TWikiVariables?template=oopsmore&param1=80&param2=80 HTTP/1.0" 200 3908
4
66.249.66.230 - - [17/Oct/2006:06:18:52 +0200] "GET /wiki_mediawiki/index.php?title=Speciaal:Recentchanges&from=20061004190657&days=3&limit=500&hi
deminor=1&hideliu=1 HTTP/1.1" 404 401
74.6.75.15 - - [17/Oct/2006:06:18:53 +0200] "GET /twiki/bin/oops.pl/Sandbox/WebStatistics?template=backlinksweb HTTP/1.0" 200 17315
74.6.67.212 - - [17/Oct/2006:06:20:11 +0200] "GET /twiki/bin/view.pl/Main/RichardVinkeLeftBar HTTP/1.0" 200 16390
66.249.66.230 - - [17/Oct/2006:06:21:12 +0200] "GET /wiki_mediawiki/index.php?title=Speciaal:Recentchanges&from=20061004202353&days=3&limit=500&hi
deminor=1&hideliu=1 HTTP/1.1" 404 401
72.30.226.222 - - [17/Oct/2006:06:21:44 +0200] "GET /twiki/bin/view.pl/TWiki/TWikiTip021 HTTP/1.0" 200 18061
74.6.70.221 - - [17/Oct/2006:06:22:32 +0200] "GET /twiki/bin/login.pl/TWiki/ResetPassword?origurl=/twiki/bin/view.pl/TWiki/ResetPassword%3Frev%3D9
 HTTP/1.0" 200 8399

I run TWiki on apachefriends, fedora 5, pentium2 300 MHz, 480 MB. All these requests are slowing down my server. (I know, I need a faster server....)

Can i do anything against these requests?

////// · 10-18-2006, 05:10 PM

Hi, to me those look like a search engine crawlers.

lj602313.inktomisearch.com = 72.30.226.222
crawl-66-249-66-230.googlebot.com = 66.249.66.230

I dont know if you should block them or not, maybe they leave you alone after some time.

Cheers

////

jcookeman · 10-19-2006, 07:18 AM

http://twiki.org/cgi-bin/view/Plugins/BlackListPlugin

Zilvermeeuw · 10-19-2006, 01:25 PM

Thanks!

I now know the source: Crawlers.

This morning I created a robot.txt file in the root and this is the only file which is accessed!

So: Problem solved!