hi,
im running vps with debian lenny, with apache2 webserver, memcached, postgres and django-framework using mod-wsgi.
Some of the pages served by django take long to generate (aprox 15 s), but there is also the memcached, compensating this.

My problem is that, when a robot visits the site, it starts traversing the site visiting all the pages and also the nongenerated, thus slowing it down to point where it is not responding.


What Im looking for is a solution to identify that the request comes from a robot (user-agent, ips, etc) and limit the resources, so that f.e. only one thread serves the robot etc...

Is it possible? has anyone come across similar problem? Any other solution?

How about robots.txt (and a sitemap) taking them to optimal 'static' versions? Or, failing that, using .htaccess to steer them elsewhere?

well, i need them to get the correct content and all sites, because I want to be indexed in search results. So I cannot block them or give them different results, because google would penalize it.