Our web server seems to be getting a ton of traffic from Microsoft bingbots. We are getting close to 100,000 lines every day in the http access log of these requests.

I could see this if the site was being indexed for the first time but to keep getting these every day seems excessive. I see a lot of activity from Google and Yahoo etc. but nowhere near as much as from Microsoft.

Are these types of numbers normal? Any ideas on slowing them down?

Thanks.
Dave.

You might want to post some snippets of your logs - obfuscated for your privacy ( remove your IP, hostname etc).

100000 lines is excessive. Maybe one or two a day, okay.

Do you have salacious pic's of Microsoft staff or something ?

Also, alot of bots fake their 'user agents'. Do a lookup on the ip that the 'bingbot' is coming from - it may not be Microsoft.

Here are a few lines from the log. DNSStuff shows the ip address as belonging to Microsoft. They are also not respecting disallowed paths in robots.txt - /profile should not be indexed.

Sorry, not allowed to discuss the "pics" situation. :-)

Thanks.
Dave.

You could exclude bingbot altogether in your robots.txt file:
My (limited) experience is that bing doesn't direct enough human traffic to my site to make its robot visits worthwhile

1 members found this post helpful.

They should. If not can you show us your robots.txt file?
Sanitize or obfuscate sensitive info, if necessary.

You could drop 157.55.0.0/16 and that's one less network block to worry about that Microsoft is assigned.'
See http://www.iplists.com/misc.txt for a list of the usual suspects.

Here is the robots.txt file

# Paths (clean URLs)
Disallow: /admin/
Disallow: /comment/reply/
Disallow: /contact/
Disallow: /logout/
Disallow: /node/add/
Disallow: /search/
Disallow: /user/register/
Disallow: /user/password/
Disallow: /user/login/
Disallow: /user/
Disallow: /profile/

# Specific File Paths of exported files
Disallow: /sites/default/files/file*

# Paths (no clean URLs)
Disallow: /?q=admin/
Disallow: /?q=comment/reply/
Disallow: /?q=contact/
Disallow: /?q=logout/
Disallow: /?q=node/add/
Disallow: /?q=search/
Disallow: /?q=user/password/
Disallow: /?q=user/register/
Disallow: /?q=user/login/
Disallow: /?q=user/
Disallow: /?q=profile/



# Paths (clean URLs) fixed!
Disallow: /admin
Disallow: /comment/reply
Disallow: /contact
Disallow: /logout
Disallow: /node/add
Disallow: /search
Disallow: /user/register
Disallow: /user/password
Disallow: /user/login
Disallow: /user/
Disallow: /profile/

# Paths (no clean URLs) fixed!
Disallow: /?q=admin
Disallow: /?q=comment/reply
Disallow: /?q=contact
Disallow: /?q=logout
Disallow: /?q=node/add
Disallow: /?q=search
Disallow: /?q=user/password
Disallow: /?q=user/register
Disallow: /?q=user/login
Disallow: /?q=user/
Disallow: /?q=profile/

Did you know that 'robots.txt' is on the "honour system" ? Bingbot, or anyone else can just ignore it.

I suggest doing as Habitual said "You could drop 157.55.0.0/16" or perhaps invest some time in Fail2ban, a software gatekeeper that will automatically drop offending IP's. It is available with most distributions packages.

http://www.fail2ban.org/wiki/index.php/Main_Page

You have no user-agent, so no wonder it "doesn't work".

Use:
User-agent: bingbot
or
User-agent: *

in the robots.txt
Near the top, I suppose, that's what all my stanzas in robots.txt do:

See also Example Robots.txt Format section.
and if they still misbehave, they get my foot up their ass network/CIDR

Hope that's helpful.

I do know that robots.txt is not mandatory. I hate to block that big a range of Microsoft ip addresses and I want Bing to accurately index our site but this much traffic is crazy. I appreciate all the help and suggestions.

Thanks.
Dave.

add the range to fail2ban
or

i still like the Apache "Allow,Deny "
< >
-- options
Order Allow, Deny
Allow from all
Deny from 123.456.789.123
</ >

the list can get long

but for a few things it works

http://httpd.apache.org/docs/2.4/mod...pat.html#order

Do you own the server or have access to iptables?
I'd just rate limit them.
With the policy of drop,
This will accept 5 new packets every minute.

Dave: try this then to slow it down a bit...
but... This will give it unrestricted access to your site. You may wish to look at and
evaluate your site directory structure to see if there's anything you don't want them to crawl.

See http://en.wikipedia.org/wiki/Robots_...elay_directive for examples and explanation.
See http://www.fail2ban.org/wiki/index.php/Whitelist for whitelisting CIDR networks