LinuxQuestions.org
Review your favorite Linux distribution.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 11-28-2012, 03:36 PM   #1
Obscurious
LQ Newbie
 
Registered: Jun 2009
Distribution: Debian, RHEL, FreeBSD
Posts: 17

Rep: Reputation: 0
Web Server Killed By Bots Crawling


I have an Ubuntu server running a website. Once or twice a week apache will spawn so many instances my server will die. Looking through the access log all the traffic is from bots, google, baidu, yandex, msn, etc. I need these bots to crawl my site so I can't just deny them. Furthermore, the bots are GETing URLs that don't exist anymore. I.E. they are trying URLs from before I rolled out the new version of the site. Does anyone have any suggestions to mitigate this situation?
 
Old 11-28-2012, 06:47 PM   #2
Obscurious
LQ Newbie
 
Registered: Jun 2009
Distribution: Debian, RHEL, FreeBSD
Posts: 17

Original Poster
Rep: Reputation: 0
I was able to have my server handle high loads better by setting MaxClients in apache. I was also able to deny bots requesting bad URLs in robots.txt and was able to set Crawl-Delay to 20 seconds in robots.txt.
 
Old 11-29-2012, 08:23 AM   #3
Habitual
LQ Addict
 
Registered: Jan 2011
Posts: 8,350
Blog Entries: 11

Rep: Reputation: 2327Reputation: 2327Reputation: 2327Reputation: 2327Reputation: 2327Reputation: 2327Reputation: 2327Reputation: 2327Reputation: 2327Reputation: 2327Reputation: 2327
Quote:
Originally Posted by Obscurious View Post
I have an Ubuntu server running a website. Once or twice a week apache will spawn so many instances my server will die. Looking through the access log all the traffic is from bots, google, baidu, yandex, msn, etc. I need these bots to crawl my site so I can't just deny them. Furthermore, the bots are GETing URLs that don't exist anymore. I.E. they are trying URLs from before I rolled out the new version of the site. Does anyone have any suggestions to mitigate this situation?
Mitigate what exactly? Your "need" to have bots crawl your site?

"google, baidu, yandex, msn" your site and see what the results are and compare to referer reference in the logs.

Have a Great Day!
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
seeing a lot of hits by bots or people against my www server can i block them unix1adm Linux - Security 7 09-22-2010 08:21 AM
Blocking Web crawlers, bots, spiders, proxies, etc from private site areas haydenyoung Linux - Security 3 11-15-2008 11:03 AM
How to program Web Bots??? chutsu Programming 3 05-30-2006 01:10 PM
stopping bots from getting to my apache server Lleb_KCir Linux - Software 2 06-15-2004 01:48 PM


All times are GMT -5. The time now is 10:18 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration