LinuxQuestions.org
Support LQ: Use code LQ3 and save $3 on Domain Registration
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices



Reply
 
Search this Thread
Old 10-11-2009, 10:32 AM   #1
linuxcroco
LQ Newbie
 
Registered: May 2009
Posts: 10

Rep: Reputation: 0
Reaching MaxClients, why ?


Hello,

I've been experiencing an issue from time to time:

For no apparent reason ( no traffic spikes, no other scripting identifiable errors), I get an entry in the errog log that MaxClients has been reached and of course httpd stops responding.

There is no load on the server, only a sudden abrupt jump in tasks: there are usually 270~300 processes running but as I mentioned, from time to time, it suddenly jumps to 900+ and crashes apache.

I have configured Apache with a MaxCLients of 800 so far, I have 16 GB of RAM and running a dual Xeon Quadcore setup with CentOS 64 bit.

Could you please advise on some methodical approach to identify this issue ?


Thank you kindly for your time.

Kind regards,

LinuxCroco
 
Old 10-12-2009, 01:25 AM   #2
chrism01
Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.6, Centos 5.10
Posts: 16,324

Rep: Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041
Try Apache's access_log & error_log.
Also /var/log/messages; see where the traffic is coming from.
If it's some kind of break-in attempt, try mod_throttle or fail2ban.
 
Old 10-14-2009, 03:40 AM   #3
linuxcroco
LQ Newbie
 
Registered: May 2009
Posts: 10

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by chrism01 View Post
Try Apache's access_log & error_log.
Also /var/log/messages; see where the traffic is coming from.
If it's some kind of break-in attempt, try mod_throttle or fail2ban.
Chris, you were correct. I missed the /var/log/messages log.

This is what I found there:

Oct 13 19:18:46 web kernel: possible SYN flooding on port 80. Sending cookies.
Oct 13 19:19:04 web kernel: TCPv6: Possible SYN flooding on port 80. Sending cookies.

Although It's defending against this possible SYN flood by sending cookies, apache gets thrown down. I'm reaching MaxClients and the server is unable to serve new requests.

Any help on how to continue ?

Thanks !
 
Old 10-14-2009, 07:18 AM   #4
linuxcroco
LQ Newbie
 
Registered: May 2009
Posts: 10

Original Poster
Rep: Reputation: 0
Another statistic that troubles me is this:

Running "netstat -s" I notice:

Tcp:
3403236 active connections openings
287134948 passive connection openings
3632799 failed connection attempts
3598067 connection resets received


There's a huge number of failed connection attempts.
 
Old 10-14-2009, 09:54 PM   #5
chrism01
Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.6, Centos 5.10
Posts: 16,324

Rep: Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041
Try fail2ban http://www.fail2ban.org/wiki/index.php/Main_Page.
You may want to disable IPv6 unless you need it (unlikely):

Quote:
To disable IPv6, we must prevent the loading of the module by adding the following two lines to /etc/modprobe.conf (& reboot system):

alias net-pf-10 off
alias ipv6 off
I'd also check your webpages; if you are getting that much traffic, they've prob been hacked to supply other info... Turn off Apache first, then figure out the problem.


Edit: This on today's slashdot.org: help from google for webmasters who (may) have been compromised:
Quote:
"In an effort to promote the 'general health of the Web,' Google will send Webmasters snippets of malicious code (http://googleonlinesecurity.blogspot...e-malware.html) in the hopes of getting infected Web sites cleaned up faster. The new information will appear as part of Google's Webmaster Tools, a suite of tools that provide data about a Web site, such as site visits. 'We understand the frustration of Webmasters whose sites have been compromised without their knowledge and who discover that their site has been flagged,' wrote Lucas Ballard on Google's online security blog. To Webmasters who are registered with Google, the company will send them an email notifying them of suspicious content along with a list of the affected pages. They'll also be able to see part of the malicious code." Another of the new Webmaster Tools is Fetch as Googlebot ( http://searchengineland.com/see-what...our-site-27623) , which shows you a page as Google's crawler sees it. This should allow Webmasters to see malicious code that bad guys have hidden on their sites via "cloaking," among other benefits.
You should prob look into those.

Last edited by chrism01; 10-15-2009 at 01:42 AM. Reason: Added info about google tools
 
Old 10-15-2009, 07:02 AM   #6
linuxcroco
LQ Newbie
 
Registered: May 2009
Posts: 10

Original Poster
Rep: Reputation: 0
Thank you.


I have done basic securing of my Linux box. I admit, it could have been done better, but I do not see any signs of intrusions. SSH is secured/

Why did you make the assumption that the box might be hacked ? I can't see any causality between the SYN flood and possible intrusions. Could you please share your thoughts ?


Thanks !
 
Old 10-15-2009, 09:53 PM   #7
chrism01
Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.6, Centos 5.10
Posts: 16,324

Rep: Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041
SYN is the first TCP pkt sent when trying to connect to a system. Unless you've got some serious info eg, winning lottery nums for next week, why would you suddenly get massive floods like that. There's also a TCP SYN Flood DOS (Denial Of Service) attack: http://en.wikipedia.org/wiki/SYN_flood.
Note I only said 'If it's some kind of break-in attempt', 'if' being the key word there.
From your own logs, the kernel says it suspects the SYN Flood attack; that's why it says its 'sending cookies' as mentioned/recommended in the Wiki article.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
http maxclients saturate often uks Linux - Server 2 10-01-2009 09:34 PM
How many more that MaxClients in apache? knobby Linux - Server 1 09-18-2008 08:16 PM
how many MaxClients should i give? zakir_123 Linux - Server 1 06-15-2008 12:17 AM
MaxClients setting lavinya Linux - Server 2 01-04-2008 03:08 PM
Apache MaxClients mphllps Linux - Software 2 03-13-2006 09:46 AM


All times are GMT -5. The time now is 03:41 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration