Reaching MaxClients, why ?

linuxcroco · 10-11-2009, 09:32 AM

Hello,

I've been experiencing an issue from time to time:

For no apparent reason ( no traffic spikes, no other scripting identifiable errors), I get an entry in the errog log that MaxClients has been reached and of course httpd stops responding.

There is no load on the server, only a sudden abrupt jump in tasks: there are usually 270~300 processes running but as I mentioned, from time to time, it suddenly jumps to 900+ and crashes apache.

I have configured Apache with a MaxCLients of 800 so far, I have 16 GB of RAM and running a dual Xeon Quadcore setup with CentOS 64 bit.

Could you please advise on some methodical approach to identify this issue ?

Thank you kindly for your time.

Kind regards,

LinuxCroco

chrism01 · 10-12-2009, 12:25 AM

Try Apache's access_log & error_log.
Also /var/log/messages; see where the traffic is coming from.
If it's some kind of break-in attempt, try mod_throttle or fail2ban.

linuxcroco · 10-14-2009, 02:40 AM

Quote:

Originally Posted by chrism01

Try Apache's access_log & error_log.
Also /var/log/messages; see where the traffic is coming from.
If it's some kind of break-in attempt, try mod_throttle or fail2ban.

Chris, you were correct. I missed the /var/log/messages log.

This is what I found there:

Oct 13 19:18:46 web kernel: possible SYN flooding on port 80. Sending cookies.
Oct 13 19:19:04 web kernel: TCPv6: Possible SYN flooding on port 80. Sending cookies.

Although It's defending against this possible SYN flood by sending cookies, apache gets thrown down. I'm reaching MaxClients and the server is unable to serve new requests.

Any help on how to continue ?

Thanks !

linuxcroco · 10-14-2009, 06:18 AM

Another statistic that troubles me is this:

Running "netstat -s" I notice:

Tcp:
3403236 active connections openings
287134948 passive connection openings
3632799 failed connection attempts
3598067 connection resets received

There's a huge number of failed connection attempts.

chrism01 · 10-14-2009, 08:54 PM

Try fail2ban http://www.fail2ban.org/wiki/index.php/Main_Page.
You may want to disable IPv6 unless you need it (unlikely):

Quote:

To disable IPv6, we must prevent the loading of the module by adding the following two lines to /etc/modprobe.conf (& reboot system):

alias net-pf-10 off
alias ipv6 off

I'd also check your webpages; if you are getting that much traffic, they've prob been hacked to supply other info... Turn off Apache first, then figure out the problem.

Edit: This on today's slashdot.org: help from google for webmasters who (may) have been compromised:

Quote:

"In an effort to promote the 'general health of the Web,' Google will send Webmasters snippets of malicious code (http://googleonlinesecurity.blogspot...e-malware.html) in the hopes of getting infected Web sites cleaned up faster. The new information will appear as part of Google's Webmaster Tools, a suite of tools that provide data about a Web site, such as site visits. 'We understand the frustration of Webmasters whose sites have been compromised without their knowledge and who discover that their site has been flagged,' wrote Lucas Ballard on Google's online security blog. To Webmasters who are registered with Google, the company will send them an email notifying them of suspicious content along with a list of the affected pages. They'll also be able to see part of the malicious code." Another of the new Webmaster Tools is Fetch as Googlebot ( http://searchengineland.com/see-what...our-site-27623) , which shows you a page as Google's crawler sees it. This should allow Webmasters to see malicious code that bad guys have hidden on their sites via "cloaking," among other benefits.

You should prob look into those.

linuxcroco · 10-15-2009, 06:02 AM

Thank you.

I have done basic securing of my Linux box. I admit, it could have been done better, but I do not see any signs of intrusions. SSH is secured/

Why did you make the assumption that the box might be hacked ? I can't see any causality between the SYN flood and possible intrusions. Could you please share your thoughts ?

Thanks !

chrism01 · 10-15-2009, 08:53 PM

SYN is the first TCP pkt sent when trying to connect to a system. Unless you've got some serious info eg, winning lottery nums for next week, why would you suddenly get massive floods like that. There's also a TCP SYN Flood DOS (Denial Of Service) attack: http://en.wikipedia.org/wiki/SYN_flood.
Note I only said 'If it's some kind of break-in attempt', 'if' being the key word there.
From your own logs, the kernel says it suspects the SYN Flood attack; that's why it says its 'sending cookies' as mentioned/recommended in the Wiki article.