Apache load problem

Sheridan · 02-11-2008, 03:18 AM

I'm operating a small Apache custer of 8 server nodes.

Randomly with all of the nodes, the following effect keeps coming back:

For some time (varies), the backend server operates examplary, serves thousands and thousands of request on an average load of 4,5-6. (The frontend is HAProxy in roundrobin mode.) Then, for no apparent reason, load skyrockets to unbelivably high values, like 115-180 or so, then the only way to save the system from crashing is with killing all apache processes with SIGKILL, wait for 20 seconds, then issue an apache service start...

I've written a script to monitor this kind of screwup, but I'd like to know the reason behind this.

What can cause something like this?

Like I said there's no apparent pattern behind this... It's sometimes happening to the same server, sometimes another... And killing the Apache processes always helps so I figure it must be something with Apache.

Sheridan · 02-11-2008, 03:45 AM

Also... when it's happening, I see lot of "uninterruptible sleep" apache processes...

Code:

apache   30594  2.1  0.9  70332 31720 ?        S    11:39   0:04 /usr/sbin/httpd
apache   30595  1.8  1.1  76128 39504 ?        D    11:39   0:04 /usr/sbin/httpd
apache   30598  2.0  0.9  69912 33072 ?        D    11:39   0:04 /usr/sbin/httpd
apache   30599  2.1  1.0  70588 33920 ?        D    11:39   0:04 /usr/sbin/httpd
apache   30603  2.1  0.9  69276 31712 ?        D    11:39   0:04 /usr/sbin/httpd
apache   30607  2.0  0.9  71676 33352 ?        D    11:39   0:04 /usr/sbin/httpd
apache   30699  1.7  0.7  70320 26496 ?        D    11:40   0:03 /usr/sbin/httpd
apache   30707  1.4  0.7  69444 25120 ?        R    11:40   0:02 /usr/sbin/httpd
apache   30708  1.3  0.7  69532 24976 ?        S    11:40   0:02 /usr/sbin/httpd
apache   30709  1.6  1.0  78324 35472 ?        D    11:40   0:02 /usr/sbin/httpd
apache   30710  1.6  0.8  71604 27496 ?        D    11:40   0:03 /usr/sbin/httpd
apache   30711  1.7  0.8  71892 27648 ?        D    11:40   0:03 /usr/sbin/httpd
apache   30712  1.4  0.7  69252 23988 ?        D    11:40   0:02 /usr/sbin/httpd
apache   30713  1.3  0.7  70072 25580 ?        D    11:40   0:02 /usr/sbin/httpd
apache   30714  1.6  0.8  72156 29204 ?        D    11:40   0:03 /usr/sbin/httpd
apache   30715  1.7  0.7  70716 25464 ?        S    11:40   0:03 /usr/sbin/httpd
apache   30716  1.4  0.8  70872 29556 ?        D    11:40   0:02 /usr/sbin/httpd
apache   30717  1.6  0.8  69500 29572 ?        S    11:40   0:02 /usr/sbin/httpd
apache   30719  1.6  0.7  69972 25748 ?        S    11:40   0:03 /usr/sbin/httpd
apache   30720  1.7  0.9  70492 31016 ?        S    11:40   0:03 /usr/sbin/httpd
apache   30721  1.5  0.9  77524 30620 ?        S    11:40   0:02 /usr/sbin/httpd