LinuxQuestions.org - Apache load problem

- Linux - Server (https://www.linuxquestions.org/questions/linux-server-73/)

- - Apache load problem (https://www.linuxquestions.org/questions/linux-server-73/apache-load-problem-620246/)

Apache load problem

I'm operating a small Apache custer of 8 server nodes.

Randomly with all of the nodes, the following effect keeps coming back:

For some time (varies), the backend server operates examplary, serves thousands and thousands of request on an average load of 4,5-6. (The frontend is HAProxy in roundrobin mode.) Then, for no apparent reason, load skyrockets to unbelivably high values, like 115-180 or so, then the only way to save the system from crashing is with killing all apache processes with SIGKILL, wait for 20 seconds, then issue an apache service start...

I've written a script to monitor this kind of screwup, but I'd like to know the reason behind this.

What can cause something like this?

Like I said there's no apparent pattern behind this... It's sometimes happening to the same server, sometimes another... And killing the Apache processes always helps so I figure it must be something with Apache.

Also... when it's happening, I see lot of "uninterruptible sleep" apache processes...

Code:

apache  30594  2.1  0.9  70332 31720 ?        S    11:39  0:04 /usr/sbin/httpd

apache  30595  1.8  1.1  76128 39504 ?        D    11:39  0:04 /usr/sbin/httpd

apache  30598  2.0  0.9  69912 33072 ?        D    11:39  0:04 /usr/sbin/httpd

apache  30599  2.1  1.0  70588 33920 ?        D    11:39  0:04 /usr/sbin/httpd

apache  30603  2.1  0.9  69276 31712 ?        D    11:39  0:04 /usr/sbin/httpd

apache  30607  2.0  0.9  71676 33352 ?        D    11:39  0:04 /usr/sbin/httpd

apache  30699  1.7  0.7  70320 26496 ?        D    11:40  0:03 /usr/sbin/httpd

apache  30707  1.4  0.7  69444 25120 ?        R    11:40  0:02 /usr/sbin/httpd

apache  30708  1.3  0.7  69532 24976 ?        S    11:40  0:02 /usr/sbin/httpd

apache  30709  1.6  1.0  78324 35472 ?        D    11:40  0:02 /usr/sbin/httpd

apache  30710  1.6  0.8  71604 27496 ?        D    11:40  0:03 /usr/sbin/httpd

apache  30711  1.7  0.8  71892 27648 ?        D    11:40  0:03 /usr/sbin/httpd

apache  30712  1.4  0.7  69252 23988 ?        D    11:40  0:02 /usr/sbin/httpd

apache  30713  1.3  0.7  70072 25580 ?        D    11:40  0:02 /usr/sbin/httpd

apache  30714  1.6  0.8  72156 29204 ?        D    11:40  0:03 /usr/sbin/httpd

apache  30715  1.7  0.7  70716 25464 ?        S    11:40  0:03 /usr/sbin/httpd

apache  30716  1.4  0.8  70872 29556 ?        D    11:40  0:02 /usr/sbin/httpd

apache  30717  1.6  0.8  69500 29572 ?        S    11:40  0:02 /usr/sbin/httpd

apache  30719  1.6  0.7  69972 25748 ?        S    11:40  0:03 /usr/sbin/httpd

apache  30720  1.7  0.9  70492 31016 ?        S    11:40  0:03 /usr/sbin/httpd

apache  30721  1.5  0.9  77524 30620 ?        S    11:40  0:02 /usr/sbin/httpd