Apache is consuming all CPU resources

romeo_tango · 12-29-2007, 12:21 AM

Can somebody help me with this Apache configuration?
My machine's CPU processing is always overloading after each one day I restarted the Apache.

I already tried to google about apache tuning and found a few tips which I already applied. Here are some of the httpd.conf configuration that I changed :

Code:

MaxKeepaliveRequests 80
KeepAliveTimeout 2

<IfModule prefork.c>
StartServers         5
MinSpareServers      5
MaxSpareServers     10
MaxClients         100
MaxRequestsPerChild  2500
</IfModule>

My website is running mostly dynamic content (php,flash, and Oracle). When the CPU process is above 90%, I found many httpd process status is sleep while they still consume the CPU and memory. Why is that happening? How could I destroy the process with sleep status automatically? Do I miss something in the configuration?

Our machine is running SPARC Solaris 10 and having 8 GB physical RAM and 16 GB swap.

Here is a capture from top command result

Code:

   PID USERNAME LWP PRI NICE  SIZE   RES STATE    TIME    CPU COMMAND
 25839 www     1  30    0  109M   43M sleep    0:00  4.10% httpd
 25831 www     1  31    0  107M   41M run      0:01  3.86% httpd
 25786 www     1  43    0  107M   41M sleep    0:03  3.42% httpd
 25824 www     1  42    0  107M   41M sleep    0:01  3.37% httpd
 25818 www     1  37    0  107M   40M sleep    0:01  3.02% httpd
 25836 www     1  42    0  107M   41M sleep    0:00  2.26% httpd
 25838 www     1  36    0  107M   41M run      0:00  2.19% httpd
 25822 www     1  52    0  107M   41M sleep    0:00  1.95% httpd
 25841 www     1  41    0  106M   40M sleep    0:00  1.45% httpd
 25808 www     1  40    0  107M   41M sleep    0:02  1.40% httpd
 25829 www     1  40    0  107M   41M sleep    0:00  1.40% httpd
 25835 www     1  40    0  107M   40M sleep    0:00  1.38% httpd
 25630 www     1  41    0  107M   41M cpu/0    0:10  0.73% httpd

I would like to limit the httpd process so that they wouldn't run untill the CPU process is > 90%. I should limit it by the MaxClients directive, shouldn't I? But the problem is eventhough I tried to lower the MaxClients allowed, there are usually many stalled httpd process that make the CPU load is always high.

Any suggestion would be appreciated.
Thanks.

jlliagre · 12-29-2007, 02:28 AM

Quote:

Originally Posted by romeo_tango

When the CPU process is above 90%, I found many httpd process status is sleep while they still consume the CPU and memory. Why is that happening? How could I destroy the process with sleep status automatically?

There is no reason to kill a process just because it has been sampled in the sleeping state. All processes do sleep at some time.

Quote:

But the problem is eventhough I tried to lower the MaxClients allowed, there are usually many stalled httpd process that make the CPU load is always high.

They aren't stalled but runaway processes, as they load the CPU.
You should use prstat and vmstat on Solaris, top is known to have some issues.
You can run pstack and/or truss on these processes to see what they are actually doing.

coolster · 12-29-2007, 04:24 AM

Quote:

Originally Posted by romeo_tango

Can somebody help me with this Apache configuration?

There is nothing wrong with (that part of) your httpd.conf.
Check Apache's access.log and error.log for some correlations with the things you 've found in prtstat (see above). Some misconfigured Apache modules may be misbehaving.

Thanks, C

romeo_tango · 01-04-2008, 02:44 AM

Quote:

Originally Posted by jlliagre

There is no reason to kill a process just because it has been sampled in the sleeping state. All processes do sleep at some time.

They aren't stalled but runaway processes, as they load the CPU.
You should use prstat and vmstat on Solaris, top is known to have some issues.
You can run pstack and/or truss on these processes to see what they are actually doing.

yeah i know that the processes is using CPU, that's what I am asking. I mean, i don't know how to keep this Apache run stable.

Here is the capture from prstat when the CPU is 90% busy :

Code:

   PID USERNAME  SIZE   RSS STATE  PRI NICE      TIME  CPU PROCESS/NLWP
 13959 www       108M   42M cpu0     0    0   0:06:07  44% httpd/1
 13873 www       108M   41M run      0    0   0:13:08  21% httpd/1
 13903 www       109M   42M sleep   40    0   0:00:18 5.8% httpd/1
 14062 www       108M   42M run     40    0   0:00:08 4.6% httpd/1
 14197 www       110M   43M sleep   42    0   0:00:01 1.0% httpd/1
 14099 www       110M   44M sleep   53    0   0:00:09 1.0% httpd/1
 14095 www       109M   42M sleep   49    0   0:00:02 0.9% httpd/1
 14185 www       109M   42M sleep   51    0   0:00:06 0.9% httpd/1
 14200 www       109M   43M sleep   42    0   0:00:01 0.8% httpd/1
 14206 www       108M   41M sleep   43    0   0:00:00 0.8% httpd/1
 14235 www       109M   42M sleep   54    0   0:00:00 0.8% httpd/1
 14149 www       110M   43M sleep   43    0   0:00:02 0.8% httpd/1
 14229 www       108M   41M sleep   55    0   0:00:00 0.8% httpd/1
 14236 www       108M   41M sleep   44    0   0:00:00 0.7% httpd/1
 14077 www       110M   44M sleep   54    0   0:00:10 0.6% httpd/1
 14139 www       109M   42M sleep   47    0   0:00:07 0.6% httpd/1
 14210 www       108M   41M sleep   51    0   0:00:00 0.6% httpd/1
 14170 www       110M   44M sleep   46    0   0:00:06 0.5% httpd/1
 14208 www       110M   43M sleep   43    0   0:00:00 0.5% httpd/1
 14225 www       108M   40M sleep   45    0   0:00:00 0.5% httpd/1
 14193 www       108M   42M sleep   47    0   0:00:01 0.4% httpd/1
 14228 www       108M   40M sleep   44    0   0:00:00 0.4% httpd/1
 14231 www       109M   42M sleep   53    0   0:00:00 0.4% httpd/1
 14223 www       109M   42M sleep   49    0   0:00:00 0.4% httpd/1
 14244 www       107M   40M sleep   54    0   0:00:00 0.3% httpd/1
 14153 www       110M   43M sleep   52    0   0:00:02 0.2% httpd/1
  5111 www       190M   72M sleep   59    0   0:02:09 0.2% java/22
 NPROC USERNAME  SIZE   RSS MEMORY      TIME  CPU
    63 www      4744M 1771M    22%   0:21:43  90%
    98 root     2262M 1066M    13%   7:36:30 0.9%
     1 nobody   3336K 2536K   0.0%   0:00:00 0.0%
     1 smmsp    7312K 1976K   0.0%   0:00:03 0.0%
     1 daemon   5224K 3304K   0.0%   0:01:18 0.0%

then I already follow your suggest to use pstack to know what are they doing there..

Code:

bash-3.00# pstack -F 13959
13959:  /usr/local/apache2/bin/httpd -k start
 fec54e38 t_splay  (6143d8, 1610, 525c50, fececbc0, fece8284, 0) + b0
 fec5451c realloc  (612dc0, 5810, 612dc8, fececbc0, e28, 6143d8) + 158
 fe530f38 _erealloc (612dd8, 5800, 0, fec54110, fece8284, fecf09b0) + 1e0
 fe542d18 op_array_alloc_ops (436bd0, ffbed568, 0, 0, 4f, 0) + 30
 fe54340c get_next_op (436bd0, ffbed544, a, fe532b84, 4f, 0) + dc
 fe5341c8 zend_do_end_variable_parse (0, 0, 1, 53, 4e, 0) + 78
 fe535d48 zend_do_pass_param (ffbed840, 42, 4, 0, 0, fecf09b0) + 268
 fe51c450 zendparse (436750, 2, 40, fe531378, fece8284, fecf09b0) + 1900
 fe521060 compile_file (ffbeecc0, 2, 0, 70, 6f64756c, 0) + 150
 fe521240 compile_filename (2, ffbeef98, ffbef500, ffbef514, 4, 0) + e0
 fe577228 execute  (5b5830, ffbf12e8, ffbf14f0, ffbf1048, 2, ffbef5a8) + 5ca0
 fe5773b8 execute  (63fd38, 673318, ffbff000, febd2000, 0, 0) + 5e30
 fe5773b8 execute  (370f28, 2, 0, 1e, 91711, 3) + 5e30
 fe552108 zend_execute_scripts (8, 0, 3, 0, ffbff614, 0) + 160
 fe4e8e64 php_execute_script (ffbff614, fe59a97c, 6, 0, 70687000, 7068702d) + 394
 fe57b8ac php_handler (35bc20, 25252525, 6f7665, 80808080, 35bc20, 3c) + 774
 00034308 ap_run_handler (35bc20, 3b3b3b3b, 70687000, 80808080, ff00, 35bc20) + 94
 00034f28 ap_invoke_handler (35bc20, 0, 4, 35bc20, 0, 348b88) + 1b8
 0002f824 ap_process_request (35bc20, 4, 35bc20, 34ac28, 34ac28, 34afe8) + 88
 00026718 ap_process_http_connection (34ac28, 34ab50, 34ab50, fffffff8, 0, 0) + 8c
 000478d8 ap_run_process_connection (34ac28, 34ab50, 34ab50, e, 348b88, 34eb28) + 94
 00047ebc ap_process_connection (34ac28, 34ab50, 34ab50, e, 348b88, 34eb28) + 80
 00031c54 child_main (e, 1, fececbc0, 0, 0, febd2000) + 7d0
 00031e90 make_child (9e0b8, e, ffbffc64, 1, 9192c, 0) + 198
 000322a4 perform_idle_server_maintenance (9c2d8, ffbffc60, ffbffc48, 9c2d8, ffbffc60, ffbffc64) + 304
 000329cc ap_mpm_run (9c2d8, c8388, 9e0b8, 9e0b8, 9c340, 9a350) + 67c
 0003d614 main     (3, ffbffdac, ffbffdbc, 91930, febd0140, febd0180) + b80
 00025e1c _start   (0, 0, 0, 0, 0, 0) + 5c

that one is for the above httpd running consume 44% of the CPU and this one is for the sleep httpd process

Code:

bash-3.00# pstack -F 14206
14206:  /usr/local/apache2/bin/httpd -k start
 fecc0568 accept   (3, 34abe4, 34ac04, 1)
 fefcda84 apr_accept (ffbff974, 9a680, 34ab18, 0, ffbff970, 348b90) + 1c
 00066f54 unixd_accept (ffbff9fc, 348b90, 34ab18, 348b90, 907d0, 66f28) + 2c
 00031b94 child_main (6, 1, fececbc0, 0, 0, febd2000) + 710
 00031e90 make_child (9e0b8, 6, ffbffc64, 1, 9192c, 0) + 198
 000322a4 perform_idle_server_maintenance (9c2d8, ffbffc60, ffbffc48, 9c2d8, ffbffc60, ffbffc64) + 304
 000329cc ap_mpm_run (9c2d8, c8388, 9e0b8, 9e0b8, 9c340, 9a350) + 67c
 0003d614 main     (3, ffbffdac, ffbffdbc, 91930, febd0140, febd0180) + b80
 00025e1c _start   (0, 0, 0, 0, 0, 0) + 5c

and to be honest.. i don't really understand the information from pstack output.

if I change the question, why the highest httpd process consume up to 44% of the CPU load? can I limit so it would not be so greedy? I changed the MaxClients directive and its friends but the result is still the same.

Thanks.

romeo_tango · 01-04-2008, 02:55 AM

Quote:

Originally Posted by coolster

There is nothing wrong with (that part of) your httpd.conf.
Check Apache's access.log and error.log for some correlations with the things you 've found in prtstat (see above). Some misconfigured Apache modules may be misbehaving.

Thanks, C

i can't find any correlations between the error.log and prstat.. my error.log mainly contains these lines :

Code:

[Fri Jan 04 15:48:37 2008] [error] [client 20x.xxx.xxx.128] Directory index forbidden by rule:

that should be happened because the .htaccess directive shouldn't it?

jlliagre · 01-04-2008, 05:09 AM

Quote:

Originally Posted by romeo_tango

yeah i know that the processes is using CPU, that's what I am asking. I mean, i don't know how to keep this Apache run stable.

You mean the server is unstable when the CPU load is high ?
Does it crashes ? Do not reply to requests ?

Quote:

Here is the capture from prstat when the CPU is 90% busy :

Code:

   PID USERNAME  SIZE   RSS STATE  PRI NICE      TIME  CPU PROCESS/NLWP
 13959 www       108M   42M cpu0     0    0   0:06:07  44% httpd/1
...
 NPROC USERNAME  SIZE   RSS MEMORY      TIME  CPU
    63 www      4744M 1771M    22%   0:21:43  90%
    98 root     2262M 1066M    13%   7:36:30 0.9%
...

then I already follow your suggest to use pstack to know what are they doing there..

Code:

bash-3.00# pstack -F 13959
13959:  /usr/local/apache2/bin/httpd -k start
 fec54e38 t_splay  (6143d8, 1610, 525c50, fececbc0, fece8284, 0) + b0
 fec5451c realloc  (612dc0, 5810, 612dc8, fececbc0, e28, 6143d8) + 158
 fe530f38 _erealloc (612dd8, 5800, 0, fec54110, fece8284, fecf09b0) + 1e0
 fe542d18 op_array_alloc_ops (436bd0, ffbed568, 0, 0, 4f, 0) + 30
 fe54340c get_next_op (436bd0, ffbed544, a, fe532b84, 4f, 0) + dc
 fe5341c8 zend_do_end_variable_parse (0, 0, 1, 53, 4e, 0) + 78
 fe535d48 zend_do_pass_param (ffbed840, 42, 4, 0, 0, fecf09b0) + 268
 fe51c450 zendparse (436750, 2, 40, fe531378, fece8284, fecf09b0) + 1900
 fe521060 compile_file (ffbeecc0, 2, 0, 70, 6f64756c, 0) + 150
 fe521240 compile_filename (2, ffbeef98, ffbef500, ffbef514, 4, 0) + e0
 fe577228 execute  (5b5830, ffbf12e8, ffbf14f0, ffbf1048, 2, ffbef5a8) + 5ca0
 fe5773b8 execute  (63fd38, 673318, ffbff000, febd2000, 0, 0) + 5e30
 fe5773b8 execute  (370f28, 2, 0, 1e, 91711, 3) + 5e30
 fe552108 zend_execute_scripts (8, 0, 3, 0, ffbff614, 0) + 160
 fe4e8e64 php_execute_script (ffbff614, fe59a97c, 6, 0, 70687000, 7068702d) + 394
 fe57b8ac php_handler (35bc20, 25252525, 6f7665, 80808080, 35bc20, 3c) + 774
 00034308 ap_run_handler (35bc20, 3b3b3b3b, 70687000, 80808080, ff00, 35bc20) + 94
 00034f28 ap_invoke_handler (35bc20, 0, 4, 35bc20, 0, 348b88) + 1b8
 0002f824 ap_process_request (35bc20, 4, 35bc20, 34ac28, 34ac28, 34afe8) + 88
 00026718 ap_process_http_connection (34ac28, 34ab50, 34ab50, fffffff8, 0, 0) + 8c
 000478d8 ap_run_process_connection (34ac28, 34ab50, 34ab50, e, 348b88, 34eb28) + 94
 00047ebc ap_process_connection (34ac28, 34ab50, 34ab50, e, 348b88, 34eb28) + 80
 00031c54 child_main (e, 1, fececbc0, 0, 0, febd2000) + 7d0
 00031e90 make_child (9e0b8, e, ffbffc64, 1, 9192c, 0) + 198
 000322a4 perform_idle_server_maintenance (9c2d8, ffbffc60, ffbffc48, 9c2d8, ffbffc60, ffbffc64) + 304
 000329cc ap_mpm_run (9c2d8, c8388, 9e0b8, 9e0b8, 9c340, 9a350) + 67c
 0003d614 main     (3, ffbffdac, ffbffdbc, 91930, febd0140, febd0180) + b80
 00025e1c _start   (0, 0, 0, 0, 0, 0) + 5c

that one is for the above httpd running consume 44% of the CPU...
and to be honest.. i don't really understand the information from pstack output.

I believe your process is busy processing some PHP code and is currently trying to allocate some memory for an array variable that is growing.
You need to run pstack a couple of times more to see where exactly the process is stuck looping. I'm not at all familiar with PHP but it could be a memory leak or something similar.

Quote:

if I change the question, why the highest httpd process consume up to 44% of the CPU load?

Because it has some code to execute. It would probably consume 100% should the CPU was fully available.

Quote:

can I limit so it would not be so greedy?

If it is busy processing something useful and it should be given the CPU it requires.
If it is stuck because of a bug or bad design, you can suspend the busy processes (pstop PID) and see if the service is still provided correctly.

coolster · 01-04-2008, 05:43 AM

Quote:

Originally Posted by romeo_tango

i can't find any correlations between the error.log and prstat.. my error.log mainly contains these lines :

Code:

[Fri Jan 04 15:48:37 2008] [error] [client 20x.xxx.xxx.128] Directory index forbidden by rule:

that should be happened because the .htaccess directive shouldn't it?

I can't say yes or no to that, since I don't know the rest of your httpd.conf or the Apache modules that you are using.

Back to the memory swallowing Apache problem:
this can be caused by any bad programmed PHP application or any thread unsafe PHP library that is used by PHP applications.

I suggest you shut down Apache modules one by one (restart the Apache webserver every time!) and see what happens with the memory usage.

Regs, C