Problem with > 256 apache processes

yepp · 11-08-2006, 12:15 PM

We have run into a strange problem with our apache processes.
If we run the serverlimit at 256 the server runs with load < 5 all the time and has a system usage < 6% and idles 50%.

== top snippet ==

Tasks: 233 total, 5 running, 226 sleeping, 0 stopped, 2 zombie
Cpu(s): 42.8%us, 6.7%sy, 0.7%ni, 45.6%id, 0.2%wa, 0.0%hi, 3.9%si, 0.0%st
Mem: 2059792k total, 2029916k used, 29876k free, 38488k buffers
Swap: 1911724k total, 3252k used, 1908472k free, 1676096k cached

== /top snippet ==

The server is a regular webserver which runs Apache 2.2.3 and php 5.1.6. It opens a connection to our database-server per connection and also it might use a nfs disk if the user uploads something.

If we use more than 256 connections the load goes through the roof, the %sy usage goes up to around 40-50% and the usertime goes down to 20% and the rest is idletime (probably doing context switches).

The context switches also jumps from arount 10k to 150k as soon as over 256 processes are reached.

If we look at our cacti-graphs the load seems to be instantanious, from ~5 to ~150 in a minute.

I have no idea on how to debug this problem, any help would be appreciated.

Regards,
Joakim

System:
Slamd64 11.0
kernel 2.6.18
Running on:
Dell PowerEdge 1950
2 x Dual Core Intel® Xeon® 5110, 4MB Cache, 1.6GHz
2Gb Ram

masterross · 11-11-2006, 06:42 PM

what are your apache settings?

yepp · 11-11-2006, 07:04 PM

Thank you for replying, I attached my apache conf. I also attached the output of opreport. According to opreport, at least as I can read it, almost all cpu time goes to scheduling.

I've been trying to google on it the last couple of days reading forums and kernel/apache/nfs mailing lists trying to find out what could be wrong. Unfortunately I don't even know what I'm really looking for.

opreport System:

Code:

root@enterprise:~/test# opreport
CPU: Core 2, speed 1596.48 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (Unhalted core cycles) count 100000
CPU_CLK_UNHALT...|
  samples|      %|
------------------
   709784 51.6418 vmlinux
   216528 15.7539 libphp5.so
   158855 11.5578 libc-2.3.6.so
   126955  9.2369 apc.so
   108630  7.9036 libz.so.1.2.3
    19890  1.4471 httpd
.....

opreport vmlinux in detail:

Code:

CPU: Core 2, speed 1596.48 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (Unhalted core cycles) count 100000
samples  %        symbol name
109047   15.3634  __down
86307    12.1596  task_rq_lock
72237    10.1773  schedule
36948     5.2055  __posix_lock_file_conf
21856     3.0792  try_to_wake_up
21214     2.9888  __wake_up
19176     2.7017  __wake_up_common
18043     2.5420  __switch_to
17395     2.4507  thread_return
15685     2.2098  mwait_idle
.....

Apache conf:

Code:

Timeout 300

KeepAlive Off
MaxKeepAliveRequests 100
KeepAliveTimeout 15


<IfModule prefork.c>
ServerLimit     256
StartServers         5
MinSpareServers      25
MaxSpareServers     50
MaxClients        256
MaxRequestsPerChild  300
</IfModule>

LoadModule php5_module        modules/libphp5.so
User nobody
UseCanonicalName On
HostnameLookups Off
AddType application/x-httpd-php .php

ExtendedStatus On

<VirtualHost xxx.xxx.xxx.xxx:80>
    ServerAdmin xxxx@xxxx
    DocumentRoot /xxxx/html/
    ServerName xxxxx
    ServerAlias xxxxx *.xxxxx
    ErrorLog logs/xxxxx-error_log

    <Location /p>
        ForceType application/x-httpd-php
    </Location>
    CacheEnable mem /bilder
     AddOutputFilterByType DEFLATE text/html text/plain text/xml
</VirtualHost>

masterross · 11-12-2006, 06:56 PM

try:

Timeout 300
KeepAlive Off
MaxKeepAliveRequests 100
ServerLimit 200
KeepAliveTimeout 4
MinSpareServers 15
MaxSpareServers 20
StartServers 15
MaxClients 200
MaxRequestsPerChild 800

but if u use mysql with a lot of queries that can be the prob too

yepp · 11-13-2006, 01:19 AM

I recompiled the kernel and changed the hz to 100 and set preemtion to none and the problem went away. Seems to be some kind of kernel feature/bug.