LinuxQuestions.org
View the Most Wanted LQ Wiki articles.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices

Reply
 
Search this Thread
Old 11-08-2006, 12:15 PM   #1
yepp
LQ Newbie
 
Registered: Oct 2005
Posts: 8

Rep: Reputation: 0
Problem with > 256 apache processes - Solved


We have run into a strange problem with our apache processes.
If we run the serverlimit at 256 the server runs with load < 5 all the time and has a system usage < 6% and idles 50%.

== top snippet ==

Tasks: 233 total, 5 running, 226 sleeping, 0 stopped, 2 zombie
Cpu(s): 42.8%us, 6.7%sy, 0.7%ni, 45.6%id, 0.2%wa, 0.0%hi, 3.9%si, 0.0%st
Mem: 2059792k total, 2029916k used, 29876k free, 38488k buffers
Swap: 1911724k total, 3252k used, 1908472k free, 1676096k cached

== /top snippet ==

The server is a regular webserver which runs Apache 2.2.3 and php 5.1.6. It opens a connection to our database-server per connection and also it might use a nfs disk if the user uploads something.


If we use more than 256 connections the load goes through the roof, the %sy usage goes up to around 40-50% and the usertime goes down to 20% and the rest is idletime (probably doing context switches).

The context switches also jumps from arount 10k to 150k as soon as over 256 processes are reached.

If we look at our cacti-graphs the load seems to be instantanious, from ~5 to ~150 in a minute.



I have no idea on how to debug this problem, any help would be appreciated.

Regards,
Joakim


System:
Slamd64 11.0
kernel 2.6.18
Running on:
Dell PowerEdge 1950
2 x Dual Core Intel® Xeon® 5110, 4MB Cache, 1.6GHz
2Gb Ram

Last edited by yepp; 11-13-2006 at 01:19 AM.
 
Old 11-11-2006, 06:42 PM   #2
masterross
Member
 
Registered: Nov 2005
Distribution: Slack 10.2, CentOS 4.x
Posts: 83

Rep: Reputation: 17
what are your apache settings?
 
Old 11-11-2006, 07:04 PM   #3
yepp
LQ Newbie
 
Registered: Oct 2005
Posts: 8

Original Poster
Rep: Reputation: 0
Thank you for replying, I attached my apache conf. I also attached the output of opreport. According to opreport, at least as I can read it, almost all cpu time goes to scheduling.

I've been trying to google on it the last couple of days reading forums and kernel/apache/nfs mailing lists trying to find out what could be wrong. Unfortunately I don't even know what I'm really looking for.

opreport System:
Code:
root@enterprise:~/test# opreport
CPU: Core 2, speed 1596.48 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (Unhalted core cycles) count 100000
CPU_CLK_UNHALT...|
  samples|      %|
------------------
   709784 51.6418 vmlinux
   216528 15.7539 libphp5.so
   158855 11.5578 libc-2.3.6.so
   126955  9.2369 apc.so
   108630  7.9036 libz.so.1.2.3
    19890  1.4471 httpd
.....
opreport vmlinux in detail:
Code:
CPU: Core 2, speed 1596.48 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (Unhalted core cycles) count 100000
samples  %        symbol name
109047   15.3634  __down
86307    12.1596  task_rq_lock
72237    10.1773  schedule
36948     5.2055  __posix_lock_file_conf
21856     3.0792  try_to_wake_up
21214     2.9888  __wake_up
19176     2.7017  __wake_up_common
18043     2.5420  __switch_to
17395     2.4507  thread_return
15685     2.2098  mwait_idle
.....
Apache conf:
Code:
Timeout 300

KeepAlive Off
MaxKeepAliveRequests 100
KeepAliveTimeout 15


<IfModule prefork.c>
ServerLimit     256
StartServers         5
MinSpareServers      25
MaxSpareServers     50
MaxClients        256
MaxRequestsPerChild  300
</IfModule>

LoadModule php5_module        modules/libphp5.so
User nobody
UseCanonicalName On
HostnameLookups Off
AddType application/x-httpd-php .php

ExtendedStatus On

<VirtualHost xxx.xxx.xxx.xxx:80>
    ServerAdmin xxxx@xxxx
    DocumentRoot /xxxx/html/
    ServerName xxxxx
    ServerAlias xxxxx *.xxxxx
    ErrorLog logs/xxxxx-error_log

    <Location /p>
        ForceType application/x-httpd-php
    </Location>
    CacheEnable mem /bilder
     AddOutputFilterByType DEFLATE text/html text/plain text/xml
</VirtualHost>
 
Old 11-12-2006, 06:56 PM   #4
masterross
Member
 
Registered: Nov 2005
Distribution: Slack 10.2, CentOS 4.x
Posts: 83

Rep: Reputation: 17
try:

Timeout 300
KeepAlive Off
MaxKeepAliveRequests 100
ServerLimit 200
KeepAliveTimeout 4
MinSpareServers 15
MaxSpareServers 20
StartServers 15
MaxClients 200
MaxRequestsPerChild 800

but if u use mysql with a lot of queries that can be the prob too
 
Old 11-13-2006, 01:19 AM   #5
yepp
LQ Newbie
 
Registered: Oct 2005
Posts: 8

Original Poster
Rep: Reputation: 0
Solved

I recompiled the kernel and changed the hz to 100 and set preemtion to none and the problem went away. Seems to be some kind of kernel feature/bug.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Apache ... wierd processes Odins_Son Linux - General 3 08-18-2005 11:38 AM
BT last 256 K download problem mimithebrain Linux - Software 1 06-24-2005 04:23 PM
apache fails on start: error 256 sohmc Linux - Networking 0 01-07-2005 08:32 AM
so many apache processes cjpsparks Linux - Software 4 01-07-2004 04:54 PM
Runaway apache processes Gardener Linux - Networking 3 01-25-2003 06:56 PM


All times are GMT -5. The time now is 09:07 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration