LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices


Reply
  Search this Thread
Old 11-26-2010, 05:51 PM   #1
kinetik
Member
 
Registered: Dec 2005
Location: The most beautiful city in the world.
Distribution: Mostly RedHat. Also Suse, Ubuntu, PHLAK etc.
Posts: 149

Rep: Reputation: 15
IBMIHS Apache 2.0.48 headaches


Hi all


Been a while since I last posted here, though I stop by often. Unfortunately I find myself in a bit of a pickle again.

Here's the situation. I have an RHEL3u5 server running Apache 2.0.48 and an equivalent of Tomcat and JBoss. IBM's WebSphere Application Server 6.0.2 to be exact, the Apache also being one of their flavors (IBMIHS).

Everything's been going dandy since forever, but lately the httpd processes seemingly just hit a brick wall of some unknown kind. Checking the /server-status page reveals a whole lot of piling up "waits".

The funny part is that the load average rarely goes higher than 1.0, I/O seems within limits, memory usage as far as I can tell seems fine, disk usage appears all good... I would've felt more at ease if something did shoot through the roof, but there's just nothing to go on.

It's definitely Apache getting its knickers all in a twist as well since I can access the servlet engine's web GUI just fine over its dedicated port via browser.


Here's what I noticed so far:

Apache processes rise steadily to over 35 or so, after which it all goes south fast. As a result and as a temporary fix, I've enabled a cronjob that checks the amount of httpd processes and just restarts Apache if they're above 28. So far this seems to keep things sort of happy, but obviously I'm still twitchy with the overall state of affairs.

I do not have the settings immediately at hand, but here are some further details so far:


Server spec:

8 x Xeon CPUs
8GB RAM


I'll post further details probably on Monday, specifically key portions of the httpd.conf file.


For now though, has anyone else experienced something like this before? Any help or advice appreciated.

Last edited by kinetik; 11-26-2010 at 05:53 PM.
 
Old 11-30-2010, 08:48 AM   #2
kinetik
Member
 
Registered: Dec 2005
Location: The most beautiful city in the world.
Distribution: Mostly RedHat. Also Suse, Ubuntu, PHLAK etc.
Posts: 149

Original Poster
Rep: Reputation: 15
Hi all


Day late, but problems are still ongoing. httpd.conf file's settings look as follows currently:

Code:
#
# Timeout: The number of seconds before receives and sends time out.
#
Timeout 300

#
# KeepAlive: Whether or not to allow persistent connections (more than
# one request per connection). Set to "Off" to deactivate.
#
KeepAlive On

#
# MaxKeepAliveRequests: The maximum number of requests to allow
# during a persistent connection. Set to 0 to allow an unlimited amount.
# We recommend you leave this number high, for maximum performance.
#
MaxKeepAliveRequests 100

#
# KeepAliveTimeout: Number of seconds to wait for the next request from the
# same client on the same connection.
#
KeepAliveTimeout 10

##
## Server-Pool Size Regulation (MPM specific)
##

# worker MPM
# ThreadLimit: maximum setting of ThreadsPerChild
# ServerLimit: maximum setting of StartServers
# StartServers: initial number of server processes to start
# MaxClients: maximum number of simultaneous client connections
# MinSpareThreads: minimum number of worker threads which are kept spare
# MaxSpareThreads: maximum number of worker threads which are kept spare
# ThreadsPerChild: constant number of worker threads in each server process
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule worker.c>
ThreadLimit         100
ServerLimit         64
StartServers         8
MaxClients         2000
MinSpareThreads     100
MaxSpareThreads     500
ThreadsPerChild     100
MaxRequestsPerChild 50
</IfModule>

Not much in the access or error logs, apart from pages being requested which do not exist. It still seems as if everything migrates south in a fast way once the httpd processes go above 35 or so, system load seemingly being unaffected.

Has anyone else had something similar? Also, if more info is required please let me know.

Thanks in advance.
 
Old 12-01-2010, 11:44 AM   #3
kinetik
Member
 
Registered: Dec 2005
Location: The most beautiful city in the world.
Distribution: Mostly RedHat. Also Suse, Ubuntu, PHLAK etc.
Posts: 149

Original Poster
Rep: Reputation: 15
OK, been doing some more investigation and so far came up with this:

Apparently the optimal MaxClients setting can be determined by taking the total size physical memory and dividing that by the average size of the httpd process (RSS size in ps aux I think). I'm working with the max size I've seen so far to be on the safe side.

So, if my maths check out, it would be this:

8GB * 1024 = 8192MB

Code:
ps aux | grep httpd
...
...
nobody   10668  0.4  0.1 1039016 11184 ?     S    14:23   0:00 /data01/IBMIHS/bin/httpd -d /data01/IBMIHS -k start
...

Which means 8192 / 12 = 682.66. I've since rather based this on the reading based in the free command, specifically the buffers/cache line, which gives me 6192MB / 12 = 540.


Next up, the MaxRequestsPerChild directive should be set, ideally at any rate, to a figure based on the number of requests the server gets per day, divided by the number of Apache processes. I assume the number of Apache processes would be based on the ServerLimit directive.

So, based on that, I think I can put something like this in place on my nonProd server:


Code:
<IfModule worker.c>
ThreadLimit         100
ServerLimit         64
StartServers         8
MaxClients         540
MinSpareThreads     100
MaxSpareThreads     500
ThreadsPerChild     100
MaxRequestsPerChild 2340
</IfModule>

Once that's done, I think the obvious next step would probably be to get some sort of test scheme together and benchmark it with ab or something. I'm hoping it's possible to just output the access file from Prod, stick that in a file, pipe that file into ab and simulate my Prod environment's load against nonProd and see if the new settings work OK or not.

Please feel free to interrupt me if my theory looks daft at any point.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Apache php postgres headaches sandWorm Mandriva 3 10-09-2006 11:51 AM
X Headaches roc Slackware - Installation 3 07-14-2005 11:53 AM
Apache and CGI is giving me headaches inpt Linux - Software 3 06-10-2005 12:24 PM
QT headaches Pi Man Linux - Software 2 10-17-2004 12:20 AM
Apache and Flash Headaches Kong Linux - Software 2 01-02-2004 02:54 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Server

All times are GMT -5. The time now is 07:49 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration