Apache 100% CPU load

UnrealX · 08-05-2007, 06:21 AM

Hello,

I have moved all my sites from one server (Dual Core Dual Xeon / 1GB RAM) to another (Core2Duo / 2GB RAM). Of course I have tweaked apache (2.2.3) settings to fit new CPU, however strangely enough some time passed and (according to `top`) one of httpd processes was taking up 100% CPU usage. I monitored the server to see if it would drop, but not only it didn`t drop, but other httpd processes started taking up more and more of CPU time.
Every time after I restart apache it takes approx. 0 - 4 hours for httpd process to take up most of CPU time again.

I`ve fiddled with httpd.conf, these are 3 configurations I`ve tried, none of them had any positive results.

Code:

<IfModule prefork.c>
StartServers       8
MinSpareServers    5
MaxSpareServers   20
ServerLimit      256
MaxClients       256
MaxRequestsPerChild  4000
</IfModule>
<IfModule worker.c>
StartServers         2
MaxClients         150
MinSpareThreads     25
MaxSpareThreads     75
ThreadsPerChild     25
MaxRequestsPerChild  4000
</IfModule>

Code:

<IfModule prefork.c>
StartServers 5
MinSpareServers 32
MaxSpareServers 64
ServerLimit 512
MaxClients 512
MaxRequestsPerChild 512
</IfModule>
<IfModule worker.c>
StartServers 5
MinSpareThreads 32
MaxSpareThreads 64
ThreadsPerChild 512
MaxClients 512
</IfModule>

Code:

<IfModule prefork.c>
StartServers 10
MinSpareServers 5
MaxSpareServers 50
MaxClients 256
MaxRequestsPerChild 5000
</IfModule>
<IfModule worker.c>
StartServers 10
MinSpareServers 5
MaxSpareServers 50
MaxClients 256
MaxRequestsPerChild 5000
</IfModule>

Naturally I thought it was some specific URL that was causing trouble, so I wrote (not-so-accurate) bash script:

Code:

#!/bin/bash
log=/var/log/httpd/domlogs/example.com
log2=/var/log/httpd/access_log
cpu=50

stop=false
while [ "$stop" == false ]
do
        load=`top -d 0 | head -8 | tail -1 | grep httpd | awk '{print $10}'`
        load=${load%%.*}
        if [ "$load" == "" ]
        then
                load="0"
        fi
        if [ "$load" -ge "$cpu" ]
        then
                tail $log
                echo "-----------------"
                tail $log2
                stop=true
        else
                echo 'Reloading ('$load'<'$cpu')'
                #sleep 1
        fi
done
exit 0

Unfortunately neither `ps` nor `top` can display accurate CPU usage if it`s run for short period of time, so this script would approximately detect when httpd process jumped up, but not accurately enough to catch the culprit script.

But then both apache and php have timeout options, so if it were some script that was causing trouble, php would simply timeout it after a minute.

I`m not sure where to go from here. Any ideas?

unSpawn · 08-06-2007, 03:37 AM

What do the sites run on? Regular PHP-based packages or home brewn stuff? Anything interesting in any logs around the time CPU starts spiking? Do you run anything like Dstat or Atsar? With those you would have a macro view of the resource usage on the box, just in case CPU hogging is just a symptom. And instead of 'top' I would run 'atop', which can save process stats which you can replay later on.

As for running a script that checks processes here's my example (and it's just an example):

Code:

#!/bin/sh
# Purpose: Show some process info
# Args: 1: CPU
# Deps: Bash, GNU utils, procstat, lsof, netstat, strings
# Run from: manual or sched

# Prepstage functions
die() { echo "No value for CPU given, exiting."; exit 1; }

case $1 in [1-9]*) cpuval="$1";; *) die;; esac
/bin/ps --no-headers --sort=c -eo %C -eo pid,command wwwee|while read cpu pid command env; do
 if [ ${cpu%%.*} -ge $cpuval ]; then
        logtag="`/bin/date +%s` [${pid}]"
        line=$(strings -an1 /proc/${pid}/cmdline|xargs);
        echo "${logtag} cmdline ${line}";
        # procstat is at senko.net
        procstat -a "${pid}"|while read line; do echo "${logtag} proc ${line}"; done;
        lsof -w -n -p "${pid}"|egrep -v "(usr|lib)/"|while read line; do echo "${logtag} lsof ${line}"; done;
        netstat -anp | grep "${pid}"|while read line; do echo "${logtag} netstat ${line}"; done;
 fi;
done

exit 0

Don't run it from cron but use Heiner's periodic instead. And if you want it in syslog replace the logging echoes with 'logger'.

UnrealX · 08-06-2007, 03:54 AM

Homebrewn PHP scripts. Although it`s not very likely one of them is causing load as I have invested quite some time in optimizing them (and as mentioned earlier, PHP has timeout limits too). Currently I am monitoring error_log. My suspicion is that mod_proxy is causing load as error messages from mod_proxy appear just about when httpd process starts taking up more and more of CPU time.

Oh, and thanks for the shell script!

UnrealX · 08-06-2007, 01:37 PM

It seems that it really was mod_proxy.
These error messages are registered every time CPU usage jumps up:

Code:

[Mon Aug 06 17:07:20 2007] [error] [client xxx.xxx.xxx.xxx] proxy: error reading status line from remote server example.com
[Mon Aug 06 17:07:20 2007] [error] [client xxx.xxx.xxx.xxx] proxy: Error reading from remote server returned by /path/to/some/file.jpg

Files are always different so I assume it`s due to mod_proxy bug.

unSpawn · 08-06-2007, 02:19 PM

Block 'em and see what happens? As in mod_security maybe?

UnrealX · 08-07-2007, 11:57 AM

I`ve managed to rewrite my scripts so they don`t rely on mod_proxy.
Hopefully httpd won`t hog all cpu resources now.

If anyone is experiencing similar mod_proxy errors, here`s a thread at apache`s bugzilla:
http://issues.apache.org/bugzilla/show_bug.cgi?id=37770

EDIT: Nope, that didn`t help. Error_log only has 404 errors (robots.txt, favico.ico, nothing too special). I have no clue what to do next.

unSpawn · 08-10-2007, 11:38 AM

I have no clue what to do next.
Maybe start by telling us what components and versions you exactly use, how your stuff is configured and maybe somebody could help shed a light on this.

proxy: Error reading from remote server returned by /path/to/some/file.jpg
Come to think of it, could this be collateral of some PHP inclusion vuln?..
Does it happen with one hosted site only?
A select set of remote URI's?
Any "weird" processes running under the Apache user perhaps?

UnrealX · 08-10-2007, 12:17 PM

Apache 2.2.3
PHP 5.2.3
CentOS 5
Core 2 Duo, 2GB RAM
httpd.conf
php.ini

There were nothing suspicious about URLs that mod_proxy was requesting. I was using mod_proxy for local url rewriting, however as I have mentioned above it is now disabled.

Currently I use my own `watcher.sh` script (posted above) to detect increased CPU usage and restart httpd in such case. I can now leave server unattended for longer periods of time, however it`s not a solution, rather a dirty hack.

niXeN · 02-04-2011, 08:04 AM

UnrealX

Hi, did you find a solution for this?