LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices

Reply
 
LinkBack Search this Thread
Old 08-05-2007, 06:21 AM   #1
UnrealX
Member
 
Registered: Oct 2005
Location: Lithuania
Distribution: Ubuntu, CentOS
Posts: 36

Rep: Reputation: 15
Apache 100% CPU load


Hello,

I have moved all my sites from one server (Dual Core Dual Xeon / 1GB RAM) to another (Core2Duo / 2GB RAM). Of course I have tweaked apache (2.2.3) settings to fit new CPU, however strangely enough some time passed and (according to `top`) one of httpd processes was taking up 100% CPU usage. I monitored the server to see if it would drop, but not only it didn`t drop, but other httpd processes started taking up more and more of CPU time.
Every time after I restart apache it takes approx. 0 - 4 hours for httpd process to take up most of CPU time again.

I`ve fiddled with httpd.conf, these are 3 configurations I`ve tried, none of them had any positive results.
Code:
<IfModule prefork.c>
StartServers       8
MinSpareServers    5
MaxSpareServers   20
ServerLimit      256
MaxClients       256
MaxRequestsPerChild  4000
</IfModule>
<IfModule worker.c>
StartServers         2
MaxClients         150
MinSpareThreads     25
MaxSpareThreads     75
ThreadsPerChild     25
MaxRequestsPerChild  4000
</IfModule>
Code:
<IfModule prefork.c>
StartServers 5
MinSpareServers 32
MaxSpareServers 64
ServerLimit 512
MaxClients 512
MaxRequestsPerChild 512
</IfModule>
<IfModule worker.c>
StartServers 5
MinSpareThreads 32
MaxSpareThreads 64
ThreadsPerChild 512
MaxClients 512
</IfModule>
Code:
<IfModule prefork.c>
StartServers 10
MinSpareServers 5
MaxSpareServers 50
MaxClients 256
MaxRequestsPerChild 5000
</IfModule>
<IfModule worker.c>
StartServers 10
MinSpareServers 5
MaxSpareServers 50
MaxClients 256
MaxRequestsPerChild 5000
</IfModule>
Naturally I thought it was some specific URL that was causing trouble, so I wrote (not-so-accurate) bash script:
Code:
#!/bin/bash
log=/var/log/httpd/domlogs/example.com
log2=/var/log/httpd/access_log
cpu=50

stop=false
while [ "$stop" == false ]
do
        load=`top -d 0 | head -8 | tail -1 | grep httpd | awk '{print $10}'`
        load=${load%%.*}
        if [ "$load" == "" ]
        then
                load="0"
        fi
        if [ "$load" -ge "$cpu" ]
        then
                tail $log
                echo "-----------------"
                tail $log2
                stop=true
        else
                echo 'Reloading ('$load'<'$cpu')'
                #sleep 1
        fi
done
exit 0
Unfortunately neither `ps` nor `top` can display accurate CPU usage if it`s run for short period of time, so this script would approximately detect when httpd process jumped up, but not accurately enough to catch the culprit script.

But then both apache and php have timeout options, so if it were some script that was causing trouble, php would simply timeout it after a minute.

I`m not sure where to go from here. Any ideas?
 
Old 08-06-2007, 03:37 AM   #2
unSpawn
Moderator
 
Registered: May 2001
Posts: 26,534
Blog Entries: 51

Rep: Reputation: 2604Reputation: 2604Reputation: 2604Reputation: 2604Reputation: 2604Reputation: 2604Reputation: 2604Reputation: 2604Reputation: 2604Reputation: 2604Reputation: 2604
What do the sites run on? Regular PHP-based packages or home brewn stuff? Anything interesting in any logs around the time CPU starts spiking? Do you run anything like Dstat or Atsar? With those you would have a macro view of the resource usage on the box, just in case CPU hogging is just a symptom. And instead of 'top' I would run 'atop', which can save process stats which you can replay later on.

As for running a script that checks processes here's my example (and it's just an example):
Code:
#!/bin/sh
# Purpose: Show some process info
# Args: 1: CPU
# Deps: Bash, GNU utils, procstat, lsof, netstat, strings
# Run from: manual or sched

# Prepstage functions
die() { echo "No value for CPU given, exiting."; exit 1; }

case $1 in [1-9]*) cpuval="$1";; *) die;; esac
/bin/ps --no-headers --sort=c -eo %C -eo pid,command wwwee|while read cpu pid command env; do
 if [ ${cpu%%.*} -ge $cpuval ]; then
        logtag="`/bin/date +%s` [${pid}]"
        line=$(strings -an1 /proc/${pid}/cmdline|xargs);
        echo "${logtag} cmdline ${line}";
        # procstat is at senko.net
        procstat -a "${pid}"|while read line; do echo "${logtag} proc ${line}"; done;
        lsof -w -n -p "${pid}"|egrep -v "(usr|lib)/"|while read line; do echo "${logtag} lsof ${line}"; done;
        netstat -anp | grep "${pid}"|while read line; do echo "${logtag} netstat ${line}"; done;
 fi;
done

exit 0
Don't run it from cron but use Heiner's periodic instead. And if you want it in syslog replace the logging echoes with 'logger'.
 
Old 08-06-2007, 03:54 AM   #3
UnrealX
Member
 
Registered: Oct 2005
Location: Lithuania
Distribution: Ubuntu, CentOS
Posts: 36

Original Poster
Rep: Reputation: 15
Homebrewn PHP scripts. Although it`s not very likely one of them is causing load as I have invested quite some time in optimizing them (and as mentioned earlier, PHP has timeout limits too). Currently I am monitoring error_log. My suspicion is that mod_proxy is causing load as error messages from mod_proxy appear just about when httpd process starts taking up more and more of CPU time.

Oh, and thanks for the shell script!
 
Old 08-06-2007, 01:37 PM   #4
UnrealX
Member
 
Registered: Oct 2005
Location: Lithuania
Distribution: Ubuntu, CentOS
Posts: 36

Original Poster
Rep: Reputation: 15
It seems that it really was mod_proxy.
These error messages are registered every time CPU usage jumps up:
Code:
[Mon Aug 06 17:07:20 2007] [error] [client xxx.xxx.xxx.xxx] proxy: error reading status line from remote server example.com
[Mon Aug 06 17:07:20 2007] [error] [client xxx.xxx.xxx.xxx] proxy: Error reading from remote server returned by /path/to/some/file.jpg
Files are always different so I assume it`s due to mod_proxy bug.
 
Old 08-06-2007, 02:19 PM   #5
unSpawn
Moderator
 
Registered: May 2001
Posts: 26,534
Blog Entries: 51

Rep: Reputation: 2604Reputation: 2604Reputation: 2604Reputation: 2604Reputation: 2604Reputation: 2604Reputation: 2604Reputation: 2604Reputation: 2604Reputation: 2604Reputation: 2604
Block 'em and see what happens? As in mod_security maybe?
 
Old 08-07-2007, 11:57 AM   #6
UnrealX
Member
 
Registered: Oct 2005
Location: Lithuania
Distribution: Ubuntu, CentOS
Posts: 36

Original Poster
Rep: Reputation: 15
I`ve managed to rewrite my scripts so they don`t rely on mod_proxy.
Hopefully httpd won`t hog all cpu resources now.

If anyone is experiencing similar mod_proxy errors, here`s a thread at apache`s bugzilla:
http://issues.apache.org/bugzilla/show_bug.cgi?id=37770

EDIT: Nope, that didn`t help. Error_log only has 404 errors (robots.txt, favico.ico, nothing too special). I have no clue what to do next.

Last edited by UnrealX; 08-07-2007 at 03:43 PM.
 
Old 08-10-2007, 11:38 AM   #7
unSpawn
Moderator
 
Registered: May 2001
Posts: 26,534
Blog Entries: 51

Rep: Reputation: 2604Reputation: 2604Reputation: 2604Reputation: 2604Reputation: 2604Reputation: 2604Reputation: 2604Reputation: 2604Reputation: 2604Reputation: 2604Reputation: 2604
I have no clue what to do next.
Maybe start by telling us what components and versions you exactly use, how your stuff is configured and maybe somebody could help shed a light on this.


proxy: Error reading from remote server returned by /path/to/some/file.jpg
Come to think of it, could this be collateral of some PHP inclusion vuln?..
Does it happen with one hosted site only?
A select set of remote URI's?
Any "weird" processes running under the Apache user perhaps?
 
Old 08-10-2007, 12:17 PM   #8
UnrealX
Member
 
Registered: Oct 2005
Location: Lithuania
Distribution: Ubuntu, CentOS
Posts: 36

Original Poster
Rep: Reputation: 15
Apache 2.2.3
PHP 5.2.3
CentOS 5
Core 2 Duo, 2GB RAM
httpd.conf
php.ini

There were nothing suspicious about URLs that mod_proxy was requesting. I was using mod_proxy for local url rewriting, however as I have mentioned above it is now disabled.

Currently I use my own `watcher.sh` script (posted above) to detect increased CPU usage and restart httpd in such case. I can now leave server unattended for longer periods of time, however it`s not a solution, rather a dirty hack.
 
Old 02-04-2011, 08:04 AM   #9
niXeN
LQ Newbie
 
Registered: Feb 2011
Posts: 1

Rep: Reputation: 0
UnrealX

Hi, did you find a solution for this?
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off
Trackbacks are Off
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
ksoftirqd cpu usage & load 100% mismacku Linux - General 1 02-25-2009 04:29 AM
Sometimes inetd makes CPU load 100% here kite Slackware 11 02-09-2006 01:20 AM
CPU Load Always at 100% freddie_leaf Debian 2 09-25-2005 06:59 AM
XMMS skipping and stopping under 100% cpu time (*not load) guardian653 Linux - Software 3 10-26-2004 07:16 PM
KDE : 100% cpu load ! Ben2210 Linux - Software 2 03-25-2004 07:55 AM


All times are GMT -5. The time now is 09:04 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration