[Apache] High Swap Usage

IPAddress · 10-02-2009, 10:59 AM

Hi.
Im running Apache 2.2.3 on a CentOS release 5.3 (Final) with 100 Sites.
I've notice that Apache is making my server Swap around 200 MBs.
"http://www.xxx.yyy.zzz/server-status" doesnt show me too much to, so i'm looking the beahaviur of specific httpd process.
ProcessID "18753" is the one for "http://www.xxx.yyy.zzz/server-status" in my browser.

This command show me (In KBs) how much virtual memory is that specific process using:
# /etc/init.d/httpd start
# grep Private_Dirty /proc/18753/smaps | awk '{ print $2 }' | xargs ruby -e 'puts ARGV.inject { |i, j| i.to_i + j.to_i }'
3012
... Running this command a lot of times it gives me the same output, but suddenly...
# grep Private_Dirty /proc/18753/smaps | awk '{ print $2 }' | xargs ruby -e 'puts ARGV.inject { |i, j| i.to_i + j.to_i }'
21708

Something make that process (and all the others httpd process too) to use a lot more memory!

• Part of my httpd.conf:
#############################################
Timeout 120
KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 3

<IfModule prefork.c>
StartServers 5
MinSpareServers 5
MaxSpareServers 20
ServerLimit 256
MaxClients 256
MaxRequestsPerChild 100
</IfModule>

<IfModule worker.c>
StartServers 2
MaxClients 150
MinSpareThreads 25
MaxSpareThreads 75
ThreadsPerChild 25
MaxRequestsPerChild 0
</IfModule>
#############################################

My server is running Mysql 5.1.34, vsftpd 2.0.5, BIND 9.3.4-P1 (as slave).
I couldnt found anything running in the specific time that httpd processes start to use that much memory.
What could be wrong?
Is there a tool i could use to track the problem?

Thanks in advance.
Bye!

johnsfine · 10-02-2009, 11:28 AM

Quote:

Originally Posted by IPAddress

This command show me (In KBs) how much virtual memory is that specific process using:
# /etc/init.d/httpd start
# grep Private_Dirty /proc/18753/smaps | awk '{ print $2 }' | xargs ruby -e 'puts ARGV.inject { |i, j| i.to_i + j.to_i }'

Why do you think the amount of Private_Dirty memory is a meaningful measure of "how much virtual memory is that specific process using"?

I don't know a meaningful measure (I know other not very meaningful measures, such as the VIRT column in top). But I think Private_Dirty memory is less meaningful than most measures.

Quote:

Running this command a lot of times it gives me the same output, but suddenly...

If I'm reading the units right, it suddenly jumped from 3MB to 21MB.

How many of these processes are there and how tight are you for memory? Is 21MB per httpd process big enough to worry about?

IPAddress · 10-06-2009, 12:35 PM

Quote:

Originally Posted by johnsfine

Why do you think the amount of Private_Dirty memory is a meaningful measure of "how much virtual memory is that specific process using"?

I don't know a meaningful measure (I know other not very meaningful measures, such as the VIRT column in top). But I think Private_Dirty memory is less meaningful than most measures.

If I'm reading the units right, it suddenly jumped from 3MB to 21MB.

How many of these processes are there and how tight are you for memory? Is 21MB per httpd process big enough to worry about?

Hi.

I read that Private_Dirty its more important than the VSZ (ps) or VIRT (top). DO you know anythink else to look for?

There are about 25-20 process running at the same time when they jump from 3 to 21 MBs (aprox) each one.
Most of the time its not something to worry about, however sometimes it is. For example... Yesterday my server go down for cause it was swaping a lot! (Arround 600 MBs, notice that its a VPS).
I would like to know why those process start to eat so much memory!

Thanks in advance.
Bye!

sleddog · 10-07-2009, 08:31 PM

Asuming you're using the prefork mpm then this:

MaxClients 256

Is probably too high for the amount of memory you have. Bring it down to maybe 25 for 512 MB RAM, or scale that number to the amount of memory you have (e.g., 50 for 1 GB). That's just a ballpark figure to start. Then monitor your memory/swap usage and watch Apache's main error_log for MaxClients notices (which will appear when Apache hits the MaxClients limit.) If you hit the MaxClients limit without any memory issues then you can safely increase the MaxClients setting a little.

IPAddress · 10-08-2009, 07:40 AM

Quote:

Originally Posted by sleddog

Asuming you're using the prefork mpm then this:

MaxClients 256

Is probably too high for the amount of memory you have. Bring it down to maybe 25 for 512 MB RAM, or scale that number to the amount of memory you have (e.g., 50 for 1 GB). That's just a ballpark figure to start. Then monitor your memory/swap usage and watch Apache's main error_log for MaxClients notices (which will appear when Apache hits the MaxClients limit.) If you hit the MaxClients limit without any memory issues then you can safely increase the MaxClients setting a little.

Hi.
I've found a script that should "guide" me in wich number should i use in "MaxClients" directive.

# cat /root/mem.stat

Code:

#!/bin/bash
echo "This is intended as a guideline only!"
if [ -e /etc/debian_version ]; then
    APACHE="apache2"
elif [ -e /etc/redhat-release ]; then
    APACHE="httpd"
fi
RSS=`ps -aylC $APACHE |grep "$APACHE" |awk '{print $8'} |sort -n |tail -n 1`
RSS=`expr $RSS / 1024`
echo "Stopping $APACHE to calculate free memory"
/etc/init.d/$APACHE stop &> /dev/null
MEM=`free -m |head -n 2 |tail -n 1 |awk '{free=($4); print free}'`
echo "Starting $APACHE again"
/etc/init.d/$APACHE start &> /dev/null
echo "MaxClients should be around" `expr $MEM / $RSS`

Running it...
# /root/mem.stat

Code:

This is intended as a guideline only!
Stopping httpd to calculate free memory
Starting httpd again
MaxClients should be around 13

I've changed my MaxClients to 15 in prefork.c, restarted apache and i'm going to monitor the logs.

I'll tell you the results!
Thanks for the help.

Bye.

IPAddress · 10-19-2009, 06:24 AM

Quote:

Originally Posted by IPAddress

Hi.
I've found a script that should "guide" me in wich number should i use in "MaxClients" directive.

# cat /root/mem.stat

Code:

#!/bin/bash
echo "This is intended as a guideline only!"
if [ -e /etc/debian_version ]; then
    APACHE="apache2"
elif [ -e /etc/redhat-release ]; then
    APACHE="httpd"
fi
RSS=`ps -aylC $APACHE |grep "$APACHE" |awk '{print $8'} |sort -n |tail -n 1`
RSS=`expr $RSS / 1024`
echo "Stopping $APACHE to calculate free memory"
/etc/init.d/$APACHE stop &> /dev/null
MEM=`free -m |head -n 2 |tail -n 1 |awk '{free=($4); print free}'`
echo "Starting $APACHE again"
/etc/init.d/$APACHE start &> /dev/null
echo "MaxClients should be around" `expr $MEM / $RSS`

Running it...
# /root/mem.stat

Code:

This is intended as a guideline only!
Stopping httpd to calculate free memory
Starting httpd again
MaxClients should be around 13

I've changed my MaxClients to 15 in prefork.c, restarted apache and i'm going to monitor the logs.

I'll tell you the results!
Thanks for the help.

Bye.

Hi.
After some days, the Swap stays at the same low level. This fix my problem!
Thanks for your help!

Bye!