Horribly high pageout rate on RHEL
 

I have 2 machines running an Apache/Subversion/Python/Trac setup.

One runs SLES, the other RHEL. Both are virtual machines. We are having performance issue with the RHEL machine, though.

Both VMs are allocated 1G of memory. Both machines are utilizing an average of 200MB, so a lot of room to go.

However, the RHEL page-out rate is very high, in the neighborhood of 180 pages/second, while SLES is at about 10 pages/sec.

SLES is clearly not maxing out the RAM, since it has, on average, about 90% free. So why this high paging rate?

The other reasons for paging are that the operating system is retrieving new data or software from the disk or writing data to a data file (not the swap partition).

You could use the top utility for a quick look at what the operating system is doing.

I recommend that you run sar. It is in the sysstats package. You can run sar against the running system. If you do this it is like running vmstats and iostats at the same time, more or less. Sar is more useful if you run its data collector in the background and then use sar to create reports based on that data.

You have to configure sar. In my distributions it is installed in completely useless mode. You have to edit the configuration file and the cron jobs to get the data collector to start. Once the data collector is running automatically it will record system activity and resource usage. Let it run for a few minutes or an hour when the system is paging heavily. Then get a few reports from sar. You pass parameters to it to tell it what parts of the system you are interesting in investigating.

Once sar is installed you can find more specific information by using the man pages.
The sar utility can make more in depth reports than top but it takes a little bit of time to collect the data.

Well, I did some work with sar and pidstat.

Unfortunately, it didn't tell me anything new. Pidstat doesn't list the page-out rate for processes, and while sar lists the pageout rate, its for the entire system. So it doesn't help me pinpoint the source paging (all I know is that it peaks around 4AM when the cron jobs run)

Is there any way to find the page out rate for an individual process?

Start by looking at what the cron jobs are doing.
Setup some logging and spreadout the cron jobs (timewise).

I found the culprit--'avahi-daemon'.


When I turned it off the memory usage and paging rate got muuuuch better