RHEL 5 Cache Memory Issue

carters2 · 09-04-2009, 08:52 AM

I have noticed something strange about one of my RHEL 5 servers and I do not know what is causing the problem. Using HypericHQ I am monitoring the health of all of my servers to catch problems before they really become a problem.

Looking at the free memory chart that it generates, every day at about 4:00am the amount of free memory plummets and the cached memory shoots up. There is noting that I know of that is set to run anywhere near that time unless it is something for the OS. Does anybody know what may be causing this?

This server is running the following..

Apache-Tomcat 6.0.18
-Hudson
-Jtrac
-vqwiki

Hyperic-HQ-Agent

Application backup script for Hudson, Jtrac, and Wiki take place at 9:00pm every day. The data directories are tarred and gzipped and sent off to a Samba share. This process used to take up a lot of memory when it ran as well but at the end of the script I added "sync; echo 3 > /proc/sys/vm/drop_caches" to fix it.

johnsfine · 09-04-2009, 09:42 AM

Quote:

Originally Posted by carters2

at the end of the script I added "sync; echo 3 > /proc/sys/vm/drop_caches" to fix it.

That was "fixing" something that wasn't broken. Your change probably won't do much harm, but it certainly won't do any good.

I have no guess what is doing a lot of file I/O on your system daily at 4AM. I hope my post won't distract anyone who might have a good guess from jumping in.

Maybe you want to do something that will reduce the amount of cache memory ever taken by your low priority file I/O intensive tasks. Doing so would be difficult and probably would have minimal or negative value, but there is at least some small chance there is some useful adjustment to be made.

But after the low priority task has already used a lot of ram for cache, there is no benefit to making the cache abruptly give all that memory back. It is easy to do that "fix" but absolutely pointless. Anything run after the low priority task that wants memory will take as much as it wants from the older cache use. Freeing the memory in advance does nothing to make that memory easier to take.

There are complex behaviors for the priority of ram use when newer demands on the cache contend against older non cache use of memory, and one or the other might win. But when newer non cache memory demands contend against older cache, it is simple: the newer memory demand wins.

carters2 · 09-04-2009, 12:55 PM

Quote:

Originally Posted by johnsfine

That was "fixing" something that wasn't broken. Your change probably won't do much harm, but it certainly won't do any good.

I have no guess what is doing a lot of file I/O on your system daily at 4AM. I hope my post won't distract anyone who might have a good guess from jumping in.

Maybe you want to do something that will reduce the amount of cache memory ever taken by your low priority file I/O intensive tasks. Doing so would be difficult and probably would have minimal or negative value, but there is at least some small chance there is some useful adjustment to be made.

But after the low priority task has already used a lot of ram for cache, there is no benefit to making the cache abruptly give all that memory back. It is easy to do that "fix" but absolutely pointless. Anything run after the low priority task that wants memory will take as much as it wants from the older cache use. Freeing the memory in advance does nothing to make that memory easier to take.

There are complex behaviors for the priority of ram use when newer demands on the cache contend against older non cache use of memory, and one or the other might win. But when newer non cache memory demands contend against older cache, it is simple: the newer memory demand wins.

Well we were having an issue that once the cache filled up it was not releasing anything for processes to use and then the machine would stop responding and need to be rebooted in a not so nice way.

johnsfine · 09-04-2009, 01:07 PM

Quote:

Originally Posted by carters2

Well we were having an issue that once the cache filled up it was not releasing anything for processes to use and then the machine would stop responding and need to be rebooted in a not so nice way.

I'm nearly certain you have diagnosed that problem incorrectly.

If you were correct, then you would want to deal with whatever was stopping the cache from properly releasing memory, rather than kludge around the problem by dropping the whole cache.

But since your diagnosis of the problem is not correct, you should find out what the real problem is rather than doing something that isn't likely to have any effect on the real problem.

One of the things you should do is a memtest. If you investigated well enough that there was some detailed support for your idea that high cache use is related to the problem, then the underlying problem is more likely hardware and would be found by a memtest (some of your ram is bad and you may get wrong results for some uses of that ram and system crashes for further use of that ram).

carters2 · 09-04-2009, 01:50 PM

Quote:

Originally Posted by johnsfine

I'm nearly certain you have diagnosed that problem incorrectly.

If you were correct, then you would want to deal with whatever was stopping the cache from properly releasing memory, rather than kludge around the problem by dropping the whole cache.

But since your diagnosis of the problem is not correct, you should find out what the real problem is rather than doing something that isn't likely to have any effect on the real problem.

One of the things you should do is a memtest. If you investigated well enough that there was some detailed support for your idea that high cache use is related to the problem, then the underlying problem is more likely hardware and would be found by a memtest (some of your ram is bad and you may get wrong results for some uses of that ram and system crashes for further use of that ram).

I am actually not the one who diagnosed the problem and I have actually never seen the problem happen myself. I was told that the cache filling up what the problem was and went with it. The server has not had problems since implementing the clearing of the cache so i don't really know whats going on with this machine. The problem is I don't want to stop clearing the caches now because this server is very important to development project. If it goes down because i remove the cache clearing i will never hear the end of it

TBC Cosmo · 09-04-2009, 02:22 PM

Quote:

There is noting that I know of that is set to run anywhere near that time unless it is something for the OS. Does anybody know what may be causing this?

Looking at /etc/crontab in a default RH insatll, I can see that the scripts in /etc/cron.daily are run. Highlighted below are housekeeping jobs that update (2) databases. The 'whatis' database and the 'locate' database.

Both of these are filesystem read intensive, and will most certainly populate the filesystem cache. Perhaps this is what is happening in your case.

However, this particular data that's now in cache is clean, and can be immediately freed without need for writing changes to disk. So any memory requirements should be readily satisfied from this.

Code:

[root@SL53-vm1 ~]# cat /etc/crontab
SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=root
HOME=/

# run-parts
01 * * * * root run-parts /etc/cron.hourly
02 4 * * * root run-parts /etc/cron.daily
22 4 * * 0 root run-parts /etc/cron.weekly
42 4 1 * * root run-parts /etc/cron.monthly
[root@SL53-vm1 ~]# man run parts
No manual entry for run
No manual entry for parts
[root@SL53-vm1 ~]# man run-parts
No manual entry for run-parts
[root@SL53-vm1 ~]# ls /etc/cron.daily/
0anacron  0logwatch  cups  logrotate  makewhatis.cron  mlocate.cron  prelink  rpm  tmpwatch  yum.cron
[root@SL53-vm1 ~]# cat /etc/cron.daily/mlocate.cron 
#!/bin/sh
nodevs=$(< /proc/filesystems awk '$1 == "nodev" { print $2 }')
renice +19 -p $$ >/dev/null 2>&1
/usr/bin/updatedb -f "$nodevs"

syg00 · 09-04-2009, 05:27 PM

Updatedb was historically appalling for polluting the cache - mlocate was designed to fix that. If you are still seeing problems every day (and RHEL really uses mlocate), you must be creating boatloads of files. Every day.
I can't imagine whatis being the problem ...