I have a number RHEL 4 64 bit boxes hosting Oracle. The two most heavily populated servers have been having an issue with swap usage. Every night during the backup window (filesystem backup), memory spikes, and the swap file nears 100% usage. The memory lowers, but whatever process is using swap holds on to it. There is nothing in top to indicate what process it is.

There is no issue with the lower used boxes. The two servers in question have 128GB ram, so it's not like the swap is needed for backup processes.

After a reboot, everything goes back to normal. The problem is, the high swap usage causes performance problems, thus requiring a reboot, then there is little to investigate, since nothing is showing in the logs.

Is there any way to determine exactly what process is holding on to swap space? That would at least give a direction to look in.

1. "top" definitely shows you which processes are consuming the most memory. If you're swapping, then the processes shown in "top" by definition include the processes using "swap".

2. But the best answer to your question is "procfs". For example:PS:
PID's don't "hold on" to swap. They *can* hold on to memory ... and if there isn't enough RAM in the system, the system will use swap. In other words, the only solution is to a) make the process use less memory, or b) add more RAM to the system.

Thanks for the response. I know nothing is <i>supposed</i> to hold on to swap space, but something is. I'll grab the top output next time it happens. It shows basically 60-some GB of free memory, and swap at 98%. I know it dowsn't make any sense, but there it is...

I think that is perfectly normal behavior and does not indicate anything that should be corrected.

During backup, the kernel decides that file caching is a more important use of memory than keeping idle pages of tasks that aren't doing anything at that moment.

Because it is a backup operation on a system with a very large amount of ram, the kernel is probably wrong in deciding to do so much file caching. There may be a way to make the backup process make the kernel understand that less file caching is appropriate. But the benefits would be trivial compared to your effort in figuring out how to modify the backup code. I think you should just leave it.

Because you have so much ram and because the backup operation has finished, the only pages that remain in swap are pages that are not accessed at all by the tasks that own them. Keeping those pages in swap is optimal for system performance.

The only non optimal swapping is the pages that are pushed out by file caching during backup and pulled back in after the backup, and that is only slightly non optimal. Pages that are pushed out and stay out are pages that should stay out.

I was in another thread recently where someone posted a simple shell command to extract the swap usage from /proc files (and I posted what I think is a modest improvement to that command).

It's not "needed" but it does slightly benefit the backup process and then the cost of bringing the pages back is tiny. So you have a tiny benefit for a tiny cost. Not something to worry about.

That is a big waste.

I'm nearly certain you are wrong. The high swap usage does not cause a performance problem and there is no benefit to rebooting.

Look at that other thread, so you can satisfy your curiosity about this non problem.
http://www.linuxquestions.org/questi...92#post3797292