Why Swap is 100% used while there is almost 30% RAM available / unused on this host?

RAM and Swap Info:


OS Info:

top and ps commands can tell me how much memory is being used by any process but how do we exactly figure out what process is making use of swap the most?

Here is the output from top command as well:

Since you have zero buffers and zero cache, I expect you are not running Linux directly on the hardware nor in a clean VM. You have some kind of quasi-virtual or more likely split OS.

Memory use stats in such systems are not generally accurate and methods of diagnosing memory use must be specific to the type of quasi-virtual system you are running (not the same as methods for diagnosing memory use in ordinary Linux systems). If you know what type of virtual system you have, post that and someone here might know more specific details.

Your 2GB use of swap might still be true and might represent a memory leak in some process that should be diagnosed. Finding actual swap use in processes is a bit tricky. I recall some scripts posted in other threads here at LW, for parsing those details out of /proc/*/smaps but at the moment I'm not sure what keywords would make an effective search to rediscover those threads.

It's an HP Server.


System Information

The one line script in the first post of this thread might answer your question about swap usage:
http://www.linuxquestions.org/questi...3/#post4906470

I misread your initial post and thought the buffers and cache indicated something strange. On a more careful look at your first post I see they are less strange than I thought.

Did you do something to drop caches as an earlier attempt to correct a memory use problem?

There are two very different (probably unrelated) memory use mysteries in the data you provided:
1) What used up all the free memory at some earlier point in time, but was no longer using it when you got the memory use data you posted?
2) What allocated and still holds (but isn't accessing) the 2GB of anonymous data in swap?

The simple explanation for (1) would be file caching, assuming this is a large active server of some kind and you recently made a misguided attempt to correct a memory use problem by dropping caches. If you did not do that, there are many other possibilities for (1) but any of them I can think of are unusual.

The simple explanation for (2) would be a memory leak in some process. That would be important to diagnose. One of many alternate explanations is some service mis-configured to pre allocate far more resources than it is actually using.

The script is very helpful indeed:

I have tweaked it to be more helpful (to me, at least):

so that the above script will not show files where Swap is "0" or null.

It appears that some process needed a lot of memory, forcing swap to be fully used, and then terminated. Pages that were written to swap during a time of heavy memory usage will not be paged back in again until they are needed.

Rounding to the nearest Gigabyte can make it hard to interpret the results. The output from free with "-m", or the default ("-k"), units might be less confusing.

Hi

The swap and buffer is not empty, just lower than 0.5Gb.

If you find the process in swap, it could be completely innocent. You can get this situation easily if you have one program start, use a lot of RAM and then quit or be killed. The kernel will put all other programs in swap, give the memory to the memory hog, and it will end up free when the program ends.

Edit:
rknichols posted while I wrote, but say the same thing.

That is one possibility for the condition of the system as reported. It is not the only possibility and not my first guess.

That is an important point for the OP to understand (and true regardless of what was previously using a lot of ram and no longer doing so).

If I was at all competent in awk, I would have made the same enhancement inside the awk portion. Your approach is less efficient, but it gets the job done and is in the Unix spirit of stacking simpler filters instead of doing a job in one more complicated step.

Yes, I feel stupid for initially misinterpreting that.

It could be completely innocent. In many threads similar to this, I'm the one explaining that some swap use is normal and doesn't indicate anything wrong.

But this time, I think the swap use is more likely to be a symptom of something wrong. 2GB of stale anonymous ram is a lot, even for a big system like that with 88GB ram in use.

Well, there was a process "Alert Engine" which was restarted and the RAM looked about 70% free and the SWAP also looked about 81% free.

Can you paste the output of below command ?

vmstat 1 40

Mybe you should scale your numbers properly.
Used swap is around 1.5% of total installed RAM
Used swap is around 2.2% of currently used RAM

Why do you care ?.

With 37GB of RAM unused, 1 GB of Swap being fully used must mean something is wrong. The basics tell me Swap is utilized only when RAM is low.

Nope.
Memory management isn't that simplistic. Long term unreferenced pages can be moved to swap anytime.

Pages are not moved to Swap when there is a large amount of ram unused. But once moved, pages stay in cache until needed again.

It is normal for very little ram to be unused, even when it would not be meaningful to say "ram is low". You might have had a very large amount of ram used by buffers and cache, causing stale anonymous pages to be moved to swap, even though ram was never "low".

You showed a snapshot of the system with a very small fraction of ram used by buffers and cache. How it got that way is an important question.

I have almost forgotten which server here had the issue. So I cannot recheck it.

Moderators:

If you feel we have come to a conclusion wrt the original question or our discussion on the differences why Swap is largely used when a large amount of RAM is available then please close the topic or I will mark it so.