From collectl's top process display, soon to be released
Naturally you can select swap as a sort column. Only trouble is findng a system with non-zero swap. All our systems have too much memory:

# TOP PROCESSES sorted by time (counters are /sec) 11:18:36
# PID User S VmSize VmLck VmRSS VmData VmStk VmExe VmLib VmSwp MajF MinF Command
21155 root R 159M 0 21044K 18836K 140K 4K 4684K 0 0 52 /usr/bin/perl
30596 root S 76192K 0 1868K 73128K 528K 212K 2216K 0 0 74 /opt/cmu/bin/MainMonitoringDaemon

-mark

Can you tell me why this particular metric seems meaningful to you? 'Using swap', as opposed to any other memory usage, is not under control of the process, itself, so you can't be thinking that something about the process (ie, a memory leak or 'excessive memory usage' without a leak...and by that I probably mean 'memory usage in excess of what you anticipated' which may or may not be memory usage in excess of what is actually reasonable) is directly the problem.

Since some of the numbers are quite large, there is clearly evidence of heavy memory usage, but whether that means that pages that are swapped out are somehow 'the problem' is not at all clear, but then you'd have to know much more to be able to make useful comments about that.

1 members found this post helpful.

After this much time, I wouldn't bet the OP is still interested in this thread.

But several other people have asked equivalent questions. I can think of several reasons a meaningful value in the SWAP column in TOP would be a useful metric.

Adding the RES column to a meaningful SWAP column, would be a valuable indication of a kind of resource use by the process. Like RES itself, that value wouldn't be directly under control of the process. It would depend on the level contention from other processes. But that is an expected aspect of a meaningful resource use measurement.

The useless SWAP currently in top added to RES just gives you VIRT.

It can't directly measure that, but it is a powerful hint. If the system doesn't have severe memory pressure, then use of swap space indicates stale anonymous memory. If there isn't some good reason for that memory to be stale, it is a good hint of a memory leak. Growing use of swap by a process over time is a much stronger hint. None of that unconditionally connects to or from a memory leak, but a hint can be useful when you don't initially know which process to suspect nor even whether to suspect a memory leak.

If you really wanted to understand the memory resource use of a process, you would want to divide VIRT into anonymous and non anonymous, then sub divide each of those into resident, non resident, and non existent.

A private page in memory is clearly "resident". A non resident page in a file mapping is clearly "non resident". An unshared page in the SWAP file is clearly "non resident". A demand zero page is clearly "non existent". But a good definition would be needed for how you would want to count shared pages if you could. In my opinion, a shared page would be an equal share fraction: A resident page shared by three processes would be 1/3 resident and 2/3 non existent in each process. A shared page in swap would similarly be 1/N non resident and (N-1)/N non existent(but that is a tricky call). (but in my opinion a shared non resident file mapped page would make more sense counted as a full non resident page in each process).

If you could decide on precise meanings for the categories and there were an efficient means to measure them (which I'm pretty sure there aren't) then you could have memory stats that really make sense and have practical meanings.

But in the true situations, we have measured memory stats that are all off (by we rarely know how much) from practical memory stats. So we must infer the situation from whatever else we know about the processes being run plus what we estimate from all the available not quite meaningful memory stats.

In the interests of science, I had a quick run at this. Brought up my F15 laptop with only a Gig and set up a bunch of tmpfs and dd'd 450 Meg into each.
Hmmm - that'll slow your system somewhat. Plenty of (fairly static) swap usage.

Observations:
- top (procps v 3.2.8 in F15) appears to use the VmSwap as discussed.
- free seems to agree with /proc/meminfo (for swap particularly)
- summing /proc/*/status for VmSwap doesn't agree with the above - and seems to drift. I'm guessing the use of kernel buffers rather than userspace acccessible RAM skews the numbers, but I can't find any correlation in the numbers (for example Firefox is the biggest swap user). Will have to knock up some code to allocate (real) memory and look again. Later maybe.

It seems that the OP has jumped to the conclusion that somehow the pages themselves that are swapped out are somehow causing the problem, or are maybe a direct symptom which points in the direction of the cause.

Actually, we don't even know that there is a real problem, or what it might be, and while it might be a case in which a reasonable guess has a reasonable probability of being correct, wouldn't it be better to get the OP to actually state the facts, rather than make a guess?



Quite possibly, but the OP's comment on that (or silence...that would probably indicate a lack of current interest) would be valuable.

The OP appears (to me, anyway) to be trying to draw conclusions from the swap usage, however measured, that swap usage won't support. Better measurements of swap don't really take us forward as much, in this case, as trying to find out what the OP has as a problem, and finding out what conclusions are being drawn from the swap usage. I'm pretty sure that there is or are some misconception(s), but without finding out what the OP was thinking, it seems impossible to clear up whatever misconceptions exist. (Or, maybe, I'm wrong and the OP has some sophisticated thinking that the rest of us don't seem to have caught on to. Knowing that would be worthwhile, too.)



Yes, yes, it is a hint that something is wrong (and in normal circumstances, that hint may just concern whether you bought enough memory, or not, but in this case, a fair amount of memory buying has already been done...) but the OP's question about tracking down the process(es) from which the pages that have been swapped come seems likely to be deceptive.

The OP hasn't explicitly stated that the intention is to point the finger of blame at the process(es) owning the pages in swap, but everyone seems to have leapt some distance in that direction; it is difficult to know whether something here needs to be debunked, if the actual thinking is only guessed at.

It is, after all, easy enough to come up with corner cases in which there is a blame free process whose pages get swapped and a culpable process which remains in memory, and that, in this kind of thinking would point the suspicion towards the blame free process...I say this is easy enough, it is perhaps less easy to get the numbers to be this large.

Yes, but I'd reckon that if (and we don't quite know this either, but it has to be a strong possibility) the OP is suffering from some very basic misapprehension about the process, this will have taken them little further on. What is the OP thinking, and does this kind of detail help? If I were to bet, my bets would be on one of:

• the OP has found some very pragmatic solution (eg, 'let's upgrade to v2.0 and see if that helps'), and, except for helping for the future, we aren't really doing anything worthwhile
• the OP has found some work around, that has made this problem 'go away' at least in the extreme form presented here
• the OP has learnt to live with it

Excellent, just excellent. The last time I heard this phrase used by anyone else, it was in a context something like "We fitted a V8 in the front of a Mini, and that made it go faster, and so, in the interests of science, we fitted another one into the back, to see whether that made it faster, too...It did."

Seeing that phrase again was like having an old friend back, and made my day.

Interesting discussion and we still don't really know what the actual problem statement is or why such a big deal is being made about swap. The one nugget that caught my eye is the one suggesting increasing swap being indicative of memory leaks. But aren't there much better ways to find leaks, such as looking at the memory usage itself?

From my experiences I've seen a great indicator of them is monitoring slab memory. If you find it steadily growing the next step would be to find if there is a specific slab type that is growing.

Of course all this assumes you've gathered the right information over time and can see if there are indeed any trends. In fact even if it's not slabs that are leaking but rather system buffers or even mapped memory, those too should be looked at.

-mark