How to check swap usage of a process
 

Hello,

I have a Red Hat database server that has some processes hitting the swap space pretty heavily. I know I can use top to see what percentage of Memory certain processes are using, but I was wodering if there is a way to see how much Swap certain process are using. If anyone knows a way to do this by either percentage or units on a specific PID, that would great.

Thanks!

Swill

Linux doesn't swap. It pages. So the "swap" partition is a misnomer.

Swapping works by transferring low priority processes to the swap partition when memory demand exceeds the available RAM. In effect low priority processes are suspended and all of their RAM is given to higher process processes. Linux does not do this.

Paging works on individual virtual pages with no reference to process. When memory demand exceeds the available RAM then Linux selects virtual pages which have had the least amount of recent usage, writes them to the swap partition, and uses that RAM to satisfy the most recent memory requests. When Linux is paging any process can have part of its memory in active RAM and part residing on the swap partition.

How much paging in going on at any given time is a function of total memory demand versus the amount of RAM available. You want to look at the total memory demand during periods of swap usage and how much memory each active process is using. Which processes happen to have some pages residing in swap is a matter of chance (as viewed from the process). You solve the swap problem by reducing peak memory demand or buying more RAM.

-------------------
Steve Stites

So?

The question was about use of that space. Not about what the word "swap" ought to mean.

Not helpful.

I think that (no reference to process) is not correct. But I don't have the detailed info to back it up. Anyway, it is off topic for the question.

Not correct. Only "dirty" pages can be written to the swap partitions. Lots of pages are "clean", meaning the page is a perfect copy of something somewhere on disk and Linux is able to hold onto that association. Typically, that means the page is part of an executable binary read from disk. Linux can just free the physical memory used by any clean page. When it needs the page again, it just rereads it.

With memory mapped file I/O, you can have even dirty pages that are written, but not to the swap space.

In top, you can enable the SWAP column, but it doesn't represent the amount in the swap space. It includes pages in files that are part of the process's virtual address space, but aren't in physical memory (or maybe are in physical memory, but not connected to that process).

Or many other places.

I don't know if the OP asked the question that really fits his underlying issue. Usually you should just care how much paging is happening, not how much space is occupied in the swap space.

But it still is a reasonable question. And I don't know the answer.

Logically, some identifiable portion of each process's memory is located in the swap space. Like pages of physical memory, a page in swap space could belong to more than one process. But, unlike pages in physical memory, that isn't very likely. Most pages in swap space belong to just one process. So you can ask what fraction of physical memory belongs to a specific process, and get an answer that may be very misleading because of various sharing, caching and buffering effects. If we knew how to ask what fraction of swap space belongs to a specific process, there wouldn't be as much making the answer misleading.

Because of sharing, that is a hard value to even define. Measuring it is even harder. Also, you really want a meaning of "using" in the above that would exclude stale pages of the process's virtual address space that are out because they aren't needed at the moment, but include pages that are out because the memory load is too high. That is pretty much impossible to measure.

Some processes have a lot of pages that when out must go to swap (as opposed to pages that are mapped to a file). Maybe this isn't what the OP thought he was asking. But there is an interesting distinction between pages that are out to the swap space and pages that are out (of physical ram) elsewhere on disk. If I was trying to diagnose an excess memory load in a database server, that is one factor I'd like to be able to measure.

My point is that the space is being used for paging, not swapping.

------------------
Steve Stites

This discussion and others would be easier if a consistent set of terminology were available. But it isn't. I'm well aware of attempts to define "swapping" to mean what you say it means. But those attempts have mostly failed. The first uses of "swapping" in computer memory management (in the 1960's) meant what you call "paging" and most uses of "swapping" in computer memory management since then have meant paging.

Certainly in Linux, "swap" does not mean what you want it to mean. It doesn't have a consistent meaning either (note the big difference between "swap space" and the "swap" column in top).

Within the real world of inconsistent terminology, the OP seems to be asking about the use of swap space (meaning what "swap space" normally means in Linux). That is a reasonable question and I wish someone would answer it here.

Even if the OP is or should be asking about some other aspect of paging, it isn't helpful for you to distract from the question by rejecting the word "swap". "Swap" is used in Linux for various aspects of paging and we all understand it as such (within that ambiguity) and not as some version of the meaning of "swap" that we all agree is something Linux doesn't do.

Wow, this went a totally different direction than I meant. It's a pretty simple question really. I type "free -m" and all my swap space is used. I just want to know if there is a way to see (at the process level) how much of that space is being used by a specific process or even just "if" certain processes are using swap space "at all". If this isn't possible, then fine, we're done... no need to go into detail about swap vs page, etc... I got bigger fish to fry.

Swill

IMHO posting something helpful instead would be a better course of action instead of trying to nail a fellow LQ member on sematics, don't you agree?

Mr. Swillis has a swap problem

He then wants to analyze the problem this way:

If Linux swapped then the second question is a good way to begin analyzing the swap problem. If Linux is paging then the answer to the second question is an irrelevant red herring. Both johnsfine and I replied with an explanation of how Linux paging works and the same conclusion of how to solve the swap problem. Neither of us had any need for the answer to the second question in our explanation of how to solve the swap problem. The answer to the second question is irrelevant to the problem. Neither johnsfine nor I know the answer to the second question and since it is irrelevant we did not bother to look it up.

Swapping came first. You could swap on the hardware that existed in the 1960s but you could not page. Paging requires hardware virtual page tables and swapping doesn't. Swapping was invented first and it meant swapping.

"Swapping works by transferring low priority processes to the swap partition when memory demand exceeds the available RAM. In effect low priority processes are suspended and all of their RAM is given to higher process processes."

Swapping was of marginal usefulness in the batch systems typical of the 1960s. Then came time sharing where each user was run for a time slice and then suspended while other users got their time slices. Swapping was essential for time sharing and as the popularity of time sharing increased rapidly swapping became widely used.

Hardware makers then invented virtual page tables. Now using a combination of hardware and software virtual memory was possible. Virtual memory using paging is a far more effective way to manage memory than swapping:

"Paging works on individual virtual pages with no reference to process. When memory demand exceeds the available RAM then the OS selects virtual pages which have had the least amount of recent usage, writes them to the page file, and uses that RAM to satisfy the most recent memory requests. When the OS is paging any process can have part of its memory in active RAM and part residing on the page file."

Paging is so superior to swapping that paging eventually drove swapping out of existence, or at least marginalized it to the point of obscurity. During the time that both swapping and paging existed there was no confusion between the terms. In the intervening years since swapping died out many people, and in particular the kernel developers, have become sloppy about the distinction between swapping and paging. This has led to a huge amount of confusion between the people who use swapping in its true definition and people who treat swapping and paging as being the same thing. The kernel discussion forums are full of wrangles between users who complain about Linux's horrible swapping behavior (they think that Linux swaps processes and want to be able to control Linux's swapping behavior) and developers trying to explain that Linux pages and does not swap at all. The same developers then confuse things further by trying to explain the difference while still referring to what Linux does as swapping. Here is an example which actually parallels the questions and answers being debated in this thread.

http://lwn.net/Articles/83588/

----------------
Steve Stites

In an attempt to answer the original question while not getting tangled in the semantics, I think that the answer is that you can't do what you want to do.

I am not positive of this, but the way I understand the memory manager to work means that the kernel does not know or care what memory for what process has been paged out. The memory manager converts requests for memory access from a process to the virtual memory page that belongs to that process, and if the page is not in RAM, it gets loaded. This results in the kernel needing to know what memory belongs to what process, but no need to know what process owns what memory, if you see the difference. Basically, you can chain forward to identify the memory that belongs to the process, but you can't chain back to find the process that owns the memory.

At least, I think that is how it works.

You could root through the process tables to identify any particular page's owner, I suppose, but the time it would take could make the knowledge useless; the system is moving pages around all the time.

There is a patch to expose this data via smaps - it's there on my 2.6.26-rc9 system, so stay tuned !!!!

That's what I thought you wanted, and if I knew even where to look for the answer, I would have said something more constructive. It sounds like syg00 has the answer.

Meanwhile (above), since I wasn't convinced the answer to that simple question (if we had it) would help you, I tried to clarify some things about paging (primarily that most of it may be to and from places other than the swap space). So whether the swap space is empty or full, the performance issues you might think are related to swapping likely are not related to the use of swap space.

Sorry about my contribution to the semantic sidetrack. I was trying to reduce that sidetrack and instead added to it.

That difference applies to specific pages. But the question is about total quantity. Top reports the total amount of resident memory belonging to a process (if Linux purely tracked just the process to page link, it would have no better access to that stat than to a swap space stat). Swap space is a resource in need of similar management. In that sense it is distinct from other kinds of virtual memory. If a process consumes its own virtual memory by a read only mapping to a file (such as the binary executable it was loaded from) the only resource that takes from the system is page table space at a fraction of a percent of the size of the mapping. But either (depending on your point of view and your "overcommit" settings) when a process maps modifiable private data, or when it actually modifies it, that must consume either physical memory or swap space.

A slightly more interesting statistic (from my point of view) related to the original request, is how many (total, not specific) pages the task has which can live only in physical ram or swap space. But I'm also interested in hearing further from syg00 or others about the specific stat originally requested.

I'm a bit surprised to hear it requires a patch. It seems like the sort of thing that would need to be tracked for good management of the system's commit level.

In most real world cases, disk space is cheap enough that it makes more sense to increase the size of the swap partition rather than worry about it. That's why the paging I/O (which is available from top) is still more interesting for diagnosing the aspect of excess paging that usually matters more.

Coming from another universe, I find the dearth of performance and tuning metrics in Linux astounding. Not to mention the total lack of historical records once a process terminates. Recently some progress has been made, but it has a looooong way to go.
Calculating accurate memory usage is damn near impossible - even the kernel devs can't agree on a decent definition. Again, that has recently changed, but I don't know I necessarily agree with what they decided on. Likewise for swap - there is swapcache in there as well folks. Go grep /proc/meminfo for "swap" and contrast it to the number "free" reports. mmmmm.
kswapd generally wouldn't care about the reverse mapping - until the memory pressure becomes so high as to trigger page reclaim, and force anonymous pages out to physical swap extent(s). A quick look at the numbers I was able to generate make it look like the smaps numbers might be for cache entries, but I'll need to look at that properly.

The linux memory mapping mechanism is really, truly, fugly. I mean, genuinely f---ing ugly. I have studied the kernel docs on this, and it is a rough ride.

Redacted:
I just ran a couple of tests, and extracted the swap numbers for all tasks listed in /proc/*/smaps.
The total was within 1.7-2.2% of the cache usage reported by free minus swapcache. So it would appear the (new) numbers exposed via smaps do indeed indicate actual (disk) swap usage on a per process basis.

(apologies for the initial posting - my scripting to reduce the data was crap .... :( )

All changes to the mainline are submitted as patches - this is not an "out of tree" patch. As I said, it's included in my -rc9, so we can expect it in the supported vanilla kernel soon. And it's merely the reporting (via smaps) side of things; quite a small change actually.
Sorry for any confusion.