LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices


Reply
  Search this Thread
Old 01-10-2011, 09:16 AM   #1
kenneho
Member
 
Registered: May 2003
Location: Oslo, Norway
Distribution: Ubuntu, Red Hat Enterprise Linux
Posts: 657

Rep: Reputation: 40
How to determine processes swap usage


Hi.


On one of my servers the "free" command tells me that a lot of swap space are in use. What I'd like to do is to determine which processes have been swapped out. I tried issuing "top" and sort by the "swap" column, but this doesn't seem to provide correct values - when performing the same excersize on another server with close to no pages swapped out, the sum when adding the swap value for each process greatly exceeds the swap usage reported by "free".

So how do I go about determining the swap space used for individual processes?

- kenneho
 
Old 01-10-2011, 11:23 PM   #2
tommylovell
Member
 
Registered: Nov 2005
Distribution: Fedora, Redhat
Posts: 372

Rep: Reputation: 101Reputation: 101
Ok. It's been 14 hours with no reply. I don't know the answer, but I can supply what I know, plus some speculation...

Quote:
On one of my servers the "free" command tells me that a lot of swap space are in use. What I'd like to do is to determine which processes have been swapped out.
You may be able to do this in a different manner than 'top', but it might be hard to make it reconcile with swap in use.

Quote:
I tried issuing "top" and sort by the "swap" column, but this doesn't seem to provide correct values - when performing the same excersize on another server with close to no pages swapped out, the sum when adding the swap value for each process greatly exceeds the swap usage reported by "free".
Here I can help. One thing that is ingenious about linux is that code is read-only; while variables within a program and malloc'd areas are read-write. You can see this if you 'cat /proc/<process-number>/maps'. Each executable is usually listed at least twice: the first one usually has r-xp attributes, this represents the "code segment"; another entry is usually rw-p and is a "data segment" containing variables. The malloc'd areas are rw-p and have 00:00 for a file descriptor.

Take the case where the system is running out of real storage and wants to steal a memory page frame. If that area is code (read-only), it does not need to write it out to swap space (there is already a copy of it on disk where it was originally loaded from); If the page frame that is being stolen has been modified then it needs to be written to swap. When a page fault occurs, if the area is code, it gets "paged" back in from the executable's location on disk; if the area is modified data, it gets paged back in from swap.

On top of that, once a page has been swapped in from swap space into real memory it is left out on swap. The rationale behind that is that if a page needs to be stolen in the future and the page has not been modified since being swapped in, the system can just steal that page again without doing any I/O. The page is not cleaned out of swap space unless swap is being exhausted or the process ends.

So, top's swap value will not equal used swap space. At times top's swap will be much larger than used swap space (mostly read-only pages stolen); and sometimes used swap space may seem bloated compared to what top says is swapped out (memory pressure relieved and everything swapped back into real memory). Or a mix of these conditions.

To complicate matters more (from an accounting perspective), shared objects (.so files) and other memory can be shared between processes...

Quote:
So how do I go about determining the swap space used for individual processes?
This is speculative (and I hope someone can correct me if I am wrong), but I think you can use the smaps data in the proc filesystem to determine what is out on swap. ('cat /proc/self/smaps' to see your own map.) I think that 'Size' and 'Rss' values can tell you how big a virtual memory area is and how much of it is in real storage. For swap usage, you'd only be interested in entries that were writable. And if it was out on swap and then swapped back in you may not be able to account for those pages left on swap.

An interesting thing to do on an unimportant server that has multiple swap filesystems and many heavy duty long running processes (like Oracle): after the system has used a lot of swap and the memory pressure has been relieved, 'swapoff' a swap space and watch how, as it gets moved to a different space, it gets trimmed down. The pages that faulted and were brought back into memory are not copied to the new swap space location.

Also, you'd have to figure out the copy-on-write mechanism ties into this.

But like I said I'm speculating here and might be completely off-base.

If you actually do figure this out I am certainly interested in your findings. But I think if it could be done someone would already have created a utility to do it.

Good luck.

Last edited by tommylovell; 01-10-2011 at 11:28 PM. Reason: typos
 
1 members found this post helpful.
Old 01-10-2011, 11:45 PM   #3
tommylovell
Member
 
Registered: Nov 2005
Distribution: Fedora, Redhat
Posts: 372

Rep: Reputation: 101Reputation: 101
I had replied based on what /proc/<pid>/smaps looks like on a RHEL 5.2 2.6.18 system looks like.

I just looked on a Fedora 14 2.6.35 system. smaps now has a Swap: field. The man page for 'proc' does not document it, but it is there.

Unfortunately, it does not appear to contain a value except '0'. Bummer.
 
Old 01-11-2011, 02:53 AM   #4
kenneho
Member
 
Registered: May 2003
Location: Oslo, Norway
Distribution: Ubuntu, Red Hat Enterprise Linux
Posts: 657

Original Poster
Rep: Reputation: 40
Thanks for your very informative answer! I learnt a couple of new things about swap I didn't know before. I'm running a RHEL 5.5 server, and created this oneliner to sum up the swap used by the java processes running on the server. If the swap field in /proc/<pid>/smaps actually indicates the amount of swap the process is using, the onliner below may give the information we're after:

Code:
[root@server   proc]# for pid in `ps -ef|grep -i java| awk '{print $2}'`; do echo -n "Pid: $pid  "; cat /proc/$pid/smaps |grep -i swap| awk '{SUM += $2} END {print "SUM: " SUM " kB (" SUM/1024 " MB)"}'; done
Pid: 1683  SUM: 333756 kB (325.934 MB)
Pid: 3850  SUM: 68996 kB (67.3789 MB)
Pid: 4092  SUM: 6800 kB (6.64062 MB)
Pid: 4111  SUM: 6568 kB (6.41406 MB)
Pid: 6879  SUM: 174040 kB (169.961 MB)
Pid: 7291  SUM: 230400 kB (225 MB)
Pid: 10660  SUM: 102060 kB (99.668 MB)
Pid: 11275  SUM: 1649528 kB (1610.87 MB)
Pid: 11850  SUM: 92388 kB (90.2227 MB)
Pid: 12478  SUM: 70912 kB (69.25 MB)
Pid: 13068  SUM: 85984 kB (83.9688 MB)
Pid: 13345  cat: /proc/13345/smaps: No such file or directory
SUM:  kB (0 MB)
Pid: 15060  SUM: 153896 kB (150.289 MB)
Pid: 17451  SUM: 162532 kB (158.723 MB)
Pid: 19645  SUM: 580336 kB (566.734 MB)
Pid: 20565  SUM: 82600 kB (80.6641 MB)
Pid: 20924  SUM: 394072 kB (384.836 MB)
Pid: 22378  SUM: 1213984 kB (1185.53 MB)
Pid: 22766  SUM: 0 kB (0 MB)
Pid: 23861  SUM: 90372 kB (88.2539 MB)
Pid: 23997  SUM: 61476 kB (60.0352 MB)
Pid: 26034  SUM: 100960 kB (98.5938 MB)
Pid: 28134  SUM: 346860 kB (338.73 MB)
Pid: 30311  SUM: 0 kB (0 MB)
Pid: 31243  SUM: 43400 kB (42.3828 MB)
Pid: 31951  SUM: 727000 kB (709.961 MB)
Do you think this gives the correct information?

- kenneho
 
Old 01-11-2011, 04:02 AM   #5
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 18,129

Rep: Reputation: 2923Reputation: 2923Reputation: 2923Reputation: 2923Reputation: 2923Reputation: 2923Reputation: 2923Reputation: 2923Reputation: 2923Reputation: 2923Reputation: 2923
Quote:
Originally Posted by kenneho View Post
Do you think this gives the correct information?
Sorta ... maybe.
I looked at this a while back, but IIRC I was unable to determine if that field included pages in swap-cache. So strictly speaking it may not indicate pages that are resident on the swap extent(s). Probably as close as we're going to get.

Last edited by syg00; 01-11-2011 at 04:04 AM. Reason: s/page-cache/swap-cache/
 
Old 01-12-2011, 01:38 AM   #6
kenneho
Member
 
Registered: May 2003
Location: Oslo, Norway
Distribution: Ubuntu, Red Hat Enterprise Linux
Posts: 657

Original Poster
Rep: Reputation: 40
Quote:
Originally Posted by syg00 View Post
Sorta ... maybe.
I looked at this a while back, but IIRC I was unable to determine if that field included pages in swap-cache. So strictly speaking it may not indicate pages that are resident on the swap extent(s). Probably as close as we're going to get.
I'm not sure what you mean by swap-cache...could you link me to some info on the subject?
 
Old 01-12-2011, 03:18 AM   #7
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 18,129

Rep: Reputation: 2923Reputation: 2923Reputation: 2923Reputation: 2923Reputation: 2923Reputation: 2923Reputation: 2923Reputation: 2923Reputation: 2923Reputation: 2923Reputation: 2923
Google found this as the first result for me.
Close enough to be useful. Swap-cache is now compressed in storage, but may not apply to your kernel.
 
Old 01-12-2011, 10:07 AM   #8
tommylovell
Member
 
Registered: Nov 2005
Distribution: Fedora, Redhat
Posts: 372

Rep: Reputation: 101Reputation: 101
Quote:
Do you think this gives the correct information?
Close. On my Fedora system, very lightly loaded, using your script, mine totals 27,860KB.
'swapon -s' shows 27,772KB used.

Code:
[root@athlon ~]# swapon -s
Filename                                Type            Size         Used      Priority
/dev/dm-1                               partition       6160380      27772     -1
But this is just one lightly used system and may not be at all representative of a heavily used system.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
processes/CPU Usage/Mem Usage desktop wallpaper ceantuco Linux - Newbie 2 04-13-2009 01:14 PM
how to determine the tape usage for a streamer? alexandrusa Linux - Software 5 10-01-2008 11:45 AM
How to determine CPU usage Gayathri.P Programming 3 03-24-2008 11:20 PM
How do I determine disk space usage? econnections Linux - Newbie 4 08-28-2005 05:16 PM
how to determine cpu usage, memory usage, I/O usage by a particular user logged on li rags2k Programming 4 08-21-2004 04:45 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Server

All times are GMT -5. The time now is 07:21 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration