LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices

Reply
 
Search this Thread
Old 07-19-2011, 12:23 PM   #1
vikas027
Senior Member
 
Registered: May 2007
Location: Sydney
Distribution: RHEL, CentOS, Debian, OS X
Posts: 1,266

Rep: Reputation: 99
Question Identify which processes are consuming swap


How to identify which processes (or PIDs) are consuming SWAP ?

In my RHEL box SWAP is nearly 100 % utilized.

Code:
$ free -m
             total       used       free     shared    buffers     cached
Mem:        144967     143212       1754          0        166     135259
-/+ buffers/cache:       7787     137180
Swap:        22367      21733        634
I googled out, but could not find anything much useful.
 
Old 07-19-2011, 02:30 PM   #2
e79
LQ Newbie
 
Registered: Aug 2010
Location: Between the Heatsink and CPU
Distribution: Ubuntu 12.04 amd64 - Windows 7 x64
Posts: 5

Rep: Reputation: 0
Well considering you have about 144 GB of ram on this server, and over 95% is used, as well as the swap, I would do a 'top' command and sort it by 'Mem' and see which application is using all your ram; I bet this same app is using your swap file like there is no tomorrow.
 
Old 07-19-2011, 02:49 PM   #3
Snark1994
Senior Member
 
Registered: Sep 2010
Location: Wales, UK
Distribution: Arch
Posts: 1,630
Blog Entries: 3

Rep: Reputation: 345Reputation: 345Reputation: 345Reputation: 345
To see the actual swap, you can fire up "top", press "f" (to add a column) and then press "p" to activate the SWAP column. Press enter and it will take you back to the the normal view, and you can sort by swap usage
 
2 members found this post helpful.
Old 07-19-2011, 02:54 PM   #4
e79
LQ Newbie
 
Registered: Aug 2010
Location: Between the Heatsink and CPU
Distribution: Ubuntu 12.04 amd64 - Windows 7 x64
Posts: 5

Rep: Reputation: 0
Thanks Snark1994 for the tip, completely forgot about this one

Cheers
 
Old 07-19-2011, 03:42 PM   #5
tommylovell
Member
 
Registered: Nov 2005
Distribution: Fedora, Redhat
Posts: 372

Rep: Reputation: 101Reputation: 101
I think top will be a good indicator but it may not represent just swap space usage.

I suspect top gets it's swap info from /proc/../smaps.
Code:
#!/bin/bash
for i in `ls /proc`
do
        if [ -f /proc/$i/smaps ]
                then
                grep -H Swap /proc/$i/smaps | awk '( $2 != "0" ) { print }'
        fi
done
Looking at this is tedious to do manually. I think memory in smaps that is "r-x" (executable code) doesn't get swapped out to the swap space, just memory that is "rw-". I'm not certain how top handles something that is swapped out, but not swapped out to the swap space. I think top may total up everything so it may show more swapped out than what is in your swap space. I could be wrong, though.

And bear in mind that unless you run out of swap, pages written there have a tendency to linger even if the page has been swapped back in. My simplified understanding is that they are only cleared when the process ends, or you've filled swap and some other page needs to be swapped out. I'm not certain how a modified swapped in memory page is handled in respect to the page left out in swap. (Adding another swap space and swapping off the full one will show you what is truly needed. I wouldn't do this on a mission critical system, though.)

Last edited by tommylovell; 07-19-2011 at 03:44 PM.
 
Old 07-19-2011, 06:53 PM   #6
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 12,271

Rep: Reputation: 1028Reputation: 1028Reputation: 1028Reputation: 1028Reputation: 1028Reputation: 1028Reputation: 1028Reputation: 1028
Quote:
Originally Posted by tommylovell View Post
I'm not certain how top handles something that is swapped out, but not swapped out to the swap space.
Can't happen - since (at least) before the 2.4 kernel series was released. Only dirty (i.e. modified) memory get swapped - and then only if there is demand for memory.
The numbers for swap in smaps should be reliable - else it's a reportable kernel bug. "top" and your script should be o.k.
A while back I wondered if those numbers included swap-cache or not - never did chase that up. Could skew the numbers for (swap) cached pages that had never been physically swapped; especially with current kernels that also compress the swap-cache to reduce the need to physically swap at all. Must put it back on my to-do list ...
 
Old 07-22-2011, 07:53 AM   #7
vikas027
Senior Member
 
Registered: May 2007
Location: Sydney
Distribution: RHEL, CentOS, Debian, OS X
Posts: 1,266

Original Poster
Rep: Reputation: 99
Unhappy

Hi All,

Thanks for your inputs. I think this might be the bug but I am unable to find all Swap entries in /proc/*/smaps of 0 KB only.

Quote:
Swap: 0 kB
Also, I don't think top then shift+O and then p or top then f and then p shows how much swap is being used by which process (or PIDs).
See this.

For Example.
Code:
$ free -m
             total       used       free     shared    buffers     cached
Mem:         64459      63240       1219          0        282      56740
-/+ buffers/cache:       6217      58241
Swap:        16351       2753      13598
I have 2753 MB of swap used here, but top then f and then p shows 22 GB and 4.2 GB
Code:
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  SWAP COMMAND                                                                                                                     
 7165 oracle    25   0 31.9g 9.3g 9.3g R 100.2 14.8   2378:49  22g oracle                                                                                                                      
12972 oracle    18   0 4339m  24m  21m S  0.3  0.0  59:01.40 4.2g oracle                                                                                                                       
    1 root      15   0 10348  636  540 S  0.0  0.0   0:45.37 9712 init
Is there some way out that I can figure what Process (or PIDs) are consuming SWAP ?


Regards,
Vikas
 
Old 07-22-2011, 03:46 PM   #8
tommylovell
Member
 
Registered: Nov 2005
Distribution: Fedora, Redhat
Posts: 372

Rep: Reputation: 101Reputation: 101
Quote:
Is there some way out that I can figure what Process (or PIDs) are consuming SWAP ?
I have no explanation for top's output. This looks like a dive into the code is in order.

But I go back to my original statement (which may have not been too clear), the free command shows you how much swap has been used (by still active processes), not necessarily what is still swapped out.

And another point, the pages that have gone out to swap have no direct relationship to the process that may be causing the swapping to occur. The system has a terribly complicated "Page Frame Reclaiming Algorithm" that determines what pages will be stolen.

Also, from the size of your cache, your system is no longer stressed.

btw, the "Swap:" entry in /proc/<pid>/smaps is only available in more recent kernels. Sorry it didn't dawn on me to mention that.

I think the bottom line answer to your question is "no".

Last edited by tommylovell; 07-22-2011 at 04:11 PM.
 
Old 07-22-2011, 03:59 PM   #9
tommylovell
Member
 
Registered: Nov 2005
Distribution: Fedora, Redhat
Posts: 372

Rep: Reputation: 101Reputation: 101
Quote:
Originally Posted by syg00 View Post
Can't happen - since (at least) before the 2.4 kernel series was released. Only dirty (i.e. modified) memory get swapped - and then only if there is demand for memory.
Agreed.

I was using the term "swap" too loosely, as many do. I was referring to "Syncable" pages, which can be reclaimed by PFRA, but do not get written to the swap space. I'm not certain if these "page steals" are represented in any metric anywhere. And I'm really too old, and too lazy to look at code.
 
Old 07-23-2011, 12:37 AM   #10
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 12,271

Rep: Reputation: 1028Reputation: 1028Reputation: 1028Reputation: 1028Reputation: 1028Reputation: 1028Reputation: 1028Reputation: 1028
Quote:
Originally Posted by vikas027 View Post
I think this might be the bug but I am unable to find all Swap entries in /proc/*/smaps of 0 KB only.
Does this mean you have swap entries in smaps, but they are all zero ?. If so you have the (kernel) support, but no processes have any swap slots allocated - as tommylovell says above, the numbers you are seeing are probably residual in that case. Easy enough to prove, swapoff followed by swapon.
NOT that I'm recommending you do that on a prod box.
Quote:
I have 2753 MB of swap used here, but top then f and then p shows 22 GB and 4.2 GB
Code:
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  SWAP COMMAND                                                                                                                     
 7165 oracle    25   0 31.9g 9.3g 9.3g R 100.2 14.8   2378:49  22g oracle                                                                                                                      
12972 oracle    18   0 4339m  24m  21m S  0.3  0.0  59:01.40 4.2g oracle                                                                                                                       
    1 root      15   0 10348  636  540 S  0.0  0.0   0:45.37 9712 init
How big is your SGA ?.
I wonder if "top" is counting that - the manpage does say this (note the last sentence)
Code:
p: SWAP  --  Swapped size (kb)
          The non-resident portion of a task's address space.
 
Old 07-23-2011, 12:58 AM   #11
johnsfine
Guru
 
Registered: Dec 2007
Distribution: Centos
Posts: 5,083

Rep: Reputation: 1110Reputation: 1110Reputation: 1110Reputation: 1110Reputation: 1110Reputation: 1110Reputation: 1110Reputation: 1110Reputation: 1110
Quote:
Originally Posted by Snark1994 View Post
To see the actual swap, you can fire up "top", press "f" (to add a column) and then press "p" to activate the SWAP column. Press enter and it will take you back to the the normal view, and you can sort by swap usage
That suggestion is not useful. The SWAP column in top is pretty much useless and in any case does not represent use of swap space, nor anything with significant correlation with use of swap space.

I would expect the information in /proc/*/smaps to include complete and accurate detail on the actual use of swap space. I don't know why the OP's results from /proc/*/smaps don't seem to identify the swap use.
 
Old 07-23-2011, 01:20 AM   #12
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 12,271

Rep: Reputation: 1028Reputation: 1028Reputation: 1028Reputation: 1028Reputation: 1028Reputation: 1028Reputation: 1028Reputation: 1028
Quote:
Originally Posted by johnsfine View Post
The SWAP column in top is pretty much useless and in any case does not represent use of swap space, nor anything with significant correlation with use of swap space.
Good point - we've been here before haven't we ? ....
Just ran strace over "top" - no indication of reference to smaps, so it (top) is still doing incantations with various fields in /proc/<pid>/stat and statm to work out it's idea of swap usage.
 
Old 09-21-2011, 09:51 AM   #13
markseger
Member
 
Registered: Jul 2003
Posts: 241

Rep: Reputation: 25
interesting enough this came up just the other day with some colleagues and someone pointed out there is a patch to top to fix this. I believe it is now looks at /proc/*/status which on newer kernels has added a VmSwap field. Check out this:

VmPeak: 100840 kB
VmSize: 100840 kB
VmLck: 0 kB
VmHWM: 504 kB
VmRSS: 504 kB
VmData: 180 kB
VmStk: 88 kB
VmExe: 44 kB
VmLib: 1612 kB
VmPTE: 44 kB
VmSwap: 0 kB
Threads: 1

can't say for sure which was the first version to support it but the info above was from a 2.6.32 kernel. I'm trying to figure out if I should add it to collectl process memory display - I'm thinking I should

if you're not aware of it, you add the switch --procopt m

# PROCESS SUMMARY (counters are /sec)
# PID User S VmSize VmLck VmRSS VmData VmStk VmExe VmLib MajF MinF Command
1 root S 19244K 0 1280K 328K 88K 128K 2256K 0 0 /sbin/init

it'd be pretty easy to slide in anywhere...

-mark
 
1 members found this post helpful.
Old 09-21-2011, 10:08 AM   #14
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 12,271

Rep: Reputation: 1028Reputation: 1028Reputation: 1028Reputation: 1028Reputation: 1028Reputation: 1028Reputation: 1028Reputation: 1028
Thanks Mark, just happened to see this post before I went to bed.
The (kernel) patch appears to maintain swap_pte counters. Wonder how swap cache compression affects this. This is to remind me to go look sometime.

Last edited by syg00; 09-21-2011 at 10:10 AM.
 
Old 09-21-2011, 11:48 AM   #15
johnsfine
Guru
 
Registered: Dec 2007
Distribution: Centos
Posts: 5,083

Rep: Reputation: 1110Reputation: 1110Reputation: 1110Reputation: 1110Reputation: 1110Reputation: 1110Reputation: 1110Reputation: 1110Reputation: 1110
Thanks Mark. I've given that answer (SWAP column in top is useless) in several threads. I appreciate the warning that I need to be more careful with that answer in the future as it might be changing.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
How to determine processes swap usage kenneho Linux - Server 7 01-12-2011 10:07 AM
[SOLVED] what kernel processes are consuming cpu resources bino_linux Linux - Kernel 2 09-21-2010 08:07 AM
Warning in Top 10 cpu consuming processes pinga123 Linux - Newbie 13 08-11-2010 07:19 AM
How to identify processes running in user mode and kernel mode subha109 Linux - Kernel 1 10-21-2009 11:37 AM
What command can identify processes running on ports? hello321_1999 Linux - Newbie 2 11-23-2004 05:01 AM


All times are GMT -5. The time now is 10:49 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration