LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices

Reply
 
Search this Thread
Old 03-15-2010, 04:49 PM   #1
Subject16
Member
 
Registered: Nov 2009
Posts: 46

Rep: Reputation: 1
Would Like Help Interpreting Output of Memory Tools in RHEL 5.3


Hi. I have been doing a lot of research on the web to piece together the components I need to understand what is probably a very simple thing. I've read up on buffers/cache vs. physical memory and using free -m and top and understanding that output, and reviewed Red Hat's article on virtual memory. I am hoping to get some assistance in putting it all together to understand what's happening in my situation because so far I have a lot of dangling factoids and no glue. I have also referenced the thread http://www.linuxquestions.org/questi...memory-309767/ but I didn't find my answer there.

My company is running a cluster of physical web servers (they are clones so running the same OS, with same RAM and applications installed). I'll be referring to a single system although the question applies to all because they're all showing similar output. The Multi Router Traffic Grapher is showing a steady swap usage for 3-4 months now. The average usage for today is 81%. There is, as far as I know, no performance hit because of it, but a developer noticed this and wanted to know what was going on, and I'd using this opp to learn.

The system is Linux version 2.6.18-128.7.1.el5, running JBOSS, postgres, Apache, and Java. Here is the output from the memory commands I have used to try and understand what's going on.

Code:
top - 21:19:27 up 159 days,  7:43,  1 user,  load average: 0.52, 0.61, 0.64
Tasks: 144 total,   1 running, 143 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.2%us,  8.0%sy,  0.0%ni, 91.9%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   4116496k total,  4090916k used,    25580k free,    12296k buffers
Swap:  2040244k total,  1538656k used,   501588k free,  1455720k cached

  PID    USER      PR  NI  VIRT     RES    SHR    S %CPU %MEM TIME+   SWAP COMMAND                                                                             
13768 root      15   0 1572m   193m 5168  S  0.0   4.8     8:01.48  1.3g  shibd                                                                               
24950 admin   20   0 1369m  39m    3604  S  0.0   1.0   55:32.86  1.3g  java                                                                                
 7651  jboss     21   0 2670m  1.6g    4200  S  0.0 40.7   1135:01   1.0g java                                                                                
 7627 postgres  15  0  520m  812     488     S  0.0  0.0   0:00.15    520m postmaster                                                                          
 7626 postgres  15   0  520m  624  516       S  0.0  0.0   0:00.15    520m postmaster                                                                          
21342 postgres  15   0  522m 6560 5148    S  0.0  0.2   0:00.51    515m postmaster                                                                          
20654 postgres  15   0  522m 7700 6220    S  0.0  0.2   0:00.16    514m postmaster                                                                          
 7575 postgres  15   0  520m 9.9m 9964     S  0.0  0.2   0:17.91     510m postmaster                                                                          
 7625 postgres  15   0  520m  29m  29m     S  0.0  0.7   0:01.58     491m postmaster                                                                          
10809 root      15   0  859m 491m  11m      S  0.0 12.2   3:23.11    367m java      

[root@server ~]# free -m
             total       used       free     shared    buffers     cached
Mem:          4020       3996         23          0         13       1387
-/+ buffers/cache:       2594       1425
Swap:         1992       1502        489

[root@server ~]# vmstat 60 5
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  1 1538668  30028  14248 1425672    1    1   300    49    2    1  6  2 88  5  0
 0  1 1538644  24020  14024 1419504   14    0   356    49 1100 2976  8  2 83  7  0
 3  0 1538644  25160  13488 1413580    0    0   791    67 1157 3068 10  2 76 12  0
 3  0 1538648  27244  12336 1412268    0    0  1400    77 1178 3014 16  4 70 10  0
 0  0 1538648  32032  12568 1403604    0    0   643   274 1161 3057 20  3 69  8  0

[root@server ~]# swapon -s
Filename                                Type            Size    Used    Priority
/dev/sda3                               partition       2040244 1538644 -1

[root@server ~]# cat /proc/meminfo 
MemTotal:      4116496 kB
MemFree:         56228 kB
Buffers:         10228 kB
Cached:        1365060 kB
SwapCached:     111536 kB
Active:        3596444 kB
Inactive:       364788 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:      4116496 kB
LowFree:         56228 kB
SwapTotal:     2040244 kB
SwapFree:       501600 kB
Dirty:             764 kB
Writeback:           0 kB
AnonPages:     2583684 kB
Mapped:         447400 kB
Slab:            46756 kB
PageTables:      29444 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:   4098492 kB
Committed_AS:  5747804 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    263516 kB
VmallocChunk: 34359474703 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

[root@server ~]# cat /proc/sys/vm/swappiness 
60
So my interpretation of this output tells me that my system has 4GB of RAM installed, 23MB of free "physical" RAM, and 1.35GB of RAM in cache (according to the second line of free -m, which represents free physical memory and useable mapped memory if I understand correctly). I have a 2GB swap partition, 1.5GB of which is currently being used, and according to the MRTG is the average that has been in use since January. So the swap size isn't decreasing. Java (JBOSS) is using the most physical RAM at 1.6GB.

My question is: if I have 1.35GB of RAM cached, why is my system using all of that swap space and not using any of the cached RAM? I thought I understood that Linux is supposed to either re-use unmodified pages in cache or send older pages to disk when more RAM was needed. It seems like my system is always sending pages to disk, which I guess would normally mean I just need more RAM but, to have close to the same swap size consistently seems like there must be more to it.

I've done the research and read the articles. My conclusion is I need more RAM but 4GB seems like it should be enough so I wanted a second (third, fourth, etc. ) opinion.

Thanks in advance.

Last edited by Tinkster; 03-16-2010 at 11:17 AM. Reason: replaced [ quote ] with [ code ] tags to aid readability
 
Old 03-15-2010, 06:12 PM   #2
ongte
Member
 
Registered: Jun 2009
Location: Penang, Malaysia
Distribution: Mandriva, CentOS, Ubuntu
Posts: 456

Rep: Reputation: 69
If you are running large applications with big chucks of data in RAM, depending on how often you need that data, the kernel may decide to swap it out & use the RAM for cache instead. This is the sensible thing to do & will probably give better performance.

Last edited by ongte; 03-15-2010 at 06:13 PM.
 
Old 03-15-2010, 06:49 PM   #3
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 12,314

Rep: Reputation: 1032Reputation: 1032Reputation: 1032Reputation: 1032Reputation: 1032Reputation: 1032Reputation: 1032Reputation: 1032
In addition to that, swap is only a problem if it's "active". To know that you need to check the rate at which swaps (in and out) are occurring. Keep an eye on the numbers in vmstat is probably easiest.
If they remain low (close to zero), you've reached a "steady state", and more RAM won't benefit you markedly - except for peace of mind when the swap usage reduces to something more "acceptable".
 
Old 03-17-2010, 08:17 AM   #4
Subject16
Member
 
Registered: Nov 2009
Posts: 46

Original Poster
Rep: Reputation: 1
Great. Sounds like I need to pay attention to vmstat and so long as there is not a lot of actual paging activity in the form of pages in and out, I should be okay. Thanks for your time and input!
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
I have a memory leak in C. Need help interpreting valgrind's output. adz Programming 13 09-05-2008 06:44 AM
Interpreting dmesg output pwabrahams Linux - Software 2 09-02-2008 01:27 PM
Interpreting output of df -h command kushalkoolwal Debian 1 06-30-2007 07:29 AM
Help interpreting Strace output rajesh_b_2k Linux - Kernel 0 12-18-2006 10:14 PM
Interpreting the output of 'lspci' command. drminix Linux - Hardware 2 06-14-2005 03:20 AM


All times are GMT -5. The time now is 11:57 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration