LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Linux - Newbie (http://www.linuxquestions.org/questions/linux-newbie-8/)
-   -   Would Like Help Interpreting Output of Memory Tools in RHEL 5.3 (http://www.linuxquestions.org/questions/linux-newbie-8/would-like-help-interpreting-output-of-memory-tools-in-rhel-5-3-a-795608/)

Subject16 03-15-2010 04:49 PM

Would Like Help Interpreting Output of Memory Tools in RHEL 5.3
 
Hi. I have been doing a lot of research on the web to piece together the components I need to understand what is probably a very simple thing. I've read up on buffers/cache vs. physical memory and using free -m and top and understanding that output, and reviewed Red Hat's article on virtual memory. I am hoping to get some assistance in putting it all together to understand what's happening in my situation because so far I have a lot of dangling factoids and no glue. I have also referenced the thread http://www.linuxquestions.org/questi...memory-309767/ but I didn't find my answer there.

My company is running a cluster of physical web servers (they are clones so running the same OS, with same RAM and applications installed). I'll be referring to a single system although the question applies to all because they're all showing similar output. The Multi Router Traffic Grapher is showing a steady swap usage for 3-4 months now. The average usage for today is 81%. There is, as far as I know, no performance hit because of it, but a developer noticed this and wanted to know what was going on, and I'd using this opp to learn.

The system is Linux version 2.6.18-128.7.1.el5, running JBOSS, postgres, Apache, and Java. Here is the output from the memory commands I have used to try and understand what's going on.

Code:

top - 21:19:27 up 159 days,  7:43,  1 user,  load average: 0.52, 0.61, 0.64
Tasks: 144 total,  1 running, 143 sleeping,  0 stopped,  0 zombie
Cpu(s):  0.2%us,  8.0%sy,  0.0%ni, 91.9%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  4116496k total,  4090916k used,    25580k free,    12296k buffers
Swap:  2040244k total,  1538656k used,  501588k free,  1455720k cached

  PID    USER      PR  NI  VIRT    RES    SHR    S %CPU %MEM TIME+  SWAP COMMAND                                                                           
13768 root      15  0 1572m  193m 5168  S  0.0  4.8    8:01.48  1.3g  shibd                                                                             
24950 admin  20  0 1369m  39m    3604  S  0.0  1.0  55:32.86  1.3g  java                                                                               
 7651  jboss    21  0 2670m  1.6g    4200  S  0.0 40.7  1135:01  1.0g java                                                                               
 7627 postgres  15  0  520m  812    488    S  0.0  0.0  0:00.15    520m postmaster                                                                         
 7626 postgres  15  0  520m  624  516      S  0.0  0.0  0:00.15    520m postmaster                                                                         
21342 postgres  15  0  522m 6560 5148    S  0.0  0.2  0:00.51    515m postmaster                                                                         
20654 postgres  15  0  522m 7700 6220    S  0.0  0.2  0:00.16    514m postmaster                                                                         
 7575 postgres  15  0  520m 9.9m 9964    S  0.0  0.2  0:17.91    510m postmaster                                                                         
 7625 postgres  15  0  520m  29m  29m    S  0.0  0.7  0:01.58    491m postmaster                                                                         
10809 root      15  0  859m 491m  11m      S  0.0 12.2  3:23.11    367m java     

[root@server ~]# free -m
            total      used      free    shared    buffers    cached
Mem:          4020      3996        23          0        13      1387
-/+ buffers/cache:      2594      1425
Swap:        1992      1502        489

[root@server ~]# vmstat 60 5
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b  swpd  free  buff  cache  si  so    bi    bo  in  cs us sy id wa st
 0  1 1538668  30028  14248 1425672    1    1  300    49    2    1  6  2 88  5  0
 0  1 1538644  24020  14024 1419504  14    0  356    49 1100 2976  8  2 83  7  0
 3  0 1538644  25160  13488 1413580    0    0  791    67 1157 3068 10  2 76 12  0
 3  0 1538648  27244  12336 1412268    0    0  1400    77 1178 3014 16  4 70 10  0
 0  0 1538648  32032  12568 1403604    0    0  643  274 1161 3057 20  3 69  8  0

[root@server ~]# swapon -s
Filename                                Type            Size    Used    Priority
/dev/sda3                              partition      2040244 1538644 -1

[root@server ~]# cat /proc/meminfo
MemTotal:      4116496 kB
MemFree:        56228 kB
Buffers:        10228 kB
Cached:        1365060 kB
SwapCached:    111536 kB
Active:        3596444 kB
Inactive:      364788 kB
HighTotal:          0 kB
HighFree:            0 kB
LowTotal:      4116496 kB
LowFree:        56228 kB
SwapTotal:    2040244 kB
SwapFree:      501600 kB
Dirty:            764 kB
Writeback:          0 kB
AnonPages:    2583684 kB
Mapped:        447400 kB
Slab:            46756 kB
PageTables:      29444 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  4098492 kB
Committed_AS:  5747804 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    263516 kB
VmallocChunk: 34359474703 kB
HugePages_Total:    0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:    2048 kB

[root@server ~]# cat /proc/sys/vm/swappiness
60

So my interpretation of this output tells me that my system has 4GB of RAM installed, 23MB of free "physical" RAM, and 1.35GB of RAM in cache (according to the second line of free -m, which represents free physical memory and useable mapped memory if I understand correctly). I have a 2GB swap partition, 1.5GB of which is currently being used, and according to the MRTG is the average that has been in use since January. So the swap size isn't decreasing. Java (JBOSS) is using the most physical RAM at 1.6GB.

My question is: if I have 1.35GB of RAM cached, why is my system using all of that swap space and not using any of the cached RAM? I thought I understood that Linux is supposed to either re-use unmodified pages in cache or send older pages to disk when more RAM was needed. It seems like my system is always sending pages to disk, which I guess would normally mean I just need more RAM but, to have close to the same swap size consistently seems like there must be more to it.

I've done the research and read the articles. My conclusion is I need more RAM but 4GB seems like it should be enough so I wanted a second (third, fourth, etc. ) opinion.

Thanks in advance.

ongte 03-15-2010 06:12 PM

If you are running large applications with big chucks of data in RAM, depending on how often you need that data, the kernel may decide to swap it out & use the RAM for cache instead. This is the sensible thing to do & will probably give better performance.

syg00 03-15-2010 06:49 PM

In addition to that, swap is only a problem if it's "active". To know that you need to check the rate at which swaps (in and out) are occurring. Keep an eye on the numbers in vmstat is probably easiest.
If they remain low (close to zero), you've reached a "steady state", and more RAM won't benefit you markedly - except for peace of mind when the swap usage reduces to something more "acceptable".

Subject16 03-17-2010 08:17 AM

Great. Sounds like I need to pay attention to vmstat and so long as there is not a lot of actual paging activity in the form of pages in and out, I should be okay. Thanks for your time and input!


All times are GMT -5. The time now is 10:51 PM.