LinuxQuestions.org
Review your favorite Linux distribution.
Go Back   LinuxQuestions.org > Forums > Enterprise Linux Forums > Linux - Enterprise
User Name
Password
Linux - Enterprise This forum is for all items relating to using Linux in the Enterprise.

Notices


Reply
  Search this Thread
Old 09-15-2016, 06:36 AM   #1
Buddhike G
Member
 
Registered: Jun 2007
Distribution: RHEL, CentOS, Solaris
Posts: 63

Rep: Reputation: 16
Can't list the highest Memory utilized processes in the "top" command output


Hi All,

I am having a RHEL 6.5 server with 32 G RAM and currently it is showing around 97% of memory utilization. I've checked the output of "free -m" and as you can see below, the memory is mostly utilized for applications only (not for buffers and cache)

Code:
[root@Server1 ~]# free -m
             total       used       free     shared    buffers     cached
Mem:         32113      30589       1523          0        602       1407
-/+ buffers/cache:      28578       3534
Swap:        32767          0      32767
[root@Server1 ~]#
when I run the "top" command with the output sorted according to the highest memory usage, the sum of the utilization percentage is way less than 50%.

Code:
top - 19:24:46 up 62 days,  7:17,  4 users,  load average: 0.16, 0.61, 0.64
Tasks: 454 total,   1 running, 453 sleeping,   0 stopped,   0 zombie
Cpu(s):  1.7%us,  0.3%sy,  0.0%ni, 98.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  32883788k total, 32090448k used,   793340k free,   600868k buffers
Swap: 33554424k total,        0k used, 33554424k free,  2290244k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                
14623 appadmin  20   0 16.9g 9.3g 7876 S 10.9 29.6  22019:03 java                                                                   
 8004 appadmin  20   0 2843m 2.2g 5100 S  0.3  7.0 851:44.57 java                                                                   
28604 root      20   0  804m 673m 9.9m S  0.7  2.1 401:11.59 java                                                                   
14839 appadmin  20   0 1343m 563m 5536 S  1.3  1.8   1093:32 java                                                                   
 9425 root      20   0  788m 559m 9820 S  0.0  1.7 296:37.00 java                                                                   
 3649 oracle    20   0  396m 292m 4288 S  0.0  0.9 240:02.63 java                                                                   
16728 oracle    20   0 10.4g  73m  19m S  0.0  0.2   2:17.31 ora_arc0_dbpemdb                                                        
 2378 root      20   0  148m  72m  18m S  0.3  0.2  42:46.95 Xvnc                                                                   
16732 oracle    20   0 10.4g  59m  17m S  0.0  0.2   2:54.23 ora_arc2_dbpemdb                                                        
16702 oracle    20   0 10.4g  53m  38m S  0.0  0.2  55:25.86 ora_mmon_dbpemdb                                                       
 2578 root      20   0  538m  50m 4820 S  0.0  0.2   4:49.24 gnome-settings-                                                        
 2974 oracle    20   0  538m  50m 4572 S  0.0  0.2   4:42.63 gnome-settings-                                                        
19877 oracle    20   0 10.4g  42m  19m S  0.0  0.1  41:45.85 ora_tt00_dbpemdb                                                        
20126 oracle    20   0 10.4g  38m  19m S  0.0  0.1   1:17.95 ora_arc1_dbpemdb                                                        
 4496 oracle    20   0 10.4g  38m  31m S  0.0  0.1   0:00.67 oracle_4496_dbpem                                                        
 4554 oracle    20   0 10.4g  37m  30m S  0.0  0.1   0:00.64 oracle_4554_dbpem                                                        
 4520 oracle    20   0 10.4g  37m  30m S  0.0  0.1   0:00.95 oracle_4520_dbpem                                                        
 4528 oracle    20   0 10.4g  37m  30m S  0.0  0.1   0:00.69 oracle_4528_dbpem                                                        
17132 oracle    20   0 10.4g  37m  31m S  0.0  0.1  23:18.77 ora_cjq0_dbpemdb1                                                        
 4490 oracle    20   0 10.4g  36m  30m S  0.0  0.1   0:00.84 oracle_4490_dbpem                                                        
 4530 oracle    20   0 10.4g  36m  30m S  0.0  0.1   0:00.47 oracle_4530_dbpem                                                        
 5750 oracle    20   0 10.4g  36m  29m S  0.0  0.1   0:00.37 oracle_5750_dbpem                                                        
 4532 oracle    20   0 10.4g  36m  29m S  0.0  0.1   0:02.51 oracle_4532_dbpem                                                        
 4482 oracle    20   0 10.4g  36m  31m S  0.0  0.1   0:01.02 oracle_4482_dbpem                                                        
 4484 oracle    20   0 10.4g  36m  30m S  0.0  0.1   0:00.92 oracle_4484_dbpem                                                        
 4522 oracle    20   0 10.4g  35m  28m S  0.0  0.1   0:00.39 oracle_4522_dbpem                                                        
 4536 oracle    20   0 10.4g  35m  28m S  0.0  0.1   0:00.50 oracle_4536_dbpem                                                        
 4542 oracle    20   0 10.4g  34m  28m S  0.0  0.1   0:00.77 oracle_4542_dbpem                                                        
18386 oracle    20   0 10.4g  33m  28m S  0.0  0.1  59:26.67 oracle_18386_sa                                                        
 6422 oracle    20   0 10.4g  33m  27m S  0.0  0.1   0:00.83 oracle_6422_dbpem                                                        
20926 oracle    20   0 10.4g  32m  28m S  0.0  0.1   0:11.71 oracle_20926_sa                                                        
12064 oracle    20   0 10.4g  32m  27m S  0.0  0.1   0:04.27 oracle_12064_sa                                                        
17504 oracle    20   0 10.4g  32m  28m S  0.0  0.1   0:23.70 oracle_17504_sa                                                        
17311 oracle    20   0 10.4g  31m  26m S  0.0  0.1   0:57.71 ora_q002_dbpemdb
Just wanted to check whether I am missing anything in my checking ?

can any of you give a clue to look ?

Thank You in advance,

Buddhike G

Last edited by Buddhike G; 09-15-2016 at 11:34 AM.
 
Old 09-15-2016, 07:12 AM   #2
pan64
LQ Addict
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 21,692

Rep: Reputation: 7275Reputation: 7275Reputation: 7275Reputation: 7275Reputation: 7275Reputation: 7275Reputation: 7275Reputation: 7275Reputation: 7275Reputation: 7275Reputation: 7275
did you read it already: www.linuxatemyram.com ?
 
1 members found this post helpful.
Old 09-15-2016, 07:35 AM   #3
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 21,103

Rep: Reputation: 4117Reputation: 4117Reputation: 4117Reputation: 4117Reputation: 4117Reputation: 4117Reputation: 4117Reputation: 4117Reputation: 4117Reputation: 4117Reputation: 4117
Edit your post and change the [quote] tags to [code] so we can read the output(s).
Then post /proc/meminfo.
 
1 members found this post helpful.
Old 09-15-2016, 11:41 AM   #4
Buddhike G
Member
 
Registered: Jun 2007
Distribution: RHEL, CentOS, Solaris
Posts: 63

Original Poster
Rep: Reputation: 16
Hello pan64 and syg00

Thank you very much for the feedback,
Yes I've gone through the www.linuxatemyram.com before. But in my case Bufferd & Cached utilization is about 2GB. And the application utilization is about 28 GB. As you can see from the "top" command output, addition of the total application utilization is less than 50% of total memory. My question is what application uses the rest of the memory.

Following is the /proc/meminfo output

Code:
[root@Server1 ~]# cat /proc/meminfo 
MemTotal:       32883788 kB
MemFree:         1568436 kB
Buffers:          617484 kB
Cached:          1448392 kB
SwapCached:            0 kB
Active:         14899156 kB
Inactive:        2431756 kB
Active(anon):   13582192 kB
Inactive(anon):  1730648 kB
Active(file):    1316964 kB
Inactive(file):   701108 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:      33554424 kB
SwapFree:       33554424 kB
Dirty:               160 kB
Writeback:             0 kB
AnonPages:      15285680 kB
Mapped:           109660 kB
Shmem:             27276 kB
Slab:             396292 kB
SReclaimable:     268252 kB
SUnreclaim:       128040 kB
KernelStack:       20680 kB
PageTables:       144920 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    43420188 kB
Committed_AS:   20363088 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      606380 kB
VmallocChunk:   34359058980 kB
HardwareCorrupted:     0 kB
AnonHugePages:  12939264 kB
HugePages_Total:    6422
HugePages_Free:     1290
HugePages_Rsvd:        5
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:       12288 kB
DirectMap2M:     2011136 kB
DirectMap1G:    31457280 kB
[root@Server1 ~]#
Please let me know if you need any more information.

Thank you very much again for the help

Buddhike G
 
Old 09-15-2016, 03:22 PM   #5
jpollard
Senior Member
 
Registered: Dec 2012
Location: Washington DC area
Distribution: Fedora, CentOS, Slackware
Posts: 4,912

Rep: Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513
Did you notice that the %MEM field is computed using the RES field - (RES/total)*100.

The RES field is what the process is currently actively using - resident in memory. What is allocated to the process is VIRT, which is much larger, but is either paged out, or in memory but not yet in use.

The other problem is that top only shows you snapshots. What was actually being used can be larger or smaller.

Next, you may not realize that a lot of that memory is shared - those bpem processes look like they are all threads - thus all the memory they are using is likely the same memory, yet reported separately...

Neither top nor ps are good at reporting memory - they are both snapshots, and poor snapshots at that. The problem is that between the time a process stats are grabbed, other processes will be using memory - and when that other process stats are taken, they may be using what was used by the first process... Getting coherent stats is not what they do.

You are more likely to get something close by using "vmstat" instead - and give it an interval to report over time. The first report from vmstat is also a snapshot...

You might also look at reports from the "sysstat" tools, as the kernel collects the data they use over the scheduling time intervals, and should be MUCH more accurate.
 
1 members found this post helpful.
Old 09-15-2016, 05:11 PM   #6
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 21,103

Rep: Reputation: 4117Reputation: 4117Reputation: 4117Reputation: 4117Reputation: 4117Reputation: 4117Reputation: 4117Reputation: 4117Reputation: 4117Reputation: 4117Reputation: 4117
AnonHugePages: 12939264 kB
Hugepages are fixed in memory, and lessen the available by that amount. You could probably blame Oracle in this case.
 
1 members found this post helpful.
Old 09-17-2016, 02:31 PM   #7
Buddhike G
Member
 
Registered: Jun 2007
Distribution: RHEL, CentOS, Solaris
Posts: 63

Original Poster
Rep: Reputation: 16
Hi jpollard, syg00

Thank you very much for your responses,

Will this consumed AnonHugePages (12939264 kB) can be used by any other application, if the system runs out of available memory (MemFree + Buffers + Cached) ?

Will it be possible to disable AnonHugePages usage, if yes, is it advisable to do so ?

Thank You & Regards,
Buddhike G
 
Old 09-17-2016, 05:42 PM   #8
jpollard
Senior Member
 
Registered: Dec 2012
Location: Washington DC area
Distribution: Fedora, CentOS, Slackware
Posts: 4,912

Rep: Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513
It all depends on what is using the pages - very likely it will be Oracle. The advantage for Oracle is speed, both for large I/O requests and for in memory usage.
 
1 members found this post helpful.
Old 09-17-2016, 07:32 PM   #9
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 21,103

Rep: Reputation: 4117Reputation: 4117Reputation: 4117Reputation: 4117Reputation: 4117Reputation: 4117Reputation: 4117Reputation: 4117Reputation: 4117Reputation: 4117Reputation: 4117
The system is a better memory manager than any of us could ever hope to be.
In need tune the users - plenty of sources online re how to manage transparent huge pages.
 
1 members found this post helpful.
Old 09-17-2016, 10:40 PM   #10
Buddhike G
Member
 
Registered: Jun 2007
Distribution: RHEL, CentOS, Solaris
Posts: 63

Original Poster
Rep: Reputation: 16
Hi Experts,

Thank you very much for your fast response on my query. I will close this thread as solved.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
top command output in "human readable" custangro Linux - General 7 12-01-2017 11:53 AM
How could a computer technician use the "top" command with "ps" and "kill" to investigate how a system is misbehaving? geckono1 Linux - Newbie 13 07-03-2016 07:51 AM
Finding the top CPU-using processes? "top" command does not help... catalytical Linux - Server 1 04-06-2009 04:50 PM
Feeding the output of "diff" or "cat" command to dpkg --purge kushalkoolwal Debian 9 06-19-2008 07:27 AM
"top" command memory question chanys Linux - General 4 03-14-2003 08:01 PM

LinuxQuestions.org > Forums > Enterprise Linux Forums > Linux - Enterprise

All times are GMT -5. The time now is 07:08 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration