How to get the memory taken by each process?

liuwenbin168@126.com · 09-14-2012, 02:40 AM

Hi gurus,

I am new to Linux, and here is a dumb question:
How can I see the memory taken by each process?

Why this question:
I have a server (CentOS 5.8), its total memory is 32 GB, and now the used memory is more than 31 GB, so I want to get which process takes most memory.
Meantime, if I can see all of the memory taken for every process, that would be best!

Till to now, what I did:
1. free -m

Code:

[root@ID5xinyong ~]# free -m
  total used free shared buffers cached
Mem: 32108 31579 528 0 6 43
-/+ buffers/cache: 31529 579
Swap: 34287 114 34173

As the output says, more than 31 GB memory has been used.

2.

Code:

top -c, then press: M
-----------------------------------------------------------------------------------------
  top - 15:03:46 up 30 days, 21 min, 2 users, load average: 0.00, 0.00, 0.00
Tasks: 351 total, 1 running, 350 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 32878968k total, 32339300k used, 539668k free, 7304k buffers
Swap: 35110904k total, 117036k used, 34993868k free, 44916k cached

  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
 3186 mqm 20 0 1498m 10m 10m S 0.0 0.0 2:19.81 /opt/mqm/bin/amqzmuc0 -m QM1
19357 mqm 20 0 1498m 10m 9.9m S 0.0 0.0 1:54.99 /opt/mqm/bin/amqzmuc0 -m QM2
 3234 mqm 20 0 935m 4284 3772 S 0.0 0.0 10:41.55 /opt/mqm/bin/amqzlaa0 -mQM1 -fip0
29364 root 20 0 97756 3888 2940 S 0.0 0.0 0:00.19 sshd: root@pts/1
29276 root 20 0 97756 3872 2940 S 0.0 0.0 0:00.14 sshd: root@pts/0
...
-----------------------------------------------------------------------------------------

As the top command output, the %MEM column has been sorted descending, but, I still cannot see the BIG process who takes much memory.

So, the dumb question is how to get the process who takes lots of memory.

Thanks in advance!

Celyr · 09-14-2012, 02:50 AM

Quote:

Originally Posted by liuwenbin168@126.com

Code:

[root@ID5xinyong ~]# free -m
  total used free shared buffers cached
Mem: 32108 31579 528 0 6 43
-/+ buffers/cache: 31529 579
Swap: 34287 114 34173

You are not seeing any process taking all that ram because actually there isn't one. You ram is just being cached by the linux kernel but is going to be freed if needed.
You are using very low ram on that server.

For example this is one of my boxes:

Code:

root@minnie:~# free -m
             total       used       free     shared    buffers     cached
Mem:          3943       3691        252          0        169       3292
-/+ buffers/cache:        228       3714
Swap:            0          0          0
root@minnie:~# ps aux | sort -b -k 4 | tail
root      2055  0.0  0.0  53308  3236 ?        Ss   Aug25   0:04 /usr/sbin/cupsd -C /etc/cups/cupsd.conf
root      2078  0.0  0.0  73404  3632 ?        Ss   Aug25   0:05 /usr/sbin/smbd -D
root      2081  0.0  0.0  73532  1940 ?        S    Aug25   0:00 /usr/sbin/smbd -D
root      6157  0.0  0.0 101256   980 ?        Ssl  Sep09   0:55 /sbin/apcupsd
data       615  0.0  0.0 152260  3436 ?        Ssl  Sep13   0:01 xxx
root      2026  0.0  0.0 184420  2432 ?        Sl   Aug25   0:00 /usr/libexec/polkitd --no-debug
root      1959  0.0  0.0 2081116 2620 ?        Ssl  Aug25   0:00 /usr/sbin/console-kit-daemon
root      2086  0.0  0.1  13556  6764 ?        Ss   Aug25   0:05 /usr/sbin/dhcpd
data      2106  0.0  0.5 263756 22476 ?        Ssl  Aug25  24:36 xxx
root      5268  0.0  0.9 372712 39396 ?        Ssl  Aug26   2:12 /usr/sbin/named
root@minnie:~#

As you see it may appear that there is only 200MB of free ram, this is true but there are at 3,2GB that can be released by the kernel if needed.

However there is something that i can't understand about your free output, can you post a

Code:

# cat /proc/meminfo

and

Code:

vmstat

It looks like your server is doing big I/O is this true ?

liuwenbin168@126.com · 09-14-2012, 03:00 AM

Quote:

Originally Posted by Celyr

You are not seeing any process taking all that ram because actually there isn't one. You ram is just being cached by the linux kernel but is going to be freed if needed.
You are using very low ram on that server.

Hi Celyr,
Thanks A Lot for your quick reply!

You mean the linux kernel cached lots of RAM? And how can I know how much RAM it cached?

Let's say the output of "free -m", the first line:

Code:

--------------------------------------------
     total used    free  shared  buffers cached
Mem: 32108 31579   528   0       6        43
--------------------------------------------

Does not it mean that only 43 mb are being cached (by kernel)? If not, how to understand the value for "cached 43"?

Thank you very much!

syg00 · 09-14-2012, 03:05 AM

@liuwenbin168@126.com, in future use [code] tags as @Celyr has - it retains data formatting and makes it easier (and more likely) others will try to help.

As for your problem, RAM gets used for all sorts of things by the kernel and products (like Oracle e.g.).
Lets see the output from

Code:

cat /proc/meminfo

. Use code tags or else you may get ignored.

liuwenbin168@126.com · 09-14-2012, 03:09 AM

Hi Celyr,

Outputs of the commands you adviced:

Code:

----------------------------------------------------
[root@ID5xinyong ~]# cat /proc/meminfo
MemTotal:       32878968 kB
MemFree:          537932 kB
Buffers:            7976 kB
Cached:            45012 kB
SwapCached:         5660 kB
Active:            33416 kB
Inactive:          33304 kB
Active(anon):      19464 kB
Inactive(anon):    16900 kB
Active(file):      13952 kB
Inactive(file):    16404 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:      35110904 kB
SwapFree:       34993868 kB
Dirty:                16 kB
Writeback:             0 kB
AnonPages:         12352 kB
Mapped:            32256 kB
Shmem:             22596 kB
Slab:              80952 kB
SReclaimable:      17164 kB
SUnreclaim:        63788 kB
KernelStack:        4296 kB
PageTables:        10052 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    47978676 kB
Committed_AS:     724816 kB
VmallocTotal:   34359738367 kB
VmallocUsed:     4536520 kB
VmallocChunk:   34329520624 kB
HardwareCorrupted:     0 kB
AnonHugePages:         0 kB
HugePages_Total:    3488
HugePages_Free:     3488
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:       10240 kB
DirectMap2M:    33544192 kB
----------------------------------------------------

Code:

[root@ID5xinyong ~]# vmstat
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0 117036 537856   7992  45012  243   61   253    67    0    0  3  1 87  9  0

----------------------------------------------------

Yes, it looks like my server is doing I/O frequently, but sorry, I am not running any special programs on it.
As the value of : swap in/out, data is being swapped in and swapped out quickly, is it also meaning that there is some fishy process?

liuwenbin168@126.com · 09-14-2012, 03:20 AM

Quote:

Originally Posted by syg00

@liuwenbin168@126.com, in future use [code] tags as @Celyr has - it retains data formatting and makes it easier (and more likely) others will try to help.

As for your problem, RAM gets used for all sorts of things by the kernel and products (like Oracle e.g.).
Lets see the output from

Code:

cat /proc/meminfo

. Use code tags or else you may get ignored.

Thank you, syg00, I will be happy to follow the rule of our forum.

syg00 · 09-14-2012, 03:50 AM

Much more readable.
It is not a rule - merely common courtesy. If you want people to read your posts, make it easy for them to help you.

You have almost 7 Gig reserved for huge pages - which aren't (currently) being used. They can be disabled (at boot), but we can't know if that will affect whatever you are running on that system. Google should help.
Personally I'd be inclined to turn it off and see what happens - but it's not my system.

chrism01 · 09-14-2012, 04:59 AM

I also recommend reading http://www.linuxatemyram.com/

Celyr · 09-14-2012, 05:27 AM

There is still something not clear to me, ok 7GB reserved for huge_page but i still think there is something wired.
can you do:

Code:

# ps aux | sort -b -k 4 | tail

and

Code:

# mount | grep ram

Code:

# mount | grep tmp

liuwenbin168@126.com · 09-14-2012, 05:31 AM

Quote:

Originally Posted by syg00

Much more readable.
It is not a rule - merely common courtesy. If you want people to read your posts, make it easy for them to help you.

You have almost 7 Gig reserved for huge pages - which aren't (currently) being used. They can be disabled (at boot), but we can't know if that will affect whatever you are running on that system. Google should help.
Personally I'd be inclined to turn it off and see what happens - but it's not my system.

Hi syg00, Thanks Very Much for pointing out my problem, my RAM issue is probably caused by huge page configuration.

Yes, my system performance slow down when I configured huge pages for it.

I need to learn more knowledge on Huge Pages then I think I will be clear on this problem.

Thanks again and have a nice day pls!

liuwenbin168@126.com · 09-14-2012, 05:38 AM

Quote:

Originally Posted by Celyr

There is still something not clear to me, ok 7GB reserved for huge_page but i still think there is something wired.
can you do:

Code:

# ps aux | sort -b -k 4 | tail

and

Code:

# df -h | grep ram

Code:

# df -h | grep tmp

Hi Celyr,

Here are the prints for these commands:

Code:

ps aux | sort -b -k 4 | tail

[root@ID5xinyong ~]# ps aux | sort -b -k 4 | tail
postfix   1508  0.0  0.0  78860   208 ?        S    Aug15   0:07 qmgr -l -t fifo -u
mqm       3202  0.0  0.0 832136  1460 ?        Sl   Aug15   1:00 /opt/mqm/bin/amqzmuf0 -m QM1
mqm      19373  0.0  0.0 832136  1460 ?        Sl   Aug27   0:34 /opt/mqm/bin/amqzmuf0 -m QM2
root      1303  0.0  0.0  93156   336 ?        S<sl Aug15   0:05 auditd
mqm       3234  0.0  0.0 958228  4284 ?        Sl   Aug15  10:42 /opt/mqm/bin/amqzlaa0 -mQM1 -fip0
mqm      19407  0.0  0.0 959368  2268 ?        Sl   Aug27   1:39 /opt/mqm/bin/amqzlaa0 -mQM2 -fip0
dbus      1340  0.0  0.0  97148     8 ?        Ssl  Aug15   0:00 dbus-daemon --system
root     29276  0.0  0.0  97756  3876 ?        S    13:22   0:00 sshd: root@pts/0
root     29364  0.0  0.0  97756  3892 ?        S    13:40   0:00 sshd: root@pts/1
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND

Code:

df -h | grep ram

[root@ID5xinyong ~]# df -h | grep ram
[root@ID5xinyong ~]#

This prints nothing when I issued it.

Code:

df -h | grep tmp
[root@ID5xinyong ~]# df -h | grep tmp
tmpfs                  16G     0   16G   0% /dev/shm

Code:

[root@ID5xinyong ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_id5xinyong-lv_root
                       50G  9.0G   38G  20% /
tmpfs                  16G     0   16G   0% /dev/shm
/dev/sda1             485M   31M  429M   7% /boot
/dev/mapper/vg_id5xinyong-lv_home
                      361G   28G  315G   9% /home

Is there anything special we can get from these prints?

Thanks a lot!!

liuwenbin168@126.com · 09-14-2012, 05:40 AM

Quote:

Originally Posted by chrism01

I also recommend reading http://www.linuxatemyram.com/

Thank you, chrism01, I will read the recommended staff.

Celyr · 09-14-2012, 06:21 AM

Quote:

Originally Posted by liuwenbin168@126.com

Is there anything special we can get from these prints?

Thanks a lot!!

I can't get anything sorry, but the situation keeps looking wired to me. The only think that I can suggest to you is to keep investigating and to lower down the number of huge pages

Code:

echo number > /proc/sys/vm/nr_hugepages

Where number can be even 10 since you are not using them at all

How many processes are you running ?

Code:

ps aux | wc -l

As you may have seen i have modified the post because ramfs doesn't get printed on df -h, try to see if you have ramfs mounted

Code:

mount | grep ramfs

liuwenbin168@126.com · 09-14-2012, 07:00 AM

Quote:

Originally Posted by Celyr

I can't get anything sorry, but the situation keeps looking wired to me. The only think that I can suggest to you is to keep investigating and to lower down the number of huge pages

Code:

echo number > /proc/sys/vm/nr_hugepages

Where number can be even 10 since you are not using them at all

How many processes are you running ?

Code:

ps aux | wc -l

As you may have seen i have modified the post because ramfs doesn't get printed on df -h, try to see if you have ramfs mounted

Code:

mount | grep ramfs

Hi Celyr,
Since I have finished work and I am at home now, I cannot run the commands you provided right now, but I will do it on Monday, and then post the prints here.

I do THANKS A MILLION for you teacher's great patience! You're really a nice man, thanks and have a nice weekend please!

liuwenbin168@126.com · 09-16-2012, 08:11 PM

Quote:

Originally Posted by Celyr

I can't get anything sorry, but the situation keeps looking wired to me. The only think that I can suggest to you is to keep investigating and to lower down the number of huge pages

Code:

echo number > /proc/sys/vm/nr_hugepages

Where number can be even 10 since you are not using them at all

How many processes are you running ?

Code:

ps aux | wc -l

As you may have seen i have modified the post because ramfs doesn't get printed on df -h, try to see if you have ramfs mounted

Code:

mount | grep ramfs

Hi Celyr,

Code:

[root@ID5xinyong ~]# echo number > /proc/sys/vm/nr_hugepages

<This is what I got:>  -bash: echo: write error: Invalid argument

and FYI:

[root@ID5xinyong ~]# cat /proc/sys/vm/nr_hugepages
3488

Code:

[root@ID5xinyong ~]# ps aux | wc -l
352

Too many processes are running. May I know how many processes are there in common? Thanks!

"mount | grep ramfs" still returns nothing.

Code:

[root@ID5xinyong ~]# mount | grep ramfs
[root@ID5xinyong ~]#

What else I can do to clarify my issue?