LinuxQuestions.org
Go Job Hunting at the LQ Job Marketplace
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices

Reply
 
Search this Thread
Old 04-19-2012, 03:59 AM   #1
listerthrawn
LQ Newbie
 
Registered: Apr 2012
Posts: 10

Rep: Reputation: Disabled
Process for investigating memory usage


Hi,

I'm interested in finding out what you can advise on finding out what is using the memory on a server I administer. The server is RHEL 5.5 running on ESX4.

free -m reports: -

Code:
             total       used       free     shared    buffers     cached
Mem:          7983       5096       2887          0          2         23
-/+ buffers/cache:       5070       2913
Swap:        10239          0      10239
So it's not cache. I'm using 5Gb of RAM.

Here's the first few lines of top sorted by memory usage.

Code:
top - 09:55:29 up 1 day, 16:09,  1 user,  load average: 0.08, 0.02, 0.01
Tasks: 102 total,   1 running, 101 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   8175272k total,  5218716k used,  2956556k free,     2664k buffers
Swap: 10485752k total,        0k used, 10485752k free,    23904k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 3748 root      18   0  149m 6724  964 S  0.0  0.1   0:00.00 app.py
23852 root      15   0  132m 5884 2208 S  0.0  0.1   0:07.90 cupsd
 3811 ntp       15   0 27632 5224 4072 S  0.0  0.1   0:00.02 ntpd
 3677 haldaemo  15   0 33280 4292 1612 S  0.0  0.1   0:00.49 hald
23404 root      16   0 90152 3640 2904 S  0.0  0.0   0:00.00 sshd
 2995 root      15   0 56872 2916 2336 S  0.0  0.0   0:06.71 vmtoolsd
How would you recommend digging further into this to find out what is using the memory? My only idea at this time is that it is a 'trick' by VMWare to ensure that Linux doesn't use up all memory for caching and buffers and therefore blocks VMWare from having that memory available to give to other VMs. I can't find anything to support this idea though.

Thanks in advance

Chris
 
Old 04-19-2012, 09:05 PM   #2
macemoneta
Senior Member
 
Registered: Jan 2005
Location: Manalapan, NJ
Distribution: Fedora x86 and x86_64, Debian PPC and ARM, Android
Posts: 4,593
Blog Entries: 2

Rep: Reputation: 326Reputation: 326Reputation: 326Reputation: 326
Your displays show a system running well, with no memory stress at all.
 
Old 04-19-2012, 09:30 PM   #3
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 12,271

Rep: Reputation: 1028Reputation: 1028Reputation: 1028Reputation: 1028Reputation: 1028Reputation: 1028Reputation: 1028Reputation: 1028
Quote:
Originally Posted by listerthrawn View Post
I'm using 5Gb of RAM.
You appear to be using that.
It's a guest - everything is "smoke and mirrors". Hipervisors these days use "ballooning" to achieve the sort of effect you describe - check lsmod. Are you (also) the admin of the ESX server or (remote) hosted. If the former you should be aware of the balloon settings and effects.
 
1 members found this post helpful.
Old 04-20-2012, 03:38 AM   #4
listerthrawn
LQ Newbie
 
Registered: Apr 2012
Posts: 10

Original Poster
Rep: Reputation: Disabled
Thanks for your replies.

I am also the VMWare admin in this case. In the vCenter it describes the server not using much memory at all (private 292m, shared 141m, swapped 0, ballooned 0, unaccessed 7.57Gb, Active 0Mb (clearly not a busy server!)), and the balloon as empty.

The reason I'm concerned is that while the server is running perfectly well, we recently had a condition where another VM started to use a lot of RAM which caused the balloon to inflate and then OOM killer started killing processes. As there is a lot of swap I was a little surprised by this as I assumed that whatever was using the memory would just page out if the balloon went up but that didn't occur. I guess whatever is using that 5G of RAM is non-pageable.

If this is VMware, it would be nice if it would show it somewhere! It has plenty of opportunities in its interface to divulge this.

Is there any way of knowing for sure what is using the memory? The kernel clearly knows it is used, as it reports as being used, but it doesn't seem to want to tell me who's using it.

Thanks again,

Chris.
 
Old 04-21-2012, 04:17 AM   #5
HahUK
LQ Newbie
 
Registered: Apr 2012
Posts: 1

Rep: Reputation: Disabled
Memory Usage

Quote:
Originally Posted by listerthrawn View Post
Thanks for your replies.

I am also the VMWare admin in this case. In the vCenter it describes the server not using much memory at all (private 292m, shared 141m, swapped 0, ballooned 0, unaccessed 7.57Gb, Active 0Mb (clearly not a busy server!)), and the balloon as empty.

The reason I'm concerned is that while the server is running perfectly well, we recently had a condition where another VM started to use a lot of RAM which caused the balloon to inflate and then OOM killer started killing processes. As there is a lot of swap I was a little surprised by this as I assumed that whatever was using the memory would just page out if the balloon went up but that didn't occur. I guess whatever is using that 5G of RAM is non-pageable.

If this is VMware, it would be nice if it would show it somewhere! It has plenty of opportunities in its interface to divulge this.

Is there any way of knowing for sure what is using the memory? The kernel clearly knows it is used, as it reports as being used, but it doesn't seem to want to tell me who's using it.

Thanks again,

Chris.
Hi Chris,

If lsmod is not telling you then you could try checking /proc/meminfo to see where the memory is allocated and then go from there.

If the memory is allocated in Slabs, then use slabtop or /proc/slabinfo to see the names of the slabs using the memory. Sometimes the slab names give away what is using the memory.

Tom
 
1 members found this post helpful.
Old 04-23-2012, 03:48 AM   #6
listerthrawn
LQ Newbie
 
Registered: Apr 2012
Posts: 10

Original Poster
Rep: Reputation: Disabled
Tom,

Here's my output of lsmod and meminfo.

I don't believe they show what is using the memory as it just shows as lowmemory that is in use.

Any ideas how to dig further?

Code:
Module                  Size  Used by
ipv6                  435488  37
xfrm_nalgo             43333  1 ipv6
crypto_api             42945  1 xfrm_nalgo
ip_conntrack_netbios_ns    36033  0
ipt_REJECT             38977  1
xt_state               35265  2
ip_conntrack           91621  2 ip_conntrack_netbios_ns,xt_state
nfnetlink              40457  1 ip_conntrack
xt_tcpudp              36289  8
iptable_filter         36161  1
ip_tables              55201  1 iptable_filter
x_tables               50505  4 ipt_REJECT,xt_state,xt_tcpudp,ip_tables
vsock                 120800  0
vmci                   68744  1 vsock
vmmemctl               47148  0
acpiphp                58841  0
loop                   48721  0
dm_multipath           56921  0
scsi_dh                42177  1 dm_multipath
video                  53197  0
backlight              39873  1 video
sbs                    49921  0
power_meter            47053  0
hwmon                  36553  1 power_meter
i2c_ec                 38593  1 sbs
dell_wmi               37601  0
wmi                    41985  1 dell_wmi
button                 40545  0
battery                43849  0
asus_acpi              50917  0
acpi_memhotplug        40517  0
ac                     38729  0
lp                     47121  0
sg                     70377  0
shpchp                 70893  0
i2c_piix4              43725  0
floppy                 95465  0
i2c_core               56641  2 i2c_ec,i2c_piix4
e1000                 161621  0
parport_pc             62313  1
ide_cd                 73825  0
parport                73165  2 lp,parport_pc
pcspkr                 36289  0
serio_raw              40517  0
cdrom                  68713  1 ide_cd
dm_raid45              99657  0
dm_message             36289  1 dm_raid45
dm_region_hash         46145  1 dm_raid45
dm_mem_cache           38977  1 dm_raid45
dm_snapshot            52233  0
dm_zero                35265  0
dm_mirror              54737  0
dm_log                 44993  3 dm_raid45,dm_region_hash,dm_mirror
dm_mod                101521  11 dm_multipath,dm_raid45,dm_snapshot,dm_zero,dm_mirror,dm_log
ata_piix               56901  0
libata                209489  1 ata_piix
mptspi                 54353  2
mptscsih               69441  1 mptspi
mptbase               121349  2 mptspi,mptscsih
scsi_transport_spi     59841  1 mptspi
sd_mod                 56513  3
scsi_mod              196953  7 scsi_dh,sg,libata,mptspi,mptscsih,scsi_transport_spi,sd_mod
ext3                  168913  2
jbd                    94769  1 ext3
uhci_hcd               57433  0
ohci_hcd               56309  0
ehci_hcd               66125  0
Code:
MemTotal:      8175272 kB
MemFree:       2718868 kB
Buffers:         46580 kB
Cached:         183396 kB
SwapCached:          0 kB
Active:         113648 kB
Inactive:       171356 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:      8175272 kB
LowFree:       2718868 kB
SwapTotal:    10485752 kB
SwapFree:     10485752 kB
Dirty:              24 kB
Writeback:           0 kB
AnonPages:       55004 kB
Mapped:          18968 kB
Slab:            21408 kB
PageTables:       4212 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  12013388 kB
Committed_AS:   138532 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    268304 kB
VmallocChunk: 34359469431 kB
HugePages_Total:  2500
HugePages_Free:   2500
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Code:
             total       used       free     shared    buffers     cached
Mem:          7983       5332       2650          0         45        179
-/+ buffers/cache:       5107       2875
Swap:        10239          0      10239
Code:
top - 09:46:33 up 11 min,  3 users,  load average: 0.00, 0.01, 0.01
Tasks: 108 total,   2 running, 106 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.1%us,  0.1%sy,  0.0%ni, 99.8%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   8175272k total,  5461772k used,  2713500k free,    47156k buffers
Swap: 10485752k total,        0k used, 10485752k free,   183680k cached
 
Old 04-23-2012, 04:43 AM   #7
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 12,271

Rep: Reputation: 1028Reputation: 1028Reputation: 1028Reputation: 1028Reputation: 1028Reputation: 1028Reputation: 1028Reputation: 1028
Do the math on that HugePages allocation. That's a fixed RAM allocation - that appears to be un{used,needed}.
Might be worth considering if you really want that.
 
1 members found this post helpful.
Old 04-23-2012, 08:28 AM   #8
listerthrawn
LQ Newbie
 
Registered: Apr 2012
Posts: 10

Original Poster
Rep: Reputation: Disabled
It does seem to be the Hugepages.

Thanks for your help. I'll mark this as solved.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
monitor memory usage for process chuikingman Linux - Server 3 01-24-2011 08:14 PM
Does process memory usage drop? Zeno McDohl Linux - Newbie 3 07-10-2008 11:40 PM
getrusage and process memory usage gearoid_murphy Programming 2 11-09-2006 04:33 AM
about displaying process memory usage maginotjr Slackware 2 09-13-2005 11:43 AM
Process memory usage wombat53 Linux - Newbie 5 07-21-2005 07:42 PM


All times are GMT -5. The time now is 10:53 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration