Hello,

i got a RHEL 5.2 machine.

and i noticed that the total used memory(as shown in "free" command) is 3GB higher than
the sum of RSS size of all processes(collected using "ps aux").

i know that its not suppose to be exact because RSS counts multiple times shared memory (so actually it suppose to be higher than the used memory).

the server runs apache which runs a pretty heavy php program.

my suspicion is that its memory leak?
or is there another reason for that?

Thanks for the help.

http://www.linuxatemyram.com/

The above URL gives a very simplified discussion of a complicated topic. I don't agree with all the details there (especially the relationship to swap).

But for your question, it is most of the answer.

Thanks for the reply.

i know about the disk cache.
but on another machine more recently installed, it has the same amount of buffers and caches being used
and it has also has free memory that is not used by the caches or the applications.
also the first machine started using swap.

about the sum of all processes RSS , isnt that should be equal to the amount of caches/buffers being used?

*** i will post the exact data tomorrow, cause i cant now.

thanks.

Maybe when I see the details, I'll have some idea what you mean by the above (the significance and/or the implied question of the above info).

That's most of what I disagree with in the LinuxAteMyRam page. Linux will use swap rather than take memory back from caching when it decides that doing so is better. That is usually an accurate choice and trying to stop it from doing so is generally a bad idea.

No.

Here are the data that i collected from the machines, notice that one machine (i will refer it as machine A) has 6GB of memory and the other (machine B)8GB.

--------------------
Machine A:

"free -m"

total used free shared buffers cached
Mem: 5938 2742 3195 0 487 1220
-/+ buffers/cache: 1034 4903
Swap: 2047 0 2047
--------
ps aux --sort rss | awk '{if($6 > 2000) print $0;}' :

------------------------------------------------------------------
Machine B:


total used free shared buffers cached
Mem: 7983 7935 47 0 428 4304
-/+ buffers/cache: 3201 4781
Swap: 2047 0 2047


ps aux --sort rss | awk '{if($6 > 2000) print $0;}'


----------------------------------------------------------------------------

the same number of apache processes(they take most of the memory) are running on both machine and are more or less the same size.

but on machine A it seems that there is more space left.

the kernel version is:

machine A: 2.6.18-194.3.1.el5

machine B: 2.6.18-128.4.1.el5

On your machine B, the memory use excluding buffers and cache is 3201MB. Both the evidence from machine A and the evidence from the ps output imply that is 2GB higher than the amount used by all those processes.

So something else is using 2GB on machine B. I don't know how to discover what.

I don't have experience with virtual machines, nor with reserving memory for large pages. So I don't know how either of those might show up in the data you displayed. I can't think at the moment of other things that might be using your missing 2GB.

these are no virtual machines.

whats disturbing to me is why the 6GB memory machine has free memory at all (except the buffers/cache).

i found something new:

on machine B , the slab is very high (2GB), here is the info from /proc/meminfo:

MemTotal: 8175136 kB
MemFree: 48636 kB
Buffers: 285236 kB
Cached: 4412088 kB
SwapCached: 0 kB
Active: 2812252 kB
Inactive: 2900016 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal: 8175136 kB
LowFree: 48636 kB
SwapTotal: 2096472 kB
SwapFree: 2096332 kB
Dirty: 136 kB
Writeback: 0 kB
AnonPages: 1014968 kB
Mapped: 19104 kB
Slab: 2361912 kB
PageTables: 24856 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
CommitLimit: 6184040 kB
Committed_AS: 1329584 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 264136 kB
VmallocChunk: 34359473527 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
Hugepagesize: 2048 kB

i wonder if its normal, cause in the other machine its a lot smaller, only 100mb

This is not a memory leak--it's the way Linux is designed.

Linux uses 'optimistic memory allocation'. In this scheme, physical frames from the RAM are not allocated after malloc until the frames are actually referenced. So, if you malloc 100MB, for example, and never reverence it, it is never reflected in RSS, but shows up as virtual memory.

I didn't think they were. I was speculating that you might be running a virtual machine under it. But now you have more info, so both my guesses were off track.

I don't think that is normal, so you should investigate further into that.

What do you get from

here is /proc/slabinfo:

2.6.16 introduced drop_caches which can be used to free unused dentries and inodes. Not sure how that'll work with NFS.
I have my doubts you'll get a lot of benefit at 2.6.18 - if the slab cache stays fragmented you may not see any benefit. A lot of work has gone into the slab (SLUB) allocator, but not so much at that kernel level I fear.

That shows that most of the the memory whose use we're trying to understand is nfs_inode_cache.

That is completely outside my expertise. I don't know how that use can get that high. Even with no memory pressure, how are there so many of whatever it is that is being cached. If memory pressure occurs later, will this cache release memory?

I assume that is a comment on the large usage by nfs_inode_cache. I don't understand much from that comment. Hopefully, kazabubu will understand whatever it is you are telling him. Is drop_caches a command you're suggesting he try?

BTW kazabubu, in the future please use CODE tags rather than QUOTE tags around output like that. That will make the info more readable for those of us trying to help you. The slabinfo data is pretty badly formatted and hard to read even with CODE tags, so I decided against suggesting CODE tags to you earlier. But now I see it is even harder to read without CODE tags.

Slab caches are outside the normally reported memory usages. Extreme examples like this led to a redesign.
drop_caches is a sysctl - see "man proc"

I wouldn't advise it's usage in a production environment normally - especially on such an old kernel. Later kernels incorporate significant changes that cause (better) consolidation of slabs, and the freeing of unused slabs. It's unlikely problems will occur, but I can see situations where memory consumption might not be helped - which could look like making things worse. Not a good position to put yourself in in a production environment.
It's possible (probable) Redhat rolled some of that back into their RHEL kernels, but I don't know that for sure.