I found from command 'top' that 8GB memory are used. However, using command 'ps' with some options to grep the running processes and then summing up the memory used by the running processes are less than 2 GB. Where has the used memory gone ? what command to check this? Thks.

The "used" memory reported by top includes "buffers" and "cached", which are categories of memory use that are not included in any individual process in ps.

The whole topic is more complex than you might expect. But if you want a nearly correct greatly simplified view, you should consider "buffers" and "cached" to be free memory.

So the actual free memory is the sum of what top reports as "free", "buffers" and "cached". The actual used memory is what top reports as "used" minus both "buffers" and "cached"

The free command gives you used and free values both before and after that adjustment. top only gives you the unadjusted values.

I expect even that value is more than the actual memory used by your running processes.

If you enable the RSS column in ps and sum that, you will get a value that is at least the total memory use of the processes, but may be significantly more than the total memory use. I don't know anything else you might have summed that wouldn't be even further from correct.

Hi Johnsfine. Thank you very much for your info.

Hi Johnsfine,

You have the best understanding of Linux memory than anyone I've worked with!
I wonder if you know about the Active memory metric reported from /proc/meminfo
and also vmstat -a. The metrics from top and free line up perfectly - you can see how used - cache - buffers = [actual] used and that number subtracted from Total = the [real] free amount.

I can see that the Inactive number from /proc/meminfo and vmstat -a map to Buffer amount, but I have tried every combination I can think of and still do not see what the Active number is derived from. Does it include cache ? How is it used ? I've been told that one can use free, top, /proc/meminfo, or vmstat and get the same results, but that does not appear to be quite true.
What am I missing ?????????

Really appreciate your time and expertise!
Thanks.

I don't know that aspect of it.

Certainly not so on my system.

Free plus Inactive plus Active appears to be total ram. So Inactive plus Active is the reported Used memory, including Buffers and Cache.

But I don't know what distinguishes Active from Inactive.

Active and inactive are merely classifications the memory manager code uses to determine suitability of pages for reclaim. Recently referenced pages stay on the "active" queue. After a (essentially indeterminate) period of non-reference, they migrate to the inactive q, where they are amongst the first considered for reclaim.
Trying to be exact with memory allocation is a waste of time - too many unknown variables. I would expect active pus inactive (including anon a+i) plus slab should approximate the total non page-cache in use on a "quiet" system.
It is possible for pages no longer allocated to remain on any of those queues even after the owning process has terminated. There is no way to account for those that I know of (in userspace).

Hi johnsfine, and syg00,

Thank you both for responding so quickly, especially considering it's a weekend!
I've been on other Q&A sites where questions never get a response - so thanks again.

I did a top, free, cat /proc/meminfo, and vmstat -a on a relatively quiet(right now) Linux machine right before my post. Here's pertinent output:

--------------------------------------------------------------------------------
bl460-5:$ top

Cpu(s): 0.0%us, 0.0%sy, 0.0%ni, 99.7%id, 0.3%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 16633024k total, 5745336k used, 10887688k free, 626460k buffers
Swap: 2097144k total, 0k used, 2097144k free, 3603556k cached

--------------------------------------------------------------------------------
bl460-5:$ free
total used free shared buffers cached
Mem: 16633024 5745832 10887192 0 626468 3603564
-/+ buffers/cache: 1515800 15117224
Swap: 2097144 0 2097144

--------------------------------------------------------------------------------
bl460-5:$ cat /proc/meminfo

MemTotal: 16633024 kB
MemFree: 10887688 kB
Buffers: 626468 kB
Cached: 3603564 kB
SwapCached: 0 kB
Active: 4941408 kB
Inactive: 646928 kB
HighTotal: 15858008 kB
HighFree: 10877288 kB
LowTotal: 775016 kB
LowFree: 10400 kB
SwapTotal: 2097144 kB
SwapFree: 2097144 kB
Dirty: 32 kB

-----------------------------------------------------------------------------------
bl460-5:$ vmstat 1 2 -a

r b swpd free inact active si so bi bo in cs us sy id wa st
0 0 0 10887316 646908 4941476 0 0 0 1 0 0 0 0 100 0 0
0 0 0 10887316 646904 4941480 0 0 0 20 1012 43 0 0 100 0 0

-----------------------------------------------------------------------------------

johnsfine, the top and /proc/meminfo Buffers number looked so close to the vmstat
inact number, that I was thinking it reflected the same thing. Not so ?

sysg00, thanks for admonition about a B-A-C-K-U-P. I ALWAYS take a backup before
ANY changes, and test changes on a machine that's not being used. It only takes one time of being burned to teach you that lesson (I was indeed once burned).

What we're really trying to determine is which number to report for the amount of
memory that was used during the benchmark of an application ? As I mentioned, several people have said that ALL the measurement tools (top, free, etc)
source the information they report from the same "place", and that /proc/meminfo
is the most reliable. Since the Active number from /proc/meminfo and vmstat
are a little different, it's not clear what the most accurate, "best" number
is to report.

Sorry this is so long, guys. Appreciate your help and interest.
Hope I can return the favor down the line.

coincidence.Using system-wide metrics to infer the usage by a single process (and children ?) would seem especially unpredictable - and unrepeatable.All these numbers are sampled from /proc/xxx
That means they are read at different times - each are as "correct" as any other. For that instant in time.
I would be inclined to use /proc/meminfo for the system-wide numbers, and /proc/<pid>/{stat,statm,status,smaps} for the process depending on what data format you need. You'll have to sample these yourself, or use something like sysstat or collectl.
Again these are only samples, so are not necessarily reproducable, or applicable to different environments.

Edit: if getrusage was fully implemented in Linux, the GNU time command would fit this requirement beautifully.