I have a virtual (VMware workstation) SLES 12 linux install (kernel 3.12) sized at 21gb. I am running a fairly large database in it. When I start the database engine it takes up ~7gb of memory. Slowly after startup my buffer memory increases and free memory goes down to ~1gb:

From free -m:
used free
-/+ buffers/cache: 18034 3196

I have no idea what is using this buffer memmory, it's not disk cache as free reports that as only 1gb. Resident memory for all running processes is ~7gb.

When I stop all database services free memory increases to ~7gb, but buffers stay at ~15gb. Nothing I try frees them. Tried umounting all database file systems. echo 1/2/3 to /proc/sys/vm/drop_caches. Nothing works.

From free -m:
used free
-/+ buffers/cache: 14368 6862

Strangely - after I suspend my guest/database system in VMware and then resume it my memory magically returns and I have the expected ~19gb free. As soon as I restart the database the process repeats.

From free -m:
used free
-/+ buffers/cache: 568 20662

Any idea how I can see what is holding these buffers? They have to be internal, i've checked vmstat, /proc/meminfo. Thinking it is disk page cache, but that should write and shrink eventually - especially after I shut down my database engine. It is definitely not disk cache (from top & free), that never goes about ~2gb...

Any help appreciated!

have you ever checked http://linuxatemyram.com?

"Buffers" are copies of information that is stashed by the file-system in memory that is not presently being used for any other purpose.

You want the "free memory" to be comparatively low: you want the memory to be doing something useful for you, and "buffers" are a typical way that this happens. They're an opportunistic use of RAM. If someone comes along that needs RAM to hold its virtual memory, the first thing that happens is that the buffer-pool size goes down. Buffer pages are the first-in-line for stealing.