Mystery memory usage

Z038 · 03-18-2006, 09:31 AM

I just started my system and logged on one user. I run top in a shell window under KDE and it shows nearly all of my 1GB of RAM in use. I can't see from the display what is using all that memmory. If you add up the memory under VIRT column, it comes nowhere near the amount shown as used.

Notice the 904496k total, 897056k used.

Also, 28188k buffers. What are those and where are they and who is using them?

Code:

top - 09:17:38 up  8:35,  1 user,  load average: 0.00, 0.02, 0.00
Tasks:  63 total,   2 running,  61 sleeping,   0 stopped,   0 zombie
Cpu(s):   2.7% user,   0.0% system,   0.0% nice,  97.3% idle
Mem:    904496k total,   897056k used,     7440k free,    28188k buffers
Swap:  2000084k total,     1132k used,  1998952k free,    80584k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 6025 s         17   0 16024  15m  12m S  1.7  1.8   0:04.95 ksysguard
 5950 root      17   0  142m  14m 2640 S  1.0  1.6   0:08.79 X
    1 root       8   0   244  244  216 S  0.0  0.0   0:04.43 init
    2 root       8   0     0    0    0 S  0.0  0.0   0:00.00 keventd
    3 root      19  19     0    0    0 S  0.0  0.0   0:00.00 ksoftirqd_CPU0
    4 root       9   0     0    0    0 S  0.0  0.0   0:00.52 kswapd
    5 root       9   0     0    0    0 S  0.0  0.0   0:00.00 bdflush
    6 root       9   0     0    0    0 S  0.0  0.0   0:00.02 kupdated
   10 root      -1 -20     0    0    0 S  0.0  0.0   0:00.00 mdrecoveryd
   27 root       9   0     0    0    0 S  0.0  0.0   0:00.00 kreiserfsd
   56 root       9   0     0    0    0 S  0.0  0.0   0:00.00 kjournald
   86 root       9   0   588  588  508 S  0.0  0.1   0:00.02 syslogd
   89 root       9   0   452  452  396 S  0.0  0.0   0:00.00 klogd
  199 root       7  -4   460  460  404 S  0.0  0.1   0:00.01 udevd
  258 root       9   0     0    0    0 S  0.0  0.0   0:00.02 khubd
  471 root       9   0     0    0    0 S  0.0  0.0   0:00.00 knodemgrd_0
 5264 root       8   0   516  516  460 S  0.0  0.1   0:00.00 inetd

KDE System Guard shows essentially the same thing as top in both graphical and process format.

This is what the free command shows:

Code:

             total       used       free     shared    buffers     cached
Mem:        904496     898540       5956          0      18516      79584
-/+ buffers/cache:     800440     104056
Swap:      2000084       1132    1998952

How do I determine what is using my memory?

Z038 · 03-18-2006, 09:59 AM

Also, this memory usage seems to be relatively static. There are variations over time, but not a lot really much. I ran watch -n 5 -d free and the changes were small and tended to average out over time; up a little, down a little, but no major change.

Z038 · 03-18-2006, 10:07 AM

Hey, cool. I'm learning all kinds of new commands. Here is a screen of vmstat output with a line being written every five seconds.

vmstat 5

Code:

s@here:~$ vmstat 5
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 2  0   1132   7484  28772  63204    0    0    28     5  113    85  1  0 99  0
 0  0   1132   7368  28884  63204    0    0     0    44  107   435  2  0 98  0
 1  0   1132   7264  28988  63204    0    0     0    41  107   443  3  0 97  0
 0  0   1132   7160  29092  63204    0    0     0    42  108   430  2  0 98  0
 1  0   1132   7056  29196  63204    0    0     0    41  107   378  2  0 98  0
 0  0   1132   6952  29300  63204    0    0     0    41  106   308  2  0 98  0
 1  0   1132   6848  29404  63204    0    0     0    41  108   349  3  0 97  0
 3  0   1132   6744  29508  63204    0    0     0    41  160   562  2  0 98  0
 0  0   1132   6640  29612  63204    0    0     0    41  108   397  3  0 97  0
 2  0   1132   6540  29716  63204    0    0     0    41  107   391  3  0 97  0
 0  0   1132   6428  29824  63204    0    0     0    42  108   392  2  0 98  0
 2  0   1132   6324  29928  63204    0    0     0    41  127   443  2  0 98  0
 1  0   1132   6220  30032  63204    0    0     0    41  289   863  2  1 97  0
 1  0   1132   6120  30136  63204    0    0     0    41  110   432  2  0 98  0
 1  0   1132   7096  29608  62764    0    0     0    41  110   421  2  0 98  0
 2  0   1132   6992  29712  62764    0    0     0    41  134   510  2  0 98  0
 2  0   1132   6888  29816  62764    0    0     0    41  107   431  2  0 98  0
 1  0   1132   6784  29920  62764    0    0     0    41  108   449  3  0 97  0
 0  0   1132   6680  30024  62764    0    0     0    41  106   357  2  0 98  0
 0  0   1132   6576  30128  62764    0    0     0    41  211   657  3  0 97  0

Unfortunately, this still doesn't tell me what is using my memory. I guess it is all the buffers and cache, but what is using them?

I still need someone to help me understand all this.

smoker · 03-18-2006, 12:10 PM

while running top you can sort the columns. If you type M then the %MEM column will be sorted according to highest usage. If you type c then the command used for that process is listed. Type h for more help or man top from a terminal.

Emerson · 03-18-2006, 02:28 PM

http://gentoo-wiki.com/FAQ_Linux_Memory_Management

Z038 · 03-18-2006, 09:32 PM

Thank you smoker for the tips on top. That's handy to know that top has many more commands.

Thanks for the link memory management info, Emerson. That's a good explanation.

My system looked this way immediately after I booted, before I'd run any user programs besides X and the monitors themselves. So all the data that is stored in the RAM that is used by the cache must have been saved to disk somewhere and read in again during boot.

Where is the disk copy of the cache saved? What tools exist to examine it and to tune it or modify it? Can it be deleted so the system starts clean with nothing (no data) from a past boot being loaded?

syg00 · 03-18-2006, 11:01 PM

Quote:

Originally Posted by Z038

My system looked this way immediately after I booted, before I'd run any user programs besides X and the monitors themselves. So all the data that is stored in the RAM that is used by the cache must have been saved to disk somewhere and read in again during boot.

Nope. All that eye-candy in KDE will be reloaded well before you can notice.
Maybe see if xrestop is any use in showing what the problem is.

sundialsvcs · 03-19-2006, 11:31 AM

There isn't any "problem" here.

Look at this the way that Linux does: memory is a resource to be used, and used completely. It's the only thing in the system that's just as fast as the CPU itself. Even the fastest disk-drive is several orders of magnitude slower. So, Linux wants to use all of it, for something, all the time. If the chosen use is simply, "well, we might need this again sometime soon," and there's no other actual demand for memory manifesting itself anywhere, then Linux will happily use the memory for that purpose -- "well, why not?"

Quite often, you see, there just isn't any competition for memory-space: no swapping is going on and it never will. So, Linux uses leftover memory for "buffers." When a request is made to read something from disk, it'll see if the information is already in a buffer. If so, no disk-I/O is required. And Linux will be absolutely, completely "lazy" about that... the memory will remain allocated until Doomsday, or until some competing use for the space actually appears.

When the demand for memory does increase, if it ever does, then you'll see that Linux first begins to reduce the "buffer" allocation, then begins to be more aggressive about cleaning-up the so-called "working sets" of active processes, and only then, as a last resort, begins to swap in earnest.

smoker · 03-19-2006, 11:45 AM

Exactly !
Here is my current top, this is with Azureus, Folding@home and firefox with multiple tabs and various panel applets and mysql and apache etc etc....
The machine is still relatively responsive but it's not using swap and almost all the memory is used. Don't worry about it unless you start seeing multiple megs of swap being used.

Code:

[smoker@kids ~]$ top

top - 17:38:41 up  7:45,  4 users,  load average: 1.13, 1.09, 1.03
Tasks: 121 total,   2 running, 119 sleeping,   0 stopped,   0 zombie
Cpu(s):  2.7% us,  2.0% sy, 95.0% ni,  0.0% id,  0.0% wa,  0.3% hi,  0.0% si
Mem:    970696k total,   952836k used,    17860k free,    79156k buffers
Swap:   915696k total,        0k used,   915696k free,   571664k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 3754 smoker    39  19 11904 5260 1412 R 95.5  0.5 445:05.27 FahCore_78.exe
 3260 root      15   0  144m  75m  10m S  3.3  7.9   6:29.07 X
 3715 smoker    15   0 41704  16m 9488 S  0.7  1.8   0:12.95 gnome-terminal
 3761 smoker    15   0  370m  68m  20m S  0.3  7.3   0:59.68 java
14745 smoker    16   0  2004 1008  776 R  0.3  0.1   0:00.06 top
    1 root      16   0  1732  560  480 S  0.0  0.1   0:00.65 init
    2 root      34  19     0    0    0 S  0.0  0.0   0:00.00 ksoftirqd/0
    3 root      10  -5     0    0    0 S  0.0  0.0   0:00.14 events/0
    4 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 khelper
    5 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 kthread
    9 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 kblockd/0
   10 root      20  -5     0    0    0 S  0.0  0.0   0:00.00 kacpid
  263 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 khubd
  316 root      15   0     0    0    0 S  0.0  0.0   0:00.00 pdflush
  317 root      15   0     0    0    0 S  0.0  0.0   0:00.00 pdflush
  319 root      11  -5     0    0    0 S  0.0  0.0   0:00.00 aio/0
  318 root      15   0     0    0    0 S  0.0  0.0   0:00.02 kswapd0

I might add that windows does exactly the same thing, it just shows more free memory in the sense of available to be used, not that that memory has not been allocated elsewhere for buffering. Linux is more honest about it that's all ;-)

Z038 · 03-20-2006, 10:17 PM

OK, all, I appreciate the info and tips and I understand. But... I still want to know what is in those allocated buffers and cache, if anything. My last question was

Quote:

Where is the disk copy of the cache saved? What tools exist to examine it and to tune it or modify it? Can it be deleted so the system starts clean with nothing (no data) from a past boot being loaded?

Why bother allocating all those buffers if there's nothing to put in them right away? Are they empty or is there data in them? If they are empty, the overhead of allocating empty buffers up front is pointless, as it takes virtually no time to allocate memory that isn't allocated. The only logical reason for doing it is to preload data that Linux thinks will be referenced again. So, again, is some data from previous boots cached and reloaded, and if so, where is that cache stored between boots?

Or am I just not getting this?

Oh well. Time to go smoke a pipe. Frog Morton on the Town in an Altinok meer tonight, I think. Lagavulin and a splash of water on the side.