Memory reporting corrupt?
 

First, apologies - I posted about this before in the SuSE board, but perhaps it's not a SuSE issue. And it really troubles me.

My system (SuSE 10 amd64, newly installed) seems to report inconsistent data for memory usage. This is what I see in top:

while free gives me

So, java is using 1.5 GIGAbytes of virtual memory? 200 mb in physical, and 1.3 gigabytes swapped out? This can't be, since I am only using 380 + 212 (physical + swap) mb of ram. Java seems to take up more virtual memory than I have physical+virtual memory in my system...

Also, I see similar things with knotify (virtual memory of over a gigabyte). It isn't just java, but is most pronounced there.

I have read the man pages several times, and googled for help on this. My only conclusion after all of this is that something is corrupt in my system's memory reporting?

it's a little daunting but its normal.

linux links demanding programs to lots of disk cache in memory which is good.

That cache (over 1gig) is phyisical RAM
its used to reduce Hadr disk reads/writes/fragmentation.

since it is cache though it will quickly go away for other programs to use, hense it is actually really free to use.

that is why you have one free 20MB with
free Cache -+ 1GB

you really have 1GB+20MB free RAM if you need it.




think of it like
Low priority for software to use the CPU.

low means a program can only use the CPU when nothing else is.

DiskCache can use the RAM only if nothing else needs it.

i love the Disk Cache Quirk
i paid a lot of money for 4GB ram and Windows uses 512MB and insists on using 200MB pagefile.

Thanks carl0ski, but I still don't understand.

I realize Linux is using all my physical RAM as cache, hence just a few megs free physical memory. I am not worried about that. What worries me is that java is reported to use 1.5 Gigabytes of RAM! A single application, that takes more RAM than physical + swap that I have... :-(

ok the thing is to understand what is virtual memory.
it doesn't have anything to do with physical anything accept proccessor bandwidth.
all 32 bit processors have 4 gigs of virtual memory
that is 2 to the 32nd power distinct values. If you allow each value to address a different byte of memory, you get 2 to the 32nd power bytes, which equals four gigabytes.
so on a 32 bit processor all things get 4 gigs virtual address space.
this is total how many unique values can be in address space that then can point or not sometimes and not to real places in memory or not on a disk or wherever some puter info might be living.

I really don't understand, I guess.

Please tell me one thing: based on the data I quoted above, is java using 1.5 gigabytes of memory (physical+swap), or not?

If not, then please tell me this: how can I tell how much physical and swap memory java (or any other process) is using? I simply want to know how much physical memory is used by it, and how much of my swap file is used by it.

Thanks in advance.

my java does it too,

in my case i have azureus leaching Suse

but that is very Hard drive intensive and data going out is frequently tthe same so it is remembered.

it is cleared if i need it.

well it's extreemly difficult to tell what physical swap is being used. all swap above is total virtual minus RSS. not a representation of actual swap use.
But the number you should be looking at is RES or "resident set size". That's basically stuff in actual physical memory or RESIDENT in physical memory. That would be 203m for your java process.

So, what you are saying, foo_bar_foo, is that there is NO WAY to know how much of my swap file a process is using?

If that is true, then what is the meaning of the SWAP field appearing in top? It's just garbage?

if you use KDE
open Kcontrol

Configure you desktop.

there will be and information tab>
memory

will give you a graphical display to give you a clear veiw of whats going on.

orange is Disk Cache.



P.S i just went back to Mandriva 32bit, it wouldnt detect more than 800MB ram :S

By the way, java is using the amount of memory it is allowed to.

Look at the script launching your java application for the -Xms and -Xmx options, and you'll find the initial and maximal amount of virtual memory set.

Like all other userland programs, java only knows about virtual memory, only the kernel deals with physical RAM.

please try following commands:

# pmap [pid of that java process]
# cat /proc/[pid]/maps

maybe we can see in detail where are all those 1.5G used. maybe just some big shared libraries that really do nothing but sitting there.

Hmm, thanks, pczou, I didn't know of those commands (well cat I of course knew, but not to do that...).

Ok, I did it, and it's very very long. So I won't post it all. But the interesting part is this:
As for cat /proc, it's also too long to post here. Also, I couldn't find a line(s) that correspond directly to the above (but this seems the closest:
2aaaae2c3000-2aaaedc93000 rwxp 2aaaae2c3000 00:00 0 - seems useless to me, though, but I don't know).

The [ anon ] means that there isn't a name corresponding to the memory chunk, I see in the documentation. So, is there anything else I can do to find out what is taking a gigabyte of RAM?

Also, I ask again: if this process is indeed of size 1.5 gigabytes, where does it reside? My physical+swap isn't big enough for it. I must not understand something fundamental here.

memory allocation pool of java can be limited by option -Xmx, e.g. 'java -Xms16m -Xmx128m'.

the 1.5G memory is just the virtual memory space taken by the process, physical memory is allocated only on demand. for example, if a large chunk of memory ls malloc'd or large file is mmap'd, the virtual memory grows but actually no physical memory is alloced (yet). so the 'RES' field of top output shows the resident memory really consumed by the java process, no need to worry too much about the 'VIRT' field.

The funny thing is, java takes 1.6 gigabytes (it rose since my original post) even though I ran it with -Xmx64m. And Knotify now uses 619 mb. Something just seems 'wrong' with all of this. Ah well...

So Linux has no problem allocating more ram than I have physical+swap? It does this and hopes that not all of it will actually be used?

that java process could have some memory leak, it could due to the bug of java itself (unlikely) or some modules (e.g. oracle JDBC module) loaded by java.

It's kind of like the way that a bank "loans money," with account-balances several times greater than the total amount of assets on-hand at any one time. You have a resource-allocation that is so-big. Even though it would not be physically possible to honor the entire request, you'll never actually demand at one time.

Thanks guys, I think I understand now. So, there are memory leaks somewhere in java and knotify, it seems.

I have a Tomcat server sitting idle at the moment and I'm watching a window running top, filtered for just the user running Tomcat. According to the info in the Swap section, my total swap usage is 51MB - but according to the process list, the swap usage is 236 MB:

The output of free confirms the 51MB figure for swap - much like the situation kripkenstein describes. If the article at http://javatechniques.com/public/jav...questions.html is correct, this is because:

It sounds like the problem is with ps (and therefore top) treating the shard memory for the Java threads as separate because they each get their own process.

Aha! Excellent stuff, gilead, thanks. So, if I understand correctly, this is like the usual issue with having to consider shared memory (the actual memory report is inflated because shared libraries are counted in the memory of each process), but WITHIN java (i.e. the java threads).

So there are many java threads, that share a chunk of memory, and ps counts that chunk once per thread - thus reaching the huge memory usage figure. This is confusing because top reports only the single process for java.

Now, is there a way to see the true memory usage, i.e. to account for this? I want to see the real amount of swap memory java is using.

Another question, this doesn't explain the memory usage of knotify that I see (760mb virtual, and counting...), right? Is this just a memory leak?

I'd say you're right, but memory management is not something I know a lot about. Since top uses ps, the following might help:

This will show my Tomcat Java process 'normally':

This is for showing threads:

top doesn't use ps, but both top and ps use the underlying APIs to retrieve process information, on Linux, the /proc directory.

Ah, my mistake - I copied a pseudo top from somewhere a while back and assumed (oops) that it worked the same way as the real one.