LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 08-18-2009, 01:26 PM   #1
Poki
LQ Newbie
 
Registered: May 2009
Posts: 21

Rep: Reputation: 0
Java processes memory consumption question


Hello,
Can somebody help on this please?

I need to find out how much memory do java processes consume at any given point

java heap space is set to -Xmx1024M on the machine i am testing this on.

the result of top command is :

PID PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
18288 15 0 1550m 1.3g 33m S 97 22.1 994:44.25 java
4175 25 0 801m 149m 15m S 16 2.6 1670:13 java
4176 15 0 812m 180m 15m S 4 3.1 646:49.49 java


So i want to verify that i understand it correctly:

all of the above java processes together consume (22.1 + 2.6 + 3.1)=27.8 which is ~28%

Does it mean that 1024*0.28 = 286.72M are being used out of 1024 available for java processes?


Waiting for your inputs
Thank you all very much!
 
Old 08-18-2009, 01:44 PM   #2
malekmustaq
Senior Member
 
Registered: Dec 2008
Location: root
Distribution: Slackware & BSD
Posts: 1,613

Rep: Reputation: 433Reputation: 433Reputation: 433Reputation: 433Reputation: 433
Poki,

Probably by analogy to the "free" command, some values reported, in case of "top" where separate PIDs are sub tasks they are already reported to the main Process bearing separate PID. There is a way to fine tune your "top" report in the manual page for top, it's very long, actually I have not even finished it yet, it requires more time and concentration, a user's luxury I don't presently have.

$ man top

hope this helps.
 
Old 08-19-2009, 12:22 AM   #3
sundialsvcs
LQ Guru
 
Registered: Feb 2004
Location: SE Tennessee, USA
Distribution: Gentoo, LFS
Posts: 7,162

Rep: Reputation: 2205Reputation: 2205Reputation: 2205Reputation: 2205Reputation: 2205Reputation: 2205Reputation: 2205Reputation: 2205Reputation: 2205Reputation: 2205Reputation: 2205
It is actually very difficult to "put a finger on" just how much memory a collection of processes is using. There are several reasons.

(1) All of the "memory" is virtual. But the amount of virtual memory does not directly correspond to the amount of physical resources being consumed to support that virtual memory.

(2) The virtual memory manager is "lazy." It'll keep pages lying around in real-memory. Ditto file-buffers and so forth. Unless there is actual pressure being exerted on the memory subsystem, pages won't be cleaned up or reclaimed.

(3) Much of the memory used is shared among several processes. If they're all running Java, then they're sharing (for example) the entire Java runtime engine. Even though the virtual-memory allocation cited for each process includes a tally of the shared segments, those shared segments are consuming the same physical-memory resources.
 
Old 08-19-2009, 05:38 PM   #4
Poki
LQ Newbie
 
Registered: May 2009
Posts: 21

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by sundialsvcs View Post
It is actually very difficult to "put a finger on" just how much memory a collection of processes is using. There are several reasons.

(1) All of the "memory" is virtual. But the amount of virtual memory does not directly correspond to the amount of physical resources being consumed to support that virtual memory.

(2) The virtual memory manager is "lazy." It'll keep pages lying around in real-memory. Ditto file-buffers and so forth. Unless there is actual pressure being exerted on the memory subsystem, pages won't be cleaned up or reclaimed.

(3) Much of the memory used is shared among several processes. If they're all running Java, then they're sharing (for example) the entire Java runtime engine. Even though the virtual-memory allocation cited for each process includes a tally of the shared segments, those shared segments are consuming the same physical-memory resources.


Thank you for the answer, it is very informative.
So you are saying I can't really calculate the number?
 
Old 08-19-2009, 05:49 PM   #5
johnsfine
LQ Guru
 
Registered: Dec 2007
Distribution: Centos
Posts: 5,286

Rep: Reputation: 1181Reputation: 1181Reputation: 1181Reputation: 1181Reputation: 1181Reputation: 1181Reputation: 1181Reputation: 1181Reputation: 1181
Quote:
Originally Posted by Poki View Post
I need to find out how much memory do java processes consume at any given point
The value in the RES column for one process is a decent approximation of how much memory that process is using at the moment. So the first Java process you listed is using about 1.3GB

Quote:
java heap space is set to -Xmx1024M on the machine i am testing this on.
That has no relation to the numbers shown in top. I'm not sure it has much relation to anything meaningful. Just one of your Java processes has a virtual size of 1550MB almost all of which is probably heap. I don't know what is actually controlled by that -Xmx1024M, but it is hard to believe that Java process would have a virtual size of 1550MB if its heap were really limited in the way you seem to expect.


Quote:
all of the above java processes together consume (22.1 + 2.6 + 3.1)=27.8 which is ~28%
There is some overlap (shared pages counted in each) so the total is less than the sum of the parts and it is usually very hard to say how much less, but the SHR column gives an upper limit on the amount of sharing and it is very low in this example, so in your case the sharing is a minor factor (putting the total % at some unknown value from 27.3% to 27.8%)

For added confusion, the effective working set of a process often includes a significant amount of memory identified as "cache" in tools such a top. So the RES and mem% shown in top might significantly undercount what that process is really using, while the sharing might mean adding processes together would overcount the memory use. You don't know which effect is larger so you don't know whether the total is more or less than the 28% it seems to be. If total system memory pressure is low, the fraction of these processes that is in the cache rather than in their resident set may be very low and the 27.?% might be accurate. But if system memory pressure is high Linux cycles pages more rapidly between resident and cache, so these processes might be using significantly more than 28%.

Quote:
Does it mean that 1024*0.28 = 286.72M are being used out of 1024 available for java processes?
No. Because it is 28% (roughly) of something entirely unrelated to the 1024M heap you specified.

Last edited by johnsfine; 08-19-2009 at 06:03 PM.
 
Old 08-19-2009, 06:38 PM   #6
jlliagre
Moderator
 
Registered: Feb 2004
Location: Outside Paris
Distribution: Solaris 11.3, Oracle Linux, Mint
Posts: 9,649

Rep: Reputation: 391Reputation: 391Reputation: 391Reputation: 391
Quote:
Originally Posted by johnsfine View Post
That has no relation to the numbers shown in top. I'm not sure it has much relation to anything meaningful.
-Xmx is defining the maximum heap size (here 1GB of virtual memory) so has a little relationship with the top VIRT column.

What is interesting is the first java process is using much more virtual memory than these 1 GB which means the JVM is using at least 550 MB of non-heap / native memory which is quite a lot. A large part of this virtual memory (1.3 GB) is not mapped out (i.e. active).

Possible explanations might be a very large number of classes created by the java application, a massive number of threads running concurrently, custom native methods, something else like a memory leak in the JVM ...

I would recommend running jconsole to monitor what is going on inside this JVM memory and also see how much of these 97% of CPU are used to process user methods vs garbage collecting.
 
Old 08-19-2009, 07:11 PM   #7
johnsfine
LQ Guru
 
Registered: Dec 2007
Distribution: Centos
Posts: 5,286

Rep: Reputation: 1181Reputation: 1181Reputation: 1181Reputation: 1181Reputation: 1181Reputation: 1181Reputation: 1181Reputation: 1181Reputation: 1181
Quote:
Originally Posted by jlliagre View Post
Possible explanations might be a very large number of classes created by the java application, a massive number of threads running concurrently, custom native methods, something else like a memory leak in the JVM ...

I would recommend running jconsole to monitor what is going on inside this JVM memory and also see how much of these 97% of CPU are used to process user methods vs garbage collecting.
Good answer. (Better than my answer on that aspect of the topic).

Quote:
Originally Posted by Poki View Post
I need to find out how much memory do java processes consume at any given point

java heap space is set to -Xmx1024M on the machine i am testing this on.
When you say "how much memory" I understand that to mean "how much physical memory" and that is the question I answered at length above. Maybe you mean "how much virtual memory" or "how much heap". Those are different questions. You see jlliagre identified reasons that virtual memory use might be far larger than heap. Then physical memory use is different from virtual.

I haven't used jconsole much, but I expect it could tell you quite a bit about heap state and use.

I expect your system has very low memory pressure and that is the reason the 1.3GB physical memory use of that task is such a large fraction of the 1.5GB virtual. (There are several possible reasons why the other two Java tasks have RES as a low fraction of VIRT even if memory pressure is low). But if memory pressure is high, then excess garbage collection would be the likely cause for the 1.3GB RES being so high and also the 97% CPU. So jlliagre's suggestion of investigating that task with jconsole is a good idea.
 
Old 08-20-2009, 10:42 AM   #8
Poki
LQ Newbie
 
Registered: May 2009
Posts: 21

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by johnsfine View Post
The value in the RES column for one process is a decent approximation of how much memory that process is using at the moment. So the first Java process you listed is using about 1.3GB



That has no relation to the numbers shown in top. I'm not sure it has much relation to anything meaningful. Just one of your Java processes has a virtual size of 1550MB almost all of which is probably heap. I don't know what is actually controlled by that -Xmx1024M, but it is hard to believe that Java process would have a virtual size of 1550MB if its heap were really limited in the way you seem to expect.




There is some overlap (shared pages counted in each) so the total is less than the sum of the parts and it is usually very hard to say how much less, but the SHR column gives an upper limit on the amount of sharing and it is very low in this example, so in your case the sharing is a minor factor (putting the total % at some unknown value from 27.3% to 27.8%)

For added confusion, the effective working set of a process often includes a significant amount of memory identified as "cache" in tools such a top. So the RES and mem% shown in top might significantly undercount what that process is really using, while the sharing might mean adding processes together would overcount the memory use. You don't know which effect is larger so you don't know whether the total is more or less than the 28% it seems to be. If total system memory pressure is low, the fraction of these processes that is in the cache rather than in their resident set may be very low and the 27.?% might be accurate. But if system memory pressure is high Linux cycles pages more rapidly between resident and cache, so these processes might be using significantly more than 28%.



No. Because it is 28% (roughly) of something entirely unrelated to the 1024M heap you specified.


Thanks for the reply, I will have to reread it several more times :-)
The site from which i took these params is a very busy site with a lot of traffic, and lot of cuncurrent threads.
When java server is configured -xms and -xmx are set.
Java server from time to time restarts itselft.
Need to find reason why it is doing so.
in syslogd see that java process is being killed from time to time, now java processes are the "heaviest" correct?
so trying to add more RAM to the server, trying to increase heap size, so that it will not spit "OutofMemory" exception.

Here i am just to trying to find out how can i know how much memory is java really using.

Last edited by Poki; 08-20-2009 at 10:44 AM.
 
Old 08-20-2009, 10:44 AM   #9
Poki
LQ Newbie
 
Registered: May 2009
Posts: 21

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by johnsfine View Post
Good answer. (Better than my answer on that aspect of the topic).



When you say "how much memory" I understand that to mean "how much physical memory" and that is the question I answered at length above. Maybe you mean "how much virtual memory" or "how much heap". Those are different questions. You see jlliagre identified reasons that virtual memory use might be far larger than heap. Then physical memory use is different from virtual.

I haven't used jconsole much, but I expect it could tell you quite a bit about heap state and use.

I expect your system has very low memory pressure and that is the reason the 1.3GB physical memory use of that task is such a large fraction of the 1.5GB virtual. (There are several possible reasons why the other two Java tasks have RES as a low fraction of VIRT even if memory pressure is low). But if memory pressure is high, then excess garbage collection would be the likely cause for the 1.3GB RES being so high and also the 97% CPU. So jlliagre's suggestion of investigating that task with jconsole is a good idea.


Thanks for the reply, I will have to reread it several more times :-)
The site from which i took these params is a very busy site with a lot of traffic, and lot of cuncurrent threads.
When java server is configured -xms and -xmx are set.
Java server from time to time restarts itselft.
Need to find reason why it is doing so.
in syslogd see that java process is being killed from time to time, now java processes are the "heaviest" correct?
so trying to add more RAM to the server, trying to increase heap size, so that it will not spit "OutofMemory" exception.

Here i am just to trying to find out how can i know how much memory is java really using.
 
Old 08-20-2009, 11:03 AM   #10
johnsfine
LQ Guru
 
Registered: Dec 2007
Distribution: Centos
Posts: 5,286

Rep: Reputation: 1181Reputation: 1181Reputation: 1181Reputation: 1181Reputation: 1181Reputation: 1181Reputation: 1181Reputation: 1181Reputation: 1181
Quote:
Originally Posted by Poki View Post
now java processes are the "heaviest" correct?
I have no idea what you're asking.

Quote:
trying to add more RAM to the server
Not likely to help. You haven't shown enough info for me to be sure, but I think the system has fairly low memory pressure, so adding more ram won't help.

If you think something is failing due to lack of ram, you can test that more easily by adding swap space than by adding ram:

1) If you add swap space and the process that was failing now slows down instead of failing, then you needed more ram.

2) If you add swap space and the process that was failing now works fine, then you just needed the swap space.

3) If you add swap space and the process that was failing still fails then adding ram wouldn't help either.

In your case, my guess is (3) neither ram nor swap space will help.

Quote:
trying to increase heap size, so that it will not spit "OutofMemory" exception.
There is a decent chance that will fix it. We can't tell whether it is hitting the heap limit, but probably it is. If it is hitting that heap limit maybe that is the result of a memory leak or other bug, so increasing the limit will just delay the failure. But maybe it is hitting the limit because the work it is doing is really that big and it needs a larger heap.

Quote:
i am just to trying to find out how can i know how much memory is java really using.
We already covered how you can get a good approximation of how much physical memory it is using and how much virtual memory it is using. That is detailed and accurate enough for your purposes. If you want to know how much heap memory it is using, I hope/expect jconsole can tell you, but I don't recall those details.
 
Old 08-20-2009, 11:04 AM   #11
jlliagre
Moderator
 
Registered: Feb 2004
Location: Outside Paris
Distribution: Solaris 11.3, Oracle Linux, Mint
Posts: 9,649

Rep: Reputation: 391Reputation: 391Reputation: 391Reputation: 391
So you have a lot of concurrent threads, that explains the large non heap memory used.

Any error log with a clue about java restarting itself ? you mentioned OutOfMemoryException.
What application server is this ?
using what JVM ?
with what garbage collection algorithm ?
Did you try running jconsole ?
 
Old 08-20-2009, 01:11 PM   #12
Poki
LQ Newbie
 
Registered: May 2009
Posts: 21

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by jlliagre View Post
So you have a lot of concurrent threads, that explains the large non heap memory used.

Any error log with a clue about java restarting itself ? you mentioned OutOfMemoryException.
What application server is this ?
using what JVM ?
with what garbage collection algorithm ?
Did you try running jconsole ?


app server - Resin
garbagy collection algorith - default
jconsole - no , didn't run jconsole
jvm - didn't find out yet
 
Old 08-20-2009, 01:43 PM   #13
malekmustaq
Senior Member
 
Registered: Dec 2008
Location: root
Distribution: Slackware & BSD
Posts: 1,613

Rep: Reputation: 433Reputation: 433Reputation: 433Reputation: 433Reputation: 433
johnsfine & jlliagre:

this may not be my OP but I have profitted from your posts, thus my blue thumbs up. Thank you.
---------------x
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Problem with memory consumption vidyavati Linux - Embedded & Single-board computer 1 07-09-2009 01:37 PM
CPP: Question about memory consumption of many vitural functions in a class of mine. RHLinuxGUY Programming 3 05-01-2007 12:01 AM
Memory Consumption of 10.2 naparhi Slackware 4 08-27-2006 04:10 AM
process memory consumption ilnli Linux - General 4 09-08-2005 08:13 PM
Slackware memory consumption muhkuhmasta Slackware 5 07-15-2004 06:51 AM


All times are GMT -5. The time now is 03:02 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration