I use linux but not well-versed with its memory management.

We have a Java application server that slows down considerably. I am quite proficient with getting the JVM details but unsure about how to interpret vmstat results.

Is there a certain way to debug the OS memory details especially inactive memory/swap etc. ? I understand this is a general question but out of 8164276 total memory, 5130836 is inactive. This is probably more about gathering vmstat reports and graphing and identifying problems. It could be that the JDBC thin client opens numerous sockets which then linger on in the OS thereby causing a bottleneck. The java application is not designed in an efficient manner and that will be addressed.

This is a production box. What other RHEL(OS-System: Linux 2.6.18-238.el5,amd64) reports should I gather ?





Thanks,
Mohan

Show the output of top, vmstat, iostat (say 5 or 10 lines of the *stat cmds).

I have attached the output of vmstat -s. Even though the JVM is limited to about 2 GB some other process seems to be running amok.

This happens one every few weeks. Don't have other reports now.

Is there some type of logging mechanism to get reports of the memory subsystem instead of actively monitoring it ? It is hard to know when it happens.

Thanks.

Hi,
We have tools like LogStash etc. to graph apache access logs. Since we are debugging a memory issue on RHEL we would like to look at graphs of sockets in TIME_WAIT state periodically.

We are aware of tools like Collectl etc. Is there a plot package built into linux that can be scheduled to graph
this socket output without installing anything new?
It is a production machine.


Thanks,
Mohan

what's wrong with installing collectl on a production machine? that's what I built it for. it's installed on some of the largest clusters in the world. we use it internally to monitor the hp public cloud?

as a curiosity what is your definition of 'built into linux'? not much is really built in but rather add things on that are part of the distro and collectl is part of the fedora, ubuntu, sles and other distros.

-mark

We will use collectl. I was trying to debug a JVM issue. We want to track native memory leaks in the JVM because we want to rule out leaks caused by open InputStream's.

Before freeing memory
VIRT RES SHR %MEM
2764m 1.6g 16m 1.0

After freeing memory by restarting JVM

VIRT RES SHR %MEM
2536M 1.1g 16m 14.7

This does not seem to show any leaks because the heap size is 1280.

Can collectl show native memory usage of Java virtual machines to track leakages ?

I believe I have found the cause. Thousands of such 'anon' blocks in the pmap output. How do I link this back to the InputStream or something else that is causing the leak ?

0000000041fcd000 4 0 0 ----- [ anon ]
0000000041fce000 1024 12 12 rwx-- [ anon ]
00000000420ce000 4 0 0 ----- [ anon ]
00000000420cf000 1024 12 12 rwx-- [ anon ]
00000000421cf000 4 0 0 ----- [ anon ]
00000000421d0000 1024 12 12 rwx-- [ anon ]
00000000422d0000 4 0 0 ----- [ anon ]
00000000422d1000 1024 12 12 rwx-- [ anon ]
00000000423d1000 4 0 0 ----- [ anon ]
00000000423d2000 1024 12 12 rwx-- [ anon ]
00000000424d2000 4 0 0 ----- [ anon ]
00000000424d3000 1024 12 12 rwx-- [ anon ]


Thanks.

Hi,
Our operations team reports that the used memory is growing and it reaches 95% of the available memory. They bounce the JBoss JVM's or free the memory manually.

I have analyzed the pmap details of the jvm's. It is about this size everytime.

total kB 2808572 1822484 1805764

'top' always shows Java as the top process. After reading a few articles about sizing memory on RHEL I turned to the forum.
I am not making much headway. Is this a memory leak ?

Since the JVM is being is bounced it is considered the cause of it. What are recommendations ?

I notice thousands of 'anon' entries in the pmap file but the total is what I showed above. It seems to be within a limit always. Each anon entry seems to correspond to the default thread stack size on a 64-bit vm which is 1024kb.

000000004013b000 1016 36 36 rwx-- [ anon ]
00000000402ed000 12 0 0 ----- [ anon ]
00000000402f0000 1016 36 36 rwx-- [ anon ]
00000000403ee000 12 0 0 ----- [ anon ]
00000000403f1000 1016 44 44 rwx-- [ anon ]
00000000404ef000 12 0 0 ----- [ anon ]

This application does not close InputStream's by calling close(). Can file handles leak memory ?

http://bugs.sun.com/view_bug.do?bug_id=6735255

How do we track leakages like they do it in Solaris which the bugid describes ?
https://blogs.oracle.com/dlutz/entry...n_with_libumem

Thanks.

If you don't close file handles, eventually you will run out. By default the max number for a process is 1024. You can check with ulimit.

"It is about this size everytime" means you don't have a leak. You can see the total virtual memory VSZ and physical memory size RSS in use by process <pid> using Are you sure you aren't just seeing free memory go to 0, which is normal? All memory in Linux gets consumed as block cache when it isn't in use by a program.

I took the liberty to merge your THREE closely related threads into one, and ask that you stick to one
thread per topic in the future to keep discussions sensibly in one place.

I think it will become very low if the admin. doesn't 'free' it.

http://www.redhat.com/magazine/001nov04/features/vm/
http://tinylan.com/index.php/article...emory-in-linux
http://www.linuxatemyram.com/

I have searched and found these. Does 'free -m' the right metric to look at ? As I mentioned the 'pmap' output of the JVM that was considered a culprit stays within about 2.5gb all the time.

Thanks,
Mohan