LinuxQuestions.org
Help answer threads with 0 replies.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 08-09-2013, 05:13 AM   #1
r_mohan
LQ Newbie
 
Registered: Aug 2013
Posts: 10

Rep: Reputation: Disabled
Inactive memory


I use linux but not well-versed with its memory management.

We have a Java application server that slows down considerably. I am quite proficient with getting the JVM details but unsure about how to interpret vmstat results.

Is there a certain way to debug the OS memory details especially inactive memory/swap etc. ? I understand this is a general question but out of 8164276 total memory, 5130836 is inactive. This is probably more about gathering vmstat reports and graphing and identifying problems. It could be that the JDBC thin client opens numerous sockets which then linger on in the OS thereby causing a bottleneck. The java application is not designed in an efficient manner and that will be addressed.

This is a production box. What other RHEL(OS-System: Linux 2.6.18-238.el5,amd64) reports should I gather ?





Thanks,
Mohan
 
Old 08-09-2013, 09:36 PM   #2
chrism01
LQ Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.8, Centos 5.10
Posts: 17,240

Rep: Reputation: 2324Reputation: 2324Reputation: 2324Reputation: 2324Reputation: 2324Reputation: 2324Reputation: 2324Reputation: 2324Reputation: 2324Reputation: 2324Reputation: 2324
Show the output of top, vmstat, iostat (say 5 or 10 lines of the *stat cmds).
 
Old 08-15-2013, 02:40 AM   #3
r_mohan
LQ Newbie
 
Registered: Aug 2013
Posts: 10

Original Poster
Rep: Reputation: Disabled
Inactive memory

Quote:
Originally Posted by chrism01 View Post
Show the output of top, vmstat, iostat (say 5 or 10 lines of the *stat cmds).
I have attached the output of vmstat -s. Even though the JVM is limited to about 2 GB some other process seems to be running amok.

This happens one every few weeks. Don't have other reports now.

Is there some type of logging mechanism to get reports of the memory subsystem instead of actively monitoring it ? It is hard to know when it happens.

Thanks.

Last edited by r_mohan; 08-15-2013 at 03:09 AM.
 
Old 08-16-2013, 07:49 AM   #4
r_mohan
LQ Newbie
 
Registered: Aug 2013
Posts: 10

Original Poster
Rep: Reputation: Disabled
Collecting socket details( SS or netstat ) and scheduled graphing

Hi,
We have tools like LogStash etc. to graph apache access logs. Since we are debugging a memory issue on RHEL we would like to look at graphs of sockets in TIME_WAIT state periodically.

We are aware of tools like Collectl etc. Is there a plot package built into linux that can be scheduled to graph
this socket output without installing anything new?
It is a production machine.


Thanks,
Mohan
 
Old 08-16-2013, 12:14 PM   #5
markseger
Member
 
Registered: Jul 2003
Posts: 242

Rep: Reputation: 26
what's wrong with installing collectl on a production machine? that's what I built it for. it's installed on some of the largest clusters in the world. we use it internally to monitor the hp public cloud?

as a curiosity what is your definition of 'built into linux'? not much is really built in but rather add things on that are part of the distro and collectl is part of the fedora, ubuntu, sles and other distros.

-mark
 
Old 08-17-2013, 12:34 PM   #6
r_mohan
LQ Newbie
 
Registered: Aug 2013
Posts: 10

Original Poster
Rep: Reputation: Disabled
We will use collectl. I was trying to debug a JVM issue. We want to track native memory leaks in the JVM because we want to rule out leaks caused by open InputStream's.

Before freeing memory
VIRT RES SHR %MEM
2764m 1.6g 16m 1.0

After freeing memory by restarting JVM

VIRT RES SHR %MEM
2536M 1.1g 16m 14.7

This does not seem to show any leaks because the heap size is 1280.

Can collectl show native memory usage of Java virtual machines to track leakages ?

I believe I have found the cause. Thousands of such 'anon' blocks in the pmap output. How do I link this back to the InputStream or something else that is causing the leak ?

0000000041fcd000 4 0 0 ----- [ anon ]
0000000041fce000 1024 12 12 rwx-- [ anon ]
00000000420ce000 4 0 0 ----- [ anon ]
00000000420cf000 1024 12 12 rwx-- [ anon ]
00000000421cf000 4 0 0 ----- [ anon ]
00000000421d0000 1024 12 12 rwx-- [ anon ]
00000000422d0000 4 0 0 ----- [ anon ]
00000000422d1000 1024 12 12 rwx-- [ anon ]
00000000423d1000 4 0 0 ----- [ anon ]
00000000423d2000 1024 12 12 rwx-- [ anon ]
00000000424d2000 4 0 0 ----- [ anon ]
00000000424d3000 1024 12 12 rwx-- [ anon ]


Thanks.

Last edited by r_mohan; 08-18-2013 at 02:14 AM. Reason: Update:
 
Old 08-19-2013, 09:34 AM   #7
r_mohan
LQ Newbie
 
Registered: Aug 2013
Posts: 10

Original Poster
Rep: Reputation: Disabled
Memory usage is odd

Hi,
Our operations team reports that the used memory is growing and it reaches 95% of the available memory. They bounce the JBoss JVM's or free the memory manually.

I have analyzed the pmap details of the jvm's. It is about this size everytime.

total kB 2808572 1822484 1805764

'top' always shows Java as the top process. After reading a few articles about sizing memory on RHEL I turned to the forum.
I am not making much headway. Is this a memory leak ?

Since the JVM is being is bounced it is considered the cause of it. What are recommendations ?

I notice thousands of 'anon' entries in the pmap file but the total is what I showed above. It seems to be within a limit always. Each anon entry seems to correspond to the default thread stack size on a 64-bit vm which is 1024kb.

000000004013b000 1016 36 36 rwx-- [ anon ]
00000000402ed000 12 0 0 ----- [ anon ]
00000000402f0000 1016 36 36 rwx-- [ anon ]
00000000403ee000 12 0 0 ----- [ anon ]
00000000403f1000 1016 44 44 rwx-- [ anon ]
00000000404ef000 12 0 0 ----- [ anon ]

This application does not close InputStream's by calling close(). Can file handles leak memory ?

http://bugs.sun.com/view_bug.do?bug_id=6735255

How do we track leakages like they do it in Solaris which the bugid describes ?
https://blogs.oracle.com/dlutz/entry...n_with_libumem

Thanks.

Last edited by r_mohan; 08-19-2013 at 09:51 AM.
 
Old 08-20-2013, 09:07 PM   #8
smallpond
Senior Member
 
Registered: Feb 2011
Location: Massachusetts, USA
Distribution: CentOS 6 (pre-systemd)
Posts: 2,610

Rep: Reputation: 702Reputation: 702Reputation: 702Reputation: 702Reputation: 702Reputation: 702Reputation: 702
If you don't close file handles, eventually you will run out. By default the max number for a process is 1024. You can check with ulimit.

"It is about this size everytime" means you don't have a leak. You can see the total virtual memory VSZ and physical memory size RSS in use by process <pid> using
Code:
ps up <pid>
Are you sure you aren't just seeing free memory go to 0, which is normal? All memory in Linux gets consumed as block cache when it isn't in use by a program.
 
Old 08-20-2013, 10:54 PM   #9
Tinkster
Moderator
 
Registered: Apr 2002
Location: in a fallen world
Distribution: slackware by choice, others too :} ... android.
Posts: 23,066
Blog Entries: 11

Rep: Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910
I took the liberty to merge your THREE closely related threads into one, and ask that you stick to one
thread per topic in the future to keep discussions sensibly in one place.
 
Old 08-28-2013, 12:42 AM   #10
r_mohan
LQ Newbie
 
Registered: Aug 2013
Posts: 10

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by smallpond View Post
Are you sure you aren't just seeing free memory go to 0, which is normal? All memory in Linux gets consumed as block cache when it isn't in use by a program.
I think it will become very low if the admin. doesn't 'free' it.

http://www.redhat.com/magazine/001nov04/features/vm/
http://tinylan.com/index.php/article...emory-in-linux
http://www.linuxatemyram.com/

I have searched and found these. Does 'free -m' the right metric to look at ? As I mentioned the 'pmap' output of the JVM that was considered a culprit stays within about 2.5gb all the time.

Thanks,
Mohan

Last edited by r_mohan; 08-28-2013 at 01:10 AM.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Inactive memory issue, Freebsd (memory leak?) JasperB *BSD 7 08-12-2008 04:19 AM
Difference between resident memory,shared memory and virtual memory in system monitor mathimca05 Linux - Newbie 1 11-11-2007 05:05 AM
IntelliMouse 1.1A PS/2 is Inactive srfpala Fedora 0 12-06-2006 08:53 PM
inactive servers LLS Linux - General 2 02-27-2005 06:48 PM
Network goes inactive beza Linux - Networking 7 10-15-2003 06:02 PM


All times are GMT -5. The time now is 10:11 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration