LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices


Reply
  Search this Thread
Old 06-09-2004, 09:56 PM   #1
BrianK
Senior Member
 
Registered: Mar 2002
Location: Los Angeles, CA
Distribution: Debian, Ubuntu
Posts: 1,334

Rep: Reputation: 51
How do you find a memory leak?


Something on my file server is leaking memory... if I top the machine, I can see the used memory go up 8K every 6 seconds, but if I sort my top by memory, I don't see anyone's memory usage increase with the total memory (maybe I'm not seeing all the options I need to see in top?).

Is there a good way to see what's causing the leak?
 
Old 06-09-2004, 10:13 PM   #2
Tinkster
Moderator
 
Registered: Apr 2002
Location: earth
Distribution: slackware by choice, others too :} ... android.
Posts: 23,067
Blog Entries: 11

Rep: Reputation: 928Reputation: 928Reputation: 928Reputation: 928Reputation: 928Reputation: 928Reputation: 928Reputation: 928
http://valgrind.kde.org/


Cheers,
Tink
 
Old 06-09-2004, 10:40 PM   #3
BrianK
Senior Member
 
Registered: Mar 2002
Location: Los Angeles, CA
Distribution: Debian, Ubuntu
Posts: 1,334

Original Poster
Rep: Reputation: 51
Quote:
Originally posted by Tinkster
http://valgrind.kde.org/
this might be my only option, however, the documentation seems to imply that it will look for
problems with specific executables only, which would mean that I would need to stop all of
my deamons and start them up with valgrind individually - not something I'd really like to do.

Is there any other way?
 
Old 06-09-2004, 10:54 PM   #4
Tinkster
Moderator
 
Registered: Apr 2002
Location: earth
Distribution: slackware by choice, others too :} ... android.
Posts: 23,067
Blog Entries: 11

Rep: Reputation: 928Reputation: 928Reputation: 928Reputation: 928Reputation: 928Reputation: 928Reputation: 928Reputation: 928
watch -n 10 -d "ps -A vm"

Make the xterminal window LONG :}



Cheers,
Tink
 
Old 06-09-2004, 11:10 PM   #5
BrianK
Senior Member
 
Registered: Mar 2002
Location: Los Angeles, CA
Distribution: Debian, Ubuntu
Posts: 1,334

Original Poster
Rep: Reputation: 51
Quote:
Originally posted by Tinkster
watch -n 10 -d "ps -A vm"
oo.. good idea. hadn't thought of that one. (I didn't know about the -d option to watch either - how nice )

unfortunately, no change other than mem usage and/or PID of watch and ps.

hmmm....
 
Old 06-10-2004, 06:28 AM   #6
spiky
LQ Newbie
 
Registered: Oct 2002
Location: Hamburg, Germany
Distribution: Suse 7.2, Debian 3.0
Posts: 11

Rep: Reputation: 0
Re: How do you find a memory leak?

Quote:
Originally posted by BrianK
Something on my file server is leaking memory
I think you are wrong. It's probably not a memory leak. Linux is just buffering / caching some used files or whatever. I've see the same on many linux boxes, on one used memory increases by 16 kb / 2sec, but also the buffer (shown in my top verion) increases, so it's just buffered, probably not a leak.

If it is a leak, then it should eat up all memory and also make the used swap-space grow. If it's not a leak, the swap-space will not be used. As far as I know...

Christoph
 
Old 06-10-2004, 12:02 PM   #7
BrianK
Senior Member
 
Registered: Mar 2002
Location: Los Angeles, CA
Distribution: Debian, Ubuntu
Posts: 1,334

Original Poster
Rep: Reputation: 51
Re: Re: How do you find a memory leak?

Quote:
Originally posted by spiky
I think you are wrong. It's probably not a memory leak. Linux is just buffering / caching some used files or whatever. I've see the same on many linux boxes, on one used memory increases by 16 kb / 2sec, but also the buffer (shown in my top verion) increases, so it's just buffered, probably not a leak.

If it is a leak, then it should eat up all memory and also make the used swap-space grow. If it's not a leak, the swap-space will not be used. As far as I know...

.. interesting. The only problem I have with this is why would Linux continue to buffer until the memory was maxed out & why would this cause programs to stop working? ... and why did it just start happening - as opposed to being that way from the get go?
 
Old 06-10-2004, 01:46 PM   #8
LoK
Member
 
Registered: May 2004
Location: Detroit MI
Distribution: RHE & FC
Posts: 31

Rep: Reputation: 15
You more than likely have a limit set on a process that is too high. Thus it is using all of the memory you give it.

In order to be any real help we need to know...

A) What is this server for (is it your webserver? are you running apache? cpanel? vsftpd/proftpd? mysql?).
B) Is the server actually crashing/locking up?
C) What does "programs won't start" mean?
D) Is your box x86?

If a program you are running indeed does have a memory leak the box will lock up intermittently. You should notice an increase in swap usage over time and once it maxes that out it will lock up. Programs will not cease to function, the whole system will freeze.
 
Old 06-10-2004, 02:09 PM   #9
BrianK
Senior Member
 
Registered: Mar 2002
Location: Los Angeles, CA
Distribution: Debian, Ubuntu
Posts: 1,334

Original Poster
Rep: Reputation: 51
Quote:
Originally posted by LoK
You more than likely have a limit set on a process that is too high. Thus it is using all of the memory you give it.
In order to be any real help we need to know...
A) What is this server for (is it your webserver? are you running apache? cpanel? vsftpd/proftpd? mysql?).

It's a simple NFS & Samba file server in addition to serving licenses for an animation package called "Houdini".
It has apache, mysql, php, etc, but none of them are running.
I attempted to make it an NIS server, but started having problems & had more pressing issues to deal with,
so abandoned it.
Quote:
B) Is the server actually crashing/locking up?

I've never let it get that far. When the memory gets to about 99% full, the Houdini license server stops
running which causes work at this office to grind to a screeching halt. The only way to get the license
back up is to reboot.
Quote:
C) What does "programs won't start" mean?

Any reference to a program not working really points back to the Houdini license server. When I said "programs
stop working" I specifically meant that the houdini license server has stopped serving licenses. I suspect that the
license server is causing the problem, but if I shut down the server, I still see the used memory increase at the
same rate as when the server is running. That said, this problem didn't show up until I installed the license server.
baffling.
Quote:
D) Is your box x86?

yes. It's an Intel P4 on an Intel mobo running RH9 with the 2.4.20-8smp kernel. Unfortunately, I have to use that
kernel because it is the only kernel the Promise RAID card will work under.

Quote:
If a program you are running indeed does have a memory leak the box will lock up intermittently. You should notice an increase in swap usage over time and once it maxes that out it will lock up. Programs will not cease to function, the whole system will freeze.
I agree with you, however, I'm not sure when I'll have time to test it being that someone is here 18 hours a day,
7 days a week. I'll likely move the license server and reconfigure all the boxes to use the new server. This is the
first time the server has been on a RH9 box (it's always been on an RH7.3 box until a few days ago).

Thanks for the help.
 
Old 06-10-2004, 05:58 PM   #10
BrianK
Senior Member
 
Registered: Mar 2002
Location: Los Angeles, CA
Distribution: Debian, Ubuntu
Posts: 1,334

Original Poster
Rep: Reputation: 51
hmm... as I was transfer some very large files via NFS, I watched the free memory go all the way to 12 MB, then it kept going from 12 to 24, then slowly back to 12, then up to 24... maybe it's just caching the way it's supposed to and maybe it was just a coincidence that the license server went down those other time.

still no idea. If there was a better way to track this sort of thing, this would all be so much easier.
 
Old 06-10-2004, 10:34 PM   #11
LoK
Member
 
Registered: May 2004
Location: Detroit MI
Distribution: RHE & FC
Posts: 31

Rep: Reputation: 15
Well, there are actually a number of ways you could monitor the processes/memory.

To keep overall tabs on the usage you can use vmstat, mrtg (there are numerous prefab packages out there in rpm form if you haven't used it before), or another cool prefab one is HotSaNIC. This is one thing the linux community loves, it's graphs, so you should have many options in that area.

For individual processes (such as NFS/Houdini/Apache) you can parse the process stats (proc/proc#/status) to see it's exact usage or intermittently do a ps -aux to look at them all.

When the system gets bogged down you should see it in a process report (ps -aux). There are a ton of options but aux should show you pretty much all you need. Just pay attention to the mem section.

If Apache is using your memory you can set "RLimitMEM" to limit how much ram each server uses (multiply that by the number of max spares). If the Houdini Lic interfaces with Apache then that could be causing a leak (the module it loads). If that was the case you could set the "maxrequestsperchild" value to 10000 (the suggested value). That way if it's causing a leak it will start new children every so often. You might also want to look around to see if any of your logs have bloated up past 1-2gb as they too can cause excessive memory use (along with anything that's constantly writing errors).

If NFS is your problem I'm at a loss because I don't use it From what I do know though, most of the tweaking is done in the kernel (at least it is in aix/hpux or was).

If it's Houdini then contact their tech support

In the interim I would suggest keeping a close eye on the resource reports and if you locate the problem process add a cron job to restart it every so often while you sort things out. It's not a pretty fix but it will buy you some time

In my personal experience "running out of memory" has always been Apache, mysql (due to Apache for the record), squid, or something that "looked cool" on freshmeat

Post up your process report if you unsure about it. I know RH installs a ton of crap that you don't always need so it could very well be something that is mis-configured (simply due to the fact that you don't use it).
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
what is memory leak blackzone Programming 6 11-05-2004 10:32 AM
Possible Memory Leak liquidcool Linux - General 3 10-07-2004 08:38 AM
possible memory leak?? matt80 Linux - General 2 07-01-2004 11:07 PM
Memory Leak don_dimo Linux - General 7 03-06-2004 06:51 PM
Memory Leak when using memory debugging C program on SuSE SLES8 babalina Linux - Distributions 0 10-06-2003 09:39 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - General

All times are GMT -5. The time now is 02:08 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration