How do you find a memory leak?
 

Something on my file server is leaking memory... if I top the machine, I can see the used memory go up 8K every 6 seconds, but if I sort my top by memory, I don't see anyone's memory usage increase with the total memory (maybe I'm not seeing all the options I need to see in top?).

Is there a good way to see what's causing the leak?

http://valgrind.kde.org/


Cheers,
Tink

this might be my only option, however, the documentation seems to imply that it will look for
problems with specific executables only, which would mean that I would need to stop all of
my deamons and start them up with valgrind individually - not something I'd really like to do.

Is there any other way?

watch -n 10 -d "ps -A vm"

Make the xterminal window LONG :}



Cheers,
Tink

oo.. good idea. hadn't thought of that one. (I didn't know about the -d option to watch either - how nice ;))

unfortunately, no change other than mem usage and/or PID of watch and ps.

:confused: hmmm....

Re: How do you find a memory leak?
 

I think you are wrong. It's probably not a memory leak. Linux is just buffering / caching some used files or whatever. I've see the same on many linux boxes, on one used memory increases by 16 kb / 2sec, but also the buffer (shown in my top verion) increases, so it's just buffered, probably not a leak.

If it is a leak, then it should eat up all memory and also make the used swap-space grow. If it's not a leak, the swap-space will not be used. As far as I know...

Christoph

Re: Re: How do you find a memory leak?
 

.. interesting. The only problem I have with this is why would Linux continue to buffer until the memory was maxed out & why would this cause programs to stop working? ... and why did it just start happening - as opposed to being that way from the get go?

You more than likely have a limit set on a process that is too high. Thus it is using all of the memory you give it.

In order to be any real help we need to know...

A) What is this server for (is it your webserver? are you running apache? cpanel? vsftpd/proftpd? mysql?).
B) Is the server actually crashing/locking up?
C) What does "programs won't start" mean?
D) Is your box x86?

If a program you are running indeed does have a memory leak the box will lock up intermittently. You should notice an increase in swap usage over time and once it maxes that out it will lock up. Programs will not cease to function, the whole system will freeze.

It's a simple NFS & Samba file server in addition to serving licenses for an animation package called "Houdini".
It has apache, mysql, php, etc, but none of them are running.
I attempted to make it an NIS server, but started having problems & had more pressing issues to deal with,
so abandoned it.

I've never let it get that far. When the memory gets to about 99% full, the Houdini license server stops
running which causes work at this office to grind to a screeching halt. The only way to get the license
back up is to reboot.

Any reference to a program not working really points back to the Houdini license server. When I said "programs
stop working" I specifically meant that the houdini license server has stopped serving licenses. I suspect that the
license server is causing the problem, but if I shut down the server, I still see the used memory increase at the
same rate as when the server is running. That said, this problem didn't show up until I installed the license server.
baffling.

yes. It's an Intel P4 on an Intel mobo running RH9 with the 2.4.20-8smp kernel. Unfortunately, I have to use that
kernel because it is the only kernel the Promise RAID card will work under.

I agree with you, however, I'm not sure when I'll have time to test it being that someone is here 18 hours a day,
7 days a week. I'll likely move the license server and reconfigure all the boxes to use the new server. This is the
first time the server has been on a RH9 box (it's always been on an RH7.3 box until a few days ago).

Thanks for the help. ;)

hmm... as I was transfer some very large files via NFS, I watched the free memory go all the way to 12 MB, then it kept going from 12 to 24, then slowly back to 12, then up to 24... maybe it's just caching the way it's supposed to and maybe it was just a coincidence that the license server went down those other time.

still no idea. If there was a better way to track this sort of thing, this would all be so much easier. ;)

Well, there are actually a number of ways you could monitor the processes/memory.

To keep overall tabs on the usage you can use vmstat, mrtg (there are numerous prefab packages out there in rpm form if you haven't used it before), or another cool prefab one is HotSaNIC. This is one thing the linux community loves, it's graphs, so you should have many options in that area.

For individual processes (such as NFS/Houdini/Apache) you can parse the process stats (proc/proc#/status) to see it's exact usage or intermittently do a ps -aux to look at them all.

When the system gets bogged down you should see it in a process report (ps -aux). There are a ton of options but aux should show you pretty much all you need. Just pay attention to the mem section.

If Apache is using your memory you can set "RLimitMEM" to limit how much ram each server uses (multiply that by the number of max spares). If the Houdini Lic interfaces with Apache then that could be causing a leak (the module it loads). If that was the case you could set the "maxrequestsperchild" value to 10000 (the suggested value). That way if it's causing a leak it will start new children every so often. You might also want to look around to see if any of your logs have bloated up past 1-2gb as they too can cause excessive memory use (along with anything that's constantly writing errors).

If NFS is your problem I'm at a loss because I don't use it :scratch: From what I do know though, most of the tweaking is done in the kernel (at least it is in aix/hpux or was).

If it's Houdini then contact their tech support ;)

In the interim I would suggest keeping a close eye on the resource reports and if you locate the problem process add a cron job to restart it every so often while you sort things out. It's not a pretty fix but it will buy you some time ;)

In my personal experience "running out of memory" has always been Apache, mysql (due to Apache for the record), squid, or something that "looked cool" on freshmeat :D

Post up your process report if you unsure about it. I know RH installs a ton of crap that you don't always need so it could very well be something that is mis-configured (simply due to the fact that you don't use it). :scratch: