LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Distributions > Slackware
User Name
Password
Slackware This Forum is for the discussion of Slackware Linux.

Notices



Reply
 
Search this Thread
Old 12-27-2007, 12:44 PM   #1
Yalla-One
Member
 
Registered: Oct 2004
Location: Norway
Distribution: Slackware, CentOS
Posts: 635

Rep: Reputation: 35
Tracing memory leak ?


Hi,

Running top, I notice that memory used increases with 124k every minute. After a couple of hours it has consumed all available (2GB) memory and begins to swap.

What is the best way to trace down the culprit? I'm on kernel 2.6.23.12 (custom, to get built-in support for the raid controller), and have disabled most services such as ntpd, nfs, rpc and so forth...

I've run top to try to visually see which application/module increases memory consumption with 124k every minute (approximately), but it's a near impossible task.

I noticed through a google search that there was a new memory leak function in the kernel (2.6.19), but this seems to be changed as Documentation/kmemleak.txt doesn't exist on any of my recent kernels.

Any hints would be greatly appreciated!

-y1
 
Old 12-27-2007, 01:11 PM   #2
Alien_Hominid
Senior Member
 
Registered: Oct 2005
Location: Lithuania
Distribution: Hybrid
Posts: 2,247

Rep: Reputation: 53
http://gentoo-wiki.com/FAQ_Linux_Memory_Management
 
Old 12-27-2007, 01:23 PM   #3
Uncle_Theodore
Member
 
Registered: Dec 2007
Location: Charleston WV, USA
Distribution: Slackware 12.2, Arch Linux Amd64
Posts: 896

Rep: Reputation: 60
I don't think OP's problem is as simple as just misinterpreting the output of the top command. Since the system starts to swap, it might have a real memory leak. Unfortunately, finding such is a very difficult problem. There are commercial tools for finding leaks on a working system (they can be easily found on Google), but overall, your best chance is to turn off services (start from the X server) and trying to see if the leakage stops...
 
Old 12-27-2007, 01:32 PM   #4
Yalla-One
Member
 
Registered: Oct 2004
Location: Norway
Distribution: Slackware, CentOS
Posts: 635

Original Poster
Rep: Reputation: 35
Thanks Uncle_Theodore,

The problem is that even without any services running, the system steadily eats 124k memory every minute, and thus after some hours of operation has to start swapping because all memory is consumed. This is clearly due to a leak somewhere and as you say has nothing to do with misinterpreting top or how Linux pageing works.

Since this is a hobby server I'm not able to rush into getting a commercial memoryleak troubleshooter. It's a brand new system though that's been operational for less than half a day, so I'll try a different, vanilla kernel and see if it's reproduced there, and then go the slow and painful route of adding one and one kernel module to see where it all goes wrong...

If anyone has some recommendations on where to start or how to best perform this, I'm all ears!

-y1
 
Old 12-27-2007, 02:15 PM   #5
David1357
Senior Member
 
Registered: Aug 2007
Location: South Carolina, U.S.A.
Distribution: Ubuntu, Fedora Core, Red Hat, SUSE, Gentoo, DSL, coLinux, uClinux
Posts: 1,302
Blog Entries: 1

Rep: Reputation: 107Reputation: 107
Quote:
Originally Posted by Yalla-One View Post
If anyone has some recommendations on where to start or how to best perform this, I'm all ears!
Well, you should be able to get the source for your kernel and "make oldconfig" to setup the original options. Then "make menuconfig" and select "Kernel Debugging" as a built-in as well as the other memory related options (Debug slab, kobject debugging, etc.).

Unfortunately, this method has the chance of turning your problem into an Heisenbug.

Last edited by David1357; 12-27-2007 at 02:16 PM. Reason: Fixed a grammatical error
 
Old 12-28-2007, 06:35 AM   #6
Yalla-One
Member
 
Registered: Oct 2004
Location: Norway
Distribution: Slackware, CentOS
Posts: 635

Original Poster
Rep: Reputation: 35
Solved!

Found the problem!
It was the Raid controller's kernel driver. When running without it, the system stays stable overnight without dying from memory exhaustion. Will either switch to soft-raid or just replace the controller with a better solution.

-y1
 
Old 12-28-2007, 08:26 AM   #7
H_TeXMeX_H
Guru
 
Registered: Oct 2005
Location: $RANDOM
Distribution: slackware64
Posts: 12,928
Blog Entries: 2

Rep: Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269
If you found the driver that has the leak, maybe you should report it (if it isn't already reported), it's usually a rather easy fix once it is found.
 
Old 12-28-2007, 08:44 AM   #8
Yalla-One
Member
 
Registered: Oct 2004
Location: Norway
Distribution: Slackware, CentOS
Posts: 635

Original Poster
Rep: Reputation: 35
Yeah - the problem is that it turns out that the driver isn't opensource at all - it's HighPoint RocketRaid which has "Open Source" all over the box, but it turns out that all they make available is the object files, so no modifications are possible. It's borderline cheating the customer, since .o files hardly qualify as "open source", so I returned the RocketRaid card and got an Areca controller which has in-kernel support and functions like a charm.

So to everyone reading this thread considering a RAID controller from HighPoint, my advise would be to consider carefully again, since the driver is not open source, leaks memory, and as such is impossible to fix.
 
Old 12-28-2007, 09:04 AM   #9
H_TeXMeX_H
Guru
 
Registered: Oct 2005
Location: $RANDOM
Distribution: slackware64
Posts: 12,928
Blog Entries: 2

Rep: Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269
Hah, well they should get sued for that. If the driver were open source, it might have been fixed by now.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
memory leak? vegas35 Linux - Newbie 2 11-03-2007 04:29 PM
Tracing memory words for pointers KareemSedki Linux - Newbie 0 10-26-2007 01:13 AM
Memory Leak?! RoaCh Of DisCor Linux - General 9 05-07-2005 01:26 AM
Memory Leak when using memory debugging C program on SuSE SLES8 babalina Linux - Distributions 0 10-06-2003 10:39 AM
Memory Leak FredrikN Linux - General 4 09-17-2002 04:16 PM


All times are GMT -5. The time now is 01:45 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration