LinuxQuestions.org
Latest LQ Deal: Linux Power User Bundle
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices


Reply
  Search this Thread
Old 10-13-2010, 03:31 AM   #1
aviasd
LQ Newbie
 
Registered: Oct 2010
Posts: 3

Rep: Reputation: 0
Diagnose memory leak problem with GDB/strace


Hi
I am having problems with a memory leak issue regarding my apache, perl ,mason installation.

for some unknown reason the apache child processes start bloating up and taking Gigs of memory off my RAM/virtual memory.
this happens at random intervals several times during the day

I have not been able to reproduce the problem myself.

I've narrowed it to the perl part of the process and noticed the following:
i've got a script that straces the proccess when they start bloating.
in strace in all the bloated proccesses these lines return all time time:
brk(0x2ae4ac5ce000) = 0x2ae4ac5ad000
mmap(NULL, 1048576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2ae90debc000
mremap(0x2ae916911000, 196222976, 196227072, MREMAP_MAYMOVE) = 0x2ae916911000
mremap(0x2ae916911000, 196227072, 196231168, MREMAP_MAYMOVE) = 0x2ae916911000
mremap(0x2ae916911000, 196231168, 196235264, MREMAP_MAYMOVE) = 0x2ae916911000
mremap(0x2ae916911000, 196235264, 196239360, MREMAP_MAYMOVE) = 0x2ae916911000
mremap(0x2ae916911000, 196239360, 196243456, MREMAP_MAYMOVE) = 0x2ae916911000
mremap(0x2ae916911000, 196243456, 196247552, MREMAP_MAYMOVE) = 0x2ae916911000
mremap(0x2ae916911000, 196247552, 196251648, MREMAP_MAYMOVE) = 0x2ae916911000
mremap(0x2ae916911000, 196251648, 196255744, MREMAP_MAYMOVE) = 0x2ae916911000
mremap(0x2ae916911000, 196255744, 196259840, MREMAP_MAYMOVE) = 0x2ae916911000


now i guess this problem is due to the file descriptor (-1) with mmap but dont know where in the code it happens
I would like to examine the returned address 0x2ae90ddbc000 and see what's there...
how do i do that ?
do you have a better way of finding out the leaker ?
( I dont have perl symbols so gdb bt isnt telling me much but here's the bt anyway :


Reading symbols from /usr/sbin/httpd...(no debugging symbols found)...done.
Attaching to program: /usr/sbin/httpd, process 15746
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
[Thread debugging using libthread_db enabled]
0x00002ae4ad8a4385 in Perl_newSV ()
from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
(gdb) #0 0x00002ae4ad8a4385 in Perl_newSV ()
from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
#1 0x00002ae4ad893c71 in ?? ()
from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
#2 0x00002ae4ad89482c in Perl_hv_fetch_ent ()
from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
#3 0x00002ae4ad89b68e in Perl_pp_helem ()
from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
#4 0x00002ae4ad89833e in Perl_runops_standard ()
from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
#5 0x00002ae4ad8456f0 in Perl_call_sv ()
from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
#6 0x00002ae4ad5ecf87 in modperl_callback ()
from /etc/httpd/modules/mod_perl.so
#7 0x00002ae4ad5ed69f in modperl_callback_run_handlers ()
from /etc/httpd/modules/mod_perl.so
#8 0x00002ae4ad5edbef in modperl_callback_per_dir ()
from /etc/httpd/modules/mod_perl.so
#9 0x00002ae4ad5e7b80 in ?? () from /etc/httpd/modules/mod_perl.so
#10 0x00002ae4ad5e7d43 in modperl_response_handler_cgi ()
from /etc/httpd/modules/mod_perl.so
#11 0x00002ae4a2c59d6a in ap_run_handler ()
#12 0x00002ae4a2c5d1ec in ap_invoke_handler ()
#13 0x00002ae4a2c67ec8 in ap_process_request ()
#14 0x00002ae4a2c650f0 in ?? ()
#15 0x00002ae4a2c611c2 in ap_run_process_connection ()
#16 0x00002ae4a2c6be5b in ?? ()
#17 0x00002ae4a2c6c0ea in ?? ()
#18 0x00002ae4a2c6c94d in ap_mpm_run ()
#19 0x00002ae4a2c47183 in main ()
(gdb) Detaching from program: /usr/sbin/httpd, process 15746



)

Thanks
Avi
 
Old 10-15-2010, 03:24 AM   #2
The MoD
LQ Newbie
 
Registered: Nov 2008
Posts: 16

Rep: Reputation: 2
Have you tried VALGRIND for memory leak detection?

The MoD
 
Old 10-16-2010, 05:32 AM   #3
aviasd
LQ Newbie
 
Registered: Oct 2010
Posts: 3

Original Poster
Rep: Reputation: 0
The server is a production web server with 10Ks hits a day. Would running valgrind on it wont kill the servers ?

As I said I cannot reproduce the error myself.
it happens randomly when clients perform requests to the server, that means that valgrind would have to wrap the perl handler on all requests to the perl section of the site ... right ?

As alternative , can valgrind attach to a running processes ? the docs say it doesn't ...

Avi
 
Old 10-17-2010, 12:13 AM   #4
The MoD
LQ Newbie
 
Registered: Nov 2008
Posts: 16

Rep: Reputation: 2
hi,

I'm not aware that Valgrind could attach to a running process. Even in case in can, I would never do that on a production server.

I have run an app (xnetpkg) on my Zenwalk desktop via Valgrind and found out that Perl 5.10.0 caused memory losses in the range of 150MB each time I run the program...

http://support.zenwalk.org/viewtopic.php?f=28&t=27806

Maybe you have a chance to run your tests on a backup or in a virtual machine.

The MoD
 
Old 10-17-2010, 06:08 PM   #5
aviasd
LQ Newbie
 
Registered: Oct 2010
Posts: 3

Original Poster
Rep: Reputation: 0
Well, putting it simply: I don't have a Dev environment to test it, and I cannot reproduce the problem anyway... It happens randomly for clients...so val grind is out of the question...
Returning to my original question: How can I inspect a memory region knowing it's address : in my example : 0x2ae916911000 - does this value mean physical or virtual mapping, how do I look at it/calculate it with gdb or something similar ?
how can trace who called the line : mmap(NULL, 1048576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2ae90debc000 ? i think the -1 parameter is suspicious..
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
LXer: Fun with strace and the GDB Debugger LXer Syndicated Linux News 0 05-14-2006 06:54 PM
Memory Leak Problem T-u-N-i-X Linux - Hardware 2 02-06-2006 02:28 PM
Memory leak problem, i need help guys ride153 Slackware 8 01-05-2006 04:07 AM
memory leak (I think) paralyses my system - how to diagnose? Moebius Linux - Software 2 12-15-2004 04:44 PM
Memory Leak when using memory debugging C program on SuSE SLES8 babalina Linux - Distributions 0 10-06-2003 10:39 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Server

All times are GMT -5. The time now is 04:43 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration