Hi
I am having problems with a memory leak issue regarding my apache, perl ,mason installation.

for some unknown reason the apache child processes start bloating up and taking Gigs of memory off my RAM/virtual memory.
this happens at random intervals several times during the day

I have not been able to reproduce the problem myself.

I've narrowed it to the perl part of the process and noticed the following:
i've got a script that straces the proccess when they start bloating.
in strace in all the bloated proccesses these lines return all time time:
brk(0x2ae4ac5ce000) = 0x2ae4ac5ad000
mmap(NULL, 1048576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2ae90debc000
mremap(0x2ae916911000, 196222976, 196227072, MREMAP_MAYMOVE) = 0x2ae916911000
mremap(0x2ae916911000, 196227072, 196231168, MREMAP_MAYMOVE) = 0x2ae916911000
mremap(0x2ae916911000, 196231168, 196235264, MREMAP_MAYMOVE) = 0x2ae916911000
mremap(0x2ae916911000, 196235264, 196239360, MREMAP_MAYMOVE) = 0x2ae916911000
mremap(0x2ae916911000, 196239360, 196243456, MREMAP_MAYMOVE) = 0x2ae916911000
mremap(0x2ae916911000, 196243456, 196247552, MREMAP_MAYMOVE) = 0x2ae916911000
mremap(0x2ae916911000, 196247552, 196251648, MREMAP_MAYMOVE) = 0x2ae916911000
mremap(0x2ae916911000, 196251648, 196255744, MREMAP_MAYMOVE) = 0x2ae916911000
mremap(0x2ae916911000, 196255744, 196259840, MREMAP_MAYMOVE) = 0x2ae916911000


now i guess this problem is due to the file descriptor (-1) with mmap but dont know where in the code it happens
I would like to examine the returned address 0x2ae90ddbc000 and see what's there...
how do i do that ?
do you have a better way of finding out the leaker ?
( I dont have perl symbols so gdb bt isnt telling me much but here's the bt anyway :


Reading symbols from /usr/sbin/httpd...(no debugging symbols found)...done.
Attaching to program: /usr/sbin/httpd, process 15746
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
[Thread debugging using libthread_db enabled]
0x00002ae4ad8a4385 in Perl_newSV ()
from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
(gdb) #0 0x00002ae4ad8a4385 in Perl_newSV ()
from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
#1 0x00002ae4ad893c71 in ?? ()
from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
#2 0x00002ae4ad89482c in Perl_hv_fetch_ent ()
from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
#3 0x00002ae4ad89b68e in Perl_pp_helem ()
from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
#4 0x00002ae4ad89833e in Perl_runops_standard ()
from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
#5 0x00002ae4ad8456f0 in Perl_call_sv ()
from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
#6 0x00002ae4ad5ecf87 in modperl_callback ()
from /etc/httpd/modules/mod_perl.so
#7 0x00002ae4ad5ed69f in modperl_callback_run_handlers ()
from /etc/httpd/modules/mod_perl.so
#8 0x00002ae4ad5edbef in modperl_callback_per_dir ()
from /etc/httpd/modules/mod_perl.so
#9 0x00002ae4ad5e7b80 in ?? () from /etc/httpd/modules/mod_perl.so
#10 0x00002ae4ad5e7d43 in modperl_response_handler_cgi ()
from /etc/httpd/modules/mod_perl.so
#11 0x00002ae4a2c59d6a in ap_run_handler ()
#12 0x00002ae4a2c5d1ec in ap_invoke_handler ()
#13 0x00002ae4a2c67ec8 in ap_process_request ()
#14 0x00002ae4a2c650f0 in ?? ()
#15 0x00002ae4a2c611c2 in ap_run_process_connection ()
#16 0x00002ae4a2c6be5b in ?? ()
#17 0x00002ae4a2c6c0ea in ?? ()
#18 0x00002ae4a2c6c94d in ap_mpm_run ()
#19 0x00002ae4a2c47183 in main ()
(gdb) Detaching from program: /usr/sbin/httpd, process 15746



)

Thanks
Avi

Have you tried VALGRIND for memory leak detection?

The MoD

The server is a production web server with 10Ks hits a day. Would running valgrind on it wont kill the servers ?

As I said I cannot reproduce the error myself.
it happens randomly when clients perform requests to the server, that means that valgrind would have to wrap the perl handler on all requests to the perl section of the site ... right ?

As alternative , can valgrind attach to a running processes ? the docs say it doesn't ...

Avi

hi,

I'm not aware that Valgrind could attach to a running process. Even in case in can, I would never do that on a production server.

I have run an app (xnetpkg) on my Zenwalk desktop via Valgrind and found out that Perl 5.10.0 caused memory losses in the range of 150MB each time I run the program...

http://support.zenwalk.org/viewtopic.php?f=28&t=27806

Maybe you have a chance to run your tests on a backup or in a virtual machine.

The MoD

Well, putting it simply: I don't have a Dev environment to test it, and I cannot reproduce the problem anyway... It happens randomly for clients...so val grind is out of the question...
Returning to my original question: How can I inspect a memory region knowing it's address : in my example : 0x2ae916911000 - does this value mean physical or virtual mapping, how do I look at it/calculate it with gdb or something similar ?
how can trace who called the line : mmap(NULL, 1048576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2ae90debc000 ? i think the -1 parameter is suspicious..