LinuxQuestions.org
Help answer threads with 0 replies.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices


Reply
  Search this Thread
Old 10-10-2012, 05:08 PM   #1
paavo256
LQ Newbie
 
Registered: Oct 2012
Posts: 1

Rep: Reputation: Disabled
Question glibc dynamic memory trimming


We are struggling with a problem of large process sizes in a large multithreaded C++ program, which does not reflect the amount of allocated memory. For example, the process size might be 8GB while the amount of allocated memory is ca 200 MB. This happens after a heavy computation, during which it is OK to have many gigabytes allocated, but it should go back down after the task is completed.

After long efforts (involving intercepting and instrumenting all malloc calls etc (to take into account any memory leaks), and scanning the memory found in /proc/PID/maps) we have found out that there remain many mmap-ed memory areas which are empty or nearly empty. For example, in some test runs there were ca 35 mmap areas, each about 53-63 MB, which were either fully or nearly empty. (the reason for nearly empty areas are some cached static data structures typically taking less than 10kB in each of the areas). (There are also many 8 MB areas containing no dynamic allocations, but these are presumably stack segments for different threads, so these are legit and I am not worrying about them.)

The glibc malloc implementation (called ptmalloc AFAIK) is supposed to allocate new mmap areas for serving large requests (we have a lot) and also in the case of multithreading contention (we have a lot of this as well). So the presence of many mmap areas is expected. However, the question is how to trim them down or release back to OS? It seems ptmalloc does this to some extent automatically, but not nearly well enough. The malloc_trim() man page only talks about sbrk, which does not apply here at all I think. Also I could not find anything relevant in the mallopt() man page. Maybe we should try some other memory allocators? (We have tried some, with no clear success).

The tests were run on a SuSE Linux (openSUSE 11.1 (x86_64)) with glibc ver 2.9, but the problem appears in newer versions as well (e.g. openSUSE 12.1 with glibc 2.14.1).

TIA
Paavo
 
Old 10-11-2012, 05:30 AM   #2
pan64
LQ Guru
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 13,580

Rep: Reputation: 4342Reputation: 4342Reputation: 4342Reputation: 4342Reputation: 4342Reputation: 4342Reputation: 4342Reputation: 4342Reputation: 4342Reputation: 4342Reputation: 4342
I think you need to use mmap directly instead of malloc otherwise you need to implement a memory manager
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] how to allocate dynamic memory to shared memory golden_boy615 Programming 7 04-25-2018 03:34 PM
C, dynamic memory trouble isamuede Programming 11 04-03-2012 12:00 AM
Memory leaks.. *** glibc detected *** ./SuffixTree: malloc(): memory corruption: 0x00 evansash Programming 12 03-21-2011 02:17 PM
Broke dynamic linker! /lib/libc.so.6 symbol _dl_out_of_memory GLIBC defa0009 Linux - Software 4 03-20-2009 09:57 AM
[C++] Classes and dynamic memory Ephracis Programming 17 12-30-2006 02:55 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 10:59 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration