glibc dynamic memory trimming
We are struggling with a problem of large process sizes in a large multithreaded C++ program, which does not reflect the amount of allocated memory. For example, the process size might be 8GB while the amount of allocated memory is ca 200 MB. This happens after a heavy computation, during which it is OK to have many gigabytes allocated, but it should go back down after the task is completed.
After long efforts (involving intercepting and instrumenting all malloc calls etc (to take into account any memory leaks), and scanning the memory found in /proc/PID/maps) we have found out that there remain many mmap-ed memory areas which are empty or nearly empty. For example, in some test runs there were ca 35 mmap areas, each about 53-63 MB, which were either fully or nearly empty. (the reason for nearly empty areas are some cached static data structures typically taking less than 10kB in each of the areas). (There are also many 8 MB areas containing no dynamic allocations, but these are presumably stack segments for different threads, so these are legit and I am not worrying about them.)
The glibc malloc implementation (called ptmalloc AFAIK) is supposed to allocate new mmap areas for serving large requests (we have a lot) and also in the case of multithreading contention (we have a lot of this as well). So the presence of many mmap areas is expected. However, the question is how to trim them down or release back to OS? It seems ptmalloc does this to some extent automatically, but not nearly well enough. The malloc_trim() man page only talks about sbrk, which does not apply here at all I think. Also I could not find anything relevant in the mallopt() man page. Maybe we should try some other memory allocators? (We have tried some, with no clear success).
The tests were run on a SuSE Linux (openSUSE 11.1 (x86_64)) with glibc ver 2.9, but the problem appears in newer versions as well (e.g. openSUSE 12.1 with glibc 2.14.1).
TIA
Paavo
|
