LinuxQuestions.org - bug in memory management

- Linux - Kernel (https://www.linuxquestions.org/questions/linux-kernel-70/)

- - bug in memory management (https://www.linuxquestions.org/questions/linux-kernel-70/bug-in-memory-management-823400/)

bug in memory management

I believe I have found a serious flaw in memory management using malloc() and free(). This sounds incredible, but please bear with me. I have made a test program that causes the problem, and the program is too simple to have a bug. It is listed below.

Test Program Logic

Code:

1. Read two input parameters: allocation amount and loop count.

2. Accept input of 'return' to continue or 'x' to exit.

3. Loop for loop count:

      malloc() a random memory amount 

      (average = step 1 allocation amount)

4. Loop for loop count:

      free() each piece of memory allocated

5. Loop back to step 2.

Note that if the program is waiting for input, there is no memory allocated.

Procedure to test:

1. Start a resource monitor so you can watch the amount of real memory in use.

2. Start the test program in one terminal. I used amount = 100 and loops = 10 million, so that an even 1 gigabyte is used.

3. Watch the monitor. It may go up and then down again, which is the expected behavior, since the program immediately releases all the memory right after allocating it. The 10 million calls to malloc() and free() need a few seconds, normally. Cycle the program with 'return' until you see the memory usage go up and not come back down. This is abnormal and deadly.

4. Leave the first test program at a state where the allocated memory stays high.

5. Start a 2nd test program in another terminal and do the same. After a few iterations it will also fail to release the memory.

6. Do the same in a 3rd terminal window.

7. The system will eventually run out of memory and slow to a crawl with swap space slowly increasing. This is happening even though there is plenty of memory wrongly retained for the test programs that have released their memory and are waiting for input.

If the randomized malloc() amount is replaced with a fixed amount, the test programs work as expected (i.e. malloc() and free() work as expected). The bug is related to the highly variable amount of memory being allocated. Uncomment the commented line of code to verify this.

I am using Ubuntu 10.04 with kernel 2.6.32, 64-bit.

Ubuntu 8.10 with kernel 2.6.27 also exhibits the problem, but with different characteristics: each test program will fail to release its memory for the fist iteration only, and then works normally forever after that. A memory crisis can be caused by running many test programs one cycle only, and leave them waiting for input.

Can someone else please test this?

If I have gotten anyone's attention, what is the next step to get the attention of someone able to fix it?

Mike

Code:

/**************************************************************************



  test malloc() and free()

  compile: gcc -o malloctest malloctest.c



***************************************************************************/



#include  <stdio.h>

#include  <stdlib.h>



int main(int argc, char *argv)

{

  int      nn, amount, loops, bytes;

  char    **pmem, x;



  amount = loops = 0;



  printf("enter allocation amount: ");              //  bytes to allocate each loop

  nn = scanf("%d",&amount);

  if (nn != 1) return 0;

  if (amount < 1) return 0;

  

  printf("enter allocation loop count: ");          //  number of loops

  nn = scanf("%d",&loops);

  if (nn != 1) return 0;

  if (loops < 1) return 0;

      

  while (1)

  {

      printf("return to repeat, x to exit \n");

      scanf("%c",&x);

      if (x == 'x') return 0;



      printf("allocating %d blocks of size %d ... \n",loops,amount);

      

      pmem = (char **) malloc(loops * sizeof(char *));

      if (! pmem) {

        printf("malloc failure \n");

        return 0;

      }

      

      for (nn = 0; nn < loops; nn++)

      {

        bytes = 1 + rand() % (2 * amount);          //  randomized amount

///      bytes = 1 + amount;                          //  fixed amount

        pmem[nn] = malloc(bytes);

        if (! pmem[nn]) {

            printf("malloc failure \n");

            return 0;

        }

      }



      printf("freeing the allocated blocks ... \n");

      

      for (nn = 0; nn < loops; nn++)

        free(pmem[nn]);

      

      free(pmem);

  }

}

Quote:

Originally Posted by kornelix (Post 4051765)

I believe I have found a serious flaw in memory management

You could call that a "flaw". It is not a bug. It operates as designed.

Quote:

Note that if the program is waiting for input, there is no memory allocated.

False.

The important point you seem to be missing is that two level of memory allocation are involved.

1) The malloc system inside your process requests memory from the OS and sometimes (but not typically) frees that memory.

2) The rest of your program allocates memory from malloc and frees that memory back to malloc.

Malloc is not normally supposed to release memory back to the OS when the program releases memory back to malloc. If you release a single very large chunk, malloc can easily release that back to the OS and normally would. If you release every small chunk that malloc had allocated from the OS as some very large chunk, then malloc could release that large chunk, but normally doesn't.

johnsfine,

Thanks for the reply. What you say may be right, but if it is then it is still a serious flaw, because the released memory is not available to other processes. I verified this in my test runs. If process A has released all its memory (back to malloc as you say), then the memory is still not available to process B and the whole system performance is impacted negatively. It seems absurd that one process could gobble up all of memory, release it, and the memory is not available to other processes. If this is deliberate in the design, then it must be a bad design.

Is there a way to release memory for real? Obviously free() does not do this, if your view is correct.

regards

More for johnsfine

The version of the test program that uses small fixed allocations instead of small random allocations works as expected: the released memory is immediately available to other processes. Why the difference?

Uncomment the line of code that is commented-out to verify this.

regards

More for johnsfine

Note also that if the test program is cycled many times, it will sometimes hold the memory and sometimes release the memory. The older kernel I tested holds the memory only for the first cycle, and then releases it immediately for all following cycles. Why the difference?

I am not convinced this is OK.

I don't believe that behavior could depend on the version of the Linux kernel.

I would expect the behavior to depend on the version of malloc inside your process.

I think ordinary programs get malloc at load time from /lib/libc.so

You can build a program statically, so it gets malloc from libc.a when built rather than from libc.so when run. You could then move the binary to a different version of Linux to verify that the behavior is determined by the version of malloc, not the version of the kernel.

You can download a better version of malloc, such as ptmalloc3, and include that when building your programs. I don't know if ptmalloc3 is better or worse for the specific behavior you are testing. I think it is generally better than the official GNU malloc, and especially better when you release and reallocate a very large number of slightly different large object sizes (triggering horrible performance because of a design flaw in ordinary malloc).

Thanks again. I will test with fedora and with ptmalloc3 to see if that makes a difference.

If standard malloc() has such a problem, then it should be fixed since this is what most apps will use by default. Such subtle, hidden performance problems are the worst kind.

I looked for a way to force the memory to be released. The only thing I found was setrlimit() which can be used to reduce the address space of a process. I tried it and this had no effect.
Perhaps this function is irrelevant for malloc() memory (mmap() memory).

My description of the test program behavior was not exactly correct.
The behavior is as follows:

When randomized allocations are used:
1st cycle: memory is (wrongly) retained after the free() calls.
2nd cycle: memory is released. Process memory is back at the initial level.
subsequent cycles: memory is alternately retained and released.

When fixed allocations are used, the memory is always released.