Memory Leaks and Processes in Linux...

Sonny · 02-21-2002, 06:33 PM

Hi Guys,

This is my understanding of how memory allocation / deallocation works in Linux(and Unix). When I call "new" (as in C or C++) I get a piece of memory that belongs to my current process/application. When the process exits, the memory is returned to the operating system. If the application has memory leaks, that's okay because all the leaked memory is recovered when the process exits. Is that correct? Does it matter how the process exits? For example, if I call "kill" on it, does the memory not get recovered?

I've been doing some development and watching with growing concern as my memory and swap indicator keep going up and up and up....

Sonny.

Malicious · 02-21-2002, 09:40 PM

You are correct, and it does not matter how the program exits; all memory is returned to the os. One of the HARDEST things about c/c++ programming is making sure that you free all your allocations.

The devil is in the details...

crabboy · 02-21-2002, 10:13 PM

This post led of off on a half hour long quest to figure out who is the memory hog. I wrote a small program with a huge memory leek to prove that all the memory is returned to the OS. My program runs, chews up memory and swap, but the numbers never added up after the program exited.
After running the program a few times, my swap rose up to about 60M used and I had about 50 of 128 free memory. As the time passed my 50 free dropped but the swap never did. My questions were answered when I issued the following command:

Code:

swapoff -a

Needless to say the kenel wasn't very happy with that command, but surprisingly it dealt with it. My mpg music jittered, my windows froze, but about 15 seconds later everything was up. My memory showed over 60M free, and the machine was running fine with no swap. I've started the swap about 10 minutes ago, and it still shows 0.
My guess is that when programs release memory to the kernel, he holds on to it until either someone else requests it, or he has to free it. Like when some idiot does a swapoff on a running system.

Use this code with caution!!! It is memory leek central.

Code:

/* memhog.c */
#include <unistd.h>

main()
{
   int i = 0;
   char * pLeekCity = 0;

   for ( i = 0; i < 100000; i++ )
   {
      pLeekCity = (char *) malloc( 1 * 1024 * sizeof( char ));
      if ( pLeekCity == NULL )
         perror( "malloc failed" );
      printf( "Allocated  %d\n", i);
   }
   printf( "Finished...\n");
   sleep(5);
}

Malicious · 02-21-2002, 10:39 PM

Try this twist in your leaker. Allocate those 100,000 chunks, look at the memory usage. Free the chunks without exiting the program and look again. Any difference? Mem still allocated to the program? Can you make the program give back any of the mem while it is still running?

crabboy · 02-21-2002, 11:06 PM

I can only free the last chunk allocated from my leaker.

Malicious · 02-21-2002, 11:19 PM

Code:

/* memhog.c */
#include <unistd.h>

main()
{
   int i = 0;
   char * pLeekCity[100000];

   for ( i = 0; i < 100000; i++ )
   {
      pLeekCity[i] = (char *) malloc( 1 * 1024 * sizeof( char ));
      if ( pLeekCity[i] == NULL )
         perror( "malloc failed" );
      printf( "Allocated  %d\n", i);
   }
   printf( "Finished Allocating...\n");
   sleep(30);
   printf( "Starting Free... \n");
   for ( i = 0; i < 100000; i++ )
   {
      free pLeekCity[i];
      pLeekCity[i] = 0;
   }
   printf( "Finished Freeing...\n");
   sleep(30);
   printf( "Exiting...\n");
}

Do it this way...

Mik · 02-22-2002, 08:41 AM

I still haven't figured out all the details on how it works. But a while I ago I was having problems with programs that where leaking memory. And I also tried monitoring it's memory usage. I noticed that the memory usage of the application didn't go down after freeing the memory while the program is still running. At first I thought the memory was still being allocated. But then I tried running an application which allocates memory and then frees it and then does the same process again several times. Monitoring the process says it's always using the same amount of memory. So I guess the application will keep the memory but not as allocated. Probably more like cache and if it is really needed by something else then it will be passed back to other applications.
I guess a study of the kernel internals will reveal more.

Malicious · 02-22-2002, 08:59 AM

Nope, sad to say that once mem is allocated to a program, it is never returned to the os until the program terminates. Allocated but idle pages are the ones that wind up in swap when the real mem is needed by something else.

Sonny · 02-22-2002, 11:56 AM

Excellent discussion guys. Thanks for the insights, Malicious, crabboy, and Mik. A mystery solved

Sonny.

Malicious · 02-22-2002, 02:38 PM

Pay it forward...

cyent · 03-04-2002, 05:54 PM

First off there are layers between your program and the OS.

Libc being the most usual. malloc requests memory from the OS with the brk routine. Having got a largish chunk, it dispenses it via malloc(). When it is "free()'d" it goes back into malloc's pool of available memory and not given back to the os until the process exits.

When the OS runs out of physical memory, it first tries to get more by "forgetting" read only pages. Why bother writing things to disk when it is already on disk? Eg. Program executable files and libraries.

Say man mmap sometime. And then strace any program and see how often things get mmap'd.

If it still requires more physical memory it can swap writable portions of memory out to disk. Slow slow slow.

So you process exits, there is lots of physical memory now. Does the OS swap the stuff back in? No, it just leaves it there. If the corresponding processes actually ever need it it will swap them in when required. Thus after a memory hog has exited, swap remains full!

You notice this as everything else behaves like treacle until it has swapped back in.