Memory allocation pattern - Linux

ArunkumarRavi · 07-31-2014, 11:43 PM

Hi johnsfine,sundialsvcs,genss and NevemTeve,
Yesterday i tested an sample try-catch block with a simple memory allocation program in the device where the UI process hangs.This device runs with only 64MB RAM with customized linux OS.My intention was to verify whether a normal try catch block will work when there is no memory on this device.Im listing the code as below.Assume this as program1.

Code:

#include "stdio.h"
#include "stdlib.h"
int main()
{
        int count;
        int *q[409600];
        printf("\nHeap Leak Starting at..");
        system("date");
        fflush(stdout);
        printf("\n, Count is located at ,%p,",&count);
        for(count=0;count<409600;count++)
        {
                printf("\nCount = ,%d,",count);
                int i=2;
                try
                {
                        fflush(stdout);
                        q[count] = new int[409600];
                        printf("New Success.Allocated %ld Bytes at %p.Difference = 0x%x,\n",(i*100000),q[count],(long)(&i) - (long)q[count]);
                        fflush(stdout);
                }
                catch(...)
                {
                                printf("\nException Caught...New Failed..No Memory Available.\n");
                                fflush(stdout);
                                exit(1);
                }
        }
}

Output of Program1:
Thu Jul 31 20:38:00 UTC 2014
Heap Leak Starting at..
, Count is located at ,0xbffffc48,
Count = ,0,New Success.Allocated 200000 Bytes at 0x402c0008.Difference = 0x7fbafc3c,

Count = ,1,New Success.Allocated 200000 Bytes at 0x40451008.Difference = 0x7fa1ec3c,

Count = ,2,New Success.Allocated 200000 Bytes at 0x405e2008.Difference = 0x7f88dc3c,

Count = ,3,New Success.Allocated 200000 Bytes at 0x40773008.Difference = 0x7f6fcc3c,

........The count goes on........

Count = ,1954,New Success.Allocated 200000 Bytes at 0xbf854008.Difference = 0x61bc3c,

Count = ,1955,New Success.Allocated 200000 Bytes at 0xbf9e5008.Difference = 0x48ac3c,

Count = ,1956,New Success.Allocated 200000 Bytes at 0xbfb76008.Difference = 0x2f9c3c,

Count = ,1957,
Exception Caught...New Failed..No Memory Available.
Now this is an expected behaviour.Since no memory is there,Exception is throwed.
Then i modified the line "q[count] = new int[409600]" to "q[count] = new int[100000]".Lets assume this as program2.I recompiled the program and executed on the same device.

Output of Program2:
Thu Jul 31 20:39:43 UTC 2014
Heap Leak Starting at..
, Count is located at ,0xbffffc48,
Count = ,0,New Success.Allocated 200000 Bytes at 0x402c0008.Difference = 0x7fbafc3c,

Count = ,1,New Success.Allocated 200000 Bytes at 0x40322008.Difference = 0x7fb4dc3c,

Count = ,2,New Success.Allocated 200000 Bytes at 0x40384008.Difference = 0x7faebc3c,

Count = ,3,New Success.Allocated 200000 Bytes at 0x403e6008.Difference = 0x7fa89c3c,

............The count goes on.............

Count = ,5215,New Success.Allocated 200000 Bytes at 0x87d00018.Difference = 0x3816fc2c,

Count = ,5216,New Success.Allocated 200000 Bytes at 0x87d61aa0.Difference = 0x3810e1a4,

Count = ,5217,New Success.Allocated 200000 Bytes at 0x87e00018.Difference = 0x3806fc2c,

Count = ,5218,New Success.Allocated 200000 Bytes at 0x87e61aa0.Difference = 0x3800e1a4,

Count = ,5219,New Success.Allocated 200000 Bytes at 0x87f00018.Difference = 0x37f6fc2c,

Count = ,5220,

And at this point,the process gets hanged forever at the New operator.I could not access the terminal even if i run the process in background.

Now im confused on noticing this different behaviour of "new".Please note that in the exception throwing case im allocating a large size(409600) than the case where the process gets hanged(100000). Even this could be root cause for the Ui process hang issue which was reported So,a breakthrough for this sample program's behaviour would show the direction for breaking the ui process hang. Can you help me in clarifying this different behvaiour of New? Pls share your thoughts and some clues..

PS:Ignore the number "200000" as i have printed it wrongly.

johnsfine · 08-01-2014, 08:20 AM

There can be a big difference between a process running out of memory and the system running out of memory.

While I didn't believe you earlier, you now seem to be making a convincing case that the OOM killer in your tiny Linux system is not working, so that when the system runs out of memory, processes are hanging rather than aborting. Either way, you would not expect an exception to work and it doesn't.

If a process fails a memory allocation, then the exception mechanism works.

You should read about the various settings controlling "over commit" in Linux. Start with a google search for "vm.overcommit_memory". You may be able to adjust those to reliably get the memory allocation in a process to fail before the system is hung for lack of memory.

The reason the two are so different is that pages are not assigned actual physical ram until written. So your process can succeed in a memory allocation request without actually taking enough memory to satisfy the request. The "over commit" settings control the point at which the kernel will refuse an additional memory allocation triggering the exception you want to catch. You should notice your first example allocated about 800MB before hitting the exception and your second example allocated about 500MB before hanging. So you can see the kernel is permitting an absurd level of over commit in a 64MB system.

In addition, the underlying allocation process has two different methods based on the size of the allocation. For small allocations, it will ask the kernel for a super big chunk, then subdivide that chunk as the program requests more. If the "over commit" settings let the super big chunk request succeed even though not that much ram was really available, later subdividing of that chunk can cause the system to run out of memory, while the process cannot recognize that to throw an exception on the subdivide step.

For very big allocations, malloc requests each chunk directly from the kernel. That gives it a better chance of throwing an exception as the system runs out of memory.

I'm not certain of over commit settings (whether you can fine tune to exactly the behavior you need). They can be made much tighter (not let you commit 500MB in a 64MB system). So when you make a small memory request, malloc will try to get a super large chunk from the kernel and immediately fail. Then malloc will fall back and select a reasonable size large chunk from the kernel and start subdividing that. As you request more small chunks, malloc will subdivide that first large chunk, soon exhausting it, then malloc will request more from the kernel, which will refuse that request and you will get the exception you expect.

Edit:
This is pretty much the same information JPollard was trying to give you in your other thread:
http://www.linuxquestions.org/questi...6/#post5210694

I didn't notice that thread until after I wrote this post.