Our application is moving from SLES10 SP2 to SLES11 SP1. When running stability test for 4 days, VIRT of some processes can jump 64M sometimes. But Res doesn't change. I am not sure whether it is memory leak.
When I check by pmap, I find that all of increased 64M are [anon]. I can confirm our application will not alloc 64M memory one time.

Pmap output when process is starting:

START SIZE RSS PSS DIRTY SWAP PERM MAPPING
0000000000400000 2548K 1112K 1112K 0K 0K r-xp /opt/***
000000000087c000 84K 52K 52K 24K 0K r--p /opt/***
0000000000891000 304K 24K 24K 12K 0K rw-p /opt/***
00000000008dd000 908K 880K 880K 880K 0K rw-p [heap]
00007f006196a000 4K 0K 0K 0K 0K ---p [anon]
00007f006196b000 8192K 12K 12K 12K 0K rwxp [anon]
00007f006216b000 4K 0K 0K 0K 0K ---p [anon]
00007f006216c000 8192K 8K 8K 8K 0K rwxp [anon]
00007f006296c000 4K 0K 0K 0K 0K ---p [anon]
00007f006296d000 8192K 8K 8K 8K 0K rwxp [anon]
00007f006316d000 4K 0K 0K 0K 0K ---p [anon]
.....................

pmap output when process is running for some time:

START SIZE RSS PSS DIRTY SWAP PERM MAPPING
0000000000400000 2548K 1112K 1112K 0K 0K r-xp /opt/***
000000000087c000 84K 52K 52K 24K 0K r--p /opt/***
0000000000891000 304K 24K 24K 12K 0K rw-p /opt/***
00000000008dd000 908K 880K 880K 880K 0K rw-p [heap]
00007f0054000000 132K 24K 24K 24K 0K rw-p [anon]
00007f0054021000 65404K 0K 0K 0K 0K ---p [anon]
00007f005c000000 132K 44K 44K 44K 0K rw-p [anon]
00007f005c021000 65404K 0K 0K 0K 0K ---p [anon]
00007f006196a000 4K 0K 0K 0K 0K ---p [anon]
00007f006196b000 8192K 12K 12K 12K 0K rwxp [anon]
00007f006216b000 4K 0K 0K 0K 0K ---p [anon]
00007f006216c000 8192K 8K 8K 8K 0K rwxp [anon]
00007f006296c000 4K 0K 0K 0K 0K ---p [anon]


It seems that some 65404 occurs

Allocation of anonymous memory is a two level process, and I think that is what is confusing you (and I think your application is running normally and what you have seen does not indicate a memory leak).

Library code inside your process manages a pool of free anonymous memory that has already been "committed" by Linux. When that pool doesn't have a place for the next requested chunk, the code managing the pool asks Linux to commit a large amount of additional memory, even if that next chunk is small.

That large amount of memory is just committed. It doesn't really exist. It takes up address space in your process's virtual address range. But it doesn't take any physical ram nor swap space.

Physical ram is allocated (in 4096 byte pages) only when part of that 4096 byte page is actually used.

1 members found this post helpful.

I have several more questions:
1. When starting the process, there are a lot of 8192K anonymous memory with "rwxp" perm which is seprated by 4k anonymous memory with "---p". Why does 65404k anonymous have "---p" perm? What does it mean?
2. You mentioned "When that pool doesn't have a place for the next requested chunk, the code managing the pool asks Linux to commit a large amount of additional memory". But when I run this process for 4 days, there are four 65404k anonymous occuring. If the first three 65404k anonymous memorys are used by process, this should be memory leak. But if the first three 65404k anonymous memorys are not used, why does the code managing the pool ask Linux to commit the forth 65404k anonymous memory?
3. Who and when will trigger the code managing the pool to ask linux to commit memory? The process? malloc function?

I'm not 100% sure, but I think there is more than one memory pool in that process. Maybe malloc is keeping separate pools per thread to reduce locking requirements.

Look at a typical pair of your lines, such as
That indicates one of your pools did fill and the library code asked for one mapping of (65404+132)K more memory for that pool. Then six 4K pages of that were actually used and physical memory allocated.

I don't quite understand the distinction (rw-p vs. ---p) between the untouched pages within the first 132K and the untouched pages in the remaining 65404K. Only 24K is allocated ram (correctly rw-p). All the rest (both inside and outside the first 132K) is just committed and wouldn't be allocated until used. So it should all be ---p. In those two lines I quoted, there ought to be only 24K of rw-p and the rest of the 132+65404 should be ---p.

I don't know whether there is a true difference in the states of the unallocated pages or whether it is just some data reduction effect in the way pmap reports things. But I'm pretty sure any difference (between the two kinds of committed unallocated pages) is not significant for your application

None of those 65404K areas are used (yet). They are committed, not allocated. I think Linux commits the fourth of those before using the first because it has four pools inside one process (maybe four active threads).

But that 24K mentioned earlier does represent memory used later in execution when part of its pool that was enough at the beginning is no longer enough. I don't know enough about your application to estimate whether that represents a small memory leak (apparently in each of four different pools).

Let's look at the following two lines. I guess when the process create a new thread, then two more lines can occur. Tomorrow I will check it.But if my guess is correct, why does OS split two parts(4K and 8192K) for one thread? 4K is used for protecting thread overlapping ?

I'm pretty sure those two lines are a thread stack (not a thread memory pool).

A no-access page is needed before each stack.

When application is call for malloc, malloc attempt to get memory from available memory pool and if none available call brk(2) to increase memory available for the application.
On the other hand, free just mnark memory as freed it's never return memory to the OS.

So it's quite normal that some time after the application startuo it's VIRT will increase.
But on some point application's VIRT should stabilized. If not - it's a memory leak.

You may run your application under valgrind to have a closer look on what is going with a memory.