Threads allocating 64M of anon space on 64-bit Linux

Hi!

I have started moving our multithreaded applications to 64-bit Linux (Fedora 11, kernel 2.6.30).

I am a little curious why every thread appears to allocate a 64 megabyte anon region. In total these regions occupy multiple GBs of address space, which if course isn't a problem in itself, but I'd like to know why it does this

Below is a partial dump of pmap -x output for the given process.
This represents the stack spaces for 6 threads together with the 64M regions for these threads.

Thanks for any input!

00007fc16405b000 65172 - - - ----- [ anon ]
00007fc168000000 944 - - - rw--- [ anon ]
00007fc1680ec000 64592 - - - ----- [ anon ]
00007fc16c000000 420 - - - rw--- [ anon ]
00007fc16c069000 65116 - - - ----- [ anon ]
00007fc170000000 496 - - - rw--- [ anon ]
00007fc17007c000 65040 - - - ----- [ anon ]
00007fc174000000 208 - - - rw--- [ anon ]
00007fc174034000 65328 - - - ----- [ anon ]
00007fc178000000 436 - - - rw--- [ anon ]
00007fc17806d000 65100 - - - ----- [ anon ]
00007fc17c000000 476 - - - rw--- [ anon ]

Originally Posted by movax222 This represents the stack spaces for 6 threads together with the 64M regions for these threads.

Notice how each RW chunk plus the following empty chunk adds to 65536.

That's the detail I don't understand, but I haven't experimented with it myself.

Each thread needs a stack, plus some unknown amount of empty space below the stack for that stack to grow into.

The system can't predict nor correct that stack growth, so it needs a large fixed allocation (I assume some parameter controls the amount).

So the chunk before each stack is the space allocated for growth of that stack.

So why is it that the fixed value 65536 the sum of each stack plus the space after it?

Are you sure those rw things are the stacks? Notice how they all start on 64MB boundaries. I'm not surprised they were allocated on such boundaries, and that causes the 65536 sum with the next, rather than previous, empty chunk. But stacks grow downward. That would mean none of the threads grew their stacks after launch of the thread. Is that reasonable for the test you ran?

Very interesting observations, johnsfine!

I have a stack size of 128K and a "guard size" of 4K, and I allocate no other per-thread heap data, so that's why I think the rw chunks are the stacks. It would be interesting to get confirmation if Linux only allocates stacks in 64M chunks on x86_64, or if something other is going on.