Why the stack size limit?

hydraMax · 05-01-2011, 04:28 AM

Why is it in Linux that there is a stack size set by default? And why is it so small? (My system is set to 8192 kbytes.) And why is there a default limit on the stack size when the max memory and virtual memory size are, by default, unlimited? (Aren't they both fed from the same place ultimately?)

Reason I ask: I want to use recursive functions in my programming a lot more. Problem is, if the language (or implementation) doesn't happen to support tail-call recursion, then I can be pretty well certain that the first huge problem that gets thrown at my function is going to kill my program because the stack size limit is going to be quickly reached. Obviously, I can change the stack size limit for my own computers, but it doesn't feel so great knowing that most of the people who copy and execute my code will have probably have overlooked this.

Anyway, does anyone know: is this small default stack size limit just one of those historical artifacts, or is there some technical reason for it?

Hidden Windshield · 05-01-2011, 04:29 PM

There has to be a set stack size, because the stack is stored contiguously in memory, and if it grows too big, it will start overwriting the heap. Increasing the stack size would involve moving the heap, which would invalidate every single heap pointer in the program (which, typically, would be the vast majority of pointers).

As for why it's so small, it's because the individual stack frames are small. For instance, on my system, gcc only uses 32 bytes per stack frame (plus whatever variables I have). I wrote a quick C program to test, and I got more than 300,000 recursions before it finally segfaulted.

My advice would be to go ahead and write your program and ignore the stack size. If you do run in to the stack size limit during testing, you can use the pthreads library to create a thread with a larger stack size, and do all your recursion in the thread.

hydraMax · 05-01-2011, 07:55 PM

Quote:

Originally Posted by Hidden Windshield

There has to be a set stack size, because the stack is stored contiguously in memory, and if it grows too big, it will start overwriting the heap. Increasing the stack size would involve moving the heap, which would invalidate every single heap pointer in the program (which, typically, would be the vast majority of pointers).

As for why it's so small, it's because the individual stack frames are small. For instance, on my system, gcc only uses 32 bytes per stack frame (plus whatever variables I have). I wrote a quick C program to test, and I got more than 300,000 recursions before it finally segfaulted.

My advice would be to go ahead and write your program and ignore the stack size. If you do run in to the stack size limit during testing, you can use the pthreads library to create a thread with a larger stack size, and do all your recursion in the thread.

Thank you for helpful response. However, some things are not quite clear to me: So, if I set stack limit x, does that meant x amount of memory is reserved for the stack of each process? If this is the case, what happens when I set the stack limit to unlimited? (My ulimit command allows me to do this.)

Code:

enigma ~ # ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 16002
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 16002
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
enigma ~ # ulimit -s unlimited
enigma ~ # ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 16002
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) unlimited
cpu time               (seconds, -t) unlimited
max user processes              (-u) 16002
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
enigma ~ #

I tried this with a small C program implementing a slow recursive function: so far it has consumed 300M of memory and presumably will continue until I run out.

So are you saying that the stack size for a process is immutably fixed to an actual number (which doesn't seem to fit the above evidence) or that there is actually an internal stack limit but that it is adjusted when necessary (during runtime)?

johnsfine · 05-02-2011, 03:08 AM

Quote:

Originally Posted by hydraMax

why is there a default limit on the stack size when the max memory and virtual memory size are, by default, unlimited?

In a 32 bit system, user space virtual memory is more fundamentally limited to 3GB. Setting ulimit above 3GB or unlimited can't get around that constraint.

Quote:

Originally Posted by Hidden Windshield

it's because the individual stack frames are small. For instance, on my system, gcc only uses 32 bytes per stack frame (plus whatever variables I have). I wrote a quick C program to test, and I got more than 300,000 recursions before it finally segfaulted.

One might have quite a lot of local variables and/or recurse deeper than 300,000. I frequently need a larger stack limit.

Quote:

Originally Posted by Hidden Windshield

My advice would be to go ahead and write your program and ignore the stack size. If you do run in to the stack size limit during testing, you can use the pthreads library to create a thread with a larger stack size, and do all your recursion in the thread.

Yuck!
If your program is expected to use a lot of stack, provide a safe margin up front. Don't trust testing to hit the cases that real use wll hit.
I also dislike that pthreads idea.

Quote:

Originally Posted by hydraMax

So, if I set stack limit x, does that meant x amount of memory is reserved for the stack of each process?

Each process has its own virtual memory space. A larger stack limit reserves virtual memory, not physical.

Quote:

If this is the case, what happens when I set the stack limit to unlimited?

I don't know.

Hidden Windshield · 05-02-2011, 08:07 PM

Quote:

Originally Posted by johnsfine

One might have quite a lot of local variables and/or recurse deeper than 300,000. I frequently need a larger stack limit.

I've never hit the stack size limit unless I had an infinite recursion bug (or I was trying to write a naïve version of the Ackermann function). Of course, most of my projects are fairly small, for myself, which might have something to do with it.

Quote:

Originally Posted by johnsfine

Yuck!
If your program is expected to use a lot of stack, provide a safe margin up front. Don't trust testing to hit the cases that real use wll hit.
I also dislike that pthreads idea.

The question is, how do you increase the margin? Have your program run ulimit, then re-exec itself? AFAIK, there's no way to change the stack size of a program once it's started, and no way for it to request more stack before it starts. (I suppose you could put "you must have x bytes of stack for this program" in the manual, but then your end users wouldn't read the manual.

)
And I didn't like the pthreads idea either. I just couldn't think of another way of getting a bigger stack on an end-user's computer.

Quote:

Originally Posted by johnsfine

Quote:

If this is the case, what happens when I set the stack limit to unlimited?

I don't know.

Ok, I guess it's been a while since I took computer architecture in college.

I had no idea that it was possible to set an unlimited stack size. As I said earlier, the stack limit basically keeps the stack from overwriting the heap. It also keeps an infinite recursion from eating all your memory and crashing your system. I presume that setting an unlimited stack size just removes that protection.

Of course, how you're going to convince your end users to do that is another problem.