process threads/stacks use up memory - which tools can detect?

My process starts up 200 threads (over time) and each one gets (defaultly) 10 MB of stack. This totals 2 GB just for stack space, and the process eventually dies.

I found this problem the hard way - by messing around with ulimit. I tried using vmastat, free, top, and looking at /proc/meminfo and /proc/slabinfo, but these tools don't seem to show that my address space (2 GB per process, right?) is being used up by the thread's stack. I looked at top output before and after the threads were started, and saw very little change. The other tools/data were inconclusive as well.

Any ideas concerning how a problem like this could be detected w/o having to guess at sizes? For example, figuring out 'ulimit -s unlimited' actually -reduced- the size of the allocated stack from 10 MB (the default on RH) to 2 MB took a while to detect.

How can I see this problem develop? Am I using the wrong tools, am I looking at the data wrong.. or the wrong data?

Thanks for any help

Steve

Originally Posted by sorcer My process starts up 200 threads (over time) and each one gets (defaultly) 10 MB of stack. This totals 2 GB just for stack space, and the process eventually dies. I found this problem the hard way - by messing around with ulimit. I tried using vmastat, free, top, and looking at /proc/meminfo and /proc/slabinfo, but these tools don't seem to show that my address space (2 GB per process, right?) is being used up by the thread's stack. I looked at top output before and after the threads were started, and saw very little change. The other tools/data were inconclusive as well. Any ideas concerning how a problem like this could be detected w/o having to guess at sizes? For example, figuring out 'ulimit -s unlimited' actually -reduced- the size of the allocated stack from 10 MB (the default on RH) to 2 MB took a while to detect. How can I see this problem develop? Am I using the wrong tools, or am I looking at the data wrong.. or the wrong data? Thanks for any help Steve

BUMP. I'm still getting, e.g., sig 11, and crashes all over. During problem periods, the process restarts and fails quickly. I'm pretty sure it is a resource problem, but I need proof. Any help is greatly appreciated. Thanks.

Steve.