What limits number of threads per system or per user?

I think I checked the obvious limits and none were low enough to explain the symptom.

Under an automated QA process, several of the programs being tested failed with the message:

OMP: Error #35: System unable to allocate necessary resources for the monitor thread:
OMP: System error #11: Resource temporarily unavailable
OMP: Hint: Try decreasing the number of threads in use simultaneously.

I don't have an exact count, but I'm pretty sure that user had between 100 and 200 threads active across about 40 processes. The rest of the system was very lightly loaded.

ulimit -a shows that user has a limit of 1024 "max user processes"

The system had 210GB ram free and only 41GB used (used includes buffers and cache) as well as 105GB swap free and less than 1GB used. So this is an absurdly underloaded system and any memory limits should not be relevant.

What other loads or limits should I investigate to understand that error?

I've hit the total open files limit (-n), which is a per-user limit of 1024 by default.

See:

Another possible scenario that can lead to an error is if the virtual address space limit is set too low. For example, if the virtual address space limit is 1 GiB and the thread stack size limit is set to 512 MiB, then the OpenMP run-time would try to allocate 512 MiB for each additional thread. With two threads one would have 1 GiB for the stacks only, and when the space for code, shared libraries, heap, etc. is added up, the virtual memory size would grow beyond 1 GiB and an error would occur: Set the virtual address space limit to 1 GiB and run with two additional threads with 512 MiB stacks (I have commented out the call to omp_set_num_threads()): $ ulimit -v 1048576 $ KMP_STACKSIZE=512m OMP_NUM_THREADS=3 ./a.out OMP: Error #34: System unable to allocate necessary resources for OMP thread: OMP: System error #11: Resource temporarily unavailable OMP: Hint: Try decreasing the value of OMP_NUM_THREADS. forrtl: error (76): Abort trap signal ... trace omitted ... zsh: abort (core dumped) OMP_NUM_THREADS=3 KMP_STACKSIZE=512m ./a.out In this case the OpenMP run-time library would fail to create a new thread and would notify you before it aborts program termination.

http://stackoverflow.com/questions/1...is-openmp-code

Thankyou for both those suggestions, but I had already considered those.

1) Each process has few open files and there are only about 40 of the processes. I actually did not figure out whether that open file limit of 1024 was per user or per process, but even if it was per user it would not be hit.

2) The virtual size is unlimited and the same processes run perfectly well when there are fewer of them, so no per process limit could be doing this. It must be a per user or per system limit, but I don't know WHAT limit.

First thing I would try is increase OMP_STACKSIZE.

I don't think this is about a thread limit, but rather stack or memory limits. Either way I don't know of any per user thread limit, but the system max is at:

cat /proc/sys/kernel/threads-max

I have gotten this error before when putting too many things on the stack with openmp. pthreads is not as conservative/restrictive with stack size. The same program written in both with seg fault with openmp but not with pthreads. Increasing OMP_STACKSIZE should fix it.

I'm pretty sure OMP_STACKSIZE could not be the problem, because the processes run fine when there are fewer of them and only fail when about 40 processes (1 to 32 threads per process, but on average few threads per process) are running at the same time. Any virtual memory or stack size related problem would be completely internal to each process and not affected by other processes.

It must be some per user or per system limit, not any per process limit.

I had already checked:

$ cat /proc/sys/kernel/threads-max 
4132206

That obviously is big enough.

The only others things I can think of:

1) Check OMP_THREAD_LIMIT, this is per program tho.
2) Check the RAM with memtest86
3) cgroups would be the only other way to limit resources besides the system method, but you would need to active and configure it for it to limit anything.

You might have more files open then you think. A way to tell (approximately) is:

lsof -u <username> |wc -l

I have 159 open files on a system just due to 6 xterm windows. root has 3000.

Originally Posted by smallpond You might have more files open then you think. A way to tell (approximately) is: lsof -u <username> |wc -l I have 159 open files on a system just due to 6 xterm windows. root has 3000.

Never thought to look at that - I now have 2322, root has 272. Very interesting data point!