-bash: fork: retry: Resource temporarily unavailable

urello · 12-15-2015, 05:50 AM

Hi.Occasionally my server cannot spawn any new process saying: "-bash: fork: retry: Resource temporarily unavailable"
I increased nproc and nofiles values in /etc/security/limits.conf to 65000, but problem remains.
While problems occurs:

Code:

root@lab7# sysctl fs.file-nr
fs.file-nr = 22336      0       9782150

Code:

root@lab7# ps -efL |wc -l 
9156

Code:

root@lab7# cat /etc/security/limits.conf
*               soft    nofile          65000
*               hard    nofile          65000
mongod          soft    nproc           65000
mongod          hard    nproc           65000

As i can see, values are far below the limit. So what could be the root cause of the problem? All 3rd party services are running under "mongod" user and the problem is limited to this user only. If get root shell before the problem occurs, i can work in it without problems.
OS: RHEL 6.6

berndbausch · 12-15-2015, 05:55 AM

Anything in the kernel buffer or /var/log/messages?

urello · 12-15-2015, 06:06 AM

Nothing useful in messages.
How can one query kernel buffer?

berndbausch · 12-15-2015, 06:24 AM

The dmesg command. Since it has a rather long output, you usually run it as dmesg | tail.
Usually, kernel messages are also written to the /var/log/messages file, so I doubt you will find much there. But it's worthwhile trying it.

Other ideas off the cuff:

The pam_limits module is not used or disabled.
The limits.conf file is overwritten by files in /etc/security/limits.d.
Do you have enough virtual memory?
Does another ulimit apply?
You can run ulimit -a to check the real limits of your shell.