LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices


Reply
  Search this Thread
Old 11-23-2017, 08:19 PM   #1
miketwenty1
LQ Newbie
 
Registered: Nov 2017
Posts: 2

Rep: Reputation: Disabled
bash: fork: retry: Resource temporarily unavailable - Limit issue?


I've looked through the other posts and feel like this question has been answered and doesn't apply to my situation or was asked and the thread has gone stale.

Ubuntu OS 16.04. Running Java Process.
Code:
java -version
openjdk version "1.8.0_151"
OpenJDK Runtime Environment (build 1.8.0_151-8u151-b12-0ubuntu0.16.04.2-b12)
OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode)
(not that it matters because I'm running into other limit issues I believe. Doesn't look like anything to do with java HEAP Xms Xmx Xss type stuff. )

I'm running a process that uses many threads i want 10,000's of threads potentially.
My process stops making threads around 12,000, I'm very confused about this. (the same exact number every time)

When it ramps up to ~12,000 threads I will also have issues running commands in the shell

my normal user: (while process is running)
Code:
-bash: fork: retry: Resource temporarily unavailable
-bash: fork: retry: Resource temporarily unavailable
-bash: fork: retry: Resource temporarily unavailable
-bash: fork: retry: Resource temporarily unavailable
-bash: fork: Resource temporarily unavailable
as root: (while process is running)
Code:
-bash: fork: retry: No child processes
(so makes me think this is a system wide thing? not a specific user issue? since both root and my normal user have shell issues)

Here's some information which might be able to help. I'm confused on what limit I'm hitting exactly.

Code:
`free -m`
              total        used        free      shared  buff/cache   available
Mem:          60395         306       59195           8         892       59349
Swap:             0           0           0
Code:
`ulimit -a`
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 241522
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 500000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 256
cpu time               (seconds, -t) unlimited
max user processes              (-u) 1000000
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
(I've messed with stack size making it 600,000 to 256 doesn't seem to affect anything)

Code:
`sysctl kernel.pid_max`
kernel.pid_max = 4194303
Code:
`cat /proc/sys/kernel/threads-max`
483045
Code:
`cat /proc/sys/fs/file-nr`
832	0	6168013
Code:
ps -T -p <pid>
(will give me how many threads are being used which stops around 12000)

I don't see anything interesting in /var/log/syslog

So, what the heck is going on exactly. What limit am I hitting? I don't feel like I'm hitting a process count limit as the ulimit -a results show the limit is very high.. the thread max count is very high. Feeling a little confused here.



Edit:
Tried this java application out on multiple cloud hardware configurations.. all yielding the same result. from 1cpu - 6gb of ram to 16cpu - 60gb of ram and several hardware configs in between.


Edit2: SOLVED
Figured out the issue
https://stackoverflow.com/questions/...57379#43157379

/etc/systemd/logind.conf UserTasksMax needs to be set higher than ~12000
Not sure how this is read in, but a reboot of the server activated this number

I think it's also worth noting that
sys.kernel.threads-max
sys.kernel.pid_max
sys.vm.max_map_count
Could probably be limitations if I was trying to go much higher with thread count, and these might be my next limiters if I decided to go bigger with thread count.

Edit3:
Instead of rebooting you can do these commands
Code:
sudo systemctl daemon-reload
sudo systemctl daemon-reexec
sudo service systemd-logind restart
Then restart your shell or.. exit.. and ssh back in

Last edited by miketwenty1; 11-24-2017 at 06:20 PM. Reason: additional information
 
Old 11-25-2017, 03:32 AM   #2
ondoho
LQ Addict
 
Registered: Dec 2013
Posts: 13,668
Blog Entries: 9

Rep: Reputation: 3768Reputation: 3768Reputation: 3768Reputation: 3768Reputation: 3768Reputation: 3768Reputation: 3768Reputation: 3768Reputation: 3768Reputation: 3768Reputation: 3768
thanks for posting your solution.
if you edit your post, it will not show up as a new post in other member's feeds, and it can also seriously disrupt the flow of communication.
it would've been better to post the solution as a separate post/reply.
that way your thread also loses zero reply status.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
-bash: fork: retry: Resource temporarily unavailable urello Linux - General 3 12-15-2015 06:24 AM
[SOLVED] "bash: fork: Resource temporarily unavailable" now occurring, did not before RandomTroll Linux - Software 3 04-18-2013 07:56 PM
Default bash: fork: Resource temporarily unavailable :SSH LittleMaster Linux - Server 3 03-13-2013 09:32 PM
fork: resource temporarily unavailable ??? str8edge Linux - General 3 06-21-2011 09:00 AM
./configure: fork: Resource temporarily unavailable doodlebug Linux - Server 3 08-23-2007 03:34 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Server

All times are GMT -5. The time now is 06:25 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration