I've looked through the other posts and feel like this question has been answered and doesn't apply to my situation or was asked and the thread has gone stale.
Ubuntu OS 16.04. Running Java Process.
Code:
java -version
openjdk version "1.8.0_151"
OpenJDK Runtime Environment (build 1.8.0_151-8u151-b12-0ubuntu0.16.04.2-b12)
OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode)
(not that it matters because I'm running into other limit issues I believe. Doesn't look like anything to do with java HEAP Xms Xmx Xss type stuff. )
I'm running a process that uses many threads i want 10,000's of threads potentially.
My process stops making threads around 12,000, I'm very confused about this. (the same exact number every time)
When it ramps up to ~12,000 threads I will also have issues running commands in the shell
my normal user: (while process is running)
Code:
-bash: fork: retry: Resource temporarily unavailable
-bash: fork: retry: Resource temporarily unavailable
-bash: fork: retry: Resource temporarily unavailable
-bash: fork: retry: Resource temporarily unavailable
-bash: fork: Resource temporarily unavailable
as root: (while process is running)
Code:
-bash: fork: retry: No child processes
(so makes me think this is a system wide thing? not a specific user issue? since both root and my normal user have shell issues)
Here's some information which might be able to help. I'm confused on what limit I'm hitting exactly.
Code:
`free -m`
total used free shared buff/cache available
Mem: 60395 306 59195 8 892 59349
Swap: 0 0 0
Code:
`ulimit -a`
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 241522
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 500000
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 256
cpu time (seconds, -t) unlimited
max user processes (-u) 1000000
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
(I've messed with stack size making it 600,000 to 256 doesn't seem to affect anything)
Code:
`sysctl kernel.pid_max`
kernel.pid_max = 4194303
Code:
`cat /proc/sys/kernel/threads-max`
483045
Code:
`cat /proc/sys/fs/file-nr`
832 0 6168013
(will give me how many threads are being used which stops around 12000)
I don't see anything interesting in /var/log/syslog
So, what the heck is going on exactly. What limit am I hitting? I don't feel like I'm hitting a process count limit as the ulimit -a results show the limit is very high.. the thread max count is very high. Feeling a little confused here.
Edit:
Tried this java application out on multiple cloud hardware configurations.. all yielding the same result. from 1cpu - 6gb of ram to 16cpu - 60gb of ram and several hardware configs in between.
Edit2: SOLVED
Figured out the issue
https://stackoverflow.com/questions/...57379#43157379
/etc/systemd/logind.conf UserTasksMax needs to be set higher than ~12000
Not sure how this is read in, but a reboot of the server activated this number
I think it's also worth noting that
sys.kernel.threads-max
sys.kernel.pid_max
sys.vm.max_map_count
Could probably be limitations if I was trying to go much higher with thread count, and these might be my next limiters if I decided to go bigger with thread count.
Edit3:
Instead of rebooting you can do these commands
Code:
sudo systemctl daemon-reload
sudo systemctl daemon-reexec
sudo service systemd-logind restart
Then restart your shell or.. exit.. and ssh back in
