Threads - Max Limit

Hello,

This error has been a nightmare for me. I am not able to deploy my server yet.

After 2 days, it gives up accepting new connections and then dies. The error that I get is a common error

java.lang.OutOfMemoryError: unable to create new native

I did my home work . My ulimits are

core file size (blocks, -c) 1000000
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 4096
pipe size (512 bytes, -p) 8
stack size (kbytes, -s) 8192 >> changed it to 1024 too
cpu time (seconds, -t) unlimited
max user processes (-u) 14335
virtual memory (kbytes, -v) unlimited


My test program stops after 600 threads

Starting Thread: 600
Starting Thread: 601
Starting Thread: 602
Starting Thread: 603
Starting Thread: 604
Starting Thread: 605
Starting Thread: 606
Starting Thread: 607
Starting Thread: 608
Starting Thread: 609
Exception in thread "main" java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:574)
at Test.startThreads(Test.java:17)
at Test.main(Test.java:29)


public static void startThreads(){

class TestThread extends Thread {
public void run(){
try{
while(true) sleep(1000);
}catch(Exception e){}
}
}

for(int i = 0;i<10000;i++){
try{
System.out.println("Starting Thread: " + i);
(new TestThread()).start();}
catch(Exception e)
{
e.printStackTrace();
e.getMessage();
System.exit(1);
}
}
}

I was actually able to run this command on my 1GB server and still get the same error.

java -Xms1000m -Xmx1000M Test

Running again gave me 400 threads only.

Here is the free memory at the time I ran this command.
free

total used free shared buffers cached
Mem: 1031644 989132 42512 0 186828 422104
-/+ buffers/cache: 380200 651444
Swap: 2040244 94472 1945772

java version "1.5.0_01"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_01-b08)
Java HotSpot(TM) Client VM (build 1.5.0_01-b08, mixed mode, sharing)


With the new 1.5
/usr/local/java/jdk1.5.0_02/bin/java -version
java version "1.5.0_02"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_02-b09)
Java HotSpot(TM) Client VM (build 1.5.0_02-b09, mixed mode)

it gives 879 threads - a little better but still cannot reach more.

uname -a
javaserver.*.com 2.4.27 #2 SMP Tue Aug 10 16:40:59 CDT 2004 i686 i686 i386 GNU/Linux

ls -l /lib/libc.so.*
lrwxrwxrwx 1 root root 13 Aug 10 2004 /lib/libc.so.6 -> libc-2.3.2.so*

Can it be my glibc? Will other JVMs do any good?

In c using the famous pthread_create, it goes till 1532 threads and stops after that.

How can I get this to create [lot] more threads?

Thanks

i personally think it's glibc
your version 2.3.2 has the ceiling set at
#define PTHREAD_THREADS_MAX 16384
take a look at /proc/sys/kernel/threads-max so that's not it but The more you reduce STACK_SIZE in LinuxThreads the more threads you can safely create.
your version of glibc has in linuxthreads/internals.h
/* The initial size of the thread stack. Must be a multiple of PAGE_SIZE. */
#ifndef INITIAL_STACK_SIZE
#define INITIAL_STACK_SIZE (4 * PAGE_SIZE)
#endif
if i am not mistaken PAGE_SIZE = 4096
so if you feel like going through the trouble of recompiling glibc the real answer might be to reduce that number to say
#define INITIAL_STACK_SIZE (PAGE_SIZE)
that is still 4Mb and that still seems like too much 4*4 mb the original seems like WAY too much if you want to make alot of threads doesn't it
i wonder if you could do (warning i havn't tried this)
#define INITIAL_STACK_SIZE (PAGE_SIZE/2) or even (PAGE_SIZE/4)

remember your version of glibc had linuxthreads as a seperate package you opened inside glibc
and you had to put --enable-add-ons as a config option

i guess you are not up against kernel file descriptor limits because your test is just a matter of thread creation not sockets but look at /proc/sys/fs/file-max

you could try ibm jre ?
http://www-128.ibm.com/developerwork...jdk/index.html

also try in the shell you are using
export LD_ASSUME_KERNEL=2.4.0
and
export LD_ASSUME_KERNEL=2.4.1
these should inhibit the use of NPTL threading
http://java.sun.com/developer/techni...dLinux/RedHat/
and at least allow you to try it with the older threading codes

good luck

Thanks for your reply foo_bar.

I realized that 2.4 depended on glibc. I loaded 2.6.11.7 and presume that it does not depend on glibc limitation.

The pthread test did not create more than 1533 threads (similar to last one) BUT java created only 53 threads and bailed out. The ulimit -s was 1k. It does not matter actually what limit I put there.

I suspect if there is a limit with ssh login. Since, it is a hosted machine, I can only ssh to it.

The more I think, I think it is time to move to windows (I know it has poor performance but atleast it works )

ulimit -a
core file size (blocks, -c) 1000000
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 4096
pipe size (512 bytes, -p) 8
stack size (kbytes, -s) 1024
cpu time (seconds, -t) unlimited
max user processes (-u) unlimited
virtual memory (kbytes, -v) unlimited

uname
2.6.11.7 #1 Sun Apr 10 23:57:56 CDT 2005 i686 i686 i386 GNU/Linux

I cannot give IBM JRE a shot as we use latest nio possible only in Tiger - 5.0. I will try blackdown. I do not think it would help in any way.

I am trying to ssh to machine, do you think it would be the problem.

Thanks. Any help would be appreciated.

Originally posted by socratesinus Thanks for your reply foo_bar. I realized that 2.4 depended on glibc. I loaded 2.6.11.7 and presume that it does not depend on glibc limitation.

if glibc says each thread takes up a certain amount of stack space then each thread takes up that amount of stack space -- the kernel doesn't matter.
the error after all is OutOfMemoryError

Thanks for your reply again.

I changed the stack using uname -s to 1k and I still get this problem.

It might be my glibc and chaning config of that might be a huge pain.

Looks like it's the kernel's threads-max setting that is limiting you. To find out what the current limit is: sysctl -a | grep threads

You can change the value on-the-fly with: echo <number> > /proc/sys/kernel/threads-max
So if you wanted to change it to 50000, for example: echo 50000 > /proc/sys/kernel/threads-max

If this works for you, make the setting permanent by adding "kernel.threads-max = 50000" in your /etc/sysctl.conf file and then issue the "sysctl -p" command to load it.

Let me know how it goes.