Threads - Max Limit
Hello,
This error has been a nightmare for me. I am not able to deploy my server yet. After 2 days, it gives up accepting new connections and then dies. The error that I get is a common error java.lang.OutOfMemoryError: unable to create new native I did my home work . My ulimits are core file size (blocks, -c) 1000000 data seg size (kbytes, -d) unlimited file size (blocks, -f) unlimited max locked memory (kbytes, -l) unlimited max memory size (kbytes, -m) unlimited open files (-n) 4096 pipe size (512 bytes, -p) 8 stack size (kbytes, -s) 8192 >> changed it to 1024 too :) cpu time (seconds, -t) unlimited max user processes (-u) 14335 virtual memory (kbytes, -v) unlimited My test program stops after 600 threads Starting Thread: 600 Starting Thread: 601 Starting Thread: 602 Starting Thread: 603 Starting Thread: 604 Starting Thread: 605 Starting Thread: 606 Starting Thread: 607 Starting Thread: 608 Starting Thread: 609 Exception in thread "main" java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Thread.java:574) at Test.startThreads(Test.java:17) at Test.main(Test.java:29) public static void startThreads(){ class TestThread extends Thread { public void run(){ try{ while(true) sleep(1000); }catch(Exception e){} } } for(int i = 0;i<10000;i++){ try{ System.out.println("Starting Thread: " + i); (new TestThread()).start();} catch(Exception e) { e.printStackTrace(); e.getMessage(); System.exit(1); } } } I was actually able to run this command on my 1GB server and still get the same error. java -Xms1000m -Xmx1000M Test Running again gave me 400 threads only. Here is the free memory at the time I ran this command. free total used free shared buffers cached Mem: 1031644 989132 42512 0 186828 422104 -/+ buffers/cache: 380200 651444 Swap: 2040244 94472 1945772 java version "1.5.0_01" Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_01-b08) Java HotSpot(TM) Client VM (build 1.5.0_01-b08, mixed mode, sharing) With the new 1.5 /usr/local/java/jdk1.5.0_02/bin/java -version java version "1.5.0_02" Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_02-b09) Java HotSpot(TM) Client VM (build 1.5.0_02-b09, mixed mode) it gives 879 threads - a little better but still cannot reach more. uname -a javaserver.*.com 2.4.27 #2 SMP Tue Aug 10 16:40:59 CDT 2004 i686 i686 i386 GNU/Linux ls -l /lib/libc.so.* lrwxrwxrwx 1 root root 13 Aug 10 2004 /lib/libc.so.6 -> libc-2.3.2.so* Can it be my glibc? Will other JVMs do any good? In c using the famous pthread_create, it goes till 1532 threads and stops after that. How can I get this to create [lot] more threads? Thanks |
i personally think it's glibc
your version 2.3.2 has the ceiling set at #define PTHREAD_THREADS_MAX 16384 take a look at /proc/sys/kernel/threads-max so that's not it but The more you reduce STACK_SIZE in LinuxThreads the more threads you can safely create. your version of glibc has in linuxthreads/internals.h /* The initial size of the thread stack. Must be a multiple of PAGE_SIZE. */ #ifndef INITIAL_STACK_SIZE #define INITIAL_STACK_SIZE (4 * PAGE_SIZE) #endif if i am not mistaken PAGE_SIZE = 4096 so if you feel like going through the trouble of recompiling glibc the real answer might be to reduce that number to say #define INITIAL_STACK_SIZE (PAGE_SIZE) that is still 4Mb and that still seems like too much 4*4 mb the original seems like WAY too much if you want to make alot of threads doesn't it i wonder if you could do (warning i havn't tried this) #define INITIAL_STACK_SIZE (PAGE_SIZE/2) or even (PAGE_SIZE/4) remember your version of glibc had linuxthreads as a seperate package you opened inside glibc and you had to put --enable-add-ons as a config option i guess you are not up against kernel file descriptor limits because your test is just a matter of thread creation not sockets but look at /proc/sys/fs/file-max you could try ibm jre ? http://www-128.ibm.com/developerwork...jdk/index.html also try in the shell you are using export LD_ASSUME_KERNEL=2.4.0 and export LD_ASSUME_KERNEL=2.4.1 these should inhibit the use of NPTL threading http://java.sun.com/developer/techni...dLinux/RedHat/ and at least allow you to try it with the older threading codes good luck |
Thanks for your reply foo_bar.
I realized that 2.4 depended on glibc. I loaded 2.6.11.7 and presume that it does not depend on glibc limitation. The pthread test did not create more than 1533 threads (similar to last one) BUT java created only 53 threads and bailed out. The ulimit -s was 1k. It does not matter actually what limit I put there. I suspect if there is a limit with ssh login. Since, it is a hosted machine, I can only ssh to it. The more I think, I think it is time to move to windows (I know it has poor performance but atleast it works :( ) ulimit -a core file size (blocks, -c) 1000000 data seg size (kbytes, -d) unlimited file size (blocks, -f) unlimited max locked memory (kbytes, -l) unlimited max memory size (kbytes, -m) unlimited open files (-n) 4096 pipe size (512 bytes, -p) 8 stack size (kbytes, -s) 1024 cpu time (seconds, -t) unlimited max user processes (-u) unlimited virtual memory (kbytes, -v) unlimited uname 2.6.11.7 #1 Sun Apr 10 23:57:56 CDT 2005 i686 i686 i386 GNU/Linux I cannot give IBM JRE a shot as we use latest nio possible only in Tiger - 5.0. I will try blackdown. I do not think it would help in any way. I am trying to ssh to machine, do you think it would be the problem. Thanks. Any help would be appreciated. |
Quote:
the error after all is OutOfMemoryError |
Thanks for your reply again.
I changed the stack using uname -s to 1k and I still get this problem. It might be my glibc and chaning config of that might be a huge pain. |
Looks like it's the kernel's threads-max setting that is limiting you. To find out what the current limit is: sysctl -a | grep threads
You can change the value on-the-fly with: echo <number> > /proc/sys/kernel/threads-max So if you wanted to change it to 50000, for example: echo 50000 > /proc/sys/kernel/threads-max If this works for you, make the setting permanent by adding "kernel.threads-max = 50000" in your /etc/sysctl.conf file and then issue the "sysctl -p" command to load it. Let me know how it goes. |
All times are GMT -5. The time now is 08:28 PM. |