LinuxQuestions.org
View the Most Wanted LQ Wiki articles.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices

Reply
 
Search this Thread
Old 04-09-2005, 12:02 PM   #1
socratesinus
LQ Newbie
 
Registered: Apr 2005
Posts: 3

Rep: Reputation: 0
Threads - Max Limit


Hello,

This error has been a nightmare for me. I am not able to deploy my server yet.

After 2 days, it gives up accepting new connections and then dies. The error that I get is a common error

java.lang.OutOfMemoryError: unable to create new native

I did my home work . My ulimits are

core file size (blocks, -c) 1000000
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 4096
pipe size (512 bytes, -p) 8
stack size (kbytes, -s) 8192 >> changed it to 1024 too
cpu time (seconds, -t) unlimited
max user processes (-u) 14335
virtual memory (kbytes, -v) unlimited


My test program stops after 600 threads

Starting Thread: 600
Starting Thread: 601
Starting Thread: 602
Starting Thread: 603
Starting Thread: 604
Starting Thread: 605
Starting Thread: 606
Starting Thread: 607
Starting Thread: 608
Starting Thread: 609
Exception in thread "main" java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:574)
at Test.startThreads(Test.java:17)
at Test.main(Test.java:29)


public static void startThreads(){

class TestThread extends Thread {
public void run(){
try{
while(true) sleep(1000);
}catch(Exception e){}
}
}

for(int i = 0;i<10000;i++){
try{
System.out.println("Starting Thread: " + i);
(new TestThread()).start();}
catch(Exception e)
{
e.printStackTrace();
e.getMessage();
System.exit(1);
}
}
}

I was actually able to run this command on my 1GB server and still get the same error.

java -Xms1000m -Xmx1000M Test

Running again gave me 400 threads only.

Here is the free memory at the time I ran this command.
free

total used free shared buffers cached
Mem: 1031644 989132 42512 0 186828 422104
-/+ buffers/cache: 380200 651444
Swap: 2040244 94472 1945772

java version "1.5.0_01"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_01-b08)
Java HotSpot(TM) Client VM (build 1.5.0_01-b08, mixed mode, sharing)


With the new 1.5
/usr/local/java/jdk1.5.0_02/bin/java -version
java version "1.5.0_02"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_02-b09)
Java HotSpot(TM) Client VM (build 1.5.0_02-b09, mixed mode)

it gives 879 threads - a little better but still cannot reach more.

uname -a
javaserver.*.com 2.4.27 #2 SMP Tue Aug 10 16:40:59 CDT 2004 i686 i686 i386 GNU/Linux

ls -l /lib/libc.so.*
lrwxrwxrwx 1 root root 13 Aug 10 2004 /lib/libc.so.6 -> libc-2.3.2.so*

Can it be my glibc? Will other JVMs do any good?

In c using the famous pthread_create, it goes till 1532 threads and stops after that.

How can I get this to create [lot] more threads?

Thanks
 
Old 04-11-2005, 01:59 AM   #2
foo_bar_foo
Senior Member
 
Registered: Jun 2004
Posts: 2,553

Rep: Reputation: 51
i personally think it's glibc
your version 2.3.2 has the ceiling set at
#define PTHREAD_THREADS_MAX 16384
take a look at /proc/sys/kernel/threads-max so that's not it but The more you reduce STACK_SIZE in LinuxThreads the more threads you can safely create.
your version of glibc has in linuxthreads/internals.h
/* The initial size of the thread stack. Must be a multiple of PAGE_SIZE. */
#ifndef INITIAL_STACK_SIZE
#define INITIAL_STACK_SIZE (4 * PAGE_SIZE)
#endif
if i am not mistaken PAGE_SIZE = 4096
so if you feel like going through the trouble of recompiling glibc the real answer might be to reduce that number to say
#define INITIAL_STACK_SIZE (PAGE_SIZE)
that is still 4Mb and that still seems like too much 4*4 mb the original seems like WAY too much if you want to make alot of threads doesn't it
i wonder if you could do (warning i havn't tried this)
#define INITIAL_STACK_SIZE (PAGE_SIZE/2) or even (PAGE_SIZE/4)

remember your version of glibc had linuxthreads as a seperate package you opened inside glibc
and you had to put --enable-add-ons as a config option

i guess you are not up against kernel file descriptor limits because your test is just a matter of thread creation not sockets but look at /proc/sys/fs/file-max

you could try ibm jre ?
http://www-128.ibm.com/developerwork...jdk/index.html

also try in the shell you are using
export LD_ASSUME_KERNEL=2.4.0
and
export LD_ASSUME_KERNEL=2.4.1
these should inhibit the use of NPTL threading
http://java.sun.com/developer/techni...dLinux/RedHat/
and at least allow you to try it with the older threading codes

good luck
 
Old 04-11-2005, 10:51 AM   #3
socratesinus
LQ Newbie
 
Registered: Apr 2005
Posts: 3

Original Poster
Rep: Reputation: 0
Thanks for your reply foo_bar.

I realized that 2.4 depended on glibc. I loaded 2.6.11.7 and presume that it does not depend on glibc limitation.

The pthread test did not create more than 1533 threads (similar to last one) BUT java created only 53 threads and bailed out. The ulimit -s was 1k. It does not matter actually what limit I put there.

I suspect if there is a limit with ssh login. Since, it is a hosted machine, I can only ssh to it.

The more I think, I think it is time to move to windows (I know it has poor performance but atleast it works )

ulimit -a
core file size (blocks, -c) 1000000
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 4096
pipe size (512 bytes, -p) 8
stack size (kbytes, -s) 1024
cpu time (seconds, -t) unlimited
max user processes (-u) unlimited
virtual memory (kbytes, -v) unlimited

uname
2.6.11.7 #1 Sun Apr 10 23:57:56 CDT 2005 i686 i686 i386 GNU/Linux

I cannot give IBM JRE a shot as we use latest nio possible only in Tiger - 5.0. I will try blackdown. I do not think it would help in any way.

I am trying to ssh to machine, do you think it would be the problem.

Thanks. Any help would be appreciated.
 
Old 04-11-2005, 01:54 PM   #4
foo_bar_foo
Senior Member
 
Registered: Jun 2004
Posts: 2,553

Rep: Reputation: 51
Quote:
Originally posted by socratesinus
Thanks for your reply foo_bar.

I realized that 2.4 depended on glibc. I loaded 2.6.11.7 and presume that it does not depend on glibc limitation.

if glibc says each thread takes up a certain amount of stack space then each thread takes up that amount of stack space -- the kernel doesn't matter.
the error after all is OutOfMemoryError
 
Old 04-11-2005, 03:07 PM   #5
socratesinus
LQ Newbie
 
Registered: Apr 2005
Posts: 3

Original Poster
Rep: Reputation: 0
Thanks for your reply again.

I changed the stack using uname -s to 1k and I still get this problem.

It might be my glibc and chaning config of that might be a huge pain.
 
Old 04-20-2005, 03:43 PM   #6
bmoy1
LQ Newbie
 
Registered: Apr 2005
Posts: 1

Rep: Reputation: 0
Looks like it's the kernel's threads-max setting that is limiting you. To find out what the current limit is: sysctl -a | grep threads

You can change the value on-the-fly with: echo <number> > /proc/sys/kernel/threads-max
So if you wanted to change it to 50000, for example: echo 50000 > /proc/sys/kernel/threads-max

If this works for you, make the setting permanent by adding "kernel.threads-max = 50000" in your /etc/sysctl.conf file and then issue the "sysctl -p" command to load it.

Let me know how it goes.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Limit Cpu Use Of Threads yakotey Programming 1 10-03-2005 09:12 AM
howto limit max concurent connections ?? fritz001 Slackware 6 08-11-2005 11:38 AM
APACHE: limit max connections from the same IP ? tclwp Linux - Networking 3 06-24-2005 11:47 AM
What is 2.6 Kernel's max concurrent threads? meeble Slackware 6 03-18-2004 08:28 PM
max process limit anoop_chandran Linux - Networking 2 01-16-2002 05:22 AM


All times are GMT -5. The time now is 09:29 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration