pthread_create returned an error. 12 12
 

Hi Folks,
We are running a 2 node Oracle RAC cluster on RHAS 3.0.
Recently we have started getting the following errors in the cluster manager logfile on 1 of the nodes :
>ERROR: ThreadSpawn: pthread_create returned an error. 12 12 804eaf0, tid = CMConnectListerner:-1257751632 file = unixinc.c, line = 537 {Tue Mar 29 14:22:12 2005 }
>WARNING: ThreadSpawn : Retrying pthread_create..., tid = CMConnectListerner:-1257751632 file = unixinc.c, line = 538 {Tue Mar 29 14:22:12 2005 }
>ERROR: ThreadSpawn: pthread_create returned an error. 12 12 804eaf0, tid = CMConnectListerner:-1257751632 file = unixinc.c, line = 537 {Tue Mar 29 14:22:12 2005 }
>WARNING: ThreadSpawn : Retrying pthread_create..., tid = CMConnectListerner:-1257751632 file = unixinc.c, line = 538 {Tue Mar 29 14:22:12 2005 }


I have run the thread-limit.c program which can be found on numerous sites and the limit seems to be 306 !!

root@prod tmp]# ./thread-limit
Creating threads ...
Address of c = 3002926 KB
Address of c = 2992682 KB
Address of c = 2982438 KB
Address of c = 2972194 KB
Address of c = 2961950 KB
Address of c = 2951706 KB
Address of c = 2941462 KB
Address of c = 2931218 KB
Address of c = 2920974 KB
Address of c = 2910730 KB
100 threads so far ...
200 threads so far ...
300 threads so far ...
Failed with return code 12 creating thread 306.

sysctl.conf looks like this..
# Oracle semaphores setting
kernel.sem = 310 32000 100 150
# Oracle Max shared memory (as per Metalink doc 252217.1
kernel.shmmax = 4294967295
#kernel.shmmax = 2147483648
# Oracle shared memory segments in the entire system
kernel.shmmni = 4096
kernel.shmseg = 4096
kernel.shmall = 3279547
kernel.threads-max = 131072
# Oracle max file handles
fs.file-max = 327679

Oracle said this ...
However, the actual cause needs to be investigated from the O/S Side ===

I would suggest to consult REDHAT to find if there is anyway to increase the max pthread limit and how it can be done


does anyone know how this can be done ? It's fairly urgent thanks.
Dave

Run ulimit -a to see user's limit (it'll show you all limits). What's the processes limit?

Thanks for the reply.


[oraprod@prod01 92rac]$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
max locked memory (kbytes, -l) 4
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
stack size (kbytes, -s) 1024
cpu time (seconds, -t) unlimited
max user processes (-u) 7168
virtual memory (kbytes, -v) unlimited

It looks correctly. Have you recently performed a system upgrade etc?

Did You EVER resolve this??!?!?!
 

We are having the exact same problem after upgrading 9i RAC to RH AS 3.0. Oracle is not helping.

>ERROR: ThreadSpawn: pthread_create returned an error. 12 12 804eaf0, tid = CMConnectListerner:-1257759824 file = unixinc.c, line = 537 {Thu May 26 04:14:35 2005 }

Errors occuring on only one node.

Hi ddenise,

Yes I did.

The solution was to decrease the ulimit stack_size for the oraprod (oracle software owner) :

We had it set to 10MB
ulimit -s 10240

This only allows 306 threads to be created.

We reduced it to 2MB
ulimit -s 2048

This allows 1532 threads to be created.

Further to this, if you are using 11i Oracle Financials, the applmgr user needs it stack_size set to 10MB otherwise you will have application problems.

Hope this helps.

Let me know if you need anymore info.
Dave


You can check how many threads are created by using the following piece of code :

/* compile with: gcc -lpthread -o thread-limit thread-limit.c */
/* originally from: http://www.volano.com/linuxnotes.html */

#include <stdio.h>
#include <unistd.h>
#include <pthread.h>

#define MAX_THREADS 10000
int i;

void run(void) {
char c;
if (i < 10)
printf("Address of c = %u KB\n", (unsigned int) &c / 1024);
sleep(60 * 60);
}

int main(int argc, char *argv[]) {
int rc = 0;
pthread_t thread[MAX_THREADS];
printf("Creating threads ...\n");
for (i = 0; i < MAX_THREADS && rc == 0; i++) {
rc = pthread_create(&(thread[i]), NULL, (void *) &run, NULL);
if (rc == 0) {
pthread_detach(thread[i]);
if ((i + 1) % 100 == 0)
printf("%i threads so far ...\n", i + 1);
}
else
printf("Failed with return code %i creating thread %i.\n",
rc, i + 1);
}
exit(0);
}

compile it like this : gcc -pthread -o thread-limit thread-limit.c
and run it like this : ./thread-limit.c

Try to run thread-limit as ROOT.