Oracle eBusiness Java process hangs then dies

MensaWater · 07-14-2009, 12:29 PM

We are attempting to launch a Java process via opmn. It just goes to wait state and eventually dies. Details follow:

Running an strace on the pid after it is started by DBA I see:

strace -p 29381

Quote:

Process 29381 attached - interrupt to quit
[ Process PID=29381 runs in 32 bit mode. ]
futex(0xc7070dc4, FUTEX_WAIT_PRIVATE, 3, NULL) = 0

That stays there until just before it dies at which point the following appears:

Quote:

futex(0x9e878c0, FUTEX_WAKE_PRIVATE, 1) = 0
gettimeofday({1247589555, 792580}, NULL) = 0
write(9, "09/07/14 12:39:15 ", 18) = 18
write(9, "Fatal error: server exiting", 27) = 27
write(9, "\n", 1) = 1
clone(child_stack=0xc26ff4b4, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARE NT_SETTID|CLONE_CHILD_CLEARTID, parent_tidptr=0xc26ffbd8, tls=0xc26ffbd8, child_tidptr=0xff8f5160) = 29472
sched_setscheduler(29472, SCHED_OTHER, { 5 }) = -1 EINVAL (Invalid argument)
futex(0xa266704, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xa266700, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
futex(0x9fc72d8, FUTEX_WAKE_PRIVATE, 1) = 1
clone(child_stack=0xc22ff4b4, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARE NT_SETTID|CLONE_CHILD_CLEARTID, parent_tidptr=0xc22ffbd8, tls=0xc22ffbd8, child_tidptr=0xff8f5160) = 29473
sched_setscheduler(29473, SCHED_OTHER, { 5 }) = -1 EINVAL (Invalid argument)
futex(0x9e85830, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x9fc739c, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x9fc7398, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
futex(0xa183bf8, FUTEX_WAKE_PRIVATE, 1) = 1
gettimeofday({1247589555, 793653}, NULL) = 0
gettimeofday({1247589555, 793721}, NULL) = 0
futex(0xc7070dc4, FUTEX_WAIT_PRIVATE, 5, NULL) = 0
futex(0x9e878c0, FUTEX_WAKE_PRIVATE, 1) = 0
unlink("/oracle/edwp/inst/apps/EDWP_atlbipd1/ora/10.1.3/j2ee/oacore/persistence/oacore_default_group_1/scheduler_notifications.lock") = -1 ENOENT (No such file or directory)
unlink("/oracle/edwp/inst/apps/EDWP_atlbipd1/ora/10.1.3/j2ee/oacore/persistence/oacore_default_group_1/scheduler_events.lock") = -1 ENOENT (No such file or directory)
unlink("/oracle/edwp/inst/apps/EDWP_atlbipd1/ora/10.1.3/j2ee/oacore/persistence/oacore_default_group_1/scheduler_jobstore.lock") = -1 ENOENT (No such file or directory)
unlink("/oracle/edwp/inst/apps/EDWP_atlbipd1/ora/10.1.3/j2ee/oacore/config/server.xml.lock") = -1 ENOENT (No such file or directory)
futex(0x9fc739c, FUTEX_WAIT_PRIVATE, 1, NULL) = 0
futex(0x9e85a20, FUTEX_WAKE_PRIVATE, 1) = 0
gettimeofday({1247589555, 838657}, NULL) = 0
futex(0xf7bce600, FUTEX_WAKE_PRIVATE, 1) = 1
sched_getparam(29381, { 0 }) = 0
sched_getscheduler(29381) = 0 (SCHED_OTHER)
gettimeofday({1247589555, 838947}, NULL) = 0
futex(0x9f420bc, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x9f420b8, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
futex(0x9e858b0, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xa151814, FUTEX_CMP_REQUEUE_PRIVATE, 1, 2147483647, 0x9e85730, 6) = 1
futex(0x9e85730, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x9e85830, FUTEX_WAIT_PRIVATE, 2, NULLPANIC: attached pid 29381 exited with 1
<unfinished ... exit status 1>

The files noted above as "No such file or directory" don't exist at any point. Since they're lock files my guess is they're supposed to be created at start of the process but never are so the fact they don't exist at end is immaterial. Not having spent a lot of time with strace I can't be sure.

Is there a good way to determine why it is hanging? I looked through the /proc stuff for the process but didn't see anything helpful.

abakalidis · 07-16-2009, 01:50 AM

This may be a long shot.

I have had similar issues with Oracle IAS 10.1.3.1 and it turned out that I had mistyped one of my hard limits on /etc/security/limits.conf.

MensaWater · 07-16-2009, 10:28 AM

Thanks.

In fact I had changed a couple of settings in limits.conf back in January but on doing a check just now the settings appear to be OK and when I switch to the desired user they take effect.