Java Daemon Dies Quietly

dswanson99 · 03-03-2010, 12:21 PM

I have a java processing running as a system service using a shell script to launch it. It runs under a user other than root via sudo -u.

The process runs just fine but seems to die sometimes for no apparent reason. When the process is killed normally (via service stop), it fires a shutdown hook in java and you can see the process going away.

What I see in the log now is nothing the process just dies. When I checked the system.log I found at the same time this:

Code:

messages:Mar  2 19:43:22 beta_web gconfd (sifagent-16786): starting (version 2.14.0), pid 16786 user 'sifagent'
messages:Mar  2 19:43:22 beta_web gconfd (sifagent-16786): Resolved address "xml:readonly:/etc/gconf/gconf.xml.mandatory" to a read-only configuration source at position 0
messages:Mar  2 19:43:22 beta_web gconfd (sifagent-16786): Resolved address "xml:readwrite:/home/sifagent//.gconf" to a writable configuration source at position 1
messages:Mar  2 19:43:22 beta_web gconfd (sifagent-16786): Resolved address "xml:readonly:/etc/gconf/gconf.xml.defaults" to a read-only configuration source at position 2

The user is just for this service, so they didn't have a home. Previously the same gconfd message appeared but with an error about not having a writable home. I fixed that and part but the apparent connection between the gconfd messages in system.log and the java process having died persist.

Any ideas?

Thanks
-Doug

kbp · 03-03-2010, 05:01 PM

Are there any application specific logs ? You should also be able to set up debug level logging in the JAVA_OPTS

dswanson99 · 03-04-2010, 11:33 AM

Yes there are, but nothing in there at stop time. When the process is shutdown in an orderly fashion (via kill) its shutdown hook fires and it does log - so its not getting stopped via service or a direct garden variety kill as nothing gets logged.

Its happened two more times - both with the gconfd activity for the daemon user at the same instant.

dswanson99 · 03-04-2010, 01:20 PM

I tried changing the daemon user's shell to noshell to see if that stopped the gconfd activity, but no luck. The daemon process just died a few hours after making that change.

dswanson99 · 03-09-2010, 09:02 AM

Any ideas?

kbp · 03-09-2010, 10:09 PM

I'm not convinced that the gconf stuff is related... can you enable debug level logging ?

dswanson99 · 03-10-2010, 02:22 PM

I'm not convinced either, but its the only lead at this point. The only thing that seems clear is that the gconfd activity happens at the same time (every time) and only when it (java process dies) so its either a symptom or disease.

The logging level isn't external to the install so I have to have the developer rebuild and see what that yields. I'll post those results when I get them.

kbp · 03-10-2010, 02:52 PM

I'm not sure what platform you're running this on, but is selinux present or active ?

dswanson99 · 03-10-2010, 02:58 PM

RHEL 5.3 and selinux is in permissive.

kbp · 03-10-2010, 05:28 PM

Which jdk are you using ?

dswanson99 · 03-11-2010, 08:21 AM

OpenJDK 64-Bit Server VM (build 1.6.0-b09, mixed mode)

We've got a version of the app now that logs everything from the 3rd party libraries that we're using, hopefully that will point out something.

dswanson99 · 03-30-2010, 07:50 AM

Just wanted to say thanks for your help. We still don't know anything but the inquiry has moved from the server to the java code, so I'm off the hook for now!