We have three production websites running on RHEL 3 AS running tomcat 5. After a reboot last night Tomcat will not start and has the following in the catalina.out log:

Mar 20, 2011 4:09:31 PM org.apache.commons.digester.Digester startElement
SEVERE: Begin event threw error
java.lang.NoClassDefFoundError: org/apache/naming/resources/ProxyDirContext
at java.lang.Class.getDeclaredConstructors0(Native Method)
at java.lang.Class.privateGetDeclaredConstructors(Class.java:2389)
at java.lang.Class.getConstructor0(Class.java:2699)
at java.lang.Class.newInstance0(Class.java:326)
at java.lang.Class.newInstance(Class.java:308)
at org.apache.commons.digester.ObjectCreateRule.begin(ObjectCreateRule.java:253)
at org.apache.commons.digester.Rule.begin(Rule.java:200)
at org.apache.commons.digester.Digester.startElement(Digester.java:1273)
at org.apache.catalina.util.CatalinaDigester.startElement(CatalinaDigester.java:65)
at org.apache.xerces.parsers.AbstractSAXParser.startElement(Unknown Source)
at org.apache.xerces.parsers.AbstractXMLDocumentParser.emptyElement(Unknown Source)
at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanStartElement(Unknown Source)
at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown Source)
at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
at org.apache.commons.digester.Digester.parse(Digester.java:1548)
at org.apache.catalina.startup.Catalina.load(Catalina.java:489)
at org.apache.catalina.startup.Catalina.load(Catalina.java:528)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:247)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:421)
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:247)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:421)
Caused by: java.lang.NoClassDefFoundError: org/apache/naming/resources/ProxyDirContext
at java.lang.Class.getDeclaredConstructors0(Native Method)
at java.lang.Class.privateGetDeclaredConstructors(Class.java:2389)
at java.lang.Class.getConstructor0(Class.java:2699)
at java.lang.Class.newInstance0(Class.java:326)
at java.lang.Class.newInstance(Class.java:308)
at org.apache.commons.digester.ObjectCreateRule.begin(ObjectCreateRule.java:253)
at org.apache.commons.digester.Rule.begin(Rule.java:200)
at org.apache.commons.digester.Digester.startElement(Digester.java:1273)
at org.apache.catalina.util.CatalinaDigester.startElement(CatalinaDigester.java:65)
at org.apache.xerces.parsers.AbstractSAXParser.startElement(Unknown Source)
at org.apache.xerces.parsers.AbstractXMLDocumentParser.emptyElement(Unknown Source)
at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanStartElement(Unknown Source)
at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown Source)
at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
at org.apache.commons.digester.Digester.parse(Digester.java:1548)
at org.apache.catalina.startup.Catalina.load(Catalina.java:489)
at org.apache.catalina.startup.Catalina.load(Catalina.java:528)
... 6 more
[root@RPSI-2 san00]#

I have spoken with our Java Developers and they seem to think it is something up between httpd and tomcat at an OS level.

I have searched and searched and found a couple of "fixes" that didn't fix the problem:

Borrowed from LinkedIn:

Tomcat in Linux problem "lock file found but no process running for pid 7789, continuing "

Hi ,
I just got a serious problem after I assigned some permissions to my webapplication inside webapp of tomcat , I got "lock file found but no process running for pid 7789, continuing " while restarting tomcat on linux , I restarted the linux machine too but the problem didn't solve . I also tried to execute
sh -x /etc/init.d/tomcat5 start for debugging the tomcat script file , But as i a newbee for linux I am unable to identify the problem , It will be great if anybody could help me please , I am stopped here .

Thanks & Regards
Sankalp

posted May 14, 2009 in Computers and Software | Closed | Report question as...
Share This Question

* Email
* del.icio.us

Permalink: To link to this question from your blog or site, copy and paste the URL below.

Share This
Report Question
Report question as

Details (optional)

Note: Your identity will not be reported to the person who asked the question.

or Cancel
Answers (3)
Thomas Marshall

Data Specialist at Farm Journal Media

see all my answers

Best Answers in: Computers and Software (3)... see more, Databases (3), Facilities Management (1), Business Analytics (1), Ethics (1), Information Storage (1) see less

Hi Sankalp,

Your problem is simple, the solution will require a little detective work. Basically your lock file is requesting the process with the process ID 7789 grant it a lock, but you don't have a process with that ID. I imagine your lock file is expecting a Tomcat process to have that process ID.

You need to find what process is expected, what process ID it has, and what script is referencing that process by that ID and change the ID 7789 to whatever ID the aforementioned process is actually referenced by.

Thanks,

posted May 14, 2009 | Report answer as...
Bil Jeschke

Analyst - Distributed System Support

see all my answers

Best Answers in: Computers and Software (2)... see more, Biotech (1), Computer Networking (1), Web Development (1) see less

Since Tomcat is not starting and giving an error that it is all ready running, I think you have a stale lock file. Look for the file /var/run/tomcat5.pid and cat or less it to see what the PID number is. If it matches the error, then delete the file and try to start Tomcat again. It should start this time.

posted May 15, 2009 | Report answer as...
Phani Bhushan Reddy V

Software Developer

see all my answers

Most common problem with Tomcat5. Try following :

1) Search PID in /tmp, if found delete it or else stop your tomcat.
2) Undeploy your application
3) Check out this link http://tomcat.apache.org/tomcat-5.5-doc/changelog.html to know what causing tomcat to generate this error.

If none of them work.. .then I would prefer to check your application configuration settings.
Links:

* http://tomcat.apache.org/tomcat-5.5-doc/changelog.html

When I do start tomcat it appears to start:

[root@RPSI-2 san00]# service tomcat5 start
Starting tomcat5: Using CATALINA_BASE: /usr/share/tomcat5
Using CATALINA_HOME: /usr/share/tomcat5
Using CATALINA_TMPDIR: /usr/share/tomcat5/temp
Using JAVA_HOME: /usr/java/jdk1.6.0

[root@RPSI-2 san00]#

But then I try to find if it is running via ps -ef | grep tomcat5, I only receive my query. I perform the same query on Java and it also only returns my query.

Any help as soon as possible would be greatly appreciated.

Sincerely,
Lee

ask red hat
that is WHY you did buy a license in the first place
http://www.redhat.com/support/

the support AND the security updates !

John VV,

Case is submitted. Have a great evening / night.

Sincerely,
Lee

All,

I'm back, RedHat came back with unless, we purchase Extended support there is no assistance they can provide. So, the latest is that the restore has completed and still have no websites. Here are the results of attempting to start Apache and Tomcat:

[root@RPSI-1 san00]# service httpd start
Starting httpd: Apache/2.0.46 mod_ssl/2.0.46 (Pass Phrase Dialog)
Some of your private key files are encrypted for security reasons.
In order to read them you have to provide the pass phrases.

Server npcrm.net:443 (RSA)
Enter pass phrase:

OK: Pass Phrase Dialog successful.
Starting tomcat5: Using CATALINA_BASE: /usr/share/tomcat5[ OK ]
Using CATALINA_HOME: /usr/share/tomcat5
Using CATALINA_TMPDIR: /usr/share/tomcat5/temp
Using JAVA_HOME: /usr/java/jdk1.6.0


[root@RPSI-1 san00]#

When I look at the services running, Tomcat is running, however, not serving any pages. Here are the entries in the messages log about the Tomcat startup:

Mar 20 23:48:18 RPSI-1 httpd: Apache/2.0.46 mod_ssl/2.0.46 (Pass Phrase Dialog)
Mar 20 23:48:18 RPSI-1 httpd: Some of your private key files are encrypted for security reasons.
Mar 20 23:48:18 RPSI-1 httpd: In order to read them you have to provide the pass phrases.
Mar 20 23:48:18 RPSI-1 httpd:
Mar 20 23:48:18 RPSI-1 httpd: Server npcrm.net:443 (RSA)
Mar 20 23:48:18 RPSI-1 httpd: Enter pass phrase:
Mar 20 23:48:24 RPSI-1 httpd:
Mar 20 23:48:24 RPSI-1 httpd: OK: Pass Phrase Dialog successful.
Mar 20 23:48:24 RPSI-1 httpd: httpd startup succeeded
Mar 20 23:48:25 RPSI-1 su(pam_unix)[2009]: session opened for user tomcat by lhigginb(uid=0)
Mar 20 23:48:25 RPSI-1 su(pam_unix)[2009]: session closed for user tomcat

Here is the /var/log/httpd/error_log after the reboot:

[Sun Mar 20 23:44:19 2011] [notice] caught SIGTERM, shutting down
[Sun Mar 20 23:48:18 2011] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Sun Mar 20 23:48:24 2011] [notice] Digest: generating secret for digest authentication ...
[Sun Mar 20 23:48:24 2011] [notice] Digest: done
[Sun Mar 20 23:48:25 2011] [notice] config.update(): done logger.apache2
[Sun Mar 20 23:48:25 2011] [notice] config.update(): done config:
[Sun Mar 20 23:48:25 2011] [notice] config.update(): done uriMap:
[Sun Mar 20 23:48:25 2011] [notice] config.update(): done workerEnv:
[Sun Mar 20 23:48:25 2011] [notice] config.update(): done shm
[Sun Mar 20 23:48:25 2011] [notice] config.update(): done status:
[Sun Mar 20 23:48:25 2011] [notice] config.update(): done channel.socket:localhost:8009
[Sun Mar 20 23:48:25 2011] [error] env.createBean2(): Error getting factory for [ajp] localhost:8009
[Sun Mar 20 23:48:25 2011] [error] config.update(): Can't create ajp:localhost:8009
[Sun Mar 20 23:48:25 2011] [notice] config.update(): done uri:www.npcdata.net/PdfServlet/pdf*
[Sun Mar 20 23:48:25 2011] [notice] config.update(): done uri:www.npcdata.net/PdfServlet/pdf?*
[Sun Mar 20 23:48:25 2011] [notice] config.update(): done uri:www.npcdata.net/PdfServlet/pdf\?*
[Sun Mar 20 23:48:25 2011] [notice] config.update(): done uri:www.npcdata.net/PdfServlet/pdf
[Sun Mar 20 23:48:25 2011] [notice] config.update(): done uri:www.npcdata.net/PdfServlet/pdf/
[Sun Mar 20 23:48:25 2011] [notice] config.update(): done uri:www.npcdata.net/PdfServlet/pdf/*
[Sun Mar 20 23:48:25 2011] [notice] config.update(): done uri:npcdata.net/PdfServlet/pdf
[Sun Mar 20 23:48:25 2011] [notice] config.update(): done uri:npcdata.net/PdfServlet/pdf*
[Sun Mar 20 23:48:25 2011] [notice] config.update(): done uri:npcdata.net/pdf
[Sun Mar 20 23:48:25 2011] [notice] config.update(): done uri:npcdata.net/pdf*
[Sun Mar 20 23:48:25 2011] [notice] config.update(): done uri:12.22.179.4/*.jsp
[Sun Mar 20 23:48:25 2011] [notice] config.update(): done uri:12.22.179.4/*/*/*/*/*/*.jsp
[Sun Mar 20 23:48:25 2011] [notice] config.update(): done uri:12.22.179.3/*.jsp
[Sun Mar 20 23:48:25 2011] [notice] config.update(): done uri:12.22.179.3/*/*/*/*/*/*.jsp
[Sun Mar 20 23:48:25 2011] [notice] config.update(): done uri:npcrm.net/*.pdf
[Sun Mar 20 23:48:25 2011] [notice] config.update(): done uri:12.22.179.2/*.jsp
[Sun Mar 20 23:48:25 2011] [notice] config.update(): done uri:12.22.179.2/*/*/*/*/*/*.jsp
[Sun Mar 20 23:48:25 2011] [error] jk2_init() Can't find child 1937 in none of the 256 scoreboard slots
[Sun Mar 20 23:48:25 2011] [error] uriEnv.init() context must start with '/' in *.jsp
[Sun Mar 20 23:48:25 2011] [notice] workerEnv.init() ok /etc/httpd/conf/workers2.properties
[Sun Mar 20 23:48:25 2011] [notice] mod_jk child -2 initialized
[Sun Mar 20 23:48:25 2011] [error] jk2_init() Can't find child 1938 in none of the 256 scoreboard slots
[Sun Mar 20 23:48:25 2011] [error] uriEnv.init() context must start with '/' in *.jsp
[Sun Mar 20 23:48:25 2011] [notice] workerEnv.init() ok /etc/httpd/conf/workers2.properties
[Sun Mar 20 23:48:25 2011] [notice] mod_jk child -2 initialized
[Sun Mar 20 23:48:25 2011] [error] jk2_init() Can't find child 1942 in none of the 256 scoreboard slots
[Sun Mar 20 23:48:25 2011] [error] uriEnv.init() context must start with '/' in *.jsp
[Sun Mar 20 23:48:25 2011] [notice] workerEnv.init() ok /etc/httpd/conf/workers2.properties
[Sun Mar 20 23:48:25 2011] [notice] mod_jk child -2 initialized
[Sun Mar 20 23:48:25 2011] [error] jk2_init() Can't find child 1943 in none of the 256 scoreboard slots
[Sun Mar 20 23:48:25 2011] [error] uriEnv.init() context must start with '/' in *.jsp
[Sun Mar 20 23:48:25 2011] [notice] workerEnv.init() ok /etc/httpd/conf/workers2.properties
[Sun Mar 20 23:48:25 2011] [notice] mod_jk child -2 initialized
[Sun Mar 20 23:48:25 2011] [error] jk2_init() Can't find child 1944 in none of the 256 scoreboard slots
[Sun Mar 20 23:48:25 2011] [error] uriEnv.init() context must start with '/' in *.jsp
[Sun Mar 20 23:48:25 2011] [notice] workerEnv.init() ok /etc/httpd/conf/workers2.properties
[Sun Mar 20 23:48:25 2011] [notice] mod_jk child -2 initialized
[Sun Mar 20 23:48:25 2011] [error] jk2_init() Can't find child 1945 in none of the 256 scoreboard slots
[Sun Mar 20 23:48:25 2011] [error] uriEnv.init() context must start with '/' in *.jsp
[Sun Mar 20 23:48:25 2011] [notice] workerEnv.init() ok /etc/httpd/conf/workers2.properties
[Sun Mar 20 23:48:25 2011] [notice] mod_jk child -2 initialized
[Sun Mar 20 23:48:25 2011] [error] jk2_init() Can't find child 1947 in none of the 256 scoreboard slots
[Sun Mar 20 23:48:25 2011] [error] uriEnv.init() context must start with '/' in *.jsp
[Sun Mar 20 23:48:25 2011] [notice] workerEnv.init() ok /etc/httpd/conf/workers2.properties
[Sun Mar 20 23:48:25 2011] [notice] mod_jk child -2 initialized
[Sun Mar 20 23:48:25 2011] [error] jk2_init() Can't find child 1948 in none of the 256 scoreboard slots
[Sun Mar 20 23:48:25 2011] [error] uriEnv.init() context must start with '/' in *.jsp
[Sun Mar 20 23:48:25 2011] [notice] workerEnv.init() ok /etc/httpd/conf/workers2.properties
[Sun Mar 20 23:48:25 2011] [notice] mod_jk child -2 initialized
[Sun Mar 20 23:48:25 2011] [notice] Apache configured -- resuming normal operations
[Sun Mar 20 23:48:34 2011] [error] channelSocket.open() connect failed 127.0.0.1:8009 111 Connection refused
[Sun Mar 20 23:48:34 2011] [error] ajp13.connect() failed ajp13:localhost:8009
[Sun Mar 20 23:48:34 2011] [error] ajp13.service() failed to connect endpoint errno=111 Connection refused
[Sun Mar 20 23:48:34 2011] [error] ajp13.service() Error forwarding ajp13:localhost:8009 1 1
[Sun Mar 20 23:48:34 2011] [notice] ajp13.done() close endpoint ajp13:localhost:8009 error_state 1
[Sun Mar 20 23:48:34 2011] [error] mod_jk.handler() Error connecting to tomcat 120000
[Sun Mar 20 23:50:24 2011] [error] channelSocket.open() connect failed 127.0.0.1:8009 111 Connection refused
[Sun Mar 20 23:50:24 2011] [error] ajp13.connect() failed ajp13:localhost:8009
[Sun Mar 20 23:50:24 2011] [error] ajp13.service() failed to connect endpoint errno=111 Connection refused
[Sun Mar 20 23:50:24 2011] [error] ajp13.service() Error forwarding ajp13:localhost:8009 1 1
[Sun Mar 20 23:50:24 2011] [notice] ajp13.done() close endpoint ajp13:localhost:8009 error_state 1
[Sun Mar 20 23:50:24 2011] [error] mod_jk.handler() Error connecting to tomcat 120000
[Sun Mar 20 23:50:46 2011] [error] channelSocket.open() connect failed 127.0.0.1:8009 111 Connection refused
[Sun Mar 20 23:50:46 2011] [error] ajp13.connect() failed ajp13:localhost:8009
[Sun Mar 20 23:50:46 2011] [error] ajp13.service() failed to connect endpoint errno=111 Connection refused
[Sun Mar 20 23:50:46 2011] [error] ajp13.service() Error forwarding ajp13:localhost:8009 1 1
[Sun Mar 20 23:50:46 2011] [notice] ajp13.done() close endpoint ajp13:localhost:8009 error_state 1
[Sun Mar 20 23:50:46 2011] [error] mod_jk.handler() Error connecting to tomcat 120000
[Sun Mar 20 23:51:04 2011] [error] channelSocket.open() connect failed 127.0.0.1:8009 111 Connection refused
[Sun Mar 20 23:51:04 2011] [error] ajp13.connect() failed ajp13:localhost:8009
[Sun Mar 20 23:51:04 2011] [error] ajp13.service() failed to connect endpoint errno=111 Connection refused
[Sun Mar 20 23:51:04 2011] [error] ajp13.service() Error forwarding ajp13:localhost:8009 1 1
[Sun Mar 20 23:51:04 2011] [notice] ajp13.done() close endpoint ajp13:localhost:8009 error_state 1
[Sun Mar 20 23:51:04 2011] [error] mod_jk.handler() Error connecting to tomcat 120000
[Sun Mar 20 23:54:54 2011] [error] channelSocket.open() connect failed 127.0.0.1:8009 111 Connection refused
[Sun Mar 20 23:54:54 2011] [error] ajp13.connect() failed ajp13:localhost:8009
[Sun Mar 20 23:54:54 2011] [error] ajp13.service() failed to connect endpoint errno=111 Connection refused
[Sun Mar 20 23:54:54 2011] [error] ajp13.service() Error forwarding ajp13:localhost:8009 1 1
[Sun Mar 20 23:54:54 2011] [notice] ajp13.done() close endpoint ajp13:localhost:8009 error_state 1
[Sun Mar 20 23:54:54 2011] [error] mod_jk.handler() Error connecting to tomcat 120000
[Sun Mar 20 23:59:58 2011] [error] channelSocket.open() connect failed 127.0.0.1:8009 111 Connection refused
[Sun Mar 20 23:59:58 2011] [error] ajp13.connect() failed ajp13:localhost:8009
[Sun Mar 20 23:59:58 2011] [error] ajp13.service() failed to connect endpoint errno=111 Connection refused
[Sun Mar 20 23:59:58 2011] [error] ajp13.service() Error forwarding ajp13:localhost:8009 1 1
[Sun Mar 20 23:59:58 2011] [notice] ajp13.done() close endpoint ajp13:localhost:8009 error_state 1
[Sun Mar 20 23:59:58 2011] [error] mod_jk.handler() Error connecting to tomcat 120000

If any other information, would be helpful, please let me know.

Sincerely,
Lee

Hi,

Looks like the ajp connector is not listening on port 8009 as it should. To verify run:
Try to restart tomcat and watch the catalina.out to see if you fond the cause. It should be before the catalina.out snippet you've posted.

Regards

1 members found this post helpful.

All,

Thank you for your recommendations. We have found the issue, it was a corrupted SAN mount. Over 1400 files in lost+found. To fix the issue we went back and restored to a month previous and applied the changes since then and all is working well now. Again, thank you all for your help!

Take care,
Lee