LinuxQuestions.org - Problem with Nutch, Tomcat, or Java

- Linux - Newbie (https://www.linuxquestions.org/questions/linux-newbie-8/)

- - Problem with Nutch, Tomcat, or Java (https://www.linuxquestions.org/questions/linux-newbie-8/problem-with-nutch-tomcat-or-java-550513/)

Problem with Nutch, Tomcat, or Java

Hoping you all might be able to help this newbie.

I recently installed Debian Etch, the latest JVM, Tomcat 5.5, and Nutch 0.9 . Was able to successfully crawl, index, and search the database from the terminal console, but got an error when trying to use the Nutch search page located at localhost:8180 (Tomcat's default).

When performing a search query I get the following error:

Code:

HTTP Status 500 -

type Exception report



message



description The server encountered an internal error () that prevented it from fulfilling this request.



exception



org.apache.jasper.JasperException: Exception in JSP: /search.jsp:66



63: --%>

64:

65: <%

66:  NutchBean bean = NutchBean.get(application, nutchConf);

67:  // set the character encoding to use when interpreting request values

68:  request.setCharacterEncoding("UTF-8");

69:





Stacktrace:

  org.apache.jasper.servlet.JspServletWrapper.handleJspException(JspServletWrapper.java:506)

  org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:395)

  org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:314)

  org.apache.jasper.servlet.JspServlet.service(JspServlet.java:264)

  javax.servlet.http.HttpServlet.service(HttpServlet.java:802)

  sun.reflect.GeneratedMethodAccessor55.invoke(Unknown Source)

  sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

  java.lang.reflect.Method.invoke(Method.java:585)

  org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:243)

  java.security.AccessController.doPrivileged(Native Method)

  javax.security.auth.Subject.doAsPrivileged(Subject.java:517)

  org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:275)

  org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:161)



root cause



java.lang.RuntimeException: java.security.AccessControlException: access denied (java.lang.reflect.ReflectPermission suppressAccessChecks)

  org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:80)

  org.apache.hadoop.fs.FileSystem.get(FileSystem.java:159)

  org.apache.hadoop.fs.FileSystem.getNamed(FileSystem.java:119)

  org.apache.hadoop.fs.FileSystem.get(FileSystem.java:91)

  org.apache.nutch.searcher.NutchBean.<init>(NutchBean.java:95)

  org.apache.nutch.searcher.NutchBean.<init>(NutchBean.java:84)

  org.apache.nutch.searcher.NutchBean.get(NutchBean.java:71)

  org.apache.jsp.search_jsp._jspService(search_jsp.java:106)

  org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)

  javax.servlet.http.HttpServlet.service(HttpServlet.java:802)

  org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:334)

  org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:314)

  org.apache.jasper.servlet.JspServlet.service(JspServlet.java:264)

  javax.servlet.http.HttpServlet.service(HttpServlet.java:802)

  sun.reflect.GeneratedMethodAccessor55.invoke(Unknown Source)

  sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

  java.lang.reflect.Method.invoke(Method.java:585)

  org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:243)

  java.security.AccessController.doPrivileged(Native Method)

  javax.security.auth.Subject.doAsPrivileged(Subject.java:517)

  org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:275)

  org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:161)



note The full stack trace of the root cause is available in the Apache Tomcat/5.5 logs.

Not sure but I think the Nutch search page (located at: /usr/share/tomcat5.5-webapps/ROOT ) doesn't know where the Nutch Java classes and/or the database files are.

Any ideas on what this means and how to fix this?

The folders for Java and Tomcat were installed in their default locations. Nutch was installed in /usr/local/nutch/ with the database index located at /usr/local/nutch/crawl .

Thanks in advance,
:newbie:

To me it looks more as if it's a matter of permissions. Are you doing this as root or as a regular user? What about password protection on the database?

Thanks for your reply, jay73. Everything was done as root when using the terminal (installing packages, running the crawl, etc.). But when using the search page via the browser (here on the local network and from outside) it was just done normally, of course.

Nothing was done to the database to password protect it. I just followed the Nutch tutorial verbatim for our first crawl.

Edit: Not sure if this is relevant. Nutch typically uses port 8080. Since installation I've never been able to get port 8080 open/listening. Ports 80 (apache2) and 8180 (tomcat) are fine. But all port scans (nmap) have showed that 8080 is closed. Could that be a problem?

That would be one explanation. I'm not familiar with that kind of application but I'm sure it has a configuration file somewhere that allows assigning a different port. It's usually as easy as tracing down the relevant line in the file and manually replacing with a different port.

I located the server.xml file for tomcat and changed port 8180 to 8080 (then restarted tomcat). I was able to access Nutch's search page at localhost:8080 but it gave "less" of an error this time when doing a search. Which I guess is progress.

Code:

type Exception report



message 



description The server encountered an internal error () that prevented it from fulfilling this request.



exception 



org.apache.jasper.JasperException: java.security.AccessControlException: access denied (java.lang.reflect.ReflectPermission suppressAccessChecks)

        org.apache.jasper.servlet.JspServletWrapper.handleJspException(JspServletWrapper.java:512)

        org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:395)

        org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:314)

        org.apache.jasper.servlet.JspServlet.service(JspServlet.java:264)

        javax.servlet.http.HttpServlet.service(HttpServlet.java:802)

        sun.reflect.GeneratedMethodAccessor50.invoke(Unknown Source)

        sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

        java.lang.reflect.Method.invoke(Method.java:585)

        org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:243)

        java.security.AccessController.doPrivileged(Native Method)

        javax.security.auth.Subject.doAsPrivileged(Subject.java:517)

        org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:275)

        org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:161)





root cause 



java.lang.RuntimeException: java.security.AccessControlException: access denied (java.lang.reflect.ReflectPermission suppressAccessChecks)

        org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:80)

        org.apache.hadoop.fs.FileSystem.get(FileSystem.java:159)

        org.apache.hadoop.fs.FileSystem.getNamed(FileSystem.java:119)

        org.apache.hadoop.fs.FileSystem.get(FileSystem.java:91)

        org.apache.nutch.searcher.NutchBean.<init>(NutchBean.java:95)

        org.apache.nutch.searcher.NutchBean.<init>(NutchBean.java:84)

        org.apache.nutch.searcher.NutchBean.get(NutchBean.java:71)

        org.apache.jsp.search_jsp._jspService(search_jsp.java:106)

        org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)

        javax.servlet.http.HttpServlet.service(HttpServlet.java:802)

        org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:334)

        org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:314)

        org.apache.jasper.servlet.JspServlet.service(JspServlet.java:264)

        javax.servlet.http.HttpServlet.service(HttpServlet.java:802)

        sun.reflect.GeneratedMethodAccessor50.invoke(Unknown Source)

        sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

        java.lang.reflect.Method.invoke(Method.java:585)

        org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:243)

        java.security.AccessController.doPrivileged(Native Method)

        javax.security.auth.Subject.doAsPrivileged(Subject.java:517)

        org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:275)

        org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:161)





note The full stack trace of the root cause is available in the Apache Tomcat/5.5 logs.

The first error had a section about /search.jsp and line 66 (NutchBean). But this time it didn't give that error. Either that's a good thing or a bad thing.

So now I need to investigate the Java security a little more.

Progress. I stumbled across this tutorial about Nutch and Debian Etch. It has some code that is suppose to be added to the policy file located at /etc/tomcat5.5/policy.d/04webapps.policy . The code is:

Code:

grant codeBase "file:/usr/share/tomcat5.5-webapps/-" { 

    permission java.util.PropertyPermission "user.dir", "read";

    permission java.util.PropertyPermission "java.io.tmpdir", "read,write";

    permission java.util.PropertyPermission "org.apache.*", "read,execute";

    permission java.io.FilePermission "/usr/local/nutch/crawls/-" , "read";

    permission java.io.FilePermission "/var/lib/tomcat5.5/temp", "read";

    permission java.io.FilePermission "/var/lib/tomcat5.5/temp/-", "read,write,execute,delete";

    permission java.lang.RuntimePermission "createClassLoader", "";

    permission java.security.AllPermission;

    };

After restarting tomcat a search could be performed without errors. :) BUT no results were found (unlike searches using the terminal inside the nutch dirctory). :( All I need to do now is figure out how to point Tomcat to the right crawl directory. I guess.

Thanks for your help, jay73. You were right, it was about permissions.

Edit: Solved. The tutorial above mentions placing a searcher.dir variable (pointing back to the nutch crawl directory - /usr/local/nutch/crawl ) inside a nutch-site.xml file. I found the file within /usr/share/tomcat5.5-webapps/ROOT/WEB-INF/classes/ folder and added:

Code:

<name>searcher.dir</name> 

<value>/usr/local/nutch/crawl</value>

Still no change. So I opened a file in the same classes directory called nutch-default.xml and hidden within that file was a variable for searcher.dir. I changed the <value> to point to the nutch crawl directory, restarted Tomcat again, and WALLA! Complete searches!

Hopefully this will help if anyone else tries to get Nutch running on Debian Etch. Thanks!