LinuxQuestions.org
Latest LQ Deal: Complete CCNA, CCNP & Red Hat Certification Training Bundle
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices


Reply
  Search this Thread
Old 10-28-2005, 04:10 PM   #1
InJesus
Member
 
Registered: Oct 2005
Location: MN
Distribution: Slack, Slamd64
Posts: 33

Rep: Reputation: 15
Java problems, JDK 1.5.x, server crashes/hangs


Hey Guys (Looking for a little direction)

I recently inherited a system, I was hired to mostly write Java and Servlets. But my secondary responsibilities are to maintain 4 servers and everything on them. Well the guy b4 me(btw, documented nothing and things aren't organized well at all) had problems with the Webserver crashing every few days to a week and he reinstalled Fedora I guess and quit shortly after. Well I came in and everything was running for about 1 month after he reinstalled everything and the same problem is surfacing(or so I think). I have a feeling this is why he quit, but can get little knowlage from him as he did no testing or anything to try to solve the problem. Here is a little background and then I'll go into the problem and what I've done and How I've come to the conclutions I've come to. (it is Fedora 3.x.x and Apache 2.x and I still can't be sure which version of tomcat he's running because the readme's and stuff say 4.1.2x but the server.xml file resembles a 5.x.x format and he said he upgraded tomcat apache and fedora regularly, yet there is no evidence of this and JDK 1.5.x)(this is the main server(Webserver that were talking about) with 3 instances of Tomcat and a Java program that listens to a port for an API). For naming sake I'll call them JavaProgram, tomcat1, tomcat2, tomcat3. So intially when the servers started to crash I checked the hardware and such(nothing wrong there). Created a bunch of Cron Jobs to see processes, load, # of files open. To see if those were exceeding thresholds and to Identify if I was getting DoS or something. Then I checked for a RootKit. Also examined the log files (which were poorly set up, not a whole lot of information there). What I found out from all that was I wasn't being attacked or Hijacked and the processes where pretty hefty close to the time of the crash, like around 100 normally to 220+ around time of crash. Most of those processes where child (/usr/bin/httpd). Then I started to watch it super close and discovered one of the 4 tomcatX and JavaProgram would hang and the load would increase rapidly from there. Usually it was JavaProgram or Tomcat1 that would hang first, sometimes not. At that point if I try to Kill the JavaProgram or TomcatX it will result in a <defunct> Zombie process and from that point it goes down hill fast, rather than gradually crashing. At that point if I try to shutdown any of the other TomcatX or JavaProgram it results in a server crash usually, or sometimes that will just turn into a <defunct> Zombie too. So I started to check for known bugs and found that when Tomcat or the like messes up Java bad enough, this same result occurs. Although the bugs they mentioned were not the culprate (threading issues with HashMap creates infinite loop, Hangs/Crashes server). As there are no HashMap.read() threads in any of my Catalina.Out. Ok so this is where I'm at and wondering if I could get some direction on where to go next. If you need anymore information b4 you can point me some where just post that and I will reply, thanks for the help.
 
Old 10-28-2005, 05:00 PM   #2
mhiggins
Member
 
Registered: Feb 2004
Posts: 140

Rep: Reputation: 15
You can try to kill the java process with a kill -3 (SIGQUIT) which will dump a stack trace to catalina.out or stdout (depending) .

Whats the state of the java process when its not responding? Is it out of memory ? Whats the CPU look like when this happens? If the vm is out of memory you may want to add -Xms and -Xmx.
 
Old 10-28-2005, 05:08 PM   #3
Tinkster
Moderator
 
Registered: Apr 2002
Location: in a fallen world
Distribution: slackware by choice, others too :} ... android.
Posts: 23,067
Blog Entries: 11

Rep: Reputation: 911Reputation: 911Reputation: 911Reputation: 911Reputation: 911Reputation: 911Reputation: 911Reputation: 911
Hi, and welcome to LQ!

No obvious solution, but a few more questions:
Code:
1.) You have java 1.5.x written in your post, which version(s)
    are you/have you been trying?
2.) From the entire post I take it that "server crashes" only refers
    to the tomcat processes rather than the entire system. Did I
    understand that correctly?
3.) At the time of those crashes, what is the memory and CPU
    utilisation like?  Have you taken batch-output of top at any
    of those times?

Cheers,
Tink
 
Old 10-28-2005, 05:14 PM   #4
InJesus
Member
 
Registered: Oct 2005
Location: MN
Distribution: Slack, Slamd64
Posts: 33

Original Poster
Rep: Reputation: 15
Hey guys thanks for the respones it is listed as JDK 1.5.0_04, I've tried Kill -QUIT, is that the same as Kill -3? When I say server Crashes, it brings the whole machine down, to where I have to go physically restart it. normally the load is .00 to .40 at the time of a hang it's 2.00 and at the time of the crash it's 7.00+ with all of the system being used up by Java. Memory is not as much I've got gigs of that, it never gets close to using it all and dosen't touch the swap at all.
 
Old 10-28-2005, 05:16 PM   #5
InJesus
Member
 
Registered: Oct 2005
Location: MN
Distribution: Slack, Slamd64
Posts: 33

Original Poster
Rep: Reputation: 15
The state of the process when it's not responding is S the same as when it's working, if I kill it then it turns into a zombie. and that speeds up the enevitable crash.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
compile java JDK 1.4 kernelvn *BSD 2 06-21-2005 10:42 PM
adding java jdk to path snecklifter Linux - Newbie 4 11-01-2004 03:58 PM
Java , Xfontsel Crashes X Server in Suse 9.1 Professional osho_gg Linux - Software 1 09-02-2004 10:53 AM
Sun jdk 1.4.1 crashes on compilation juby Programming 1 06-28-2003 06:36 AM
Mandrake 9.0 and Java jdk + Forte problems lolmc Linux - Software 0 12-14-2002 01:02 AM

LinuxQuestions.org > Forums > Non-*NIX Forums > Programming

All times are GMT -5. The time now is 04:50 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration