I'm an IT Admin for a larger company. we have 2 dev servers and 4 live servers that pull source code from netapp server. They're using CentOS 4.4 and Apache Tomcat. They will randomly restart and I can't find anything weird in the logs...with the exception of some css reference errors that are pretty normal. It's rather annoying to have an email from the load balancer every 2 hours saying a server went down.

any ideas?

Questions:

Are all four boxes in a pool that is load balanced?

Do you have your four boxes paired off on Active / Standby or Active / Active?

When the boxes restart if in the Active / Active state is it only one of the boxes that restarts or both?

What application or device do you use for load balancing? (F5?)

When the boxes restart if in the Active / Standby is it the Active one that restarts which triggers the Standby to become active and then does the reverse happen for the was Standby now Active, Then was Active now Standby comes back up and later on they switch states?

What version of Tomcat 5.0 / 5.5 / 6.0?

Quick assumption here:

Your four prod servers share the same "/share" off the netapp so you update in one place and it is available on all four.

Ideas:

One thing I would check would be the time on all boxes. I've had issues with JBoss freaking out because our Intel Dual Core was skipping ahead b/c of ACPI being turned on. So actual time would be 5:55 but it would have a hwclk time of 5:57. NTP corrects this and then JBoss freaks out because it doesn't understand how the time is 5:56 when it was just 5:57 and the drops its connections. The only error you get is "connection time out" which only makes sense when you figure out the problem. If your tomcats are clustered you may want to see if it is a multi cast issue. Also check the netapp to see if there is an issue with communication between your app servers and the netapp at periodic or consistent times. I've seen the netapp kickback some error messages that are more informative.

yes, all 4 boxes are in a pool thats load balanced br 2 load balancers. but one LB is a failsafe.

The time idea you have makes sense. im looking into it now, and ill let you know what i see.

ill get back to you on your other questions at a later time. i have a more serious issue to address right now =[