Server down for approx 14 hours, can't find anything useful in logs
Linux - ServerThis forum is for the discussion of Linux Software used in a server related context.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
Server down for approx 14 hours, can't find anything useful in logs
I'm getting weird behavior on one of our servers, and I haven't been able to track down the root cause. This problem happens on Sundays, but does not happen every Sunday. The server will become unresponsive for 12 hours or more, and there will be a gap in all of the server's logs that corresponds with the downtime. Looking at /var/log/syslog, it appears that the machine is rebooting itself several times.
Notice the 14-hour gap there. This same gap exists in all the log files I checked. Something appears to be overloading the system and causing it to reboot, but I can't tell what. The timing of these outages seem to be erratic as well. One week the outage happened at 2:30 am, another time at approx 4:00 pm, and this time at around 2:20 pm. I've tried watching the server during the middle of the night to see if I could catch some script running wild with system resources, but I've never caught anything. Is there anything else I can be doing in order to investigate this issue?
The server is running Ubuntu Feisty, and serves up a webapp running on Apache2 w/ PostgreSQL 8.2.
Last edited by retrovertigo; 07-28-2008 at 01:38 PM.
Interesting problem, especially with reboots in such short succession... When did this start to happen? Can you trace back what changed at that time? Can you check hardware conditions? No overheating? No filesystem problems? Does the server run some HW/SW watchdog? Is the hardware identical to another server that runs OK? What automated processes cronjobs does the server run? Any apps that respond to logged strings? What monitoring do you run right now? Any chance of letting Atop, Dstat or Collectl run? Is the kernel and userland software identical to another server that runs OK? What services does this server provide exactly? What other processes does it run? Can you make the machine run syslog from init and make it log to a remote syslog server or like a serial console? Do the logs show other anomalies than this?
It began happening a couple months ago. The last time it happened we feared power issues and moved the power to a separate UPS on a dedicated power line separate from where it was plugged in before. The problem went away for a couple weekends, so we thought this took care of it, however the problem has cropped up again now.
There have been no recent hardware changes, and servers with similar hardware have not experienced any downtime. Overheating was initially a concern as our server closet is on the small side, but we addressed that by powering off several unneeded internal servers on the weekend when we're not in business.
The weekly cron jobs do run every Sunday morning at 6:47am. There are three of them:
None of these would appear to be the problem as one of the outages occurred at approx. 2am on a Sunday morning, several hours before the weekly cronjobs were set to run. The only self-created cronjob is a small Perl script I wrote to handle the daily, weekly, and monthly local backups of the PGSQL database.
Unfortunately no other servers have identical software configurations. The server is a web portal for our company, and serves up a Perl-based webapp. The only additional services running are apache2 and postgresql.
I do not have experience using those monitoring tools, can you recommend one of them and I'll study it more in-depth and leave it to run this Sunday?
Last edited by retrovertigo; 07-30-2008 at 09:15 AM.
Is there anything else running on an automated interval, like backups? How much disk space is used (is it possible that there's something like a log rotate that's sending it over the edge?)
If you're worried about environmentals or power, I'd plug it into a UPS that supports monitoring (like an APC SmartUPS) and have that monitored remotely every few minutes. For environmental, an real environment sensor, or one of the Dallas 1-wire sensors, will be able to give you a handle on temperature.
Most importantly, I can't begin to stress the importance of a good host/network monitoring system, such as Nagios or Zenoss, in tracking down the root cause of the problem. You'll be able to monitor nearly every aspect of your system (in this case, the most important seems to be hardware (SMART, as well as lmsensors/onboard temps), system load and uptime, RAID health (if applicable), and *wink* logged in users.
What type of hardware is this running? If it's Proliant or Sun, the management log should be able to give you helpful information.
I think jantman has some excellent remarks there and you should go with those.
In short: 'atop' is like 'top' except I find it easier to save output, "step" through saved records, grep for terms, 'dstat' is like 'sar -A' and collectl I have no experience with but it's somewhat similar to Dstat. I'd use Atop plus Dstat but it depends on how you want to look for info. You could for instance make Dstat save to CSV then graph it looking for spikes and then "zoom in" on a certain period using 'atop -r savedlog -b start:time -e end:time'.
The "problem" is there's no indication of anything, so saving SAR info is just something of a hedge. Since you've got stability problems I'd first look at replacing the webserver (if possible) with a stable machine and *then* troubleshoot it.
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.
