Server repeatedly crashing

imadsani · 03-11-2014, 03:24 AM

Hey,

I have this new server which I just set up over the weekend, I've installed Atomic Secure Linux as well. After an update for WAF yesterday afternoon the server has started to go down every couple of hours, by this I mean it is completely unresponsive, can't dial in or hit apache.

I need to debug this sooner rather than later and the time difference between me and official support is really wasting time, so until they respond I need to see if I can get to the bottom of this.

Could anyone here please tell me which logs to look at?

Server specs:
AMD Octacore
32GB RAM
2x250GB SSD's
CentOS 6.5
Plesk 11.5.30

Edit: I have newrelic installed and it does not show any spike in any resource before it went down.

tombelcher7 · 03-11-2014, 07:54 AM

I know it is hard to do so when the box keeps crashing but perhaps look at dmesg or /var/log/kern.log or /var/log/kern.log.x

I haven't had the opportunity to analyse linux servers crashing before so my advice might be tenuous and any information from others is welcome....

tombelcher7 · 03-11-2014, 07:57 AM

Booting into rescue mode might also help determine if something as part of normal mode is causing issue:

http://www.openlogic.com/wazi/bid/18...S-Linux-Server

imadsani · 03-11-2014, 08:13 AM

Thank you for the reply, because this was a brand new server and I wasn't hosting anything I went and nuked it. Just got done redoing everything. Let's see if it works now.

mpyusko · 03-17-2014, 08:39 AM

One tactic I use, is to wipe all the logs.... I mean wipe them. Get rid of them. Delete them. Shut down your server and boot with DSL, System Rescue, etc... anything live. Mount your log partition and move every log to a flash drive. Don't leave any on the system. Then power off your server and remove the live CD. Power it back up like you normally would and let it run. If it becomes unrsponsive again, kill the power, don't shutdown. Reboot with that live CD again and have a look at your logs. You will only have data from the one time it booted to the point where you killed power. I'm not going to say the answer will stare you in the face, but it is much easier to try to piece together what is going wrong without all the clutter.

And the FWIW category, test you RAM with Memtest86+. You might have a corrupting chip.