Hello everyone. This is my first post so I apologize for any failed etiquette.

Last night I had the head node for one of my clusters go down twice and I am
trying to sift through the log files to identify what has happened.

Here is the log from /var/log/dmesg:
https://pastebin.com/vDaHX157

Here is a snapshot of the most recent crash:
https://pastebin.com/JBr7RxGM

Can someone point me in the right direction to figuring out what went wrong and how can I fix it?
I am running CentOS 6.9
Please let me know if any additional information is needed.


As a huge warning, as of about a month ago I was ...drafted in... to become the system administrator for these clusters. As far as notes left behind, I have 1 page of passwords, and 1 page of IPs. It was only about 2 months ago I learned what the bash command "vi" was. I have a lot to learn and I appreciate any help and patience given to me.

Thank you.

Edit 08/17/2018 12:09PM CST

The logs I posted above appears to be after the system reboot. Prior to the system shutdown, there were no logs that indicated why the machine shut down. I do not believe that there was a temperature issue. After taking the machine apart, everything seemed fairly clean. No HDD issues, raid controller seems fine. Network switch logged nothing out of the ordinary. APC backup power also did not log anything weird with respect to the power supply of the head node.

Both logs do not contain errors about a crash.

So no indication about the error or cause of the crash.

is this the same issue? https://www.linuxquestions.org/quest...og-4175636599/

Yes it is. I realized that this is most likely not a Kernel issue and I moved the topic to there. I will mark this thread as solved. I apologize for the confusion. If I can remove this post, it would be all the better

Thread reported for closure.

Continue here:

https://www.linuxquestions.org/quest...og-4175636599/