kernel panic- not syncing : fatal exception on RHEL 5
 

Hi All,

One of the RHEL 5 server suddenly went in to kernel panic with error "kernel panic- not syncing : fatal exception on RHEL 5".

In order to revive the system we hard rest the system using ILO. No core dump was generated due to kdump mistakenly not operational due to which we are not able to figure out the root cause of the kernel panic.

I am sharing the logs before server was restarted, Could the trap divide error be the root cause as it could be seen before restart.

Note: This server is a part of Oracle cluster.

May 28 03:33:34 nxdbsd05 snmpd[11718]: Connection from UDP: [127.0.0.1]:59803
May 28 03:33:49 nxdbsd05 snmpd[11718]: Connection from UDP: [127.0.0.1]:57607
May 28 03:33:49 nxdbsd05 snmpd[11718]: Connection from UDP: [127.0.0.1]:57607
May 28 03:33:49 nxdbsd05 snmpd[11718]: Connection from UDP: [127.0.0.1]:37767
May 28 03:33:49 nxdbsd05 snmpd[11718]: Connection from UDP: [127.0.0.1]:10862
May 28 03:34:03 nxdbsd05 kernel: nmupm[8938] trap divide error rip:804a36a rsp:fffed274 error:0
May 28 03:34:04 nxdbsd05 snmpd[11718]: Connection from UDP: [127.0.0.1]:62322
May 28 03:34:04 nxdbsd05 snmpd[11718]: Connection from UDP: [127.0.0.1]:62322
May 28 03:34:04 nxdbsd05 snmpd[11718]: Connection from UDP: [127.0.0.1]:63851
May 28 03:34:04 nxdbsd05 snmpd[11718]: Connection from UDP: [127.0.0.1]:43416
May 28 03:34:18 nxdbsd05 kernel: nmupm[9015] trap divide error rip:804a36a rsp:ffc40334 error:0
May 28 03:34:19 nxdbsd05 snmpd[11718]: Connection from UDP: [127.0.0.1]:62934
May 28 03:34:19 nxdbsd05 snmpd[11718]: Connection from UDP: [127.0.0.1]:62934
May 28 03:34:19 nxdbsd05 snmpd[11718]: Connection from UDP: [127.0.0.1]:62302
May 28 03:34:19 nxdbsd05 snmpd[11718]: Connection from UDP: [127.0.0.1]:10581
May 28 03:34:23 nxdbsd05 snmpd[11718]: Connection from UDP: [10.40.198.110]:55385
May 28 03:34:23 nxdbsd05 last message repeated 2 times
May 28 03:34:23 nxdbsd05 xinetd[18803]: START: nrpe pid=9043 from=10.40.111.152
May 28 03:34:23 nxdbsd05 nrpe[9043]: INFO: SSL/TLS initialized. All network traffic will be encrypted.
May 28 03:34:23 nxdbsd05 xinetd[18803]: EXIT: nrpe status=0 pid=9043 duration=0(sec)
May 28 03:56:30 nxdbsd05 syslogd 1.4.1: restart.
May 28 03:56:30 nxdbsd05 kernel: klogd 1.4.1, log source = /proc/kmsg started.

If you've nothing else, it's a good place to start. Your choice of colour made the message illegible, btw.

My guess is: Something went badly wrong; the kernel trapped it and logged it; then there was a crash. Is a hack attempt a possibility?

This was the error logged before the kernel panic and reset. No possibility of hacking.

May 28 03:34:18 nxdbsd05 kernel: nmupm[9015] trap divide error rip:804a36a rsp:ffc40334 error:0

the server rebooted again (This time no panic) with the same error logged before reboot. Please let me know if anybody has seen this error before


kernel: nmupm[30394] trap divide error rip:804a36a rsp:ffd09094 error:0

This thread started 1st June. Haven't you updated the kernel since then?