Hi All

I am hoping that someone can perhaps assist with this one as I cant seem to find any exact info on this.

An oracle upgrade was done on one of our clients' servers (Oracle Rac setup runnign on RHEL 4.5 ES)

After the upgrade (about a day later) the server rebooted and the following errors were found in the log files relating to the reboot:




Mar 23 06:03:55 oradb02 kernel: (13312,3):dlm_print_one_lock_resource:461 lockres: M000000000000000015ba6bbab5afdb, owner=0, state=64
Mar 23 06:03:55 oradb02 kernel: (13312,3):__dlm_print_one_lock_resource:476 lockres: M000000000000000015ba6bbab5afdb, owner=0, state=64
Mar 23 06:03:55 oradb02 kernel: (13312,3):__dlm_print_one_lock_resource:478 last used: 861607677, on purge list: yes
Mar 23 06:03:55 oradb02 kernel: (13312,3):dlm_print_lockres_refmap:444 refmap nodes: [ ], inflight=0
Mar 23 06:03:55 oradb02 kernel: (13312,3):__dlm_print_one_lock_resource:480 granted queue:
Mar 23 06:03:55 oradb02 kernel: (13312,3):__dlm_print_one_lock_resource:495 converting queue:
Mar 23 06:03:55 oradb02 kernel: (13312,3):__dlm_print_one_lock_resource:510 blocked queue:
Mar 23 06:03:55 oradb02 kernel: ------------[ cut here ]------------
Mar 23 06:03:55 oradb02 kernel: kernel BUG at /rpmbuild/smushran/BUILD/ocfs2-1.2.9/fs/ocfs2/dlm/dlmmaster.c:2300!
Mar 23 06:03:55 oradb02 kernel: invalid operand: 0000 [#1]
Mar 23 06:03:55 oradb02 kernel: SMP
Mar 23 06:03:55 oradb02 kernel: Modules linked in: hangcheck_timer cpqci(U) mptctl mptbase sg ipmi_si(U) ipmi_devintf(U) ipmi_msghandler(U) md5 ipv6 parport_pc lp parport autofs4 i2c_dev i2c_core ocfs2(U) debugfs(U) ocfs2_dlmfs(U) ocfs2_dlm(U) ocfs2_nodemanager(U) configfs(U) sunrpc 8021q dm_round_robin dm_multipath button battery ac joydev ehci_hcd uhci_hcd bnx2(U) dm_snapshot dm_zero dm_mirror ext3 jbd dm_mod qla2400(U) qla2300(U) qla2xxx(U) qla2xxx_conf(U) cciss(U) sd_mod scsi_mod
Mar 23 06:03:55 oradb02 kernel: CPU: 3
Mar 23 06:03:55 oradb02 kernel: EIP: 0060:[<f8f626a0>] Tainted: P VLI
Mar 23 06:03:55 oradb02 kernel: EFLAGS: 00010246 (2.6.9-55.ELsmp)
Mar 23 06:03:55 oradb02 kernel: EIP is at dlm_drop_lockres_ref+0x24f/0x261 [ocfs2_dlm]
Mar 23 06:03:55 oradb02 kernel: eax: 00000000 ebx: cb802b40 ecx: f7cd9e00 edx: cb802b88
Mar 23 06:03:55 oradb02 kernel: esi: 00000000 edi: f7cd9f03 ebp: 0000001f esp: f7cd9ecc
Mar 23 06:03:55 oradb02 kernel: ds: 007b es: 007b ss: 0068
Mar 23 06:03:55 oradb02 kernel: Process dlm_thread (pid: 13312, threadinfo=f7cd9000 task=f66f8330)
Mar 23 06:03:55 oradb02 kernel: Stack: f12ace00 cb802b40 f58a1000 ffffffea 00000000 1f010000 3030304d 30303030
Mar 23 06:03:55 oradb02 kernel: 30303030 30303030 62353130 62623661 61356261 00626466 00000000 00000000
Mar 23 06:03:55 oradb02 kernel: 00000000 00000000 00000000 00000000 00000000 00000000 cb802b40 cb802b40
Mar 23 06:03:55 oradb02 kernel: Call Trace:
Mar 23 06:03:55 oradb02 kernel: [<f8f7494d>] dlm_purge_lockres+0x121/0x2c9 [ocfs2_dlm]
Mar 23 06:03:55 oradb02 kernel: [<f8f74b72>] dlm_run_purge_list+0x7d/0xc6 [ocfs2_dlm]
Mar 23 06:03:55 oradb02 kernel: [<f8f758ff>] dlm_thread+0x72/0x46b [ocfs2_dlm]
Mar 23 06:03:55 oradb02 kernel: [<c012052d>] autoremove_wake_function+0x0/0x2d
Mar 23 06:03:55 oradb02 kernel: [<c012052d>] autoremove_wake_function+0x0/0x2d
Mar 23 06:03:55 oradb02 kernel: [<f8f7588d>] dlm_thread+0x0/0x46b [ocfs2_dlm]
Mar 23 06:03:55 oradb02 kernel: [<c01342ed>] kthread+0x73/0x9b
Mar 23 06:03:55 oradb02 kernel: [<c013427a>] kthread+0x0/0x9b
Mar 23 06:03:55 oradb02 kernel: [<c01041f5>] kernel_thread_helper+0x5/0xb
Mar 23 06:03:55 oradb02 kernel: Code: 72 40 68 fa 08 00 00 68 f2 64 f7 f8 ff 70 10 ff b1 94 00 00 00 68 81 ab f7 f8 e8 5e 02 1c c7 83 c4 28 8b 44 24 04 e8 f6 64 ff ff <0f> 0b fc 08 8b 8d f7 f8 83 c4 60 89 f0 5b 5e 5f 5d c3 55 57 56
Mar 23 06:03:55 oradb02 kernel: <0>Fatal exception: panic in 5 seconds
Mar 23 04:10:46 oradb02 syslogd 1.4.1: restart.
Mar 23 04:10:46 oradb02 syslog: syslogd startup succeeded
Mar 23 04:10:46 oradb02 kernel: klogd 1.4.1, log source = /proc/kmsg started.
Mar 23 04:10:46 oradb02 kernel: Linux version 2.6.9-55.ELsmp (brewbuilder@ls20-bc2-14.build.redhat.com) (gcc version 3.4.6 20060404 (Red Hat 3.4.6-3)) #1 SMP Fri Apr 20 17:03:35 EDT 2007





Can someone perhaps assist in pinpointing the problem and perhaps solution or even direct me in the correct direction for this?


Thank you in advance.


Machiel

Yes, the best people to assist you in this would be Oracle. There are too many variables to be dealt with here, and since you're in a supported environment (RHEL and Oracle), and you're paying for support from both of them, Oracle would be your best starting point. Especially since this problem didn't start until you did an Oracle upgrade.