Redhat 5 Crashing after adding extra disk.

Blushus · 03-01-2011, 11:51 PM

Hi All,

I'm running Redhat 5 which is running as an email server with Lotus Domino 8.5. Its a VM on ESX 3.5. The server was running out of space to so I added a new hardrive and mounted it to /domino/mail2, everything was fine for a couple of days, but then the machine became unresponsive, I couldn't even see anything on the console screen so I rebooted, it then did a file check and everything came up fine. This has now happened 2 more times over the past few days, I haven't seen any evidence that the extra disk is the problem but if I can't find anything else I will remove it and see if the problem goes away. There isn't really a consistent time it occurs.

Here's the output of the logs from the last time it happened if anybody who know more about this stuff then me ( which isn't a lot) wants to take a quick look as see if the problem is obvious or what I could be trying. Much appreciated.

Mar 2 00:17:30 DOM2 kernel: mptscsih: ioc0: attempting task abort! (sc=ce6afbc0)
Mar 2 00:17:30 DOM2 kernel: sd 0:0:0:0:
Mar 2 00:17:30 DOM2 kernel: command: Write(10): 2a 00 01 3c 8c eb 00 00 08 00
Mar 2 00:17:30 DOM2 kernel: mptscsih: ioc0: task abort: SUCCESS (sc=ce6afbc0)
Mar 2 00:17:30 DOM2 kernel: sd 0:0:0:0: timing out command, waited 360s
Mar 2 00:17:49 DOM2 kernel: INFO: task server:3878 blocked for more than 120 seconds.
Mar 2 00:17:49 DOM2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 2 00:17:49 DOM2 kernel: server D 00005C63 2372 3878 2822 3879 3877 (NOTLB)
Mar 2 00:17:59 DOM2 kernel: f4a22f4c 00000086 52cfd60e 00005c63 f4a22f54 c047e2a1 f4a22ef4 00000009
Mar 2 00:18:00 DOM2 kernel: c43a6000 52d76833 00005c63 00079225 00000002 c43a610c c40e8588 f60433c0
Mar 2 00:18:00 DOM2 kernel: e6a4bd20 f2a7a2c0 00000000 00000000 f7e623c0 f5f763c0 c40e8a44 00000000
Mar 2 00:18:00 DOM2 kernel: Call Trace:
Mar 2 00:18:06 DOM2 kernel: [<c047e2a1>] cp_new_stat64+0xfc/0x10e
Mar 2 00:18:06 DOM2 kernel: [<c061f1dd>] __mutex_lock_slowpath+0x4d/0x7c
Mar 2 00:18:08 DOM2 kernel: [<c061f21b>] .text.lock.mutex+0xf/0x14
Mar 2 00:18:08 DOM2 kernel: [<c0486e61>] vfs_readdir+0x4b/0x8d
Mar 2 00:18:08 DOM2 kernel: [<c0486c98>] filldir64+0x0/0xc5
Mar 2 00:18:08 DOM2 kernel: [<c0486f06>] sys_getdents64+0x63/0xa5
Mar 2 00:18:08 DOM2 kernel: [<c0404f4b>] syscall_call+0x7/0xb
Mar 2 00:18:13 DOM2 kernel: =======================
Mar 2 00:19:35 DOM2 kernel: INFO: task server:3878 blocked for more than 120 seconds.
Mar 2 00:19:35 DOM2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 2 00:19:35 DOM2 kernel: server D 00005C63 2372 3878 2822 3879 3877 (NOTLB)
Mar 2 00:19:35 DOM2 kernel: f4a22f4c 00000086 52cfd60e 00005c63 f4a22f54 c047e2a1 f4a22ef4 00000009
Mar 2 00:19:35 DOM2 kernel: c43a6000 52d76833 00005c63 00079225 00000002 c43a610c c40e8588 f60433c0
Mar 2 00:19:35 DOM2 kernel: e6a4bd20 f2a7a2c0 00000000 00000000 f7e623c0 f5f763c0 c40e8a44 00000000
Mar 2 00:19:35 DOM2 kernel: Call Trace:
Mar 2 00:19:35 DOM2 kernel: [<c047e2a1>] cp_new_stat64+0xfc/0x10e
Mar 2 00:19:35 DOM2 kernel: [<c061f1dd>] __mutex_lock_slowpath+0x4d/0x7c
Mar 2 00:19:35 DOM2 kernel: [<c061f21b>] .text.lock.mutex+0xf/0x14
Mar 2 00:19:35 DOM2 kernel: [<c0486e61>] vfs_readdir+0x4b/0x8d
Mar 2 00:19:35 DOM2 kernel: [<c0486c98>] filldir64+0x0/0xc5
Mar 2 00:19:35 DOM2 kernel: [<c0486f06>] sys_getdents64+0x63/0xa5
Mar 2 00:19:35 DOM2 kernel: [<c0404f4b>] syscall_call+0x7/0xb
Mar 2 00:19:35 DOM2 kernel: =======================
Mar 2 00:19:35 DOM2 kernel: INFO: task server:3878 blocked for more than 120 seconds.
Mar 2 00:19:35 DOM2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 2 00:19:35 DOM2 kernel: server D 00005C63 2372 3878 2822 3879 3877 (NOTLB)
Mar 2 00:19:35 DOM2 kernel: f4a22f4c 00000086 52cfd60e 00005c63 f4a22f54 c047e2a1 f4a22ef4 00000009
Mar 2 00:19:35 DOM2 kernel: c43a6000 52d76833 00005c63 00079225 00000002 c43a610c c40e8588 f60433c0
Mar 2 00:19:35 DOM2 kernel: e6a4bd20 f2a7a2c0 00000000 00000000 f7e623c0 f5f763c0 c40e8a44 00000000
Mar 2 00:19:35 DOM2 kernel: Call Trace:
Mar 2 00:19:35 DOM2 kernel: [<c047e2a1>] cp_new_stat64+0xfc/0x10e
Mar 2 00:19:35 DOM2 kernel: [<c061f1dd>] __mutex_lock_slowpath+0x4d/0x7c
Mar 2 00:19:35 DOM2 kernel: [<c061f21b>] .text.lock.mutex+0xf/0x14
Mar 2 00:19:35 DOM2 kernel: [<c0486e61>] vfs_readdir+0x4b/0x8d
Mar 2 00:19:35 DOM2 kernel: [<c0486c98>] filldir64+0x0/0xc5
Mar 2 00:19:36 DOM2 kernel: [<c0486f06>] sys_getdents64+0x63/0xa5
Mar 2 00:19:36 DOM2 kernel: [<c0404f4b>] syscall_call+0x7/0xb
Mar 2 00:19:36 DOM2 kernel: =======================
Mar 2 00:19:36 DOM2 kernel: INFO: task server:3878 blocked for more than 120 seconds.
Mar 2 00:19:36 DOM2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 2 00:19:36 DOM2 kernel: server D 00005C63 2372 3878 2822 3879 3877 (NOTLB)
Mar 2 00:19:36 DOM2 kernel: f4a22f4c 00000086 52cfd60e 00005c63 f4a22f54 c047e2a1 f4a22ef4 00000009
Mar 2 00:19:36 DOM2 kernel: c43a6000 52d76833 00005c63 00079225 00000002 c43a610c c40e8588 f60433c0
Mar 2 00:19:36 DOM2 kernel: e6a4bd20 f2a7a2c0 00000000 00000000 f7e623c0 f5f763c0 c40e8a44 00000000
Mar 2 00:19:36 DOM2 kernel: Call Trace:
Mar 2 00:19:36 DOM2 kernel: [<c047e2a1>] cp_new_stat64+0xfc/0x10e
Mar 2 00:19:36 DOM2 kernel: [<c061f1dd>] __mutex_lock_slowpath+0x4d/0x7c
Mar 2 00:19:36 DOM2 kernel: [<c061f21b>] .text.lock.mutex+0xf/0x14
Mar 2 00:19:36 DOM2 kernel: [<c0486e61>] vfs_readdir+0x4b/0x8d
Mar 2 00:19:36 DOM2 kernel: [<c0486c98>] filldir64+0x0/0xc5
Mar 2 00:19:36 DOM2 kernel: [<c0486f06>] sys_getdents64+0x63/0xa5
Mar 2 00:19:36 DOM2 kernel: [<c0404f4b>] syscall_call+0x7/0xb
Mar 2 00:19:36 DOM2 kernel: =======================
Mar 2 00:19:36 DOM2 kernel: INFO: task server:3878 blocked for more than 120 seconds.
Mar 2 00:19:36 DOM2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 2 00:19:36 DOM2 kernel: server D 00005C63 2372 3878 2822 3879 3877 (NOTLB)
Mar 2 00:19:36 DOM2 kernel: f4a22f4c 00000086 52cfd60e 00005c63 f4a22f54 c047e2a1 f4a22ef4 00000009
Mar 2 00:19:36 DOM2 kernel: c43a6000 52d76833 00005c63 00079225 00000002 c43a610c c40e8588 f60433c0
Mar 2 00:19:36 DOM2 kernel: e6a4bd20 f2a7a2c0 00000000 00000000 f7e623c0 f5f763c0 c40e8a44 00000000
Mar 2 00:19:36 DOM2 kernel: Call Trace:
Mar 2 00:19:36 DOM2 kernel: [<c047e2a1>] cp_new_stat64+0xfc/0x10e
Mar 2 00:19:36 DOM2 kernel: [<c061f1dd>] __mutex_lock_slowpath+0x4d/0x7c
Mar 2 00:19:37 DOM2 kernel: [<c061f21b>] .text.lock.mutex+0xf/0x14
Mar 2 00:19:37 DOM2 kernel: [<c0486e61>] vfs_readdir+0x4b/0x8d
Mar 2 00:19:37 DOM2 kernel: [<c0486c98>] filldir64+0x0/0xc5
Mar 2 00:19:37 DOM2 kernel: [<c0486f06>] sys_getdents64+0x63/0xa5
Mar 2 00:19:37 DOM2 kernel: [<c0404f4b>] syscall_call+0x7/0xb
Mar 2 00:19:37 DOM2 kernel: =======================
Mar 2 00:19:37 DOM2 kernel: INFO: task server:3878 blocked for more than 120 seconds.
Mar 2 00:19:37 DOM2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 2 00:19:37 DOM2 kernel: server D 00005C63 2372 3878 2822 3879 3877 (NOTLB)
Mar 2 00:19:38 DOM2 kernel: f4a22f4c 00000086 52cfd60e 00005c63 f4a22f54 c047e2a1 f4a22ef4 00000009
Mar 2 00:19:38 DOM2 kernel: c43a6000 52d76833 00005c63 00079225 00000002 c43a610c c40e8588 f60433c0
Mar 2 00:19:38 DOM2 kernel: e6a4bd20 f2a7a2c0 00000000 00000000 f7e623c0 f5f763c0 c40e8a44 00000000
Mar 2 00:19:38 DOM2 kernel: Call Trace:
Mar 2 00:19:38 DOM2 kernel: [<c047e2a1>] cp_new_stat64+0xfc/0x10e
Mar 2 00:19:38 DOM2 kernel: [<c061f1dd>] __mutex_lock_slowpath+0x4d/0x7c
Mar 2 00:19:38 DOM2 kernel: [<c061f21b>] .text.lock.mutex+0xf/0x14
Mar 2 00:19:38 DOM2 kernel: [<c0486e61>] vfs_readdir+0x4b/0x8d
Mar 2 00:19:38 DOM2 kernel: [<c0486c98>] filldir64+0x0/0xc5
Mar 2 00:19:38 DOM2 kernel: [<c0486f06>] sys_getdents64+0x63/0xa5
Mar 2 00:19:38 DOM2 kernel: [<c0404f4b>] syscall_call+0x7/0xb
Mar 2 00:19:38 DOM2 kernel: =======================
Mar 2 00:19:38 DOM2 kernel: INFO: task server:3878 blocked for more than 120 seconds.
Mar 2 00:19:38 DOM2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 2 00:19:38 DOM2 kernel: server D 00005C63 2372 3878 2822 3879 3877 (NOTLB)
Mar 2 00:21:31 DOM2 kernel: f4a22f4c 00000086 52cfd60e 00005c63 f4a22f54 c047e2a1 f4a22ef4 00000009
Mar 2 00:26:43 DOM2 kernel: c43a6000 52d76833 00005c63 00079225 00000002 c43a610c c40e8588 f60433c0
Mar 2 00:26:43 DOM2 kernel: e6a4bd20 f2a7a2c0 00000000 00000000 f7e623c0 f5f763c0 c40e8a44 00000000
Mar 2 00:26:43 DOM2 kernel: Call Trace:
Mar 2 00:26:43 DOM2 kernel: [<c047e2a1>] cp_new_stat64+0xfc/0x10e
Mar 2 00:26:43 DOM2 kernel: [<c061f1dd>] __mutex_lock_slowpath+0x4d/0x7c
Mar 2 00:26:43 DOM2 kernel: [<c061f21b>] .text.lock.mutex+0xf/0x14
Mar 2 00:26:43 DOM2 kernel: [<c0486e61>] vfs_readdir+0x4b/0x8d
Mar 2 00:26:43 DOM2 kernel: [<c0486c98>] filldir64+0x0/0xc5
Mar 2 00:26:43 DOM2 kernel: [<c0486f06>] sys_getdents64+0x63/0xa5
Mar 2 00:26:43 DOM2 kernel: [<c0404f4b>] syscall_call+0x7/0xb
Mar 2 00:26:43 DOM2 kernel: =======================
Mar 2 00:26:43 DOM2 kernel: INFO: task server:3878 blocked for more than 120 seconds.
Mar 2 00:26:43 DOM2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 2 00:26:43 DOM2 kernel: server D 00005C63 2372 3878 2822 3879 3877 (NOTLB)
Mar 2 00:26:43 DOM2 kernel: f4a22f4c 00000086 52cfd60e 00005c63 f4a22f54 c047e2a1 f4a22ef4 00000009
Mar 2 00:26:43 DOM2 kernel: c43a6000 52d76833 00005c63 00079225 00000002 c43a610c c40e8588 f60433c0
Mar 2 00:26:43 DOM2 kernel: e6a4bd20 f2a7a2c0 00000000 00000000 f7e623c0 f5f763c0 c40e8a44 00000000
Mar 2 00:26:43 DOM2 kernel: Call Trace:
Mar 2 00:26:43 DOM2 kernel: [<c047e2a1>] cp_new_stat64+0xfc/0x10e
Mar 2 00:26:43 DOM2 kernel: [<c061f1dd>] __mutex_lock_slowpath+0x4d/0x7c
Mar 2 00:26:43 DOM2 kernel: [<c061f21b>] .text.lock.mutex+0xf/0x14
Mar 2 00:26:43 DOM2 kernel: [<c0486e61>] vfs_readdir+0x4b/0x8d
Mar 2 00:27:11 DOM2 kernel: [<c0486c98>] filldir64+0x0/0xc5
Mar 2 00:27:11 DOM2 kernel: [<c0486f06>] sys_getdents64+0x63/0xa5
Mar 2 00:27:11 DOM2 kernel: [<c0404f4b>] syscall_call+0x7/0xb
Mar 2 00:27:11 DOM2 kernel: =======================
Mar 2 00:27:11 DOM2 kernel: INFO: task server:3878 blocked for more than 120 seconds.
Mar 2 00:27:11 DOM2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 2 00:27:11 DOM2 kernel: server D 00005C63 2372 3878 2822 3879 3877 (NOTLB)
Mar 2 00:27:11 DOM2 kernel: f4a22f4c 00000086 52cfd60e 00005c63 f4a22f54 c047e2a1 f4a22ef4 00000009
Mar 2 00:27:14 DOM2 kernel: c43a6000 52d76833 00005c63 00079225 00000002 c43a610c c40e8588 f60433c0
Mar 2 00:27:14 DOM2 kernel: e6a4bd20 f2a7a2c0 00000000 00000000 f7e623c0 f5f763c0 c40e8a44 00000000
Mar 2 00:27:14 DOM2 kernel: Call Trace:
Mar 2 00:27:14 DOM2 kernel: [<c047e2a1>] cp_new_stat64+0xfc/0x10e
Mar 2 00:27:14 DOM2 kernel: [<c061f1dd>] __mutex_lock_slowpath+0x4d/0x7c
Mar 2 00:27:14 DOM2 kernel: [<c061f21b>] .text.lock.mutex+0xf/0x14
Mar 2 00:27:14 DOM2 kernel: [<c0486e61>] vfs_readdir+0x4b/0x8d
Mar 2 00:27:14 DOM2 kernel: [<c0486c98>] filldir64+0x0/0xc5
Mar 2 00:27:14 DOM2 kernel: [<c0486f06>] sys_getdents64+0x63/0xa5
Mar 2 00:27:14 DOM2 kernel: [<c0404f4b>] syscall_call+0x7/0xb
Mar 2 00:27:14 DOM2 kernel: =======================
Mar 2 00:27:25 DOM2 kernel: INFO: task server:3878 blocked for more than 120 seconds.
Mar 2 00:27:55 DOM2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 2 00:27:57 DOM2 kernel: server D 00005C63 2372 3878 2822 3879 3877 (NOTLB)
Mar 2 00:27:57 DOM2 kernel: f4a22f4c 00000086 52cfd60e 00005c63 f4a22f54 c047e2a1 f4a22ef4 00000009
Mar 2 00:27:57 DOM2 kernel: c43a6000 52d76833 00005c63 00079225 00000002 c43a610c c40e8588 f60433c0
Mar 2 00:27:57 DOM2 kernel: e6a4bd20 f2a7a2c0 00000000 00000000 f7e623c0 f5f763c0 c40e8a44 00000000
Mar 2 00:27:57 DOM2 kernel: Call Trace:
Mar 2 00:27:57 DOM2 kernel: [<c047e2a1>] cp_new_stat64+0xfc/0x10e
Mar 2 00:27:57 DOM2 kernel: [<c061f1dd>] __mutex_lock_slowpath+0x4d/0x7c
Mar 2 00:27:57 DOM2 kernel: [<c061f21b>] .text.lock.mutex+0xf/0x14
Mar 2 00:27:57 DOM2 kernel: [<c0486e61>] vfs_readdir+0x4b/0x8d
Mar 2 00:27:57 DOM2 kernel: [<c0486c98>] filldir64+0x0/0xc5
Mar 2 00:27:57 DOM2 kernel: [<c0486f06>] sys_getdents64+0x63/0xa5
Mar 2 00:27:57 DOM2 kernel: [<c0404f4b>] syscall_call+0x7/0xb
Mar 2 00:27:57 DOM2 kernel: =======================
Mar 2 00:36:14 DOM2 kernel: set_rtc_mmss: can't update from 2 to 58
Mar 2 00:36:15 DOM2 kernel: set_rtc_mmss: can't update from 3 to 59
Mar 2 01:21:41 DOM2 kernel: hda: lost interrupt
Mar 2 01:21:41 DOM2 kernel: ide-cd: cmd 0x3 timed out
Mar 2 01:21:41 DOM2 kernel: hda: lost interrupt
Mar 2 01:21:41 DOM2 kernel: ide-cd: cmd 0x3 timed out
Mar 2 01:21:41 DOM2 kernel: hda: lost interrupt
Mar 2 01:38:06 DOM2 kernel: hda: lost interrupt
Mar 2 01:38:06 DOM2 kernel: hda: lost interrupt
Mar 2 01:38:06 DOM2 kernel: hda: cdrom_pc_intr: The drive appears confused (ireason = 0x01). Trying to recover by ending request.
Mar 2 01:39:15 DOM2 kernel: hda: status error: status=0x18 { SeekComplete DataRequest }
Mar 2 01:39:15 DOM2 kernel: ide: failed opcode was: unknown
Mar 2 01:39:15 DOM2 kernel: hda: drive not ready for command
Mar 2 01:39:15 DOM2 kernel: hda: status error: status=0x18 { SeekComplete DataRequest }
Mar 2 01:39:15 DOM2 kernel: ide: failed opcode was: unknown
Mar 2 01:39:15 DOM2 kernel: hda: drive not ready for command
Mar 2 09:20:51 DOM2 syslogd 1.4.1: restart.

MensaWater · 03-02-2011, 08:08 AM

The messages about /dev/hda, ide and CDROM lead me to believe your issue may be that your new drive is an IDE (or ATA) and you added it to the system without setting master/slave designation on it. (Or more likely it is set to same as the existing device which was a CDROM.

So the questions that arise:
1) What was the original hard drive? (IDE/ATA?, SCSI? SAS? USB?)
2) What is the new hard drive?
3) If both are IDE/ATA do you have more than one IDE/ATA bus in the system? If so have you checked master/slave designations?
4) If both are SCSI have you verified each has its own SCSI ID and that it is not in conflict with the ID on the SCSI Controller HBA?
5) Did you add another HBA to the system to support this drive? If so it might be the HBA causing conflicts. (Years ago I had a system continually have hard drive issues when I added a new RS232c serial adapter - it didn't make sense that it impacted the hard drive but removing it solved the issue.)

Check out this link for more info on IDE stuff:
http://www.buildyourown.org.uk/pc-ar...configuration/

Blushus · 03-03-2011, 05:25 AM

Hi Thanks for the reply,

Its a VM but the Hard Drives are presented as SCSI

Original Hard Drive: SCSI0:1
New Hard Drive SCSI0:2
CDROM Drive IDE0:0

I wouldn't have thought so but your right it does look like it could be a conflict with the CD drive, I'm going to try and remove the CD drive from the VM and see if that makes a difference. It seems to crash about every 2 days, so I will hold my breath until then..