LinuxQuestions.org - RAID1 array rebuild fails at 99.9% recovery

I am running SuSE 10.1 with kernel 2.6.16.13-4-smp.

I have 4 SCSI drives. /dev/sda and /dev/sdb are partitioned and RAID1-arrayed into /dev/md0 /dev/md1 /dev/dm2 and /dev/dm3. /dev/sdc and /dev/sdd only have 1 partition each and form /dev/md4.

For some reason I don't understand /dev/sdd and /dev/sdb are not actually in the arrays. The system works fine like this but I want to have mirroring for redundancy. Here is /proc/mdstat:

Code:

Personalities : [raid1]

md4 : active raid1 sdd1[2](F) sdc1[0]

      312560512 blocks [2/1] [U_]

      [===================>.]  recovery = 99.9% (312559296/312560512) finish=0.0min speed=17772K/sec



md3 : active raid1 sda5[0]

      285193792 blocks [2/1] [U_]



md0 : active raid1 sda1[0]

      104320 blocks [2/1] [U_]



md2 : active raid1 sda3[0]

      20972736 blocks [2/1] [U_]



md1 : active raid1 sda2[0]

      6297408 blocks [2/1] [U_]



unused devices: <none>

As you can see, I have tried to add /dev/sdd. As the recovery progressed there were no problems, but once it got to 99.9% it froze up. Now any operation involving /dev/md4 just hangs. This includes any file access or mdadm-related query. In particular removing it from the array does not work because it says the device is busy. Additionally the computer freezes randomly for a few seconds every minute, which did not happen before I "added" /dev/sdd to /dev/md4.

If I try to reboot with shutdown -r the computer hangs, I think maybe when it is trying to unmount /dev/md4. I then have to hit the power button or the reset button. It reboots OK and even runs OK for a few minutes while it tries to recover the array. Once it gets to 99.9% recovered, though, the hanging starts all over again. The only way to break the cycle is to unplug the hard drive. Then the computer runs great again, with no hanging, except that I am back where I started, with no mirroring.

I checked /var/log/messages and see error messages such as the following:

Code:

Jun  5 22:05:53 innateimmunity kernel: ata4: command 0x35 timeout, stat 0xd0 host_stat 0x21

Jun  5 22:05:53 innateimmunity kernel: ata4: translated ATA stat/err 0xd0/00 to SCSI SK/ASC/ASCQ 0xb/47/00

Jun  5 22:05:53 innateimmunity kernel: ata4: status=0xd0 { Busy }

Jun  5 22:05:53 innateimmunity kernel: sd 3:0:0:0: SCSI error: return code = 0x8000002

Jun  5 22:05:53 innateimmunity kernel: sdd: Current: sense key: Aborted Command

Jun  5 22:05:53 innateimmunity kernel:    Additional sense: Scsi parity error

Jun  5 22:05:53 innateimmunity kernel: end_request: I/O error, dev sdd, sector 191836447

They appear roughly once per minute, with the sector number is increasing by 8 each minute. If I reboot and recover to 99.9% the same thing happens, but the sector number might be completely different. I have no idea what these messages mean.

/dev/sdb appears to suffer from exactly the same problem.

I have tried replacing the hard drive but this doesn't help. I also ran SeaTools on both /dev/sdb and /dev/sdd and both drives passed the LONG TEST. So I don't think there is anything physically wrong with the drives.

I would greatly appreciate anyone's thoughts on how to fix this!