LinuxQuestions.org - RAID-5 with multiple disk failure, need help

Hello,

I have a 3x200 GB Linux software RAID-5 system with multiple disk failure. One disk failed, hdc9. Then I did a stupid thing, I rebooted. Then another disk failed, hde7. Now I have a system with a RAID-5 device with multiple disk failure.

My question: is there some miracle that I can do to recover data from /dev/md6?

Thanks beforehand

Regards

Greger ( greger(at)lufk.nu )

System:
* Slackware 10.0
* Kernel 2.4.26
* Maxtor DiamondMax Plus 9 200 GB (6Y200P0) disks
* ABIT AI7 motherboard
* Promise TX2 Ultra133 PCI controller card

Part of my /etc/raidtab:

raiddev /dev/md6
raid-level 5
nr-raid-disks 3
nr-spare-disks 0
persistent-superblock 1
chunk-size 128
parity-algorithm left-symmetric
device /dev/hda9
raid-disk 0
device /dev/hdc9
raid-disk 1
device /dev/hde7
raid-disk 2

Part of /var/log/syslog

Mar 5 20:41:18 vector kernel: hdc: dma_intr: status=0x51 { DriveReady SeekComplete Error }
Mar 5 20:41:18 vector kernel: hdc: dma_intr: error=0x40 { UncorrectableError }, LBAsect=346367672, high=20, low=10823352, sector=322832384
Mar 5 20:41:18 vector kernel: end_request: I/O error, dev 16:09 (hdc), sector 322832384
Mar 5 20:41:18 vector kernel: raid5: Disk failure on hdc9, disabling device. Operation continuing on 2 devices
Mar 5 20:41:18 vector kernel: md6: no spare disk to reconstruct array! -- continuing in degraded mode

Part of /var/log/messages

Mar 5 20:41:18 vector kernel: md: recovery thread got woken up ...
Mar 5 20:41:18 vector kernel: md: updating md6 RAID superblock on device
Mar 5 20:41:18 vector kernel: md: hde7 [events: 00000022]<6>(write) hde7's sb offset: 187374016
Mar 5 20:41:18 vector kernel: md: (skipping faulty hdc9 )
Mar 5 20:41:18 vector kernel: md: hda9 [events: 00000022]<6>(write) hda9's sb offset: 187374016
Mar 5 20:41:18 vector kernel: md: recovery thread finished ...

Part of /var/log/syslog after the reboot

Mar 5 22:12:17 vector kernel: md: superblock update time inconsistency -- using the most recent one
Mar 5 22:12:17 vector kernel: md: kicking non-fresh hdc9 from array!
Mar 5 22:12:17 vector kernel: raid5: md6, not all disks are operational -- trying to recover array
Mar 5 22:12:17 vector kernel: raid5: raid level 5 set md6 active with 2 out of 3 devices, algorithm 2
Mar 5 22:12:17 vector kernel: RAID5 conf printout:
Mar 5 22:12:17 vector kernel: --- rd:3 wd:2 fd:1
Mar 5 22:12:17 vector kernel: disk 0, s:0, o:1, n:0 rd:0 us:1 dev:hda9
Mar 5 22:12:17 vector kernel: disk 1, s:0, o:0, n:1 rd:1 us:1 dev:[dev 00:00]
Mar 5 22:12:17 vector kernel: disk 2, s:0, o:1, n:2 rd:2 us:1 dev:hde7
Mar 5 22:12:17 vector kernel: RAID5 conf printout:
Mar 5 22:12:17 vector kernel: --- rd:3 wd:2 fd:1
Mar 5 22:12:17 vector kernel: disk 0, s:0, o:1, n:0 rd:0 us:1 dev:hda9
Mar 5 22:12:17 vector kernel: disk 1, s:0, o:0, n:1 rd:1 us:1 dev:[dev 00:00]
Mar 5 22:12:17 vector kernel: disk 2, s:0, o:1, n:2 rd:2 us:1 dev:hde7
Mar 5 22:12:17 vector kernel: md6: no spare disk to reconstruct array! -- continuing in degraded mode
Mar 5 22:14:21 vector kernel: hde: dma_intr: status=0x51 { DriveReady SeekComplete Error }
Mar 5 22:14:21 vector kernel: hde: dma_intr: error=0x40 { UncorrectableError }, LBAsect=276373495, high=16, low=7938039
, sector=252838144
Mar 5 22:14:21 vector kernel: end_request: I/O error, dev 21:07 (hde), sector 252838144
Mar 5 22:14:21 vector kernel: raid5: Disk failure on hde7, disabling device. Operation continuing on 1 devices
Mar 5 22:14:21 vector kernel: md6: no spare disk to reconstruct array! -- continuing in degraded mode
Mar 5 22:14:22 vector kernel: journal_bmap: journal block not found at offset 12 on md(9,6)
Mar 5 22:14:22 vector kernel: Aborting journal on device md(9,6).
Mar 5 22:14:22 vector kernel: hde: dma_intr: status=0x51 { DriveReady SeekComplete Error }
Mar 5 22:14:22 vector kernel: hde: dma_intr: error=0x40 { UncorrectableError }, LBAsect=276373494, high=16, low=7938038
, sector=252838152
Mar 5 22:14:22 vector kernel: end_request: I/O error, dev 21:07 (hde), sector 252838152
Mar 5 22:14:24 vector kernel: hde: dma_intr: status=0x51 { DriveReady SeekComplete Error }
Mar 5 22:14:24 vector kernel: hde: dma_intr: error=0x40 { UncorrectableError }, LBAsect=276373494, high=16, low=7938038
, sector=252838160
Mar 5 22:14:24 vector kernel: end_request: I/O error, dev 21:07 (hde), sector 252838160
Mar 5 22:14:25 vector kernel: hde: dma_intr: status=0x51 { DriveReady SeekComplete Error }
Mar 5 22:14:25 vector kernel: hde: dma_intr: error=0x40 { UncorrectableError }, LBAsect=276373494, high=16, low=7938038
, sector=252838168
Mar 5 22:14:25 vector kernel: end_request: I/O error, dev 21:07 (hde), sector 252838168
Mar 5 22:14:27 vector kernel: hde: dma_intr: status=0x51 { DriveReady SeekComplete Error }
Mar 5 22:14:27 vector kernel: hde: dma_intr: error=0x40 { UncorrectableError }, LBAsect=276373494, high=16, low=7938038
, sector=252838176
Mar 5 22:14:27 vector kernel: end_request: I/O error, dev 21:07 (hde), sector 252838176
Mar 5 22:14:28 vector kernel: hde: dma_intr: status=0x51 { DriveReady SeekComplete Error }
Mar 5 22:14:28 vector kernel: hde: dma_intr: error=0x40 { UncorrectableError }, LBAsect=276373494, high=16, low=7938038
, sector=252838184
Mar 5 22:14:28 vector kernel: end_request: I/O error, dev 21:07 (hde), sector 252838184
Mar 5 22:14:29 vector kernel: hde: dma_intr: status=0x51 { DriveReady SeekComplete Error }
Mar 5 22:14:29 vector kernel: hde: dma_intr: error=0x40 { UncorrectableError }, LBAsect=276373494, high=16, low=7938038
, sector=252838192
Mar 5 22:14:29 vector kernel: end_request: I/O error, dev 21:07 (hde), sector 252838192
Mar 5 22:14:33 vector kernel: hde: dma_intr: status=0x51 { DriveReady SeekComplete Error }
Mar 5 22:14:33 vector kernel: hde: dma_intr: error=0x40 { UncorrectableError }, LBAsect=276373496, high=16, low=7938040
, sector=252838200
Mar 5 22:14:33 vector kernel: end_request: I/O error, dev 21:07 (hde), sector 252838200

Part of /var/log/messages after the reboot

Mar 5 22:12:17 vector kernel: md: considering hde7 ...
Mar 5 22:12:17 vector kernel: md: adding hde7 ...
Mar 5 22:12:17 vector kernel: md: adding hdc9 ...
Mar 5 22:12:17 vector kernel: md: adding hda9 ...
Mar 5 22:12:17 vector kernel: md: created md6
Mar 5 22:12:17 vector kernel: md: bind<hda9,1>
Mar 5 22:12:17 vector kernel: md: bind<hdc9,2>
Mar 5 22:12:17 vector kernel: md: bind<hde7,3>
Mar 5 22:12:17 vector kernel: md: running: <hde7><hdc9><hda9>
Mar 5 22:12:17 vector kernel: md: hde7's event counter: 00000029
Mar 5 22:12:17 vector kernel: md: hdc9's event counter: 00000021
Mar 5 22:12:17 vector kernel: md: hda9's event counter: 00000029
Mar 5 22:12:17 vector kernel: md: freshest: hde7
Mar 5 22:12:17 vector kernel: md: unbind<hdc9,2>
Mar 5 22:12:17 vector kernel: md: export_rdev(hdc9)
Mar 5 22:12:17 vector kernel: md6: max total readahead window set to 1024k
Mar 5 22:12:17 vector kernel: md6: 2 data-disks, max readahead per data-disk: 512k
Mar 5 22:12:17 vector kernel: raid5: device hde7 operational as raid disk 2
Mar 5 22:12:17 vector kernel: raid5: device hda9 operational as raid disk 0
Mar 5 22:12:17 vector kernel: raid5: allocated 3284kB for md6
Mar 5 22:12:17 vector kernel: md: updating md6 RAID superblock on device
Mar 5 22:12:17 vector kernel: md: hde7 [events: 0000002a]<6>(write) hde7's sb offset: 187374016
Mar 5 22:12:17 vector kernel: md: recovery thread got woken up ...
Mar 5 22:12:17 vector kernel: md: recovery thread finished ...
Mar 5 22:12:17 vector kernel: md: hda9 [events: 0000002a]<6>(write) hda9's sb offset: 187374016
Mar 5 22:14:21 vector kernel: md: recovery thread got woken up ...
Mar 5 22:14:21 vector kernel: md: updating md6 RAID superblock on device
Mar 5 22:14:21 vector kernel: md: (skipping faulty hde7 )
Mar 5 22:14:21 vector kernel: md: hda9 [events: 0000002b]<6>(write) hda9's sb offset: 187374016
Mar 5 22:14:21 vector kernel: md: recovery thread finished ...