Hi adiehl,
Sounds like we may both be in the same (sinking?) boat and neither of us have been rescued yet. I was going to simply post a link to all my sordid details over on Linux Forums, but I'm not allowed, so I'll repost them here.
You might want to step through and see how closely they match your details. Maybe we can gang up on this at least...
I have
got to get this array back up today -- the natives are getting restless...
-cw-
Post 1:
Ok, I'm a Linux software raid veteran and I have the scars to prove it (google for mddump if you're bored), but that's not doing me much good now. I'm at the end of my rope... er... SATA cable. Help? Please??
The subject platform is a PC running FC5 (Fedora Core 5, patched latest) with eight 400gb SATA drives (/dev/sd[b-i]1) assembled into a RAID6 md0 device. Originally built with mdadm. No LVM or other exotics. /dev/md0 is a /data filesystem, nothing there needed at boot time. It's been humming along nicely for months.
Then... This morning I found that /dev/sdb1 had been kicked out of the array and there was the requisite screaming in /var/log/messages about failed read/writes, SMART errors, highly miffed SATA controllers, etc., all associated with /dev/sdb1. (It appears to have been a temporary failure -- badblocks found no problems.) Tried shutting the system down cleanly, which didn't seem to be working, so finally crossed my fingers and hit the reset button.
No surprise, it booted back up refusing to assemble the array. More specfically:
Code:
Nov 27 19:03:52 ornery kernel: md: bind<sdb1>
Nov 27 19:03:52 ornery kernel: md: bind<sdd1>
Nov 27 19:03:52 ornery kernel: md: bind<sde1>
Nov 27 19:03:52 ornery kernel: md: bind<sdf1>
Nov 27 19:03:52 ornery kernel: md: bind<sdg1>
Nov 27 19:03:52 ornery kernel: md: bind<sdh1>
Nov 27 19:03:52 ornery kernel: md: bind<sdi1>
Nov 27 19:03:52 ornery kernel: md: bind<sdc1>
Nov 27 19:03:52 ornery kernel: md: kicking non-fresh sdb1 from array!
Nov 27 19:03:52 ornery kernel: md: unbind<sdb1>
Nov 27 19:03:52 ornery kernel: md: export_rdev(sdb1)
Nov 27 19:03:52 ornery kernel: md: md0: raid array is not clean -- starting back
ground reconstruction
Nov 27 19:03:52 ornery kernel: raid5: device sdc1 operational as raid disk 1
Nov 27 19:03:52 ornery kernel: raid5: device sdi1 operational as raid disk 7
Nov 27 19:03:52 ornery kernel: raid5: device sdh1 operational as raid disk 6
Nov 27 19:03:52 ornery kernel: raid5: device sdg1 operational as raid disk 5
Nov 27 19:03:52 ornery kernel: raid5: device sdf1 operational as raid disk 4
Nov 27 19:03:52 ornery kernel: raid5: device sde1 operational as raid disk 3
Nov 27 19:03:52 ornery kernel: raid5: device sdd1 operational as raid disk 2
Nov 27 19:03:52 ornery kernel: raid5: cannot start dirty degraded array for md0
Nov 27 19:03:52 ornery kernel: RAID5 conf printout:
Nov 27 19:03:52 ornery kernel: --- rd:8 wd:7 fd:1
Nov 27 19:03:52 ornery kernel: disk 1, o:1, dev:sdc1
Nov 27 19:03:52 ornery kernel: disk 2, o:1, dev:sdd1
Nov 27 19:03:52 ornery kernel: disk 3, o:1, dev:sde1
Nov 27 19:03:52 ornery kernel: disk 4, o:1, dev:sdf1
Nov 27 19:03:52 ornery kernel: disk 5, o:1, dev:sdg1
Nov 27 19:03:52 ornery kernel: disk 6, o:1, dev:sdh1
Nov 27 19:03:52 ornery kernel: disk 7, o:1, dev:sdi1
Nov 27 19:03:52 ornery kernel: raid5: failed to run raid set md0
Nov 27 19:03:52 ornery kernel: md: pers->run() failed ...
Code:
[root@ornery ~]# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : inactive sdc1[1] sdi1[7] sdh1[6] sdg1[5] sdf1[4] sde1[3] sdd1[2]
2734961152 blocks
unused devices: <none>
Attempts to force assembly fail:
Code:
[root@ornery ~]# mdadm -S /dev/md0
[root@ornery ~]# mdadm --assemble --force --scan /dev/md0
mdadm: failed to RUN_ARRAY /dev/md0: Input/output error
Leaving out the bad drive:
Code:
[root@ornery ~]# mdadm -S /dev/md0
[root@ornery ~]# mdadm --assemble --force /dev/md0 /dev/sd[c-i]1
mdadm: failed to RUN_ARRAY /dev/md0: Input/output error
[root@ornery ~]# mdadm -S /dev/md0
[root@ornery ~]# mdadm --assemble --force --run /dev/md0 /dev/sd[c-i]1
mdadm: failed to RUN_ARRAY /dev/md0: Input/output error
Trying to fail or remove the bad drive doesn't work either:
Code:
[root@ornery ~]# mdadm -f /dev/md0 /dev/sdb1
mdadm: set device faulty failed for /dev/sdb1: No such device
[root@ornery ~]# mdadm -r /dev/md0 /dev/sdb1
mdadm: hot remove failed for /dev/sdb1: No such device
A quick check of the event counters shows that only /dev/sdb is stale:
Code:
[root@ornery ~]# mdadm -E /dev/sd[b-i]1 | grep Event
Events : 0.851758
Events : 0.854919
Events : 0.854919
Events : 0.854919
Events : 0.854919
Events : 0.854919
Events : 0.854919
Events : 0.854919
Here's a full examine from one of the good drives:
Code:
[root@ornery ~]# mdadm -E /dev/sdc1
/dev/sdc1:
Magic : a92b4efc
Version : 00.90.03
UUID : d57cea81:3be21b7d:183a67d9:782c3329
Creation Time : Tue Mar 21 11:14:56 2006
Raid Level : raid6
Device Size : 390708736 (372.61 GiB 400.09 GB)
Array Size : 2344252416 (2235.65 GiB 2400.51 GB)
Raid Devices : 8
Total Devices : 8
Preferred Minor : 0
Update Time : Mon Nov 27 10:10:36 2006
State : active
Active Devices : 7
Working Devices : 7
Failed Devices : 0
Spare Devices : 0
Checksum : ebd6e3a8 - correct
Events : 0.854919
Number Major Minor RaidDevice State
this 1 8 33 1 active sync /dev/sdc1
0 0 0 0 0 removed
1 1 8 33 1 active sync /dev/sdc1
2 2 8 49 2 active sync /dev/sdd1
3 3 8 65 3 active sync /dev/sde1
4 4 8 81 4 active sync /dev/sdf1
5 5 8 97 5 active sync /dev/sdg1
6 6 8 113 6 active sync /dev/sdh1
7 7 8 129 7 active sync /dev/sdi1
And detail for the array:
Code:
[root@ornery ~]# mdadm -D /dev/md0
/dev/md0:
Version : 00.90.03
Creation Time : Tue Mar 21 11:14:56 2006
Raid Level : raid6
Device Size : 390708736 (372.61 GiB 400.09 GB)
Raid Devices : 8
Total Devices : 7
Preferred Minor : 0
Persistence : Superblock is persistent
Update Time : Mon Nov 27 10:10:36 2006
State : active, degraded
Active Devices : 7
Working Devices : 7
Failed Devices : 0
Spare Devices : 0
Chunk Size : 256K
UUID : d57cea81:3be21b7d:183a67d9:782c3329
Events : 0.854919
Number Major Minor RaidDevice State
9421816 0 0 1912995864 removed
1 8 33 1 active sync /dev/sdc1
2 8 49 2 active sync /dev/sdd1
3 8 65 3 active sync /dev/sde1
4 8 81 4 active sync /dev/sdf1
5 8 97 5 active sync /dev/sdg1
6 8 113 6 active sync /dev/sdh1
7 8 129 7 active sync /dev/sdi1
So I've obviously got a degraded array. Where does the "dirty" part come in? Why can't I simply force this thing back together in active degraded mode with 7 drives and then add a fresh /dev/sdb1?
I know as a last resort I can create a "new" array over my old one and as long as I get everything juuuuust right, it'll work, but that seems a rather drastic solution to what should be a trivial (and all to common) situation -- dealing with a single failed drive. I mean... I run RAID6 to provide a little extra protection, not to slam into these kinds of brick walls. Heck, I might as well run RAID0! ARGH!!! Ok... ok... I'll calm down.
FWIW, here's my mdadm.conf:
Code:
[root@ornery ~]# grep -v '^#' /etc/mdadm.conf
DEVICE /dev/sd[bcdefghi]1
ARRAY /dev/md0 UUID=d57cea81:3be21b7d:183a67d9:782c3329
MAILADDR root
Have I missed something obvious? Thanks in advance for any clues...
Followup Post:
Ok, done a bit more poking around... I tried zeroing out the superblock on the failed device and adding it back into the array. It just sat there looking stupid. The status of the new drive became "sync", the array status remained inactive, and no resync took place:
Code:
[root@ornery ~]# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : inactive sdb1[0](S) sdc1[1] sdi1[7] sdh1[6] sdg1[5] sdf1[4] sde1[3] sdd1[2]
3125669888 blocks
unused devices: <none>
Another thing I noticed was the new drive didn't fill the slot for the missing drive, but instead occupied a new slot. Here's a detail for the array:
Code:
[root@ornery ~]# mdadm --detail /dev/md0
/dev/md0:
Version : 00.90.03
Creation Time : Tue Mar 21 11:14:56 2006
Raid Level : raid6
Device Size : 390708736 (372.61 GiB 400.09 GB)
Raid Devices : 8
Total Devices : 8
Preferred Minor : 0
Persistence : Superblock is persistent
Update Time : Mon Nov 27 10:10:36 2006
State : active
Active Devices : 8
Working Devices : 8
Failed Devices : 0
Spare Devices : 0
Chunk Size : 256K
UUID : d57cea81:3be21b7d:183a67d9:782c3329
Events : 0.854919
Number Major Minor RaidDevice State
4150256 0 0 1912995872 removed
1 8 33 1 active sync /dev/sdc1
2 8 49 2 active sync /dev/sdd1
3 8 65 3 active sync /dev/sde1
4 8 81 4 active sync /dev/sdf1
5 8 97 5 active sync /dev/sdg1
6 8 113 6 active sync /dev/sdh1
7 8 129 7 active sync /dev/sdi1
0 8 17 - active sync /dev/sdb1
It's like it's just adding the new /dev/sdb1 in as a spare or something. My hunch is that the problem stems from the superblock indicating that the bad device is simply "removed" rather than failed. Yet trying to fail the device... well, failed.
Barring any sudden insights from my fellow Linuxens, it's looking like I have another romp with mddump looming in my future. By my reckoning, I would need to set the SB's to indicate that device 0's status is failed rather than removed, and set the counters to indicate 1 failed device and 7 active/working devices.
If anyone has suggestions, feel free to jump in at any time!! :-)