RAID6 rebuild weirdness

mvanhorn · 11-02-2010, 09:46 AM

I am in a weird situation with a raid6 array which has just had a disk replaced. It is composed of 8 disks, sd[b-i], and sdh went bad. I took the system down, replaced the disk, brought it back up, and I can't mount the array (/dev/md0), and I'm not sure what it happening. Here's some output:

Code:

[root@hostname ~]# mdadm --query /dev/md0 
/dev/md0: 0.00KiB raid6 8 devices, 1 spare. Use mdadm --detail for more detail.

How is this possible? Doesn't raid 6, by definition, have *2* spares?

Code:

[root@hostname ~]# mdadm --detail /dev/md0
/dev/md0:
        Version : 00.90.03
  Creation Time : Mon Jun 28 10:46:51 2010
     Raid Level : raid6
    Device Size : 1953511936 (1863.01 GiB 2000.40 GB)
   Raid Devices : 8
  Total Devices : 8
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Tue Nov  2 08:19:52 2010
          State : dirty, degraded
 Active Devices : 7
Working Devices : 8
 Failed Devices : 0
  Spare Devices : 1

     Chunk Size : 64K

           UUID : 6b8b4567:327b23c6:643c9869:66334873
         Events : 0.34591155

    Number   Major   Minor   RaidDevice State
       0       8       17        0      active sync   /dev/sdb1
       1       8       33        1      active sync   /dev/sdc1
       2       8       49        2      active sync   /dev/sdd1
       3       8       65        3      active sync   /dev/sde1
       4       8       81        4      active sync   /dev/sdf1
       5       8       97        5      active sync   /dev/sdg1
       6       8      113        6      spare rebuilding   /dev/sdh1
       7       8      129        7      active sync   /dev/sdi1

If there are 7 disks working, and it's rebuilding the other, why can't I go ahead and mount the array; I've only lost one disk!

Code:

[root@hostname ~]# cat /proc/mdstat
Personalities : [raid6] 
md0 : inactive sdh1[6] sdb1[0] sdi1[7] sdg1[5] sdf1[4] sde1[3] sdd1[2] sdc1[1]
      15628095488 blocks
       
unused devices: <none>

I don't understand why I can't just mount the array. Instead, I get a superblock error:

Code:

root@hostname ~]# mount /dev/md0
mount: /dev/md0: can't read superblock

Can someone please shed some light on what it going on?

Thanks!

mesiol · 11-02-2010, 10:02 AM

Hi,

the RAID cannot be mounted, because it is inactive as mdstat shows.

Did you try the

Code:

mdadm --run /dev/md0

Be careful with all commands in case there is data on your RAID.

mvanhorn · 11-02-2010, 10:07 AM

I've now downloaded a new mdadm, and this shows that the array is active:

Code:

[root@chanute04 mdadm-3.1.4]# mdadm --detail /dev/md0
/dev/md0:
        Version : 0.90
  Creation Time : Mon Jun 28 10:46:51 2010
     Raid Level : raid6
  Used Dev Size : 1953511936 (1863.01 GiB 2000.40 GB)
   Raid Devices : 8
  Total Devices : 8
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Tue Nov  2 08:19:52 2010
          State : active, degraded, Not Started
 Active Devices : 7
Working Devices : 8
 Failed Devices : 0
  Spare Devices : 1

         Layout : left-symmetric
     Chunk Size : 64K

           UUID : 6b8b4567:327b23c6:643c9869:66334873
         Events : 0.34591155

    Number   Major   Minor   RaidDevice State
       0       8       17        0      active sync   /dev/sdb1
       1       8       33        1      active sync   /dev/sdc1
       2       8       49        2      active sync   /dev/sdd1
       3       8       65        3      active sync   /dev/sde1
       4       8       81        4      active sync   /dev/sdf1
       5       8       97        5      active sync   /dev/sdg1
       6       8      113        6      spare rebuilding   /dev/sdh1
       7       8      129        7      active sync   /dev/sdi1

So, why can't I mount it?

And, yes, I have tried --run:

Code:

[root@hostname]# mdadm --run /dev/md0
mdadm: failed to run array /dev/md0: Input/output error

And, yes, the raid does have data on it; I was expecting to still be able to get to the data while it was adding the new drive back in. I don't understand why I can't do so.

mesiol · 11-02-2010, 10:12 AM

Hi,

did they RAID autostart during boot, or did you assemble it manualy? I'm also not sure what creates this behaviour.

Can you stop your raid and start it manually, i can not see any reason why mdstat reports the array as inactive.

mvanhorn · 11-02-2010, 10:20 AM

Quote:

Originally Posted by mesiol

did they RAID autostart during boot, or did you assemble it manualy? I'm also not sure what creates this behaviour.

Can you stop your raid and start it manually, i can not see any reason why mdstat reports the array as inactive.

No, it does not start at boot. I'm trying to start it manually, and that isn't working. I also don't understand why it's showing up as inactive; with (now) 8 disks of an 8-disk raid6 array, it should be just fine.

mvanhorn · 11-02-2010, 10:33 AM

This may help. I just tried to assemble it again, and now it says:

Code:

[root@hostname]# mdadm -A /dev/md0
mdadm: /dev/sdh1 has no superblock - assembly aborted

So, how do I put a superblock on /dev/sdh1 (this is the drive that was replaced). I thought the system would do this automatically?

mesiol · 11-02-2010, 11:22 AM

Hi,

could please post output of

Code:

 fdisk -l /dev/sdh

Do you replace your disk by only replacing the physical hardware? Or do you also replace it with the appropriate fdisk/mdadm commands?

Did you add the disk by creating a software RAID partition with fdisk and than using

Code:

mdadm manage /dev/md0 --add /dev/sdh1

?

Software RAID partitions contains private regions where information about the RAID is stored. This is created during initialization of the RAID. If you replace a disk this information is lost, so it cannot really be taken into the RAID. So you have to tell the RAID to use this disk as an replacement, therefor are commands like --remove and --add.

mvanhorn · 11-03-2010, 07:55 AM

It turns out I had a second disk bad. Or, well, maybe I do. I started checking the output of 'mdadm --examine' on each individual disk in the array, and on a couple of them it showed that a different disk (sdd, rather than sdh) had been removed from the array as well. So, I did a 'mdadm --assemble' without either sdd or sdh, and it took right off. Then, I was able to add sdh back in, and it's rebuilding now.

What's odd is that I've now run what I thought was the bad sdh through the Western Digital diagnostics, and everything seems okay with this disk. So, I'm going to try replacing sdd with this (formerly sdh) disk, and see what happens.

Thanks for your help!

mesiol · 11-03-2010, 08:05 AM

Hi,

fine to hear that your rebuild is working. Wish you success on your problem.