mdadm issues

lecnt · 02-20-2009, 07:02 AM

I have/had a RAID5 array with four 250GB drives. As a result of some poking around inside the computer while it was running, the system froze and I had to hard reset it. After booting up it looks like the RAID array got messed up. It looks as if one of the drives were removed. Here's what mdadm says:

Code:

# mdadm -D /dev/md0
/dev/md0:
        Version : 00.90.03
  Creation Time : Thu Dec 27 08:47:02 2007
     Raid Level : raid5
    Device Size : 244198464 (232.89 GiB 250.06 GB)
   Raid Devices : 4
  Total Devices : 3
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Wed Feb 11 22:04:35 2009
          State : active, degraded, Not Started
 Active Devices : 3
Working Devices : 3
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 64K

           UUID : 654d2ad9:ca55af99:8b394a0b:cda00542
         Events : 0.5

    Number   Major   Minor   RaidDevice State
       0      33        0        0      active sync   /dev/hde
       1      33       64        1      active sync   /dev/hdf
       2      34        0        2      active sync   /dev/hdg
       3       0        0        3      removed

and this is what dmesg says:

Code:

# dmesg | grep md0
md: md0 stopped.
md: md0: raid array is not clean -- starting background reconstruction
raid5: cannot start dirty degraded array for md0
raid5: failed to run raid set md0

and it won't even explain:

Code:

# mdadm -E /dev/md0
mdadm: No md superblock detected on /dev/md0.

Do I still have hope or did I lose all my data? I stopped messing around with it before I do something bad and mess it up even more (if that's even possible).

Any help would be greatly appreciated.

- George

jschiwal · 02-20-2009, 09:37 AM

When you say "poking around inside the computer", do you mean physically?

Go into the bios. Does the computer physically detect the forth drive? You might want to check the cables and determine whether there is a hardware problem with the drive/cable/controller before proceeding.

lecnt · 02-20-2009, 09:57 AM

Oh, yes, I forgot to mention that the drive shows up ok and it looks like it's working. It was removed from the array. But the array should recover without that drive, right?

And yes, it was a physical poking. I didn't really get inside, I was just playing around the fan on the side but I guess by moving the fan's power cord I moved something else too.

jschiwal · 02-20-2009, 10:23 AM

Try replacing the drive (/dev/hdh) and run "mdadm --query /dev/hdh". Does it recognize it as a member of an array?

I think you will want to monitor mdadm and determine if it is trying to rebuild the array. Did you have 3 drives & 1 spare? If it is rebuilding, then leave it alone until finished.

lecnt · 02-20-2009, 10:39 AM

How do I replace the drive? Do I do the following?

Code:

mdadm --manage /dev/md0 --add /dev/hdh

The fourth drive wasn't a spare, all drives were used in the array. I'm so scared to do anything to the array.

lecnt · 02-20-2009, 03:27 PM

Or do I have to do

Code:

mdadm --manage /dev/md0 --re-add /dev/hdh

lecnt · 02-23-2009, 08:38 AM

OK, I was impatient so I had to try something and it worked.

Here's what I did in case someone else has the same problem.

Code:

[root@LECBackup ~]# mdadm --manage /dev/md0 --re-add /dev/hdh
mdadm: re-added /dev/hdh
[root@LECBackup ~]# cat /sys/block/md0/md/array_state
inactive
[root@LECBackup ~]# echo "clean" > /sys/block/md0/md/array_state
[root@LECBackup ~]# cat /sys/block/md0/md/array_state
clean
[root@LECBackup ~]# mdadm -D /dev/md0
/dev/md0:
        Version : 00.90.03
  Creation Time : Thu Dec 27 08:47:02 2007
     Raid Level : raid5
     Array Size : 732595392 (698.66 GiB 750.18 GB)
    Device Size : 244198464 (232.89 GiB 250.06 GB)
   Raid Devices : 4
  Total Devices : 4
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Wed Feb 11 22:04:35 2009
          State : clean
 Active Devices : 4
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 64K

           UUID : 654d2ad9:ca55af99:8b394a0b:cda00542
         Events : 0.5

    Number   Major   Minor   RaidDevice State
       0      33        0        0      active sync   /dev/hde
       1      33       64        1      active sync   /dev/hdf
       2      34        0        2      active sync   /dev/hdg
       3      34       64        3      active sync   /dev/hdh

After this I mounted the volume and I've been copying the data off of it ever since.