RAID 5 won't mount

millenium · 06-26-2015, 01:29 AM

I have a RAID5 array with 3 WD RED disks (sdb, sdc sdd), where one drive gave up. I replaced the drive and rebuild. Now the array won't mount.

My mdadm conf:

Code:

# mdadm.conf#
# Please refer to mdadm.conf(5) for information about this file.
#


# by default (built-in), scan all partitions (/proc/partitions) and all
# containers for MD superblocks. alternatively, specify devices to scan, using
# wildcards if desired.
#DEVICE partitions containers


# auto-create devices with Debian standard permissions
CREATE owner=root group=disk mode=0660 auto=yes


# automatically tag new arrays as belonging to the local system
HOMEHOST <system>


# instruct the monitoring daemon where to send mail alerts
MAILADDR root


# definitions of existing MD arrays
ARRAY /dev/md0 UUID=76d1edff:a1188431:0cf33d4d:3ef09af2


# This file was auto-generated on Fri, 24 Apr 2015 06:49:33 +0200
# by mkconf $Id$

'cat /proc/mdstat' gives:

Code:

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]md0 : active raid5 sdd[2] sdc[0] sdb[3]
      5860270080 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/3] [UUU]

'fsck /dev/md0' gives

Code:

fsck from util-linux 2.20.1e2fsck 1.42.9 (4-Feb-2014)
fsck.ext4: Filesystem revision too high while trying to open /dev/md0
The filesystem revision is apparently too high for this version of e2fsck.
(Or the filesystem superblock is corrupt)


The superblock could not be read or does not describe a valid ext2/ext3/ext4
filesystem.  If the device is valid and it really contains an ext2/ext3/ext4
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
    e2fsck -b 8193 <device>
 or
    e2fsck -b 32768 <device>

'mdadm -E /dev/sd[bcd]' gives

Code:

/dev/sdb:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 76d1edff:a1188431:0cf33d4d:3ef09af2
           Name : movano:0
  Creation Time : Mon Apr  6 00:07:18 2015
     Raid Level : raid5
   Raid Devices : 3


 Avail Dev Size : 5860271024 (2794.40 GiB 3000.46 GB)
     Array Size : 5860270080 (5588.79 GiB 6000.92 GB)
  Used Dev Size : 5860270080 (2794.39 GiB 3000.46 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 85875e25:589122cb:689b974e:1cf4b988


    Update Time : Fri Apr 24 02:56:16 2015
       Checksum : d8b33362 - correct
         Events : 232


         Layout : left-symmetric
     Chunk Size : 512K


   Device Role : Active device 1
   Array State : AAA ('A' == active, '.' == missing)
millenium@ubuntu:~$ sudo mdadm -E /dev/sd[bcd]
/dev/sdb:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 76d1edff:a1188431:0cf33d4d:3ef09af2
           Name : movano:0
  Creation Time : Mon Apr  6 00:07:18 2015
     Raid Level : raid5
   Raid Devices : 3


 Avail Dev Size : 5860271024 (2794.40 GiB 3000.46 GB)
     Array Size : 5860270080 (5588.79 GiB 6000.92 GB)
  Used Dev Size : 5860270080 (2794.39 GiB 3000.46 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 85875e25:589122cb:689b974e:1cf4b988


    Update Time : Fri Apr 24 02:56:16 2015
       Checksum : d8b33362 - correct
         Events : 232


         Layout : left-symmetric
     Chunk Size : 512K


   Device Role : Active device 1
   Array State : AAA ('A' == active, '.' == missing)
/dev/sdc:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 76d1edff:a1188431:0cf33d4d:3ef09af2
           Name : movano:0
  Creation Time : Mon Apr  6 00:07:18 2015
     Raid Level : raid5
   Raid Devices : 3


 Avail Dev Size : 5860271024 (2794.40 GiB 3000.46 GB)
     Array Size : 5860270080 (5588.79 GiB 6000.92 GB)
  Used Dev Size : 5860270080 (2794.39 GiB 3000.46 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 9732beb4:34a4d203:a8c351d1:788be3c0


    Update Time : Fri Apr 24 02:56:16 2015
       Checksum : 5ba6b0ea - correct
         Events : 232


         Layout : left-symmetric
     Chunk Size : 512K


   Device Role : Active device 0
   Array State : AAA ('A' == active, '.' == missing)
/dev/sdd:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 76d1edff:a1188431:0cf33d4d:3ef09af2
           Name : movano:0
  Creation Time : Mon Apr  6 00:07:18 2015
     Raid Level : raid5
   Raid Devices : 3


 Avail Dev Size : 5860271024 (2794.40 GiB 3000.46 GB)
     Array Size : 5860270080 (5588.79 GiB 6000.92 GB)
  Used Dev Size : 5860270080 (2794.39 GiB 3000.46 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : b2db48cc:b885a2b6:d1085a95:ecbe42a4


    Update Time : Fri Apr 24 02:56:16 2015
       Checksum : cd68b428 - correct
         Events : 232


         Layout : left-symmetric
     Chunk Size : 512K


   Device Role : Active device 2
   Array State : AAA ('A' == active, '.' == missing)

When I try to mount the array as EXT4 (originally EXT4 before dead disk) I get this:

Code:

mount: wrong fs type, bad option, bad superblock on /dev/md0,
       missing codepage or helper program, or other error
       In some cases useful info is found in syslog - try
       dmesg | tail  or so

'dmesg | tail' gives

Code:

[   85.150872] EXT4-fs (md0): Couldn't mount because of unsupported optional features (1fd00001)

S.Haran · 06-26-2015, 11:35 PM

You might want to try Testdisk to see if it finds a filesystem. If not try Photorec which will scan for your data based on file extension.

Further action, if needed, will depend on the results you get.

syg00 · 06-27-2015, 01:58 AM

And did you try the fsck with the alternate superblock(s) as the message told you ?.

wpeckham · 06-27-2015, 07:02 AM

I would work on that for about an hour, then just rebuild the array blank, and restore from backup.

You DO have a backup, yes?

Actually, being somewhat adventuresome, I might rebuild using btrfs with some of the raid features turned on.

millenium · 06-28-2015, 02:39 AM

I will try #2 and #3

Unfortunately my backup system didn't include the RAID array ;-( Lesson learned, the hard way.

syg00 · 06-28-2015, 05:32 AM

Nor should it - the array is at the (emulated) device level. You seem to have reconstructed that ok. For the superblock problem, if fsck won't fix it, do a mkfs and restore your data. Tools like photorec are for people that don't have backups - it scans the disk looking for tell-tale signs of files. Sometimes works, sometimes not. And what is the level of confidence ?.

As for @wpeckham comments on btrfs RAID, no better in some respects - I use it (RAID5) for my important data, but you still need backups. Differential snaps are a boon, but for RAID[56] device replacement you need to be reasonably current on kernel and btrfs-progs (say Feb 2015). Still the best solution IMHO.

wpeckham · 06-28-2015, 08:07 AM

I have been a professional SYSADMIN for over 20 years, a NETADMIN for half that.
If I have learned one thing, it is that you can be good or bad at almost any part of this business and succeed ONLY if you are good at backup and recovery.
We wear a dozen hats, and it is good to be excellent at many or all of our jobs, but this is the ONE thing that is survival: for the businesses we support and for our career.

If you are not a professional, it is less of a priority, but still a good lesson. Every newbie must suffer through learning this at some point: no system/hardware lasts forever.

My first rules of backups (not my invention, any experienced sysadmin might list the same):
1. Don't back up everything, only back up what you do not want to lose.
2. No storage solution takes the place of backups. Notice that is plural: if one failure can kill it, one is not secure.
3. It cannot be called a good backup, until restore is tested.

I hope that you can recover your data, and wish you the best of luck!

S.Haran · 06-28-2015, 10:51 AM

@syg00, To clarify

Quote:

if fsck won't fix it, do a mkfs and restore your data

I expect you mean restore from backup as running mkfs will in no way restore data by itself. As stated the OP has no backup.

As for Photorec one reason to run it here is as a test to validate the RAID is assembled correctly. If for example Photorec recovers a 2MB .jpg file that renders fine then chances are the RAID is in good shape.

For the best chance of recovery the rule is to avoid any operation that writes to the RAID member drives, including fsck. If this is high value data for maximum safety it is best to take drive images and perform recovery work on the images.

syg00 · 06-28-2015, 07:46 PM

Quote:

Originally Posted by S.Haran

As stated the OP has no backup.

Hmmm- maybe I misunderstood. I read that as "I have backup (of data), but not of the array metadata."
My mistake.

Quote:

for maximum safety it is best to take drive images and perform recovery work on the images.

Agreed.

millenium · 06-28-2015, 11:35 PM

Thanks for all the answers. I will get a hold of some drives and make drive images and try the suggestions.