RAID5 Array Recovery after OS upgrade

rtspitz · 04-17-2007, 11:24 AM

Quote:

Originally Posted by hazmatt20

That German thread is exactly what mine is doing. If you don't mind translating the solution, I'd be grateful. I tried

translation of the last part with the "solution":

Code:

at the time I created the raid I must have made a mistake, which showed up right now. 
apparently I had created persistent superblocks on the devices (/dev/sd[a-e]) as well as
on the partitions (/dev/sd[a-e]1). 
after zero-ing the superblocks with "mdadm --zero-superblock /dev/sd[-e]" and rebooting,
the partitions showed up in /proc/partitions again and the raid was operational and could
be mounted without any errors.

this night was not subject to entertainment tax. (:

hazmatt20 · 04-17-2007, 02:01 PM

Well, I decided to give mdadm --create a shot.

Code:

mdadm --create /dev/md0 --verbose --level=5 --raid-devices=6 /dev/sd[d-i]1
mdadm: layout defaults to left-symmetric
mdadm: chunk size defaults to 64K
mdadm: /dev/sdd1 appears to contain an ext2fs file system
    size=1953543680K  mtime=Sun Apr 15 18:40:03 2007
mdadm: /dev/sdd1 appears to be part of a raid array:
    level=raid5 devices=6 ctime=Sun Mar 11 00:22:58 2007
mdadm: /dev/sde1 appears to be part of a raid array:
    level=raid5 devices=6 ctime=Sun Mar 11 00:22:58 2007
mdadm: /dev/sdf1 appears to be part of a raid array:
    level=raid5 devices=6 ctime=Sun Mar 11 00:22:58 2007
mdadm: /dev/sdg1 appears to contain an ext2fs file system
    size=1953543680K  mtime=Sun Apr 15 18:40:02 2007
mdadm: /dev/sdg1 appears to be part of a raid array:
    level=raid5 devices=6 ctime=Sun Mar 11 00:22:58 2007
mdadm: /dev/sdh1 appears to be part of a raid array:
    level=raid5 devices=6 ctime=Sun Mar 11 00:22:58 2007
mdadm: /dev/sdi1 appears to be part of a raid array:
    level=raid5 devices=6 ctime=Sun Mar 11 00:22:58 2007
mdadm: size set to 390708736K
Continue creating array? y
mdadm: array /dev/md0 started.

#cat /proc/mdstat
Personalities : [raid5] [raid4]
md0 : active raid5 sdi1[6] sdh1[4] sdg1[3] sdf1[2] sde1[1] sdd1[0]
      1953543680 blocks level 5, 64k chunk, algorithm 2 [6/5] [UUUUU_]
      [>....................]  recovery =  0.1% (419712/390708736) finish=340.8min speed=19077K/sec

# mdadm --detail /dev/md0
/dev/md0:
        Version : 00.90.03
  Creation Time : Tue Apr 17 14:57:02 2007
     Raid Level : raid5
     Array Size : 1953543680 (1863.04 GiB 2000.43 GB)
    Device Size : 390708736 (372.61 GiB 400.09 GB)
   Raid Devices : 6
  Total Devices : 6
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Tue Apr 17 14:57:41 2007
          State : clean, degraded, recovering
 Active Devices : 5
Working Devices : 6
 Failed Devices : 0
  Spare Devices : 1

         Layout : left-symmetric
     Chunk Size : 64K

 Rebuild Status : 1% complete

           UUID : ce16308c:c13226e7:126d5cca:b4ac2ebe
         Events : 0.3

    Number   Major   Minor   RaidDevice State
       0       8       49        0      active sync   /dev/sdd1
       1       8       65        1      active sync   /dev/sde1
       2       8       81        2      active sync   /dev/sdf1
       3       8       97        3      active sync   /dev/sdg1
       4       8      113        4      active sync   /dev/sdh1
       6       8      129        5      spare rebuilding   /dev/sdi1

So, I'll let it go for a few hours and check back.

hazmatt20 · 04-17-2007, 06:31 PM

Alright, so it finished resyncing. Now we get

Code:

# mount /dev/md0 md0
mount: wrong fs type, bad option, bad superblock on /dev/md0,
       missing codepage or other error
       In some cases useful info is found in syslog - try
       dmesg | tail  or so

# fsck /dev/md0
fsck 1.39 (29-May-2006)
e2fsck 1.39 (29-May-2006)
Group descriptors look bad... trying backup blocks...
fsck.ext3: Bad magic number in super-block while trying to open /dev/md0

The superblock could not be read or does not describe a correct ext2
filesystem.  If the device is valid and it really contains an ext2
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
    e2fsck -b 8193 <device>

Is there anything else we can try, or is it game over?

Quakeboy02 · 04-17-2007, 06:37 PM

You've kind of lost me here. I was under the impression that the create option created a new array and threw away anything that previously existed. As far as I was understand, the data was gone when you ran create.

hazmatt20 · 04-17-2007, 07:00 PM

I was going by this article posted earlier. http://kev.coolcavemen.com/2007/03/h...d-superblocks/

rtspitz · 04-17-2007, 07:26 PM

there is a utility called testdisk http://www.cgsecurity.org/wiki/TestDisk which can scan devices with ext2/ext3 for backup superblocks and help recover them.

some hints:

http://www.cgsecurity.org/wiki/Advan...kup_SuperBlock

you could run:

testdisk /dev/md0

then: [PROCEED], [NONE], [Advanced], [Superblock]

if this works you should get some output like this:

superblock 0, blocksize=1024
superblock 8193, blocksize=1024
...
...

with that you can tell fsck.ext3 (or equivalent on your system) to use a backup superblock like e.g.:

/sbin/fsck.ext3 -b 8193 -B 1024 /dev/md0

if that doesn't work I'm at my wits' end.

hazmatt20 · 04-17-2007, 10:02 PM

Well, testdisk didn't show any partitions under advanced, so I'm running analyse. It's going to take a good while, but I'm going to start making plans to start reloading data. I'll post an update when it finishes.

hazmatt20 · 04-18-2007, 06:11 PM

Alright, well, it analyse didn't detect stuff correctly and just gave a bunch of garbage, so I'm pretty positive it's gone. So many DVDs to reload! Oh, well. Thanks for your help.

One last thing, what precautions should I take in the future to increase my chances of recovery? I know now to run dist-upgrade install of installing from disk, but other than that and backing up my mdadm.conf, what should I do?

Quakeboy02 · 04-18-2007, 06:34 PM

I've been thinking about this today, and I wonder if the problem could have been avoided if you hadn't had your drives connected when you installed mdadm. I mentioned that I installed mdadm once and it created a bunch of junk on the drives I had connected. It may or may not be a problem, but it's something to think about if you have to reinstall for any reason. You might also think about creating a backup of your non-data files. There are a number of good backup systems out there. I just used tar along with a trivial script I wrote. I actually did a restore (from Knoppix) of the boot/non-data image I keep on my data disks recently, and it worked just fine.

hazmatt20 · 04-19-2007, 12:52 AM

Well, I've almost got everything working, but I've got a few snags. Two parts.

First, I want the 6 400GB drives to start as md0 and the 3 500GB drives to start as md1. When I reboot, md0 starts with 2 of the 3 500GB drives and resyncs with the third while md1 starts with 4 of the 6 400GB drives. mdadm.conf is currently

Code:

# cat mdadm.conf
DEVICE partitions
MAILADDR root
ARRAY /dev/md0 level=raid5 num-devices=6 UUID=d295489e:1146f6bf:10e91e6c:42385ae5
ARRAY /dev/md1 level=raid5 num-devices=3 UUID=27088f9d:f6aea8a8:60e614e7:ea4536bf

Secondly, I'm setting up the two arrays on an LVM. I've already done this much.

Code:

pvcreate /dev/md0
pvcreate /dev/md1
vgcreate RAID_GROUP /dev/md0 /dev/md1
modprobe dm-mod
lvcreate -L2.72T -nmedia RAID_GROUP
mkfs.ext3 /dev/RAID_GROUP/media

After that, I could mount /dev/RAID_GROUP/media normally. After it reboots and I get the raid arrays back up, I activate it with

Code:

vgchange -a y RAID_GROUP

If I want it to activate on startup, should I just add that line to rc.local (assuming the raid arrays come up first), or is there a better way to do it before local file systems are mounted?

hazmatt20 · 04-19-2007, 08:41 AM

Well, now on reboot, md0 comes up correctly but as the 3 500GB drives, so I'll just leave that the way it is. md1, however, only comes up with 4 drives on startup. Once I get a console,

Code:

mdadm -A /dev/md1 /dev/sd[d-i]

works, so I could run it under rc.local, but is there a cleaner way?