RAID5 doesn't work after new Ubuntu install

kilbasar · 02-10-2009, 04:18 PM

Hi all. First of all, thanks for taking the time to lend me a hand

I recently re-installed Ubuntu 8.10 on my system. Prior to the re-install, my ext3-formatted raid5 array was working perfectly. When I did the re-install, all I touched was my (non-raid) boot disk, which houses my "/" partition and my windows partition. I did a manual partition, so I know this to be the case.

After the re-install, I installed mdadm, and rebooted my computer. When it came back up, I checked "mdadm --detail /dev/md0". It had auto-detected the three raid5 disks, and had re-assembled the array. However, it also thought the array was inconsistent, and it had started to rebuild the third drive. I didn't want to mess with it, so I let it finish rebuilding, after which it said the array was clean:

Code:

# sudo mdadm --detail /dev/md0 
/dev/md0:
        Version : 00.90
  Creation Time : Wed Apr  2 16:08:44 2008
     Raid Level : raid5
     Array Size : 1465148928 (1397.27 GiB 1500.31 GB)
  Used Dev Size : 732574464 (698.64 GiB 750.16 GB)
   Raid Devices : 3
  Total Devices : 3
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Mon Feb  9 19:47:20 2009
          State : clean
 Active Devices : 3
Working Devices : 3
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 64K

           UUID : 28ce99d8:12dc7f14:80dcd6ba:f1e671fb
         Events : 0.22

    Number   Major   Minor   RaidDevice State
       0       8       16        0      active sync   /dev/sdb
       1       8       32        1      active sync   /dev/sdc
       2       8       64        2      active sync   /dev/sde

As you can see, it sees all 3 drives, sees the correct amount of space that had been used, and sees the date I originally created the array. However, when I try to mount the array, it does not recognize the ext3 partition that I know is on the array:

Code:

# sudo mount -t ext3 /dev/md0 /raid
mount: wrong fs type, bad option, bad superblock on /dev/md0,
       missing codepage or helper program, or other error
       In some cases useful info is found in syslog - try
       dmesg | tail  or so
# dmesg|tail -1
[118517.394256] VFS: Can't find ext3 filesystem on dev md0.

The only other commands I have run have been "mdadm --stop" and "mdadm --assemble". I know my data must still be there, but I can't figure out why it won't mount. Any help would be greatly appreciated! In case it helps, here's what fdisk sees:

Code:

# sudo fdisk -l /dev/md0

Disk /dev/md0: 1500.3 GB, 1500312502272 bytes
255 heads, 63 sectors/track, 182402 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x0009860b

    Device Boot      Start         End      Blocks   Id  System
/dev/md0p1               1       91201   732572001   fd  Linux raid autodetect

mdadm.conf:

Code:

# cat mdadm.conf
ARRAY /dev/md0 level=raid5 num-devices=3 UUID=28ce99d8:12dc7f14:80dcd6ba:f1e671fb

blkid:

Code:

#  blkid
/dev/sda1: UUID="a08797de-3906-43c4-81fb-b9777be9165f" SEC_TYPE="ext2" TYPE="ext3" 
/dev/sdb1: UUID="2dc0dedb-3565-3ec5-bad6-dc80fb71e6f1" TYPE="mdraid" 
/dev/sdc1: UUID="2dc0dedb-3565-3ec5-bad6-dc80fb71e6f1" TYPE="mdraid" 
/dev/sdd1: UUID="86CA6959CA694711" TYPE="ntfs" 
/dev/sdd2: UUID="039c5fed-edf6-40a3-bd44-71a66954db2a" TYPE="ext3" 
/dev/sdd5: UUID="e63d7b46-6baa-439e-87f8-5a5807727708" TYPE="swap" 
/dev/sde1: UUID="2dc0dedb-3565-3ec5-bad6-dc80fb71e6f1" TYPE="mdraid" 
/dev/sdj1: SEC_TYPE="msdos" UUID="18B0-8D32" TYPE="vfat" 
/dev/loop0: TYPE="squashfs"

Thanks, and I really hope I can get my 700gb of data back...

Dankles · 02-11-2009, 02:26 PM

I've seen this before. You can sometimes mount it as an ext2 volume instead of ext3 because sometimes the journaling becomes messed up on ext3

kilbasar · 02-11-2009, 05:01 PM

Hi,

Thanks for the idea. Unfortunately, it won't mount as ext2 either:

Code:

# sudo mount -t ext2 /dev/md0 /raid
mount: wrong fs type, bad option, bad superblock on /dev/md0,
       missing codepage or helper program, or other error
       In some cases useful info is found in syslog - try
       dmesg | tail  or so

# dmesg| tail -1
[208795.081508] VFS: Can't find an ext2 filesystem on dev md0.

custangro · 02-11-2009, 05:34 PM

What is the output of...

Code:

user@ubuntu$ sudo mdadm --detail --scan

kilbasar · 02-11-2009, 06:40 PM

Code:

# sudo mdadm --detail --scan
ARRAY /dev/md0 level=raid5 num-devices=3 metadata=00.90 UUID=28ce99d8:12dc7f14:80dcd6ba:f1e671fb

thanks!

kilbasar · 02-16-2009, 07:38 PM

Hmmm... On a hunch, I booted from an Ubuntu 8.04 LiveCD, and installed mdadm. I then ran "mdadm --assemble --scan", and lo and behold, my array came back (degraded), now able to mount. So, my data is all there. But how do I get the array working in my normal system? Here's what things look like from inside the LiveCD:

Code:

# mdadm --detail /dev/md0
/dev/md0:
        Version : 00.90.03
  Creation Time : Wed Apr  2 20:21:23 2008
     Raid Level : raid5
     Array Size : 1465143808 (1397.27 GiB 1500.31 GB)
  Used Dev Size : 732571904 (698.64 GiB 750.15 GB)
   Raid Devices : 3
  Total Devices : 2
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Tue Feb 17 01:19:29 2009
          State : clean, degraded
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 64K

           UUID : dbdec02d:c53e6535:80dcd6ba:f1e671fb
         Events : 0.62

    Number   Major   Minor   RaidDevice State
       0       8       17        0      active sync   /dev/sdb1
       1       8       33        1      active sync   /dev/sdc1
       2       0        0        2      removed

Code:

# fdisk -l /dev/md0

Disk /dev/md0: 1500.3 GB, 1500307259392 bytes
2 heads, 4 sectors/track, 366285952 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Disk identifier: 0x00000000

Disk /dev/md0 doesn't contain a valid partition table

Code:

# blkid
/dev/sda1: UUID="a08797de-3906-43c4-81fb-b9777be9165f" SEC_TYPE="ext2" TYPE="ext3" 
/dev/sdb1: UUID="2dc0dedb-3565-3ec5-bad6-dc80fb71e6f1" TYPE="mdraid" 
/dev/sdc1: UUID="2dc0dedb-3565-3ec5-bad6-dc80fb71e6f1" TYPE="mdraid" 
/dev/sdd1: UUID="86CA6959CA694711" TYPE="ntfs" 
/dev/sdd2: UUID="039c5fed-edf6-40a3-bd44-71a66954db2a" SEC_TYPE="ext2" TYPE="ext3" 
/dev/sdd5: TYPE="swap" UUID="e63d7b46-6baa-439e-87f8-5a5807727708" 
/dev/sde: UUID="d899ce28-147f-dc12-bad6-dc80fb71e6f1" TYPE="mdraid" 
/dev/sdj1: UUID="4999-F967" TYPE="vfat" 
/dev/loop0: TYPE="squashfs" 
/dev/md0: UUID="8c0a7fa3-6c52-4674-b095-c770454b8634" TYPE="ext3"

Code:

# mdadm --detail --brief /dev/md0
ARRAY /dev/md0 level=raid5 num-devices=3 UUID=dbdec02d:c53e6535:80dcd6ba:f1e671fb