Cannot recover software RAID5

tlagren · 03-05-2015, 03:28 PM

Hi, I'm having big problem with a RAID5 recovery on my home server.

I was running CentOS 6, latest patchlevel and had a mdadm created RAID5 for data storage with three disks.

After a failure of the boot disk which wasn't part of any raid configuration I'm not able to assemble the old raid again.

Here is what I found so far.

The disks used in the array is /dev/sdc, /dev/sdd and /dev/sde and they all contains one raid partition.

Code:

[root@tor ~]# fdisk -l /dev/sdc

Disk /dev/sdc: 320.1 GB, 320072933376 bytes
255 heads, 63 sectors/track, 38913 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sdc1               1       38913   312568641   fd  Linux raid autodetect

When I try to assemble the array with scan it doesn't find anything.

Code:

[root@tor ~]# mdadm --assemble --verbose --scan
mdadm: looking for devices for further assembly
mdadm: no recogniseable superblock on /dev/dm-1
mdadm: Cannot assemble mbr metadata on /dev/dm-0
mdadm: Cannot assemble mbr metadata on /dev/sde
mdadm: Cannot assemble mbr metadata on /dev/sdd
mdadm: Cannot assemble mbr metadata on /dev/sdc
mdadm: Cannot assemble mbr metadata on /dev/sdb
mdadm: Cannot assemble mbr metadata on /dev/sda
mdadm: No arrays found in config file or automatically

When executed mdadm --examine I noticed strange things.
It looks like mdadm cannot see the partition sdc1 and it's the same thing with sdd and sde drives.

Code:

[root@tor ~]# mdadm --examine /dev/sdc
/dev/sdc:
   MBR Magic : aa55
Partition[0] :    625137282 sectors at           63 (type fd)
[root@tor ~]# 
[root@tor ~]# 
[root@tor ~]# mdadm --examine /dev/sdc1
mdadm: cannot open /dev/sdc1: No such file or directory

If I execute the blkid command I noticed that the TYPE of all my three raid member drives has the TYPE="hpt45x_raid_member". I think this have something to do with that the disk are connected to an old Rocket Raid controller, but due to the lack of good Linux drivers for the raid usage I don't use the hardware raid features and used the software raid instead. So the rocket card are just used an extended SATA controller.
Do you think that the disk TYPE can cause my problems?

Code:

[root@tor ~]# blkid /dev/sdc
/dev/sdc: TYPE="hpt45x_raid_member"

One thing I was thinking about but haven't tried it yet because I'm not sure if it's safe to do. Some people mention that mdadm is smart when creating a new array so if I would create a new raid5 array with my disk mdadm will find that the disk already have a raid signature and don't wipe the disks. Perhaps it would be safe if I'm using the assume-clean arument? What do you think?

Do you have any ideas what I can do to recreate my raid array and get the data back again? I don't have any copy of the old /etc/mdadm.conf file left.

/Tomas

tlagren · 03-05-2015, 03:50 PM

Some more information

Code:

[root@tor ~]# mdadm --assemble --verbose /dev/md0 /dev/sdc1 /dev/sdd1 /dev/sde1
mdadm: looking for devices for /dev/md0
mdadm: cannot open device /dev/sdc1: No such file or directory
mdadm: /dev/sdc1 has no superblock - assembly aborted

smallpond · 03-05-2015, 03:58 PM

RocketRaid uses the Highpoint RAID controller chip, which does have a Linux driver. Are you sure you didn't have that enabled? Before modifying the drives check with dmraid to make sure they weren't using the controller RAID. It's very suspicious that you lost your partitions. That sounds like whole-drive RAID was enabled.

tlagren · 03-05-2015, 04:37 PM

I see I've got the driver loaded

Code:

[root@tor ~]# lsmod | grep hpt
hptiop                 13838  0 
pata_hpt37x            12778  0

I also see the disks with dmraid -r but I'm not still sure I've used the hpt features. I almost 100% sure that i used md raid, but can there have been some mix with the hpt driver or similar, I don't really understand this.
The sda/sdb devices below my current boot disk and are not included in the old array.

Code:

[root@tor ~]# dmraid -r
/dev/sde: hpt45x, "hpt45x_SPARE", spare, ok, 625142437 sectors, data@ 0
/dev/sdd: hpt45x, "hpt45x_SPARE", spare, ok, 625142437 sectors, data@ 0
/dev/sdc: hpt45x, "hpt45x_SPARE", spare, ok, 625142437 sectors, data@ 0
/dev/sdb: nvidia, "nvidia_adadfdhf", mirror, ok, 488397166 sectors, data@ 0
/dev/sda: nvidia, "nvidia_adadfdhf", mirror, ok, 488397166 sectors, data@ 0

/Tomas

tlagren · 03-05-2015, 05:14 PM

Now I found something interesting. I unloaded the hptiop and pata_hpt37x module and loaded them again and then the array was available.

Code:

[root@tor ~]# mdadm --detail --scan
ARRAY /dev/md/2 metadata=1.2 name=tor.lagren.com:2 UUID=1e183967:b225b24d:0e4381ad:ac1a3fbf

[root@tor ~]# mdadm --detail /dev/md2
/dev/md2:
        Version : 1.2
  Creation Time : Sun Oct 30 14:47:31 2011
     Raid Level : raid5
     Array Size : 625134592 (596.17 GiB 640.14 GB)
  Used Dev Size : 312567296 (298.09 GiB 320.07 GB)
   Raid Devices : 3
  Total Devices : 3
    Persistence : Superblock is persistent

    Update Time : Fri Mar  6 00:47:49 2015
          State : clean 
 Active Devices : 3
Working Devices : 3
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 512K

           Name : tor.lagren.com:2  (local to host tor.lagren.com)
           UUID : 1e183967:b225b24d:0e4381ad:ac1a3fbf
         Events : 1535

    Number   Major   Minor   RaidDevice State
       0       8       33        0      active sync   /dev/sdc1
       1       8       49        1      active sync   /dev/sdd1
       3       8       65        2      active sync   /dev/sde1

I created a new mdadm.conf and activated the LVM volume group and now all my logical volumes are in perfect condition.

After reboot it's the same problem again, reload the hpt modules and everything is Ok...

/Tomas

syg00 · 03-05-2015, 05:37 PM

Quote:

Originally Posted by tlagren

I was running CentOS 6, latest patchlevel and had a mdadm created RAID5 for data storage with three disks.

After a failure of the boot disk which wasn't part of any raid configuration I'm not able to assemble the old raid again.

What distro/version are you using when attempting the recovery ?.

tlagren · 03-05-2015, 05:50 PM

I'm using CentOS 6.6 and all updates are installed.

It feels like the modules are loaded in the wrong order or so. I can't remember I did something special with this the last time I installed the server. Everything works, but when booted I need to reload the hpt module and when I do that the array loads directly.

/Tomas

tlagren · 03-05-2015, 06:00 PM

Here is the complete process I do to make the array running after a reboot.

Code:

[root@tor ~]# cat /proc/mdstat 
Personalities : 
unused devices: <none>


[root@tor ~]# lsmod | grep hpt
hptiop                 13838  0 
pata_hpt37x            12778  0 


[root@tor ~]# service mdmonitor stop
Killing mdmonitor:                                         [  OK  ]


[root@tor ~]# modprobe -r hptiop ; modprobe -r pata_hpt37x ; modprobe pata_hpt37x ; modprobe hptiop


[root@tor ~]# cat /proc/mdstat 
Personalities : [raid6] [raid5] [raid4] 
md2 : active raid5 sde1[3] sdd1[1] sdc1[0]
      625134592 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/3] [UUU]
      
unused devices: <none>


[root@tor ~]# service mdmonitor start
Starting mdmon:                                            [  OK  ]
Starting mdmonitor:                                        [  OK  ]

/Tomas

syg00 · 03-05-2015, 06:10 PM

I missed your post #5 before I posted.
Have you tried just unloading the modules (not reloading them again) ?
If that works, just blacklist the htp.

tlagren · 03-06-2015, 01:09 AM

If i unload the modules without loading them again the drives completely disappear so I think I need to have the driver loaded.
I just got back my /etc catalog from the old server installation and I checked /etc/mdadm.conf and also /etc/modprobe.d/blacklist.conf and there was no special configuration in mdadm.conf and no blacklists. I copied back the mdadm.conf but are stuck in the same situation.

/Tomas

tlagren · 03-06-2015, 01:41 AM

I'm replying to my self again :-) Now I got it working!
I added the nodmraid option to the grub configuration and now the drives shows up correctly at boot and the array is autostarting.

So why did I need to add that option? The old server was installed with CentOS 6.0 and upgraded whenever new releases was available. The restored server was installed with CentOS 6.2 (that was the image I had available and then upgraded to latest version with yum update).

I don't have the old /boot partition left so I can't very that I didn't put the nodmraid option in the old installation but I don't think so.

/Tomas

syg00 · 03-06-2015, 02:49 AM

I'm guessing old format meta-data - nope, maybe not.