Software Raid 1 behaving strangely after power outage

Krigslund · 01-12-2009, 03:39 PM

Hi,
I have set up a software raid 1 device (md0) on my Ubuntu Server using two identical disks (sda1 and sdb1). The other day I experienced a power outage and after reboot I examined the system for any errors caused by the loss of power.

The raid array mounted nicely. However, I still think something is off, and here is why:

My mdstat gives the following:

Code:

root@ghost:~# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid1 sda1[0]
      312568576 blocks [2/1] [U_]

unused devices: <none>

So my raid array appears to be degraded. Running a "mdadm -E /dev/sda1" gives:

Code:

root@ghost:~# mdadm -E /dev/sda1
/dev/sda1:
          Magic : a92b4efc
        Version : 00.90.00
           UUID : 63a5c86e:cabfad07:855774c4:31e91d18 (local to host ghost)
  Creation Time : Fri Aug 10 00:46:13 2007
     Raid Level : raid1
  Used Dev Size : 312568576 (298.09 GiB 320.07 GB)
     Array Size : 312568576 (298.09 GiB 320.07 GB)
   Raid Devices : 2
  Total Devices : 1
Preferred Minor : 0

    Update Time : Tue Jan 11 19:22:28 2000
          State : clean
 Active Devices : 1
Working Devices : 1
 Failed Devices : 1
  Spare Devices : 0
       Checksum : 20d13784 - correct
         Events : 0.1291208


      Number   Major   Minor   RaidDevice State
this     0       8        1        0      active sync   /dev/sda1

   0     0       8        1        0      active sync   /dev/sda1
   1     1       0        0        1      faulty removed

Apparently my sdb1 disk is down. But out of curiosity I ran the same command on sdb1, which returned the following:

Code:

root@ghost:~# mdadm -E /dev/sdb1
/dev/sdb1:
          Magic : a92b4efc
        Version : 00.90.00
           UUID : 63a5c86e:cabfad07:855774c4:31e91d18 (local to host ghost)
  Creation Time : Fri Aug 10 00:46:13 2007
     Raid Level : raid1
  Used Dev Size : 312568576 (298.09 GiB 320.07 GB)
     Array Size : 312568576 (298.09 GiB 320.07 GB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 0

    Update Time : Thu Jan 24 20:16:03 2008
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0
       Checksum : 2fc73f71 - correct
         Events : 0.34


      Number   Major   Minor   RaidDevice State
this     1       8       17        1      active sync   /dev/sdb1

   0     0       8        1        0      active sync   /dev/sda1
   1     1       8       17        1      active sync   /dev/sdb1

I may be a newbie regarding software raid in linux and I don't know much about interpreting these outputs, but this seems nevertheless fairly strange to me. Even though the array (md0) is mountable I don't feel safe (data-wise), as the above simply cannot be right.

I hope some of you can see what may be wrong and has some suggestions on how I get a healthy raid array again.

Best regards,
Rasmus Krigslund
Denmark

madman_lxl · 01-14-2009, 12:09 AM

Krigslund,

whats the state of md0 ? mdadm --detail /dev/md0 for the command.

Regards,

Madman

Krigslund · 01-15-2009, 03:25 AM

Thank you for replying madman_lxl!

The command you specified gives the following output:

Code:

root@ghost:~$ mdadm --detail /dev/md0
/dev/md0:
        Version : 00.90.03
  Creation Time : Fri Aug 10 00:46:13 2007
     Raid Level : raid1
     Array Size : 312568576 (298.09 GiB 320.07 GB)
  Used Dev Size : 312568576 (298.09 GiB 320.07 GB)
   Raid Devices : 2
  Total Devices : 1
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Mon Jan 17 15:00:47 2000
          State : clean, degraded
 Active Devices : 1
Working Devices : 1
 Failed Devices : 0
  Spare Devices : 0

           UUID : 63a5c86e:cabfad07:855774c4:31e91d18 (local to host ghost)
         Events : 0.1296612

    Number   Major   Minor   RaidDevice State
       0       8        1        0      active sync   /dev/sda1
       1       0        0        1      removed

Can you say anything from this?

madman_lxl · 01-15-2009, 05:46 AM

Krigslund,

raid md0 can only see disk /dev/sda1 . sdb has been removed from array.
As i'm always double checking commands before entering them i'd really suggest that you make sure you have a backup of data and then try the following tasks.

You'll need to do is use gparted or fdisk to wipe sdb so that no partitions remain and then add a partition that is selected to be tagged as a software raid partition once done then you'll need to use mdadm to add the /dev/sdb1 partition back to the array. I would also recommend testing the drive as well with the manufacturers tools i.e. seagate tools.

Regards,

Madman

Krigslund · 01-15-2009, 06:00 AM

Madman_lxl,
I have a backup of all data on the server so data loss is, for once, not an issue. I see your point in wiping and testing sdb before trying to add it to the array again, but could you please elaborate on what you mean by "add a partition that is selected to be tagged as a software raid partition". I understand "adding a partition" of course, but I am not sure what you mean by tagging it as a raid partition.

Best regards
Krigslund

madman_lxl · 01-15-2009, 06:23 AM

Krigslund

Sorry, my bad wording. Just format the drives partition to be Linux raid should be fd in fdisk.

fdisk /dev/sdb
then use command d to delete partitions use command p to print and make sure none are left.
then use n to add a partition (make it fill all free space should be default from memory) and t to change type select 1 as first partition and fd as the type to make it a raid drive.
use p again to confirm setup.
then use w to write the changes.

That should now make it a raid formatted drive.

Regards,

Madman

Krigslund · 01-16-2009, 01:09 PM

Madman,
I tried wiping and adding sdb1 to the raid array again. The recovery started immediately, but nothing happens - It is simply stuck at 0.0%:

Code:

root@ghost:~# cat /proc/mdstat 
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active raid1 sdb1[2] sda1[0]
      312568576 blocks [2/1] [U_]
      [>....................]  recovery =  0.0% (120576/312568576) finish=172.7min speed=30144K/sec
      
unused devices: <none>

Is this a common issue, or maybe the disk is really damaged after all?

Best,
Krigslund