mdadm: raid1 mirror's spare won't activate

hallucinative · 02-15-2012, 06:56 PM

g'day Guys and girls!

So... I'm stumped... Basicly I had a faulty sata cable, ran the server degraded while getting a new cable, shut it down, installed the cable and booted it up and for some reason I can't get it to become active again..

Through numerous google searches on this problem I've come up with null, I've zeroed the super block, re-formatted, removed 512 KB of the beginning of the drive and still, she doesn't want to activate.. she syncs up with "Spare synchronizing" finishes and drops back in to the spare group..

I would recreate the array but this is my root drive..

Some one point me in the right direction -.-"

-Donny

Running Ubuntu Server, latest updates..

mdadm -D /dev/md0 shows...

Code:

root@LoM:~# mdadm -D /dev/md0
/dev/md0:
        Version : 1.2
  Creation Time : Sat Feb 11 14:56:56 2012
     Raid Level : raid1
     Array Size : 976758841 (931.51 GiB 1000.20 GB)
  Used Dev Size : 976758841 (931.51 GiB 1000.20 GB)
   Raid Devices : 2
  Total Devices : 2
    Persistence : Superblock is persistent

    Update Time : Thu Feb 16 08:46:56 2012
          State : clean, degraded
 Active Devices : 1
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 1

           Name : LoM:0  (local to host LoM)
           UUID : 45a552f7:5e854deb:26a64954:01f3aaec
         Events : 116263

    Number   Major   Minor   RaidDevice State
       1       8        1        0      active sync   /dev/sda1
       1       0        0        1      removed

       2       8       17        -      spare   /dev/sdb1

cat /proc/mdstat shows...

Code:

root@LoM:~# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid1 sdb1[2](S) sda1[1]
      976758841 blocks super 1.2 [2/1] [U_]

unused devices: <none>

fdisk -l shows...

Code:

root@LoM:~# fdisk -l

Disk /dev/sda: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0008e75b

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1              63  1953520064   976760001   fd  Linux RAID autodetect

Disk /dev/sdb: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1              63  1953520064   976760001   fd  Linux RAID autodetect

Disk /dev/md0: 1000.2 GB, 1000201053184 bytes
2 heads, 4 sectors/track, 244189710 cylinders, total 1953517682 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/md0 doesn't contain a valid partition table

Disk /dev/sdc: 3997 MB, 3997171712 bytes
213 heads, 47 sectors/track, 779 cylinders, total 7806976 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000d3836

   Device Boot      Start         End      Blocks   Id  System
/dev/sdc1   *        2048     7804927     3901440   83  Linux

hallucinative · 02-16-2012, 06:59 AM

Shameless bump..

romagnolo · 02-16-2012, 07:11 AM

Have you tried to remove and re-add the the drive from the array? like:

Code:

mdadm /dev/md0 -r /dev/sdb1
mdadm /dev/md0 -a /dev/sdb1

hallucinative · 02-16-2012, 08:19 AM

Quote:

Originally Posted by romagnolo

Have you tried to remove and re-add the the drive from the array? like:

Code:

mdadm /dev/md0 -r /dev/sdb1
mdadm /dev/md0 -a /dev/sdb1

Thanks romagnolo for your reply

I've tried re-adding many time, with a combination of zeroing the super block, reformatting, dding away the first 512 KB of the hard drive and rebooting in case of some unknown cache

Recently I forced the array in to a 1 device mirror with

Code:

mdadm -G /dev/md0 --force -n 1

Rebooted and grew the array back to 2 with

Code:

mdadm -G /dev/md0 -n 2

Re-added the device with

Code:

mdadm --add /dev/md0 /dev/sdb1

waited the 2 hours for it to resync, the whole time showing

Code:

root@LoM:~# mdadm -D /dev/md0
/dev/md0:
        Version : 1.2
  Creation Time : Sat Feb 11 14:56:56 2012
     Raid Level : raid1
     Array Size : 976758841 (931.51 GiB 1000.20 GB)
  Used Dev Size : 976758841 (931.51 GiB 1000.20 GB)
   Raid Devices : 2
  Total Devices : 2
    Persistence : Superblock is persistent

    Update Time : Thu Feb 16 22:15:55 2012
          State : active, degraded, recovering
 Active Devices : 1
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 1

 Rebuild Status : 0% complete

           Name : LoM:0  (local to host LoM)
           UUID : 45a552f7:5e854deb:26a64954:01f3aaec
         Events : 153215

    Number   Major   Minor   RaidDevice State
       1       8        1        0      active sync   /dev/sda1
       2       8       17        1      spare rebuilding   /dev/sdb1

and when its finished it drops in to spare and the original slot becomes 'Removed'

Have I missed something?

hallucinative · 02-17-2012, 09:38 AM

A more desperate bump...

sag47 · 02-17-2012, 03:13 PM

I would recommend removing sdb1, zeroing the super block on sdb1 and then re-adding it. I've encountered re-adding problems in the past with a bad, but still readable, superblock.

Code:

mdadm /dev/md0 -r /dev/sdb1
mdadm --zero-superblock /dev/sdb1
mdadm /dev/md0 -a /dev/sdb1

rleibman · 06-14-2012, 03:34 PM

I ended up in the same situation, an active drive, a spare and a degraded array.... it's been a few months, were you able to figure out how to activate the spare?

hallucinative · 06-15-2012, 05:32 AM

g'day rleibman

It turns out in my situation the hard drive was failing but not reporting it back to SmartMon or mdadm, so mdadm failed silently giving no indication of why. Try with another hard drive if you have one and see if that activates, I gave the faulty 1 TB hard drive to a mate who partitioned it avoiding the faulty sector (Through trial and error) and to this day is running fine.

Good luck, Hall