[SOLVED] mdadm raid inactive, assemble fails with Device or resource busy

blackman890 · 08-18-2012, 11:40 AM

Hi there.

Revcently my raid 5 decided to suddenly mark itself as failing and mounted as read-only. After trying to view the hard drives I didn't see any S.M.A.R.T. failures on the drives it said was failing.

So I thought I should just reboot the machine (big mistake, I know).
After googling some more, trying some more solutions, I seem to have actually made the matter worse. So now I'm trying to make amends for that.

First thing first, my raid setup:

I have a Raid 5 with 6x2TB setup.
2 of those 2TB are actually Raid 0 of 2x1TB

Here is the general setup:

/dev/md0 - Raid 0 on /dev/sda and /dev/sdb
/dev/md1 - Raid 0 on /dev/sdc and /dev/sdd
/dev/md2 - Raid 5 on /dev/md0, /dev/md1, /dev/sde, /dev/sdf, /dev/sdh, /dev/sdg

Here is what I've tried:

Code:

# mdadm --assemble /dev/md2 /dev/md[01] /dev/sd[efhg]
mdadm: cannot open device /dev/md1: Device or resource busy
mdadm: /dev/md1 has no superblock - assembly aborted

Code:

# mdadm --misc --scan --detail /dev/md2
/dev/md2:
        Version : 1.2
  Creation Time : Tue Apr 19 21:09:20 2011
     Raid Level : raid5
  Used Dev Size : 1953513344 (1863.02 GiB 2000.40 GB)
   Raid Devices : 6
  Total Devices : 4
    Persistence : Superblock is persistent

    Update Time : Sat Aug 18 06:50:11 2012
          State : active, FAILED, Not Started
 Active Devices : 4
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 128K

           Name : Jupiter:2  (local to host Jupiter)
           UUID : 11946496:1687adf6:4df8ad86:bcaacf99
         Events : 80768

    Number   Major   Minor   RaidDevice State
       0       0        0        0      removed
       1       9        1        1      active sync   /dev/md1
       2       0        0        2      removed
       3       8       80        3      active sync   /dev/sdf
       4       8      112        4      active sync   /dev/sdh
       6       8       96        5      active sync   /dev/sdg

So it has removed both md0 and /dev/sde from the raid 5. Ok, let's try adding them back in.

Code:

~# mdadm --manage /dev/md2 --add /dev/md0
mdadm: add new device failed for /dev/md0 as 7: Invalid argument

Huh, no luck. Trying to get more information:

Code:

~# mdadm -A --verbose /dev/md2 /dev/md[01] /dev/sd[efhg]
mdadm: looking for devices for /dev/md2
mdadm: cannot open device /dev/md1: Device or resource busy
mdadm: /dev/md1 has no superblock - assembly aborted

Well, so much for that. So now I try using scan and verbose:

Code:

~# mdadm --assemble --scan --verbose
mdadm: looking for devices for /dev/md0
mdadm: cannot open device /dev/md1: Device or resource busy
mdadm: /dev/md1 has wrong uuid.
mdadm: /dev/md0 has wrong uuid.
mdadm: cannot open device /dev/sdi5: Device or resource busy
mdadm: /dev/sdi5 has wrong uuid.
mdadm: no recogniseable superblock on /dev/sdi2
mdadm: /dev/sdi2 has wrong uuid.
mdadm: cannot open device /dev/sdi1: Device or resource busy
mdadm: /dev/sdi1 has wrong uuid.
mdadm: cannot open device /dev/sdi: Device or resource busy
mdadm: /dev/sdi has wrong uuid.
mdadm: cannot open device /dev/sdh: Device or resource busy
mdadm: /dev/sdh has wrong uuid.
mdadm: cannot open device /dev/sdg: Device or resource busy
mdadm: /dev/sdg has wrong uuid.
mdadm: cannot open device /dev/sdf: Device or resource busy
mdadm: /dev/sdf has wrong uuid.
mdadm: /dev/sde has wrong uuid.
mdadm: cannot open device /dev/sdd: Device or resource busy
mdadm: /dev/sdd has wrong uuid.
mdadm: cannot open device /dev/sdc: Device or resource busy
mdadm: /dev/sdc has wrong uuid.
mdadm: cannot open device /dev/sdb: Device or resource busy
mdadm: /dev/sdb has wrong uuid.
mdadm: cannot open device /dev/sda: Device or resource busy
mdadm: /dev/sda has wrong uuid.
mdadm: looking for devices for /dev/md1
mdadm: cannot open device /dev/md1: Device or resource busy
mdadm: /dev/md1 has wrong uuid.
mdadm: /dev/md0 has wrong uuid.
mdadm: cannot open device /dev/sdi5: Device or resource busy
mdadm: /dev/sdi5 has wrong uuid.
mdadm: no recogniseable superblock on /dev/sdi2
mdadm: /dev/sdi2 has wrong uuid.
mdadm: cannot open device /dev/sdi1: Device or resource busy
mdadm: /dev/sdi1 has wrong uuid.
mdadm: cannot open device /dev/sdi: Device or resource busy
mdadm: /dev/sdi has wrong uuid.
mdadm: cannot open device /dev/sdh: Device or resource busy
mdadm: /dev/sdh has wrong uuid.
mdadm: cannot open device /dev/sdg: Device or resource busy
mdadm: /dev/sdg has wrong uuid.
mdadm: cannot open device /dev/sdf: Device or resource busy
mdadm: /dev/sdf has wrong uuid.
mdadm: /dev/sde has wrong uuid.
mdadm: cannot open device /dev/sdd: Device or resource busy
mdadm: /dev/sdd has wrong uuid.
mdadm: cannot open device /dev/sdc: Device or resource busy
mdadm: /dev/sdc has wrong uuid.
mdadm: cannot open device /dev/sdb: Device or resource busy
mdadm: /dev/sdb has wrong uuid.
mdadm: cannot open device /dev/sda: Device or resource busy
mdadm: /dev/sda has wrong uuid.
mdadm: looking for devices for /dev/md2
mdadm: cannot open device /dev/md1: Device or resource busy
mdadm: /dev/md1 has wrong uuid.
mdadm: cannot open device /dev/sdi5: Device or resource busy
mdadm: /dev/sdi5 has wrong uuid.
mdadm: no RAID superblock on /dev/sdi2
mdadm: /dev/sdi2 has wrong uuid.
mdadm: cannot open device /dev/sdi1: Device or resource busy
mdadm: /dev/sdi1 has wrong uuid.
mdadm: cannot open device /dev/sdi: Device or resource busy
mdadm: /dev/sdi has wrong uuid.
mdadm: cannot open device /dev/sdh: Device or resource busy
mdadm: /dev/sdh has wrong uuid.
mdadm: cannot open device /dev/sdg: Device or resource busy
mdadm: /dev/sdg has wrong uuid.
mdadm: cannot open device /dev/sdf: Device or resource busy
mdadm: /dev/sdf has wrong uuid.
mdadm: cannot open device /dev/sdd: Device or resource busy
mdadm: /dev/sdd has wrong uuid.
mdadm: cannot open device /dev/sdc: Device or resource busy
mdadm: /dev/sdc has wrong uuid.
mdadm: cannot open device /dev/sdb: Device or resource busy
mdadm: /dev/sdb has wrong uuid.
mdadm: cannot open device /dev/sda: Device or resource busy
mdadm: /dev/sda has wrong uuid.
mdadm: /dev/md2 is already in use.
root@Jupiter:~# mdadm --stop /dev/md2
mdadm: stopped /dev/md2
root@Jupiter:~# mdadm -A --scan --verbose /dev/md2
mdadm: looking for devices for /dev/md2
mdadm: cannot open device /dev/sdi5: Device or resource busy
mdadm: /dev/sdi5 has wrong uuid.
mdadm: no RAID superblock on /dev/sdi2
mdadm: /dev/sdi2 has wrong uuid.
mdadm: cannot open device /dev/sdi1: Device or resource busy
mdadm: /dev/sdi1 has wrong uuid.
mdadm: cannot open device /dev/sdi: Device or resource busy
mdadm: /dev/sdi has wrong uuid.
mdadm: cannot open device /dev/sdd: Device or resource busy
mdadm: /dev/sdd has wrong uuid.
mdadm: cannot open device /dev/sdc: Device or resource busy
mdadm: /dev/sdc has wrong uuid.
mdadm: cannot open device /dev/sdb: Device or resource busy
mdadm: /dev/sdb has wrong uuid.
mdadm: cannot open device /dev/sda: Device or resource busy
mdadm: /dev/sda has wrong uuid.
mdadm: /dev/md1 is identified as a member of /dev/md2, slot 1.
mdadm: /dev/md0 is identified as a member of /dev/md2, slot -1.
mdadm: /dev/sdh is identified as a member of /dev/md2, slot 4.
mdadm: /dev/sdg is identified as a member of /dev/md2, slot 5.
mdadm: /dev/sdf is identified as a member of /dev/md2, slot 3.
mdadm: /dev/sde is identified as a member of /dev/md2, slot -1.
mdadm: no uptodate device for slot 0 of /dev/md2
mdadm: no uptodate device for slot 2 of /dev/md2
mdadm: added /dev/sdf to /dev/md2 as 3
mdadm: added /dev/sdh to /dev/md2 as 4
mdadm: added /dev/sdg to /dev/md2 as 5
mdadm: added /dev/md0 to /dev/md2 as -1
mdadm: failed to add /dev/sde to /dev/md2: Device or resource busy
mdadm: added /dev/md1 to /dev/md2 as 1
mdadm: /dev/md2 assembled from 4 drives and 1 spare - not enough to start the array.

Finally some information. Seems it's "detecting" md0 as slot -1. Not good

This is how my mdstat currently look like:

Code:

$ cat /proc/mdstat
Personalities : [raid0] [raid6] [raid5] [raid4]
md2 : inactive md1[1](S) md0[7](S) sdg[6](S) sdh[4](S) sdf[3](S)
      9767587464 blocks super 1.2

md1 : active raid0 sdc[0] sdd[1]
      1953523840 blocks 128k chunks

md0 : active raid0 sda[0] sdb[1]
      1953524992 blocks 128k chunks

unused devices: <none>

And now, I have no idea how I add the previos discs to their designated places.

Please, can someone help me restore the raid.

Update:
Here is my mdadm.conf:

Code:

# mdadm.conf
# by default, scan all partitions (/proc/partitions) for MD superblocks.
# alternatively, specify devices to scan, using wildcards if desired.
DEVICE partitions

# auto-create devices with Debian standard permissions
CREATE owner=root group=disk mode=0660 auto=yes

# automatically tag new arrays as belonging to the local system
HOMEHOST <system>

# instruct the monitoring daemon where to send mail alerts
MAILADDR root

# definitions of existing MD arrays
ARRAY /dev/md0 UUID=f8598a1b:c41aa725:7d57d188:5a244fcc
ARRAY /dev/md1 UUID=f5329c76:2bd13f68:7d57d188:5a244fcc
ARRAY /dev/md2 UUID=11946496:1687adf6:4df8ad86:bcaacf99

sKaar · 08-18-2012, 05:01 PM

is the raid volume in fstab?

blackman890 · 08-18-2012, 08:06 PM

Only /dev/md2 is in fstab and it's defined as following:

Code:

UUID=4e67cd8b-bbd9-44e0-805e-b81f02e1f420       /mnt/data       ext4    auto,users,rw,relatime 0 0

So I doubt that's the problem.

Update:

Some more investigations and I find this:

Code:

# mdadm --manage /dev/md1 -S
mdadm: failed to stop array /dev/md1: Device or resource busy
Perhaps a running process, mounted filesystem or active volume group?

So maybe that's my problem all along? When I try to activate md2 I always get this error:

Code:

# mdadm -A --verbose /dev/md2 /dev/md[01] /dev/sd[efhg]
mdadm: looking for devices for /dev/md2
mdadm: cannot open device /dev/md1: Device or resource busy
mdadm: /dev/md1 has no superblock - assembly aborted

Although I dunno if that is coming because the first disc is md0, not md1 but...

sKaar · 08-18-2012, 10:47 PM

was a shot, if it was being mounted by fstab... maybe udev is mounting it?

blackman890 · 08-19-2012, 12:43 AM

It shouldn't be able to mount either /dev/md0 and /dev/md1 cause neither contain a comprehensive file structure on their own.

So right now I'm just stuck on what I can do.

Here's the contents of /sys/block/md2/md:

Code:

root@Jupiter:/sys/block/md2/md# ls
array_size   chunk_size      dev-md1  dev-sdg  layout  metadata_version  raid_disks        resync_start
array_state  component_size  dev-sdf  dev-sdh  level   new_dev           reshape_position  safe_mode_delay

As you can see, it's missing both directoried for md0 and sde.

blackman890 · 08-19-2012, 02:10 PM

So I FINALLY fixed the problem \o/

Code:

root@Jupiter:~# mdadm --create --assume-clean --level=5 --chunk 128 --raid-devices=6 /dev/md3 /dev/md0 /dev/md1 /dev/sde /dev/sdf /dev/sdh /dev/sdg
mdadm: /dev/md0 appears to be part of a raid array:
    level=raid5 devices=6 ctime=Tue Apr 19 21:09:20 2011
mdadm: partition table exists on /dev/md0 but will be lost or
       meaningless after creating array
mdadm: super1.x cannot open /dev/md1: Device or resource busy
mdadm: /dev/md1 is not suitable for this array.
mdadm: /dev/sde appears to be part of a raid array:
    level=raid5 devices=6 ctime=Tue Apr 19 21:09:20 2011
mdadm: super1.x cannot open /dev/sdf: Device or resource busy
mdadm: /dev/sdf is not suitable for this array.
mdadm: super1.x cannot open /dev/sdh: Device or resource busy
mdadm: /dev/sdh is not suitable for this array.
mdadm: super1.x cannot open /dev/sdg: Device or resource busy
mdadm: /dev/sdg is not suitable for this array.
mdadm: create aborted
root@Jupiter:~# mdadm --manage --stop /dev/md2
mdadm: stopped /dev/md2
root@Jupiter:~# mdadm --create --assume-clean --level=5 --chunk 128 --raid-devices=6 /dev/md3 /dev/md0 /dev/md1 /dev/sde /dev/sdf /dev/sdh /dev/sdg
mdadm: /dev/md0 appears to be part of a raid array:
    level=raid5 devices=6 ctime=Tue Apr 19 21:09:20 2011
mdadm: partition table exists on /dev/md0 but will be lost or
       meaningless after creating array
mdadm: /dev/md1 appears to be part of a raid array:
    level=raid5 devices=6 ctime=Tue Apr 19 21:09:20 2011
mdadm: /dev/sde appears to be part of a raid array:
    level=raid5 devices=6 ctime=Tue Apr 19 21:09:20 2011
mdadm: /dev/sdf appears to be part of a raid array:
    level=raid5 devices=6 ctime=Tue Apr 19 21:09:20 2011
mdadm: /dev/sdh appears to be part of a raid array:
    level=raid5 devices=6 ctime=Tue Apr 19 21:09:20 2011
mdadm: partition table exists on /dev/sdh but will be lost or
       meaningless after creating array
mdadm: /dev/sdg appears to be part of a raid array:
    level=raid5 devices=6 ctime=Tue Apr 19 21:09:20 2011
Continue creating array? y
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md3 started.
root@Jupiter:~#

My heart nearly skipped a beat, doing this but wow it worked. Got all my data back and everything (had to run testdisk to recover the partition but other than that, all the data is there).

For others encountering this problem, here's the link I used for reference: http://www.linuxforums.org/forum/ser...id6-array.html

sKaar · 08-22-2012, 12:54 AM

so much nicer when there's decent backups, no need to fix things, just write over an oops.

lleb · 12-14-2014, 04:43 PM

im sorry this is a bit late in asking, but when you rebuilt the array to create the raid, did you lose any data?

im in a similar situation with my system. im attempting to recover the raid via a liveusb and im stuck with portions that are busy and portions that are inactive.