LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Linux - Server (https://www.linuxquestions.org/questions/linux-server-73/)
-   -   mdadm raid inactive, assemble fails with Device or resource busy (https://www.linuxquestions.org/questions/linux-server-73/mdadm-raid-inactive-assemble-fails-with-device-or-resource-busy-4175422782/)

blackman890 08-18-2012 11:40 AM
mdadm raid inactive, assemble fails with Device or resource busy
 
Hi there.

Revcently my raid 5 decided to suddenly mark itself as failing and mounted as read-only. After trying to view the hard drives I didn't see any S.M.A.R.T. failures on the drives it said was failing.

So I thought I should just reboot the machine (big mistake, I know).
After googling some more, trying some more solutions, I seem to have actually made the matter worse. So now I'm trying to make amends for that.

First thing first, my raid setup:

I have a Raid 5 with 6x2TB setup.
2 of those 2TB are actually Raid 0 of 2x1TB

Here is the general setup:

/dev/md0 - Raid 0 on /dev/sda and /dev/sdb
/dev/md1 - Raid 0 on /dev/sdc and /dev/sdd
/dev/md2 - Raid 5 on /dev/md0, /dev/md1, /dev/sde, /dev/sdf, /dev/sdh, /dev/sdg

Here is what I've tried:

Code:
# mdadm --assemble /dev/md2 /dev/md[01] /dev/sd[efhg]
mdadm: cannot open device /dev/md1: Device or resource busy
mdadm: /dev/md1 has no superblock - assembly aborted
Code:
# mdadm --misc --scan --detail /dev/md2
/dev/md2:
        Version : 1.2
  Creation Time : Tue Apr 19 21:09:20 2011
    Raid Level : raid5
  Used Dev Size : 1953513344 (1863.02 GiB 2000.40 GB)
  Raid Devices : 6
  Total Devices : 4
    Persistence : Superblock is persistent

    Update Time : Sat Aug 18 06:50:11 2012
          State : active, FAILED, Not Started
 Active Devices : 4
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 0

        Layout : left-symmetric
    Chunk Size : 128K

          Name : Jupiter:2  (local to host Jupiter)
          UUID : 11946496:1687adf6:4df8ad86:bcaacf99
        Events : 80768

    Number  Major  Minor  RaidDevice State
      0      0        0        0      removed
      1      9        1        1      active sync  /dev/md1
      2      0        0        2      removed
      3      8      80        3      active sync  /dev/sdf
      4      8      112        4      active sync  /dev/sdh
      6      8      96        5      active sync  /dev/sdg
So it has removed both md0 and /dev/sde from the raid 5. Ok, let's try adding them back in.

Code:
~# mdadm --manage /dev/md2 --add /dev/md0
mdadm: add new device failed for /dev/md0 as 7: Invalid argument
Huh, no luck. Trying to get more information:

Code:
~# mdadm -A --verbose /dev/md2 /dev/md[01] /dev/sd[efhg]
mdadm: looking for devices for /dev/md2
mdadm: cannot open device /dev/md1: Device or resource busy
mdadm: /dev/md1 has no superblock - assembly aborted
Well, so much for that. So now I try using scan and verbose:

Code:
~# mdadm --assemble --scan --verbose
mdadm: looking for devices for /dev/md0
mdadm: cannot open device /dev/md1: Device or resource busy
mdadm: /dev/md1 has wrong uuid.
mdadm: /dev/md0 has wrong uuid.
mdadm: cannot open device /dev/sdi5: Device or resource busy
mdadm: /dev/sdi5 has wrong uuid.
mdadm: no recogniseable superblock on /dev/sdi2
mdadm: /dev/sdi2 has wrong uuid.
mdadm: cannot open device /dev/sdi1: Device or resource busy
mdadm: /dev/sdi1 has wrong uuid.
mdadm: cannot open device /dev/sdi: Device or resource busy
mdadm: /dev/sdi has wrong uuid.
mdadm: cannot open device /dev/sdh: Device or resource busy
mdadm: /dev/sdh has wrong uuid.
mdadm: cannot open device /dev/sdg: Device or resource busy
mdadm: /dev/sdg has wrong uuid.
mdadm: cannot open device /dev/sdf: Device or resource busy
mdadm: /dev/sdf has wrong uuid.
mdadm: /dev/sde has wrong uuid.
mdadm: cannot open device /dev/sdd: Device or resource busy
mdadm: /dev/sdd has wrong uuid.
mdadm: cannot open device /dev/sdc: Device or resource busy
mdadm: /dev/sdc has wrong uuid.
mdadm: cannot open device /dev/sdb: Device or resource busy
mdadm: /dev/sdb has wrong uuid.
mdadm: cannot open device /dev/sda: Device or resource busy
mdadm: /dev/sda has wrong uuid.
mdadm: looking for devices for /dev/md1
mdadm: cannot open device /dev/md1: Device or resource busy
mdadm: /dev/md1 has wrong uuid.
mdadm: /dev/md0 has wrong uuid.
mdadm: cannot open device /dev/sdi5: Device or resource busy
mdadm: /dev/sdi5 has wrong uuid.
mdadm: no recogniseable superblock on /dev/sdi2
mdadm: /dev/sdi2 has wrong uuid.
mdadm: cannot open device /dev/sdi1: Device or resource busy
mdadm: /dev/sdi1 has wrong uuid.
mdadm: cannot open device /dev/sdi: Device or resource busy
mdadm: /dev/sdi has wrong uuid.
mdadm: cannot open device /dev/sdh: Device or resource busy
mdadm: /dev/sdh has wrong uuid.
mdadm: cannot open device /dev/sdg: Device or resource busy
mdadm: /dev/sdg has wrong uuid.
mdadm: cannot open device /dev/sdf: Device or resource busy
mdadm: /dev/sdf has wrong uuid.
mdadm: /dev/sde has wrong uuid.
mdadm: cannot open device /dev/sdd: Device or resource busy
mdadm: /dev/sdd has wrong uuid.
mdadm: cannot open device /dev/sdc: Device or resource busy
mdadm: /dev/sdc has wrong uuid.
mdadm: cannot open device /dev/sdb: Device or resource busy
mdadm: /dev/sdb has wrong uuid.
mdadm: cannot open device /dev/sda: Device or resource busy
mdadm: /dev/sda has wrong uuid.
mdadm: looking for devices for /dev/md2
mdadm: cannot open device /dev/md1: Device or resource busy
mdadm: /dev/md1 has wrong uuid.
mdadm: cannot open device /dev/sdi5: Device or resource busy
mdadm: /dev/sdi5 has wrong uuid.
mdadm: no RAID superblock on /dev/sdi2
mdadm: /dev/sdi2 has wrong uuid.
mdadm: cannot open device /dev/sdi1: Device or resource busy
mdadm: /dev/sdi1 has wrong uuid.
mdadm: cannot open device /dev/sdi: Device or resource busy
mdadm: /dev/sdi has wrong uuid.
mdadm: cannot open device /dev/sdh: Device or resource busy
mdadm: /dev/sdh has wrong uuid.
mdadm: cannot open device /dev/sdg: Device or resource busy
mdadm: /dev/sdg has wrong uuid.
mdadm: cannot open device /dev/sdf: Device or resource busy
mdadm: /dev/sdf has wrong uuid.
mdadm: cannot open device /dev/sdd: Device or resource busy
mdadm: /dev/sdd has wrong uuid.
mdadm: cannot open device /dev/sdc: Device or resource busy
mdadm: /dev/sdc has wrong uuid.
mdadm: cannot open device /dev/sdb: Device or resource busy
mdadm: /dev/sdb has wrong uuid.
mdadm: cannot open device /dev/sda: Device or resource busy
mdadm: /dev/sda has wrong uuid.
mdadm: /dev/md2 is already in use.
root@Jupiter:~# mdadm --stop /dev/md2
mdadm: stopped /dev/md2
root@Jupiter:~# mdadm -A --scan --verbose /dev/md2
mdadm: looking for devices for /dev/md2
mdadm: cannot open device /dev/sdi5: Device or resource busy
mdadm: /dev/sdi5 has wrong uuid.
mdadm: no RAID superblock on /dev/sdi2
mdadm: /dev/sdi2 has wrong uuid.
mdadm: cannot open device /dev/sdi1: Device or resource busy
mdadm: /dev/sdi1 has wrong uuid.
mdadm: cannot open device /dev/sdi: Device or resource busy
mdadm: /dev/sdi has wrong uuid.
mdadm: cannot open device /dev/sdd: Device or resource busy
mdadm: /dev/sdd has wrong uuid.
mdadm: cannot open device /dev/sdc: Device or resource busy
mdadm: /dev/sdc has wrong uuid.
mdadm: cannot open device /dev/sdb: Device or resource busy
mdadm: /dev/sdb has wrong uuid.
mdadm: cannot open device /dev/sda: Device or resource busy
mdadm: /dev/sda has wrong uuid.
mdadm: /dev/md1 is identified as a member of /dev/md2, slot 1.
mdadm: /dev/md0 is identified as a member of /dev/md2, slot -1.
mdadm: /dev/sdh is identified as a member of /dev/md2, slot 4.
mdadm: /dev/sdg is identified as a member of /dev/md2, slot 5.
mdadm: /dev/sdf is identified as a member of /dev/md2, slot 3.
mdadm: /dev/sde is identified as a member of /dev/md2, slot -1.
mdadm: no uptodate device for slot 0 of /dev/md2
mdadm: no uptodate device for slot 2 of /dev/md2
mdadm: added /dev/sdf to /dev/md2 as 3
mdadm: added /dev/sdh to /dev/md2 as 4
mdadm: added /dev/sdg to /dev/md2 as 5
mdadm: added /dev/md0 to /dev/md2 as -1
mdadm: failed to add /dev/sde to /dev/md2: Device or resource busy
mdadm: added /dev/md1 to /dev/md2 as 1
mdadm: /dev/md2 assembled from 4 drives and 1 spare - not enough to start the array.
Finally some information. Seems it's "detecting" md0 as slot -1. Not good :(

This is how my mdstat currently look like:

Code:
$ cat /proc/mdstat
Personalities : [raid0] [raid6] [raid5] [raid4]
md2 : inactive md1[1](S) md0[7](S) sdg[6](S) sdh[4](S) sdf[3](S)
      9767587464 blocks super 1.2

md1 : active raid0 sdc[0] sdd[1]
      1953523840 blocks 128k chunks

md0 : active raid0 sda[0] sdb[1]
      1953524992 blocks 128k chunks

unused devices: <none>
And now, I have no idea how I add the previos discs to their designated places.

Please, can someone help me restore the raid.


Update:
Here is my mdadm.conf:

Code:
# mdadm.conf
# by default, scan all partitions (/proc/partitions) for MD superblocks.
# alternatively, specify devices to scan, using wildcards if desired.
DEVICE partitions

# auto-create devices with Debian standard permissions
CREATE owner=root group=disk mode=0660 auto=yes

# automatically tag new arrays as belonging to the local system
HOMEHOST <system>

# instruct the monitoring daemon where to send mail alerts
MAILADDR root

# definitions of existing MD arrays
ARRAY /dev/md0 UUID=f8598a1b:c41aa725:7d57d188:5a244fcc
ARRAY /dev/md1 UUID=f5329c76:2bd13f68:7d57d188:5a244fcc
ARRAY /dev/md2 UUID=11946496:1687adf6:4df8ad86:bcaacf99

sKaar 08-18-2012 05:01 PM
is the raid volume in fstab?

blackman890 08-18-2012 08:06 PM
Only /dev/md2 is in fstab and it's defined as following:

Code:
UUID=4e67cd8b-bbd9-44e0-805e-b81f02e1f420      /mnt/data      ext4    auto,users,rw,relatime 0 0
So I doubt that's the problem.

Update:

Some more investigations and I find this:

Code:
# mdadm --manage /dev/md1 -S
mdadm: failed to stop array /dev/md1: Device or resource busy
Perhaps a running process, mounted filesystem or active volume group?
So maybe that's my problem all along? When I try to activate md2 I always get this error:

Code:
# mdadm -A --verbose /dev/md2 /dev/md[01] /dev/sd[efhg]
mdadm: looking for devices for /dev/md2
mdadm: cannot open device /dev/md1: Device or resource busy
mdadm: /dev/md1 has no superblock - assembly aborted
Although I dunno if that is coming because the first disc is md0, not md1 but...

sKaar 08-18-2012 10:47 PM
was a shot, if it was being mounted by fstab... maybe udev is mounting it?

blackman890 08-19-2012 12:43 AM
It shouldn't be able to mount either /dev/md0 and /dev/md1 cause neither contain a comprehensive file structure on their own.

So right now I'm just stuck on what I can do.

Here's the contents of /sys/block/md2/md:

Code:
root@Jupiter:/sys/block/md2/md# ls
array_size  chunk_size      dev-md1  dev-sdg  layout  metadata_version  raid_disks        resync_start
array_state  component_size  dev-sdf  dev-sdh  level  new_dev          reshape_position  safe_mode_delay
As you can see, it's missing both directoried for md0 and sde.

blackman890 08-19-2012 02:10 PM
So I FINALLY fixed the problem \o/

Code:
root@Jupiter:~# mdadm --create --assume-clean --level=5 --chunk 128 --raid-devices=6 /dev/md3 /dev/md0 /dev/md1 /dev/sde /dev/sdf /dev/sdh /dev/sdg
mdadm: /dev/md0 appears to be part of a raid array:
    level=raid5 devices=6 ctime=Tue Apr 19 21:09:20 2011
mdadm: partition table exists on /dev/md0 but will be lost or
      meaningless after creating array
mdadm: super1.x cannot open /dev/md1: Device or resource busy
mdadm: /dev/md1 is not suitable for this array.
mdadm: /dev/sde appears to be part of a raid array:
    level=raid5 devices=6 ctime=Tue Apr 19 21:09:20 2011
mdadm: super1.x cannot open /dev/sdf: Device or resource busy
mdadm: /dev/sdf is not suitable for this array.
mdadm: super1.x cannot open /dev/sdh: Device or resource busy
mdadm: /dev/sdh is not suitable for this array.
mdadm: super1.x cannot open /dev/sdg: Device or resource busy
mdadm: /dev/sdg is not suitable for this array.
mdadm: create aborted
root@Jupiter:~# mdadm --manage --stop /dev/md2
mdadm: stopped /dev/md2
root@Jupiter:~# mdadm --create --assume-clean --level=5 --chunk 128 --raid-devices=6 /dev/md3 /dev/md0 /dev/md1 /dev/sde /dev/sdf /dev/sdh /dev/sdg
mdadm: /dev/md0 appears to be part of a raid array:
    level=raid5 devices=6 ctime=Tue Apr 19 21:09:20 2011
mdadm: partition table exists on /dev/md0 but will be lost or
      meaningless after creating array
mdadm: /dev/md1 appears to be part of a raid array:
    level=raid5 devices=6 ctime=Tue Apr 19 21:09:20 2011
mdadm: /dev/sde appears to be part of a raid array:
    level=raid5 devices=6 ctime=Tue Apr 19 21:09:20 2011
mdadm: /dev/sdf appears to be part of a raid array:
    level=raid5 devices=6 ctime=Tue Apr 19 21:09:20 2011
mdadm: /dev/sdh appears to be part of a raid array:
    level=raid5 devices=6 ctime=Tue Apr 19 21:09:20 2011
mdadm: partition table exists on /dev/sdh but will be lost or
      meaningless after creating array
mdadm: /dev/sdg appears to be part of a raid array:
    level=raid5 devices=6 ctime=Tue Apr 19 21:09:20 2011
Continue creating array? y
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md3 started.
root@Jupiter:~#
My heart nearly skipped a beat, doing this but wow it worked. Got all my data back and everything (had to run testdisk to recover the partition but other than that, all the data is there).

For others encountering this problem, here's the link I used for reference: http://www.linuxforums.org/forum/ser...id6-array.html

sKaar 08-22-2012 12:54 AM
so much nicer when there's decent backups, no need to fix things, just write over an oops.

lleb 12-14-2014 04:43 PM
im sorry this is a bit late in asking, but when you rebuilt the array to create the raid, did you lose any data?

im in a similar situation with my system. im attempting to recover the raid via a liveusb and im stuck with portions that are busy and portions that are inactive.


All times are GMT -5. The time now is 02:25 PM.