LinuxQuestions.org
Register a domain and help support LQ
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices

Reply
 
Search this Thread
Old 08-18-2012, 11:40 AM   #1
blackman890
Member
 
Registered: Oct 2004
Location: Iceland
Posts: 94

Rep: Reputation: 15
mdadm raid inactive, assemble fails with Device or resource busy


Hi there.

Revcently my raid 5 decided to suddenly mark itself as failing and mounted as read-only. After trying to view the hard drives I didn't see any S.M.A.R.T. failures on the drives it said was failing.

So I thought I should just reboot the machine (big mistake, I know).
After googling some more, trying some more solutions, I seem to have actually made the matter worse. So now I'm trying to make amends for that.

First thing first, my raid setup:

I have a Raid 5 with 6x2TB setup.
2 of those 2TB are actually Raid 0 of 2x1TB

Here is the general setup:

/dev/md0 - Raid 0 on /dev/sda and /dev/sdb
/dev/md1 - Raid 0 on /dev/sdc and /dev/sdd
/dev/md2 - Raid 5 on /dev/md0, /dev/md1, /dev/sde, /dev/sdf, /dev/sdh, /dev/sdg

Here is what I've tried:

Code:
# mdadm --assemble /dev/md2 /dev/md[01] /dev/sd[efhg]
mdadm: cannot open device /dev/md1: Device or resource busy
mdadm: /dev/md1 has no superblock - assembly aborted
Code:
# mdadm --misc --scan --detail /dev/md2
/dev/md2:
        Version : 1.2
  Creation Time : Tue Apr 19 21:09:20 2011
     Raid Level : raid5
  Used Dev Size : 1953513344 (1863.02 GiB 2000.40 GB)
   Raid Devices : 6
  Total Devices : 4
    Persistence : Superblock is persistent

    Update Time : Sat Aug 18 06:50:11 2012
          State : active, FAILED, Not Started
 Active Devices : 4
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 128K

           Name : Jupiter:2  (local to host Jupiter)
           UUID : 11946496:1687adf6:4df8ad86:bcaacf99
         Events : 80768

    Number   Major   Minor   RaidDevice State
       0       0        0        0      removed
       1       9        1        1      active sync   /dev/md1
       2       0        0        2      removed
       3       8       80        3      active sync   /dev/sdf
       4       8      112        4      active sync   /dev/sdh
       6       8       96        5      active sync   /dev/sdg
So it has removed both md0 and /dev/sde from the raid 5. Ok, let's try adding them back in.

Code:
~# mdadm --manage /dev/md2 --add /dev/md0
mdadm: add new device failed for /dev/md0 as 7: Invalid argument
Huh, no luck. Trying to get more information:

Code:
~# mdadm -A --verbose /dev/md2 /dev/md[01] /dev/sd[efhg]
mdadm: looking for devices for /dev/md2
mdadm: cannot open device /dev/md1: Device or resource busy
mdadm: /dev/md1 has no superblock - assembly aborted
Well, so much for that. So now I try using scan and verbose:

Code:
~# mdadm --assemble --scan --verbose
mdadm: looking for devices for /dev/md0
mdadm: cannot open device /dev/md1: Device or resource busy
mdadm: /dev/md1 has wrong uuid.
mdadm: /dev/md0 has wrong uuid.
mdadm: cannot open device /dev/sdi5: Device or resource busy
mdadm: /dev/sdi5 has wrong uuid.
mdadm: no recogniseable superblock on /dev/sdi2
mdadm: /dev/sdi2 has wrong uuid.
mdadm: cannot open device /dev/sdi1: Device or resource busy
mdadm: /dev/sdi1 has wrong uuid.
mdadm: cannot open device /dev/sdi: Device or resource busy
mdadm: /dev/sdi has wrong uuid.
mdadm: cannot open device /dev/sdh: Device or resource busy
mdadm: /dev/sdh has wrong uuid.
mdadm: cannot open device /dev/sdg: Device or resource busy
mdadm: /dev/sdg has wrong uuid.
mdadm: cannot open device /dev/sdf: Device or resource busy
mdadm: /dev/sdf has wrong uuid.
mdadm: /dev/sde has wrong uuid.
mdadm: cannot open device /dev/sdd: Device or resource busy
mdadm: /dev/sdd has wrong uuid.
mdadm: cannot open device /dev/sdc: Device or resource busy
mdadm: /dev/sdc has wrong uuid.
mdadm: cannot open device /dev/sdb: Device or resource busy
mdadm: /dev/sdb has wrong uuid.
mdadm: cannot open device /dev/sda: Device or resource busy
mdadm: /dev/sda has wrong uuid.
mdadm: looking for devices for /dev/md1
mdadm: cannot open device /dev/md1: Device or resource busy
mdadm: /dev/md1 has wrong uuid.
mdadm: /dev/md0 has wrong uuid.
mdadm: cannot open device /dev/sdi5: Device or resource busy
mdadm: /dev/sdi5 has wrong uuid.
mdadm: no recogniseable superblock on /dev/sdi2
mdadm: /dev/sdi2 has wrong uuid.
mdadm: cannot open device /dev/sdi1: Device or resource busy
mdadm: /dev/sdi1 has wrong uuid.
mdadm: cannot open device /dev/sdi: Device or resource busy
mdadm: /dev/sdi has wrong uuid.
mdadm: cannot open device /dev/sdh: Device or resource busy
mdadm: /dev/sdh has wrong uuid.
mdadm: cannot open device /dev/sdg: Device or resource busy
mdadm: /dev/sdg has wrong uuid.
mdadm: cannot open device /dev/sdf: Device or resource busy
mdadm: /dev/sdf has wrong uuid.
mdadm: /dev/sde has wrong uuid.
mdadm: cannot open device /dev/sdd: Device or resource busy
mdadm: /dev/sdd has wrong uuid.
mdadm: cannot open device /dev/sdc: Device or resource busy
mdadm: /dev/sdc has wrong uuid.
mdadm: cannot open device /dev/sdb: Device or resource busy
mdadm: /dev/sdb has wrong uuid.
mdadm: cannot open device /dev/sda: Device or resource busy
mdadm: /dev/sda has wrong uuid.
mdadm: looking for devices for /dev/md2
mdadm: cannot open device /dev/md1: Device or resource busy
mdadm: /dev/md1 has wrong uuid.
mdadm: cannot open device /dev/sdi5: Device or resource busy
mdadm: /dev/sdi5 has wrong uuid.
mdadm: no RAID superblock on /dev/sdi2
mdadm: /dev/sdi2 has wrong uuid.
mdadm: cannot open device /dev/sdi1: Device or resource busy
mdadm: /dev/sdi1 has wrong uuid.
mdadm: cannot open device /dev/sdi: Device or resource busy
mdadm: /dev/sdi has wrong uuid.
mdadm: cannot open device /dev/sdh: Device or resource busy
mdadm: /dev/sdh has wrong uuid.
mdadm: cannot open device /dev/sdg: Device or resource busy
mdadm: /dev/sdg has wrong uuid.
mdadm: cannot open device /dev/sdf: Device or resource busy
mdadm: /dev/sdf has wrong uuid.
mdadm: cannot open device /dev/sdd: Device or resource busy
mdadm: /dev/sdd has wrong uuid.
mdadm: cannot open device /dev/sdc: Device or resource busy
mdadm: /dev/sdc has wrong uuid.
mdadm: cannot open device /dev/sdb: Device or resource busy
mdadm: /dev/sdb has wrong uuid.
mdadm: cannot open device /dev/sda: Device or resource busy
mdadm: /dev/sda has wrong uuid.
mdadm: /dev/md2 is already in use.
root@Jupiter:~# mdadm --stop /dev/md2
mdadm: stopped /dev/md2
root@Jupiter:~# mdadm -A --scan --verbose /dev/md2
mdadm: looking for devices for /dev/md2
mdadm: cannot open device /dev/sdi5: Device or resource busy
mdadm: /dev/sdi5 has wrong uuid.
mdadm: no RAID superblock on /dev/sdi2
mdadm: /dev/sdi2 has wrong uuid.
mdadm: cannot open device /dev/sdi1: Device or resource busy
mdadm: /dev/sdi1 has wrong uuid.
mdadm: cannot open device /dev/sdi: Device or resource busy
mdadm: /dev/sdi has wrong uuid.
mdadm: cannot open device /dev/sdd: Device or resource busy
mdadm: /dev/sdd has wrong uuid.
mdadm: cannot open device /dev/sdc: Device or resource busy
mdadm: /dev/sdc has wrong uuid.
mdadm: cannot open device /dev/sdb: Device or resource busy
mdadm: /dev/sdb has wrong uuid.
mdadm: cannot open device /dev/sda: Device or resource busy
mdadm: /dev/sda has wrong uuid.
mdadm: /dev/md1 is identified as a member of /dev/md2, slot 1.
mdadm: /dev/md0 is identified as a member of /dev/md2, slot -1.
mdadm: /dev/sdh is identified as a member of /dev/md2, slot 4.
mdadm: /dev/sdg is identified as a member of /dev/md2, slot 5.
mdadm: /dev/sdf is identified as a member of /dev/md2, slot 3.
mdadm: /dev/sde is identified as a member of /dev/md2, slot -1.
mdadm: no uptodate device for slot 0 of /dev/md2
mdadm: no uptodate device for slot 2 of /dev/md2
mdadm: added /dev/sdf to /dev/md2 as 3
mdadm: added /dev/sdh to /dev/md2 as 4
mdadm: added /dev/sdg to /dev/md2 as 5
mdadm: added /dev/md0 to /dev/md2 as -1
mdadm: failed to add /dev/sde to /dev/md2: Device or resource busy
mdadm: added /dev/md1 to /dev/md2 as 1
mdadm: /dev/md2 assembled from 4 drives and 1 spare - not enough to start the array.
Finally some information. Seems it's "detecting" md0 as slot -1. Not good

This is how my mdstat currently look like:

Code:
$ cat /proc/mdstat
Personalities : [raid0] [raid6] [raid5] [raid4]
md2 : inactive md1[1](S) md0[7](S) sdg[6](S) sdh[4](S) sdf[3](S)
      9767587464 blocks super 1.2

md1 : active raid0 sdc[0] sdd[1]
      1953523840 blocks 128k chunks

md0 : active raid0 sda[0] sdb[1]
      1953524992 blocks 128k chunks

unused devices: <none>
And now, I have no idea how I add the previos discs to their designated places.

Please, can someone help me restore the raid.


Update:
Here is my mdadm.conf:

Code:
# mdadm.conf
# by default, scan all partitions (/proc/partitions) for MD superblocks.
# alternatively, specify devices to scan, using wildcards if desired.
DEVICE partitions

# auto-create devices with Debian standard permissions
CREATE owner=root group=disk mode=0660 auto=yes

# automatically tag new arrays as belonging to the local system
HOMEHOST <system>

# instruct the monitoring daemon where to send mail alerts
MAILADDR root

# definitions of existing MD arrays
ARRAY /dev/md0 UUID=f8598a1b:c41aa725:7d57d188:5a244fcc
ARRAY /dev/md1 UUID=f5329c76:2bd13f68:7d57d188:5a244fcc
ARRAY /dev/md2 UUID=11946496:1687adf6:4df8ad86:bcaacf99

Last edited by blackman890; 08-18-2012 at 11:44 AM.
 
Old 08-18-2012, 05:01 PM   #2
sKaar
Member
 
Registered: Jun 2006
Location: dartmouth, nova scotia
Distribution: slackware 12.1
Posts: 74

Rep: Reputation: 2
is the raid volume in fstab?
 
Old 08-18-2012, 08:06 PM   #3
blackman890
Member
 
Registered: Oct 2004
Location: Iceland
Posts: 94

Original Poster
Rep: Reputation: 15
Only /dev/md2 is in fstab and it's defined as following:

Code:
UUID=4e67cd8b-bbd9-44e0-805e-b81f02e1f420       /mnt/data       ext4    auto,users,rw,relatime 0 0
So I doubt that's the problem.

Update:

Some more investigations and I find this:

Code:
# mdadm --manage /dev/md1 -S
mdadm: failed to stop array /dev/md1: Device or resource busy
Perhaps a running process, mounted filesystem or active volume group?
So maybe that's my problem all along? When I try to activate md2 I always get this error:

Code:
# mdadm -A --verbose /dev/md2 /dev/md[01] /dev/sd[efhg]
mdadm: looking for devices for /dev/md2
mdadm: cannot open device /dev/md1: Device or resource busy
mdadm: /dev/md1 has no superblock - assembly aborted
Although I dunno if that is coming because the first disc is md0, not md1 but...

Last edited by blackman890; 08-18-2012 at 08:13 PM.
 
Old 08-18-2012, 10:47 PM   #4
sKaar
Member
 
Registered: Jun 2006
Location: dartmouth, nova scotia
Distribution: slackware 12.1
Posts: 74

Rep: Reputation: 2
was a shot, if it was being mounted by fstab... maybe udev is mounting it?
 
Old 08-19-2012, 12:43 AM   #5
blackman890
Member
 
Registered: Oct 2004
Location: Iceland
Posts: 94

Original Poster
Rep: Reputation: 15
It shouldn't be able to mount either /dev/md0 and /dev/md1 cause neither contain a comprehensive file structure on their own.

So right now I'm just stuck on what I can do.

Here's the contents of /sys/block/md2/md:

Code:
root@Jupiter:/sys/block/md2/md# ls
array_size   chunk_size      dev-md1  dev-sdg  layout  metadata_version  raid_disks        resync_start
array_state  component_size  dev-sdf  dev-sdh  level   new_dev           reshape_position  safe_mode_delay
As you can see, it's missing both directoried for md0 and sde.
 
Old 08-19-2012, 02:10 PM   #6
blackman890
Member
 
Registered: Oct 2004
Location: Iceland
Posts: 94

Original Poster
Rep: Reputation: 15
So I FINALLY fixed the problem \o/

Code:
root@Jupiter:~# mdadm --create --assume-clean --level=5 --chunk 128 --raid-devices=6 /dev/md3 /dev/md0 /dev/md1 /dev/sde /dev/sdf /dev/sdh /dev/sdg
mdadm: /dev/md0 appears to be part of a raid array:
    level=raid5 devices=6 ctime=Tue Apr 19 21:09:20 2011
mdadm: partition table exists on /dev/md0 but will be lost or
       meaningless after creating array
mdadm: super1.x cannot open /dev/md1: Device or resource busy
mdadm: /dev/md1 is not suitable for this array.
mdadm: /dev/sde appears to be part of a raid array:
    level=raid5 devices=6 ctime=Tue Apr 19 21:09:20 2011
mdadm: super1.x cannot open /dev/sdf: Device or resource busy
mdadm: /dev/sdf is not suitable for this array.
mdadm: super1.x cannot open /dev/sdh: Device or resource busy
mdadm: /dev/sdh is not suitable for this array.
mdadm: super1.x cannot open /dev/sdg: Device or resource busy
mdadm: /dev/sdg is not suitable for this array.
mdadm: create aborted
root@Jupiter:~# mdadm --manage --stop /dev/md2
mdadm: stopped /dev/md2
root@Jupiter:~# mdadm --create --assume-clean --level=5 --chunk 128 --raid-devices=6 /dev/md3 /dev/md0 /dev/md1 /dev/sde /dev/sdf /dev/sdh /dev/sdg
mdadm: /dev/md0 appears to be part of a raid array:
    level=raid5 devices=6 ctime=Tue Apr 19 21:09:20 2011
mdadm: partition table exists on /dev/md0 but will be lost or
       meaningless after creating array
mdadm: /dev/md1 appears to be part of a raid array:
    level=raid5 devices=6 ctime=Tue Apr 19 21:09:20 2011
mdadm: /dev/sde appears to be part of a raid array:
    level=raid5 devices=6 ctime=Tue Apr 19 21:09:20 2011
mdadm: /dev/sdf appears to be part of a raid array:
    level=raid5 devices=6 ctime=Tue Apr 19 21:09:20 2011
mdadm: /dev/sdh appears to be part of a raid array:
    level=raid5 devices=6 ctime=Tue Apr 19 21:09:20 2011
mdadm: partition table exists on /dev/sdh but will be lost or
       meaningless after creating array
mdadm: /dev/sdg appears to be part of a raid array:
    level=raid5 devices=6 ctime=Tue Apr 19 21:09:20 2011
Continue creating array? y
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md3 started.
root@Jupiter:~#
My heart nearly skipped a beat, doing this but wow it worked. Got all my data back and everything (had to run testdisk to recover the partition but other than that, all the data is there).

For others encountering this problem, here's the link I used for reference: http://www.linuxforums.org/forum/ser...id6-array.html
 
Old 08-22-2012, 12:54 AM   #7
sKaar
Member
 
Registered: Jun 2006
Location: dartmouth, nova scotia
Distribution: slackware 12.1
Posts: 74

Rep: Reputation: 2
so much nicer when there's decent backups, no need to fix things, just write over an oops.
 
1 members found this post helpful.
  


Reply

Tags
debian, mdadm, raid


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] mdadm RAID 5 assemble sag47 Linux - Software 1 12-17-2011 06:13 PM
Debian 5 mdadm "device or resource busy" raid edwardzc Debian 1 12-27-2010 04:52 AM
[SOLVED] mdadm raid1 partially works, however device or resource busy when adding sda1 q630 Linux - Software 6 07-13-2010 08:11 PM
mdadm fails to assemble reconnected drives (debian, raid0) SandRock Linux - Software 12 11-09-2008 12:06 PM
mdadm fails to assemble my RAID device tomhildebrand Fedora 6 06-28-2007 12:08 AM


All times are GMT -5. The time now is 09:43 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration