LinuxQuestions.org
View the Most Wanted LQ Wiki articles.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices

Reply
 
Search this Thread
Old 06-29-2009, 09:55 AM   #1
twinge
LQ Newbie
 
Registered: Jun 2009
Posts: 1

Rep: Reputation: 0
Help recoving a failed MD RAID-5 on openSUSE 11.1


Hello all,
I set up a server running openSuSE 11.1 a few weeks back for my own use at home. It was configured with 4x1TB hard drives and I used software raid to set these four drives into a raid-5 array. This morning I turned on the server to find it had re-booted itself, possibly from a power failure or brownout, and failed to boot back up. The software raid is giving me an error and I have spent all of today searching the internet with a way to get the array back up but come up with nothing. Any help would be appreciated. Here is some info on what is going on:

- physically the hard drives are fine - I did a full sector-by-sector scan on each one using Western Digitals SMART utilities and they all passed without a flaw.
- /dev/md0 is a 16gb raid-0 array for swap consisting of /dev/sda2, /dev/sdb2, /dev/sdc2, /dev/sdd2 - this array works fine
- /dev/md1 is a 3TB raid-5 array for the OS and Data consisting of /dev/sda3, /dev/sdb3, /dev/sdc3, /dev/sdd3 - this is the array giving me issues
- the /boot partition is /dev/sda1

When I boot I receive this error message and then I am dumped into the SH shell early in the boot process. I have access to some basic command line utils like mdadm but lots of stuff is missing.

Code:
md: md1 stopped.
md: bind<sdb3>
md: bind<sdc3>
md: bind<sdd3>
md: bind<sda3>
raid5: device sda3 operational as raid disk 0
raid5: device sdd3 operational as raid disk 3
raid5: device sdc3 operational as raid disk 2
raid5: device sdb3 operational as raid disk 1
raid5: allocated 4288kB for md1
raid5: raid level 5 set md1 active with 4 out of 4 devices, algorithm 0
RAID5 conf printout:
 --- rd:4 wd:4
 disk 0, o:1, dev:sda3
 disk 1, o:1, dev:sdb3
 disk 2, o:1, dev:sdc3
 disk 3, o:1, dev:sdd3
md1: bitmap file is out of date, doing full recovery
md1: bitmap initialisation failed: -5
md1: failed to create bitmap (-5)
mdadm: failed to RUN_ARRAY /dev/md/1: Input/output error.
invalid root filesystem -- exiting to /bin/sh
$
Here is the info for my array and the four disks attached to it:

Code:
$mdadm -D /dev/md1
/dev/md1:
        Version : 1.00
  Creation Time : Thu Jun 18 15:15:16 2009
     Raid Level : raid5
  Used Dev Size : 972462464 (927.41 GiB 995.80 GB)
   Raid Devices : 4
  Total Devices : 4
    Persistence : Superblock is persistent

    Update Time : Sun Jun 28 10:50:07 2009
          State : active, Not Started
 Active Devices : 4
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-asymmetric
     Chunk Size : 128K

           Name : linux:1
           UUID : e9a0da25:0bce6c41:0330678f:44257bba
         Events : 6303

    Number   Major   Minor   RaidDevice State
       0       8        3        0      active sync   /dev/sda3
       1       8       19        1      active sync   /dev/sdb3
       2       8       35        2      active sync   /dev/sdc3
       4       8       51        3      active sync   /dev/sdd3

$mdadm --examine /dev/sda3
/dev/sda3:
          Magic : a92b4efc
        Version : 1.0
    Feature Map : 0x1
     Array UUID : e9a0da25:0bce6c41:0330678f:44257bba
           Name : linux:1
  Creation Time : Thu Jun 18 15:15:16 2009
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 1944925016 (927.41 GiB 995.80 GB)
     Array Size : 5834774784 (2782.24 GiB 2987.40 GB)
  Used Dev Size : 1944924928 (927.41 GiB 995.80 GB)
   Super Offset : 1944925272 sectors
          State : clean
    Device UUID : eabd52cf:8b404ce5:57000a4a:74617399

Internal Bitmap : -233 sectors from superblock
    Update Time : Sun Jun 28 10:50:07 2009
       Checksum : 5858713e - correct
         Events : 6303

         Layout : left-asymmetric
     Chunk Size : 128K

    Array Slot : 0 (0, 1, 2, failed, 3)
   Array State : Uuuu 1 failed

$mdadm --examine /dev/sdb3
/dev/sdb3:
          Magic : a92b4efc
        Version : 1.0
    Feature Map : 0x1
     Array UUID : e9a0da25:0bce6c41:0330678f:44257bba
           Name : linux:1
  Creation Time : Thu Jun 18 15:15:16 2009
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 1944925016 (927.41 GiB 995.80 GB)
     Array Size : 5834774784 (2782.24 GiB 2987.40 GB)
  Used Dev Size : 1944924928 (927.41 GiB 995.80 GB)
   Super Offset : 1944925272 sectors
          State : active
    Device UUID : dbe61e90:6a957602:8ad6d54c:b561a4f6

Internal Bitmap : -233 sectors from superblock
    Update Time : Sun Jun 28 10:50:07 2009
       Checksum : 964bc582 - correct
         Events : 6303

         Layout : left-asymmetric
     Chunk Size : 128K

    Array Slot : 1 (0, 1, 2, failed, 3)
   Array State : uUuu 1 failed

$mdadm --examine /dev/sdc3
/dev/sdc3:
          Magic : a92b4efc
        Version : 1.0
    Feature Map : 0x1
     Array UUID : e9a0da25:0bce6c41:0330678f:44257bba
           Name : linux:1
  Creation Time : Thu Jun 18 15:15:16 2009
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 1944925016 (927.41 GiB 995.80 GB)
     Array Size : 5834774784 (2782.24 GiB 2987.40 GB)
  Used Dev Size : 1944924928 (927.41 GiB 995.80 GB)
   Super Offset : 1944925272 sectors
          State : active
    Device UUID : cbd92876:238d1eb7:4bc2e26e:ca7d581a

Internal Bitmap : -233 sectors from superblock
    Update Time : Sun Jun 28 10:50:07 2009
       Checksum : 76beb802 - correct
         Events : 6303

         Layout : left-asymmetric
     Chunk Size : 128K

    Array Slot : 2 (0, 1, 2, failed, 3)
   Array State : uuUu 1 failed

$mdadm --examine /dev/sdd3
/dev/sdd3:
          Magic : a92b4efc
        Version : 1.0
    Feature Map : 0x1
     Array UUID : e9a0da25:0bce6c41:0330678f:44257bba
           Name : linux:1
  Creation Time : Thu Jun 18 15:15:16 2009
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 1944925016 (927.41 GiB 995.80 GB)
     Array Size : 5834774784 (2782.24 GiB 2987.40 GB)
  Used Dev Size : 1944924928 (927.41 GiB 995.80 GB)
   Super Offset : 1944925272 sectors
          State : active
    Device UUID : b8eaa407:238e9d80:7b8e7eef:a7b56aaa

Internal Bitmap : -233 sectors from superblock
    Update Time : Sun Jun 28 10:50:07 2009
       Checksum : e267cdfe - correct
         Events : 6303

         Layout : left-asymmetric
     Chunk Size : 128K

    Array Slot : 4 (0, 1, 2, failed, 3)
   Array State : uuuU 1 failed
Note: /dev/sdd3 says it is using slot 4 but the valid slots are 0,1,2,3,failed - this looks off to me. It also says the state of the array has 1 failed HDD but I cant find which one it is and the result from mdadm -D /dec/md1 says none are failed?

I have tried forcing the array to mount but the result is the same as when booting:

Code:
$mdadm -A -f /dev/md1
md: md1 stopped.
md: bind<sdb3>
md: bind<sdc3>
md: bind<sdd3>
md: bind<sda3>
raid5: device sda3 operational as raid disk 0
raid5: device sdd3 operational as raid disk 3
raid5: device sdc3 operational as raid disk 2
raid5: device sdb3 operational as raid disk 1
raid5: allocated 4288kB for md1
raid5: raid level 5 set md1 active with 4 out of 4 devices, algorithm 0
RAID5 conf printout:
 --- rd:4 wd:4
 disk 0, o:1, dev:sda3
 disk 1, o:1, dev:sdb3
 disk 2, o:1, dev:sdc3
 disk 3, o:1, dev:sdd3
md1: bitmap file is out of date, doing full recovery
md1: bitmap initialisation failed: -5
md1: failed to create bitmap (-5)
mdadm: failed to RUN_ARRAY /dev/md/1: Input/output error.
Does anyone have any suggestions on what else I could try to get this array up so I can get my data off if it? This is my first experience using MD. I have used a lot of hardware raid solutions but never software before. I am beginning to get afraid my data may be gone forever . Any help would be very much appreciated.
Thank you,
David
 
Old 07-01-2009, 12:34 AM   #2
leandean
Member
 
Registered: Oct 2005
Location: Burley, WA
Distribution: Sabayon
Posts: 276

Rep: Reputation: Disabled
Boot into rescue mode. Mount md1 and check menu.lst to make sure you are booting the array and not trying to boot off just one drive. Chances are the machine rebooted after a kernel update and it was overwritten or modified. You didn't say what the hardware is. Some have their own personalities There is also a hot-key combo that will let the machine boot but I can't remember what it is I have it written down at work but I'm not there. Getting old sucks!
 
Old 07-02-2009, 11:33 AM   #3
leandean
Member
 
Registered: Oct 2005
Location: Burley, WA
Distribution: Sabayon
Posts: 276

Rep: Reputation: Disabled
<ctrl> - d is the hotkey to boot the machine. After it boots you need to edit /lib/mkinitrd/boot-md.sh and make it like this::

if [ "$md_dev" ] ; then
/sbin/mdadm $mdconf --auto=md $md_dev || /sbin/mdadm -Ac partitions
$mdarg --auto=md $md_dev
fi
sleep 1
echo change > /sys/block/md$md_minor/uevent
wait_for_events
fi

Then run 'mkinitrd'.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Opensuse 11 software RAID 5 failed: how to recover ? laufandreas Linux - Server 3 06-30-2009 04:51 AM
Recoving Partition eggoz Linux - Software 1 09-05-2006 11:35 AM
Recoving compact flash card phantompgr Linux - Hardware 4 08-03-2005 05:14 PM
Recoving Data from Other Partition PulsarSL *BSD 4 06-26-2005 01:37 PM
recoving my old windows hdd into linux? deee Linux - Newbie 1 11-13-2003 04:46 AM


All times are GMT -5. The time now is 09:50 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration