LinuxQuestions.org
Register a domain and help support LQ
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices

Reply
 
Search this Thread
Old 06-08-2007, 08:35 AM   #1
phyros
LQ Newbie
 
Registered: Jun 2007
Posts: 1

Rep: Reputation: 0
Raid5 Corruption Recovery Help for embedded-linux NAS device


Hey guys, I'd like some help recovering from a failed software Raid-5 setup. The raid-5 setup is on an embedded linux NAS (the Bufallo Terastation Pro, if anyone's familiar with it), so I can't really give all that many details as to the distro, version, setup, etc. All of that is hidden and proprietary. The tech support told me that all I can do is scrap my data, but this is stupid... they're manufacturing a redundant data server; they should know better.

Anyways, a hacked firmware does allow me to telnet into the device as root (and void my warrenty probably, but whatever), so if any pertinent information is discoverable, I can attempt to reverse engineer this thing if you tell me what to do (my linux experience is about a few month's worth... enough to get by but lacking in the deeper understandings of things). Google has been surprisingly unhelpful in finding a comprehensive tutorial on troubleshooting a raid configuration, so I'm hoping someone here can help me.

Anyways, here's what I do know about the setup: it uses four 500gb hard-drives in a RAID-5 configuration, and the raid arrays are mounted as md devices. [edit]The file system is XFS.[/edit] There's two main partitions of interest: /md0 is a system partition and /md1 is the partition of data that I'm trying to recover. I suspect the problem is a bricked superblock, but I'm not quite sure on how to recover from that.

Here's what I've discovered by poking around with mdadm. Looking at the system partition...
Code:
root@HAXD_HELPER:/etc# mdadm --examine /dev/md0
mdadm: No super block found on /dev/md0 (Expected magic a92b4efc, got 00000000)


root@HAXD_HELPER:/etc# mdadm --detail /dev/md0
/dev/md0:
        Version : 00.90.02
  Creation Time : Sat Jan 14 12:32:49 2006
     Raid Level : raid1
     Array Size : 385408 (376.38 MiB 394.66 MB)
    Device Size : 385408 (376.38 MiB 394.66 MB)
   Raid Devices : 4
  Total Devices : 4
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Wed Jun  6 21:26:53 2007
          State : active
Active Devices : 4
Working Devices : 4
Failed Devices : 0
  Spare Devices : 0

           UUID : e87531ac:9fe1f96a:121f55a1:1220867e
         Events : 0.110

    Number   Major   Minor   RaidDevice State
       0       8        1        0      active sync   /dev/sda1
       1       8       33        1      active sync   /dev/sdc1
       2       8       49        2      active sync   /dev/sdd1
       3       8       17        3      active sync   /dev/sdb1
I may not be understanding it correctly (or just not knowing what a good working config looks like), but it seems that all the --details are fine while the --examine says uh-oh. This is also weird since this is supposed to be the system partition (and the system works since, well, I'm in it and running commands), but it supposedly has a bad superblock.

Anyways, there's probably some implementation magic that makes things happen. Thats not too important. I'm really just concerned about my data, which is on /md1.

Code:
root@HAXD_HELPER:/etc# mdadm --examine /dev/md1     
mdadm: No super block found on /dev/md1 (Expected magic a92b4efc, got 7d7d7d7d)


root@HAXD_HELPER:/etc# mdadm --detail /dev/md1
/dev/md1:
        Version : 00.90.02
  Creation Time : Tue Dec 27 16:09:40 2005
     Raid Level : raid5
     Array Size : 1462862592 (1395.09 GiB 1497.97 GB)
    Device Size : 487620864 (465.03 GiB 499.32 GB)
   Raid Devices : 4
  Total Devices : 1
Preferred Minor : 1
    Persistence : Superblock is persistent

    Update Time : Wed Jun  6 22:22:04 2007
          State : active, degraded
Active Devices : 1
Working Devices : 1
Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 64K

           UUID : 37d97fb5:083ede07:8d3e9c16:0f299b85
         Events : 0.300

    Number   Major   Minor   RaidDevice State
       0       8        3        0      active sync   /dev/sda3
       1       8       19        1      active sync   /dev/sdb3
       2       8       35        2      active sync   /dev/sdc3
       3       8       51        3      active sync   /dev/sdd3
What concerns me here are the lines that say there are 4 raid devices, but only 1 total device. The md device doesn't have a good superblock, but when I --examine the individual sd*3 partitions, they do appear to have good superblocks, so this makes me think that all hope is not yet lost...
Code:
root@HAXD_HELPER:/etc# mdadm -E /dev/sd[abcd]3
/dev/sda3:
          Magic : a92b4efc
        Version : 00.90.02
           UUID : 37d97fb5:083ede07:8d3e9c16:0f299b85
  Creation Time : Tue Dec 27 16:09:40 2005
     Raid Level : raid5
   Raid Devices : 4
  Total Devices : 1
Preferred Minor : 1

    Update Time : Wed Jun  6 22:22:04 2007
          State : active
Active Devices : 4
Working Devices : 1
Failed Devices : 0
  Spare Devices : 0
       Checksum : 2cd505c9 - correct
         Events : 0.300

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     0       8        3        0      active sync   /dev/sda3

   0     0       8        3        0      active sync   /dev/sda3
   1     1       8       19        1      active sync   /dev/sdb3
   2     2       8       35        2      active sync   /dev/sdc3
   3     3       8       51        3      active sync   /dev/sdd3
/dev/sdb3:
          Magic : a92b4efc
        Version : 00.90.02
           UUID : 37d97fb5:083ede07:8d3e9c16:0f299b85
  Creation Time : Tue Dec 27 16:09:40 2005
     Raid Level : raid5
   Raid Devices : 4
  Total Devices : 1
Preferred Minor : 1

    Update Time : Wed Jun  6 22:22:04 2007
          State : active
Active Devices : 4
Working Devices : 1
Failed Devices : 0
  Spare Devices : 0
       Checksum : 2cd505db - correct
         Events : 0.300

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     1       8       19        1      active sync   /dev/sdb3

   0     0       8        3        0      active sync   /dev/sda3
   1     1       8       19        1      active sync   /dev/sdb3
   2     2       8       35        2      active sync   /dev/sdc3
   3     3       8       51        3      active sync   /dev/sdd3
/dev/sdc3:
          Magic : a92b4efc
        Version : 00.90.02
           UUID : 37d97fb5:083ede07:8d3e9c16:0f299b85
  Creation Time : Tue Dec 27 16:09:40 2005
     Raid Level : raid5
   Raid Devices : 4
  Total Devices : 1
Preferred Minor : 1

    Update Time : Wed Jun  6 22:22:04 2007
          State : active
Active Devices : 4
Working Devices : 1
Failed Devices : 0
  Spare Devices : 0
       Checksum : 2cd505ed - correct
         Events : 0.300

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     2       8       35        2      active sync   /dev/sdc3

   0     0       8        3        0      active sync   /dev/sda3
   1     1       8       19        1      active sync   /dev/sdb3
   2     2       8       35        2      active sync   /dev/sdc3
   3     3       8       51        3      active sync   /dev/sdd3
/dev/sdd3:
          Magic : a92b4efc
        Version : 00.90.02
           UUID : 37d97fb5:083ede07:8d3e9c16:0f299b85
  Creation Time : Tue Dec 27 16:09:40 2005
     Raid Level : raid5
   Raid Devices : 4
  Total Devices : 1
Preferred Minor : 1

    Update Time : Wed Jun  6 22:22:04 2007
          State : active
Active Devices : 4
Working Devices : 1
Failed Devices : 0
  Spare Devices : 0
       Checksum : 2cd505ff - correct
         Events : 0.300

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     3       8       51        3      active sync   /dev/sdd3

   0     0       8        3        0      active sync   /dev/sda3
   1     1       8       19        1      active sync   /dev/sdb3
   2     2       8       35        2      active sync   /dev/sdc3
   3     3       8       51        3      active sync   /dev/sdd3
Soo... it seems to me like the individual sd*3 devices have the right superblock info, but the superblock info on the md1 device got bust. Is there any way I can tell the md1 device to look at the individual sd*3 devices for its superblock? I'm not sure how to phrase this in terms of proper raid/mdadm terminology (or if I even have the right idea).

Finally, it may help to figure out how these devices are scripted to be setup at boot-time. Again, this is a embedded linux NAS device, so all of this is hidden and would have to be reverse-engineered. I've been told that creating a /initrd directory un-hides all of the boot-time scripts/ramdisk (and indeed this is true for my device), but I have no idea what to look for in here.

Any help from a raid guru would be infinitely helpful.

Last edited by phyros; 06-08-2007 at 02:35 PM. Reason: making title more specific
 
Old 06-09-2007, 09:35 PM   #2
rtspitz
Member
 
Registered: Jan 2005
Location: germany
Distribution: suse, opensuse, debian, others for testing
Posts: 307

Rep: Reputation: 32
if you run mdadm --examine on an md device it will complain about a missing superblock - this is normal and happens on my raid systems as well - which all work fine.

the 4 partitions sd[abcd]3 seem to contain all the info of the raid they should form.

so you could try this (no guarantee for data recovery though):

- unmount /dev/md1 and stop the md1 array with:

Code:
mdadm --stop /dev/md1
- reassemble the array:

Code:
mdadm --assemble /dev/md1 /dev/sda3 /dev/sdb3 /dev/sdc3 /dev/sdc3
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
LXer: Device Profile: Hammer MyShare NAS device LXer Syndicated Linux News 0 04-26-2007 01:01 PM
RAID5 Array Recovery after OS upgrade hazmatt20 Linux - Server 25 04-19-2007 08:41 AM
Linux embedded NAS ch4os Linux - Hardware 1 03-22-2005 11:17 AM
Need help with embedded linux device nabeelakhtar Linux - Laptop and Netbook 3 12-25-2004 12:12 AM


All times are GMT -5. The time now is 09:19 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration