LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices

Reply
 
LinkBack Search this Thread
Old 06-09-2009, 02:52 PM   #1
sxa
Senior Member
 
Registered: Aug 2003
Location: Austin, TX
Distribution: Mac OS 10.7 / CentOS 6(servers) / xubuntu 13.04
Posts: 1,186

Rep: Reputation: 49
Trying to recover/mount RAID array


To make a long story short, I am responsible for a media server that has two RAID array's (/dev/md0 and /dev/md2). I still have a ton to learn about software RAID and LVM, etc. We had a drive fail in /dev/md0, okay not a big deal I replaced the bad drive, reassembled the RAID array (which took most of last night to complete) and this morning I came in and was able to mount /dev/md0 with no data loss.

Little did I know, something became corrupt with /dev/md2. Now /dev/md0 was set-up by my co-worker two years ago, I set-up /dev/md2 about six months ago. Always having done RAID using the hardware controllers built into servers from Dell and HP, this is the first time I have had to deal with software RAID. I followed the instructions on http://www.gagme.com/greg/linux/raid-lvm.php which I felt was the best idea as LVM would give me the ability to grow the RAID array in the future (I just now learned that mdadm can do the same). There was no issues with this until today. After rebooting the machine after it repaired /dev/md0 it was unable to mount /dev/lmv-raid/lvm0 (said the file or directory couldn't be found). It almost looks like the logical volume is gone. I do not understand what would have caused this. No drives in the second array were bad, I didn't touch the second array. Nothing should have gone bad with the array. But it looks like something isn't right.

Now /dev/md2 is running and is in a clean state, but I cannot mount it. I am afraid that data is gone. The really terrible thing is that there is no explanation in my mind as to what happened. I am assuming that I can't mount /dev/md2 because the RAID array is expecting to be controlled by LVM. Maybe I do not understand the system well enough, but if I have a clean array shouldn't the data still be in the array? Any idea how I can go about recovering the data? The last thing I want to do is have to go back to my boss and tell him that in the process of fixing one array another array magically broke and all your data is gone. I have looked at multiple sites, but since I don't know what actually broke with it all I don't know if I have really lost data or if there is something else that I am missing.

Any advice would be greatly appreciated, I have been struggling with this all morning long and am at a total loss.

Here is some more information:

Code:
[root@viamao ~]# mdadm --detail /dev/md2
/dev/md2:
        Version : 00.90.01
  Creation Time : Tue Jun  9 11:32:06 2009
     Raid Level : raid5
     Array Size : 4395415488 (4191.79 GiB 4500.91 GB)
    Device Size : 1465138496 (1397.26 GiB 1500.30 GB)
   Raid Devices : 4
  Total Devices : 4
Preferred Minor : 2
    Persistence : Superblock is persistent

    Update Time : Tue Jun  9 15:54:39 2009
          State : clean
 Active Devices : 4
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 64K

    Number   Major   Minor   RaidDevice State
       0       8      128        0      active sync   /dev/sdi
       1       8      144        1      active sync   /dev/sdj
       2       8      160        2      active sync   /dev/sdk
       3       8      176        3      active sync   /dev/sdl
           UUID : 88bbe155:b6d852b1:3ad8a77d:5eba3ee2
         Events : 0.36
Code:
[root@viamao archive2]# mount -t ext3 /dev/md2 /share/archive2
mount: wrong fs type, bad option, bad superblock on /dev/md2,
       or too many mounted file systems
Code:
[root@viamao archive2]# mount
/dev/hda3 on / type ext3 (rw)
none on /proc type proc (rw)
none on /sys type sysfs (rw)
none on /dev/pts type devpts (rw,gid=5,mode=620)
usbfs on /proc/bus/usb type usbfs (rw)
/dev/hda1 on /boot type ext3 (rw)
none on /dev/shm type tmpfs (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
/dev/md0 on /share/archive1 type ext3 (rw)
Code:
[root@viamao share]# cat /proc/mdstat
Personalities : [raid5] 
md2 : active raid5 sdl[3] sdk[2] sdj[1] sdi[0]
      4395415488 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
      
md0 : active raid5 sda[0] sdh[7] sdg[6] sdf[5] sde[4] sdd[3] sdc[2] sdb[1]
      5128021248 blocks level 5, 64k chunk, algorithm 2 [8/8] [UUUUUUUU]
      
unused devices: <none>

Anything else I can show you that might be able to help? I have a active RAID array that I can't mount, thinking this has something to do with the fact that this used to be on lvm, but now when I try to mount /dev/lvm-raid/lvm0 it tells me the special device cannot be found. lvdisplay doesn't give me anything pvdisplay only shows me the drive that my OS is on. vgdisplay also yields nothing.

Thoughts? I am seriously at a loss here.

Last edited by sxa; 06-09-2009 at 03:02 PM. Reason: added /proc/mdstat
 
Old 06-09-2009, 04:13 PM   #2
eco
Member
 
Registered: May 2006
Location: BE
Distribution: Debian/Gentoo
Posts: 412

Rep: Reputation: 48
Could you send us the steps you took to repair the raid so that we can have a clearer picture of what you did.

The results of pvscan, vgscan, lvscan, ... would be a plus.

If you have time and space, you might do a disk copy (dd) of your "failed" RAID and run your tests from the image rather than from the live RAID.

Do you have any backups?
 
Old 06-09-2009, 04:30 PM   #3
sxa
Senior Member
 
Registered: Aug 2003
Location: Austin, TX
Distribution: Mac OS 10.7 / CentOS 6(servers) / xubuntu 13.04
Posts: 1,186

Original Poster
Rep: Reputation: 49
I didn't do anything to /dev/md2 when recovering /dev/md0. I booted the machine up after putting the new drive in, /dev/md0 could not start (at this point /dev/lvm-raid/lvm0 could not be found). For getting /dev/md0 started again I ran

Code:
mdadm --assemble --run --force -v /dev/md0 /dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh
It took several hours to reassemble, but when it did I was able to mount /dev/md0. That still didn't explain why /dev/lvm-raid/lvm0 could not be found. Now /dev/md0 has nothing to do with lvm.

Code:
[root@viamao archive]# lvscan
  Incorrect metadata area header checksum
Code:
[root@viamao archive]# pvscan
  Incorrect metadata area header checksum
  Incorrect metadata area header checksum
  Incorrect metadata area header checksum
  PV /dev/hda2         lvm2 [37.17 GB]
  Total: 1 [37.17 GB] / in use: 0 [0   ] / in no VG: 1 [37.17 GB]
Code:
[root@viamao archive]# vgscan
  Reading all physical volumes.  This may take a while...
  Incorrect metadata area header checksum
I have been told the incorrect metadata area header just means that it is trying to talk to non lvm volumes. I ran pvscan -vvv and confirmed it is trying to talk to /dev/md2 but says:

Code:
pened /dev/md2 RO
      /dev/md2: size is 8790830976 sectors
        Closed /dev/md2
      /dev/md2: size is 8790830976 sectors
        Opened /dev/md2 RO O_DIRECT
        /dev/md2: block size is 4096 bytes
        Closed /dev/md2
        Using /dev/md2
        Opened /dev/md2 RO O_DIRECT
        /dev/md2: block size is 4096 bytes
      /dev/md2: No label detected
        Closed /dev/md2
As for backups, some of the stuff they have is backed up, I am not sure what all they might have lost, and it looks to me like they have lost data and I am pretty much screwed.
 
Old 06-09-2009, 04:54 PM   #4
ramram29
Member
 
Registered: Jul 2003
Location: Miami, Florida, USA
Distribution: Debian
Posts: 848
Blog Entries: 1

Rep: Reputation: 47
You want to run "foremost" and see if you can recover any raw data - it's a mess but it does recover files with random names.

I would suggest also to clone that bad disk and try to play around with the copy, leave the original around.

Try the following commands:

pvcreate /dev/md2
vgscan
vgchange -ay
mount /dev/lvm-raid/lvm0 /mnt
 
Old 06-09-2009, 04:59 PM   #5
sxa
Senior Member
 
Registered: Aug 2003
Location: Austin, TX
Distribution: Mac OS 10.7 / CentOS 6(servers) / xubuntu 13.04
Posts: 1,186

Original Poster
Rep: Reputation: 49
Quote:
Originally Posted by ramram29 View Post
You want to run "foremost" and see if you can recover any raw data - it's a mess but it does recover files with random names.

I would suggest also to clone that bad disk and try to play around with the copy, leave the original around.

Try the following commands:

pvcreate /dev/md2
vgscan
vgchange -ay
mount /dev/lvm-raid/lvm0 /mnt
Code:
[root@viamao archive]# pvcreate /dev/md2
  Physical volume "/dev/md2" successfully created
[root@viamao archive]# vgscan
  Reading all physical volumes.  This may take a while...
  Incorrect metadata area header checksum
[root@viamao archive]# vgchange -ay
  Incorrect metadata area header checksum
[root@viamao archive]# mount /dev/lvm-raid/lvm0 /mnt
mount: special device /dev/lvm-raid/lvm0 does not exist
[root@viamao archive]#
 
Old 06-10-2009, 09:32 AM   #6
ramram29
Member
 
Registered: Jul 2003
Location: Miami, Florida, USA
Distribution: Debian
Posts: 848
Blog Entries: 1

Rep: Reputation: 47
fubar
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Recover data from RAID 0 (fake RAID) from Win XP KiAnKo Linux - Hardware 4 04-23-2009 12:28 PM
Raid array resized - Cannot mount LVM - Fedora Core 5 rimfire Linux - Software 0 11-29-2006 12:08 AM
mount windows raid array psirek Linux - Desktop 0 10-04-2006 06:28 PM
Unable to Mount NTFS RAID O Array lpboarder23 Suse/Novell 1 03-14-2006 04:29 PM
Mount a raid array with SGID bit set, howto? anand_kt Linux - General 4 04-01-2005 01:30 AM


All times are GMT -5. The time now is 10:37 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration