Linux - ServerThis forum is for the discussion of Linux Software used in a server related context.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
Looks like ended up in quite a situation in the last few days. Here is my story in a nutshell, I hope what I have done can be undone.
I have a file server where I stored a lot of my business information as well as all my personal information especially pictures of my kids from the last 6 years or so. I am sure you can already see where this is going.
The file server is a virtual openfiler machine that has the device passed in to it as /dev/sda upon which I have created a physical volume with 2 logical volume (business / media).
The vm host machine is running suse 11 with xen server 3. I created a raid 5 array using 4 x 500GB Seagate SATA hard drives. (/dev/sd[bcde]1) .. in the last little while I had to occurences when the array would become degrated because different disks would "fail" on it... was not able to isolate the issue but have been able to re-assemble the array each time, rebuild and keep going. This has not been the case this last time. (btw this machine creates the md0 device and passes it to openfiler as /dev/sda).
The predicament I find myself in now is as follows:
This morning I had 2 drives drop out of the array and ofcourse my array became inaccessible. I should also note at this point that I have checked all the connections and smart information all drives are in good physical shape. The OS can see all drives just fine... I can do an fdisk -l on each drive and get what is expected. With much research and reading I have tried to re-assemble the array many times always failing regardless of wether I used the --force --run attributes with mdadm it kept complaing about /dev/sdc1 being out of sync or something and just gave up. After doing some research I found that a lot of people had success in a similar situations by rebuilding the array with --assume-clean argument, so I felt this might be my only other option left so I tried that. Each time I tried it the process was successful and I got my array up and running but as you might have guessed when firing up the openfiler server my logical volumes were deemed not found. Doing an fdisk -l on /dev/md0 so far has always said "No valid partition table has been found".
Here is what I have done recently that seem to be making my stomach all tense...
I tried to reassemble the array with the partitions listed in different orders (I started with the order I thought would be most logical after looking at my notes which I will post). Every time the array would assemble successfully but no access to data, at least pvscan did not find any traces of any volume groups on the md0 device. I then --zero-superblock on /dev/sd[bcd]1 and tried to recreate the array again.. once again to no success. The part that is confusing though it seem to be correct about how much of the array is "used" which gives me a bit of hope.
One important thing to is I did not realize at first that somehow my original md0 superblock was created with metadata version 1.0 so by default mdadm created v 0.9 the first few times... I also had to mess around with different chunk sizes until I got the new superblock to correspond to the one I noted prior to this whole disaster, I really hope that did not affect my data, as far as I know I did not write anything to the disk, I am hoping that all my manipulation so far has been with the superblocks only.
One question I have right now is if it is possible to recreate my original superblocks on each drive and try to reassemble again using the same uuid of the array as well as each individual device.
I am running out of ideas and sure could use any of your guys' help... should also note that I had an external 1tb hard drive where I was backing up my most important information but that got stolen less than a week ago when he had a break in.
Here is some technical information:
sdb1 -> Device 0 in 4 device array
sdc1 -> Device 1 in 4 device array
sdd1 -> Device 2 in 4 device array
sde1 -> Device 3 in 4 device array
mdadm --create --metadata=1.0 --raid-devices=4 --chunk=512 --level=5 --layout=left-asymmetric --assume-clean /dev/md0 /dev/sd[bcd]1 missing
(order of devices is a guess in this one)
mdadm --zero-superblock /dev/sd[bcd]1
mdadm --create --metadata=1.0 --raid-devices=4 --chunk=128 --level=5 --layout=left-asymmetric --assume-clean /dev/md0 /dev/sdd1 /dev/sdb1 /dev/sdc1 missing
(this I feel is probably very very close to my initial superblock setup)
I also have a backup of the metadata of the creation of my physical volume so if I can manage to recover this raid5 array to a usable state worse comes to worse I might be able to recover the lvm information unto the device from that...
ANY help and suggestions would be appreciated much more than you realize... I have not felt this desperate for a very long time.
After spending too many nights to 4am working to resolve this, it was really starting to get to me so I have decided to take a break until I get a chance to go back at it.
I have bough a couple more hard drives 1 2TB drive adn 1 1TB drive and then dd_rescued all the data that I could into images on those drives in the drives original state as the system went down. I also have 4 replacement 500GB drives that I have been restoring those images unto and trying different scenarios. I have only once been able to recreate the raid5 array and actually have the PVs visible. When trying to access though XFS complained that it could not find superblock or something along those lines so when running the XFS repair tools to make it "healthy" again I was able to see some of the directory structures on the volume as before but not the majority of it. Majority of the files showing up were not valid files either (ie jpg files that you could not view, etc).
So out of the 4 drives I should have the following currently:
sdb1 -> failed physically, I have already tried a head swap on it and got the same results (can not get more than about 100MBs of data off it until it quits.) I suspect there might be spindle damage to it as it seems to get a bit of a wobble when it runs with the cover off. I do not have any tools to attempt and perform a dual platter move to one of my other drives so until I do I am leaving this oen alone.
sdc1 -> imaged to a separate drive in its original state.
sdd1 -> imaged to a separate drive in its original state.
sde1 -> imaged on a separate drive in original state. In its original state however this drive got kicked out of the array first and its events are less than the other 3... I am hoping that somehow using this drive instead of sdb1 I will be able to recreate this array again and at least get some of my data back.
Not giving up yet, but it is looking really grim for me right now
If anyone has any other ideas, I'd love to hear them.
after doing XFS repair some content came back, some content showed up as if it was there but was not..
a lot of content was completely gone.
- next thing to try will be to dd the drives again, and try to assemble the array back and have the XFS header still valid.
- I thin key drives will be sdb1, sdc1, sdd1 maybe try without the sde1 or sdc1
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.
