Linux - ServerThis forum is for the discussion of Linux Software used in a server related context.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
Hello, one of our servers (a XENServer) had a hardware problem in a RAID5, that RAID5 got inconsistent and XEN never worked again making all the Virtual Machines not accessible through the xenapi.
All the information in the virtual machines is irrelevant except for one "small" mysql database. I don't know the actual size of the mysql database but I doubt it is bigger than 10Mb, its a mysql database of a mediawiki that contains some articles we would like to recover.
The disk structure is as follows: first partition is the xen operating system, and the second partition is an LVM (62GB). Inside that LVM resides several VHDs that are the virtual disks of our machines. Considering that I couldnt access them I did a dd of the entire disk to a file, and started working on it on my pc.
I can list the partitions:
Code:
fdisk -l ./dd_disk.img
but I don't seem to be able to use the LVM, and even if I did, all it would get me where some VHD's that I don't think I can open anyway (I tested dd on a healthy xenserver and extracted one VHD and I couldn't open it (tried it with winimage)).
What I am trying to do is the following:
I picked a word that I knew was written in the wiki articles, then I used
Code:
od -S 5 ./dd_disk.img | grep MYWORD
and I can see the phrases in perfect and clear English, so... my data is there!!, phrases that belong to my mediawiki database, however they are "lost" inside of my full disk dump.
I started to read some information about ext filesystems (the file system where the mediawiki was installed was an ext4) and found that sometimes you can recover a filesystem that doesnt even has a valid superblock by making a mke2fs -S (http://animeshdas.wordpress.com/2009...ad-superblock/) that would write only the superblock and leave the data, then an fsck might allow me to access the files. I wonder if I can do this: I find the "probable beggining" of the file system inside my disk dump, I dd it to another file, I mke2fs -S to it, and then fseek it?
My problem is that I can't seem to find any info on the ext4 filesystem specification that says how to detect the beggining of the filesystem.
Doest anyone know if there is some kind of static string inside and ext4 file system that marks its beggining? Or some kind of marker I can use to know where an ext4 file system begins?
If there are users here with experience on this kind of recoveries..am I going in a good direction? I really only need to recover the files that make the mysql database (/var/lib/mysql) all the rest is irrelevant...
Hmmm - haven't use Xen in (quite) a while, but will be again soon, so this caught my eye. this looks interesting re mounting the vhd elsewhere. Might save you a lot of angst with other methods.
As for the "mkfs -S...", I tested this a while back, and it worked fine. However, it was a controlled test - I built a new filesystem (ext3 from memory), created some files, formatted the partition as NTFS, then "mkfs -S ...". fsck then I was able to find all the files. But as I said, controlled - no fragmentation, no long-term activity on the f/s. There is a warning in the manpage I presume you saw.
I used to have the doco for ext3 structure (ext4 should be similar), but that machine died - I'll check my backups later.
I hope you weren't relying on raid (only) and not taking backups ... ???. A simple restore would quickly fix your problem. Even if you can get the vhd mounted, if the raid died with inconsistencies, how can you trust the (on-disk) image you have ?.
Hello, I have tried to dd the VHD's and mount them but had no luck... I just can't open them, first I thought it was because they might be corrupted, but then I tried to export a VHD from a healthy XEN and couldn't mount it anyhow... that gave me some hope!
Even if the RAID is corrupted the recovery is all about text documents that are "easily" verifiable by hand... not a problem, this is just to save time re-making all of them.
About backups?? well...you are absolutely right... RAID is not designed and is not enough to offer the protection a production server needs...however this was a test machine... several VM's that where not important...but when this problem occured...we found that a wiki that had been there for test purposes never left and kept being used there
If I can't recover this we will need about one month work to re-do the documents in the wiki, it isn't something mission critical, but I would really like not to waste that month and recover all the files
I'm going to try the mke2fs -S and retry the mounting of the VHD's like you said... if you could give me some hint on how to detect the starting of ext I will be very grateful! I have created a empty file (10MB) inside I created two partitions and formatted them with ext3 so that I could try to figure out some marker on the beginning but the pattern escaped me...
If the partition table is correct then it would be easiest to access the LVM by loop-mounting the dd image. Having done that you should be able to use pvscan and vgscan to check PV and VG details. Ensure (configure?) they get added and not replace any existing VGs on your workstation. Then 'vgchange -ay' should load the LVM (for non-LVM partitions the easiest way is to use 'kpartx -va') and 'dmsetup status' show the result. Now you should be able to find the VHD inside the LVM.
Quote:
Originally Posted by circuit_PT
and even if I did, all it would get me where some VHD's that I don't think I can open anyway
I don't think (IOW: best check that yourself) Photorec or testdisk read VHD format but they do read 'dd' raw images. VirtualBox comes with a tool that can convert Virtual HD images to raw dd.
Quote:
Originally Posted by circuit_PT
an fsck might allow me to access the files.
If the file systems contents are not corrupted then running 'fsck' makes no sense. If file systems contents are corrupted then running 'fsck' will re-allocate space in that partition and write recovered blocks to that file systems /lost+found directory, ensuring unallocated space which by then has been overwritten will no longer be recoverable. In short I suggest you don't.
Quote:
Originally Posted by circuit_PT
My problem is that I can't seem to find any info on the ext4 filesystem specification that says how to detect the beggining of the filesystem. Doest anyone know if there is some kind of static string inside and ext4 file system that marks its beggining? Or some kind of marker I can use to know where an ext4 file system begins?
Even though it is about ext3 a nice explanation of extn headers is here: http://carlo17.home.xs4all.nl/howto/undelete_ext3.html. Ext4 documentation resides at https://ext4.wiki.kernel.org/. For efficiency sake I suggest you first use existing tools like testdisk or Autopsy to access the file system. You shouldn't use scalpel, foremost or Photorec and there's two reasons for that: 0) you stated the file system inside the converted VHD is not corrupted, meaning contents should be readily accessible and 1) file carvers recover files by their header / footer which works well for files with a distinct signature like MySQL .frm and .MYI files but not for files that contain data like the .MYD file (it's even worse, see: http://stam.blogs.com/8bits/2009/07/...d-headers.html).
Quote:
Originally Posted by circuit_PT
(..) one "small" mysql database. I don't know the actual size of the mysql database but I doubt it is bigger than 10Mb, (..) we would like to recover. (..) I picked a word (..) and I can see the phrases in perfect and clear English, so... my data is there!!, phrases that belong to my mediawiki database, however they are "lost" inside of my full disk dump.
Once you got the VHD converted to a raw 'dd' image you could "walk" the inode tree pointing testdisk to it. If you want a more elaborate view of things you could load the Helix 2008R1(2.0) ISO (MD5 hash 93a285bfa8ab93d664d508e5b12446d3) burned as CDROM (or boot it from inside your favorite virtualization tool) and "walk" the file system with Autopsy. As the file system inside the converted VHD is not corrupted you should have no problem exporting the coveted .MYD file using testdisk or Autopsy.
Hello Thank you for your reply, I'm going to try and follow you tips.
For example when i fdisk -l this is what I get:
Code:
# fdisk -l full_disk.img
Disk full_disk.img: 72.8 GB, 72816967680 bytes
255 heads, 32 sectors/track, 17429 cylinders, total 142220640 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
Device Boot Start End Blocks Id System
full_disk.img1 * 63 8388670 4194304 83 Linux
full_disk.img2 8388671 16777278 4194304 83 Linux
full_disk.img3 16777279 142220639 62721680+ 8e Linux LVM
#
so then I thought I could do losetup:
Code:
# losetup ./full_disk.img
loop: can't get info on device ./full_disk.img: Inappropriate ioctl for device
but that error happens do you know what it is? If I go to my "defective" xenserver I can do the pvscan, vgscan , etc... and I get a file in /dev/mapper/VG_XexStorage--<LOTS OF NUMBERS>-VHD-<MORE NUMBERS> (the numbers are the UUID of the VDI inside xen) there I can do for example:
Code:
hexdump -C /dev/mapper/VG_XexStorage--<LOTS OF NUMBERS>-VHD-<MORE NUMBERS> | head
And I see the beginning of the VHD (the conectix string).
Code:
fdisk -l /dev/mapper/VG_XexStorage--<LOTS OF NUMBERS>-VHD-<MORE NUMBERS>
The fdisk says it cant find a partition table (maybe this is normal because this is a VHD???)
I'm going to dd now only this VHD...maybe then I can convert it with the tools you told me...
Code:
dd if=/dev/mapper/VG_XexStorage--<LOTS OF NUMBERS>-VHD-<MORE NUMBERS> bs=1024000 of=/mnt/tmp/vhd.img
now I'm going to do a conversion to raw:
Code:
qemu-img convert -f vpc ./vhd.img -O raw vhd.raw
everything works ok......
Code:
# ls -lh
total 27G
-rw-r--r-- 1 root root 26G Feb 13 10:02 vhd.img
-rw-r--r-- 1 root root 25G Feb 13 10:38 vhd.raw
#
but....listing the partitions now should work???? lets see:
Code:
# fdisk -l ./vhd.raw
Disk ./vhd.raw: 26.8 GB, 26843185152 bytes
255 heads, 63 sectors/track, 3263 cylinders, total 52428096 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
Disk ./vhd.raw doesn't contain a valid partition table
#
even though I think it woldn't make sense I tried a mount -o loop ./vhd.raw /mnt/a it didnt work...no superblock...
What I was trying to do in my last post with fseek is something I just attempted now.... I copied vhd.raw to vhd2.raw and did a mk2efs -S -b 1024 -f 1024 -I 128 ./vhd2.raw I am aware that this RAW device should represent a disk and no a partition, even though.. I thought that if I gave it any superblock, then fsck could restore the files somehow.... but it didn't work either.
I'm starting to loose hope, and what is more frustrating is that when I do an od -S4 I see tons of data from my articles..... the data is there
# losetup ./full_disk.img
loop: can't get info on device ./full_disk.img: Inappropriate ioctl for device
but that error happens do you know what it is?
Yes, you didn't specify a loop device: 'losetup /dev/loopn /path/to/full_disk.img' (or 'losetup -o $[512*16777279] /dev/loopn /path/to/full_disk.img' for only the LVM partition). Alternatively you should be able to use 'kpartx -va /path/to/full_disk.img'.
Quote:
Originally Posted by circuit_PT
now I'm going to do a conversion to raw:
Code:
qemu-img convert -f vpc ./vhd.img -O raw vhd.raw
everything works ok......
AFAIK you shouldn't have to use "-f vpc" as qemu-img will recognise the VHD format, but be aware some versions of qemu-img didn't convert the VirtualPC format well resulting in a seemingly OK image w/o access and there's a difference between single volume and dynamic virtual disks. (Also see 'VBoxManage.exe /path/to/vhd.img /path/to/vhd.raw --format RAW'). But it seems VirtualBox (how about VMware?) can load VHD files natively.
Hello, first of all let me thank you for all the help you are giving! Without it I would be completely lost.
About the latest commands unfortunately I still haven't been able to recover my data....
The losetup wasn't working even with the losetup /dev/loop0 /path/to/full_disk.img however then I tried the second command you gave me:
Code:
# losetup -o $[512*16777279] /dev/loop1 /mnt/sdb1/full_disk.img
# pvscan
PV /dev/loop1 VG VG_XenStorage-b9b79283-9957-6708-067d-a2cad1c4053e lvm2 [59.80 GiB / 5.60 GiB free]
Total: 1 [59.80 GiB] / in use: 1 [59.80 GiB] / in no VG: 0 [0 ]
# vgscan
Reading all physical volumes. This may take a while...
Found volume group "VG_XenStorage-b9b79283-9957-6708-067d-a2cad1c4053e" using metadata type lvm2
# vgs
VG #PV #LV #SN Attr VSize VFree
VG_XenStorage-b9b79283-9957-6708-067d-a2cad1c4053e 1 13 0 wz--n- 59.80g 5.60g
# vgchange -ay VG_XenStorage-b9b79283-9957-6708-067d-a2cad1c4053e
13 logical volume(s) in volume group "VG_XenStorage-b9b79283-9957-6708-067d-a2cad1c4053e" now active
# ls -l /dev/VG_XenStorage-b9b79283-9957-6708-067d-a2cad1c4053e/
MGT VHD-576f19cd-6a73-4fea-9bf8-4471a3908a4c VHD-b8b17e22-5008-4313-8ad2-64d2c5398a70
VHD-0cc0bebb-e0e7-4f45-8cd6-88be82f578b6 VHD-6a8ab0f1-9606-414a-8b98-425ac5d04bc9 VHD-c67d743d-3d84-4c6e-a602-1fd7233ca789
VHD-1aab97a0-f455-4c72-81e7-c7ee1a1ce984 VHD-78972128-cfe3-4961-a81d-093e3f90dad9 VHD-ea737393-03dc-4dd1-8511-69df7a187d86
VHD-208ba32d-e4a1-4ef9-bd5d-cb231bba9cc5 VHD-94749908-1a0a-44c5-b921-d021535c2baa
VHD-3dcf8b24-9eb7-464b-acdd-7f7a10ae2e91 VHD-b2037158-69a1-4417-af36-730702e0188f
So...It seems the LVM is up, and the VHD's appear...the one that interests me is VHD-1aab97a0-f455-4c72-81e7-c7ee1a1ce984 (/dev/mapper/VG_XenStorage--b9b79283--9957--6708--067d--a2cad1c4053e-VHD--1aab97a0--f455--4c72--81e7--c7ee1a1ce984)
Now I tried VBoxManage.... but:
Code:
# VBoxManage clonehd /dev/mapper/VG_XenStorage--b9b79283--9957--6708--067d--a2cad1c4053e-VHD--1aab97a0--f455--4c72--81e7--c7ee1a1ce984 /mnt/sdb1/lq/vobxconvert.raw --format RAW
VBoxManage: error: The medium '/dev/mapper/VG_XenStorage--b9b79283--9957--6708--067d--a2cad1c4053e-VHD--1aab97a0--f455--4c72--81e7--c7ee1a1ce984' can't be used as the requested device type
VBoxManage: error: Details: code NS_ERROR_FAILURE (0x80004005), component Medium, interface IMedium, callee nsISupports
Context: "OpenMedium(Bstr(pszFilenameOrUuid).raw(), enmDevType, AccessMode_ReadWrite, pMedium.asOutParam())" at line 209 of file VBoxManageDisk.cpp
Now what can I do (yes,..I'm tired...running out of ideas and desperate for more help )? I tried qemu again...no luck.... I tried VBoxManage
Code:
#VBoxManage clonehd ./vhd.raw --format RAW
VBoxManage: error: Could not get the storage format of the medium '/mnt/sdb1/lq/./vhd.raw' (VERR_NOT_SUPPORTED)
VBoxManage: error: Details: code VBOX_E_IPRT_ERROR (0x80bb0005), component Medium, interface IMedium, callee nsISupports
Context: "OpenMedium(Bstr(pszFilenameOrUuid).raw(), enmDevType, AccessMode_ReadWrite, pMedium.asOutParam())" at line 209 of file VBoxManageDisk.cpp
#
I'm not shure if I should be using the CLONENHD but... well... I googled it and I think that is the right command... In desperation I tried (I don't think I wan't to convert from RAW but... well..:
Code:
# VBoxManage convertfromraw /mnt/sdb1/lq/vhd.raw new.raw --format RAW
Converting from raw image file="/mnt/sdb1/lq/vhd.raw" to file="new.raw"...
Creating dynamic image with size 26843185152 bytes (25600MB)...
# file novo.raw
novo.raw: x86 boot sector; partition 1: ID=0x83, active, starthead 32, startsector 2048, 1024000 sectors, code offset 0x31
[root@war lq]# mount -o loop ./novo.raw mount_point
mount: you must specify the filesystem type
[root@war lq]# mount -o loop ./novo.raw mount_point -t ext4
mount: wrong fs type, bad option, bad superblock on /dev/loop2,
missing codepage or helper program, or other error
In some cases useful info is found in syslog - try
dmesg | tail or so
completely running out of ideas.... Is there anything else I can do?
# disktype /dev/mapper/VG_XenStorage--b9b79283--9957--6708--067d--a2cad1c4053e-VHD--1aab97a0--f455--4c72--81e7--c7ee1a1ce984
--- /dev/mapper/VG_XenStorage--b9b79283--9957--6708--067d--a2cad1c4053e-VHD--1aab97a0--f455--4c72--81e7--c7ee1a1ce984
Block device, size 25.05 GiB (26902265856 bytes)
Connectix Virtual PC hard disk image, differential
Disk size 25 GiB (26843545600 bytes)
#
I believe each VHD is the virtual disk that I created in the xenserver for each virtual machine. The VG_XenStorage--b9b79283--9957--6708--067d--a2cad1c4053e-VHD--1aab97a0--f455--4c72--81e7--c7ee1a1ce984 is the disk that belongs to the wiki server I am trying to recover... what is inside I would say a Linux Swap partition and an ext4....but I just remmembered.... inside the VHD there is another LVM.... well... let me explain... we have several templates for our virtual machines, and I just connected to another virutal machine that is in another pool but that uses the same template, and this is what I have:
Code:
# fdisk -l
Disk /dev/xvda: 26.8 GB, 26843545600 bytes
255 heads, 63 sectors/track, 3263 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0005897f
Device Boot Start End Blocks Id System
/dev/xvda1 * 1 64 512000 83 Linux
Partition 1 does not end on cylinder boundary.
/dev/xvda2 64 3264 25701376 8e Linux LVM
Disk /dev/dm-0: 24.2 GB, 24201134080 bytes
255 heads, 63 sectors/track, 2942 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
Disk /dev/dm-0 doesn't contain a valid partition table
Disk /dev/dm-1: 2113 MB, 2113929216 bytes
255 heads, 63 sectors/track, 257 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
Disk /dev/dm-1 doesn't contain a valid partition table
# mount
/dev/mapper/VolGroup-lv_root on / type ext4 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
/dev/xvda1 on /boot type ext4 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
none on /proc/xen type xenfs (rw)
This machine that I just "fdisked -l and mount" is also in a XEN, has also a VHD in the same format, inside an LVM in a XENSERVER...exactly the same... so what is inside the VHD:
- An ext (my boot partition)
- Another LVM that has my swapp and my ext;
I had forgotten that there was this additional LVM layer inside the VHD.
So...can I losetup the LVM that is inside the VHD? I believe I have to convert the VHD to RAW and losetup it right? But the thing is that I already converted it using qemu-img before and then I fdisk -l the raw image and I could see any partition, I should see some partitions right?
Anyhow I repeated the steps:
Code:
# qemu-img convert /dev/mapper/VG_XenStorage--b9b79283--9957--6708--067d--a2cad1c4053e-VHD--1aab97a0--f455--4c72--81e7--c7ee1a1ce984 -O raw vhd-convert-to-raw.raw
# fdisk -l ./vhd-convert-to-raw.raw
Disk ./vhd-convert-to-raw.raw: 26.8 GB, 26843185152 bytes
255 heads, 63 sectors/track, 3263 cylinders, total 52428096 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
Disk ./vhd-convert-to-raw.raw doesn't contain a valid partition table
# disktype ./vhd-convert-to-raw.raw
--- ./vhd-convert-to-raw.raw
Regular file, size 25.00 GiB (26843185152 bytes)
First 1.001 MiB (1049600 bytes) are blank
--- /dev/mapper/VG_XenStorage--b9b79283--9957--6708--067d--a2cad1c4053e-VHD--1aab97a0--f455--4c72--81e7--c7ee1a1ce984
Block device, size 25.05 GiB (26902265856 bytes)
Connectix Virtual PC hard disk image, differential
(..)I just remmembered.... inside the VHD there is another LVM.
There's four different VPC formats and I think that basic tools only cope with basic disk layout (as in "fixed"). So I think you should skip the extraction at that stage but copy the VHD out as VHD. Configure a virtual machine inside your favorite virtualization tool (VirtualBox or VMware), make it boot from a Live CD (KNOPPIX ISO?) and use the copied VHD as hard disk. After Linux is loaded you can access the LVM, mount the partition and then copy the .frm, MYI and .MYD out.
So I dd the file like I described in the last post, it got me a disk.vhd...then I already had VirtualBox installed so I google in how to add a vhd to my virtualbox, I found this: http://security.goldsby.com/2010/12/...lbox-windows7/
so I did:
Code:
# qemu-img.exe convert -f vpc disk.vhd -O raw disk.bin
# qemu-img convert -f vpc ./disk.vhd -O raw disk.img
# VBoxManage convertdd disk.img disk.vdi
Converting from raw image file="disk.img" to file="disk.vdi"...
Creating dynamic image with size 26843185152 bytes (25600MB)...
# VBoxManage modifyvdi disk.vdi compact
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
# file disk.vdi
disk.vdi: data
# disktype disk.vdi
--- disk.vdi
Regular file, size 2.024 GiB (2172755968 bytes)
Nothing in this process gave any error, I just think it is "strange" that the resulting file only has 2GB but the server is a basic Linux install nothing more...so...maybe no problem there..
The I created a new machine in virtual box, and gave it a Live CD of centos I have, it boots without a problem in the CD... however when I fdisk -l it lists a 26.8GB disk but without a partition table... when I do a pvscan it says "No matching volumes found", pvdisplay show no volumes...
so what you recommend is that I dd this to a file righ?
Right. A 'dd' copy is an exact copy of the VHD.
Quote:
Originally Posted by circuit_PT
I google in how to add a vhd to my virtualbox
VirtualBox appears to support the VHD format since version 2.x:
- create a virtual machine,
- add a Live CD ISO as medium to boot from,
- add VHD as hard disk,
- now run, load, boot or whatever you call it,
- enable LVM, copy files out.
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.