gzipped dd VM backups on LVM + soft RAID-1 failing

Gerard Lally · 04-01-2012, 06:12 PM

I have a Slackware amd64 server running kvm-qemu with 5 Windows XP guests.

The disk layout is as follows:

4 brand new SATA disks: 2x500GB and 2x1TB

Partitioning:

/dev/sda1 + /dev/sdb1 = /dev/md0 = /boot
/dev/sda2 + /dev/sdb2 = /dev/md1
/dev/sda3 + /dev/sdb3 = /dev/md2
/dev/sdc1 + /dev/sdd1 = /dev/md3
/dev/sdc2 + /dev/sdd2 = /dev/md4

LVM:

system-volgroup on /dev/md1 is for root, tmp, swap, usr, var and home Logical Volumes.

vm-volgroup across /dev/md2 + /dev/md3 is for virtual machines (four underlying disks)

/dev/md4 is mounted as /data, for samba shares and public data generally.

So far I have tried to backup as follows, with all methods failing:

Licensed copy of Terabyte Image for Linux - imaging proceeds to about 98% and then reports corruption.

Clonezilla - same thing, although I later read that Clonezilla doesn't support software RAID.

dd - I shut one of the virtual machines down, and then make a LVM snapshot as follows:

Code:

# lvcreate -s -n xpmachine1-snapshot -L 5G /dev/vm-volgroup/xpmachine1

I then back it up to a file with dd and gzip:

Code:

# dd if=/dev/vm-volgroup/xpmachine1-snapshot | gzip -c > /data/backup/xpmachine1.dd.gz

I then try to restore:

Code:

# dd if=/data/backup/xpmachine1.dd.gz | gzip -d | dd of=/dev/vm-volgroup/xpmachine1

However, I keep getting an error, that the data stream is corrupt. I have tried it without gzip, and I tried it with bzip2, and I get the same result.

The disks appear to be in fine condition. I am wondering if I need to shut all virtual machines down while creating the snapshot and then backing it up, in case some underlying RAID or LVM activity is corrupting the backup.

Perhaps I should boot the entire system from SystemRescueCD and backup the Virtual Machines that way?

Would appreciate some guidance. Thank you.

dyasny · 04-02-2012, 04:35 AM

what virtualization technology are you using?
how is the VM image LV set up - what format?

Gerard Lally · 04-02-2012, 09:34 AM

Quote:

Originally Posted by dyasny

what virtualization technology are you using?
how is the VM image LV set up - what format?

I am using the kvm modules included with the vanilla kernel 3.0.22, and the qemu-kvm 1.0 userspace from the KVM website.

Storage backend for each virtual machine is an unformatted logical volume, created like this so that it is striped (LVM striped) across the two underlying RAID-1 volumes:

Code:

lvcreate -i 2 -n stores1 -L 40G vm-volgroup

I start the VM as follows:

Code:

# Start virtual machine ... 
#
echo "Starting Windows XP Professional Virtual Machine ... "
echo "MAC address is ranmac1 ... "
echo "TAP is tap1 ... "
echo "Storage is /dev/vm-volgroup/stores1 ... "
echo "VNC is :11 ... "
echo "IP address is 10.40.51.51 ... "
echo "User is stores1 ... "
sleep 3
qemu-kvm -cpu host -localtime \
        -vga std -daemonize \
        -usbdevice tablet -soundhw sb16 \
        -net nic,macaddr=$ranmac1,model=e1000 \
        -net tap,ifname=tap1,script=tap1-up.sh,downscript=tap1-down.sh \
        -drive file=/dev/vm-volgroup/stores1,cache=none,aio=native \
        -cdrom /data/iso/system-rescue-cd-2-5-1.iso \
        -boot c -m 1536 -vnc :11 -k en-gb

dyasny · 04-02-2012, 09:59 AM

I think I've missed that last part - you're doing this while the VM is running? And the running VM uses the base LVM snapshot, while you're cloning the snapshot itself?

The idea behind snapshots is that you will take a snapshot, move the VM over to the snapshot, leaving the base image in r/o mode, so it doesn't change, and then backup the base image. if you keep changing the base image while the VM is working, the snapshot will definitely be corrupted, because it relies on data that gets changed under its feet.

Gerard Lally · 04-02-2012, 06:10 PM

Quote:

Originally Posted by dyasny

I think I've missed that last part - you're doing this while the VM is running? And the running VM uses the base LVM snapshot, while you're cloning the snapshot itself?

Yes I am cloning the snapshot. At first I had the VM running but then I tried creating the snapshot and cloning it when the VM was turned off.

The procedure is outlined here.

Quote:

The idea behind snapshots is that you will take a snapshot, move the VM over to the snapshot, leaving the base image in r/o mode, so it doesn't change, and then backup the base image. if you keep changing the base image while the VM is working, the snapshot will definitely be corrupted, because it relies on data that gets changed under its feet.

Well I wasn't aware I had to move the VM over to the snapshot and clone the base image! All the guides I've seen do it the other way around - create the snapshot and then clone the snapshot with dd.

What you're saying is that I should
1) create the snapshot;
2) shut down the VM;
3) adjust the script so that the next time the VM starts it starts with the snapshot as its storage backend;
and
4) dd the base image instead of the snapshot?

It sounds right, but why do the guides on the internet have it the other way around? All of them tell us to clone the snapshot.

dyasny · 04-03-2012, 04:08 AM

Had to go read up (and ask a colleague) about LVM snapshots, and indeed they do not behave like a typical COW chain. So the guides are right - you need to be backing up the snapshot, not the base.

So the procedure is correct, however, corruption still occurs. I wonder if this happens because you don't quiesce your VM. With Linux, it can be done using fsfreeze, in Windows, probably, something around vssadmin should do the job, but I always used a backup software vendor provided vss agent for that.