DD with Netcat restoring images
 

Hi all

I am new to Linux and have been trying various ways of backing up Linux servers at work, without having to purchase any software...

I have managed to take snapshots of the full hard drives in the servers and copied them to a central location server for them to be stored on:

Server config:
nc –l 1234 | gzip | dd of=/home/sda.image.gzip
host config:
dd if= /dev/sda | gzip | nc 192.168.0.1 1234

the above creates an image of the host hard drive on the server which is fine (i think) and I know how to schedule using cronjobs so am also ok with that.

The issue i have is when it comes to restoring the image to a new hard drive!!! I have used countless live cd's including Helix, Knoppix, systemrescuecd, ping, clonezilla, g4u etc however it just will not create a replica of the original!

I have also uploaded the image from the server to a seperate FTP server to see if i can restore from FTP via the live cds - just cannot figure out how to do it!

I have also tried DD over netcat the other way to create a replica of the original:

Server config:
Dd if=/home/sda.image.gz | gzip | nc 192.168.0.2 1234
Host config:
Nc –l 1234 | dd of=/dev/sda

Could anyone suggest firstly what I am doing wrong and how to do it correctly or if i am missing something :S

Lastly I have setup a working version of FOG server however being live production servers they cannot be rebooted to grab an image via PXE! (testing has been done in a lab environment by the way)...

Many thanks in advance for any help :)

Dave

Welcome to Linux Questions!

From your post, I take it that this is the crux of your complaint: May I ask how you are making the comparison? Does the image fail to operate? What is different and what problem are you experiencing?

When you use DD to make the image of the drive, it is important that the drive not be mounted and that you don't make changes to it while engaging in the process. This is where the part of the livecd comes into play. When you use the livecd, if it detects that there is a swap partition on the hard drive, it may still mount it and use it. I encountered this a few days ago when I tried to make a backup image of an entire drive before performing some distribution replacement surgery on it. I found that the md5sum of the image did not match that of the drive. I then used the cat command to pipe the image and compare it with /dev/sda. I found that the byte addresses in the range of the swap partition had differences. I then compared the active partition table (df -h) with the swap on and swap off and discovered that the live cd was using the detected swap partition, hence the modifications.

Another thing to consider is that DD will copy everything, including error sectors. If your drive has errors, you need to tell it to at least write the correct length of data. I thought that this link had a pretty good discussion of the subject, save the swap partition part.

Hi Noway2

Thanks for the quick reply - in short the restored image does not work at all!

I think it must be where i am not unmounting the drive as i DD it - i am running the DD on the machine on command line, i was using the live cd to restore it!

To summarize.... I would have to unmount the drive first before using DD - is there no way of DD 'ing while the server is on?
Can you suggest anything else that may be used to clone a hard drive under linux without having to bring the server offline and/or unmount the drive?

Many thanks

Dave

well you *should* umount it, but really it is unlikely to matter.

Why do you think the image is legitimate? You appear to be compressing the data twice on your first stage, once on the client, and once on the server. So your processes end to end result in you dd-ing a compressed image on to the disk, not the original decompressed data.

Hi acid_kewpie

In all honesty i dont know if the image is legit - its just that it doesnt throw up any errors when it finishes - i am pretty new to all this so sorry you will have to make allowances for that :p

I see about the compression now so for compression to be done on the fly via netcat would the commands be the following?:

Server config:
nc –l 1234 | dd of=/home/sda.image.gzip
host config:
dd if= /dev/sda | gzip | nc 192.168.0.1 1234

or would it be:

Server config:
nc –l 1234 | gzip | dd of=/home/sda.image.gzip
host config:
dd if= /dev/sda | nc 192.168.0.1 1234

maybe thats where i have been going wrong then!!!!

thanks for the insight so far guys....

Dave

well gzip on the client side, don't waste all that bandwidth.

ok update to my original post:

copied full disk via netcat and compressed:
Server config:
nc –l 1234 | dd of=/home/sda.image.gzip
host config:
dd if= /dev/sda | gzip | nc 192.168.0.2 1234

booted new target machine with system rescue cd. Configured eth0 with ip address. Target machine already had standard Linux installed with same partition layout as original host machine (192.168.0.2). Sent image via netcat to new machine:

dd if=/sda.image.gzip | nc 192.168.0.3 1234
new target machine set to accept image and write to existing drive:
nc –l 1234 | gunzip –d | dd of=/dev/sda
waited until write process was complete – rebooted machine……still nothing!!!

When the image was copied to the server the image only sized at 2.5 Gb - the disk was 40Gb so i am presuming that the image did not complete correctly??

Am i missing anything else from the above that anyone can see???

Many thanks

Dave

the image is compressed, that's the point of compression... making things smaller!

on the server, just decompress the image and see how big it is.

Hi Chris

I understand that compression makes things smaller i just didnt think it would compress 40Gb to 2.5Gb thats all - it just seemed quite small....

do you have any thoughts on the rest of the post?

Many thanks

Dave

A compression of 40Gb to 2.5Gb does seem small. Is 40Gb the size of data on the drive or the entire size of the drive? Did you fill up the free space with zeros prior to beginning the procedure?

Regarding the rest of the posts, please note that Chris was pointing out that you had double compression, which when you re-inflated upon attempted to restore you were still writing a compressed image to disk, not the image.

Hi Again

40Gb is the size of the entire disk, I did not fill up the free space with zeros no - is that a procedure I will have to follow?

I thought that the command 'nc –l 1234 | gunzip –d | dd of=/dev/sda' would decompress the image onto the target drive thus restoring the original image....is that not the case then?

Thanks for the input so far guys :)

Dave

you can compress a blank 10tb file to 1k or whatever, if there's nothing to store, it's not going to use up space storing it.

the drive is partitioned to 2 20Gb partitions:

there is a basic Centos 5.7 Server install without any extra feature enabled....there is also a Broadworks R17 XSP install completed on the server.

df -h shows:

Filesystem Size Used Avail Use% Mounted on
/dev/sda1 20G 2.1G 17G 12% /
/dev/sda2 20G 1.5G 17G 8% /bw
tmpfs 741M 0 741M 0% /dev/shm

So i am presuming due to the fact i didnt write the unused space with zeros thats why it only shows as 2.5Gb in the image file....?.....

fdisk -l shows:

Disk /dev/sda: 42.9 GB, 42949672960 bytes
255 heads, 63 sectors/track, 5221 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sda1 * 1 2611 20972826 83 Linux
/dev/sda2 2612 5221 20964825 83 Linux

the target host has the same partition layout and same linux server install - just not the Broadworks install.

I will try and create an image and zero out the free space and see if i have anymore joy with that approach.....

Many thanks

I have just read that the dd command should write zeros to the non used space already...is this true?

could anyone give an example of the command i would need please?

i have tried:

dd if=/dev/sda conv=sync,noerror | gzip | nc192.168.0.2 1234

and

dd if=/dev/sda conv=notrunc,noerror | gzip | nc192.168.0.2 1234

however both have been unsuccessful!!!

Many thanks

See the following: http://www.hanckmann.net/?q=node/19

You need to be careful that you compress the data before you send it through netcat and then save it to a file but don't compress it again. Then when you wish to restore the image, you send the compressed image across the network, decompress it and write the data to disk. You should be able to verify the image by comparing an md5sum of the entire hard drive against the decompressed image. When you decompress the image use the flag to send the image to the stdout and then pipe this through md5sum. This is where the use of a livecd and turning the swap off comes into play because it enables you to avoid making changes to the drive for the duration of your process so that you can verify the sums. You can use gzip to both decompress and compress, but you need to use the -d flag to decompress.
This will write the compressed image to the file sda.image.gzip.

Now while you have the livecd running and the swawp OFF: to get the sum of the drive.
Then on the machine with the image:
You should get the same checksum, validating your image as being an exact mirror copy. Notice what we are doing, we are decompressing the image to the stdout and piping this through the md5sum utility.

Here is an example.
In your case, you will want to pipe the data across the network with netcat.

You can use the same kind of procedure, only in reverse to restore the image. Use netcat with dd to stream the compressed image across the network, decompress it on the reciving end and use dd to write the image to the disk. You can then use md5sum again to verify that the image was written to disk correctly.