Learn the DD command

Emmanuel_uk · 12-19-2006, 01:16 AM

Quote:

wanted to know if it is possible for dd to clone a 80gb hdd to a 20gb hdd. Their is only about 4gb of data on the 80gb hdd

I have already asked the question, if you look in the thread. Mateer also gave some advice about resizing partition before using dd. Anyhow, still as indicated by other posters cp -R is best

AwesomeMachine,
I have heard of people having problem with the journal of ext3 when using dd, and resorting to transforming ext3 to ext2 before dd'ing. Do you know when this technique ext3->ext2 *really* need to be applied? I am not sure this is really needed (sorry I have no example at hand)

yamez · 12-21-2006, 06:13 AM

I want to create an image of a USB flash drive then write that image back to another same sized USB flash drive when needed. I need to image the whole drive. Using Knoppix 5.0, I enter the command...

dd if=/dev/sdb of=/home/knoppix/usb.img bs=1M

When writing back after I swap USB flash drives, I use...

dd if=/home/knoppix/usb.img of=/dev/sdb bs=1M

The problem...

The source USB flash drive has 2816 files, 58 folders, total 94MB.
The target USB flash drive has 2190 files, 45 folders, total 64MB.

However, in Windows under drive properties, both USB flash drives report using the same space of 98MB. I've tried 2 different USB flash drives, both 128MB. No errors result when running dd.

Any ideas?

AwesomeMachine · 12-26-2006, 09:18 PM

Quote:

Originally Posted by saikee

I believe that is down to the kernel. If the kernel recognises the RAID disks as you want it to then you are home dry.

The kernel does have to cope with different chipsets and if it fails the two disks in the RAID may be treated as two separate disks.

You should have a pretty good idea of the hard disk recognition if you issue the "fdisk -l" command requesting all disks to be displayed with their partitions tabulated.

There are too many variations with RAID for dd to be able to reliably work with copying raid arrays to other raid arrays, unless the arrays are external. An external RAID array looks like one big disk to dd, but when you try to put two discreet RAID setups in a single PC it is very difficult. If you have four disks, any controller will try to configure this as RAID 5. If you really want (2) RAID 0 you probably won't get that.

With small arrays there are ways to copy the whole array onto a single disk, and then rebuild the array by adding disks.

AwesomeMachine · 12-26-2006, 09:31 PM

AwesomeMachine,
I have heard of people having problem with the journal of ext3 when using dd, and resorting to transforming ext3 to ext2 before dd'ing. Do you know when this technique ext3->ext2 *really* need to be applied? I am not sure this is really needed (sorry I have no example at hand)[/quote]

I hadn't heard of converting from ext3->ext2 before using dd, but there are some theoretical concerns with the superblock and journals on ext3, and use of dd. By careful guessing, I would assume this is currently a problem, or was a problem at some point. I can't find anything in any books on this, but I think if a partition is copied by itself, and it isn't the first primary partition on the HDD, and it is ext3, you need to convert to ext2 before using dd. If someone could blackbox this theory for me I would appreciate it.

AwesomeMachine · 12-26-2006, 09:43 PM

Quote:

Originally Posted by yamez

I want to create an image of a USB flash drive then write that image back to another same sized USB flash drive when needed. I need to image the whole drive. Using Knoppix 5.0, I enter the command...

dd if=/dev/sdb of=/home/knoppix/usb.img bs=1M

When writing back after I swap USB flash drives, I use...

dd if=/home/knoppix/usb.img of=/dev/sdb bs=1M

The problem...

The source USB flash drive has 2816 files, 58 folders, total 94MB.
The target USB flash drive has 2190 files, 45 folders, total 64MB.

However, in Windows under drive properties, both USB flash drives report using the same space of 98MB. I've tried 2 different USB flash drives, both 128MB. No errors result when running dd.

Any ideas?

With hotplug, /dev/sdb isn't the drive, it is the disk. If you want to copy to the drive, you use - /dev/sdb1. With conventional HDD's /dev/hda, sda are drives, but not with USB drives using hotplug. Another thing, sometimes with hotplug, if the device is detected once, and then it is unplugged, it won't be recognized again when it is plugged back in. It usually doesn't take long for /dev/sdb1 to disappear after plugging in a usb stick, so look in /dev quickly for the proper partition. With sticks that size there is probably one drive on each stick. There is a 512 MB boundary on FAT12, which is usually what small sticks are formatted with. You can also use iso9660, and put a bootable iso image file on the stick. USB boot is compliant with the el torito boot specification CDRoms use for booting.

drhousbr · 01-01-2007, 06:21 PM

This was very helpful. The only extra thing I had
to do was run reiserfscks --check prior to running
resize_reiserfs. This was under SLES 9 AMD64.

Worked for me. I had a cloned disk that booted fine.

Cheers, DRHO

Quote:

Originally Posted by jlinkels

Almost.

I assume you have been partitioning hdb correctly and that 64M as first partition is enough to hold you boot stuff etc.

Then do:
dd if=/dev/hda1 of=/dev/hdb1 bs=4k conv=notrunc,noerror
to copy the first partition
and
dd if=/dev/hda2 of=/dev/hdb2 bs=4k conv=notrunc,noerror
to copy the second partition.

(You were right here, but might have forgotten to clone the first partition)

Still, the drive is not bootable, because you did not copy the master boot record yet. You can this with:

dd if=/dev/hda of=/dev/hdb bs=446 count=1

Note that you do a copy on a disk and not on a partition. The bs=446 is important because this is exactly the boot sector, without the partition table.

Then, resize both partitions:
resize_reiserfs /dev/hdb1
resize_reiserfs /dev/hdb2

And NOT /dev/hdb!

Don't forget you might want to create a swap partition if you don't have one yet. On the partition you designated as swap, do mkswap to initialize it as swap partition, no need to do this with dd.

And yes, you can do this in a live system.

jlinkels

user99099099 · 01-02-2007, 11:27 AM

Wow those first two posts (the ones with all the dd related information) were huge. Great information though.

AQG · 01-04-2007, 11:14 AM

Excellent !!!!!

yamez · 01-04-2007, 08:42 PM

Quote:

Originally Posted by AwesomeMachine

With hotplug, /dev/sdb isn't the drive, it is the disk. If you want to copy to the drive, you use - /dev/sdb1. With conventional HDD's /dev/hda, sda are drives, but not with USB drives using hotplug.

I have tried your recommendations but the same error occurs. I did find that it appears to be the USB flash drives. I have 100 identical flash drives and 30 have this error when using dd, the remaining do not. However, when I do a normal file copy ALL the flash drives work. I would like to use imaging though for support of bootable flash drives and different file systems/partitioning when duplicating multiple drives. Is there a way with to determine how many written blocks are on the drive and then use that number to limit how much dd writes? Thanks for your reply.

AwesomeMachine · 01-04-2007, 11:52 PM

Yamez,

You weren't very detailed in your response. Were you able to locate a partition on each drive (i.e. sdb1, sdb2)? If so, did you try writing to the partition instead of the sdb device? Did you use fdisk -l to look at the partitions? You don't give a lot of information. On a USB flash drive, 512 byte logical sectors are used. You can do:

dd if=/dev/zero of=/dev/sdb1

and when it gets finished it will say how many 512 byte blocks it copied zeroes to.

dwcoffin · 01-08-2007, 01:26 PM

Is it possible to dd to a SMB share using Netcat or some other network mechanism? I have been playing with DD in VM, but now want to learn how to create a gz file of the image to a network location. I'm currently using different flavors of RedHat our production servers are AS4 and my desktop VM's that I'm playing with are FC5.

AwesomeMachine · 01-12-2007, 03:49 AM

Quote:

Originally Posted by dwcoffin

Is it possible to dd to a SMB share using Netcat or some other network mechanism? I have been playing with DD in VM, but now want to learn how to create a gz file of the image to a network location. I'm currently using different flavors of RedHat our production servers are AS4 and my desktop VM's that I'm playing with are FC5.

Dd is specifically designed to copy a certain amount of blocks from a block device to a file or block device. Block devices are found in '/dev' directory. I've never used samba, so I don't know if samba makes a block device of the remote share on the host. Also, dd does not work with directories. You can copy files only. Since block devices are virtual files, dd can image complete drives. Dd can also duplicate other files, but not mount points.

nirmaltom · 01-12-2007, 03:55 AM

hi,
copied the page to my self linux helper directory
regards,
Tom.

Duck2006 · 01-12-2007, 07:23 PM

Thanks a lot, this is great

alisou · 01-15-2007, 01:03 PM

Hi all,
Verry good post!
AIR (Automated Image and Restore) is a GUI front-end at dd/dcfldd.
http://air-imager.sourceforge.net/
and this live CD use AIR:
Helix, http://www.e-fense.com/helix/
BackTrack, http://www.remote-exploit.org/index.php/BackTrack
4BAK, http://4bak.org/
AliXe, http://alixe.org

Regards!
alisou, :-)