One of the things that dd doesn't do is write to optical media. If growisofs uses dd, it is inside a scsi wrapper that communicates with the CD/DVD device. CD/DVD devices are all slightly different, and dd doesn't bother to query the drive, it just assumes. The scsi module translates the dd communication to scsi communication for optical drives. Or at least that's probably the way it works. I have read of someone using:

hdc=ide-scsi in the /boot/grub/menu.lst kernel line

and then using the device:

/dev/scd0
like so:
dd if=/dev/hdd of=/dev/scd0

But I could never get it to work, so I think it was a fluke.

I started my clone from a 80GB to 160GB hard drive 17 hours ago with this command
dd if=/dev/hda of=/dev/hdb conv=notrunc,noerror
As of right now, dd is still running on my system. Is there a way to find the current progress on dd? How much longer should I wait? FYI, the 80GB does have some back block on it. When I cloned my good 80GB to the same 160GB drive it only took 7 hours to completed. What is my options?

John

17 hours is a bit long to wait. You might also use the block size, 'bs' parameter like so

dd if=/dev/hda of=/dev/hdb conv=notrunc,noerror bs=4k

If the drive has bad blocks you first want to do:

dd if=/dev/hda of=/dev/hda or
dd if=/dev/zero of=/dev/hda or
Both.

Thanks...I finally break out after 29 hours. Since, the ps was in S+ state. The weird part was I was able to boot up with the new drive for 2 hours then I get readonly on the / mount point. When I rebooted the server it won't boot all the way. It took me to single mode and said I need to run fsck to the mount point. So, I'm going to do a fsck this weekend and see if that will fixed my problem.

Great article! This site is much better than I thought! Great work guys!

Great overview for a Linux newbie like me! I came here after discovering that both Partition Magic and gparted wouldn't copy a TrueCrypt partition (presumably because they didn't understand its internal format - which is entirely reasonable when asking such a utility to resize a partition but seems a bit arbitrary for a simple copy operation, where a blind byte-by-byte copy should do the job if it doesn't understand the internal format sufficiently to do something more efficient) - and then seeing references to the use of dd in the latter (and in clonezilla as well). While I appreciate the added layer of safety that such GUI utilities provide for the fumble-fingered, for the task I was actually interested in performing dd looks like an entirely acceptable option.

Question: why use noerror for copy operations? Your post seems to say that on an input error dd will skip the data in the output (rather than leave a space where it should have gone), resulting in an unusable file/partition/disk beyond that point, whereas if you let dd stop on an error you can just restart at the appropriate offset to retry and keep going (assuming that the error was transient in nature).

Question: if you use, say, a 4 KB block size (as you do in some of your full-partition and full-disk copy examples), what happens to the last few disk sectors if the entity you're operating upon does not contain a multiple of 8 sectors (which, for example, will *usually* be the case when copying a partition comprising 16065b cylinders)? Since you recommended this option I had expected dd to do the right thing (copy the remainder sectors at single-sector granularity), but toward the end of the thread you seem to say that it does not (which could prove disastrous for any file system or database that used the last few sectors in its partition for something important). In a somewhat related vein, some systems (some IBM Thinkpads, for example) use a nominal 240 heads rather than 255 heads, making their nominal cylinder size 15120b rather than 16065b - another case where having dd 'do the right thing' if an awkward block size were chosen could avoid unpleasant surprises. A third potential problem is that both 'logical' partitions (in the 'extended' partition) and any partition beginning at the very start of the disk do not include the first track of their first cylinder, because that's where the MBR (in the latter case) or the EPBR (for a logical partition) resides - once again screwing up the expected alignment (whether 16065b or 15120b). If dd indeed does *not* do the right thing in such cases, using a 512-byte block size (or possibly a 63b block size, since all modern hard disk cylinders seem to use nominal 63-sector tracks) would seem the only prudent course.

For that matter, will dd copy files exactly (not that it normally should be used for this) if the last sector of the input file is only partially filled? Not all file types will interpret zero-fill after the last valid byte in the manner desired, so unless dd honors the byte-granularity end-of-file value in an input file there could be a problem.

As others have already observed, your advice in the OP about copying a partition to a larger or smaller one is inappropriate: a file system (or database) resident in a partition usually has internal information regarding that partition's size and does not necessarily re-check this size as described in the partition table entry (which it may not even know how to find) every time it starts; even if it did perform this check it could have structures spread throughout the original partition's space that could be inconsistent with its revised size - even after defragmenting; and even if all the filesystem's structures were nicely compacted at the start of the partition, the remaining space would not necessarily be filled with zeros (and hence compactable by dd) if data had resided there in the past. The bottom line is that you really should use a file-system- (or database-)aware utility to resize any partition containing a file system (or raw database).

Furthermore, my understanding is that the partition table in the MBR does not describe the size of the disk but just the sizes and locations of up to 4 primary partitions on that disk (one of which - or possibly more than one of which in Linux - may be an 'extended' partition). So skipping the MBR in a whole-disk-to-whole-disk copy is not helpful, and in fact skipping it leaves the output disk's partition table (if any) in its previous state while the source disk's (likely quite different) partition layout is copied to the output disk: instead, if you want the same partition layout on the output disk, copy the MBR along with everything else as a whole-disk-to-whole-disk copy, and if you don't want the same partition layout on the output disk, create the new layout there (using the sizes of the relevant partitions on the source disk) before doing anything else (and copy over the first 446 bytes of the MBR if the master boot code on the source differs from that created on the destination), then copy partition-by-partition, then resize as desired using some utility that understands how to do that properly.

Overall, though, a great resource: thanks for the time and effort that you have put into it.

Haven't tried dd yet (pending answers here), but did try a little (runs from two floppies) NetBSD utility called g4u that seemed to do the job (i.e., perform a dumb bit-for-bit copy of a TrueCrypt partition, though I haven't yet tried doing so to a different offset on the target disk from the source disk offset).

Thank you!

Great help article!

My old drive had been causing XP to lock up each day. Ran Spinrite on it and got a warning that the drive was overheating. I shut it down, unplugged the old drive, popped in my fav Linux LiveCD distro to get online, did some research and luckily found this site.

I didn't know if using dd would be the best choice for a drive in trouble of failing (or with errors) so I ended up using Antonio Diaz's Gnu ddrescue instead of dd.

Using an Ubuntu LiveCD (and installing Gnu ddrescue, which is "gddrescue" on that distro), I was able to successfully clone my "old" Maxtor 120GB to a new WD 200GB. Then rebooted with just the new drive and everything worked fine. I then used gparted to "claim" the extra free space on the new drive, and it all worked great!

The information I got from this forum topic helped trememdously. Thanks everyone.

An interesting tip to checkout

http://fosswire.com/category/tips-tricks/

I want to thank you for posting this. To say it has been most useful would be a gross understatement.

In playing with FreeBSD, on one of my computers, I can't say how many times I've had to reinstall after screwing something up, and reinstalling software from ports can be unbelievably time consuming. Now, working in steps, as soon as I get something right, I move it to the slave drive. If I later screw up, it's a simple matter to get back where I began, like you said.

I might add that the two drives on this computer aren't a matched set, but both are the same size (well, very close) and it works. Also, thanks for mentioning the Helix CD; much better than using dd.

I have attempted several times to backup my 20Gb hdd running xp to a file on my 200Gb hdd which has about 170Gb free space to work with using variations of

but every time I try, dd doesn't stop copying at the end of my 20Gb hdd, but instead continues writing to the "of" until I receive an error message stating:

this gives me no room to work from on my hdd.
so now what do I try?

ps. great write-up!

SORTED OUT
stupid mistake. mixed up my hdd's. was copying my hda into a file on itself....
now to give it a go the right way....

While I'm sure you're cute, I have the benefit of experience. I have tried all the examples in the OP. They might be theoretically wrong if dd is a dumb bitstreamer, but it isn't. It's a smart tool, although this is supposed to be transparent to the user.

I see you have a lot of information. If one could learn by watching, dogs would become butchers. Now get some HDDs, and start doing some real work with dd. If you have another question, feel free to come back and answer it for yourself in the same post you ask it. Take a break from whatever it is you're not taking a break from often enough. And, You're Welcome

Awesome

You're Welcome,

ezor

I have been looking for a way to securely delete a file from a reiserfs partition. There doesn't seem to be any good solution. Having read through or scanned every post here, would dd be worth trying?

$ dd if=/dev/urandom of=somefile bs=721 count=1 conv=notrunc

Secure delete utilities such as shred apparently don't work well with journalling file systems. It's just a single file I want to get rid of securely. I suppose I could write and then print off the file using a live cd, which wouldn't leave any trace either. Any suggestions?