Well, my wife sometimes thinks so...

So do I, sonny: I've been designing and implementing file systems since I began doing so at DEC over 30 years ago (were you even in diapers back then?). I'm only a *Linux* newbie.

With every file system and database supported not only by Linux itself but in any other environment which uses disks that you so blithely suggest people meddle with using dd?

I thought not - and that, of course, was my point.

It would have to be an *omniscient* tool to be adequately knowledgeable to fiddle around with the internals of every file system and database that can be found on the disks whose *partition* layout it understands (again, that was my point).

Which apparently went completely over your head.

No need, at least at the moment: as I noted earlier, NetBSD has solved my immediate problem.

I think not, given your attitude: if I need to know more about dd I'll just look at the code (or ask someone who actually knows the internals if I happen across him/her).

AwesomeMachine thank you very much for all the work you have done and mainly sharing your knowledge. I learned a lot.

All the best
-=terry=-

I just found this. Learned a lot. Thanks.

Would you mind telling me, in
# dd if=/dev/zero of=loop bs=4K seek=256K count=1 > /dev/null 2>&1

what does "> /dev/null 2>&1" mean?

Thanks.

great stuff
everything you didn't need to know about dd
More importantly, it's got all the things you do need to know.
More comprehensive than man and info.
Bookmarked.



for the poster above
> /dev/null 2>&1 redirects STDERR to STDOUT.
(prints errors to the terminal)

BT, I'm sorry for my little joke, but you sounded like a 18 year old girl, with a double major in English composition and computer science in your first post. Let's just say I believe you are who you say you are. At least people reading this thread will know now.

Don't get down about what I said before. I was messing with you to make the thread a little more entertaining. I don't really think you're a young girl. Your post was just so perfect, I couldn't resist.

Please accept my apologies. I didn't know I would provoke such a response. Although I do admire your ability to stand up for yourself.

If people were only allowed to give instructions that were 100% reliable in every theoretical possibility, no one could ever assist anyone else, and humanity would cease. I do the best job I can on this thread, and that's pretty darn good.

This thread is the best information on dd anywhere. You're correct. Dd is not magic. On the systems dd does not work, it is generally taken out of the available tools, or rewritten so it does work.

I will place a warning at the beginning of the OP to the effect: Caution should be observed when using dd to copy encrypted file systems.

-Awesome

Certainly: had I had any idea that's what you were doing, I would have responded differently.

That's pretty much what I tried to say in my original post by thanking you for what was valid and trying to help correct what was not.

Actually, I'm not sure that's a problem (the difficulty I had using higher-level utilities based on dd for that purpose may have been in their logic, not in dd's): my main concern is about using dd to change (especially, to shrink) partition sizes when copying partitions. dd may (or may not: I still haven't checked the code) understand how to modify the internal data structures of *some* file systems and/or databases when changing the size of the partition in which they reside, but I seriously doubt that it understands how to do so for *all* that it may encounter. Any corruption thereby introduced will not necessarily be immediately apparent: the file system or database may continue to work just fine for quite a while, until it tries to write beyond the end of the partition and receives errors it is not prepared to deal with.

I found The Open Group's specification of dd at http://www.opengroup.org/onlinepubs/...lities/dd.html, and it's pretty comprehensive in describing the nit-picky details of dd's operation. One interesting wrinkle is that in the absence of the sync operand bs=xxx does not work quite the same way that the combination ibs=xxx and obs=xxx does when it comes to partial input blocks - one of my original questions and the reason why, in the absence of sync, dd does the right thing with any partial block at the end of a copy operation as long as bs is used rather than the ibs/obs combination. It also confirms that using noerror in the absence of sync is probably not very useful when copying a partition (since every data block beyond the point of error will be shifted to a lower partition address, hence may well be actively confusing as well as unusable); using sync would keep the later blocks in the right position but would then introduce a partial-block problem at the partition end if a block size other than 512 bytes was used that did not integrally divide the partition size (another of my original concerns).

Hello!

Can anyone tell me why dd sometimes prints the statistics as

nnnnnn+0 records in
nnnnnn+0 records out

and sometimes there are non-zero numbers after '+', like

nnnnnn+22 records in
nnnnnn+22 records out

Does it indicate an error?

I have some problems after cloning the NTFS partition over the net by piping it through nc (the clone does not boot). The partitions have the same size, but dd ended with "not enough space" error and I noticed the "extra" records reported. I cannot just put two disks into one box, as they are in laptops

BTW: great thread.

Good, thanks

AwesomeMachine,

I've studied all 24 pages of this thread, and none of the suggestions have worked for me. Here's my story:

I installed FC6 on an 80G drive and accepted default partition sizes (who am I to argue with Linux?). I set up an Apache web server, MySql, and Bugzilla on that drive (many hours of work). To keep it safe, I cloned the 80G (using G4U) on to a 320G that I could safely "mess with". I messed with it so well, that the 320G now contains exactly the system I want to run, so I need to clone it onto a 40G for daily use and abuse. Unfortunately, G4U doesn't want to clone a big drive onto a small one. In search of a solution, I started playing with dd, trying to clone the now "expendable" 80G on to the empty 40G (I'll worry about 320 to 40 later). There is more than enough space on the 40G to hold my stuff.

THESE ARE MY DRIVES:
********************************************************************
Disk /dev/hda: 80.0 GB, 80026361856 bytes
255 heads, 63 sectors/track, 9729 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/hda1 * 1 13 104391 83 Linux
/dev/hda2 14 9729 78043770 8e Linux LVM

Disk /dev/hdb: 40.0 GB, 40060403712 bytes
16 heads, 63 sectors/track, 77622 cylinders
Units = cylinders of 1008 * 512 = 516096 bytes

Device Boot Start End Blocks Id System
/dev/hdb1 1 208 104800+ 83 Linux
/dev/hdb2 209 77622 39016656 8e Linux LVM
********************************************************************
I created the above partitions on hdb to resemble those of hda. hda is mounted, hdb is not.

I've tried to dd partitions to partitions and then dd the boot record without partition info from hda to hdb, but things still only boot so far before necessary files can't be found and error messages abound.

Trying another method, after taring to a mounted hdb, I couldn't chroot:
[root@localhost ~]# chroot /mnt/hdb1
chroot: cannot run command `/bin/bash': No such file or directory

I would like to do what you recommended in one of your posts (#300) regarding small to large cloning, if it will work in the other direction as well:
"use cp -r * from the root directory of the old drive to the root of the new drive. Then, use the linux install CD in rescue mode, and reinstall the bootloader to the MBR."


My questions:

1) Will the cp -r method copy files from and to the proper partitions, or will it just dump stuff where it finds room, totally jumbling up a necessary structure?

2) How does one "...use the linux install CD in rescue mode, and reinstall the bootloader to the MBR." (my hda uses grub)

3) Is there a better way to move my system onto a smaller drive?

Thanks.

really really good!

Congratulations!

Yes, IMO you're doing things the hard way. Dd is very versatile but IMO is really best used for emergency data transfer (for instance from a failing hard disk)
Why don't you boot from Knoppix recovery cd (for intance). These usually contain partition magic type facilities Gpart, PartEd or Gnome partition editor which are ideally suited to copying and resizing partitions.

Jasee,

Thanks for the tips. I researched them and found that they cannot handle LVM. I'm now trying to resize my logical volumes and file system before cloning. Maybe then dd will do the trick.

Sorry I didn't notice the reference to LVM. I am a Linux newbie, I knew nothing about LVM.

From a brief reading it seems like LVM is somewhat similar to software raiding, striping etc: Dynamic disks (XP or later) . Nothing I would like to deal with! Hardware solutions are much better! I'm amazed dd works at all (as presumably the partition tables don't reflect the volumes directly) In NT you would back up the data and restore to a simple disk.

I'm going to rephrase your question so people know what this reply is about. For some reason you are configuring laptops with a drive image stored remotely, inaccessible except by Internet. So the image is not on the same lan as the laptops. The image file will not make a bootable partition on the lappies, or laptops. The network copy almost works, but has minor problems.

Microsoft Vista NTFS is not compatible with Linux tools, and I doubt if it ever will be. I think the days of Linux trying to be compatible with Windows file systems is over. If you're working with XP partitions, you have to transfer the image to the same partition number it came from. If you back up a Windows root partition, or c:\, you need to put it where the hidden file, c:\boot.ini says it is supposed to be, or the boot loader, NTLDR will not find it. You can edit boot.ini so it will look anywhere you tell it, even on an extended partition. So if you have more than one partition on the laptop drives you'll need to consider this.

If you're simply backing up, and restoring to the same place the backup came from, I would try using 'bs=16065b', and don't try to use bzip2 compression. You can use gzip, which isn't anywhere near as memory intensive. If the images to be restored are raw, or in raw format files, ssh into the machine they're on, and try to mount them.

mount -t ntfs /home/sam/ntfs.img /mnt/windows -o loop, and see if the image is any good. If all this fails, come back and tell me about it.

Everyone wants everything the way it was before it got the way it is now.

Believe it or not, in the 1980s, it could more than an entire work day to install a hard drive in a PC, because there were so many parameters to configure, nothing was marked on the drive itself, and there was no WWW to Google, and if you didn't get through to the engineer who designed the drive, no one could tell you how to set the bios.

This is all calculated automatically by the bios now, but there are some people who would love it if we could get rid of everything that hides the truth from people, like anything over 64k of ram, any programming languages other than assembler, any disks over 10 MB, and some people just quit using PCs when you couldn't boot off a floppy, then take the boot disk out of the drive and put a program disk in so you could run your handwritten software without practicing voodoo with hard drives. After all, you can't open a hard drive and look at it, so anything could be inside. With a floppy, you can examine the entire thing, and if one got really suspicious the devil was involved, the floppy could be easily burned. Not so with a hard drive.

Those people thought there might be a demand for at least five computers, world wide, and with enough practice one could be manufactured that weighed under 5 tons. I kid you not. The chairman of IBM said, 'One day we'll make a computer that weighs under 5 tons'. The Chairman of Digital Equipment corporation said, "Consumers would never use a computer".

The first hard drive had (50) 2 foot in diameter platters, would run in the open air, but had a cover, with a capacity of 5.0 MB; 500 alpha/numeric characters per track * 100 tracks per cylinder = 50,000 characters per cylinder * 100 total cylinders = 5 MB total capacity. An 80 GB drive today would have around 10,000 cylinders/8.2 MB per ONE cylinder. These cylinders are virtual, but they still count.

A cylinder is all the tracks that line up exactly with each other, vertically. If each disk surface has 1024 tracks, and there 2 platters in the drive = 4 disk surfaces, the drive has 4 heads/1024 cylinders. Platter surface 1 contains tracks 0 - 1023. Platter surface two, the bottom of the same platter, contains tracks 1024-2047. Platter suface 3, the top of the second platter contains tracks 2048-3071. Platter surface 4, the bottom of the same platter contains tracks 3072-4095. Tracks 0, 1024, 2048, and 3072 are directly aligned vertically, to the micrometer, and form one cylinder.

Because the heads on a drive are all attached to each other, and when they swing across the platter surfaces they are always perfectly aligned with one another, vertically. This allows the drive to function, because if the heads moved independent to each other, the drive would never know where to go next using the head that is currently reading data.

If the heads are aligned perfectly, a slight calculation will allow the drive to locate data on another platter surface by the position of the head currently reading data. The ability to use drive geometry in the form heads/sectors per track/cylinders is an inescapable foundation of 32 bit PC architecture.