Learn the DD command

jschiwal · 03-09-2007, 05:31 AM

I haven't read all 15 pages of comments, so forgive me if this has already been mentioned. I would suggest changing the example where you erase a hard drive.
Instead of using /dev/random, use /dev/urandom. The random device collects entropy information for the random numbers. It's source of entropy will run out after 1 or 2 K. The urandom device will supply a large number of psuedo-random numbers.

AwesomeMachine · 03-09-2007, 05:33 AM

Quote:

Originally Posted by Quakeboy02

AwsomeMachine, add one specific example to your first page:

Copy the bootstrap code from the MBR and the first partition from sda to sdb without overwriting the partition table on sdb.

If I knew linux well enough I could make waffles with it.

dd if=/dev/sda bs=1 count=446 of /dev/sdb
dd if=/dev/sda1 skip=63 of=/dev/sdb1 seek=63 bs=1b

If sdb doesn't boot when you get done install grub from a rescue CD and it will. But, it should work with those two dd command lines. Let me know how it goes. I have faith in you. You can badmouth dd all you want. It won't care.

AwesomeMachine · 03-09-2007, 05:45 AM

Quote:

Originally Posted by jschiwal

I've read all 15 pages of comments. I would suggest changing the example where you wipe a hard drive. Instead of using /dev/random, use /dev/urandom. The random device collects entropy information for the random numbers. It's source of entropy will run out after 1 or 2 K. The urandom device will supply a large number of psuedo-random numbers.

When a user overwrites a hard drive it doesn't matter if a wide pattern can be established for the sequence used to overwrite the disk. If a user is making crytographic keys they would preferably use /dev/urandom to gather true randomness, but if /dev/urandom runs out of entropy it sits there and waits until it can gather more, whereas /dev/random will spit out the same sequence over and over. Trust me, /dev/urandom will never contain 250 GB of random bytes, and if the user leaves the machine it will take forever to generate enough truly pseudo-random bytes to wipe a drive.

But, your heart is in the right spot. I know you wanted to help this thread, so thanks.

jschiwal · 03-09-2007, 07:37 AM

I think you have it backwards. /dev/random will run out quickly, but /dev/urandom goes on like the energizer bunny.

You need to move the mouse or use the keyboard to continue using /dev/random.

From the "man urandom" man page

Code:

       When  read,  the  /dev/random  device will only return random bytes within the estimated number of
       bits of noise in the entropy pool.  /dev/random should be suitable for uses that  need  very  high
       quality  randomness such as one-time pad or key generation.  When the entropy pool is empty, reads
       from /dev/random will block until additional environmental noise is gathered.

       A read from the /dev/urandom device will not block waiting for more  entropy.   As  a  result,  if
       there is not sufficient entropy in the entropy pool, the returned values are theoretically vulner‐
       able to a cryptographic attack on the algorithms used by the driver.  Knowledge of how to do  this
       is  not  available in the current non-classified literature, but it is theoretically possible that
       such an attack may exist.  If this is a concern in your application, use /dev/random instead.

Sleepmore · 03-09-2007, 11:13 AM

Thanks for this thread, it has helped a lot. I just want to make sure I have everything correct. Here is my situation. I have 40GB drive which I want to backup onto a new 250GB drive. From what I have gleamed from the post all I need to do is:

dd if=/dev/hda of=/dev/hdb bs=4k conv=noerror

when the copy is finished, I:

resize2fs /dev/hdb

to resize the new drive to take up the rest of the new drive's space.

Now the questions:

1. First and foremost, is the commands I have above correct for making a copy of the old drive and resizing the new drive?

2. Do I need to place an entry into /etc/fstab for the new drive?

3. Do I need to mount the new drive?

3. Is there anyway to "see" the progress of the backup? Any command-line option?

Thanks for you help and I'm sorry if I'm the 10,000th person who has asked these questions.

Tinkster · 03-09-2007, 02:20 PM

Hi, and welcome to LQ!

Quote:

Originally Posted by Sleepmore

Thanks for this thread, it has helped a lot. I just want to make sure I have everything correct. Here is my situation. I have 40GB drive which I want to backup onto a new 250GB drive. From what I have gleamed from the post all I need to do is:

dd if=/dev/hda of=/dev/hdb bs=4k conv=noerror

when the copy is finished, I:

resize2fs /dev/hdb

to resize the new drive to take up the rest of the new drive's space.

Now the questions:

1. First and foremost, is the commands I have above correct for making a copy of the old drive and resizing the new drive?

No. You don't want to copy the raw-device but the
partitions; same with the resizefs - unless of course
you actually *didn't* partition the old drive, and
created a file-system on the raw-device.

Quote:

Originally Posted by Sleepmore

2. Do I need to place an entry into /etc/fstab for the new drive?

Not for dd to work.

Quote:

Originally Posted by Sleepmore

3. Do I need to mount the new drive?

Not for dd to work.

Quote:

Originally Posted by Sleepmore

3. Is there anyway to "see" the progress of the backup? Any command-line option?

No, not to the best of my knowledge.

Cheers,
Tink

Quakeboy02 · 03-09-2007, 02:27 PM

Posted by AwsomeMachine:

Quote:

If sdb doesn't boot when you get done install grub from a rescue CD and it will. But, it should work with those two dd command lines. Let me know how it goes. I have faith in you. You can badmouth dd all you want. It won't care.

AwsomeMachine,

I posted a howto on moving from IDE to SATA. I had hoped to use "dd" in penance, but the receiving partition is much larger than the origin partition, and I could only figure out how to do it using "cp -axu". If I understand dd correctly, when copying from one partition to another, the filesystem in receiving partition will be sized the same as that from the sending filesystem. If I've got that wrong, could you correct me? That's the part of "dd" that I'm just not sure of.

Added:
OH! I think that jschiwal in the following post has also answered my question: use resize2fs to match the copied filesystem to the partition it resides in.

Added again:
So, your example would be modified as follows? Or do I still have it wrong?

Code:

dd if=/dev/sda bs=1 count=446 of /dev/sdb
dd if=/dev/sda1 skip=63 of=/dev/sdb1 seek=63 bs=1b
resize2fs /dev/sdb1

jschiwal · 03-09-2007, 02:29 PM

Quote:

Originally Posted by Sleepmore

Thanks for this thread, it has helped a lot. I just want to make sure I have everything correct. Here is my situation. I have 40GB drive which I want to backup onto a new 250GB drive. From what I have gleamed from the post all I need to do is:

dd if=/dev/hda of=/dev/hdb bs=4k conv=noerror

when the copy is finished, I:

resize2fs /dev/hdb

to resize the new drive to take up the rest of the new drive's space.

Now the questions:

1. First and foremost, is the commands I have above correct for making a copy of the old drive and resizing the new drive?

2. Do I need to place an entry into /etc/fstab for the new drive?

3. Do I need to mount the new drive?

3. Is there anyway to "see" the progress of the backup? Any command-line option?

Thanks for you help and I'm sorry if I'm the 10,000th person who has asked these questions.

1. The dd command is correct. You need to change the size of the partition using the fdisk command. Then use resize2fs to increase the size of the filesystem to match. The resize2fs command is missing the size to resize to. Also, it will only work for ext2 or ext3 filesystems. It would probably be a better idea A) clone the old drive to the new drive. B) resize it using gparted or your distro's partitioner tool. Many have a resize option. When you copy /dev/hda to /dev/hdb, you will be copying the MBR as well which contains the partition table of the disk drive.

2. When you are done, you will need to check the contents of /etc/fstab. These are the partitions you want mounted at boot. Since you are replacing the drive, it probably will be /dev/hda after removing your old /dev/hda and replacing it with the new drive. You will need to configure the drive for the ide cable connection, e.g. master, auto, lone master.

3. Don't mount the new drive at first. You are writing on the device for the entire drive, /dev/hdb and not a partition on the drive. You mount a partition's filesystem and not a drive.

4. Read the man pages for dd and resize2fs and fdisk before you start. The dd manpage and the intial posting show how to send a USR1 signal to the process to have it print out the progress on stderr. If you have SuSE, or Mandriva, or others, you may be able to do all of these steps graphically using the respective distro's partitioner program.

It also might be a good idea to backup important data on the source drive first in case something goes wrong. Sometimes backing up /home and reinstalling will give you a fresh start. Especially if you have done several distro version upgrades. A fresh install might work better. Some features of xgl for example worked after I did a fresh install versus the previous upgrade. Plus, installing an OS fresh is kind of fun!

AwesomeMachine · 03-09-2007, 03:38 PM

Quote:

Originally Posted by jschiwal

I think you have it backwards. /dev/random will run out quickly, but /dev/urandom goes on like the energizer bunny.

You need to move the mouse or use the keyboard to continue using /dev/random.

From the "man urandom" man page

Code:

       When  read,  the  /dev/random  device will only return random bytes within the estimated number of
       bits of noise in the entropy pool.  /dev/random should be suitable for uses that  need  very  high
       quality  randomness such as one-time pad or key generation.  When the entropy pool is empty, reads
       from /dev/random will block until additional environmental noise is gathered.

       A read from the /dev/urandom device will not block waiting for more  entropy.   As  a  result,  if
       there is not sufficient entropy in the entropy pool, the returned values are theoretically vulner‐
       able to a cryptographic attack on the algorithms used by the driver.  Knowledge of how to do  this
       is  not  available in the current non-classified literature, but it is theoretically possible that
       such an attack may exist.  If this is a concern in your application, use /dev/random instead.

Instead of using /dev/random to generate random bytes, use /dev/urandom for the above reasons. I will change the original post.

iframe · 03-10-2007, 03:29 AM

AwsomeMachine:
Thanks for your advise.

Documenting what I did: The output of

dd if=/dev/zero of=/dev/sdb bs=1024 count=256000

was:

dd: writing `/dev/sdb': No space left on device
251905+0 records in
251904+0 records out
257949696 bytes (258 MB) copied, 67.7849 seconds, 3.8 MB/s

and built the file system with:

mkdosfs -I -F 32 -n KINGSTON -v /dev/sdb

and the output was:

mkdosfs 2.11 (12 Mar 2005)
/dev/sdb has 8 heads and 62 sectors per track,
logical sector size is 512,
using 0xf8 media descriptor, with 503808 sectors;
file system has 2 32-bit FATs and 1 sector per cluster.
FAT size is 3876 sectors, and provides 496024 clusters.
Volume ID is 45f27263, volume label KINGSTON .

Tortanick · 03-10-2007, 03:43 PM

just out of curiosoty dose dd create byte for byte perfect copies? bit for bit copies? imperfect copies?

AwesomeMachine · 03-10-2007, 04:58 PM

Dd is a bitstream duplicator, so it copies bit for bit. It has been tested by The United States Department of Justice computer forensics lab and found to be an exact duplicator.

Tortanick · 03-11-2007, 03:10 AM

thank you

Sleepmore · 03-13-2007, 02:17 PM

To recap, I'm attempting to replace a 40GB drive with a new 150GB drive. I want the remaining space on the new drive to be given to the / partition.

I did:

dd if=/dev/hda of=/dev/hdb bs=4k conv=noerror

it seemed to work. I did a

fdisk -l /dev/hda

and compared it to

fdisk -l /dev/hdb

and the results are the same. Here is the output from the list of hdb:

/dev/hdb1 * 1 6 486136+ 83 Linux
/dev/hdb2 7 260 2040255 82 Linux swap
/dev/hdb3 261 4865 36989662+ 83 Linux

I then tried (because it looks like the / partition, which is the rest of the disk):

resize2fs /dev/hdb3

and got:

resize2fs: Filesystem has unsupported feature(s) (/dev/hdb3)

Here is my /etc/fstab (just in case it's an issue, I have only included line pertaining to the hard drive):

LABEL=/ / ext3 defaults 1 1
LABEL=/boot /boot ext3 defaults 1 2
/dev/hda2 swap swap defaults 0 0

My questions:

1.) Did I need to format the new drive before I did the dd? I don't think I need to do this, but just checking. If I do need to "format" the new drive, what command do I use?

2.) What does the error I got from resize2fs mean? I have done the "goggle" search and the responses are unclear.

Thanks for your help.

AwesomeMachine · 03-19-2007, 09:02 PM

From the end of the thread look backwards until you get to the part where I discuss moving smaller partitions to larger ones. You do not need to format before dd'ing. If you want to keep the same system, partition the new drive, format it, and use cp -r * from the root directory of the old drive to the root of the new drive. Then, use the linux install CD in rescue mode, and reinstall the bootloader to the MBR.