Learn the DD command

arjaya · 11-20-2005, 05:10 AM

Thank you for your help! It is much appreciated!

Emmanuel_uk · 11-28-2005, 07:06 AM

AwesomeMachine, what happens if one copies a given partition onto a smaller partition? I was going to test this because I want to backup a partition which is far from full onto a smallish partition that I know is big enough to take all the data.
dd if=/dev/hda5 (30gb but only 3 gb of data) of=/dev/hdb14 (5gb of space)
Will this work? Will dd spit an error and stop when the destination is full? (wihthout trying to write to the next partition for example).

I'd rather know beforehand than being sorry

AwesomeMachine · 11-29-2005, 01:38 PM

Quote:

Originally posted by Emmanuel_uk
AwesomeMachine, what happens if one copies a given partition onto a smaller partition? I was going to test this because I want to backup a partition which is far from full onto a smallish partition that I know is big enough to take all the data.
dd if=/dev/hda5 (30gb but only 3 gb of data) of=/dev/hdb14 (5gb of space)
Will this work? Will dd spit an error and stop when the destination is full? (wihthout trying to write to the next partition for example).

I'd rather know beforehand than being sorry

The only difference between a big partition and a small partition, besides size, is the partition table. If you are copying say sda to sdb, an entire drive with a single partition, sdb being smaller than sda, then you have to do:

dd if=/dev/sda skip=1 of=/dev/sdb seek=1 bs=4k conv=noerror

You can put the skip and seek anywhere you want, I just put them there to show what they go to. Skip skips input blocks at the beginning of the media(sda). Seek skips over so many blocks on the output media before writing(sdb). By doing this, you leave the first 4k bytes on each drive the same. This is eight sectors. The first sector is the Master Boot Record (MBR). This contains the partition table. You don't want to tell a drive it is bigger than it really is by writing a partition table from a larger drive to a smaller drive. In my example, we wanted to use a byte size (bs) of 4k or 4096 bytes. This speeds up the copy from the default 512 byte bs. But, your are limited in how small of a skip or seek you can make. In this example, the smallest skip or seek is 1, or 4096 bytes. Since the first 63 sectors of the drives are empty, except sector 1, the MBR, we can skip eight sectors without any problem missing part of the boot sector. One sector on a hard drive is 512 bytes.

Now, if you are copying say sda3 to sda2, this is different. What you want to do is this:

dd if=/dev/sda3 of=/dev/sda2 bs=4096 conv=noerror

Even if these are primary partitions, they are still logical to the main partition. Do not use the conv=notrunc option. Without notrunc, multiple blocks of zeros get abbreviated with a string of asterisks, saving a lot of space. if you use notrunc, all the zeros will be written out. On a smaller partition you want to save room. If you write out all the zeros, the smaller partition won't be big enough.

Emmanuel_uk · 11-30-2005, 04:20 PM

Thanks. This was eductional. And my little test went half ok

As expected

Code:

dd: writing `/dev/hdb15': No space left on device
3957708+0 records in
3957707+0 records out

This was from ext3 to ext3, different size partitions (source 12Gb onto 4 Gb destination, on purpose,
although I want to test 12 Gb onto 14 Gb as well)

Fdisk says the destination was 15830829 blocks
3957708 * 4=15830832
So about ok

#ls -l
ls: blablafolder: Input/output error
some files and folder are visible are catable good

Filesystem Size Used Avail Use% Mounted on
/dev/hdb15 51G 12G 39G 24% /mnt/toto
Look really wrong because hdb15 is only 4 Gb/ So it took the info from the source directory

Not good

Code:

# e2fsck -f /dev/hdb15
e2fsck 1.36 (05-Feb-2005)
/dev/hdb15: recovering journal
e2fsck: unable to set superblock flags on /dev/hdb15

I went reading a bit more about superblocks
http://www.win.tue.nl/~aeb/linux/lk/lk-7.html
I am afraid for now, I need to read more about debug2fs etc.

arjaya · 11-30-2005, 09:17 PM

Hi,

I'm trying to extract the contents of a 4MM tape using dd.

The tape contains backup data from a novell netware 6.5 server.

Can I use dd to extract the contents of the tape?

I've tried using this command

dd if=/dev/st0 of=tapedata

But LINUX responds

dd: /dev/st0: Input/output error
0+0 records in
0+0 records out

I checked that all was well with the tape drive by successfully extracting
data from another tape containing linux backup data.

We are using Fedora Core 4, and an internal 4MM tape.

I have tried numerous block sizes to extract the data from this tape. I have also tried the command:

Code:

mt -f /dev/st0 setblk 0

Is there any other way to determine the block size of the tape?

Can anyone point me in the right direction on what I am doing wrong?

Any help is much appreciated!!!

Here are some details on the scsi st0 device.

Code:

[root@localhost /]# dmesg | grep scsi
scsi0 : Adaptec AIC7XXX EISA/VLB/PCI SCSI HBA DRIVER, Rev 6.2.36
scsi1 : Adaptec AIC7XXX EISA/VLB/PCI SCSI HBA DRIVER, Rev 6.2.36
scsi1:A:0:0: Tagged Queuing enabled.  Depth 4
(scsi1:A:0): 6.600MB/s transfers (16bit)
(scsi1:A:0): 160.000MB/s transfers (80.000MHz DT, offset 127, 16bit)
Attached scsi disk sda at scsi1, channel 0, id 0, lun 0
scsi1:A:1:0: Tagged Queuing enabled.  Depth 4
(scsi1:A:1): 6.600MB/s transfers (16bit)
(scsi1:A:1): 160.000MB/s transfers (80.000MHz DT, offset 127, 16bit)
Attached scsi disk sdb at scsi1, channel 0, id 1, lun 0
(scsi1:A:6): 6.600MB/s transfers (16bit)
(scsi1:A:6): 80.000MB/s transfers (40.000MHz, offset 32, 16bit)
scsi2 : Adaptec AIC7XXX EISA/VLB/PCI SCSI HBA DRIVER, Rev 6.2.36
Attached scsi tape st0 at scsi1, channel 0, id 6, lun 0
[root@localhost /]#
[root@localhost /]#
[root@localhost /]# dmesg | grep st0
Attached scsi tape st0 at scsi1, channel 0, id 6, lun 0
st0: try direct i/o: yes (alignment 512 B), max page reachable by HBA 1048575
[root@localhost /]#

Code:

[root@localhost /]# mt -f /dev/st0 status
SCSI 2 tape drive:
File number=0, block number=0, partition=0.
Tape block size 512 bytes. Density code 0x26 (DDS-4 or QIC-4GB).
Soft error count since last status=0
General status bits on (41010000):
 BOT ONLINE IM_REP_EN
[root@localhost /]#

Code:

[root@localhost /]# dd if=/dev/st0 of=/st0.img
dd: reading `/dev/st0': Input/output error
0+0 records in
0+0 records out
[root@localhost /]#

Code:

Nov 22 14:44:57 localhost kernel: st0: Block limits 1 - 16777215 bytes.
Nov 22 14:47:17 localhost kernel: st0: Incorrect block size.
[root@localhost log]#

AwesomeMachine · 12-02-2005, 12:31 AM

This seems to indicate one of two things:

When dd gives an i/o error, it is always because of the device being read from, in this case, /dev/st0. What you probably have is an unreadable tape.

Try this:

dd if=/dev/st0 of=tape.img bs=1b conv=notrunc,noerror

If this fails, try looking toward the end of the post for a link to dd_rescue. dd_rescue can start at the end of the tape, and restore backwards. You can also try "skip=1" to skip the first sector of the tape. If the tape is bad, at least the first sector is bad because you are getting 0 + 0 records. That means the tape fails before sector 1. If this fails, Sanderson Forensics sells a utility to get data off damaged tape media. Once you buy it, you can just download it. I've never used it, but I have used other programs made by Paul Sanderson and they are good.

I have also heard, but not verified, that dd has trouble with QIC-80 drives.

microsoft/linux · 12-25-2005, 02:07 PM

I'd like to back up my root partition, so as to be able to resize the partition, and I want to make sure I'd be doing this right before I mess anything up.

I'd need to run

Code:

dd if=/dev/hda3 of=/backup<an HD mounted across a network> bs=4096 conv=notrunc,noerror

to backup my root
and

Code:

dd if=/backup skip=1 of=/dev/hda3<being the new, larger, partition> seek=1 bs=4k conv=noerror

to put the backed up / on the new partition.

is this right?

AwesomeMachine · 12-27-2005, 04:09 AM

Quote:

Originally Posted by microsoft/linux

I'd like to back up my root partition, so as to be able to resize the partition, and I want to make sure I'd be doing this right before I mess anything up.

I'd need to run

Code:

dd if=/dev/hda3 of=/backup<an HD mounted across a network> bs=4096 conv=notrunc,noerror

to backup my root
and

Code:

dd if=/backup skip=1 of=/dev/hda3<being the new, larger, partition> seek=1 bs=4k conv=noerror

to put the backed up / on the new partition.

is this right?

If you are going across a network, there are instructions in my original post for using netcat with dd. Just go back to the first page and do a search for "netcat" with you browser. I don't think dd works with NFS mount points. With netcat you can specify the actual device file on the remote backup machine. Also, you can back up root to a file, and even mount the file as a file system, which is a good test to see if the backup file is any good, on the remote machine. If you want the new partition smaller, don't use notrunc. Notrunc copies all the zeroes in empty sectors. Without notrunc, sections of the disk which are solid zeroes are abbreviated with a string of asterisks. So if you have 3000 sectors in a row, which are all blank, without notrunc, all 3000 sectors only consume 10 bytes. Any sector that has anything in it besides zeroes will not be truncated. I use notrunc for copying complete drives where everything has to end up in the right spot. For a single partition you don't need notrunc. If you want to see how much is on the root partition now do a <df -h>. Then you'll have some idea of the actual space needed. I really appreciate these replies because they give me all kinds of new ideas.

microsoft/linux · 12-28-2005, 09:59 PM

So in order to back up my / partition to a NFS mounted HD I'd have to do

on my laptop(whose root would be backed up)

Code:

dd if=/dev/hda3 bs=16065b | netcat <ip goes here> <port goes here>

where the ip is that of the remote machine?

Code:

netcat -l -p <port goes here> | dd of=/dev/hdb1(the HD I want the backup to go to) bs=16065b

is this correct? Would I just reverse the process to copy it back?

AwesomeMachine · 12-29-2005, 12:15 AM

Yes, that is how you do it. The "bs=16065b" is an example of how to specify one cylinder per block. In reality, smaller bs values, such as 2048 or 2k, 4096 or 4k, 8192 or 8k are good for network backups. In fstab, make sure the NFS drives specify "rsize=8192,wsize=8192". This will make the backup work better.

microsoft/linux · 12-29-2005, 12:30 AM

what does all of that mean? A block is just a section of an HD right? Why do I need to be worried about the blocks?

AwesomeMachine · 01-08-2006, 04:15 AM

Quote:

Originally Posted by microsoft/linux

what does all of that mean? A block is just a section of an HD right? Why do I need to be worried about the blocks?

Optimal block size, specified in the dd command by bs= gives optimal transfer rates. On a machine with even a minimal amount of ram, dd's default bs=512 is quite slow. 512 is one sector at a time. I have found this to be slow. I prefer larger block sizes to speed transfers. These larger block sizes, specified with bs=, enable less read/writes, because the dd program reads and writes larger chuncks of data. Of course, too big of a block size will slow things down because flushing the dd buffers does take some time.

Lotharster · 01-08-2006, 08:07 AM

Quote:

Originally Posted by AwesomeMachine

With urandom it is possible to read a byte size of 100. With /dev/random you need to do bs=1. urandom does not have this restriction.

This is not correct. Both random and urandom can in theory produce output of any byte size. They use the kernel's entropy pool to compute the random numbers. This entropy pool is filled with random noise gathered from device drivers and other sources. The important difference between the two is that /dev/random produces only as many random bits as the entropy pool contains. This results in high quality randomness, which is neccessary for kryptographic keys. In case the user demands more random bytes than currently contained in the entropy pool, the process stops until the entropy pool is refilled (waggling your mouse helps). /dev/urandom, however, does not have this restriction. If the user demands more bits than currently in the entropy pool, it produces them using a pseudo random number generator. These pseudo random numbers should not be used as cryptographic keys.[/quote]

jon_flanders · 01-08-2006, 08:02 PM

I would like to clone my 13 gb windows 98 drive, divided into c: programs and d: data to a 100 gb drive via usb which I would pre-partition to say, c: 20gb , d: 30 gb.

The goal would be to then simply pop the 100 gb drive into the ide ribbon in place of the 13 gb drive. My Linux drive is currently 30gb which is sufficient for my purposes. Here's the output of df currently:

[jon@localhost ~]$ df
Filesystem Size Used Avail Use% Mounted on
/dev/hdb5 4.9G 3.6G 1.1G 78% /
/dev/hdb3 5.7G 3.3G 2.3G 60% /home/data
/dev/hda1 5.8G 5.1G 663M 89% /mnt/win_c
/dev/hda5 6.9G 4.2G 2.8G 60% /mnt/win_d
[jon@localhost ~]$

I have unmounted partitions and unused space on the 30gb drive.

Is this doable? I know I can partition the hard drive in the usb via Mandriva's Harddrake. I've done that already with an old 3gb drive.

Jon Flanders

tonnnn · 01-10-2006, 02:52 AM

greate tutorial !!!