LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Linux - Newbie (http://www.linuxquestions.org/questions/linux-newbie-8/)
-   -   Learn The DD command (http://www.linuxquestions.org/questions/linux-newbie-8/learn-the-dd-command-366442/)

AwesomeMachine 09-23-2005 11:18 PM

Learn The DD command
 
This thread is the original incarnation of the much better, and maintained dd thread, here: http://www.linuxquestions.org/questi...ommand-362506/

PenguinPwrdBox 09-23-2005 11:30 PM

This - is without question - the most excellent thing I have seen in a long time - maybe ever.
Fantastic job.
I'll post this anywhere I can - but you will be the one getting the credit.

Charred 09-24-2005 01:59 AM

Awesome Machine,

Excellent work, I'm really impressed!

Where are you going?

AwesomeMachine 09-24-2005 12:22 PM

I am writing a book. I'm on page 282. I have to focus. If I get a little brilliant linux idea I will post it. I forgot one thing

I told you about hidden partitions, but I left something out. Every PC partition has to end with 0x55aa, that's 55aa hex. So if you want to find out if you have a hidden partition on your drive:

dd if=/dev/sda | hexdump -C | grep '55<space>aa'

this will find every instance of this hex string on the physical drive. This is called a magic signature. Since grep will give you the byte offset, you can use dd to go investigate if it seems like there are too many partitions.

AwesomeMachine 09-24-2005 12:23 PM

Please feel free to copy freely and redistribute.

shamrock_uk 09-24-2005 03:59 PM

Amazing - thank-you! I've been looking for something as clear as this for a while now!

sfrazier9999 04-30-2006 11:03 AM

I want to duplicate a drive. It is a 80 Gig drive and I have a 120 Gig drive. I wanted to duplicate the 80 GIG drive to the 120 Gig drive.
If I do the following:
dd if=/dev/hda of=/dev/hdb
hda = 80 Gig hdb = 120 Gig Drive
Will it set up the exactly partitions on the 120 Gig drive as it is on the 80 Gig drive?

How could I use the additional 40 Gigs without having to do a different partition? Could I some how expand the /var on the 120 Gig Drive to use the additional 120 Gig drive?
TIA

AwesomeMachine 04-30-2006 06:43 PM

Partitions, Drives
 
Quote:

Originally Posted by sfrazier9999
I want to duplicate a drive. It is a 80 Gig drive and I have a 120 Gig drive. I wanted to duplicate the 80 GIG drive to the 120 Gig drive. If I do the following: dd if=/dev/hda of=/dev/hdb hda = 80 Gig hdb = 120 Gig Drive Will it set up the exactly partitions on the 120 Gig drive as it is on the 80 Gig drive? How could I use the additional 40 Gigs without having to do a different partition? Could I some how expand the /var on the 120 Gig Drive to use the additional 120 Gig drive? TIA

Your starting out with an 80 GB drive as source. This drive has some partitions on it. If the drive is /dev/hda, then the partitions are: /dev/hda1 /dev/hda2 /dev/hda5 /dev/hda6 /dev/hdax So, here is some information on using dd with partitions. You have a table of primary and extended partitions in the Master Boot Record. There are four entries possible, which are a combination of up to four partitions. The hitcher is you can make an extended partition within the MBR, and put more logical partitions in that partition. Dd doesn't care, if you are copying partitions to other partitions, if you copy a primary partition to a logical partition, or vice versa. The only time this matters is with the first primary partition. You have to dd the first primary partition from the source to the first primary partition on the target. The bootable stuff has to go in the first primary partition on both drives. That said, you can copy any partition to any other partition, given enough space on the target. If you don't use the conv=notrunc option, adjacent sectors of all zeroes will be abbreviated with "*****". So, you can have fifty thousand sectors of zeroes on the source, and without notrunc this would take up five bytes on the target. If you want to do a one shot drive copy, all you need to do is make sure the drives have the same LBA geometry. Drives from 40 GB up to about 200 GB use 63 sectors per track 255 heads. But, if you copy a smaller drive to a larger one there will be some space left over. So, here is what you do with that: You partition the target before you use dd. Copy one partition at a time: dd if=/dev/hda1 of=/dev/hdb1 etc

AwesomeMachine 04-30-2006 06:54 PM

More stuff about DD
 
Note that sending a SIGUSR1 signal to a running 'dd' process makes it
print to standard error the number of records read and written so far,
then to resume copying.


$ dd if=/dev/zero of=/dev/null& pid=$!
$ kill -USR1 $pid; sleep 1; kill $pid

10899206+0 records in 10899206+0 records out




BLOCKS and BYTES may be followed by the following multiplicative suffixes: c 1, w 2, b 512, kB 1000, K 1024, MB 1000*1000, M 1024*1024, GB 1000*1000*1000, G 1024*1024*1024
So,

dd if=/dev/sda of=/dev/sdb bs=1GB
Will use one gigabyte block sizes.
bs=4b would give dd a block size of 4 disk sectors. 1 sector=512 bytes.
bs=4k would indicate dd use a 4 kilobyte block size. I have found bs=4k to be the fastest for copying disk drives on a modern machine.

OPERANDS The following operands are supported:
if=file Specifies the input path. Standard input is the default.
of=file Specifies the output path.
Standard output is the default.

seek=blocks Skip this many blocks in the output file.
ibs=n Specifies the input block size in n bytes (default is 512).
obs=n Specifies the output block size in n bytes (default is 512).
If no conversion other than sync, noerror, and, notrunc is specified, each input block is copied to the output as a single block without aggregating short blocks.
cbs=n Specifies the conversion block size for block and unblock in bytes by n (default is 0). If cbs= is omitted or given a value of 0, using block or unblock produces unspecified results. This option is used only if ASCII or EBCDIC conversion is specified.
ascii and asciib operands, the input is handled as described for the unblock operand except that characters are converted to ASCII before the trailing SPACE characters are deleted.
ebcdic, ebcdicb, ibm, and ibmboperands, the input is handled as described for the block operand except that the characters are converted to EBCDIC or IBM EBCDIC after the trailing SPACE characters are added. files=n Copies and concatenates n input files before terminating (makes sense only where input is a magnetic tape or similar device).
skip=n Skips n input blocks (using the specified input block size) before starting to copy. On seekable files, the implementation reads the blocks or seeks past them. On non-seekable files, the blocks are read and the data is discarded. iseek=n Seeks n blocks from beginning of input file before copying (appropriate for disk files, where skip can be incredibly slow).
oseek=n Seeks n blocks from beginning of output file before copying. seek=n Skips n blocks (using the specified output block size) from beginning of output file before copying. On non-seekable files, existing blocks are read and space from the current end-of-file to the specified offset, if any, is filled with null bytes. On seekable files, the implementation seeks to the specified offset or reads the blocks as described for non-seekable files.
count=n Copies only n input blocks.
conv=value[,value. . . ] Where values are comma-separated symbols from the following list:
conv=notrunc Tells dd not to abbreviate blocks of all zero value, or multiple adjacent blocks of zeroes, with five asterisks (when you want to maintain size) Do not use notrunc for copying a larger volume to a smaller volume. Without conv=notrunc you could have 30,000 adjacent blocks with nothing but zeroes written to them. Dd, by default, would abbreviate those 30,000 blocks *****. Then it doesn't have to write all those zeroes. But, when you copy, sometimes you want to maintain size, so then use notrunc.
ascii Converts EBCDIC to ASCII.
asciib Converts EBCDIC to ASCII using BSD-compatible character translations. ebcdic Converts ASCII to EBCDIC. If converting fixed-length ASCII records without NEWLINEs, sets up a pipeline with dd conv=unblock beforehand.
ebcdicb Converts ASCII to EBCDIC using BSD-compatible character translations. If converting fixed-length ASCII records without NEWLINEs, sets up a pipeline with
dd conv=unblock beforehand.
ibm Slightly different map of ASCII to EBCDIC. If converting fixed-length ASCII records without NEWLINEs, sets up a pipeline with dd dd conv=unblock beforehand.
I]ibmb[/I] [B][COLOR="DarkGreen"]Slightly different map of ASCII to EBCDIC using BSD-compatible character translations. If converting fixed-length ASCII records without NEWLINEs, sets up a pipeline with dd conv=unblock beforehand. The ascii (or asciib), ebcdic (or ebcdicb), and ibm (or ibmb) values are mutually exclusive. block Treats the input as a sequence of NEWLINE-terminated or EOF-terminated variable-length records independent of the input block boundaries. Each record is converted to a record with a fixed length specified by the conversion block size. Any NEWLINE character is removed from the input line. SPACE characters are appended to lines that are shorter than their conversion block size to fill the block. Lines that are longer than the conversion block size are truncated to the largest number of characters that will fit into that size. The number of truncated lines is reported. unblock Converts fixed-length records to variable length. Reads a number of bytes equal to the conversion block size (or the number of bytes remaining in the input, if less than the conversion block size), delete all trailing SPACE characters, and append a NEWLINE character. The block and unblock values are mutually exclusive.
lcase Maps upper-case characters specified by the LC_CTYPE keyword tolower to the corresponding lower-case character. Characters for which no mapping is specified are not modified by this conversion. ucase Maps lower-case characters specified by the LC_CTYPE keyword toupper to the corresponding upper-case character. Characters for which no mapping is specified are not modified by this conversion. The lcase and ucase symbols are mutually exclusive.
swab Swaps every pair of input bytes. If the current input record is an odd number of bytes, the last byte in the input record is ignored.
noerror Does not stop processing on an input error. When an input error occurs, a diagnostic message is written on standard error, followed by the current input and output block counts in the same format as used at completion. If the sync conversion is specified, the missing input is replaced with null bytes and processed normally. Otherwise, the input block will be omitted from the output. notrunc Does not truncate the output file. Preserves blocks in the output file not explicitly written by this invocation of dd. (See also the preceding of=file operand.) sync Pads every input block to the size of the ibs= buffer, appending null bytes. (If either block or unblock is also specified, appends SPACE characters, rather than null bytes.)

ENVIRONMENT VARIABLES

The following environment variables affect the messages and errors messages of dd:

LANG
Provide a default value for the internationalisation variables that are unset or null. If LANG is unset or null, the corresponding value from the implementation-dependent default locale will be used. If any of the internationalisation variables contains an invalid setting, the utility will behave as if none of the variables had been defined.
LC_ALL
If set to a non-empty string value, override the values of all the other internationalisation variables.
LC_CTYPE
Determine the locale for the interpretation of sequences of bytes of text data as characters (for example, single- as opposed to multi-byte characters in arguments and input files), the classification of characters as upper- or lower-case, and the mapping of characters from one case to the other.
LC_MESSAGES
Determine the locale that should be used to affect the format and contents of diagnostic messages written to standard error and informative messages written to standard output.
NLSPATH
Determine the location of message catalogues for the processing of LC_MESSAGES.


Please reply if you find this thread useful

mjolnir 05-14-2006 06:44 PM

PM me when the book is ready man I want one.

////// 05-23-2006 06:34 AM

Hi, I'm having a little problem when grepping my hd for email addresses.

This works just fine:
Code:

egrep -o -e "^(([A-Za-z0-9]+_+)|([A-Za-z0-9]+\-+)|([A-Za-z0-9]+\.+)|([A-Za-z0-9]+\++))*[A-Za-z0-9]+@((\w+\-+)|(\w+\.))*\w{1,63}\.[a-zA-Z]{2,6}$" /root/Desktop/mails
It finds all kinds of addresses:
name.surname@blah.com
a+b_c@d-e_f.gh
jim@jim.c.dc.ca
eee@e-e.com
eee@ee.eee.museum
joe.tillis@unit.army.mil
foo99@foo.co.uk
lazy@yahoo.com

But it doesnt find anything with when I use it with dd:
Code:

dd if=/dev/hda |hexdump -C |egrep -o -e "^(([A-Za-z0-9]+_+)|([A-Za-z0-9]+\-+)|([A-Za-z0-9]+\.+)|([A-Za-z0-9]+\++))*[A-Za-z0-9]+@((\w+\-+)|(\w+\.))*\w{1,63}\.[a-zA-Z]{2,6}$"
Thanks for great howto :)

MikeM-LQ 05-25-2006 08:12 AM

Very useful
 
But it looks all crammed together - is the formatting off or is it just me?

The second posting looks great - Its just the first one that looks like all the CR/LFs are gone.

KarlosDaJackel 05-25-2006 08:33 AM

Formating is off, but it put it in a text file on my desktop and fixed it myself.

Great doc. Congrats to you, and do let us know the title of the book when it comes out. I feel like buying a linux toolkit tricks type book.

AwesomeMachine 05-25-2006 11:59 PM

searching a HDD
 
To search an entire HDD for email addresses:

dd if=/dev/sda | hexdump -C | grep '@' > email.file

It won't be formatted in the file, so you will need take out all the clutter.

////// 05-30-2006 08:38 AM

Quote:

Originally Posted by AwesomeMachine
dd if=/dev/sda | hexdump -C | grep '@' > email.file

(very) Small correction:
Code:

dd if=/dev/sda | hexdump -C | grep -r2 '@' > email.file
with -r2 you get 2 lines above and below '@'
Code:

00000000  71 71 71 71 71 71 71 71  71 71 71 71 71 0a 71 71  |qqqqqqqqqqqqq.qq|
00000010  71 71 71 71 71 71 71 71  71 71 71 0a 71 71 71 71  |qqqqqqqqqqq.qqqq|
00000020  71 71 71 71 71 71 71 71  71 0a 66 61 6b 65 40 6d  |qqqqqqqqq.fake@m|
00000030  61 69 6c 2e 63 6f 6d 0a  71 71 71 71 71 71 71 71  |ail.com.qqqqqqqq|
00000040  71 71 71 71 71 0a 71 71  71 71 71 71 71 71 71 71  |qqqqq.qqqqqqqqqq|

with..
Code:

dd if=/dev/sda | hexdump -C | grep '@' > email.file
Code:

00000020  71 71 71 71 71 71 71 71  71 0a 66 61 6b 65 40 6d  |qqqqqqqqq.fake@m|
Cheers.


All times are GMT -5. The time now is 02:48 PM.