Learn The DD Command Revised
This post contains comprehensive documentation with examples for one of the most useful Linux/UNIX/Windows commands: dd. Dd is a bit-stream duplicator. If you have questions, post them. The latest addition, How To Encrypt an 8.0 GB SDHC MicroSD Card was on 06-19-2011.
First Time visitors please reply. How To Encrypt an 8.0 GB SDHC MicroSD Card Put the card into an USB adapter. Such devices are not perfect. One might have to push the MicroSD card into the reader as far as it will go, and others might have to pull it back a millimeter or two. If the kernel does not detect a partition on a new card, it's detecting the USB adapter only. Adjust the card slighty, and replace the adapter if necessary. Should show some new device(s): Code:
ls /dev/sd* Code:
dd if=/dev/urandom of=/dev/sdb bs=4k Code:
apt-get install cryptsetup Code:
parted Code:
cryptsetup --verbose --verify-passphrase luksFormat /dev/sdb1 Code:
cryptsetup luksOpen /dev/sdb1 vol_1 Code:
mkfs.xfs -imaxpct=3 /dev/mapper/vol_1 Code:
mkdir /AES_Drive Code:
umount /AES_Drive && cryptsetup luksClose /dev/mapper/vol_1 Linux DD The basic command structure is as follows: Code:
dd if=<source> of=<target> bs=<byte size> ("USUALLY" some power of 2, and usually not less than 512 bytes (ie, 512, 1024, 2048, 4096, 8192, 16384, but can be any reasonable whole integer value.) skip= seek= conv=<conversion> Warning!! If you reverse the source and target, you can wipe out a lot of data. This feature has inspired the nickname "dd" Data Destroyer. Warning!! Caution should be observed when using dd to duplicate encrypted partitions. Examples: duplicate one hard disk partition to another hard disk partition: Sda2 and sdb2 are partitions. You want to duplicate sda2 to sdb2. Code:
dd if=/dev/sda2 of=/dev/sdb2 bs=4096 conv=notrunc,noerror Code:
rsync To make an iso image of a CD: This duplicates sector for sector. MyCD.iso will be a hard disk image file of the CD. Code:
dd if=/dev/hdc of=/home/sam/myCD.iso bs=2048 conv=sync,notrunc Code:
mkdir /mnt/myCD Code:
cd /mnt/myCD Code:
dd if=/dev/fd0 of=/home/sam/floppy.image Code:
#!/bin/bash Code:
chmod +x anti_swqid Best Laptop Backup: Purchase a laptop drive and an USB 2.0 drive enclosure (Total cost $100.00USD). Assemble the lappy drive into the external enclosure. Plug the external drive into the lappy USB port, and boot with The Knoppix live CD. Launch a terminal. This command will backup the existing drive: Code:
dd if=/dev/hda of=/dev/sda bs=64k conv=notrunc,noerror Code:
dd if=/dev/sda of=/dev/hda bs=64k conv=notrunc,noerror This series will make a DVD backup of hard drive partition: Code:
dd if=/dev/hda3 of=/home/sam/backup_set_1.img bs=1M count=4430 Code:
wodim -dev=/dev/hdc --driveropts=burnfree /home/sam/backup_set_1.img Code:
dd if=/media/dvd/backup_set_1.img of=/dev/hda3 bs=1M conv=sync,noerror Code:
dd if=/media/dvd/backup_set_2.img of=/dev/hda3 seek=4430 bs=1M conv=sync,noerror Code:
dd if=/media/dvd/backup_set_3.img of=/dev/hda3 seek=8860 bs=1M conv=sync,noerror If you wrote chat messages and emails to another girl, on your girlfriend's computer, you can't be sure the files you deleted are unrecoverable. But you can make sure if anyone were to recover them, that you wouldn't get busted. Code:
dd if=/dev/sda | sed 's/Wendy/Janet/g' | dd of=/dev/sda This command will overwrite the drive with zeroes Code:
dd if=/dev/zero of=/dev/sda bs=4k conv=notrunc Code:
dd if=/dev/sda | hexdump -C | grep [^00] The following method of ouputting statistics applies to any dd command invocation. This is an example dd command so you can try it. Code:
/bin/dd if=/dev/zero of=/dev/null count=100MB Code:
ps aux | awk '/bin\/dd/ && !/awk/ {print $2}' | xargs kill -s USR1 $1 Code:
33706002+0 records in Code:
58596452+0 records in Code:
74473760+0 records in Code:
100000000+0 records in Code:
dd if=/home/sam/file.file | clamscan - FYI: duplicating smaller partition or drive to larger partition or drive; or vice versa: Code:
rsync -avH --exclude=/other_mount_point/ /mount_point/* /other_mount_point/ Code:
rsync -avH --exclude=/mnt/destination_drive/ /* /mnt/destination_drive/ Code:
grub-install Making a NTFS partition, is not easy without using Windows based tools. I was formatting an external drive for my brother, who uses MS Windows XP. I wasn't going to admit Linux couldn't make a NTFS partition. Make an ext3 partition on the drive. Open a hex editor and make a file containing Code:
07 Code:
dd if=/home/sam/file.bin of=/dev/sdb bs=1 seek=450 count=1 Code:
0x1c2 Code:
0x83 Code:
0x07 The four primary partition type byte offsets are: Code:
0x1c2=450 To be revised at a later date: To make a bootable flash drive: Download 50 MB Debian based distro here: http://sourceforge.net/projects/insert/ Plug in the thumb drive into a USB port. Do: Code:
dmesg | tail Code:
dd if=/home/sam/insert.iso of=/dev/sdb ibs=4b obs=1b conv=notrunc,noerror End to be revised This command will duplicate the MBR and boot sector of a floppy disk to hard drive image: Code:
dd if=/dev/fd0 of=/home/sam/MBRboot.image bs=512 count=2 Code:
dd if=/dev/sda of=/dev/sdb bs=4096 conv=notrunc,noerror Duplicate MBR, but not partition table. This will duplicate the first 446 bytes of the hard drive to a file: Code:
dd if=/dev/sda of=/home/sam/MBR.image bs=446 count=1 To wipe a hard drive: (Boot from a live CD distro to do this.) Code:
dd if=/dev/zero of=/dev/sda conv=notrunc To overwrite all the free disk space on a partition (deleted files you don't want recovered): Code:
dd if=/dev/urandom of=/home/sam/bigfile.file Code:
no room left on device Code:
rm bigfile.file Sometimes one wants to look inside a binary file, looking only for clues. The output of the command line: Code:
less /home/sam/file.bin Code:
dd if=/home/sam/file.bin | hexdump -C | less Code:
dd if=/home/sam/file.file | strings -n 8 -t d | less Code:
dd if=/home/sam/JPEG.jpg bs=1w count=2 | hexdump -C Code:
dd if=JPEG.jpg | hexdump -C Code:
dd if=/dev/sda3 | hexdump -C | "grep 'ff d8 ff e0' | 'ff d9'" Code:
grep Code:
0xba0002f Code:
0xbaff02a We need to use skip= parameter to skip to our start point: 195 035 183 / bs=4096 = 47 616.011. We always round down, so dd will start reading before the beginning of the file. In this case we round down to skip=47615. The following writes a file containing the JPEG with some unwanted bytes before and after. Code:
dd if=/dev/sda3 skip=47615 of=/home/sam/work_file.bin count=257 bs=4096 Code:
hexdump -C work_file.bin | "grep 'ff d8 ff e0' | 'ff d9'" The principle of file carving negates the need for Linux undelete programs. So if your from a MS Windows world, don't google for linux undelete, but rather, foremost NEXT ... I put two identical drives in every one of my machines. Before I do anything that most probably spells disaster, like an untested command line in a root shell, that contains Code:
find / -regex ?*.???* -type f | xargs rm -f "$1" Code:
dcfldd if=/dev/sda of=/dev/sdb bs=4096 conv=notrunc,noerror Code:
dd if=/dev/sdb of=/dev/sda bs=4096 conv=notrunc,noerror Code:
dd if=/dev/sda2 (root) of=/home/sam/root.img bs=4096 conv=notrunc,noerror Code:
dd if /home/sam/root.img of=/dev/sda2 (root) bs=4096 conv=notrunc,noerror How to make a swap file, or another swapfile on a running system: Code:
dd if=/dev/zero of=/swapspace bs=4k count=250000 How to pick proper block size: Code:
dd if=/dev/zero bs=1024 count=1000000 of=/home/sam/1Gb.file Read: Code:
dd if=/home/sam/1Gb.file bs=64k | dd of=/dev/null Code:
dd if=/dev/zero bs=1024 count=1000000 of=/home/sam/1Gb.file Play with 'bs=' and 'count=', always having them multiply out to the same toal size. You can calculate bytes/second like this: 1Gb/total seconds = Gb/s. You can get more realistic results using a 3Gb file. Rejuvenate a hard drive To cure input/output errors experienced when using dd. Over time the data on a drive, especially a drive that hasn't been used for a year or two, grows into larger magnetic flux points than were originally recorded. It becomes more difficult for the drive heads to decipher these magnetic flux points. This results in I/O errors. Sometimes sector 1 goes bad, resulting in a useless drive. Try: Code:
dd if=/dev/sda of=/dev/sda Make a file of 100 random bytes: Code:
dd if=/dev/urandom of=/home/sam/myrandom bs=100 count=1 Randomize data over a file before deleting it: Code:
ls -l In this case it is 3769 Code:
ls -l afile -rw------- ... 3769 Nov 2 13:41 <filename> Code:
dd if=/dev/urandom of=afile bs=3769 count=1 conv=notrunc Warning!! Do not write a partition image file to the same partition. Code:
dd if=/dev/sdb2 of=/home/sam/partition.image bs=4096 conv=notrunc,noerror Code:
dd if=/dev/sdb2 ibs=4096 | gzip > partition.image.gz conv=noerror Code:
dd if=partition.image.gz | gunzip | dd of=/dev/sdb2 Code:
< filename >.bz2 Code:
dd if=/home/sam/partition.image of=/dev/sdb2 bs=4096 conv=notrunc,noerror Code:
dd if=filename of=filename conv=ucase The Linux kernel makes a number a ramdisks you can make into ramdrives. You have to populate the drive with zeroes like so: Code:
dd if=/dev/zero of=/dev/ram7 bs=1k count=16384 Code:
mke2fs -m0 /dev/ram7 4096 Code:
debian:/home/sam # hdparm -t /dev/ram7 Code:
mkdir /mnt/mem Duplicate ram memory to a file: Code:
dd if=/dev/mem of=/home/sam/mem.bin bs=1024 Code:
/dev/mem Dd will print to the terminal window if you omit the Code:
of=/dev/output Code:
dd if=/home/sam/myfile To search the system memory: Code:
dd if=/dev/mem | strings | grep 'some-string-of-words-in-the-file-you-forgot-to-save-before-the-power-failed' Code:
mkdir /mnt/mem You can get arrested in 17 states for doing this next thing. Make an AES encrypted loop device: Code:
dd if=/dev/urandom of=/home/sam/aes-drv bs=16065b count=100 Code:
umount /aes If you are curious about what might be on you disk drive, or what an MBR looks like, or maybe what is at the very end of your disk: Code:
dd if=/dev/sda count=1 | hexdump -C To see the end of the disk you have to know the total number of sectors, and the MAS must be set equal to the MNA. The helix CD has a utility to set this correctly. In the dd command, your skip value will be one less than MNA of the disk. For a 120 GB Seagate SATA drives Code:
dd if=/dev/sda of=home/sam/myfile skip=234441646 bs=512 So this reads sector for sector, and writes the last sector to myfile. Even with LBA addressing, disks still secretly are read in sectors, cylinders, and heads. There are 63 sectors per track, and 255 heads per cylinder. There is a total cylinder count. 512_bytes/sector*63_sectors/track*255heads=16065*512bytes/cylinder=8,225,280_bytes/cylinder. 63_sectors/track*255_heads=sectors/cylinder. With 234441647 total sectors, and 16065 sectors per cylinder, you get some trailing sectors which do not make up an entire cylinder: 14593.317584812_cylinders/drive. This leaves 5102 sectors which cannot be partitioned, because to be in a partition you have to be a whole cylinder. It's like having part of a person. That doesn't really count as a person. These become surplus sectors after the last partition. You can't ordinarily read past the last partition. But dd can. It's a good idea to check for anything writing to surplus sectors. For our Seagate 120 GB drive, 234,441,647_sectors/drive - 5102_surplus_sectors = 234,436,545 partitionable sectors. Code:
dd if=/dev/sda of=/home/sam/myfile skip=234436545 Code:
dd if=/dev/urandom of=/dev/sda bs=512 seek=234436545 Block size: One cylinder in LBA mode = 255_heads*63_sectors/track=16065_sectors=16065*512_bytes=8,225,280_bytes. The b means '* 512'. 32130b represents a two cylinder block size. Cylinder block size always works to cover every sector in a partition, because partitions are made of a whole number of cylinders. One cylinder is 8,225,280 bytes. If you want to check out some random area of the disk: Code:
dd if=/dev/sda of=/home/sam/myfile bs=4096 skip=2000 count=1000 Code:
dd if=/home/sam/myfile of=/dev/sda bs=4096 seek=2000 count=1000 On source machine: Code:
dd if=/dev/hda bs=16065b | netcat < targethost-IP > 1234 Code:
netcat -l -p 1234 | dd of=/dev/hdc bs=16065b Code:
netcat -l -p 1234 | bzip2 > partition.img Code:
netcat -l -p 1234 | gzip > partition.img Alert!! Don't hit enter yet. Hit enter on the target machine. THEN hit enter on the source machine. Netcat is a program, available by default, on most linux installations. It's a networking swiss army knife. In the preceding example, netcat and dd are piped to one another. One of the functions of the linux kernel is to make pipes. The pipe character looks like two little lines on top of one another, both vertical. Here is how this command behaves: This byte size is a cylinder. bs=16065b equals one cylinder on an LBA drive. The dd command is piped to netcat, which takes as its arguments the IP address of the target(like 192.168.0.1, or any IP address with an open port) and what port you want to use (1234). You can also use ssh. Code:
dd if=/dev/sdb2 | ssh sam@192.168.0.121 "sudo dd of=/home/sam/sdb2.img" Dd is like Symantec Norton Ghost, Acronis True Image, Symantec Drive Image. You can perform disk drive backup, restore, imaging, disk image, cloning, clone, drive cloning, transfer image, transfer data, clone to another drive or clone to another machine, move Windows XP to a new hard drive, clone Windows XP, clone Windows, transfer Windows, hard drive upgrade, duplicate a boot drive, duplicate a bootable drive, upgrade your operating system hard drive, Tired of reinstalling WinXP Windows XP? Copyright 2008, 2010 by AwesomeMachine. All Rights Reserved. |
How to rescue a CD or DVD, HDD, flash drive or photos:
Code:
ddrescue -b 2048 -d -e 20 -r 300 -v /dev/hdc /home/sam/dvd.iso log.txt Code:
foremost -t all -k 256 -v -b 2048 -i ~/dvd.iso -o ~/dvd/ MS Windows Section Use dd for drive cloning, backup, drive upgrading, and restore tasks. Boot a Windows XP machine with a Knoppix Linux live CD. Download Knoppix, burn the iso image file to a CD, boot with it, and clone drives. Drives are described to the dd command using device files. Boot into Knoppix, and open a root shell. It's in the penguin menu. (80 conductor grey ribbon cable) = /dev/hda for master; /dev/hdb for slave. SATA are /dev/sda and /dev/sdb. The partitions on the first drive: Code:
fdisk -l /dev/hda Code:
man fdisk Code:
man parted Many games written for Windows are virtually impossible back-up. This leaves only the original media. If it gets damaged, the user SOL. But there is a way to make back-ups using Linux. Download Adrienne Knoppix, and choose the option to copy it to a flash drive. Boot the flash-drive. If there are two optical drives available, load the CD/DVD source disk in one drive, and the blank disk in the other. Open a terminal from the menu, and get the root account by typing Code:
su Code:
ls /dev/hd* Code:
ls /dev/sr* Code:
wodim dev=/dev/hdc fs=16m speed=8 -raw96r driveropts=noburnfree,hidecdr -dummy -overburn -copy /dev/hdc/ Feel free to read the entire posts. Dd is not presently able to clone Microsoft Windows Vista OEM Partitions. How to make a bootable CD from a bootable floppy Put the floppy in the floppy drive, and don't mount it, YET! Code:
dd if=/dev/fd0 of=/home/sam/floppy.img Code:
mkdir /mnt/floppy.img Code:
cd /mnt/floppy.img/ Code:
df -h Code:
cd .. Code:
mkisofs -o /home/sam/floppy.img.iso -b /home/sam/floppy.img /home/sam/floppy.img Code:
wodim dev=/dev/hdc -sao driveropts=burnfree -dummy /home/sam/floppy.img.iso You want to find out if your girlfriend is cheating on you, having cyber whoopie, or your a control freak. Even if the computer is secured with a password, you can boot with the: Knoppix Live CD and search the entire drive partition for text strings: Code:
dd if=/dev/sda2 bs=16065 | strings -n 10 -t d | grep -i -B 20 -A 20 'luv U' This particular command string prints the search results, preceded by the decimal offset of its location on the drive, to the screen. If the offfset is: 34,409,872, we want manageable numbers, custom designed for speed and ease of use. The decimal disk offset is roughly 34 million, so the data we want to view is 34 MB into the partition. We divide 34,409,872 by some power of 2. Experience says 2^13 is about what we want, to get a quotient in the thousands. 34,409,872/8192=~4200. The data we want is 8,192 4,200 byte blocks, OR, 4,200 8,192 byte blocks, into the partition. We check: 4200*8192=34406400; 34,409,872-34406400=3472. This means the following command line will start reading 3,472 bytes before the string location. Code:
dd if=/dev/sda2 bs=4200 skip=8192 count=2 | strings > file.txt [color="red"]This next thing does not work anymore, because the Linux kernel wised up a bit. I'm working on an alternate method.[color] You can search system memory with this method, by substituting Code:
/dev/mem Code:
/dev/sda2 Write system memory to a CD. This is useful for documenting memory contents without contaminating the HDD. I recommend using a CD-RW so you can practice a little. This doesn't involve dd, but it's cool. Code:
wodim /dev=/dev/scd0 -raw driveropts=burnfree /dev/mem Code:
wodim --devices Code:
dd if=/dev/hdd | hexdump -C | less Code:
/dev/hdd Code:
/dev/sda2 Code:
dd if=/dev/hdd | strings -n 6 -t d | grep 'string' list: Code:
'[[:alnum:]]' any alphanumeric characters Code:
dd if=/dev/sda | hexdump -C | grep '[:punct:]' | less Code:
hexdump -C Back up your MBR: Code:
dd if=/dev/sda of=mbr.bin count=1 Code:
dd if=boot.img of=/dev/fd0 Code:
dd if=/dev/fd0 of=/home/sam/floppies/backup.bin conv=notrunc Code:
dd if=/home/sam/floppies/backup.bin of=/dev/fd0 conv=notrunc Code:
dd if=/dev/mem bs=1k skip=768 count=256 2>/dev/null | strings -n 8 Dd need not be black boxed like other inexpensive forensic software: http://www.cftt.nist.gov/ For a low cost bootable CD based professional ghosting solution, that supports all operating systems and file systems: http://www.feyrer.de/g4u/ Variation of dd for data rescue off defective media: http://www.garloff.de/kurt/linux/ddrescue/ Department of Defense implementation of dd: http://dcfldd.sourceforge.net/ Sdd is useful when input block size is different than output block size, and will succeed in some instances where dd fails: http://linux.maruhn.com/sec/sdd.html This is one of the best links I haven't written about dd: http://www.softpanorama.org/Tools/dd.shtml Copyright 2008 by AwesomeMachine. All Rights Reserved. Public Domain Copyright Material Begins Here: Note that sending a SIGUSR1 signal to a running 'dd' process makes it print to standard error the number of records read and written so far, then to resume copying. Code:
$ dd if=/dev/zero of=/dev/null& pid=$! So, Code:
dd if=/dev/sda of=/dev/sdb bs=1GB bs=4b would give dd a block size of 4 disk sectors. 1 sector=512 bytes. bs=4k would indicate dd use a 4 kilobyte block size. I have found bs=4k to be the fastest for copying disk drives on a modern machine. OPERANDS The following operands are supported: Code:
if=file Code:
of=file Standard output is the default. seek=blocks Skip this many blocks in the output file. Code:
ibs=n Code:
obs=n If no conversion other than Code:
sync, noerror, and, notrunc Code:
cbs=n Code:
block or unblock Code:
ascii and asciib Code:
ebcdic, ebcdicb, ibm, and ibmb Code:
files=n Code:
skip=n Code:
iseek=n Code:
oseek=n Code:
seek=n Code:
count=n Code:
conv=value Code:
conv=notrunc Code:
ascii Code:
asciib Code:
ebcdic Code:
dd conv=unblock Code:
ebcdicb Code:
dd conv=unblock Code:
ibm Code:
dd conv=unblock Code:
ibmb Code:
dd conv=unblock Code:
ascii (or asciib), ebcdic (or ebcdicb), and ibm (or ibmb) Code:
lcase Code:
ucase Code:
swab Code:
noerror Code:
sync Code:
of=file Code:
sync ENVIRONMENT VARIABLES The following environment variables affect the messages and errors messages of dd: Code:
LANG Code:
LANG Code:
LC_ALL Code:
LC_CTYPE Code:
LC_MESSAGES Code:
NLSPATH Public Domain Copyright Material Ends Here |
:cool: thanks, mate!
|
personally...i found this thread *****AWESOME*****
thanks man...:) |
Great thread. New most of it, but I sure didn't a few years ago! Would've been great to have!
|
It's useful man!It's useful
|
Bookmarked!
|
very cool............
|
can we stickify this?
|
Straight to my personal HelpFiles folder.
|
great article, however, explaining what "notrunc" is would be good.
|
True. Also, what is the difference between those:
dd if=/dev/urandom bs=1 count=100 dd if=/dev/urandom bs=100 count=1 Yves. |
dd if=/dev/urandom bs=1 count=100
reads/writes 100 times 1 byte, whereas dd if=/dev/urandom bs=100 count=1 reads/writes 1 time 100 bytes; the result is the same (but the latter is faster). I've also experimented with blocksizes of below L1/L2 cache size but it wasn't faster, probably because the hard disk was the bottle neck, not the RAM ;) |
it is acknowledged by the author that:
dd if=/dev/urandom bs=100 count=1 Is faster than dd if=/dev/urandom bs=1 count=100 With urandom it is possible to read a byte size of 100. With /dev/random you need to do bs=1. Urandom does not have this restriction. conv=notrunc is to prevent the output file from being truncated. If you just want to write 1024 bytes to the beginning of other_existing__file.file, but leave the remainder of the file intact, this command will not work: Code:
dd if=/home/sam/existing_file.file bs=1k count=1 of=/home/sam/other_existing_file.file |
great article, im in the process of expanding my / partition and this artilcle will help clear a lot of questions, thanks
|
All times are GMT -5. The time now is 05:34 PM. |