LinuxQuestions.org
Review your favorite Linux distribution.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 10-18-2010, 09:53 AM   #661
JZL240I-U
Senior Member
 
Registered: Apr 2003
Location: Germany
Distribution: openSuSE 42.1_64+Tumbleweed-KDE, Mint 17.3
Posts: 3,949

Rep: Reputation: Disabled

Ahhh, man is every time it is mentioned a marvel -- I'd never have guessed it has an entry on null . Thanks for sharing, onebuck.
 
Click here to see the post LQ members have rated as the most helpful post in this thread.
Old 10-30-2010, 06:37 AM   #662
AwesomeMachine
Senior Member
 
Registered: Jan 2005
Location: USA and Italy
Distribution: Debian testing/sid; OpenSuSE; Fedora
Posts: 1,828

Original Poster
Rep: Reputation: 256Reputation: 256Reputation: 256
Post

Quote:
Originally Posted by JZL240I-U View Post
Hi AwesomeMachine and other knowledgeable persons .

How is /dev/zero working? I mean what mechanism tells dd to use an endless stream of binary zeros e.g. for an input file in dd if=/dev/zero ... ? I can't imagine that the zeros truly get freshly created for more than one blocksize (bs) -- or do they?

Reason for my inquiry (apart from the joy in generalized knowledge) is this: last night I wiped an old 10 GB disk with "dd if=/dev/urandom of=/dev/sdb". That took more than 1 1/2 hours on my old 700 Mhz Athlon.

After this experience my idea was to first create a file of, say, 1 MB of random data and then using this block of data repeatedly as input for dd. Would that be possible and how would one go about that?
First of all, if you write zeroes over an entire drive, only sophisticated recovery methods can possibly recover anything useful. /dev/zero is a kernel feature. The linux kernel produces the zeroes.

/dev/urandom is held in a kernel entropy pool. If entropy runs out, it uses the same entropy over again. /dev/random stops until the kernel has gathered unique entropy.

To use a random file you would have to write a shell script that loops an runs the dd command repeatedly. You would use a variable for the seek='' parameter, and increment it by the size of the random file, so you didn't keep writing the same spot on the disk.

I've never tried this before, but it's theoretically possible. I don't overwrite whole disks with random data. If you really want to make it tough to recover an overwritten drive, first tighten up the disk tracks:

dd if=/dev/sda of=/dev/sda

which rewrites the data to the drive, which gets rid of magnetic flux on the outer edges of the tracks. Then:

dd if=/dev/zero of=/dev/sda

There will be nothing recoverable unless someone takes the drive apart and uses electron micrography.

Last edited by AwesomeMachine; 10-31-2010 at 04:58 PM.
 
1 members found this post helpful.
Old 11-01-2010, 04:33 AM   #663
JZL240I-U
Senior Member
 
Registered: Apr 2003
Location: Germany
Distribution: openSuSE 42.1_64+Tumbleweed-KDE, Mint 17.3
Posts: 3,949

Rep: Reputation: Disabled
Understood. Thanks for clarification and elaboration, AwesomeMachine.
 
Old 11-01-2010, 10:49 PM   #664
sundry_50
Member
 
Registered: Jan 2005
Distribution: knoppix/debian
Posts: 31
Blog Entries: 2

Rep: Reputation: 0
Great information. Thanks! Going in my delicious NOW.
 
Old 11-12-2010, 12:39 AM   #665
kwdaniels
LQ Newbie
 
Registered: Nov 2010
Posts: 2

Rep: Reputation: 0
Thanks AwesomeMachine.

A lot of effort and skill went into this post... I'm seriously impressed.

A short question. I just swithced to ext4 and Ubuntu 10.10.

Are there any known problems with using dd on ext4 file systems?
 
Old 11-21-2010, 11:44 PM   #666
AwesomeMachine
Senior Member
 
Registered: Jan 2005
Location: USA and Italy
Distribution: Debian testing/sid; OpenSuSE; Fedora
Posts: 1,828

Original Poster
Rep: Reputation: 256Reputation: 256Reputation: 256
No problems with ext4 that I've heard of. The only problem file systems I'm aware of are encrypted file systems in which sometimes dd will not copy the encryption header, because someone put the encryption header on a CD and deleted it off the hard drive.
 
Old 11-22-2010, 05:53 PM   #667
kwdaniels
LQ Newbie
 
Registered: Nov 2010
Posts: 2

Rep: Reputation: 0
Thanks AwesomeMachine

I will do some testing in the next month and if I have problems with dd and my encrypted folders I will report the results.
 
Old 11-27-2010, 05:37 PM   #668
ubuntunoob613
LQ Newbie
 
Registered: Nov 2010
Posts: 1

Rep: Reputation: 0
Using DD to image my HDD

Hi,

This would be my first post. I just needed to make an image of my newly installed windows system. The HDD in the system is a TB large and I wouldnt be able to copy the whole thing bit for bit. Is there a way I could use the DD command to copy only what is being used on the Hard drive. Currently I have a little over 900GB free.

Scenario: I'm using Ubuntu to boot from CD and hoping to image only what's being used on the 1TB HDD. I'm looking to output the file to a USB which is 32GB.

Any help is greatly appreciated.

David
 
Old 11-29-2010, 04:42 PM   #669
HGraeber
LQ Newbie
 
Registered: Nov 2010
Posts: 1

Rep: Reputation: 0
First of all, thanks AwesomeMachine for this great post!

I'm fairly new to Linux/Ubuntu. I've Ubuntu 10.10 installed on a 32GB Corsair USB (fdisk -l shows 32.5 GB) and I want to do a 1:1 copy of thsi installation to another USB drive. The new one is a 32GB SanDisk (fdisk -l shows 32.0 GB for this one). What I tried: Boot with SystemRescueCD, plug in source and destination USB (fdisk - l shows me source as /dev/sdc and dest as /dev/sdd. I simply did "dd if=/dev/sdc of=/dev/sdd", but the new USB doesn't boot (from dd I got a success message). Is it because source is 32.5 GB and dest 32.0 GB? Another weird thing that I notice: fdisk -l shows the 3 partitions /dev/sdc and /dev/sdd exactly the same; gparted instead shows /dev/sdd "unallocated". Any thoughs? What would happen, if I shrink a little the ext4 partition of my Ubuntu 10.10 installation, try to create the partions manually and try to copy partitions instead of the whole device?
 
Old 12-10-2010, 11:03 PM   #670
ekamlesh
Member
 
Registered: Jun 2008
Distribution: Debian Wheezy
Posts: 43

Rep: Reputation: 17
Thanks AwesomeMachine for such a wonderful thread.

I have a simple question. Suppose I am using dd without "notrunc" option and I have two partitions sda1 and sda2 of same size. Assume that sda1(as well as sda2) has 10000(512 bytes) blocks. For sda1, first 2000 and last 2000 blocks have data on it and rest are all zero blocks. When i run
Code:
dd if=/dev/sda1 of=/dev/sda2 bs=512
blocks which are zero in between will be abbreviated by *****.
My question. Will the operating system understand those 5 asteriks?
That is, when I mount my partition sda2, will the data on last 2000 blocks of sda1 be accessible?

Last edited by ekamlesh; 12-10-2010 at 11:08 PM. Reason: elaborate
 
Old 12-13-2010, 04:22 AM   #671
AwesomeMachine
Senior Member
 
Registered: Jan 2005
Location: USA and Italy
Distribution: Debian testing/sid; OpenSuSE; Fedora
Posts: 1,828

Original Poster
Rep: Reputation: 256Reputation: 256Reputation: 256
Post DD using bzip2 or gzip

Quote:
Originally Posted by ubuntunoob613 View Post
Hi,

This would be my first post. I just needed to make an image of my newly installed windows system. The HDD in the system is a TB large and I wouldnt be able to copy the whole thing bit for bit. Is there a way I could use the DD command to copy only what is being used on the Hard drive. Currently I have a little over 900GB free.

Scenario: I'm using Ubuntu to boot from CD and hoping to image only what's being used on the 1TB HDD. I'm looking to output the file to a USB which is 32GB.

Any help is greatly appreciated.

David
Hi David,

Try this:

dd if=/dev/sda bs=32k | gzip | dd of=/dev/sdb1

OR

mount the flash drive to /mnt/sdb1 and try:

dd if=/dev/sda bs=32k | bunzip2 > /mnt/sdb1/partition.img

But bzip2 compression of a disk really stresses the memory subsystem. Gzip is easier on the hardware.

Certain memory weaknesses cause problems only when running bzip2 full blast for a half hour or more!

Some unrelated trivia: the most common cause of unrecoverable memory errors are gamma radiation and other cosmic rays!
 
Old 12-13-2010, 04:44 AM   #672
AwesomeMachine
Senior Member
 
Registered: Jan 2005
Location: USA and Italy
Distribution: Debian testing/sid; OpenSuSE; Fedora
Posts: 1,828

Original Poster
Rep: Reputation: 256Reputation: 256Reputation: 256
Smile A little trick I learned about flash drives!

Quote:
Originally Posted by HGraeber View Post
First of all, thanks AwesomeMachine for this great post!

I'm fairly new to Linux/Ubuntu. I've Ubuntu 10.10 installed on a 32GB Corsair USB (fdisk -l shows 32.5 GB) and I want to do a 1:1 copy of thsi installation to another USB drive. The new one is a 32GB SanDisk (fdisk -l shows 32.0 GB for this one). What I tried: Boot with SystemRescueCD, plug in source and destination USB (fdisk - l shows me source as /dev/sdc and dest as /dev/sdd. I simply did "dd if=/dev/sdc of=/dev/sdd", but the new USB doesn't boot (from dd I got a success message). Is it because source is 32.5 GB and dest 32.0 GB? Another weird thing that I notice: fdisk -l shows the 3 partitions /dev/sdc and /dev/sdd exactly the same; gparted instead shows /dev/sdd "unallocated". Any thoughs? What would happen, if I shrink a little the ext4 partition of my Ubuntu 10.10 installation, try to create the partions manually and try to copy partitions instead of the whole device?
I think the culprit is something called: U3. You can google it if you wish. Write 400 MB of /dev/zero to the target drive, twice.

dd if=/dev/zero of=/dev/sdd bs=1M count=400

After it's finished, hit the up arrow and do it again. It'll take 2 minutes or so. Pull the target drive out, wait 10 seconds, and put it back.

Now it should work, unless you have U3 on the source drive! If that's the case, you can mount the drive, use rsync -rv <drive_mount_point/*> <some_empty_directory/>,

Wait for rsync to finish! Do the dd trick to the source drive the same way. Partition the drive and format as vfat. Mount it and rsync the directory tree you just copied off it, back to it.

Then install isolinux to the drive. I'm not promising this will for sure work in every possible case. You might come out ahead using the Ubuntu method to reinstall Linux to the source drive.

But I want readers to learn something about what's really going on underneath those USB installation scripts!

I'm going to bow and say, "Thank you," to everyone for the recent applause! Thank you.

-Awesome Bows.
 
Old 12-13-2010, 04:50 AM   #673
AwesomeMachine
Senior Member
 
Registered: Jan 2005
Location: USA and Italy
Distribution: Debian testing/sid; OpenSuSE; Fedora
Posts: 1,828

Original Poster
Rep: Reputation: 256Reputation: 256Reputation: 256
Thumbs up That isn't true any longer.

Quote:
Originally Posted by ekamlesh View Post
Thanks AwesomeMachine for such a wonderful thread.

I have a simple question. Suppose I am using dd without "notrunc" option and I have two partitions sda1 and sda2 of same size. Assume that sda1(as well as sda2) has 10000(512 bytes) blocks. For sda1, first 2000 and last 2000 blocks have data on it and rest are all zero blocks. When i run
Code:
dd if=/dev/sda1 of=/dev/sda2 bs=512
blocks which are zero in between will be abbreviated by *****.
My question. Will the operating system understand those 5 asteriks?
That is, when I mount my partition sda2, will the data on last 2000 blocks of sda1 be accessible?
The discussion thread sort of evolved, although I'm impressed you read that far into it. Hexdump is the program that represents successive zero blocks as *****.

You are correct. It is my mistake. If you read a little further on, I corrected it.
 
Old 12-13-2010, 10:26 PM   #674
ekamlesh
Member
 
Registered: Jun 2008
Distribution: Debian Wheezy
Posts: 43

Rep: Reputation: 17
Thankyou again AwesomeMachine for the clarification. I read most of the pages of the thread, some how I missed your earlier clarification. Sorry for bothering you.
 
Old 12-14-2010, 01:57 AM   #675
AwesomeMachine
Senior Member
 
Registered: Jan 2005
Location: USA and Italy
Distribution: Debian testing/sid; OpenSuSE; Fedora
Posts: 1,828

Original Poster
Rep: Reputation: 256Reputation: 256Reputation: 256
Arrow No Bother ...

Quote:
Originally Posted by ekamlesh View Post
Thankyou again AwesomeMachine for the clarification. I read most of the pages of the thread, some how I missed your earlier clarification. Sorry for bothering you.
I started this tread to help others. Feel free to ask questions. I'm a no nonsense writer, so sometimes I come across indifferent or stern. I encourage readers to experiment. That was my original intention for the OP.

Last edited by AwesomeMachine; 05-13-2011 at 08:03 PM.
 
  


Reply

Tags
backup, best, clonezilla, cloning, command, data, dd, disk, drive, duplicate, erase, explanation, formatting, ghost, hard, image, iso, memory, ping, popular, recover, recovery, rescue, search, security, stick, upgrade, usb, wipe


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Learn The DD command AwesomeMachine Linux - Newbie 17 08-17-2006 05:22 AM
The best way to learn? iz3r Programming 7 02-07-2005 12:00 AM
Best way to learn Linux from the command line patpawlowski Linux - General 2 03-01-2004 04:37 PM
I want to learn C. KptnKrill Programming 14 12-18-2003 02:03 PM
Best way to learn.... InEeDhElPlInUx Linux - Newbie 5 10-11-2003 02:02 AM


All times are GMT -5. The time now is 06:52 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration