LinuxQuestions.org
View the Most Wanted LQ Wiki articles.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices

Reply
 
Search this Thread
Old 07-27-2007, 12:11 PM   #331
AwesomeMachine
Senior Member
 
Registered: Jan 2005
Location: USA and Italy
Distribution: Debian jessie/sid; OpenSuSE; Fedora
Posts: 1,593

Original Poster
Rep: Reputation: 162Reputation: 162
Optical Media and dd.


Quote:
Originally Posted by dive
I'm sure I read a way of copying a CD with if=/dev/hdc of=/dev/hdd but this gives a 'read only file system error'. I know that growisofs uses an 'in-built' dd so is there some way of getting dd to do this (and also DVDs)?
One of the things that dd doesn't do is write to optical media. If growisofs uses dd, it is inside a scsi wrapper that communicates with the CD/DVD device. CD/DVD devices are all slightly different, and dd doesn't bother to query the drive, it just assumes. The scsi module translates the dd communication to scsi communication for optical drives. Or at least that's probably the way it works. I have read of someone using:

hdc=ide-scsi in the /boot/grub/menu.lst kernel line

and then using the device:

/dev/scd0
like so:
dd if=/dev/hdd of=/dev/scd0

But I could never get it to work, so I think it was a fluke.
 
Old 08-05-2007, 10:33 AM   #332
cojo
Member
 
Registered: Feb 2003
Location: St. Louis
Distribution: RedHat 8
Posts: 262

Rep: Reputation: 31
I started my clone from a 80GB to 160GB hard drive 17 hours ago with this command
dd if=/dev/hda of=/dev/hdb conv=notrunc,noerror
As of right now, dd is still running on my system. Is there a way to find the current progress on dd? How much longer should I wait? FYI, the 80GB does have some back block on it. When I cloned my good 80GB to the same 160GB drive it only took 7 hours to completed. What is my options?

John
 
Old 08-09-2007, 04:56 PM   #333
AwesomeMachine
Senior Member
 
Registered: Jan 2005
Location: USA and Italy
Distribution: Debian jessie/sid; OpenSuSE; Fedora
Posts: 1,593

Original Poster
Rep: Reputation: 162Reputation: 162
Dd, what's the wait.

Quote:
Originally Posted by cojo
I started my clone from a 80GB to 160GB hard drive 17 hours ago with this command
dd if=/dev/hda of=/dev/hdb conv=notrunc,noerror
As of right now, dd is still running on my system. Is there a way to find the current progress on dd? How much longer should I wait? FYI, the 80GB does have some back block on it. When I cloned my good 80GB to the same 160GB drive it only took 7 hours to completed. What is my options?

John
17 hours is a bit long to wait. You might also use the block size, 'bs' parameter like so

dd if=/dev/hda of=/dev/hdb conv=notrunc,noerror bs=4k

If the drive has bad blocks you first want to do:

dd if=/dev/hda of=/dev/hda or
dd if=/dev/zero of=/dev/hda or
Both.
 
Old 08-09-2007, 05:54 PM   #334
cojo
Member
 
Registered: Feb 2003
Location: St. Louis
Distribution: RedHat 8
Posts: 262

Rep: Reputation: 31
Thanks...I finally break out after 29 hours. Since, the ps was in S+ state. The weird part was I was able to boot up with the new drive for 2 hours then I get readonly on the / mount point. When I rebooted the server it won't boot all the way. It took me to single mode and said I need to run fsck to the mount point. So, I'm going to do a fsck this weekend and see if that will fixed my problem.
 
Old 08-10-2007, 09:07 PM   #335
marco18
Member
 
Registered: Jul 2007
Location: Argentina
Distribution: Ubuntu 13.04 , Debian Lenny 5.0.7
Posts: 223

Rep: Reputation: 21
Great article! This site is much better than I thought! Great work guys!
 
Old 08-13-2007, 02:00 AM   #336
BT+1
LQ Newbie
 
Registered: Aug 2007
Posts: 11

Rep: Reputation: 0
Well, since you wanted a reply...

Great overview for a Linux newbie like me! I came here after discovering that both Partition Magic and gparted wouldn't copy a TrueCrypt partition (presumably because they didn't understand its internal format - which is entirely reasonable when asking such a utility to resize a partition but seems a bit arbitrary for a simple copy operation, where a blind byte-by-byte copy should do the job if it doesn't understand the internal format sufficiently to do something more efficient) - and then seeing references to the use of dd in the latter (and in clonezilla as well). While I appreciate the added layer of safety that such GUI utilities provide for the fumble-fingered, for the task I was actually interested in performing dd looks like an entirely acceptable option.

Question: why use noerror for copy operations? Your post seems to say that on an input error dd will skip the data in the output (rather than leave a space where it should have gone), resulting in an unusable file/partition/disk beyond that point, whereas if you let dd stop on an error you can just restart at the appropriate offset to retry and keep going (assuming that the error was transient in nature).

Question: if you use, say, a 4 KB block size (as you do in some of your full-partition and full-disk copy examples), what happens to the last few disk sectors if the entity you're operating upon does not contain a multiple of 8 sectors (which, for example, will *usually* be the case when copying a partition comprising 16065b cylinders)? Since you recommended this option I had expected dd to do the right thing (copy the remainder sectors at single-sector granularity), but toward the end of the thread you seem to say that it does not (which could prove disastrous for any file system or database that used the last few sectors in its partition for something important). In a somewhat related vein, some systems (some IBM Thinkpads, for example) use a nominal 240 heads rather than 255 heads, making their nominal cylinder size 15120b rather than 16065b - another case where having dd 'do the right thing' if an awkward block size were chosen could avoid unpleasant surprises. A third potential problem is that both 'logical' partitions (in the 'extended' partition) and any partition beginning at the very start of the disk do not include the first track of their first cylinder, because that's where the MBR (in the latter case) or the EPBR (for a logical partition) resides - once again screwing up the expected alignment (whether 16065b or 15120b). If dd indeed does *not* do the right thing in such cases, using a 512-byte block size (or possibly a 63b block size, since all modern hard disk cylinders seem to use nominal 63-sector tracks) would seem the only prudent course.

For that matter, will dd copy files exactly (not that it normally should be used for this) if the last sector of the input file is only partially filled? Not all file types will interpret zero-fill after the last valid byte in the manner desired, so unless dd honors the byte-granularity end-of-file value in an input file there could be a problem.

As others have already observed, your advice in the OP about copying a partition to a larger or smaller one is inappropriate: a file system (or database) resident in a partition usually has internal information regarding that partition's size and does not necessarily re-check this size as described in the partition table entry (which it may not even know how to find) every time it starts; even if it did perform this check it could have structures spread throughout the original partition's space that could be inconsistent with its revised size - even after defragmenting; and even if all the filesystem's structures were nicely compacted at the start of the partition, the remaining space would not necessarily be filled with zeros (and hence compactable by dd) if data had resided there in the past. The bottom line is that you really should use a file-system- (or database-)aware utility to resize any partition containing a file system (or raw database).

Furthermore, my understanding is that the partition table in the MBR does not describe the size of the disk but just the sizes and locations of up to 4 primary partitions on that disk (one of which - or possibly more than one of which in Linux - may be an 'extended' partition). So skipping the MBR in a whole-disk-to-whole-disk copy is not helpful, and in fact skipping it leaves the output disk's partition table (if any) in its previous state while the source disk's (likely quite different) partition layout is copied to the output disk: instead, if you want the same partition layout on the output disk, copy the MBR along with everything else as a whole-disk-to-whole-disk copy, and if you don't want the same partition layout on the output disk, create the new layout there (using the sizes of the relevant partitions on the source disk) before doing anything else (and copy over the first 446 bytes of the MBR if the master boot code on the source differs from that created on the destination), then copy partition-by-partition, then resize as desired using some utility that understands how to do that properly.

Overall, though, a great resource: thanks for the time and effort that you have put into it.
 
Old 08-13-2007, 04:20 AM   #337
BT+1
LQ Newbie
 
Registered: Aug 2007
Posts: 11

Rep: Reputation: 0
(fyi)

Haven't tried dd yet (pending answers here), but did try a little (runs from two floppies) NetBSD utility called g4u that seemed to do the job (i.e., perform a dumb bit-for-bit copy of a TrueCrypt partition, though I haven't yet tried doing so to a different offset on the target disk from the source disk offset).
 
Old 08-15-2007, 02:25 PM   #338
alar
Member
 
Registered: Sep 2003
Location: Canada
Distribution: CentOS 5.5, Ubuntu 10.04, Linux Mint 11 gnome 64 bit
Posts: 336

Rep: Reputation: 36
Thank you!
 
Old 08-19-2007, 08:29 PM   #339
fleetfoot
LQ Newbie
 
Registered: Aug 2007
Posts: 1

Rep: Reputation: 0
Great help article!

My old drive had been causing XP to lock up each day. Ran Spinrite on it and got a warning that the drive was overheating. I shut it down, unplugged the old drive, popped in my fav Linux LiveCD distro to get online, did some research and luckily found this site.

I didn't know if using dd would be the best choice for a drive in trouble of failing (or with errors) so I ended up using Antonio Diaz's Gnu ddrescue instead of dd.

Using an Ubuntu LiveCD (and installing Gnu ddrescue, which is "gddrescue" on that distro), I was able to successfully clone my "old" Maxtor 120GB to a new WD 200GB. Then rebooted with just the new drive and everything worked fine. I then used gparted to "claim" the extra free space on the new drive, and it all worked great!

The information I got from this forum topic helped trememdously. Thanks everyone.
 
Old 08-20-2007, 12:58 AM   #340
lpn1160
Member
 
Registered: May 2004
Location: USA
Distribution: pclinuxos
Posts: 37

Rep: Reputation: 15
An interesting tip to checkout

http://fosswire.com/category/tips-tricks/
 
Old 08-26-2007, 12:57 AM   #341
ezor
Member
 
Registered: Mar 2006
Location: Toytown, NY
Distribution: Suse10.2 FreeBSD6.2
Posts: 83

Rep: Reputation: 15
I want to thank you for posting this. To say it has been most useful would be a gross understatement.

In playing with FreeBSD, on one of my computers, I can't say how many times I've had to reinstall after screwing something up, and reinstalling software from ports can be unbelievably time consuming. Now, working in steps, as soon as I get something right, I move it to the slave drive. If I later screw up, it's a simple matter to get back where I began, like you said.

I might add that the two drives on this computer aren't a matched set, but both are the same size (well, very close) and it works. Also, thanks for mentioning the Helix CD; much better than using dd.
 
Old 08-27-2007, 01:08 AM   #342
blazoner
LQ Newbie
 
Registered: Aug 2007
Distribution: Xubuntu 13.10
Posts: 1

Rep: Reputation: 0
Question dd not stopping at the end of "if=hda"

I have attempted several times to backup my 20Gb hdd running xp to a file on my 200Gb hdd which has about 170Gb free space to work with using variations of

Code:
sudo dd if=/dev/hda of=/home/blazoner/.disk-images/XP-MediaCenter2005-HDD bs=4k
but every time I try, dd doesn't stop copying at the end of my 20Gb hdd, but instead continues writing to the "of" until I receive an error message stating:

Quote:
dd: writing `/home/blazoner/.disk-images/XP-MediaCenter2005-HDD': No space left on device
45523587+0 records in
45523586+0 records out
186464608256 bytes (186 GB) copied, 9283.24 seconds, 20.1 MB/s
this gives me no room to work from on my hdd.
so now what do I try?

ps. great write-up!

SORTED OUT
stupid mistake. mixed up my hdd's. was copying my hda into a file on itself....
now to give it a go the right way....

Last edited by blazoner; 08-27-2007 at 07:04 AM.
 
Old 09-07-2007, 02:33 AM   #343
AwesomeMachine
Senior Member
 
Registered: Jan 2005
Location: USA and Italy
Distribution: Debian jessie/sid; OpenSuSE; Fedora
Posts: 1,593

Original Poster
Rep: Reputation: 162Reputation: 162
Smile This girl requires a special reply.

Quote:
Originally Posted by BT+1 View Post
Great overview for a Linux newbie like me! I came here after discovering that both Partition Magic and gparted wouldn't copy a TrueCrypt partition (presumably because they didn't understand its internal format - which is entirely reasonable when asking such a utility to resize a partition but seems a bit arbitrary for a simple copy operation, where a blind byte-by-byte copy should do the job if it doesn't understand the internal format sufficiently to do something more efficient) - and then seeing references to the use of dd in the latter (and in clonezilla as well). While I appreciate the added layer of safety that such GUI utilities provide for the fumble-fingered, for the task I was actually interested in performing dd looks like an entirely acceptable option.

Question: why use noerror for copy operations? Your post seems to say that on an input error dd will skip the data in the output (rather than leave a space where it should have gone), resulting in an unusable file/partition/disk beyond that point, whereas if you let dd stop on an error you can just restart at the appropriate offset to retry and keep going (assuming that the error was transient in nature).

Question: if you use, say, a 4 KB block size (as you do in some of your full-partition and full-disk copy examples), what happens to the last few disk sectors if the entity you're operating upon does not contain a multiple of 8 sectors (which, for example, will *usually* be the case when copying a partition comprising 16065b cylinders)? Since you recommended this option I had expected dd to do the right thing (copy the remainder sectors at single-sector granularity), but toward the end of the thread you seem to say that it does not (which could prove disastrous for any file system or database that used the last few sectors in its partition for something important). In a somewhat related vein, some systems (some IBM Thinkpads, for example) use a nominal 240 heads rather than 255 heads, making their nominal cylinder size 15120b rather than 16065b - another case where having dd 'do the right thing' if an awkward block size were chosen could avoid unpleasant surprises. A third potential problem is that both 'logical' partitions (in the 'extended' partition) and any partition beginning at the very start of the disk do not include the first track of their first cylinder, because that's where the MBR (in the latter case) or the EPBR (for a logical partition) resides - once again screwing up the expected alignment (whether 16065b or 15120b). If dd indeed does *not* do the right thing in such cases, using a 512-byte block size (or possibly a 63b block size, since all modern hard disk cylinders seem to use nominal 63-sector tracks) would seem the only prudent course.

For that matter, will dd copy files exactly (not that it normally should be used for this) if the last sector of the input file is only partially filled? Not all file types will interpret zero-fill after the last valid byte in the manner desired, so unless dd honors the byte-granularity end-of-file value in an input file there could be a problem.

As others have already observed, your advice in the OP about copying a partition to a larger or smaller one is inappropriate: a file system (or database) resident in a partition usually has internal information regarding that partition's size and does not necessarily re-check this size as described in the partition table entry (which it may not even know how to find) every time it starts; even if it did perform this check it could have structures spread throughout the original partition's space that could be inconsistent with its revised size - even after defragmenting; and even if all the filesystem's structures were nicely compacted at the start of the partition, the remaining space would not necessarily be filled with zeros (and hence compactable by dd) if data had resided there in the past. The bottom line is that you really should use a file-system- (or database-)aware utility to resize any partition containing a file system (or raw database).

Furthermore, my understanding is that the partition table in the MBR does not describe the size of the disk but just the sizes and locations of up to 4 primary partitions on that disk (one of which - or possibly more than one of which in Linux - may be an 'extended' partition). So skipping the MBR in a whole-disk-to-whole-disk copy is not helpful, and in fact skipping it leaves the output disk's partition table (if any) in its previous state while the source disk's (likely quite different) partition layout is copied to the output disk: instead, if you want the same partition layout on the output disk, copy the MBR along with everything else as a whole-disk-to-whole-disk copy, and if you don't want the same partition layout on the output disk, create the new layout there (using the sizes of the relevant partitions on the source disk) before doing anything else (and copy over the first 446 bytes of the MBR if the master boot code on the source differs from that created on the destination), then copy partition-by-partition, then resize as desired using some utility that understands how to do that properly.

Overall, though, a great resource: thanks for the time and effort that you have put into it.
While I'm sure you're cute, I have the benefit of experience. I have tried all the examples in the OP. They might be theoretically wrong if dd is a dumb bitstreamer, but it isn't. It's a smart tool, although this is supposed to be transparent to the user.

I see you have a lot of information. If one could learn by watching, dogs would become butchers. Now get some HDDs, and start doing some real work with dd. If you have another question, feel free to come back and answer it for yourself in the same post you ask it. Take a break from whatever it is you're not taking a break from often enough. And, You're Welcome

Awesome

Last edited by AwesomeMachine; 09-07-2007 at 02:51 AM. Reason: forgot something
 
Old 09-07-2007, 02:47 AM   #344
AwesomeMachine
Senior Member
 
Registered: Jan 2005
Location: USA and Italy
Distribution: Debian jessie/sid; OpenSuSE; Fedora
Posts: 1,593

Original Poster
Rep: Reputation: 162Reputation: 162
You Are Welcome

Quote:
Originally Posted by ezor View Post
I want to thank you for posting this. To say it has been most useful would be a gross understatement.

In playing with FreeBSD, on one of my computers, I can't say how many times I've had to reinstall after screwing something up, and reinstalling software from ports can be unbelievably time consuming. Now, working in steps, as soon as I get something right, I move it to the slave drive. If I later screw up, it's a simple matter to get back where I began, like you said.

I might add that the two drives on this computer aren't a matched set, but both are the same size (well, very close) and it works. Also, thanks for mentioning the Helix CD; much better than using dd.
You're Welcome,

ezor
 
Old 09-07-2007, 08:11 AM   #345
simcox1
Member
 
Registered: Mar 2005
Location: UK
Distribution: Slackware
Posts: 794
Blog Entries: 2

Rep: Reputation: 30
I have been looking for a way to securely delete a file from a reiserfs partition. There doesn't seem to be any good solution. Having read through or scanned every post here, would dd be worth trying?

$ dd if=/dev/urandom of=somefile bs=721 count=1 conv=notrunc

Secure delete utilities such as shred apparently don't work well with journalling file systems. It's just a single file I want to get rid of securely. I suppose I could write and then print off the file using a live cd, which wouldn't leave any trace either. Any suggestions?
 
  


Reply

Tags
backup, best, clonezilla, cloning, command, data, dd, disk, drive, duplicate, erase, explanation, formatting, ghost, hard, image, iso, memory, ping, popular, recover, recovery, rescue, search, security, stick, upgrade, usb, wipe


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Learn The DD command AwesomeMachine Linux - Newbie 17 08-17-2006 05:22 AM
The best way to learn? iz3r Programming 7 02-07-2005 12:00 AM
Best way to learn Linux from the command line patpawlowski Linux - General 2 03-01-2004 04:37 PM
I want to learn C. KptnKrill Programming 14 12-18-2003 02:03 PM
Best way to learn.... InEeDhElPlInUx Linux - Newbie 5 10-11-2003 02:02 AM


All times are GMT -5. The time now is 03:46 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration