LinuxQuestions.org
Did you know LQ has a Linux Hardware Compatibility List?
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices

Reply
 
Search this Thread
Old 10-09-2006, 03:14 AM   #241
Emmanuel_uk
Senior Member
 
Registered: Nov 2004
Distribution: Mandriva mostly, vector 5.1, tried many.Suse gone from HD because bad Novell/Zinblows agreement
Posts: 1,604

Rep: Reputation: 47
Any more information about error messages from dd


AwesomeMachine, I found the info pages on dd quite short on error messages with dd.
When I backup partitions (using notrunc,noerror and from a live cd)
I always get a +1 next to the number of records transferred.
Saying that the OS on the backup partition seems to be working fine.
(I know it is short on details; I am not asking for a solution to this;
also I may have a "wobbly" HD)

I wondered if you knew a bit more about dd error messages (or lack of),
that could be added to your tutorial.
Thanks
 
Old 10-09-2006, 05:20 AM   #242
AwesomeMachine
Senior Member
 
Registered: Jan 2005
Location: USA and Italy
Distribution: Debian jessie/sid; OpenSuSE; Fedora
Posts: 1,593

Original Poster
Rep: Reputation: 162Reputation: 162
Smile

Quote:
Originally Posted by Emmanuel_uk
AwesomeMachine, I found the info pages on dd quite short on error messages with dd.
When I backup partitions (using notrunc,noerror and from a live cd)
I always get a +1 next to the number of records transferred.
Saying that the OS on the backup partition seems to be working fine.
(I know it is short on details; I am not asking for a solution to this;
also I may have a "wobbly" HD)

I wondered if you knew a bit more about dd error messages (or lack of),
that could be added to your tutorial.
Thanks
As cryptic as its syntax are dd error messages. Dd is a bit lacking in verboseness. There is no -v switch. If I could find a place that lists all the dd messages I would add them to the post. The 'xxxxxxxxx records +1' means there was one piece of data on the source device that was less than one block. Block size is determined by 'bs=' in the dd command. Say you specify 'bs=8192'. That's 16 sectors per block. Sometimes the last block isn't a full block. It might only be 13 sectors. Dd skips the data at the end of the partition which doesn't make a full block. You can get around this by leaving out 'bs=', and block size will default to 'bs=512'. 512 bytes is one sector on an IDE or SATA (which are also IDE (Integrated Drive Electronics) drives. With 512 byte block size there can't be an incomplete block because drives are are divided in 512 byte chunks called sectors. You can also use cylinders for blocks, as each partition is made of only complete cylinders:

bs=16065b

Will accomplish this, but that is a very large block size for dd and may degrade performance. Cylinders are 255 (heads) * 63 (sectors per track) = sectors per cylinder * 512, or in dd language 'b' = bytes per cylinder. Partitions must have complete cylinders. There are no fractions of cylinders in a partition.
 
Old 10-09-2006, 05:30 AM   #243
Emmanuel_uk
Senior Member
 
Registered: Nov 2004
Distribution: Mandriva mostly, vector 5.1, tried many.Suse gone from HD because bad Novell/Zinblows agreement
Posts: 1,604

Rep: Reputation: 47
Quote:
records +1' means there was one piece of data on the source device that was less than one block
Thanks, that makes sense and is enlightening (much more than the doc itself).
I always use bs=4096.

Quote:
If I could find a place that lists all the dd messages I would add them to the post.
Well, I will be on the lookout, one never knows
 
Old 11-30-2006, 04:50 AM   #244
Loonyjuice
LQ Newbie
 
Registered: Nov 2006
Posts: 2

Rep: Reputation: 0
Does it matter if I use dd while the system is running other processes, or do I need to switch to single user mode? What happens to the contents of /proc when you restore a image?
 
Old 12-01-2006, 01:35 AM   #245
karelvdm
Member
 
Registered: Sep 2004
Location: Pietermaritzburg South Africa
Distribution: Home:OpenSuse 11, Office:FreeBSD 7 PBX:Trixbox 2 (CentOS 4.4) Home Automation : LinuxMCE
Posts: 123

Rep: Reputation: 15
Hey Guys

I'm new to linux, but wanted to know if it is possible for dd to clone a 80gb hdd to a 20gb hdd. Their is only about 4gb of data on the 80gb hdd.

Thanx in advance
 
Old 12-02-2006, 11:10 PM   #246
AwesomeMachine
Senior Member
 
Registered: Jan 2005
Location: USA and Italy
Distribution: Debian jessie/sid; OpenSuSE; Fedora
Posts: 1,593

Original Poster
Rep: Reputation: 162Reputation: 162
Running System

Quote:
Originally Posted by Loonyjuice
Does it matter if I use dd while the system is running other processes, or do I need to switch to single user mode? What happens to the contents of /proc when you restore a image?
It really doesn't matter if the system is running, or even if a drive is mounted or not. /proc is a virtual file system. It doesn't exist on the HDD.
 
Old 12-02-2006, 11:15 PM   #247
AwesomeMachine
Senior Member
 
Registered: Jan 2005
Location: USA and Italy
Distribution: Debian jessie/sid; OpenSuSE; Fedora
Posts: 1,593

Original Poster
Rep: Reputation: 162Reputation: 162
Theory and Practice

Quote:
Originally Posted by karelvdm
Hey Guys

I'm new to linux, but wanted to know if it is possible for dd to clone a 80gb hdd to a 20gb hdd. Their is only about 4gb of data on the 80gb hdd.

Thanx in advance
In theory it is possible to do this, but in practice you are better off just copying the files. If you need the drive bootable, use a linux install CD, boot into rescue mode, and install grub.
 
Old 12-03-2006, 03:49 AM   #248
john_smith
Member
 
Registered: Apr 2006
Posts: 42

Rep: Reputation: 15
alot of thanks
 
Old 12-03-2006, 03:17 PM   #249
stockguy
LQ Newbie
 
Registered: Dec 2006
Posts: 1

Rep: Reputation: 0
Excellent DD article!

Thank you for this article. It helped me very much.
 
Old 12-03-2006, 03:46 PM   #250
gendoikari
LQ Newbie
 
Registered: Sep 2006
Distribution: xandros
Posts: 12

Rep: Reputation: 0
Quote:
Originally Posted by karelvdm
Hey Guys

I'm new to linux, but wanted to know if it is possible for dd to clone a 80gb hdd to a 20gb hdd. Their is only about 4gb of data on the 80gb hdd.

Thanx in advance
Since there is only 4gb it's possible as long as you don't wipe the fs
 
Old 12-04-2006, 01:02 AM   #251
karelvdm
Member
 
Registered: Sep 2004
Location: Pietermaritzburg South Africa
Distribution: Home:OpenSuse 11, Office:FreeBSD 7 PBX:Trixbox 2 (CentOS 4.4) Home Automation : LinuxMCE
Posts: 123

Rep: Reputation: 15
Quote:
Originally Posted by gendoikari
Since there is only 4gb it's possible as long as you don't wipe the fs
What would the command look like.
 
Old 12-16-2006, 02:34 AM   #252
AwesomeMachine
Senior Member
 
Registered: Jan 2005
Location: USA and Italy
Distribution: Debian jessie/sid; OpenSuSE; Fedora
Posts: 1,593

Original Poster
Rep: Reputation: 162Reputation: 162
Smile Copying different partition sizes

Quote:
Originally Posted by karelvdm
What would the command look like.
If you copy a drive, or partition, dd copies the entire file system. Dd does not discriminate based on what is usable data, and what is empty, but formatted space on the partition. For copying user data you use "cp -r". Dd is a pretty useful tool, and in my post I made a little more out of it than is necessary. Some things are just cool to do with dd. But, copying user data is not something dd was meant to do. That's why there are other linux tools besides dd. If you could use one command for everything, linux wouldn't be so hard to learn, but you can't. Use the best tool for the job. If you take the time to read this entire thread you will understand why you can't use dd for what you want to do.
 
Old 12-18-2006, 10:48 AM   #253
saikee
Senior Member
 
Registered: Sep 2005
Location: Newcastle upon Tyne UK
Distribution: Any free distro.
Posts: 3,398
Blog Entries: 1

Rep: Reputation: 112Reputation: 112
Here is an example of using tar or dd

dd can naturally clone a hard disk if the target is identical in size or larger as the excess space will become unallocated space by default.

The opposite way doesn't work because if a large disk is cloned into a small disk the partition table will be ruinned and cannot be understood by the operating system or ourselves since the big disk addresses are outisde the bound of the small disk. If a 80Gb disk is fully used up then its partition table will have a record of the address of the end of the 80Gb disk. How can an operating system find the same address in a small disk with only 20Gb?

To migrate a 80Gb disk say with only one partition of hda1 into a smaller 20Gb disk formatted with only one partition say hdc1 the following steps are one of the many alternatives

(1) make temporary directories to mount hda1 and hdc1 if they are not parts of the booted up filing system.
Code:
mkdir /mnt/hda1
mkdir /mnt/hdc1
mount /dev/hda1 /mnt/hda1
mount /dev/hdc1 /mnt/hdc1
(2) change root to the directory you wish to clone the information out. In this case it is the source hda1
Code:
cd /mnt/hda1
(3) Just one tar command is needed to clone the filing system
Code:
tar cf - . | (cd /mnt/hdc1; tar xvf -)
The entire original filing system with all the settings and protections are cloned from hda1 to hdc1. As Awesomemachine pointed out previously the filing-system cloning does not copy the boot sector across because the boot loader is not part of any filing system. This makes dd different from all the other archiving commands because it duplicates or mirrors every byte in the partition or disk to be cloned and so the boot sector is automatically included. If the partition happens to be the first one then the partition table will be copied too.

If a user has a need to make the new disk hdc bootable then he/she will need to change root into the new partition to restore the boot loader.
(4)
Code:
chroot /mnt/hdc1
If the disk hda1 was originally booted by Grub the command to put Grub into the MBR of the new home hdc is simply
Code:
grub-install /dev/hdc
For Lilo booting hda1 originally the equivalent command is
Code:
lilo -b /dev/hdc
Please be aware after change-root a user is actually inside hdc1 and must issue
Code:
exit
to return back to the Linux where everything starts.

Had the target disk been the same size or bigger dd would eat it for breakfast with just one line command (omitting parameters to show its simplest form) of
Code:
dd if=/dev/hda of=/dev/hdc
Hope the above clarifies the main difference between dd and other archiving commands.

A user will appreciate the usefulness of dd if he/she is cloning a large disk with inside many different partitions for Dos, Windows, Linux, BSD... and can rely on the target disk to boot exactly same as the orginal.

Observed difference between dd and other commercial software like Ghost/Acronis True Image

dd is simpler, usually faster and more reliable.

To be fair the commercial software can clone a big source disk into a small target disk, assuming the files can be accommodated in the small target disk, by the same command as cloning a small source into a big target. Due to the same technique being used the type of operating systems inside the disk/partitions is immaterial too. In order word commercial cloning software are equally capable.

I used to rely on Ghost for cloning but since into Linux I never have had a need to go back to the commercial software, except for comparison studies.

Last edited by saikee; 12-18-2006 at 07:07 PM.
 
Old 12-18-2006, 01:07 PM   #254
Micro420
Senior Member
 
Registered: Aug 2003
Location: Berkeley, CA
Distribution: Mac OS X Leopard 10.6.2, Windows 2003 Server/Vista/7/XP/2000/NT/98, Ubuntux64, CentOS4.8/5.4
Posts: 2,986

Rep: Reputation: 45
THANK YOU! Very helpful post with examples. Why can't MAN pages, in general, contain examples such as this?

Question: I have my Ubuntu system running on RAID 1 with 2x160GB hard drives. I am wanting to expand them to 2x750GB hard drives also running in RAID 1. Will `dd` work with a RAID 1 system and will the new drives know that they will be working in RAID 1??

Last edited by Micro420; 12-18-2006 at 01:10 PM.
 
Old 12-18-2006, 07:06 PM   #255
saikee
Senior Member
 
Registered: Sep 2005
Location: Newcastle upon Tyne UK
Distribution: Any free distro.
Posts: 3,398
Blog Entries: 1

Rep: Reputation: 112Reputation: 112
Quote:
Originally Posted by Micro420
THANK YOU! Very helpful post with examples. Why can't MAN pages, in general, contain examples such as this?

Question: I have my Ubuntu system running on RAID 1 with 2x160GB hard drives. I am wanting to expand them to 2x750GB hard drives also running in RAID 1. Will `dd` work with a RAID 1 system and will the new drives know that they will be working in RAID 1??
I believe that is down to the kernel. If the kernel recognises the RAID disks as you want it to then you are home dry.

The kernel does have to cope with different chipsets and if it fails the two disks in the RAID may be treated as two separate disks.

You should have a pretty good idea of the hard disk recognition if you issue the "fdisk -l" command requesting all disks to be displayed with their partitions tabulated.
 
  


Reply

Tags
backup, best, clonezilla, cloning, command, data, dd, disk, drive, duplicate, erase, explanation, formatting, ghost, hard, image, iso, memory, ping, popular, recover, recovery, rescue, search, security, stick, upgrade, usb, wipe


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Learn The DD command AwesomeMachine Linux - Newbie 17 08-17-2006 05:22 AM
The best way to learn? iz3r Programming 7 02-07-2005 12:00 AM
Best way to learn Linux from the command line patpawlowski Linux - General 2 03-01-2004 04:37 PM
I want to learn C. KptnKrill Programming 14 12-18-2003 02:03 PM
Best way to learn.... InEeDhElPlInUx Linux - Newbie 5 10-11-2003 02:02 AM


All times are GMT -5. The time now is 03:54 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration