Can dd restore the entire hard disk including partition table and MBR?
 

Hi!

I issued the following command before.

dd if=/dev/hda | gzip -c | split -b 2000m - /mnt/sda1/backup.img.gz

I recently bought a new hard drive which is larger than my old backup that I copied. I just want to ask if it is safe to restore everything. I heard that the partition table or the MBR stores information of the hard drive itself. Such as the size of the hard disk, how many partitions it has. If I restore the entire harddisk image copy from my previous hard disk, will it affect the performance of the new hard drive? My previous hard disk is just 40 GB and the new one is 300 GB.

If ever I issue :

cat /mnt/sda1/backup.img.gz.* | gzip -dc | dd of=/dev/hda1

Can I still maximize the hard disk space of the new 300 GB hard drive? What are your recommendations? I backed up my hard drive before using partimage but it seems that partimage only back up partition one-by-one and not the entire hard disk. I needed to re-install the MBR and bootloader which I think is not a complete backup. Hope someone can help me out. Thanks and happy new year to everyone in LQ.

I've only used dd to restore individual partitions, but there should be no problem doing what you are describing. It will copy the MBR, including the partition table, and all partitions. It is a variation on one of the hacks (#48) described in the O'Reilly book Knoppix Hacks by Kyle Rankin.

The only issue you should have is that the new disk is larger. The image you copy will be the same size as the old, so you will have unpartitioned space on your new disk. So you will probably want to add new partitions or expand your last existing partition using something like Gparted or Qtparted. If all of your partitions are primary, this is straightforward. If you have logical partition(s) (on Linux, those numbered 5 and above), then you will first need to expand the extended partition (presumably to the end of the disk). You can also use the tools mentioned above for this.

The worst that can happen (if you don't tell dd to write its output to the wrong disk!) is your new disk will contain garbage. You won't damage it. If you end up with garbage, just figure out what you did wrong and try again. :)

NOTE: The above is assuming you are running an x86 system, i.e. not PPC or something which might use disks with a different structure.

Okay...very well said. Do you think "dd" command is better than partimage to backup hard drive? I tried partimage but it didn't restore the bootloader and the MBR? How can I copy the MBR and bootloader using "dd"? How about Ipod? Do you know a way to transfer my music and videos from a 30 GB to a 80 GB Ipod without the use of Itunes? I always use gtkpod and not Itunes and I just want to transfer my music and videos to a bigger Ipod which is 80 GB. I wanted to try "dd" but I am just to scared to mess things up. If you're saying that if I transfer the partition image from a smaller to a bigger HD that I might have unpartitioned space. How hard it is to do in qtparted? Thanks.

Imaging disks is a mugs game.
Restoring the MBR (especially partition table) to a different disk is even worse.

I know, I know, everyone ('cept me maybe) says it is the way to go.
There are a bazillion (give or take a few) ways to back up data - almost all of them are preferable to "imaging".
If you absolutely *have* to, create the new (hopefully larger) disk partitions and restore individual partitions as needed. Resize as desired. If it works, all well and good. Else start copying to yet another (correctly sized) partition.
Like should have been done initially.

Sorry folks, I've seen this fail too often.
The only time I use dd to image a partition is when I need a copy of a partition that I am about to attempt recovery on. If (when) it all goes to crap, I use the image to start over, as blackhole54 mentions.

syg00, I completely agree. I've moved entire installations from one disk to another. I would never ever use dd for that. I do it as follows:

First, as the target disk has a different size, I create the file system manually. This allows the mkfs program to choose the correct number of inodes (in case of ext2/3), for example. Or I can change the file system if I wish.

Then,
- If both are mountable on a single computer, I just use a LiveCD and "cp -a".
- If this cannot be done, I use a different medium (DVDs, external hdd), and copy either with "cp -a" or "tar".

Finally,
I "chroot" into the target partition, reinstall the boot loader and adapt /etc/fstab.

This has never failed for me.

depam, I have no experience with partimage or ipods.

If you use dd to copy the partition table (by itself, or as part of a full disk copy) to a larger disk, you will have unpartitioned space as a result. The programs I mentioned (and I would guess partimage) are supposed to have the ability to enlarge petitions, but uselpa might have a good point about inodes. I have never done it, so I can't speak from experience.

What I have done w/o problem is use parted (a command line program that uses some of the same libraries as Gparted and Qtparted) to shrink a FAT32 partition, and used dd to save and restore FAT32 partitions and MBRs. And I have only done this a few times.

For the unpartitioned space, in addition to (trying to) expand the size of a partition, you have the option of adding one or more new partitions. (I don't know what the consequence of this is for an ipod). As I've already mentioned, you might have to first expand the size of the extended partition (which "houses" the logical partitions), which I think is just a matter of changing the primary partition table (on the MBR), and as such I would think would be no problem for any of these programs.

If this is for an ipod, you might want to consider what type of filesystem it uses. I thought it was some flavor of FAT. If so, I don't think FAT uses inodes, and so you might have a higher probability of success.

In any case, TMK, you can't physically screw up a disk this way. So if something goes wrong, all you've done is failed. But I would not alter the original disk until you are sure you have the new one working properly.

blackhole54,

Thank you for providing a very useful post. Can you please post a detailed commands that you usually do to backup and restore your hard disk? Though I have experiences linux, I just don't do this very often. I have read alot of things about backup and restore using linux and it seems really confusing because most of them offer options which I am really not sure which one is going to work. I am kinda looking for a step-by-step backup and recovery which is already tested. And since you have been doing this, I am thinking if you can help me out by letting me know the commands that you've typed and the process itself. My other problem is that I have to backup ntfs which some of the threads that I've read is not supported yet by other linux kernels. Do you have experiences on this? The "parted" command which you stated does not support NTFS. Maybe you can give me a hand on this. Thanks.

deleted due to double post

I am backing up my hda and hdc disks, both 300Gb, one with 63 partitions and the other 59 partitions. They are cloned to disks of identical size. Here is the command I am using.
I use mobile racks which are internal frames for accepting hard disks in caddies. I removed the hdc disk and in its position put a backup disk for cloning the hda disk. Hardware wise the new disk is recognised as hdc by the Bios.

1st operation took 5784 second at 51.9 Mb/s speed. That is 1.6 hours. dd was issued inside Slax housed in hda62.

2nd operation took 6197 second at 48.4 Mb/s. That is 1.72 hours. I had to use a Slax Live CD because the system boot loader is in hda disk which has been removed temporarily

The block size of 32768 is 64 sectors of 512 bytes representing one full track, an optimal choice for speed from my experience as without bs being specified dd defaults to 512 bytes for each transfer operation.

In the second operation I slotted the hdc disk into the hda position and put a back up in the vacant hdc slot.

Were the operations successful? Well I am answering the post with Wolvix Linux, booted by Grub residing in the new hda, called up from the partition hdc59 of the newly backup hdc disk.

For those worry about dd here is my experience

(1) dd always work for identical size disks or from a small one into a big one.

(2) dd does not work for a large disk into a small one.

(3) If the target disk is bigger the excess capacity becomes unallocated space. This is not hard to understand because the cloned partition table has no reference of the excess capacity.

(4) Linux has made a significant inroad into resizing partitions and the current leaders are gParted V3.3 and Parted Magic v 1.0. The current Parted Magic mainatainer used to work for gparted. These two software have Live CD iso images freely available for download. You will find them on par if not better than the commercial products. My two disks had 60 and 57 partitions last week and I used Parted Magic to resize the partitions to get 5 more and have just finished installing distros in them.

(5) I used to rely on the commercial software to clone disks but never have a need to go back to them after I gone into Linux. When conducting comparison tests I found dd faster and more reliable.

Two above disks provide the materials for writing this thread

My purpose of ansering this thread is I think unwarranted criticisms of dd are unjustified. It is a simple Linux Bash command that only duplicates faithfully the binary pattern of the hard disk sector by sector. The number of sectors is dictated by the user. It is simple and fast. qParted document states many of their tasks are accomplished by dd.

G'day saikee - I wasn't knocking dd, I use it all the time.
But I don't use it (or partimage, or anything else) for imaging my data. If you hit an error with dd guess what it does (by default) ???.
Now go back and look at the command used by the OP. How would you know if it had an error ???.
I just won't do it - but everybody can make their own choice.

dd by default in cloning a hard disk must do all the sectors in the source disk and the MBR is the first sector to be copied. It is therefore impossible for the target disk not to boot exactly as the original, unless the geometries of the two disks differ significantly. I use large disks all with 255 heads and 63 sectors per track and so never run into problem myself.

dd doesn't not give any information on its progress untill the work is done. Generally the hard disk light flickering frequency tells us how busy dd is with its work. If a user wants to know the time it takes he/she can issue the "date" command before and after the dd command so that the exact second is recorded. Newer kernels report automatically the time taken by dd. From experience between 35 to 50Mb/s would be normal speed for dd when doing internal disks. Sata normally exceeds the upper limit but not by much. External disks are bottlenecked by the USB transmission rate.

In cloning dd reads the binary pattern of one disk and write the same pattern on the other disk as a mirror image. As such if one has rubbish in the source he/she gets a 100% of the same rubbish in the target. dd does not know how to cheat.

An area dd can get into trouble is when one of the disk is unreadable or cannot be written on due to hardware corruptions. In such a case no system in the world can do better than dd.

I have just cloned two 300Gb disks with 122 Dos, Windows and Linux inside and immediately use a Linux from the newly cloned disks to do the reply here. I do not expect dd to fail me with the command indicated in Post #9.

I feel sad for users not understand how simple dd is and develop misconceptions about it ability.

For all intent and purpose dd can be viewed as a carbon paper that a user uses to produce a duplicate for his/her work. The second page gets whatever he/she writes on the first page. If he/she uses the second page larger than the first then there is no information lost. However if the second page is smaller than the first page then information is truncated as the latter part of the written work has no where to copy onto.

Can we blame the carbon paper?

-----------------------------------------------

I am a beneficiary of dd command and so I put my case forward with the following result just came out a few minutes ago.

Only half an hour ago I decided to do a backup for my office latop which has a 2.5" 80Gb disk fully of years of work. It has 3 partitions which I initially wanted to dd one partition at a time using command like
as I am using an external 200Gb 3.5" hard disk via a USB port (that is why the source is called sda).

I noticed this little 80Gb has geometry of 240 heads and 63 sectors per track while the large 200Gb disk has the standard 255 heads and the same 63 sectors per track. I was uneasy about fitting the backup into the external hard disk which already holds other backups.

To play safe I got hold of a spare 400Gb disk that also has 255 heads 63 sectors/track geometry and dd the entire content of the 80Gb laptop disk across. The 400Gb has several partitions inside that I could afford to lose.

Without bothering with deleting the existing partitions or formatting the 400Gb disk I jsut hooked the used external disk to the USB port of the Laptop, slip a SLax Live CD and boot it up. The command I used was exactly as per Post #9 except the hdc is changed to sda as I was cloning an internal 2.5" hard disk into an 3.5" external hard disk.

The job took 3200 seconds and the speed was 25Mb/s.

Immediately after cloning the whole of the 400Gb disk has the geometry changed to 240 heads and 63 sectors/track same as the one inside the Laptop but the partitions refuses to mount claiming superblock error in the ntfs partition. Did I worry? No. This is what I expect from dd. I suspected the Linux kernel could be reading the 400Gb with the old geometry, as no one tells it otherwise, and so I rebooted.

After a reboot I see a complete mirror image of all the laptop disk partitions and Slax has mounted all of them automatically. As I could log in as root in Slax and so I inherit XP's Admin right and so can access every file of my XP system now in the external hard disk.

In the past I had used dd on 2.5" external hard disks to clone a few laptop XP systems and then swapped the disks because the owners wanted a larger one inside the laptop.

I really don't find dd a problem to work with at all. I delivers exactly what it is spposed to do ----> just like a carbon paper would do between two sheets of papers.

saikee has already shown how to do a disk to disk copy (and provided a deeper level of discussion of the issues than my knowledge would allow -- thank you saikee). My experience with copying FAT32 was to backup a win95 (don't ask) partition for (possible) later restoration to the same partition. I used ssh to send the data to a file on another machine using the default block size. The backup and restore (run as root) commands would look somthing like:

If you want to do compression, you can pipe the data through gzip/gunzip at either end.

I have no experience with that. But since dd does a bit-by-bit copy, it doesn't have to understand what it is copying (the carbon paper analogy was good). That is one of the reasons I used it for copying FAT32; to my knowledge Linux doesn't understand the FAT file attributes, but to dd they are just bits w/o any particular meaning. So I would expect it to handle NTFS fine.

You may have noticed I kind of hedged my language when I talked about parted. That was because I was well aware that it did not handle NTFS while Gparted and Qtparted did. These GUI programs use one or more additional libraries for that. Or so I understand -- I have no direct experience with them. If perchance you are using MS Windows Vista, you should be aware that MS has changed NTFS yet again, and the last I knew, none of the free(dom) software programs can handle it yet. But I think they work fine on the older versions of NTFS.

You could always use a md5 checksum or some such to compare original with final. This was exactly what I did back when, when I broke up "large" files and transferred them to another machine with floppies and reassembled them. Of course, the files were a lot smaller so calculating the checksum would go much faster. And in the case of filesystems, it would never hurt to use their native integrity checker.

blackhole54,


What? MS changed their filesystem again? Maybe they are already talking to other backup and restore utility such as Norton Ghost and others to provide them bigger profit. If you believe that dd uses a bit-by-bit copying then what difference does it make if MS Vista would change their filesystem? I don't know but this makes sense but you also told me that dd doesn't know what is being copied and works like a carbon paper? But anyway, thanks for the info man...I really love people like you who share their experiences and ideas without confusing other members and pointing them to other websites. Honestly, I've read a lot of infos which I've searched in google which kinda confused me since one website says absolutely different from the other.

saikee,

So, you have tried to copy windows xp filesystem (NTFS) before using dd? That is exactly what I wanted to know because I do have workstations which I wanted to backup using dd into an image file. I am hesitant to do this since others said that dd will not be able to do the job and they have suggested to use ntfsclone instead. What can you say regarding our discussion on the new Windows Vista filesystem? Do you think dd can't copy this new filesystem? I would also like to thank you for sharing your experiences. Maybe other people have tried different techniques on backing and restoring up data. But for me, I wanted dd because it will copy everything including the MBR which I think is a handy tool and you don't have to do some partitioning or anything. You're right and I think that the qtparted can do the resizing. How about ipod? Have you ever encountered backing up backup using dd? I do have a 30 GB Ipod. We all know that this device is equipped with a real hard drive. Now, I wanted to buy a larger Ipod which is 80 GB. Can I just use dd to copy from one Ipod to another? My only concern is that if it is possible to do this with dd, I think everything will be written on the new one and will overwrite even the software, firmware upgrades and maybe some of the features of the new one. If you have another idea, please let me know.


Thanks everyone and I am very pleased that there are still people like you who is willing to help other members.

depam,

I must state up front that I do not keep the hard disk image as file, compress it for a later restoration. I always clone the hard disk to hard disk and use the second image while putting away the original disk as back up. Thus I know dd works but could not comment on effect to moving the compressed file around and its ultimate restoration. However it is technically sound and many users have reported doing it this way. Having said that it may be relevant to remember the original hard disk geometry and to have it matched in the the eventual restoration if difficulties are encountered. In a disk-to-disk cloning the target disk is forced to to same geometry and one can check it immediately.

Regard the new ntfs system (which I know nothing about if it is there) and if Vista can be cloned my answer is dd wouldn't give a toss. If you write "Vista with a new ntfs system" on the first page, has a carbon paper sandwished in between, what you get in the second page is what exactly dd gives you.

In the Post #9 you will find I have a Vista installed in hdc3. The post describes how I installed it on a stand-alone disk and "dd" it into its final position, as I could not afford Vista to mesh up the 59 partitions in my hdc disk when I installed it the very first time. My reply in Post #9 confirms that I have cloned the 300Gb hdc disk to a second copy and it is the Vista in the second copy I am using now to write this very reply. In other word I am using the clone of a clone of Vista. Therefore as far as dd is concerned you can do whatever with your ipod but if you hook it up as a USB disk to dd it will clone it to any media of your choice.

All OSs, files, boot loaders of software of any description must be stored in bits of either "1" or "0" on a hard disk. dd just copies the "1" and "0" across. Why should you worry about it a Vista, a BSD, a Dos, a Linux...?

saikee,

Very well said. This topic became really, really interesting thanks to all who posted. I already tried "dd" to save the whole hard disk (/dev/hda) into an image file (backup.img.gz). Since FAT32 has a file size limitation, I managed to split the image into 2GB each. After backing up, I just erased everything including the partition table and mbr. When I restored it using dd, whew!! It really worked like a charm. Well, I guess dd can really copy everything or any OS and that makes me really proud of dd and open source. Regarding the ipod issue, my argument is this. Ipod has different models and firmwares installed. For example, I have Ipod 5G Video 30 GB. I tried to plug it in my Linux OS and guess what, Linux doesn't recognize its filesystem. But when you power on the IPOD, it says that the format is "Windows" which i believe is either FAT32 or NTFS. Even if I mount the Ipod, I don't get any information regarding the drive. I also tried "fdisk -l" but the actual capacity reflected is the used space and not the total space of the hard drive. If you have come across backing up and restoring ipod into a higher version (Ipod 80 GB for instance), please let me know how you do it. Thanks alot saikee. You rock man!