Making a bootable clone with dd, partprobe and tune2fs

yendorian · 04-24-2017, 06:23 PM

I know there are alternatives to dd for cloning a disk. I am specifically asking how to clone with dd to create a bootable disk. Please direct your opinions, statistics and assertions of other backup tools elsewhere.

I have successfully copied a drive with dd. I now want to make it bootable. lsblk shows that the original drive and the cloned drive each have partitions of identical size. Then I wanted to let the kernel know that the cloned drive existed so I can boot from it.

I have used partprobe /dev/sdbx on each partition, but I get an error:

Code:

Error: Partition(s) 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13,
 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 
31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 
48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64
 on /dev/sdb1 have been written, but we have been unable to inform the kernel of the change, probably because it/they are in use.  
As a result, the old partition(s) will remain in use. 
 You should reboot now before making further changes.

I first attempted to use umount on each of the /dev/sdb partitions, however all of them said that they were 'not mounted'. I rebooted, figuring the kernel would figure it out where things were.

When I went to use tune2fs to give new IDs to any of the partitions, I got this problem:

Code:

sudo tune2fs /dev/sdbx -U random
tune2fs 1.42.12 (29-Aug-2014)
tune2fs: Bad magic number in super-block while trying to open /dev/sdb1
Couldn't find valid filesystem superblock.

I'm not sure what in the process is preventing the commands from running smoothly. Any ideas on what I can do to get the kernel to recognize the new drive as a bootable device?

schneidz · 04-25-2017, 07:25 AM

what was your dd command ?
whats the result of this command:

Code:

file /path/to/image

syg00 · 04-25-2017, 07:29 AM

Quote:

Originally Posted by yendorian

II have successfully copied a drive with dd. I now want to make it bootable.

Had that been true, it would be bootable.

My recommendations are, as you demanded, elsewhere.

jailbait · 04-25-2017, 10:13 AM

Welcome to linuxquestions

Quote:

Originally Posted by yendorian

I have successfully copied a drive with dd.

No, you haven't. The following error messages indicate that the file systems on your cloned partitions are broken.

Quote:

Originally Posted by yendorian

I have used partprobe /dev/sdbx on each partition, but I get an error:

Code:

Error: Partition(s) 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13,
 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 
31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 
48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64
 on /dev/sdb1 have been written, but we have been unable to inform the kernel of the change, probably because it/they are in use.  
As a result, the old partition(s) will remain in use. 
 You should reboot now before making further changes.

I first attempted to use umount on each of the /dev/sdb partitions, however all of them said that they were 'not mounted'. I rebooted, figuring the kernel would figure it out where things were.

When I went to use tune2fs to give new IDs to any of the partitions, I got this problem:

Code:

sudo tune2fs /dev/sdbx -U random
tune2fs 1.42.12 (29-Aug-2014)
tune2fs: Bad magic number in super-block while trying to open /dev/sdb1
Couldn't find valid filesystem superblock.

------------------------
Steve Stites

yendorian · 04-25-2017, 10:24 AM

In case it wasn't clear, I did not use dd to move an .iso or image to a drive. I used a bootable linux distro, used dd on my day-to-day, unmounted hard drive containing my data and primary OS, then moved it to a backup hard drive.

Quote:

What was your dd command ?

dd if=/dev/sda of=/dev/sdb bs=64K conv=noerror,sync

Quote:

whats the result of this command:
file /path/to/image

I didn't use an image, I used an actual hard drive I was trying to clone. The results of file /dev/sdb are:
/dev/sdb: block special (8/16)

If it matters, file /dev/sda outputs
/dev/sda: block special (8,0)

My understanding is that dd in and of itself is not sufficient for creating a bootable disk when cloning a hard drive. Part of it is that partition changes made by dd are not seen by the kernel, and the other part of it is that the UUID the disk you are backing up to is overwritten with the UUID of the disk you are cloning, ultimately preventing the bootloader from being able to distinguish between the two. Partprobe and tune2fs are solutions for these respectively.

Thank you schneidz for a congenial response free from snarky comments. Users like you are what give the linux community a good name and make it fun to be a part of

schneidz · 04-25-2017, 11:26 AM

^ i experimented with stuff like this in the past.
making sure that my /etc/fstab, /boot/grub/grub.lst, ... didnt use uuid's (e.g.- i would edit the (hda,0) entries to whatever was appropriate so that grub could find the init.rd and vmlinuz, ...).

yendorian · 04-25-2017, 01:56 PM

I ran blkid to see what my drives actually have and found some interesting output:

Code:

/dev/sda1: UUID="4075-504B" TYPE="vfat" PARTUUID="6e2fb032-414a-4f53-8efe-d93a241d0e7a"
/dev/sda2: UUID="b1fedb24-548d-4161-9aa1-233fbbfbd8cd" TYPE="ext2" PARTUUID="fb8a79da-a2a4-40d3-abd6-1c666db96b7c"
/dev/sda3: UUID="31de72cc-44fa-448d-9ef9-9fd09e2c3121" TYPE="crypto_LUKS" PARTUUID="6ad5156a-eb7f-4892-95b0-deab00dd0b25"
/dev/mapper/polymorph--vg-swap_1: UUID="9917639d-edd4-410a-8aa1-65f4f04285f2" TYPE="swap"
/dev/sdb1: PARTUUID="6e2fb032-414a-4f53-8efe-d93a241d0e7a"
/dev/sdb2: PARTUUID="fb8a79da-a2a4-40d3-abd6-1c666db96b7c"
/dev/sdb3: PARTUUID="6ad5156a-eb7f-4892-95b0-deab00dd0b25"

My cloned drive partitions each has a PARTUUID but no UUID. Each PARTUUID was copied identically from the original drive. I ran
sgdisk --partition-guid=1:R /dev/sdb
sgdisk --partition-guid=2:R /dev/sdb
sgdisk --partition-guid=3:R /dev/sdb

Which changed the PARTUUID of each of the cloned drives partitions to something unique as I figured that would be something that needed to be done eventually. Later I'll see if I can get the drive to boot by giving it a manual label, although I'm not sure why the disk clone has no UUID at all. Maybe it has something to do with the original being full disk encrypted?

jefro · 04-25-2017, 03:54 PM

https://superuser.com/questions/6225...ync-noerror-do

If one dd'd a drive to a drive then we'd (linux users) tend to assume that the bit by bit copy of one to another created an exact copy. Since you are playing with trying to fix it, we have to assume as noted above that the clone was in fact not performed.

So it leads us to question why it failed. Was is the block size in statement? Some problem with the disk(s)? Some problem with the device naming? Some padding places in the dd command by the sync option?

You will have to try clone operation again instead of trying to play with fixing the clone.

yendorian · 04-25-2017, 11:13 PM

I attempted to make a second clone by doing some things differently. This time I formatted the drives to have the file systems and partition sizes, even though I don't think the partition sizes are necessary. I ran the same dd command with a block size of 64K. I then ran dd on the 3 existing partitions on the drive I was trying to clone, i.e
if=/dev/sda1 of=/dev/sdb1
if=/dev/sda2 of=/dev/sdb2
if=/dev/sda3 of=/dev/sdb3

Oddly, the two small partitions (< 500MB) responded well to partprobe and tune2fs. Only the ~230GB partition failed with the same errors. I think I'll try this again tomorrow with a smaller block size. That's the only reason I can think of that would allow the other partitions to succeed but the large one to fail, unless the 'failing' partition is failing because it is a crypto_LUKS encrypted partition. Thank you for the feedback about the clone not succeeding.

Brains · 04-25-2017, 11:29 PM

Quote:

Originally Posted by yendorian

dd if=/dev/sda of=/dev/sdb bs=64K conv=noerror,sync

I prefer using dcfldd which defaults to block size, something like:

Code:

dcfldd if=/dev/sda of=/dev/sdb

Works flawlessly every time, notice there is no "conv" stuff, especially "noerror". If I use dd I tend to stick to a byte size of "bs=4096" maximum for hard drives/hard drive partitions, but again, no "conv" options.

In your example, if I'm not mistaken, if there is an error in a 64K block, zeros follow the error for the remainder of that block and dd continues on, so if the file system has 4K blocks, there should be file system errors.
UUIDs are assigned by partition utilities, since the cloned drive was not partitioned may explain the lack of a UUID. Assign the UUIDs that are in /etc/fstab for each partition from a live session, the PARTUUIDs don't mean much from a booting perspective and can also remain the same as the original.

If both drives are hooked up and visible to each other, the operating system may use files from both drives when they are identical, for example:
I have two almost identical Debian testing installations on one computer, only difference is one has Nvidia proprietary drivers, the other does not and uses the nouveau drivers, if I let the non Nvidia installation see the other and boot it, it will boot with full Nvidia support without having it installed.
So it is best to only have the clone attached when trying it out unless you have a way to hide the original.

yendorian · 04-26-2017, 04:57 PM

I think I know what's happening now. After two more attempts using dd and adjusting block size and other things, while still getting the same errors I started to investigate encryption as a factor.

From a liveUSB Linux distro, I used

Code:

cryptsetup luksOpen /dev/sdxn some_label
lvscan
vgchange -ay
mount /dev/mapper/[name of my drive found by lvscan] /media

and the hard drive mounted beautifully, appearing to be a completely valid clone.

My guess is that partprobe and tune2fs don't work innately with encrypted disks without some coalescing. This would explain why they worked just fine on the other unencrypted partitions without error, but choked on my encrypted partition. I have not found out how to modify the UUID of an encrypted drive, but I consider that outside the scope of my original question, so I will troubleshoot that elsewhere.

I hypothesize that the drive would mount with the 3 commands mentioned in the title of this thread if it just wasn't encrypted.

yendorian · 04-26-2017, 05:58 PM

*Update: After decrypting and mounting my partition using the steps above, I was able to assign a new UUID through by running partprobe and tune2fs

Code:

lvscan
vgchange -ay
partprobe /dev/mapper/[name of my drive found by lvscan]
tune2fs -U random /dev/mapper/[name of my drive found by lvscan]

This gave it a unique UUID. I then ran partprobe and tune2fs on my unencrypted swap partition without doing anything special and it worked as it has in the past. Ironically, my boot partition is now throwing the superblock error when I use tune2fs, which is unfortunate because it didn't always do that. Cloning the boot partition again did not resolve the issue.

Anyway, I should be pretty close now. If I can get it working I'll post all steps to reproduce for those interested. I don't think I've seen any walkthrough for creating a bootable clone from an encrypted drive with dd.