How can I dd copy all files from 1 hdd to another but without the target being in an image file?

blooperx3 · 11-30-2020, 03:01 AM

Basically, i want to copy all files from one hdd to another and have the 2 hdd's exactly the same. But on the target hdd, i do NOT want an image file.

Reason for this: when using "cp" not all of my files transfer over because of long names, etc. So it is much faster to just dd them over because all of them do transfer over with dd.

Here is a list of known dd commands, though i am not necessarily sure which situation goes with which command:

Code:

1- “sudo dd if=/dev/sdx bs=8M of=/filepath/directory.name”

2- “sudo dd if=/dev/sde bs=8M of=/directory.name.external.media.mounted.to/directory.name.new.files.go.here”

3- “sudo dd if=/directory/filename bs=8M of=/filepath/new.file.name” ((this seems to be for a single file only))

4- “sudo dd if=/dev/sdx bs=4M of=/path.to.backup.img”

5- “sudo dd if=/dev/sdx bs=4M of=/dev/sdy”

6- “sudo dd if=/filepath/filename bs=8M of=/filepath/new.file.name”

7- “sudo dd if=/filepath/filename bs=8M of=/dev/sdx” (converting an image to an OS???)

Turbocapitalist · 11-30-2020, 03:05 AM

I would use rsync for that, with the -a and -v options. Which filesystem are you using on the source and destination partitions?

blooperx3 · 11-30-2020, 03:17 AM

Quote:

Originally Posted by Turbocapitalist

I would use rsync for that, with the -a and -v options. Which filesystem are you using on the source and destination partitions?

ext4 filesystems for both

Turbocapitalist · 11-30-2020, 03:21 AM

Thanks. If they had been OpenZFS, then there would have been additional options but you will be able to make a fast and accurate copy across two EXT4 partitions using rsync, unless you are dealing with millions of files. I'm not sure where the threshold is but at that level you'd have to split up the transfer into smaller hierarchies.

Code:

sudo rsync -av --dry-run /source/dir/ /destination/dir/

blooperx3 · 11-30-2020, 04:39 AM

Quote:

Originally Posted by Turbocapitalist

Thanks. If they had been OpenZFS, then there would have been additional options but you will be able to make a fast and accurate copy across two EXT4 partitions using rsync, unless you are dealing with millions of files. I'm not sure where the threshold is but at that level you'd have to split up the transfer into smaller hierarchies.

Code:

sudo rsync -av --dry-run /source/dir/ /destination/dir/

What is the benefit of rsync vs dd? I used dd everyday

Code:

sudo dd if=/dev/zero bs=8M of=/dev/sdy

Turbocapitalist · 11-30-2020, 04:45 AM

With rsync you won't make a copy of the file system itself, only the contents. Thus you have a proper blkid and so on. That is a big advantage. However, the primary advantage is that it copies only the changes so you can use it for ongoing updates at a lower resource and time cost. For the same reason it is useful for transfers that might get interrupted, because it can resume where it left off unlike dd.

computersavvy · 11-30-2020, 06:08 PM

Another advantage to rsync is that it will copy files with long names and with spaces in the names where cp does not.

michaelk · 11-30-2020, 06:55 PM

To elaborate using dd i.e. dd if=/dev/sdy of=... is basically a full metal backup of the entire drive. Your copying the drive byte for byte, the partition table, partitions, filesystem metadata, files and of course empty space. Any existing data on the destination drive is overwritten. Not a good method for backing up just files.

rsync copies files from a source directory to a destination directory just like the cp command but does a better job and copying lots of files. As a backup tool it will only copy files that have changed on the destination. In your case the destination directory would be the mount point of your 2nd hard drive.

Brains · 11-30-2020, 09:52 PM

Quote:

Originally Posted by blooperx3

I used dd everyday

Code:

sudo dd if=/dev/zero bs=8M of=/dev/sdy

So what you're saying...
Is that you destroy files every day.
Because that's what that dd command you posted does.

rnturn · 11-30-2020, 11:31 PM

Quote:

Originally Posted by blooperx3

Basically, i want to copy all files from one hdd to another and have the 2 hdd's exactly the same. But on the target hdd, i do NOT want an image file.

Two options come to mind:

Code:

rsync -av $SRCDIR $DESTDIR

and

Code:

cd $SRCDIR ; find . -depth -print0 | cpio --null -pVd $DESTDIR

where SRCDIR and DESTDIR are appropriately defined before executing either command line.

I can see using "dd if=PATH-TO-SRCFILE of=PATH-TO-DESTFILE" on an individual file. But... why?

HTH...

blooperx3 · 12-01-2020, 04:48 PM

Quote:

Originally Posted by computersavvy

Another advantage to rsync is that it will copy files with long names and with spaces in the names where cp does not.

Seems like a good choice!

blooperx3 · 12-01-2020, 04:50 PM

I am going to copy a 1TB hdd bit-for-bit using the following command. To get the most precise dd copy over, what number would work best for "bs=XM"? I've read that the bs=xM has to do with the speed of copying. I'm wondering if the higher speeds result in more errors or a poorer quality clone. For this hdd, I need dd due to deleted files.

Code:

sudo dd if=/dev/sdx bs=16M of=/dev/sdy

I imagine I can use: 1,4,8,16...

blooperx3 · 12-02-2020, 12:02 AM

I did the dd using

Code:

sudo dd if=/dev/sdx bs=16M of=/dev/sdy

At around the 333GB mark, it stopped copying and gave an Input/Output error. Then when i tried to mount it, it said it couldn't mount b/c it couldn't open a certain directory. I restarted the computer and then it mounted.

I don't actually need the files on the desktop of the drive, I just need the deleted files - to try and get them back. My source hdd with 'fuse' filesystem became infected with malware and the hdd itself told me there was a problem with the drive and asked me if I wanted to fix it. I said yes and it deleted many GB's of files.

The hdd was an external hdd, not an internal like my other hdds. Now I am trying to dd my 1TB external hdd to a 1TB internal hdd in a docking station.

blooperx3 · 12-02-2020, 12:08 AM

Quote:

Originally Posted by Brains

So what you're saying...
Is that you destroy files every day.
Because that's what that dd command you posted does.

yes - to clean my micro-sd cards.

blooperx3 · 12-02-2020, 12:12 AM

Disregarding the error causing the hdd to stop copying...

If I put a partion table on, partion, and format/ext4 - can i accurately dd the source hdd using the following?

Code:

sudo dd if=/dev/sdx1 bs=16M of=/dev/sdy1

...this way not removing the partition table, etc...

??