LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 12-02-2020, 12:36 AM   #16
Brains
Senior Member
 
Registered: Apr 2009
Distribution: All OS except Apple
Posts: 1,591

Rep: Reputation: 389Reputation: 389Reputation: 389Reputation: 389

Quote:
Originally Posted by blooperx3 View Post
Disregarding the error causing the hdd to stop copying...

If I put a partion table on, partion, and format/ext4 - can i accurately dd the source hdd using the following?

Code:
sudo dd if=/dev/sdx1 bs=16M of=/dev/sdy1
...this way not removing the partition table, etc...

??
Yes, that should work as long as the destination partition is of same or larger size as the source partition. However, you do not need to format the destination partition, format structure will be carried over from the source partition. If the destination partition is larger than the source partition, one "resize2fs" command can add remaining space to the filesystem.
 
1 members found this post helpful.
Old 12-02-2020, 01:29 AM   #17
ondoho
LQ Addict
 
Registered: Dec 2013
Posts: 19,872
Blog Entries: 12

Rep: Reputation: 6053Reputation: 6053Reputation: 6053Reputation: 6053Reputation: 6053Reputation: 6053Reputation: 6053Reputation: 6053Reputation: 6053Reputation: 6053Reputation: 6053
Quote:
Originally Posted by blooperx3 View Post
I did the dd using

Code:
sudo dd if=/dev/sdx bs=16M of=/dev/sdy
At around the 333GB mark, it stopped copying and gave an Input/Output error.
Whatever happened there has nothing to do with the blocksize.

Please always show us full commands & full output, no paraphrasing.
 
1 members found this post helpful.
Old 12-02-2020, 12:25 PM   #18
computersavvy
Senior Member
 
Registered: Aug 2016
Posts: 3,345

Rep: Reputation: 1484Reputation: 1484Reputation: 1484Reputation: 1484Reputation: 1484Reputation: 1484Reputation: 1484Reputation: 1484Reputation: 1484Reputation: 1484
Quote:
Originally Posted by blooperx3 View Post
Disregarding the error causing the hdd to stop copying...

If I put a partion table on, partion, and format/ext4 - can i accurately dd the source hdd using the following?

Code:
sudo dd if=/dev/sdx1 bs=16M of=/dev/sdy1
...this way not removing the partition table, etc...

??
NO! While that would work exactly as you asked, I don't recommend it. The new partition will be exactly the same as the one on sdx and will gain nothing since it is now on the new drive with the same warts and blemishes. You could (space permitting) mount the new partition you just created somewhere like /mnt then copy the device to an image file with
Code:
sudo mount /dev/sdy1 /mnt
sudo dd if=/dev/sdx1 of=/mnt/sdx1.img bs=16M
Now you have an image file to work with instead of the original, possibly failing, sdx. You might then look at ddrescue or photorec as a tool to assist in recovery of your files from the image. Ddrescue scans the entire drive or image attempting to make a clean image. Photorec scans the entire drive or image attempting to recover files.

You also could use ddrescue or photorec on the original drive or partition but that is not recommended since the recovery efforts would hammer the mechanical drive and potentially cause cascading failure.
 
1 members found this post helpful.
Old 12-02-2020, 04:00 PM   #19
rnturn
Senior Member
 
Registered: Jan 2003
Location: Illinois (SW Chicago 'burbs)
Distribution: openSUSE, Raspbian, Slackware. Previous: MacOS, Red Hat, Coherent, Consensys SVR4.2, Tru64, Solaris
Posts: 2,803

Rep: Reputation: 550Reputation: 550Reputation: 550Reputation: 550Reputation: 550Reputation: 550
Quote:
Originally Posted by blooperx3 View Post
Disregarding the error causing the hdd to stop copying...

If I put a partion table on, partion, and format/ext4 - can i accurately dd the source hdd using the following?

Code:
sudo dd if=/dev/sdx1 bs=16M of=/dev/sdy1
...this way not removing the partition table, etc...

??
I'm scratchin' my head trying to understand why you're trying to bend 'dd' -- which is intended to do block-by-block copying of objects like devices and/or individual files and not directory trees -- into a tool for backing up multiple files. There are much, much better tools whose purpose in life is to copy files. Some of them have even been mentioned above. And most of those -- esp. rsync, find|cpio, and, I'm reasonably sure, "cp -R" (actually don't use that one too much myself) -- avoid the problem with spaces-in-filenames that your original post mentioned that you were worried about.
 
1 members found this post helpful.
Old 12-03-2020, 09:15 PM   #20
blooperx3
Member
 
Registered: Nov 2020
Posts: 67

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by Turbocapitalist View Post
Thanks. If they had been OpenZFS, then there would have been additional options but you will be able to make a fast and accurate copy across two EXT4 partitions using rsync, unless you are dealing with millions of files. I'm not sure where the threshold is but at that level you'd have to split up the transfer into smaller hierarchies.

Code:
sudo rsync -av --dry-run /source/dir/ /destination/dir/
There are actually millions of "items" - will this command still work without error?
 
Old 12-03-2020, 09:22 PM   #21
blooperx3
Member
 
Registered: Nov 2020
Posts: 67

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by computersavvy View Post
NO! While that would work exactly as you asked, I don't recommend it. The new partition will be exactly the same as the one on sdx and will gain nothing since it is now on the new drive with the same warts and blemishes. You could (space permitting) mount the new partition you just created somewhere like /mnt then copy the device to an image file with
Code:
sudo mount /dev/sdy1 /mnt
sudo dd if=/dev/sdx1 of=/mnt/sdx1.img bs=16M
Now you have an image file to work with instead of the original, possibly failing, sdx. You might then look at ddrescue or photorec as a tool to assist in recovery of your files from the image. Ddrescue scans the entire drive or image attempting to make a clean image. Photorec scans the entire drive or image attempting to recover files.

You also could use ddrescue or photorec on the original drive or partition but that is not recommended since the recovery efforts would hammer the mechanical drive and potentially cause cascading failure.
Does photorec look for "all files on the hdd" including both non-deleted and deleted, or just 'deleted' files?

I used
Code:
sudo dd if=/dev/sdx bs=16M of=/dev/sdy
...worked without an apparent problem, copying almost 1 million items, but the 'target' hdd did have 3,000 less files in the end. The most important thing are the deleted files.
 
Old 12-03-2020, 09:28 PM   #22
blooperx3
Member
 
Registered: Nov 2020
Posts: 67

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by rnturn View Post
I'm scratchin' my head trying to understand why you're trying to bend 'dd' -- which is intended to do block-by-block copying of objects like devices and/or individual files and not directory trees -- into a tool for backing up multiple files. There are much, much better tools whose purpose in life is to copy files. Some of them have even been mentioned above. And most of those -- esp. rsync, find|cpio, and, I'm reasonably sure, "cp -R" (actually don't use that one too much myself) -- avoid the problem with spaces-in-filenames that your original post mentioned that you were worried about.
Decided not to use the partitions in the command. My reason was just trying to copy "to" an ext4 hdd filesystem - the source hdd is not ext4.

For just copying files I will use rsync.
 
Old 12-04-2020, 05:26 AM   #23
blooperx3
Member
 
Registered: Nov 2020
Posts: 67

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by blooperx3 View Post
There are actually millions of "items" - will this command still work without error?
After rereading the answers posted, i see you say to break up the files if a large amount. Thanks.
 
Old 12-04-2020, 05:27 AM   #24
blooperx3
Member
 
Registered: Nov 2020
Posts: 67

Original Poster
Rep: Reputation: Disabled
It seems safer to break up the whole hdd into multiple partitions in case there is a problem with opening 1 partition, and that were the only partition on the entire hdd. With multiple partitions, less chance data loss - my opinion after reading.

Agree/ disagree?
 
Old 12-04-2020, 06:00 AM   #25
Brains
Senior Member
 
Registered: Apr 2009
Distribution: All OS except Apple
Posts: 1,591

Rep: Reputation: 389Reputation: 389Reputation: 389Reputation: 389
Quote:
Originally Posted by blooperx3 View Post
It seems safer to break up the whole hdd into multiple partitions in case there is a problem with opening 1 partition, and that were the only partition on the entire hdd. With multiple partitions, less chance data loss - my opinion after reading.

Agree/ disagree?
Absolutely, data recovery a lot faster on smaller partitions also. Which gets the partition re-formatted and back in service much faster after a Whoops! also. But ultimately, have a backup drive of same or larger size and back up your data, then you don't have to mess with recovery efforts in a Whoops! event. Not hard to back up operating systems in compressed files also.
 
1 members found this post helpful.
Old 12-04-2020, 06:40 AM   #26
blooperx3
Member
 
Registered: Nov 2020
Posts: 67

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by Brains View Post
Absolutely, data recovery a lot faster on smaller partitions also. Which gets the partition re-formatted and back in service much faster after a Whoops! also. But ultimately, have a backup drive of same or larger size and back up your data, then you don't have to mess with recovery efforts in a Whoops! event. Not hard to back up operating systems in compressed files also.
Thanks.
 
Old 12-04-2020, 08:55 PM   #27
blooperx3
Member
 
Registered: Nov 2020
Posts: 67

Original Poster
Rep: Reputation: Disabled
NOT WORKING...

Quote:
Originally Posted by Turbocapitalist View Post
Thanks. If they had been OpenZFS, then there would have been additional options but you will be able to make a fast and accurate copy across two EXT4 partitions using rsync, unless you are dealing with millions of files. I'm not sure where the threshold is but at that level you'd have to split up the transfer into smaller hierarchies.

Code:
sudo rsync -av --dry-run /source/dir/ /destination/dir/
Code:
sudo rsync -av --dry-run /media/mnt/K /mnt/6BLACK.COPY.HDD2 
sending incremental file list
K/
K/K.1 (copy 1).odt
K/K.1.odt
K/COPIES/
K/COPIES/K.1 (copy 1).odt
K/COPIES/K.1.odt

sent 248 bytes  received 36 bytes  568.00 bytes/sec
total size is 91,592  speedup is 322.51 (DRY RUN)
NOTHING SHOWS UP IN DESTINATION DIRECTORY "/mnt/6BLACK.COPY.HDD2"

TRIED 2 WAYS – WITH & WITHOUT “/” AFTER SOURCE DIRECTORY AND DESTINATION DIRECTORY

sudo rsync -av --dry-run /media/mnt/K /mnt/6BLACK.COPY.HDD2

sudo rsync -av --dry-run /media/mnt/K/ /mnt/6BLACK.COPY.HDD2/

The result above is from the first one; using second one, nothing reported to transfer based on terminal output --- just zeros show up for 'bytes received' 'speedup'...........

Last edited by blooperx3; 12-04-2020 at 09:01 PM.
 
Old 12-04-2020, 09:28 PM   #28
Turbocapitalist
LQ Guru
 
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 7,313
Blog Entries: 3

Rep: Reputation: 3723Reputation: 3723Reputation: 3723Reputation: 3723Reputation: 3723Reputation: 3723Reputation: 3723Reputation: 3723Reputation: 3723Reputation: 3723Reputation: 3723
Quote:
Originally Posted by blooperx3 View Post
NOTHING SHOWS UP IN DESTINATION DIRECTORY "/mnt/6BLACK.COPY.HDD2"
Correct. That is by design.

Code:
man rsync

sudo rsync -av --dry-run /media/mnt/K/ /mnt/6BLACK.COPY.HDD2/
There are three options there, each explained in the manual page. Even if you don't like to, it is important to look at each of the options at least once in the manual page, especially before trying random formulas from the net.
 
1 members found this post helpful.
Old 12-04-2020, 09:30 PM   #29
blooperx3
Member
 
Registered: Nov 2020
Posts: 67

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by Turbocapitalist View Post
Correct. That is by design.

Code:
man rsync

sudo rsync -av --dry-run /media/mnt/K/ /mnt/6BLACK.COPY.HDD2/
There are three options there, each explained in the manual page. Even if you don't like to, it is important to look at each of the options at least once in the manual page, especially before trying random formulas from the net.
Thanks. I didn't think that the command you posted was random - i felt i could trust it at face value.
 
Old 12-04-2020, 09:47 PM   #30
Turbocapitalist
LQ Guru
 
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 7,313
Blog Entries: 3

Rep: Reputation: 3723Reputation: 3723Reputation: 3723Reputation: 3723Reputation: 3723Reputation: 3723Reputation: 3723Reputation: 3723Reputation: 3723Reputation: 3723Reputation: 3723
Quote:
Originally Posted by blooperx3 View Post
Thanks. I didn't think that the command you posted was random - i felt i could trust it at face value.
I think everyone here tries, but errors do creep in. The formula has a safety added to reduce the likelihood of damage in the event of an error in either the formula or its application. Again, the idea is that you must take a quick look in the manual page at the program's description and each of the three options before it will actually write.
 
1 members found this post helpful.
  


Reply



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] Copying over files from 1 hdd to another hdd - will CL transfer faster than "copy/paste" ? duupunisher2x Linux - Newbie 15 09-01-2020 10:30 PM
copied files from SOURCE to TARGET but now want to rsync SOURCE to existing TARGET teddymills1 Linux - Newbie 3 10-28-2019 07:07 AM
copy installation from one HDD to another, which directory NOT to copy? bfzhou Linux - Newbie 9 03-16-2016 11:59 AM
Want to change how pasted files are named "(copy)," "(another copy)," "(3rd copy)" L a r r y Linux - Desktop 3 08-24-2013 03:39 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 12:53 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration