[SOLVED] Splitting and merging large file (~1.5TB) with dd involving multiple hard drives
Linux - SoftwareThis forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
Splitting and merging large file (~1.5TB) with dd involving multiple hard drives
Hi All,
First, I hope this is the correct forum/category to post this question, it's kind of in between hardware and software.
I describe my hardware/OS environment at the end.
The issue I'm about to solve is that I've experienced recently a hard drive failure on a 2TB hard drive. As I write, I'm cloning the failed/corrupted drive with the help of ddrescue to another 2TB hard drive into an image. As of good practice, I'll copy the cloned image so that I can safely use the cloned image for data recovery. Here are the critical issues in actually copying this cloned image:
1. I only have free space of 2TB distributed across multiple drives (please note that I'm already using 2 x 2TB drives, i.e. the original and the one hosting the cloned image)
2. The drive hosting the cloned image gave me this warning when running fdisk:
"The device presents a logical sector size that is smaller than
the physical sector size. Aligning to a physical sector (or optimal
I/O) size boundary is recommended, or performance may be impacted.
3. When I'll copy the cloned image, I'll do it by splitting the image into multiple parts, so that each part will fit to a drive providing that much of space.
Questions:
1. When I define the block size for splitting the cloned image, what block size should I be using (512, 4096 or else)? Please note the warning message and the difference in logical / physical sector size above.
2. When it comes to merging the parts, is it a problem that different drives might have different sector sizes? If so, do I need to use the block size of the actual drive hosting the part at hand, or the one I used to split the cloned image file? Or should I use the block size of the target drive?
Thank you so much for your help!
Best,
zahoo
Environment: Debian testing (installed ~8 months before the current release of Wheezy) for 64bit arch (amd64), kernel: 3.2, Intel Core i5-750 CPU, Asus P7P55D motherboard.
Personally I would trust the software. Pick a (large) multiple of 4k as a "bs" - "dd" just acts on sequential sectors regardless. There might be a performance penalty, but in a recovery scenario that should be the last of your concerns.
For "ddrescue" on the split file, that's a bit more interesting as it only reads the allocated sectors - and tends to be all over the place (as in deliberately non-sequential). I might be inclined to also let it default to 512 - you might be able to just feed it the segments separately and set the appropriate (hardware) sector size, but I don't know if I'd bother.
So I started the ddrescue with default settings. In the first step I defined the source partition, the target image and a logfile and used the -n flag to avoid bad sector splitting. No blocksize was defined. In the next step, I used the -d -r3 flag, and source partition, target image and logfile to zero in on the bad sectors. Once the procedure was over, took about 2 days (step 1 and 2), I tried to mount the image which I could. Since everything seemed to be the way it was before the disk error, I started copying the files instead of my original goal, i.e. first backing up the cloned image by splitting it. Anyway, your answer was very helpful, it has reassured me that getting the blocksize right has to do with performance as opposed to the outcome of dd, i.e. success/failure.
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.