By Disillusionist at 2009-10-27 13:05
Splitting and Merging files using dd
When copying a 15G binary file onto a temporary storage device (to transfer it from a failing machine to a replacement machine), I hit a 4G file size limit and the copy process crapped out.
Rather than use split and cat, I decided to play with dd as I didn't trust split and cat to play well with my binary data file.
*** Always make sure you have the Input (if) and Output (of) files specified correctly! ***
Split the file
First task was to split the 15G file into 3G chunks using dd
Getting the first part of the file is pretty straight forward, my block size is 1024 bytes and I want 3145728 blocks (3G)
Getting the next part of the file is a little more complex, I want to skip the blocks that I have already copied and get the next 3G portion:
dd if=big_file of=part1 bs=1024 count=3145728
Subsequent parts are just a case of increasing the value to be skipped:
dd if=big_file of=part2 bs=1024 count=3145728 skip=3145728
Merge the files
dd if=big_file of=part3 bs=1024 count=3145728 skip=6291456
dd if=big_file of=part4 bs=1024 count=3145728 skip=9437184
dd if=big_file of=part5 bs=1024 count=3145728 skip=12582912
When it comes time to put the pieces back together, it is a very similar syntax, just replace skip with seek and make sure you have the if and of files specified correctly!
dd if=part1 of=new_file bs=1024 count=3145728
dd if=part2 of=new_file bs=1024 count=3145728 seek=3145728
dd if=part3 of=new_file bs=1024 count=3145728 seek=6291456
dd if=part4 of=new_file bs=1024 count=3145728 seek=9437184
dd if=part5 of=new_file bs=1024 count=3145728 seek=12582912