Visit Jeremy's Blog.
Go Back > Linux Answers > Applications / GUI / Multimedia
User Name


By Disillusionist at 2009-10-27 13:05
Splitting and Merging files using dd

When copying a 15G binary file onto a temporary storage device (to transfer it from a failing machine to a replacement machine), I hit a 4G file size limit and the copy process crapped out.

Rather than use split and cat, I decided to play with dd as I didn't trust split and cat to play well with my binary data file.

*** Always make sure you have the Input (if) and Output (of) files specified correctly! ***

Split the file

First task was to split the 15G file into 3G chunks using dd

Getting the first part of the file is pretty straight forward, my block size is 1024 bytes and I want 3145728 blocks (3G)

dd if=big_file of=part1 bs=1024 count=3145728
Getting the next part of the file is a little more complex, I want to skip the blocks that I have already copied and get the next 3G portion:

dd if=big_file of=part2 bs=1024 count=3145728 skip=3145728
Subsequent parts are just a case of increasing the value to be skipped:

dd if=big_file of=part3 bs=1024 count=3145728 skip=6291456
dd if=big_file of=part4 bs=1024 count=3145728 skip=9437184
dd if=big_file of=part5 bs=1024 count=3145728 skip=12582912
Merge the files

When it comes time to put the pieces back together, it is a very similar syntax, just replace skip with seek and make sure you have the if and of files specified correctly!

dd if=part1 of=new_file bs=1024 count=3145728
dd if=part2 of=new_file bs=1024 count=3145728 seek=3145728
dd if=part3 of=new_file bs=1024 count=3145728 seek=6291456
dd if=part4 of=new_file bs=1024 count=3145728 seek=9437184
dd if=part5 of=new_file bs=1024 count=3145728 seek=12582912

by colo on Sun, 2010-06-13 14:39
While nice, this is a somewhat pointless exercise - neither `split` nor `cat` (nor any other program using fopen() or open() and friends) treat their stdin and stdout in any special way, and it's just as safe to use them for "binary" data as it is for "text" data. POSIX OS are not fundamentally broken like MS-DOS and its successors, insofar as they don't differentiate between "text" and "binary" files; it's "binary" all the way, every time.

by ikeeki on Fri, 2014-03-07 10:00
Hi, this post is old, but I want to thank you anyway. I'm Ikeeki from cubieforums and I need this info to develop an online flasher for armhf based socs devices. Thanks again, info is NEVER useless


All times are GMT -5. The time now is 03:11 AM.

Main Menu
Write for LQ is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration