-   Linux - Software (
-   -   Segmenting archives into a variety of smaller archives. (

ktek 09-11-2008 09:44 PM

Segmenting archives into a variety of smaller archives.
Hey, can one cut an archive into a variety of smaller archives of x size?
Certainly there are many applications that do that, I was just wondering of there was anything that came with basic linux, or, failing that, what is a recommended full-featured, well docced, command-line based app to do just that?

i92guboj 09-11-2008 09:56 PM

The command line "split" comes with coreutils, so it should be present in most linux, even in a minimal installation or livecds.

It's easy enough to use, and all the info is in "man split".

To join the pieces you can use cat:


cat file1 file2 file3 > big_file
It really can't be any simpler than that.

ktek 09-11-2008 11:41 PM

Thanks. I did a little research before posting but was going at it from the wrong angle it seems.

jschiwal 09-12-2008 01:06 AM

I once backed up my home directory before performing a fresh install on my laptop. The external drive available was formatted in FAT32, and the tar archive would be larger than the 2 or 4 GB limit of the filesystem. I used spit.

To restore you don't need to reassemble the parts. You can pipe the output of the pipe command into your tar command.
cat backup* | tar -C /dest/dir/ -xf -


An option of tar allows you to create a multivolume archive at the start. You restore from the first volume and it will prompt you for the second. I don't remember the full details. I think you can use the -f option multiple times for each volume on the command line. If it is a local file and not a tape device, then either you won't be prompted or you can supply a default prompt with < <(yes y) at the end of the command. Please check the info tar manual to make sure.

If you are creating a backup on a local hard drive in order to backup each slice of the backup to a DVD, it can be tricky restoring from DVD's if you can't reassemble the original archive. This can happen if you don't have an external drive, or a spare partition to re-assemble the archive on. The way I've found that works is to create an fifo file for each disc. Then in one terminal cat the file in that dvd into the corresponding fifo. In another shell run a loop cat'ting from each fifo. This way, coming to the end of the file doesn't cause the pipe to be broken.

So for 12 DVDs
Shell 1:
mkfifo fifo-{1..12}
#for each respective disc insert and mount the dvd and cat into the corresponding fifo file interactively
cat /media/dvd/backup_part-1 >fifo-1
cat /media/dvd/backup_part-2 >fifo-2

Shell 2:
for part in fifo-{1..12}; do
cat $part
done | tar -C /dest/dir/ -f -


Forgive me if I went off on a tangent. Given the size of hard disks these days relative to DVDs and the fact that you aren't keeping the archive intact, you might be backing up the slices to a CD or DVD to free up storage.


I'm operating from memory here, from when I tested out the multiple fifo technique on slices of a dd created image. If you may not have the drive space available to reassemble the original, I'd recommend practicing on a smaller archive and slices to perform a practice dry run. Although trying out the multivolume option may be a better idea. Each tar volume is an independent archive unless a large file needed to be split between volumes. So if you can't find disc 3 but you know a program is on disc 4, you can restore from disc 4 directly.


For DVD backups I like the dar command. I cheat by using kdar to set up the backup & restore jobs and then export the jobs as a bash script. If you have dar and libdar installed on a live distro or rescue disk, you can use it and a copy of the bash restore script in an emergency where you con't have X!! access.

ktek 09-25-2008 05:47 AM

Thanks for the edifying post, jschiwal. :)

All times are GMT -5. The time now is 05:37 AM.