Linux - GeneralThis Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
I have a huge tar file with several subdirectories. Since managing this junk with my old PC is very slow, I'd like to split the tar into several smaller tars, one for each sub-folder. Instead of extract-to-disk, and then create-from-disk, I'd like to extract|create on the fly with a pipe.
From what I read on the manual, and on internet, I should do something like this:
Code:
$ tar -xOf test.tar sub1 | tar -cf -
$ tar -xOf test.tar sub1 | tar -cf sub1.tar
$ tar -xOf test.tar sub1 | tar -cf - sub1.tar -f -
Where the '-O' option sends to STDOUT, and the '-f -' reads from STDIN/OUT. But I am unable to get it right.
Try tar -xOf test.tar sub1 | tar -cf sub1.tar - since you want to create sub1.tar from STDIN.
Note that this may not work as you'd wish if sub1 contains subdirectories or more than one file. In fact, I'd suggest that you try a tar -xOf test.tar sub1 | less before you go too far down this path so you can get a clear idea of what will be in the input stream to the second tar. It seems to me that the -O option will concatenate the contents of all the files in sub1 into a single stream, and that your new sub1.tar file will contain a single, un-named, block of text, which will, probable, be quite difficult for you to disentangle.
Thank you both very much for your reply. Both suggestions produce the following error:
Code:
Cannot stat: No such file or directory.
The second suggestion additionally creates an empty sub1.tar file.
This is an old backup with hundreds (or thousands) of files in subdirectories, and it's stored on an external drive. I didn't want to untar the whole thing, since writing many files is much slower than reading/writing one single file. I'll have to do it overnight.
I don't know if you can do that with tar. Maybe you could do that with cpio however.
It's one thing to do something like this:
tar cf - file1 ... | ( cd targetdir/ ; tar xf - )
The stream traveling down the pipe is the archive, so the first tar command was able to do the serializing of the files & filenames. The stdio stream contains both the files and the names. The second tar command marshals the serial stream into the separate files.
I once performed a backup of my /home directory before performing a fresh installation of another Linux version. The external drive I had handy used the fat32 filesystem which has a limited file size limit. So I piped the output of tar through the "split" command to produce several files. I then used par2create to produce par files in case one of the slices became damages. I could extract from the archive without reassembling it, simply by cat'ing the pieces together and piping the output to a tar extract command. As an alternative, you could use the multi-volume option.
Also look at using dar for producing fixed sized back slices.
Thanks, jschiwal, your cpio suggestion is really interesting. I never heard about that command, and it does, indeed, read directly from stdin, which gives a lot of flexibility (together with the use of 'find'). But I already started the extraction of the large file. In the long term, it will be easier for me to maintain it that way, since I have a lot of "cleaning" to do with the contents.
One drawback I find to cpio is that the archive format is not compatible with tar.
When I was reading about tar pipes, I found a lot of examples with tar used together with split. That remembers me of my old days with floppy drives, and a huge backup I did once on several tens of CDs... I would've been very happy to use Linux these days... I'm happy I have a large (even if slow) external hard drive now.
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.