Help answer threads with 0 replies.
Go Back > Forums > Linux Forums > Linux - General
User Name
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.


  Search this Thread
Old 12-28-2007, 02:04 PM   #1
LQ Newbie
Registered: Dec 2007
Distribution: Xubuntu
Posts: 23

Rep: Reputation: 15
Question how to do a tar extract | create pipe on the fly?

I have a huge tar file with several subdirectories. Since managing this junk with my old PC is very slow, I'd like to split the tar into several smaller tars, one for each sub-folder. Instead of extract-to-disk, and then create-from-disk, I'd like to extract|create on the fly with a pipe.

From what I read on the manual, and on internet, I should do something like this:

$ tar -xOf test.tar sub1 | tar -cf -
$ tar -xOf test.tar sub1 | tar -cf sub1.tar
$ tar -xOf test.tar sub1 | tar -cf - sub1.tar -f -
Where the '-O' option sends to STDOUT, and the '-f -' reads from STDIN/OUT. But I am unable to get it right.

Any Ideas?
Old 12-28-2007, 02:43 PM   #2
harry edwards
Registered: Nov 2007
Location: Lincolnshire, UK
Distribution: CentOS, Fedora, and Suse
Posts: 365

Rep: Reputation: 48
As you are piping the output as input to tar you may need the verbose (v) switch? i.e.

$ tar -xvOf test.tar sub1 | tar -cf - sub1.tar
Old 12-28-2007, 03:13 PM   #3
Senior Member
Registered: Dec 2004
Location: Olympia, WA, USA
Distribution: Fedora, (K)Ubuntu
Posts: 4,186

Rep: Reputation: 354Reputation: 354Reputation: 354Reputation: 354
Try tar -xOf test.tar sub1 | tar -cf sub1.tar - since you want to create sub1.tar from STDIN.

Note that this may not work as you'd wish if sub1 contains subdirectories or more than one file. In fact, I'd suggest that you try a tar -xOf test.tar sub1 | less before you go too far down this path so you can get a clear idea of what will be in the input stream to the second tar. It seems to me that the -O option will concatenate the contents of all the files in sub1 into a single stream, and that your new sub1.tar file will contain a single, un-named, block of text, which will, probable, be quite difficult for you to disentangle.
Old 12-29-2007, 07:13 AM   #4
LQ Newbie
Registered: Dec 2007
Distribution: Xubuntu
Posts: 23

Original Poster
Rep: Reputation: 15
Thank you both very much for your reply. Both suggestions produce the following error:
Cannot stat: No such file or directory.
The second suggestion additionally creates an empty sub1.tar file.

This is an old backup with hundreds (or thousands) of files in subdirectories, and it's stored on an external drive. I didn't want to untar the whole thing, since writing many files is much slower than reading/writing one single file. I'll have to do it overnight.
Old 12-29-2007, 10:51 AM   #5
LQ Guru
Registered: Aug 2001
Location: Fargo, ND
Distribution: SuSE AMD64
Posts: 15,733

Rep: Reputation: 680Reputation: 680Reputation: 680Reputation: 680Reputation: 680Reputation: 680
I don't know if you can do that with tar. Maybe you could do that with cpio however.

It's one thing to do something like this:
tar cf - file1 ... | ( cd targetdir/ ; tar xf - )
The stream traveling down the pipe is the archive, so the first tar command was able to do the serializing of the files & filenames. The stdio stream contains both the files and the names. The second tar command marshals the serial stream into the separate files.

I once performed a backup of my /home directory before performing a fresh installation of another Linux version. The external drive I had handy used the fat32 filesystem which has a limited file size limit. So I piped the output of tar through the "split" command to produce several files. I then used par2create to produce par files in case one of the slices became damages. I could extract from the archive without reassembling it, simply by cat'ing the pieces together and piping the output to a tar extract command. As an alternative, you could use the multi-volume option.

Also look at using dar for producing fixed sized back slices.
Old 12-29-2007, 08:50 PM   #6
LQ Newbie
Registered: Dec 2007
Distribution: Xubuntu
Posts: 23

Original Poster
Rep: Reputation: 15
Thanks, jschiwal, your cpio suggestion is really interesting. I never heard about that command, and it does, indeed, read directly from stdin, which gives a lot of flexibility (together with the use of 'find'). But I already started the extraction of the large file. In the long term, it will be easier for me to maintain it that way, since I have a lot of "cleaning" to do with the contents.

One drawback I find to cpio is that the archive format is not compatible with tar.

When I was reading about tar pipes, I found a lot of examples with tar used together with split. That remembers me of my old days with floppy drives, and a huge backup I did once on several tens of CDs... I would've been very happy to use Linux these days... I'm happy I have a large (even if slow) external hard drive now.


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Similar Threads
Thread Thread Starter Forum Replies Last Post
LXer: Tips from an RHCE: Splitting tar archives on the fly LXer Syndicated Linux News 0 10-24-2007 09:20 PM
how to pipe my tar file linux_10_1 Linux - Newbie 2 04-11-2006 11:28 AM
tar problems, trying to pipe through split microsoft/linux Linux - Software 13 03-04-2006 08:20 PM
tar get on the fly nifflerX Linux - General 3 07-27-2005 10:09 AM
tar and ftp put on the fly nifflerX Linux - General 6 06-14-2005 11:45 AM > Forums > Linux Forums > Linux - General

All times are GMT -5. The time now is 09:11 PM.

Main Menu
Write for LQ is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration