LinuxQuestions.org
Review your favorite Linux distribution.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices

Reply
 
Search this Thread
Old 10-21-2007, 12:49 AM   #1
Pi Man
LQ Newbie
 
Registered: Sep 2004
Distribution: Gentoo
Posts: 16

Rep: Reputation: 0
Make tar,zip,rar play nice with named pipes?


Hello all.

I'm working on a scripting project and it would be very helpful if I could make tar, zip, and rar treat named pipes just like regular files when creating an archive. That is, it would be nice if these programs would, upon encountering a named pipe, read all of the input from it and place that input into an entry in the resulting archive named the same way as the named pipe.

What I need is similar to what tar h, zip without the -y option, and rar without the -ol option do for symlinks, only I need to do that for named pipes.

So, for instance, this sequence of instructions:
Code:
mkdir ~/bashtest
cd ~/bashtest
mkdir arcdir
echo "This is a file" > arcdir/file1
echo "This is a symlinked file" > sfile
ln -s sfile arcdir/file2
mkfifo arcdir/file3
echo "This is a fifo" > arcdir/file3 &
tar c?hf archive.tar arcdir
mkdir ~/untarred
cd ~/untarred
tar xf ~/bashtest/archive.tar
ls -l arcdir
cat arcdir/file1 arcdir/file2 arcdir/file3
Would produce this as output:
Code:
total 12
-rw-r--r-- 1 user user 15 Oct 20 22:57 file1
-rw-r--r-- 1 user user 25 Oct 20 22:57 file2
-rw-r--r-- 1 user user 15 Oct 20 22:57 file3
This is a file
This is a symlinked file
This is a fifo
Instead of this:
Code:
total 12
-rw-r--r-- 1 user user 15 Oct 20 22:57 file1
-rw-r--r-- 1 user user 25 Oct 20 22:57 file2
prw-r--r-- 1 user user 9 Oct 20 22:57 file3
This is a file
This is a symlinked file
This is a fifo
I've thoroughly combed man tar, man zip, and man rar and googled for information on this, but I've found no solution so far.

If you want the long version of the story, I'm writing a script which will reside on a web server to which I and a few others admins have SSH access. This script, when invoked, will backup a directory structure and a mysql database and output the resulting .tar.gz, tar.bz2, .zip, or .rar data directly to stdout. That way, on our respective boxen, we will be able to make a backup with something like:

Code:
ssh admin@server.com ~/backup.sh > mybackup.tar.gz
(Or its equivalent on Win or Mac).

I'm working with limited disk space on the server, so I didn't want to duplicate the existing files and dump the MySQL data into a temporary file so that I could tar zcf - the directory. So, I had the bright idea to create a temporary directory containing a symlink to the directory to be backed up and a named pipe to which I would dump the output of mysqldump.

I was thinking something like this would work:
Code:
cd ~
mkdir ~/tempdir
ln -s ~/public_html tempdir/files
mkfifo tempdir/mysql_dump.sql
mysqldump ... > tempdir/mysql_dump.sql &
tar zcvh?f - tempdir
rm -rf tempdir
(Of course, it will need to be a bit more involved than that, what with supporting multiple archive/compression formats and such.)

But, I've found no way to make these archivers do what I need them to do for this to work.

I tried using a symlink to a named pipe with tar h, but tar was too smart to be tricked so easily.

Any and all help would be greatly appreciated. Thanks very much for your time.
 
Old 10-21-2007, 03:44 AM   #2
gnashley
Amigo developer
 
Registered: Dec 2003
Location: Germany
Distribution: Slackware
Posts: 4,775

Rep: Reputation: 481Reputation: 481Reputation: 481Reputation: 481Reputation: 481
Why bot just use the normal pipe?

mysql dump | tar | gzip

You may need to use '-' with tar to properly take input from stdin

Here's a line I use in a program which shows some syntax that may help you:
This backs up files ($i) one at a time into a dir
for i in $filelist ; do
(tar -cpf - "$i"| tar -f - -xvpC somedir)
done

Then this restores them after wards:
cd somedir
tar -cpf - . | tar -f - -xlUvpC

Since you want compressed files you'd need to use another pipe at the end to pipe into gzip/bzip. (note that the xlUvpC is syntax for tar-1.13)

Ah, I forgot, I have a line which creates compressed archives of the backup dir:
cd somedir
tar -cpf - . | gzip -9 > "${BACKUPS_SAVE_DIR}/backup-$(date +%m%d%Y%H%M).tar.gz"

Hopefully there' something there to help you out. The -l and -U options I use for tar-1.13 have to do with preservation of file metadata. I use the old tar-1.13 exactly because of the way it handles links. They make sure that a directory which is a link doesn't get overwritten as a dir. They make tar follow the link instead of replacing it with a real directory. Later version of tar have different default beahviour and syntax for doing this. You should probably check the verison you are using very carefully as thh usage and syntax is different for each version from 1.13, 1.14, 1.15 and 1.16. 1.17 seems to have finally settled the transition in syntax. But you may need to write code which checks the version and supplies alternate syntax for each version or be sure that everyone is using the same version. Have fun...
 
Old 10-21-2007, 01:55 PM   #3
Pi Man
LQ Newbie
 
Registered: Sep 2004
Distribution: Gentoo
Posts: 16

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by gnashley View Post
Why bot just use the normal pipe?

mysql dump | tar | gzip

You may need to use '-' with tar to properly take input from stdin
I don't want to do that because I want the sql dump and the files in one tar file.

And... does tar even take input from stdin when creating a tar archive? It wouldn't know what to name the entry in the resulting tar file, would it? Or would it name it "-", or is there a way to give it a specific name?

If there is a way to create a named entry in the tar file which has contents taken from stdin, then I could create the tar file and then use the "r" option to add the other files to the archive.

Here's a quick test of taking input from stdin when creating a tar archive:
Code:
echo "to be tarred" | tar cf blah.tar -
tar: -: Cannot stat: No such file or directory
tar: Error exit delayed from previous errors
Quote:
Originally Posted by gnashley View Post
Here's a line I use in a program which shows some syntax that may help you:
This backs up files ($i) one at a time into a dir
for i in $filelist ; do
(tar -cpf - "$i"| tar -f - -xvpC somedir)
done

Then this restores them after wards:
cd somedir
tar -cpf - . | tar -f - -xlUvpC
Yes, but in your examples, tar is only taking input from stdin when it's extracting, not when it's creating a tar archive.

Quote:
Originally Posted by gnashley View Post
Since you want compressed files you'd need to use another pipe at the end to pipe into gzip/bzip. (note that the xlUvpC is syntax for tar-1.13)

Ah, I forgot, I have a line which creates compressed archives of the backup dir:
cd somedir
tar -cpf - . | gzip -9 > "${BACKUPS_SAVE_DIR}/backup-$(date +%m%d%Y%H%M).tar.gz"

Hopefully there' something there to help you out. The -l and -U options I use for tar-1.13 have to do with preservation of file metadata. I use the old tar-1.13 exactly because of the way it handles links. They make sure that a directory which is a link doesn't get overwritten as a dir. They make tar follow the link instead of replacing it with a real directory. Later version of tar have different default beahviour and syntax for doing this. You should probably check the verison you are using very carefully as thh usage and syntax is different for each version from 1.13, 1.14, 1.15 and 1.16. 1.17 seems to have finally settled the transition in syntax. But you may need to write code which checks the version and supplies alternate syntax for each version or be sure that everyone is using the same version. Have fun...
Actually, this script will sit on a webserver. I should be able to just run tar --version, take note of what version is installed, and tailor my script to that version. I'll need to check to make sure they haven't updated their version of tar from time to time, but I think it will still be less of a hassle than running most of this on our individual machines.

As for the named pipe stuff. I do have one idea... and it may sound a little bit insane, but it could work.

I could study the tar file format and create a script that basically produces a tar archive containing only one file (the mysql dump) and then pipe the output of that into tar which then adds the rest of the directory structure to the archive. I'll work on that and see what I come up with.

Thanks for your help.
 
Old 10-22-2007, 03:43 AM   #4
gnashley
Amigo developer
 
Registered: Dec 2003
Location: Germany
Distribution: Slackware
Posts: 4,775

Rep: Reputation: 481Reputation: 481Reputation: 481Reputation: 481Reputation: 481
The last example does create a tar.gz from stdin. The '-' following the tar command is taking the stdin from the ouput of '.' which is equivalent to 'ls'.

I think if you play with the syntax I provided you can get tar to do what you want. Just change the last part so that it creates a tar instead of just copying the files to a new dir.
Another idea is that it might be better to just use cpio instead of tar. It may be less finicky and handles long names better.
I'm too busy to work it out for you, but I'm quite sure that tar will do what you want. I just provided the above syntax because it shows the proper way to handle tar with stdin.

Last edited by gnashley; 10-22-2007 at 03:50 AM.
 
Old 10-22-2007, 02:27 PM   #5
Pi Man
LQ Newbie
 
Registered: Sep 2004
Distribution: Gentoo
Posts: 16

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by gnashley View Post
The last example does create a tar.gz from stdin. The '-' following the tar command is taking the stdin from the ouput of '.' which is equivalent to 'ls'.

I think if you play with the syntax I provided you can get tar to do what you want. Just change the last part so that it creates a tar instead of just copying the files to a new dir.
Actually, the last example you gave doesn't take anything from stdin. It archives the current directory (which is what the dot refers to) and writes the resulting tar to stdout (which is what "-" refers to).

Quote:
Originally Posted by gnashley View Post
Another idea is that it might be better to just use cpio instead of tar. It may be less finicky and handles long names better.
Hmm. That's a tool I've not run across in the past. Thanks. I'll take a look.

Quote:
Originally Posted by gnashley View Post
I'm too busy to work it out for you, but I'm quite sure that tar will do what you want.
No problem. Thanks for your help so far.
 
Old 10-22-2007, 03:47 PM   #6
Disillusionist
Senior Member
 
Registered: Aug 2004
Location: England
Distribution: Ubuntu
Posts: 1,013

Rep: Reputation: 83
The power of named pipes comes when the two scripts/programs do not directly talk to each other.

Example:
Code:
###
### Make a named pipe and list files to it
###
mkfifo /tmp/myfifo
ls /etc/init.d > /tmp/myfifo
Code:
###
### Tar files selected from named pipe
###
cat /tmp/myfifo|xargs tar cvf /tmp/backup_initd.tar
When the first piece of code runs, it lists the files and then waits.

When the second piece of code runs, the first piece finishes and the files get passed to tar.

xargs - build and execute command lines from standard input

man xargs
 
Old 10-22-2007, 04:40 PM   #7
Pi Man
LQ Newbie
 
Registered: Sep 2004
Distribution: Gentoo
Posts: 16

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by Disillusionist View Post
The power of named pipes comes when the two scripts/programs do not directly talk to each other.

Example:
Code:
###
### Make a named pipe and list files to it
###
mkfifo /tmp/myfifo
ls /etc/init.d > /tmp/myfifo
Code:
###
### Tar files selected from named pipe
###
cat /tmp/myfifo|xargs tar cvf /tmp/backup_initd.tar
When the first piece of code runs, it lists the files and then waits.

When the second piece of code runs, the first piece finishes and the files get passed to tar.

xargs - build and execute command lines from standard input

man xargs
Yes, but your example just creates a tar containing all of the files in /etc/init.d. (Actually, if I'm not mistaken, unless the current directory happens to be /etc/init.d, it will return a "cannot find file" error several times.) I want to be able to create a tar containing only one entry/file which has the name of the named pipe and which contains the output read from the pipe.

What I'm wanting is something like this:
Code:
###
### Make a named pipe and list files to it
###
mkfifo /tmp/myfifo
ls /etc/init.d > /tmp/myfifo
Code:
###
### Tar files selected from named pipe
###
tar cv?f /tmp/initd_files.tar /tmp/myfifo
# This preceding line isn't correct, but kindof shows what I'm looking for.
Code:
###
### Check the contents of /tmp/initd_files.tar
###
tar tf /tmp/initd_files.tar # will output only one entry: "myfifo"
tar xf /tmp/initd_files.tar
# This preceding line will create one file (not a named pipe) 
# in the current directory called "myfifo"
# with contents something like: 
# "kdm\nnet.eth0\nnet.lo\nsshd\n..."
Now, if you run this next script as written:
Code:
cd ~
mkfifo /tmp/myfifo
echo "This is a fifo" > /tmp/myfifo &
tar cf tarball.tar /tmp/myfifo
tar xf tarball.tar
You get a named pipe called "myfifo" in your home directory. The named pipe is stored in the tar file as a named pipe. I just need it to store the contents of the named pipe as a regular file so that the preceding command would create a file in your home directory named "myfifo" with contents "This is a fifo".

Thanks for your help.

Last edited by Pi Man; 10-22-2007 at 04:42 PM.
 
Old 10-22-2007, 05:07 PM   #8
Disillusionist
Senior Member
 
Registered: Aug 2004
Location: England
Distribution: Ubuntu
Posts: 1,013

Rep: Reputation: 83
Fair point on the errors, I was in the directory I was testing with.

If you want to keep the contents of the pipe, you would first have to pass it to a regular file, when a tar backs up a named pipe it doesn't store the contents.

This isn't what you are after (you would need to tar two files and then to recover would have to restore, then load the data into the named pipe)

I guess the reason I have never tried is that the data in a named pipe is typically only used to pass information between disconnected processes and is temporary (so why back it up)
 
Old 10-22-2007, 05:14 PM   #9
Pi Man
LQ Newbie
 
Registered: Sep 2004
Distribution: Gentoo
Posts: 16

Original Poster
Rep: Reputation: 0
Yeah. I was afraid of that.

At least now I know I should give up on that route and try something different.

Thanks again for your help.
 
  


Reply

Tags
backup, bash, pipe, script, symlink, tar, tarball, tarbz2, targz


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
How to make Ubuntu 6.06 play nice with NVIDIA 7800? Scarlett_Oh Linux - Hardware 1 09-03-2006 05:29 PM
.ZIP .RAR passwords help! SLP Suse/Novell 2 05-17-2006 11:45 PM
Write command in named pipes sahel Programming 7 12-28-2005 07:05 AM
python - make curses play nice w/ os.system() datadriven Programming 0 02-17-2005 04:31 PM
Use of Named Pipes casey0999 Linux - Software 3 08-03-2003 02:21 PM


All times are GMT -5. The time now is 02:54 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration