LinuxQuestions.org - Make tar,zip,rar play nice with named pipes?

- Programming (https://www.linuxquestions.org/questions/programming-9/)

- - Make tar,zip,rar play nice with named pipes? (https://www.linuxquestions.org/questions/programming-9/make-tar-zip-rar-play-nice-with-named-pipes-593392/)

Make tar,zip,rar play nice with named pipes?

Hello all.

I'm working on a scripting project and it would be very helpful if I could make tar, zip, and rar treat named pipes just like regular files when creating an archive. That is, it would be nice if these programs would, upon encountering a named pipe, read all of the input from it and place that input into an entry in the resulting archive named the same way as the named pipe.

What I need is similar to what tar h, zip without the -y option, and rar without the -ol option do for symlinks, only I need to do that for named pipes.

So, for instance, this sequence of instructions:

Code:

mkdir ~/bashtest

cd ~/bashtest

mkdir arcdir

echo "This is a file" > arcdir/file1

echo "This is a symlinked file" > sfile

ln -s sfile arcdir/file2

mkfifo arcdir/file3

echo "This is a fifo" > arcdir/file3 &

tar c?hf archive.tar arcdir

mkdir ~/untarred

cd ~/untarred

tar xf ~/bashtest/archive.tar

ls -l arcdir

cat arcdir/file1 arcdir/file2 arcdir/file3

Would produce this as output:

Code:

total 12

-rw-r--r-- 1 user user 15 Oct 20 22:57 file1

-rw-r--r-- 1 user user 25 Oct 20 22:57 file2

-rw-r--r-- 1 user user 15 Oct 20 22:57 file3

This is a file

This is a symlinked file

This is a fifo

Instead of this:

Code:

total 12

-rw-r--r-- 1 user user 15 Oct 20 22:57 file1

-rw-r--r-- 1 user user 25 Oct 20 22:57 file2

prw-r--r-- 1 user user 9 Oct 20 22:57 file3

This is a file

This is a symlinked file

This is a fifo

I've thoroughly combed man tar, man zip, and man rar and googled for information on this, but I've found no solution so far.

If you want the long version of the story, I'm writing a script which will reside on a web server to which I and a few others admins have SSH access. This script, when invoked, will backup a directory structure and a mysql database and output the resulting .tar.gz, tar.bz2, .zip, or .rar data directly to stdout. That way, on our respective boxen, we will be able to make a backup with something like:

Code:

ssh admin@server.com ~/backup.sh > mybackup.tar.gz

(Or its equivalent on Win or Mac).

I'm working with limited disk space on the server, so I didn't want to duplicate the existing files and dump the MySQL data into a temporary file so that I could tar zcf - the directory. So, I had the bright idea to create a temporary directory containing a symlink to the directory to be backed up and a named pipe to which I would dump the output of mysqldump.

I was thinking something like this would work:

Code:

cd ~

mkdir ~/tempdir

ln -s ~/public_html tempdir/files

mkfifo tempdir/mysql_dump.sql

mysqldump ... > tempdir/mysql_dump.sql &

tar zcvh?f - tempdir

rm -rf tempdir

(Of course, it will need to be a bit more involved than that, what with supporting multiple archive/compression formats and such.)

But, I've found no way to make these archivers do what I need them to do for this to work.

I tried using a symlink to a named pipe with tar h, but tar was too smart to be tricked so easily. :D

Any and all help would be greatly appreciated. Thanks very much for your time. :)

Why bot just use the normal pipe?

mysql dump | tar | gzip

You may need to use '-' with tar to properly take input from stdin

Here's a line I use in a program which shows some syntax that may help you:
This backs up files ($i) one at a time into a dir
for i in $filelist ; do
(tar -cpf - "$i"| tar -f - -xvpC somedir)
done

Then this restores them after wards:
cd somedir
tar -cpf - . | tar -f - -xlUvpC

Since you want compressed files you'd need to use another pipe at the end to pipe into gzip/bzip. (note that the xlUvpC is syntax for tar-1.13)

Ah, I forgot, I have a line which creates compressed archives of the backup dir:
cd somedir
tar -cpf - . | gzip -9 > "${BACKUPS_SAVE_DIR}/backup-$(date +%m%d%Y%H%M).tar.gz"

Hopefully there' something there to help you out. The -l and -U options I use for tar-1.13 have to do with preservation of file metadata. I use the old tar-1.13 exactly because of the way it handles links. They make sure that a directory which is a link doesn't get overwritten as a dir. They make tar follow the link instead of replacing it with a real directory. Later version of tar have different default beahviour and syntax for doing this. You should probably check the verison you are using very carefully as thh usage and syntax is different for each version from 1.13, 1.14, 1.15 and 1.16. 1.17 seems to have finally settled the transition in syntax. But you may need to write code which checks the version and supplies alternate syntax for each version or be sure that everyone is using the same version. Have fun...

Quote:

Originally Posted by gnashley (Post 2931347)

Why bot just use the normal pipe?

mysql dump | tar | gzip

You may need to use '-' with tar to properly take input from stdin

I don't want to do that because I want the sql dump and the files in one tar file.

And... does tar even take input from stdin when creating a tar archive? It wouldn't know what to name the entry in the resulting tar file, would it? Or would it name it "-", or is there a way to give it a specific name?

If there is a way to create a named entry in the tar file which has contents taken from stdin, then I could create the tar file and then use the "r" option to add the other files to the archive.

Here's a quick test of taking input from stdin when creating a tar archive:

Code:

echo "to be tarred" | tar cf blah.tar -

tar: -: Cannot stat: No such file or directory

tar: Error exit delayed from previous errors

Quote:

Originally Posted by gnashley (Post 2931347)

Here's a line I use in a program which shows some syntax that may help you:
This backs up files ($i) one at a time into a dir
for i in $filelist ; do
(tar -cpf - "$i"| tar -f - -xvpC somedir)
done

Then this restores them after wards:
cd somedir
tar -cpf - . | tar -f - -xlUvpC

Yes, but in your examples, tar is only taking input from stdin when it's extracting, not when it's creating a tar archive.

Quote:

Originally Posted by gnashley (Post 2931347)

Since you want compressed files you'd need to use another pipe at the end to pipe into gzip/bzip. (note that the xlUvpC is syntax for tar-1.13)

Ah, I forgot, I have a line which creates compressed archives of the backup dir:
cd somedir
tar -cpf - . | gzip -9 > "${BACKUPS_SAVE_DIR}/backup-$(date +%m%d%Y%H%M).tar.gz"

Hopefully there' something there to help you out. The -l and -U options I use for tar-1.13 have to do with preservation of file metadata. I use the old tar-1.13 exactly because of the way it handles links. They make sure that a directory which is a link doesn't get overwritten as a dir. They make tar follow the link instead of replacing it with a real directory. Later version of tar have different default beahviour and syntax for doing this. You should probably check the verison you are using very carefully as thh usage and syntax is different for each version from 1.13, 1.14, 1.15 and 1.16. 1.17 seems to have finally settled the transition in syntax. But you may need to write code which checks the version and supplies alternate syntax for each version or be sure that everyone is using the same version. Have fun...

Actually, this script will sit on a webserver. I should be able to just run tar --version, take note of what version is installed, and tailor my script to that version. I'll need to check to make sure they haven't updated their version of tar from time to time, but I think it will still be less of a hassle than running most of this on our individual machines.

As for the named pipe stuff. I do have one idea... and it may sound a little bit insane, but it could work.

I could study the tar file format and create a script that basically produces a tar archive containing only one file (the mysql dump) and then pipe the output of that into tar which then adds the rest of the directory structure to the archive. I'll work on that and see what I come up with.

Thanks for your help. :D

The last example does create a tar.gz from stdin. The '-' following the tar command is taking the stdin from the ouput of '.' which is equivalent to 'ls'.

I think if you play with the syntax I provided you can get tar to do what you want. Just change the last part so that it creates a tar instead of just copying the files to a new dir.
Another idea is that it might be better to just use cpio instead of tar. It may be less finicky and handles long names better.
I'm too busy to work it out for you, but I'm quite sure that tar will do what you want. I just provided the above syntax because it shows the proper way to handle tar with stdin.

Quote:

Originally Posted by gnashley (Post 2932275)

Actually, the last example you gave doesn't take anything from stdin. It archives the current directory (which is what the dot refers to) and writes the resulting tar to stdout (which is what "-" refers to).

Quote:

Originally Posted by gnashley (Post 2932275)

Another idea is that it might be better to just use cpio instead of tar. It may be less finicky and handles long names better.

Hmm. That's a tool I've not run across in the past. Thanks. I'll take a look. :)

Quote:

Originally Posted by gnashley (Post 2932275)

I'm too busy to work it out for you, but I'm quite sure that tar will do what you want.

No problem. Thanks for your help so far. :D

The power of named pipes comes when the two scripts/programs do not directly talk to each other.

Example:

Code:

###

### Make a named pipe and list files to it

###

mkfifo /tmp/myfifo

ls /etc/init.d > /tmp/myfifo

Code:

###

### Tar files selected from named pipe

###

cat /tmp/myfifo|xargs tar cvf /tmp/backup_initd.tar

When the first piece of code runs, it lists the files and then waits.

When the second piece of code runs, the first piece finishes and the files get passed to tar.

xargs - build and execute command lines from standard input

man xargs

Quote:

Originally Posted by Disillusionist (Post 2932874)

The power of named pipes comes when the two scripts/programs do not directly talk to each other.

Example:

Code:

###

### Make a named pipe and list files to it

###

mkfifo /tmp/myfifo

ls /etc/init.d > /tmp/myfifo

Code:

###

### Tar files selected from named pipe

###

cat /tmp/myfifo|xargs tar cvf /tmp/backup_initd.tar

Yes, but your example just creates a tar containing all of the files in /etc/init.d. (Actually, if I'm not mistaken, unless the current directory happens to be /etc/init.d, it will return a "cannot find file" error several times.) I want to be able to create a tar containing only one entry/file which has the name of the named pipe and which contains the output read from the pipe.

What I'm wanting is something like this:

Code:

###

### Make a named pipe and list files to it

###

mkfifo /tmp/myfifo

ls /etc/init.d > /tmp/myfifo

Code:

###

### Tar files selected from named pipe

###

tar cv?f /tmp/initd_files.tar /tmp/myfifo

# This preceding line isn't correct, but kindof shows what I'm looking for.

Code:

###

### Check the contents of /tmp/initd_files.tar

###

tar tf /tmp/initd_files.tar # will output only one entry: "myfifo"

tar xf /tmp/initd_files.tar

# This preceding line will create one file (not a named pipe) 

# in the current directory called "myfifo"

# with contents something like: 

# "kdm\nnet.eth0\nnet.lo\nsshd\n..."

Now, if you run this next script as written:

Code:

cd ~

mkfifo /tmp/myfifo

echo "This is a fifo" > /tmp/myfifo &

tar cf tarball.tar /tmp/myfifo

tar xf tarball.tar

You get a named pipe called "myfifo" in your home directory. The named pipe is stored in the tar file as a named pipe. I just need it to store the contents of the named pipe as a regular file so that the preceding command would create a file in your home directory named "myfifo" with contents "This is a fifo".

Thanks for your help. :)

Fair point on the errors, I was in the directory I was testing with.

If you want to keep the contents of the pipe, you would first have to pass it to a regular file, when a tar backs up a named pipe it doesn't store the contents.

This isn't what you are after (you would need to tar two files and then to recover would have to restore, then load the data into the named pipe)

I guess the reason I have never tried is that the data in a named pipe is typically only used to pass information between disconnected processes and is temporary (so why back it up)

Yeah. I was afraid of that.

At least now I know I should give up on that route and try something different.

Thanks again for your help. :)