tar - write to stdout & create log files ( from stdout and stderr ) [solved]

paziulek · 02-22-2014, 01:53 PM

Hello,

I got stuck on this one, and this time I will not continue banging my had against the wall before I open this thread.
here is my deal: tar write to stdout, pipe to a CMD for post processing, redirect the verbose output of tar to tar.log and stderr to tar_err.log while not touching stdout or stderr of the CMD of the post proc.

this is my cmd line from the script:

Code:

tar cv . | mbuffer -m 16G -O 10.0.0.113:8000

adding redirect before the pipe causes writing all tar output contents to the redirect instead of pipe: ( oops )

Code:

tar cv . 1>tar.log | mbuffer -m 16G -O 10.0.0.113:8000

adding it at EOL, makes the strout of mbuffer unvisible.

I also tried with nohup and "cvf -" same "oops" result, which put the tar archive result, and the verbose output of tar in a single file 8-|

From what I see, tar does not provide any options for redirecting the verbose output via one of its options...
I thought about redirecting "cvf xxx" somewhere else, an other stream.... well no positive results..

Thanks for suggestions!

Mike

edit:

testing it with:

Code:

 tar cv . > log.out | cat - > bin.out

catkin · 02-23-2014, 08:47 AM

What do you want to achieve by this?

Depending on what it is, bash process substitution could be a solution.

paziulek · 02-23-2014, 12:26 PM

Quote:

Originally Posted by catkin

What do you want to achieve by this?

Depending on what it is, bash process substitution could be a solution.

Perfect suggestion!!! Thanks CATKIN!

Here is what I want to achieve:
I got an used LTO3 scsi tape drive, I have some weakness for tape drives, but have not used one at home for quite a few years, since when DDS became vintage, plus I do not feel comfortable with a singe "to disk" NAS backup, and I do not have money to keep investing in HDs, that would either land on a shelf, or became vulnerable in a RAID configuration... in any case, I would not trust in a HD much for long term archiving.

I connected the LTO to my other box, over 1Gb, transferring backups to that box and dumping them to a tape would be kind of inefficient... dumping the data via tcp directly to the tape device is what I was looking for. I looked at number of ways, most involving ssh - no..... I played with mbuffer once a couple of years ago - it is a great tool for fast streaming devices ( and not only ), that suffer from buffer underruns, killing the device, and the media... Well, if you got a backup made, it is also good to know what files have been backed up ( or not ), and the errors during the process... that is what I wanted to achieve, and with your priceless suggestion catkin I got it to work.

backup client side:

Code:

tar cjv --preserve-permissions --numeric-owner -f >(mbuffer -m 8G -O 10.0.0.113:8000) /root > root_backup.log

backup "server"

Code:

mbuffer -I 8000 -m 12G -f -P 80 -o /dev/nst0

BTW, to increase the compression speed I recompiled the latest tar to use lbzip2 as the bzip2 compressor, to enable multithreading during tar.bz2 archive creation... now the 40-80MB/s native tape drive does not choke during the backup creation ( if it does, than it is once/twice per 200GB backup ) - a multithreaded alternative to lbzip2 is pigz ( gzip )... faster with lower compression ratio...

Thanks again catkin!

Mike