LinuxQuestions.org
Help answer threads with 0 replies.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices


Reply
  Search this Thread
Old 02-22-2014, 01:53 PM   #1
paziulek
Member
 
Registered: Mar 2009
Location: N.C.
Distribution: "gentoo|debian|arch"
Posts: 106

Rep: Reputation: 11
tar - write to stdout & create log files ( from stdout and stderr ) [solved]


Hello,

I got stuck on this one, and this time I will not continue banging my had against the wall before I open this thread.
here is my deal: tar write to stdout, pipe to a CMD for post processing, redirect the verbose output of tar to tar.log and stderr to tar_err.log while not touching stdout or stderr of the CMD of the post proc.

this is my cmd line from the script:
Code:
tar cv . | mbuffer -m 16G -O 10.0.0.113:8000
adding redirect before the pipe causes writing all tar output contents to the redirect instead of pipe: ( oops )
Code:
tar cv . 1>tar.log | mbuffer -m 16G -O 10.0.0.113:8000
adding it at EOL, makes the strout of mbuffer unvisible.

I also tried with nohup and "cvf -" same "oops" result, which put the tar archive result, and the verbose output of tar in a single file 8-|

From what I see, tar does not provide any options for redirecting the verbose output via one of its options...
I thought about redirecting "cvf xxx" somewhere else, an other stream.... well no positive results..

Thanks for suggestions!

Mike

edit:

testing it with:
Code:
 tar cv . > log.out | cat - > bin.out

Last edited by paziulek; 02-23-2014 at 12:27 PM.
 
Old 02-23-2014, 08:47 AM   #2
catkin
LQ 5k Club
 
Registered: Dec 2008
Location: Tamil Nadu, India
Distribution: Debian
Posts: 8,578
Blog Entries: 31

Rep: Reputation: 1208Reputation: 1208Reputation: 1208Reputation: 1208Reputation: 1208Reputation: 1208Reputation: 1208Reputation: 1208Reputation: 1208
What do you want to achieve by this?

Depending on what it is, bash process substitution could be a solution.
 
1 members found this post helpful.
Old 02-23-2014, 12:26 PM   #3
paziulek
Member
 
Registered: Mar 2009
Location: N.C.
Distribution: "gentoo|debian|arch"
Posts: 106

Original Poster
Rep: Reputation: 11
Thumbs up

Quote:
Originally Posted by catkin View Post
What do you want to achieve by this?

Depending on what it is, bash process substitution could be a solution.

Perfect suggestion!!! Thanks CATKIN!

Here is what I want to achieve:
I got an used LTO3 scsi tape drive, I have some weakness for tape drives, but have not used one at home for quite a few years, since when DDS became vintage, plus I do not feel comfortable with a singe "to disk" NAS backup, and I do not have money to keep investing in HDs, that would either land on a shelf, or became vulnerable in a RAID configuration... in any case, I would not trust in a HD much for long term archiving.

I connected the LTO to my other box, over 1Gb, transferring backups to that box and dumping them to a tape would be kind of inefficient... dumping the data via tcp directly to the tape device is what I was looking for. I looked at number of ways, most involving ssh - no..... I played with mbuffer once a couple of years ago - it is a great tool for fast streaming devices ( and not only ), that suffer from buffer underruns, killing the device, and the media... Well, if you got a backup made, it is also good to know what files have been backed up ( or not ), and the errors during the process... that is what I wanted to achieve, and with your priceless suggestion catkin I got it to work.

backup client side:
Code:
tar cjv --preserve-permissions --numeric-owner -f >(mbuffer -m 8G -O 10.0.0.113:8000) /root > root_backup.log
backup "server"
Code:
mbuffer -I 8000 -m 12G -f -P 80 -o /dev/nst0

BTW, to increase the compression speed I recompiled the latest tar to use lbzip2 as the bzip2 compressor, to enable multithreading during tar.bz2 archive creation... now the 40-80MB/s native tape drive does not choke during the backup creation ( if it does, than it is once/twice per 200GB backup ) - a multithreaded alternative to lbzip2 is pigz ( gzip )... faster with lower compression ratio...

Thanks again catkin!

Mike
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] Bash: Stderr & Stdout 2 file Ztcoracat Linux - General 9 11-18-2012 09:23 AM
[SOLVED] stderr stdout to a log file troubles metallica1973 Linux - Software 3 11-12-2011 04:28 PM
bash output stderr to 2 files and stdout vascot Linux - Software 1 04-05-2011 08:04 AM
copying stdout and stderr to seperate files dasy2k1 Linux - General 2 02-09-2007 06:41 AM
redirecting stdout to /dev/null and stderr to stdout? Thinking Programming 1 05-18-2006 02:36 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - General

All times are GMT -5. The time now is 06:08 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration