[SOLVED] Copy not redirect stdout and stderr in bash

budgie26 · 05-01-2014, 01:01 AM

I'm converting a script from csh to bash for use with a PBS batch job system, and I need to add commands which redirect stdout and stderr (hence the conversion to bash). What I'm after is not a simple case of using:

Code:

command_to_run $1 $2 $3 1> output.log 2> error.log

as I need to COPY the stdout and stderr to files and not just redirect.

My script processes multiple files via a single PBS batch job. Each file needs to have its own output and error log file (which the above code will do), but the single PBS job also needs to have the stdout and stderr output for each processed file put into its own output and error log file. The PBS batch job system uses stdout for its output log file and stderr for its error log file. I need to use the PBS error log file to check for any errors in the processed files, rather than going into each file's error log file (I'm dealing with thousands of files!). I don't want to have to create a separate script to consolidate the errors into a new file if the PBS error log file can provide this information with a small amount of coding.

I've done lots of searching to solve this but haven't had any luck. I'm new to bash, so it may be a case of me not fully understanding the coding.

What I require is:
1. copy the stdout results to 'output.log' for each processed file and keep the results in stdout so it can be picked up by the PBS batch job output results file.
2. copy the stderr results to 'error.log' for each processed file and keep the results in stderr so it can be picked up by the PBS batch job error results file.

Is there a way to have the std* go to both the *.log file and to the PBS log file?

In addition, I need to record the command line and arguments to a 'command_log' file. This currently happens with the csh file, but I can't work out the coding in bash. The csh script has the command:

Code:

alias GM 'echo \!* | tee -a log; echo; \!* >>! log1'
GM command_to_run $1 $2 $3

The alias GM is put before the commands I want to capture and enables the command line input and arguments to be put into the 'log' file and the stdout into the 'log1' file. All I want is to capture what's in the 'log' file.

Any advice would be greatly appreciated!!

Thanks in advance

grail · 05-01-2014, 03:24 AM

Well I am not sure I exactly follow, however, the tee command from your csh example is what would be used to have output streamed to 2 separate locations (i think this is what you are asking)

As for the alias, I think you need to create a function as opposed to an alias (I could be wrong but am not familiar with '!*'):

Code:

GM()
{
  echo $* | tee -a log
  echo
  $* >> log1
}

NevemTeve · 05-01-2014, 04:47 AM

bash only:

Code:

command >(tee output.log) 2>(tee error.log)

to log commands:

Code:

set -vx

budgie26 · 05-04-2014, 09:16 PM

Thanks. I found that NevemTeve's suggestions didn't work, but I've found that doing the following gets me closer to a suitable solution:

Code:

GM()
{
    echo $* | tee -a command.log
    echo
    $* >> output.log
}

This gives me the command and output details I want in the relevent files, but I can't get stderr to the error.log to work within this function. I've tried adding:

Code:

2> error.log
cat error.log 1>&2

But this doesn't work, as the error.log file is blank. Any ideas?

NevemTeve · 05-04-2014, 10:25 PM

> I found that NevemTeve's suggestions didn't work

What did you try? Did you get an error message? Or what happened?
...

Code:

2> error.log
cat error.log 1>&2

> But this doesn't work, as the error.log file is blank. Any ideas?

What should have happened?

budgie26 · 05-04-2014, 10:40 PM

'set -vx' printed all the commands within the script,instead of just the single command line I'm interested in (eg. command $1 $2) within the script. When I did 'command >(tee output.log) 2>(tee error.log)' these files were empty and I didn't get any errors.

When I tried:

Code:

GM()
{
    echo $* | tee -a command.log
    echo
    $* >> output.log
    2> error.log
    cat error.log 1>&2
}

The command.log file shows the single command line I'm interested in, the output.log file has the stdout details but the error.log file is empty. The PBS job shows the stderr, so it looks like the errors haven't been piped to the error.log file and then copied back to the stderr.

If I run:

Code:

command $1 $2 1> output.log 2> error.log
cat error.log 1>&2
cat output.log

The output.log and error.log files have the right details, and the stderr and stdout are still piped to the PBS job output files. This provides everything I want except for the command.log. I'm not keen on this solution as it doesn't have the command.log details and I'd have to manually write the redirections and cat commands after each command in the script. Having a function instead would be much easier as there are quite a few commands in the script.

SaintDanBert · 05-04-2014, 11:52 PM

Quote:

Originally Posted by NevemTeve

> I found that NevemTeve's suggestions didn't work
What did you try? Did you get an error message? Or what happened?
...

Code:

2> error.log
cat error.log 1>&2

But this doesn't work, as the error.log file is blank. Any ideas?
What should have happened?

As I understand things 1>&2 says to direct output on "channel 1" (STDOUT) onto "channel 2" (STDERR)?

Don't you want to send STDERR to STDOUT instead as 2>&1?

~~~ 0;-Dan

budgie26 · 05-04-2014, 11:57 PM

No I want send stdout to stderr (1>&2). By doing 'cat error.log 1>&2' this will display the error.log contents as an output not an error, so I want to redirect the contents of error.log back to stderr.

pan64 · 05-05-2014, 01:57 AM

to copy means (at least for me) to duplicate. The normal stdout (and stderr) should be saved and printed too. In that case you may try to use screen -L <your command> (this will mix stdout and stderr, but save it), or you can try to solve it with tee.
The redirection like:

Code:

2>error.log
cat error.log 1>&2

will not work, because the first line will just create an empty error.log file and the second line will immediately print it (and of course it is completed). Redirection to error.log may work, but the cat command already finished.

Here is a small example, probably it can be used:

Code:

mkfifo /tmp/fifo.stdout
cat /tmp/fifo.stdout | tee /tmp/stdout.log
mkfifo /tmp/fifo.stderr
cat /tmp/fifo.stderr | tee /tmp/stderr.log

your_command >/tmp/fifo.stdout 2>/tmp/fifo.stderr

this will save stdout and stderr in /tmp and also print

NevemTeve · 05-05-2014, 02:47 AM

Ok, I was plain wrong. Now it is tested:

Code:

#!/bin/bash
./command_that_writes_stdout_and_stderr \
	 > >(tee output.log) 		\
	2> >(tee error.log >&2)

grail · 05-05-2014, 03:31 AM

As pointed out by pan64, if you wish to redirect stderr, you will need to do this at the point of calling the command, ie. not after:

Code:

GM()
{
    echo $* | tee -a command.log
    echo
    $* >> output.log 2> error.log
    
    cat error.log 1>&2  # I was not sure of the purpose of this line so left as is
}

budgie26 · 05-07-2014, 08:47 PM

Thank you for all your help, I now have the solution!

Code:

GM()
{
    echo $* | tee -a command.log
    echo
    $* >> output.log 2> error.log
    cat error.log 1>&2
}

This will create three files (command.log, output.log, error.log) for each processed file as well as copy the stdout and stderr to the PBS job output files.