Directing Script output to Memory but without using a pipe |

telecom_is_me · 06-27-2008, 01:18 PM

I'm trying to run a series of commands to read a live data dump an strip it in various ways on the fly and send those various stripped pieces of data to corresponding files.

For instance I have the following:

Code:

./dumpscript.sh | sed '/print/,/raw/d' | cut -c 1-47 > output.txt

In this case my dumpscript.sh output is piped through "sed" and "cut" to give me the specific stream of data that I need for archiving.

Now at the same time I would like to run a second "cut" with different parameters and output the results to another file. Again this needs to be live on the fly.

I've tried:

Code:

./dumpscript.sh | sed '/print/,/raw/d' | cut -c 1-47 > output.txt ; cut -c 1-18 > output2.txt

However, there's nothing in the second output file.

Any help would be appreciated.

telecom_is_me · 06-27-2008, 08:24 PM

Is there some way that I could use "tee" perhaps? I read in the man page that:

Code:

The `tee' command copies standard input to standard output and also to any files given as arguments.  This is useful when you want not only to
send some data down a pipe, but also to save a copy.

Any suggestions?

Mr. C. · 06-27-2008, 11:48 PM

The problem with your last command line is that the semicolon is a command separator. Its just like entering the first command, hitting Enter, and then the second.

What you need is a way to send output to a file... and use tee to send the same stream of bytes to another pipeline. The problem is that you cannot both redirect (>) and pipe (|) STDOUT in the same command.

However, this is where sub-shells are useful. There is nothing to prevent you from sending your output to another shell, which can then redirect away. Try this example:

Code:

$ echo Help Me | sed 's/Me/You/' | tee file1 | (cut -c1-4 > file2)
$ cat file1
Help You
$ cat file2
Help

Note that the STDOUT for the current shell is sent along a pipeline, but the STDOUT of the sub-shell is redirected.

Tinkster · 06-28-2008, 12:39 AM

How about awk?

Code:

./dumpscript.sh | sed '/print/,/raw/d' | awk '{printf substr($0,1,47) > "output1.txt"; printf substr($0,1,18) > "output2.txt"}'

Cheers,
Tink

telecom_is_me · 06-29-2008, 12:17 AM

Quote:

Originally Posted by Mr. C.

The problem with your last command line is that the semicolon is a command separator. Its just like entering the first command, hitting Enter, and then the second.

What you need is a way to send output to a file... and use tee to send the same stream of bytes to another pipeline. The problem is that you cannot both redirect (>) and pipe (|) STDOUT in the same command.

However, this is where sub-shells are useful. There is nothing to prevent you from sending your output to another shell, which can then redirect away. Try this example:

Code:

$ echo Help Me | sed 's/Me/You/' | tee file1 | (cut -c1-4 > file2)
$ cat file1
Help You
$ cat file2
Help

Note that the STDOUT the current shell is sent along a pipeline, but the STDOUT of the sub-shell is redirected.

Mr. C.

Thank you so much, I really appreciate your help... That works exactly as I would like, I can't wait to try it on a live system on Monday, It will be far more efficient than my current methods which require quite a bit of manual work.

Thanks, again.