Using tee and concatenate
 

Please provide a narrative on what the following two commands are doing and highlight the differences. For instance, something like "the first one concatenates the text located after the first line break after the deliminator and before the deliminator located on a new line, and tees it both to the screen and appends (not replaces since it has double arrows) it to the file". Not saying my description is right, and would appreciate a little more detail and especially about what the direction of the arrows represent.

Thank you

Perhaps a simple explanation of the function of tee?

tee, from the man page for it, "tee - read from standard input and write to standard output and files".

Well, yeah.

Typical use is to monitor a running job on the console while also writing the output to a file for later analysis if something is going wrong.

Of the various options commonly used, -a (to append to an existing file rather than overwriting) might be the most common.

Keep in mind that you automatically get three channels opened when a program executes: standard in (0), standard out (1) and standard error (2). The less than (<) and greater than (>) characters let you direct stuff where you want it to go. You direct the content of a file into a program with <, you direct the output of a program into a file with > and you combine the standard out and standard error with 2>&1 to redirect output and error messages; e.g., prog < file 2>&1 > other_file. Using >> appends to the end of existing file other_file rather than overwriting it (what the -a option does with tee).

The sort of funky usage in a shell program of the EOF (or any other pattern) in a quick and dirty shell just makes sure that the program stops after it has done what you want it to do, otherwise it'll just hang and you kill it with ^C or ^D.

Your first example both shows you hello! on the standard output and writes to bla.txt where the second only appends hello! to bla.txt, it concatenates the text to the end of blah.txt because of the cat >> bla.txt which, essentially, is what tee -a bla.txt would do.

You use > to redirect to a file, erasing the existing content of the file, you use >> to append to the file. In either case, if the file does not exist, it will be created.

Hope this helps some.

Thanks tronayne,

So, the first one has <<, but it doesn't appear to be directing the content into a file, but the opposite. What am I missing?

And this one has both >> and <<. How is it working?

Thanks

To add to what tronayne said: bash processes redirects like < and << before it runs the rest of the commands, so sometimes you'll see things like

< input-file > output-file command

These can go before and/or after command and still work fine - whatever seems more clear to read. It just gets a bit more tricky when used in conjunction with something like 2>&1 - which has to be placed precisely to get the desired result.

See http://www.tldp.org/LDP/abs/html/io-redirection.html for all the details with some good examples.

The way this question was stated, it sounds like a homework assignment or an exam study question.

Try reading the link I provided in my last post (written before your most recent post got to me.)

One or more <s pointing left gets input from somewhere and assigns it to stdin - the place where commands get their normal input. There are at least three variations <, <<, and <<<, the last of which is for use with bash variables instead of files. There are also variations for use with subshells <( ... ).

The same is true for one or more >s, but these take the output of something (from stdout) and redirect it somewhere else - usually into a file.

So the two instances in your commands are each doing two separate things to separate data sources and sinks. One is connecting input to the command and the other is disposing of the output from the command.

It will all make sense after you experiment with the variations and see more examples in other people's code which you come across.

You just have to continue coding and debugging your own scripts to get experience. bash appears to be deceptively simple, but it has a lot of wrinkles and odd behaviors that take time to master.

Thanks Josephj,

Yea I suppose it did sound like a homework problem. Definitely not as I am too old to be allowed in school!

Let me check out your link, it it still seems ambiguous.

Ps written on a phone and before reading your last post!!!

This is exactly what I was looking for! Thank you!!!

So, this takes the the text between the deliminators and assigns it to standard in because it is prefaced by an "<" (or we know actually appends it because it uses "<<"),
and then takes the output of cat (who's input happens to be stndin) and sends it (oh yea, appends it) to bla.txt?

Guess that makes sense... Am I right? Can you give a similar narrative about the following? I've started to write one about 5 times, but keep on deleting what I typed (thankfully not on a phone!).

The blow by blow - more than you ever wanted to know
 

The only way I learned this stuff was by buying a bunch of books and reading them - often, more than once. Stuff on the web is too fragmentary to get the overall picture from.

At this point, your local library (I'm guessing you're old enough to remember what one of them is ;) ), should have a bunch of books on Linux. Go for Linux for Dummies or a SAMS book to start with. The stuff from O'Reilly or academic publishers is best kept for later when you understand more.

Eventually, you'll probably want to buy a few so you can write in them and leave bookmarks.

In any case make notes of what you learn. I have a whole subdirectory tree on my computer: ~/public/howto which has more than a hundred little tutorials and notes I made for myself right after I figured out how to do something. I refer to it when I get stuck. I also have a much smaller directory elsewhere of working pieces of bash and other languages that I can refer to.

I don't keep all of this stuff in my head.

Here goes:

cat <<EOF | tee bla.txt
hello!
EOF

Create a "Here Document" (temporary file - stored in memory) containing everything starting on the next line and going until just before a line with just EOF on it.

(Advanced: Perform shell expansions on everything in that document - but there's nothing to do in this example.)

Assign that file to standard input (stdin).

Run the cat command which takes its input from a file, but if no argument is specified, it will use stdin (which is where we just put the contents of the here document) and copies it to stdout.

stdout gets redirected to stdin by the pipe (that's what simple pipes do).

run the command tee

(Advanced: not waiting for cat to finish. The whole pipeline runs at roughly the same time.)

which takes stdin and copies it to stdout while also copying it to the file bla.txt.

(Advanced: with no flags, the file bla.txt will be created if it does not already exist or truncated to zero length if it does exist - before copying the data to it. This happens before tee runs, so, even if tee died immediately, bla.txt would probably already be truncated. That's where I'm not sure of the exact details.)

Since this is the end of the pipeline, stdout will be copied to wherever the default is. If you are running this in a terminal, then the text will be displayed there. If not, the text may be thrown away - into /dev/null ...

(Advanced: when cat comes to the end of the here document, it will close the pipe and exit. tee will see this event and know that it has all the input and can terminate itself after closing bla.txt.

If instead of the here document, the input was from a very large or very slow source, it would be buffered into blocks by the file system. tee would get this input block by block rather than character by character. Depending on how things were setup, output from tee might be delayed until it got the next full block to work with.)

(Advanced, this script is a prime example of what is looked down upon and called a "useless cat". It uses cat and extra processes to do what the shell would do anyway. It would be better as:

tee bla.txt <<EOF
hello!
EOF

but then it wouldn't be a (bad) example of how to use a pipe.)

There are a couple of things to see here.

1) There's a whole lot more going on here than meets the eye.

As a beginner, you can pretty much ignore all the advanced stuff, but sooner or later it may bite you and you may end up scratching your head as to why code that looks perfectly good is doing something other than what you intended.

2) People who provide examples don't always know the best ways to do things.

Take a look at https://en.wikipedia.org/wiki/Here_document for a complete explanation of here-document along with other forms and with a number of examples.

I would suggest that you might want to give some thought to actually using here-document in programming; over the 41 years or so that I've been programming (a lot of shell programs) I can't recall ever having used one (except trying examples in the manual).

Here's an example that might make here-document a little more clear; this is from The KornShell Command and Programming Language.

You might use here-document as a basis for a form letter generator or program generator. Specify cat with the template for the form or program you want to generate. Use parameter expansion and/or command substitution to generate the variable information in the form or program.

Note the use of <<-, which deletes leading tabs.
Dear Morris,

I am writing this on Tue Oct 18 09:00:58 EDT 2016.
Remember to go to the dentist at two-thiry.

I saved the code in a file, Morris and executed it.

Practical? I don't know. You would need to wrap a lot of shell program around it; e.g., piping the message into mail (with the user mail address) and a few other things to make it actually usable. All you really need to know is that whatever is between cat <<- EOF and EOF will be output.

Hope this helps some.

PS - I forgot to mention in the previous post that we were talking about here-document, sorry about that.

tee and the Here Document
 

[QUOTE=NotionCommotion;5619102]Please provide ... (some explanation)....
================
the code -- not valid syntax
--------------

You are dealing with the tee - splits output to 2 or more locations.
..for more than 2, one side of tee is
..fed to another tee and so forth
You are dealing with Here Documents.
..man bash [enter]
..1692G [enter]
..should be close. I use Slackware.

..the contents of the heredoc takes the place of YOU!
..It takes over the input of instructions to a given program.
..sample:
..vars are loaded (perhaps in another program) and
..the following feeds my slice-n-dice big images program.
..when the cutup program asks for boundary corners it gets answered by
..the heredoc.
#
ZYS=$ZXS
..ZW=`dc <<++
.... $ZXI $ZXS 2.0 / - p q
.... ++`
..ZN=`dc <<++
.... $ZYI $ZYS 2.0 / + p q
.... ++`
..ZE=`dc <<++
.... $ZW $ZXS $ZCOL * + p q
.... ++`
..ZS=`dc <<++
.... $ZN $ZYS $ZROW * - p q
.... ++`
# ignore periods above. some things do need spacing to understand.
# order counts. above - dc is a calculator (HP style)

The ZW, ZN, ZE, ZS are corner coordinates.
This portion can feed State Plane, Arbitrary grid, Lat/Long - whatever
and I don't need to be present when it runs. Nor do I need a gazillion
feeder scripts. :)

Yes - I use the tee to load the vars, typically, to a list in a file.
That way I can see what gets loaded into a list of a large group of
corners, start the process and go home for the weekend.

CAUTION: when using tee and you are loading "data" from several sources,
BE SURE output to "data" file is via the '>>' (append) and not '>' [start over].


Norseman01