bash 'read' built-in with multiple processes reading same descriptor

colucix · 09-13-2009, 12:48 PM

Here we go! I changed the line inside the inner loop to

Code:

echo $line $I >> outfile_$I

Note that single redirection cannot work in this case, because the file will be overwritten at every cycle. The result is that I didn't lost any line, but two characters. Here is the first line of output_2:

Code:

$ head -5 output_2
ne06964 2
line06965 2
line06966 2
line06967 2
line06968 2

the first two characters "li" are missing from both files. They are completely lost, for a reason I cannot explain. I verified that they are really missing using character count from the wc command. The rest of the output is correct: 10000 lines total, perfect comparison between the input and the "catted" and sorted output, except for the truncated line, of course.

Ai posteri l'ardua sentenza! (A. Manzoni, Il Cinque Maggio, 1821)

colucix · 09-13-2009, 12:56 PM

An aside note: I'd like to bring to your attention a project made specifically to run shell scripts in parallel. Here is the relevant thread, which saw the participation of the developer himself.

catkin · 09-13-2009, 01:08 PM

Quote:

Originally Posted by colucix

Maybe you missed my last post...

You're right, I did

Writing at the same time?

Quote:

Originally Posted by colucix

The reason lies in the definition of standard input itself. Yes, without cat it tries to read from the standard input, but the standard input is the keyboard input, unless redirected. Cat performs this redirection: it takes the standard input redirected from the first cat into the outer loop and flushes its output through the pipe, which becomes the input source for the inner loop.

In other words, if the inner loop is left "alone":

Code:

  while read line; do
    echo $line $I
  done &

the read statement takes (or better, tries to take) standard input from the keyboard, since there is no redirection at all in this block of code!

Perhaps I am being dense here but ... why does the inner loop cat take the standard input redirected from the first cat into the outer loop but the "read" does not and picks up stdin from the keyboard?

catkin · 09-13-2009, 01:14 PM

Quote:

Originally Posted by colucix

the first two characters "li" are missing from both files.

Ai posteri l'ardua sentenza! (A. Manzoni, Il Cinque Maggio, 1821)

Unexpected -- a very similar experiment worked perfectly for me as posted earlier. What does Ai posteri l'ardua sentenza! mean? BabelFish translated it as "To the posteri l' arduous sentence" which was not much advance on the original! "Sentence" as in text or as in "prison sentence"?!

colucix · 09-13-2009, 02:29 PM

Quote:

Originally Posted by catkin

Perhaps I am being dense here but ... why does the inner loop cat take the standard input redirected from the first cat into the outer loop but the "read" does not and picks up stdin from the keyboard?

Outer loop: standard input redirected by cat $* through pipe:

Code:

cat $* | for I in `seq 1 "$max_count"`; do

this means the code inside the loop takes standard input from cat $*.

Inner loop: standard input redirected by cat through pipe:

Code:

cat | while read line; do

the loop takes input from the pipe.

The matter is: redirection must be explicit for a command to take input from some file descriptor. Otherwise standard input is expected by the keyboard (file descriptor 0 in C, unit 5 in fortran).

The man page for stdin (C programming) explains it well:

Quote:

Under normal circumstances every Unix program has three streams opened for it when it starts up, one for input, one for output, and one for printing diagnostic or error messages. These are typically attached to the user's terminal (see tty(4) but might instead refer to files or other devices, depending on what the parent process chose to set up.

In other words: who tells the inner loop to take input from the first cat $* and not from the keyboard? The second cat tells: "Hey hey... standard input is here! Don't look at the keyboard! I'm passing it through pipe...!"

colucix · 09-13-2009, 02:38 PM

Quote:

Originally Posted by catkin

Unexpected -- a very similar experiment worked perfectly for me as posted earlier. What does Ai posteri l'ardua sentenza! mean? BabelFish translated it as "To the posteri l' arduous sentence" which was not much advance on the original! "Sentence" as in text or as in "prison sentence"?!

This is a quote from the first and greatest Italian novelist, Alessandro Manzoni. The quote is from one of his greatest poems (The Five of May) and literally means: "To posterity the arduous decision!". It is used when you report some facts, but you don't express a judgement.

catkin · 09-13-2009, 04:16 PM

Quote:

Originally Posted by colucix

The quote is from one of his greatest poems (The Five of May) and literally means: "To posterity the arduous decision!". It is used when you report some facts, but you don't express a judgement.

Thanks for your translation which is more insightful (and useful) than the "Sentence waits posterity" given here. According to the Free Dictionary, the English word "sentence" has an obsolete meaning of "An opinion, especially one given formally after deliberation" which is close to Manzoni's use of sentenza.

catkin · 09-13-2009, 04:27 PM

Thank you for explaining but I still haven't got it. I'm OK with,

Quote:

Originally Posted by colucix

Code:

cat $* | for I in `seq 1 "$max_count"`; do

this means the code inside the loop takes standard input from cat $*.

but I don't understand why the inner (while) loop's standard input isn't also coming from cat $*. AIUI (obviously wrongly!) the whole of the for-do-done compound statement is executed with standard input from cat $* and that includes the while-do-done compound statement. What causes the while-do-done to take standard input from somewhere else? What causes it to change?

colucix · 09-13-2009, 04:32 PM

Quote:

Originally Posted by catkin

According to the Free Dictionary, the English word "sentence" has an obsolete meaning of "An opinion, especially one given formally after deliberation" which is close to Manzoni's use of sentenza.

That's true. Indeed, the poem was inspired to Manzoni from the conversion of Napoleon, a few days before his death. After narrating his deeds, he asked a question:

Code:

Fu vera gloria? Ai posteri l'ardua sentenza

that is: Was it true glory? To posterity the arduous decision!
Hence sentence as in the terms you depicted.

What do you think? Are we a little off-topic?

colucix · 09-13-2009, 04:41 PM

Quote:

Originally Posted by catkin

... and that includes the while-do-done compound statement. What causes the while-do-done to take standard input from somewhere else? What causes it to change?

Don't forget that the while loop is launched in background, so that it is detached from the default standard input (tty), so that you're forced to make it to read from a pipe or from a file.

catkin · 09-13-2009, 04:41 PM

Quote:

Originally Posted by colucix

What do you think? Are we a little off-topic?

We sure are but I've just become a Senior Member so I'm allowed some "senior moments"! Enough -- back to standard inputs!

catkin · 09-13-2009, 04:58 PM

Quote:

Originally Posted by colucix

Don't forget that the while loop is launched in background, so that it is detached from the default standard input (tty), so that you're forced to make it to read from a pipe or from a file.

Thanks, I've got it now. As it says in the GNU Bash Reference on backgrounding, "If a command is terminated by the control operator ‘&’, the shell executes the command asynchronously in a subshell. [snip] the standard input for asynchronous commands, in the absence of any explicit redirections, is redirected from /dev/null.". I was getting mixed up with fork(3C) which duplicates file descriptors and their buffers -- useless for an interactive shell backgrounding! Here's an alternative idiom, dispensing with the cat

Code:

max_count="$1"
shift

cat $* | for I in `seq 1 "$max_count"`; do
  while read line; do
    echo $line >> output$I.txt
  done <&0 &
done