LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices


Reply
  Search this Thread
Old 09-13-2009, 12:48 PM   #31
colucix
LQ Guru
 
Registered: Sep 2003
Location: Bologna
Distribution: CentOS 6.5 OpenSuSE 12.3
Posts: 10,509

Rep: Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981

Here we go! I changed the line inside the inner loop to
Code:
echo $line $I >> outfile_$I
Note that single redirection cannot work in this case, because the file will be overwritten at every cycle. The result is that I didn't lost any line, but two characters. Here is the first line of output_2:
Code:
$ head -5 output_2
ne06964 2
line06965 2
line06966 2
line06967 2
line06968 2
the first two characters "li" are missing from both files. They are completely lost, for a reason I cannot explain. I verified that they are really missing using character count from the wc command. The rest of the output is correct: 10000 lines total, perfect comparison between the input and the "catted" and sorted output, except for the truncated line, of course.

Ai posteri l'ardua sentenza! (A. Manzoni, Il Cinque Maggio, 1821)
 
Old 09-13-2009, 12:56 PM   #32
colucix
LQ Guru
 
Registered: Sep 2003
Location: Bologna
Distribution: CentOS 6.5 OpenSuSE 12.3
Posts: 10,509

Rep: Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981
An aside note: I'd like to bring to your attention a project made specifically to run shell scripts in parallel. Here is the relevant thread, which saw the participation of the developer himself.
 
Old 09-13-2009, 01:08 PM   #33
catkin
LQ 5k Club
 
Registered: Dec 2008
Location: Tamil Nadu, India
Distribution: Debian
Posts: 8,577
Blog Entries: 31

Rep: Reputation: 1197Reputation: 1197Reputation: 1197Reputation: 1197Reputation: 1197Reputation: 1197Reputation: 1197Reputation: 1197Reputation: 1197
Quote:
Originally Posted by colucix View Post
Maybe you missed my last post...
You're right, I did Writing at the same time?

Quote:
Originally Posted by colucix View Post
The reason lies in the definition of standard input itself. Yes, without cat it tries to read from the standard input, but the standard input is the keyboard input, unless redirected. Cat performs this redirection: it takes the standard input redirected from the first cat into the outer loop and flushes its output through the pipe, which becomes the input source for the inner loop.

In other words, if the inner loop is left "alone":
Code:
  while read line; do
    echo $line $I
  done &
the read statement takes (or better, tries to take) standard input from the keyboard, since there is no redirection at all in this block of code!
Perhaps I am being dense here but ... why does the inner loop cat take the standard input redirected from the first cat into the outer loop but the "read" does not and picks up stdin from the keyboard?
 
Old 09-13-2009, 01:14 PM   #34
catkin
LQ 5k Club
 
Registered: Dec 2008
Location: Tamil Nadu, India
Distribution: Debian
Posts: 8,577
Blog Entries: 31

Rep: Reputation: 1197Reputation: 1197Reputation: 1197Reputation: 1197Reputation: 1197Reputation: 1197Reputation: 1197Reputation: 1197Reputation: 1197
Quote:
Originally Posted by colucix View Post
the first two characters "li" are missing from both files.

Ai posteri l'ardua sentenza! (A. Manzoni, Il Cinque Maggio, 1821)
Unexpected -- a very similar experiment worked perfectly for me as posted earlier. What does Ai posteri l'ardua sentenza! mean? BabelFish translated it as "To the posteri l' arduous sentence" which was not much advance on the original! "Sentence" as in text or as in "prison sentence"?!
 
Old 09-13-2009, 02:29 PM   #35
colucix
LQ Guru
 
Registered: Sep 2003
Location: Bologna
Distribution: CentOS 6.5 OpenSuSE 12.3
Posts: 10,509

Rep: Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981
Quote:
Originally Posted by catkin View Post
Perhaps I am being dense here but ... why does the inner loop cat take the standard input redirected from the first cat into the outer loop but the "read" does not and picks up stdin from the keyboard?
Outer loop: standard input redirected by cat $* through pipe:
Code:
cat $* | for I in `seq 1 "$max_count"`; do
this means the code inside the loop takes standard input from cat $*.

Inner loop: standard input redirected by cat through pipe:
Code:
cat | while read line; do
the loop takes input from the pipe.

The matter is: redirection must be explicit for a command to take input from some file descriptor. Otherwise standard input is expected by the keyboard (file descriptor 0 in C, unit 5 in fortran).

The man page for stdin (C programming) explains it well:
Quote:
Under normal circumstances every Unix program has three streams opened for it when it starts up, one for input, one for output, and one for printing diagnostic or error messages. These are typically attached to the user's terminal (see tty(4) but might instead refer to files or other devices, depending on what the parent process chose to set up.
In other words: who tells the inner loop to take input from the first cat $* and not from the keyboard? The second cat tells: "Hey hey... standard input is here! Don't look at the keyboard! I'm passing it through pipe...!"
 
Old 09-13-2009, 02:38 PM   #36
colucix
LQ Guru
 
Registered: Sep 2003
Location: Bologna
Distribution: CentOS 6.5 OpenSuSE 12.3
Posts: 10,509

Rep: Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981
Quote:
Originally Posted by catkin View Post
Unexpected -- a very similar experiment worked perfectly for me as posted earlier. What does Ai posteri l'ardua sentenza! mean? BabelFish translated it as "To the posteri l' arduous sentence" which was not much advance on the original! "Sentence" as in text or as in "prison sentence"?!
This is a quote from the first and greatest Italian novelist, Alessandro Manzoni. The quote is from one of his greatest poems (The Five of May) and literally means: "To posterity the arduous decision!". It is used when you report some facts, but you don't express a judgement.
 
Old 09-13-2009, 04:16 PM   #37
catkin
LQ 5k Club
 
Registered: Dec 2008
Location: Tamil Nadu, India
Distribution: Debian
Posts: 8,577
Blog Entries: 31

Rep: Reputation: 1197Reputation: 1197Reputation: 1197Reputation: 1197Reputation: 1197Reputation: 1197Reputation: 1197Reputation: 1197Reputation: 1197
Quote:
Originally Posted by colucix View Post
The quote is from one of his greatest poems (The Five of May) and literally means: "To posterity the arduous decision!". It is used when you report some facts, but you don't express a judgement.
Thanks for your translation which is more insightful (and useful) than the "Sentence waits posterity" given here. According to the Free Dictionary, the English word "sentence" has an obsolete meaning of "An opinion, especially one given formally after deliberation" which is close to Manzoni's use of sentenza.
 
Old 09-13-2009, 04:27 PM   #38
catkin
LQ 5k Club
 
Registered: Dec 2008
Location: Tamil Nadu, India
Distribution: Debian
Posts: 8,577
Blog Entries: 31

Rep: Reputation: 1197Reputation: 1197Reputation: 1197Reputation: 1197Reputation: 1197Reputation: 1197Reputation: 1197Reputation: 1197Reputation: 1197
Thank you for explaining but I still haven't got it. I'm OK with,
Quote:
Originally Posted by colucix View Post
Code:
cat $* | for I in `seq 1 "$max_count"`; do
this means the code inside the loop takes standard input from cat $*.
but I don't understand why the inner (while) loop's standard input isn't also coming from cat $*. AIUI (obviously wrongly!) the whole of the for-do-done compound statement is executed with standard input from cat $* and that includes the while-do-done compound statement. What causes the while-do-done to take standard input from somewhere else? What causes it to change?
 
Old 09-13-2009, 04:32 PM   #39
colucix
LQ Guru
 
Registered: Sep 2003
Location: Bologna
Distribution: CentOS 6.5 OpenSuSE 12.3
Posts: 10,509

Rep: Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981
Quote:
Originally Posted by catkin View Post
According to the Free Dictionary, the English word "sentence" has an obsolete meaning of "An opinion, especially one given formally after deliberation" which is close to Manzoni's use of sentenza.
That's true. Indeed, the poem was inspired to Manzoni from the conversion of Napoleon, a few days before his death. After narrating his deeds, he asked a question:
Code:
Fu vera gloria? Ai posteri l'ardua sentenza
that is: Was it true glory? To posterity the arduous decision!
Hence sentence as in the terms you depicted.

What do you think? Are we a little off-topic?
 
Old 09-13-2009, 04:41 PM   #40
colucix
LQ Guru
 
Registered: Sep 2003
Location: Bologna
Distribution: CentOS 6.5 OpenSuSE 12.3
Posts: 10,509

Rep: Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981
Quote:
Originally Posted by catkin View Post
... and that includes the while-do-done compound statement. What causes the while-do-done to take standard input from somewhere else? What causes it to change?
Don't forget that the while loop is launched in background, so that it is detached from the default standard input (tty), so that you're forced to make it to read from a pipe or from a file.

Last edited by colucix; 09-13-2009 at 05:15 PM. Reason: To focus attention on the most important part of the argumentation
 
Old 09-13-2009, 04:41 PM   #41
catkin
LQ 5k Club
 
Registered: Dec 2008
Location: Tamil Nadu, India
Distribution: Debian
Posts: 8,577
Blog Entries: 31

Rep: Reputation: 1197Reputation: 1197Reputation: 1197Reputation: 1197Reputation: 1197Reputation: 1197Reputation: 1197Reputation: 1197Reputation: 1197
Quote:
Originally Posted by colucix View Post
What do you think? Are we a little off-topic?
We sure are but I've just become a Senior Member so I'm allowed some "senior moments"! Enough -- back to standard inputs!
 
Old 09-13-2009, 04:58 PM   #42
catkin
LQ 5k Club
 
Registered: Dec 2008
Location: Tamil Nadu, India
Distribution: Debian
Posts: 8,577
Blog Entries: 31

Rep: Reputation: 1197Reputation: 1197Reputation: 1197Reputation: 1197Reputation: 1197Reputation: 1197Reputation: 1197Reputation: 1197Reputation: 1197
Quote:
Originally Posted by colucix View Post
Don't forget that the while loop is launched in background, so that it is detached from the default standard input (tty), so that you're forced to make it to read from a pipe or from a file.
Thanks, I've got it now. As it says in the GNU Bash Reference on backgrounding, "If a command is terminated by the control operator &, the shell executes the command asynchronously in a subshell. [snip] the standard input for asynchronous commands, in the absence of any explicit redirections, is redirected from /dev/null.". I was getting mixed up with fork(3C) which duplicates file descriptors and their buffers -- useless for an interactive shell backgrounding! Here's an alternative idiom, dispensing with the cat
Code:
max_count="$1"
shift

cat $* | for I in `seq 1 "$max_count"`; do
  while read line; do
    echo $line >> output$I.txt
  done <&0 &
done

Last edited by catkin; 09-14-2009 at 02:36 AM. Reason: Harmonise with colucix' edit
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
LXer: Reading Multiple Files with Bash LXer Syndicated Linux News 0 08-22-2009 12:00 AM
multiple processes read from a device jiankunli Linux - Kernel 3 05-09-2008 05:15 PM
bash: read multiple lines in a file 300zxkyle Programming 7 07-29-2007 04:38 AM
Bash: How to read tab character when reading input new_to_bash Programming 7 12-09-2006 07:31 PM
bash: create a descriptor reading/writing from/to different files? Thinking Programming 0 04-19-2006 03:28 PM

LinuxQuestions.org > Forums > Non-*NIX Forums > Programming

All times are GMT -5. The time now is 07:12 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration