LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices


Reply
  Search this Thread
Old 01-17-2008, 06:25 PM   #1
jamesj629
LQ Newbie
 
Registered: Jan 2008
Posts: 14

Rep: Reputation: 0
Question Q: are pipelines processed in parallel/serial?


Do pipelines start multiple threads so that data potentially flows from start to final output at once? (Given enough CPUs)

For instance...

cat data | awk 'some conditional operations' | sed 's/xyz/def/' | tr '@' ':' | awk 'some more' | sort

Do each of these stages have their own process?
Is cat assured to finish before handing over control to awk?

Many thanks for clearing this up :P

~James
 
Old 01-17-2008, 06:49 PM   #2
Tinkster
Moderator
 
Registered: Apr 2002
Location: in a fallen world
Distribution: slackware by choice, others too :} ... android.
Posts: 23,067
Blog Entries: 11

Rep: Reputation: 912Reputation: 912Reputation: 912Reputation: 912Reputation: 912Reputation: 912Reputation: 912Reputation: 912
hi,

And welcome to LQ!

Just a thought: I'd guess this depends on the receiving process, not on
the pipe. E.g. it may make sense for awk as the receiver (right-hand side
of a pipe) to do its thing to each line as they dash past, but it certainly
doesn't make sense to start sorting before you've seen all data.


Pure speculation, I never really thought about it in the past ;}


Cheers,
Tink
 
Old 01-17-2008, 07:22 PM   #3
ilikejam
Senior Member
 
Registered: Aug 2003
Location: Glasgow
Distribution: Fedora / Solaris
Posts: 3,109

Rep: Reputation: 96
Hi.

The pipes are set up in parallel, but as Tink suggests, it's up to the utilities whether they pass the data through 'live' or not.

Try ping into awk:
$ ping localhost | awk '{print $(NF-1)}'
and it works in parallel

Then stick sed on the end:
$ ping localhost | awk '{print $(NF-1)}' | sed '1d'
and sed waits for the end of the input stream before doing anything.

Dave
 
Old 01-17-2008, 07:26 PM   #4
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 18,229

Rep: Reputation: 2964Reputation: 2964Reputation: 2964Reputation: 2964Reputation: 2964Reputation: 2964Reputation: 2964Reputation: 2964Reputation: 2964Reputation: 2964Reputation: 2964
"man 7 pipe"
 
Old 01-17-2008, 07:43 PM   #5
jamesj629
LQ Newbie
 
Registered: Jan 2008
Posts: 14

Original Poster
Rep: Reputation: 0
thanks all
 
Old 01-17-2008, 08:17 PM   #6
sundialsvcs
LQ Guru
 
Registered: Feb 2004
Location: SE Tennessee, USA
Distribution: Gentoo, LFS
Posts: 9,078
Blog Entries: 4

Rep: Reputation: 3177Reputation: 3177Reputation: 3177Reputation: 3177Reputation: 3177Reputation: 3177Reputation: 3177Reputation: 3177Reputation: 3177Reputation: 3177Reputation: 3177
A "pipe" is an inter-process communication (IPC) channel. (It's one of several.)

Basically, a "pipe" presents itself as a file, which a particular process can either read from or write to. The "pipe," then, becomes a buffered communications-mechanism between its "reader" and its "writer," always appearing to both of them as "just a file." But here's the magic...
  1. If "you" are reading from the pipe, but the writer (still exists and...) has not yet written anything (more) to the pipe, "you" will be put to sleep until the writer does write something. Then, you'll be able to read what the writer has just written.
  2. If "you" are writing, and the pipe becomes "full," you will be put to sleep until the pipe is no longer full.
So, with all that having been said, the two processes .. the "reader" and the "writer" .. are free to run on whatever CPUs they can find, as best they can manage, at the sole discretion of the system scheduler.

When you, in the shell, type something like ls | grep foo, you actually cause two processes to be launched: one is ls, which writes its output to its STDOUT, and the other is grep, which reads its input from its STDIN. And... (magic time!) the STDOUT from the one is the STDIN of the other! It's a pipe.

(Please step outside the room while your brain explodes. We've all been there... we don't mind. Now, when you come back into the room, you ought to be saying either "Sweet!" or else, "That is so way k-e-w-e-l!"]

Yeah, those dudes at Bell Labs way back in the 1970's {I was almost-there, but nevermind!} had some pretty mind-blowing ideas...
 
Old 01-18-2008, 07:18 AM   #7
ilikejam
Senior Member
 
Registered: Aug 2003
Location: Glasgow
Distribution: Fedora / Solaris
Posts: 3,109

Rep: Reputation: 96
There are two major products that come out of Berkeley: LSD and UNIX. We don't believe this to be a coincidence.

- Jeremy S. Anderson
 
Old 01-18-2008, 08:58 AM   #8
jamesj629
LQ Newbie
 
Registered: Jan 2008
Posts: 14

Original Poster
Rep: Reputation: 0
There are times when I think everything would make a lot more sense if I were on LSD. Then I remember I prefer the purple pills the doctor gives me. *munch* *munch*
 
Old 01-18-2008, 07:55 PM   #9
osor
HCL Maintainer
 
Registered: Jan 2006
Distribution: (H)LFS, Gentoo
Posts: 2,450

Rep: Reputation: 77
In addition to the great explanation by sundialsvcs, there is another thing about unix pipes which is useful but requires care in some circumstances (actually there are a few other such things, but I will only talk about one that relates somewhat to your question): When a process is writing to a pipe and the file descriptor on the “read” end has been closed, the process is sent a SIGPIPE signal. The default action for receiving such a signal (i.e., the action taken unless the program explicitly handles or ignores the signal) is to terminate the program.

As you might imagine, this behavior presents great potential for use. Here is a prototypical example of the kind of situation for which it was intended: suppose you have a really big gzip file, of which you want to read the first few lines to see what is inside. You could do something like this:
Code:
zcat reallybigfile.gz | head
The under normal circumstances, the zcat might take a minute or more to execute, but since only the first ten lines are desired, the execution of zcat is “short-circuited”. What happens is this: A pipe is opened with the “write” end replacing stdout of zcat and the “read” end replacing stdin of head. The process for zcat does its job and starts deflating the file chunk-by-chunk and writing it to stdout. The head utility does its job and reads 10 lines from its stdin and writes them to its stdout. Afterward, it exits (and part of that exiting involves the closing of the “read” end of the pipe). When zcat tries to write additional data to the “broken pipe”, it receives a SIGPIPE signal and terminates without completely deflating the entire file. This saves you a lot of time/cpu cycles, and satisfies “do what I mean”.

As you might also imagine, you can also abuse this functionality, and it might even get you into trouble if you aren’t careful.
 
Old 01-19-2008, 03:33 AM   #10
jamesj629
LQ Newbie
 
Registered: Jan 2008
Posts: 14

Original Poster
Rep: Reputation: 0
This is probably one of the more interesting linux topics I've covered so far. Thanks a lot guys
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
serial/parallel loopback testing Serena Linux - Hardware 2 08-30-2007 05:17 AM
Serial/Parallel/Game Port Interfacing sparkyssb Programming 8 06-29-2006 02:41 PM
serial and parallel port with DMESG command froglinux Linux - Hardware 1 06-16-2006 09:51 PM
Parallel to serial adapter traybore19 Linux - Hardware 3 07-21-2005 11:34 PM
serial/parallel port marco_aq Linux - Networking 1 10-12-2001 03:44 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - General

All times are GMT -5. The time now is 04:19 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration