> If my command launches 15 processes, are 14 of them sleeping at any given time?

The number of CPU-cores limits the number of paralel running processes, also the fact that the reader of a pipe has to wait for the writer of that pipe. (And vice verse, as the pipe has a limited capacity.)

1 members found this post helpful.

this is the official documentation: https://docs.kernel.org/scheduler/index.html
The people who created that kernel code made a lot of work to optimize this, so it is definitely not sub-optimal. Anyway, if you think you can do a better job just do it (or at least explain how can it be improved).

1 members found this post helpful.

Actually, the short answer is "no". Given a pipeline of 15 stages, 14 won't always be sleeping. The in/out data rates of the individual stages of the pipeline will determine which individual components are blocking on input or output — owing to full or empty fifo buffers — at any given time.

Assuming there is no competition for cpu resource, the overall performance of the pipeline will be dependent upon data-flow: constrained by the rate of its slowest component.

I see that the OP tagged this thread with “asynchronous task”.

Does he understand that pipelines are not “asynchronous”?

You can't just take a single example and use it as a basis for a performance discussion, unless the command sequence is one you specifically wish to optimize. But when you expand the question to cite "if 15 processes ...", then that's a different, but still specific and singular question.

Recommend reviewing the link provided by pan64 about the scheduler.

Imagine if there was a terrabyte of data. Would it be worthwhile to consider optimizations?

Sometimes there's a better way to do something, and I don't know about it.

The prior script was IO bound. The new one should be CPU bound.

The backslash is needed to escape backslashes.

When jq is processing data, bunzip (theoretically at least) can read more data from disk and decompress it. Likewise, when sed is processing data, jq can read from stdin and process that.

In other words, if the programs do not execute simultaneously, that would be like having assembly line with one car. In the (newer) code example, there's clearly room for eight cars.

Sed cannot read from stdin when jq is processing data. If you still don’t understand that, then don’t come back until you do.

True. However, jq does not need to wait for sed!

I think you are reffering to this code:
In theory, sed can read from stdin while bunzip is processing data!

What do you want to be told?

EDIT: Or, more specifically: what, exactly, do you want to get out of this thread?

The link looks potentially useful. Although it's a lot to digest for a question that I had intended to be a broadly generalizable about Bash programming.

But yet you expanded your question beyond bash.

That's all I can help with here, sorry but the question continues to change scope and it's unclear what you're looking for.

The generalization is: Bash is a tool for convenience, not performance.

yes, it is completely unclear what is it all about?
Using a pipe chains like that mentioned bzip| jq | cut | rev | whatever mean: all these commands will be started in the same time, will be executed independently from each other, only the output of one will be sent to the next in the chain. They will not wait for completion of any other member of the chain, but will wait for something to work with (=input data).

And it is actually completely independent from bash, this is handled, executed, processed and driven by the kernel. bash is just a language where you can construct pipe chains like this, but not the only one.
(therefore the speed of the execution again does not depend on bash at all)

Well, this is at least part of the story. For halfpower:

Process Life Cycle: States