Below is an example that extracts links from a web page.
It's clear to me that each command will work before the others have completed, but will they block each other too? Will curl, grep, and sed run simultaneously on different logical cores?

The pipes take care of it.

The output of the first one is the input to the next one. And so on.

Yes they will run at the same time but the pipes feed the following two where they depend on the earlier process.

What does it matter how a CPU chooses to run processes?

They run simultaneously, and each one blocks the one to its right.

Sed blocks until it gets a line from grep, and grep blocks until it gets a line from curl.

When the primary factor limiting execution time is CPU power, then, it can affect total runtime. Theoretically, it should become a big issue when there's a large amount of data, a lot of commands chained together, and many unutilized CPU cores.

Right here the network speed will be the limiting factor.

1 members found this post helpful.

And it goes without saying that processing HTML without a proper parser can be quite brittle. An unexpected space or line break, although valid html, with choke the sed script.

The xmlstarlet utility is just one option. The are parsers for Perl, Python3, and other scripting languages.

PS. clownflare tries to block me from posting the 'concat' part above. >:(

1 members found this post helpful.

No. Obviously the computing capabilities will limit the speed of execution, but linux (the kernel) will be able to utilize all the cores (not only one) and therefore there will be no unutilized cores (don't forget, all the system is running, containing at least several hundred processes).

1 members found this post helpful.

Which is ugly, flawed, and unnecessarily slow.

Asking about CPU core allocation is a micro optimization; if you have a performance issue, you have bigger factors to address first.

Is there a larger problem that prompted this line of thinking, or is it just a "what if" thought?

Depends on the web page, grep will do that by itself. What the OP is trying to do anyway. Although a web browser or html/xml parser will do it better.

The more pipes the more subshells.

Are you actually having a performance issue that you’re trying to solve here, or was this meant to be a model?

Keep in mind that IPC isn’t fast. The typical performance-centric approach is to to break up the data, distribute each chunk to a program that does not know about the others and which processes its part of the data to a central location, and then wait for all of the individual programs to finish.

BTW, you do understand that the Linux kernel puts processes to "sleep" (in quotes because it's a technical word) when they're waiting for input, right?

I was trying demonstrate the issue. This might be a better example:

Neither example would benefit from multiple cores. The reasons have been said many times, but jq in particular needs to wait for the entire curl or bunzip command to finish before it even starts.

Also, you can use bzcat instead of bunzip2 --stdout.

It's not. End of story.

Or have you actually seen top/iostat/sar output showing that?

The question is more theoretical in nature. The code (which has been edited) is only intended to illustrate the issue.

Some data is stored in a monolithic format. At the present time, I have no method or on-the-fly splitting.

My question was whether or not the processes would block the execution of the other processes. In other words: If my command launches 15 processes, are 14 of them sleeping at any given time? If they are, it is very sub-optimal.

What issue?

Put another way: You don't have a performance issue until you can demonstrate a measurable issue.

Whether the answer to your question is yes or no, what difference is it going to make?

If it runs quickly enough, nobody cares what core it executes on.

If it doesn't run quickly enough, switching to forced parallel execution is going to make the code less maintainable, and very likely having less of an impact than optimising whatever algorithm(s) might be involved and/or using a lower-level language for the task.


It's really not.

Even if you add the missing -z argument to sed, the backslash shouldn't be escaped and the group is unnecessary, but it's far simpler to use tr to replace newlines.

But if we pretend you did use tr, you still have to consider that jq (unlike grep/sed) will wait for stdin to complete before parsing the object, so it doesn't demonstrate any meaningful simultaneous execution.

Yes. The short answer is yes, and it's been explained to you exactly how that works. How many times do you need to hear "yes" before it gets through to you?

I'm starting to think you're deliberately ignoring the actual answers.