LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices


Reply
  Search this Thread
Old 04-27-2021, 04:37 AM   #1
igadoter
Senior Member
 
Registered: Sep 2006
Location: wroclaw, poland
Distribution: many, primary Slackware
Posts: 2,248
Blog Entries: 1

Rep: Reputation: Disabled
Process substitution in bash


I want to discuss the feature - and try to understand how it works. Here real-life example to count md5sum on a fly https://www.linuxquestions.org/quest...ml#post6245276. Ok start with this
Code:
% echo >(:)
what's the output? And the question: what really is "process substitution" when working in command line?
 
Old 04-27-2021, 06:23 AM   #2
pan64
LQ Guru
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 16,490

Rep: Reputation: 5532Reputation: 5532Reputation: 5532Reputation: 5532Reputation: 5532Reputation: 5532Reputation: 5532Reputation: 5532Reputation: 5532Reputation: 5532Reputation: 5532
https://tldp.org/LDP/abs/html/process-sub.html
 
Old 04-27-2021, 06:39 AM   #3
igadoter
Senior Member
 
Registered: Sep 2006
Location: wroclaw, poland
Distribution: many, primary Slackware
Posts: 2,248

Original Poster
Blog Entries: 1

Rep: Reputation: Disabled
I can read manuals - can find manuals - can make a coffee. I was counting on discussion here - but perhaps LQ is not a place to discuss - just watch.
 
Old 04-27-2021, 06:49 AM   #4
boughtonp
Senior Member
 
Registered: Feb 2007
Location: UK
Distribution: Debian
Posts: 1,339

Rep: Reputation: 1132Reputation: 1132Reputation: 1132Reputation: 1132Reputation: 1132Reputation: 1132Reputation: 1132Reputation: 1132Reputation: 1132
Quote:
Originally Posted by igadoter View Post
I want to discuss the feature - and try to understand how it works.
High level or low-level?


Quote:
Originally Posted by igadoter
Ok start with this
Code:
% echo >(:)
what's the output?
"bash: fg: %: no such job"


Quote:
Originally Posted by igadoter
And the question: what really is "process substitution" when working in command line?
Answered by the link Pan provided, as well as in the Bash manual: "Process substitution allows a processís input or output to be referred to using a filename."

 
Old 04-27-2021, 06:57 AM   #5
pan64
LQ Guru
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 16,490

Rep: Reputation: 5532Reputation: 5532Reputation: 5532Reputation: 5532Reputation: 5532Reputation: 5532Reputation: 5532Reputation: 5532Reputation: 5532Reputation: 5532Reputation: 5532
This was written on the link I posted:
Quote:
Process substitution uses /dev/fd/<n> files to send the results of the process(es) within parentheses to another process.
I can't really explain it better. What is unclear? Actually what do you want to discuss?
 
Old 04-27-2021, 07:01 AM   #6
igadoter
Senior Member
 
Registered: Sep 2006
Location: wroclaw, poland
Distribution: many, primary Slackware
Posts: 2,248

Original Poster
Blog Entries: 1

Rep: Reputation: Disabled
Quote:
Originally Posted by boughtonp View Post
"bash: fg: %: no such job"
Your bash version is different than mine
Code:
 
% echo >(:)
/dev/fd/63
To discuss on every possible level. Bash can be really pita - it is good to stop sometime to resume. Say -u option (read from file descriptor) in read internal command is also a pita - but that's different story.
 
Old 04-27-2021, 07:02 AM   #7
igadoter
Senior Member
 
Registered: Sep 2006
Location: wroclaw, poland
Distribution: many, primary Slackware
Posts: 2,248

Original Poster
Blog Entries: 1

Rep: Reputation: Disabled
Quote:
Originally Posted by pan64 View Post
I can't really explain it better. What is unclear? Actually what do you want to discuss?
Possible applications.
 
Old 04-27-2021, 07:40 AM   #8
boughtonp
Senior Member
 
Registered: Feb 2007
Location: UK
Distribution: Debian
Posts: 1,339

Rep: Reputation: 1132Reputation: 1132Reputation: 1132Reputation: 1132Reputation: 1132Reputation: 1132Reputation: 1132Reputation: 1132Reputation: 1132
Quote:
Originally Posted by igadoter View Post
Your bash version is different than mine
Code:
$ bash --version
GNU bash, version 5.0.3(1)-release (x86_64-pc-linux-gnu)

$ % echo >(:)
-bash: fg: %: no such job

$ help % echo >(:)
%: job_spec [&]
    Resume job in foreground.

    Equivalent to the JOB_SPEC argument to the `fg' command.  Resume a
    stopped or background job.  JOB_SPEC can specify either a job name
    or a job number.  Following JOB_SPEC with a `&' places the job in
    the background, as if the job specification had been supplied as an
    argument to `bg'.

    Exit Status:
    Returns the status of the resumed job.
echo: echo [-neE] [arg ...]
    Write arguments to the standard output.

    ...
Achieving the output in your post requires removing the % from the command...


Quote:
To discuss on every possible level.
It's generally easier to understand subjects by focusing on one area at a time.

 
Old 04-27-2021, 07:54 AM   #9
igadoter
Senior Member
 
Registered: Sep 2006
Location: wroclaw, poland
Distribution: many, primary Slackware
Posts: 2,248

Original Poster
Blog Entries: 1

Rep: Reputation: Disabled
I see "% " is command line prompt here - just I started to use % (only) as prompt PS1="% "
 
Old 04-27-2021, 08:16 AM   #10
GazL
LQ Veteran
 
Registered: May 2008
Posts: 5,923

Rep: Reputation: 3899Reputation: 3899Reputation: 3899Reputation: 3899Reputation: 3899Reputation: 3899Reputation: 3899Reputation: 3899Reputation: 3899Reputation: 3899Reputation: 3899
>( cmd ) sets up a fifo(pipe) connected to the stdin of cmd and substitutes bash's internal filename for that pipe on the command-line.

Unlike $( ) it substitutes a filename rather than the output of cmd so you have to use it were a filename is expected.

The <( cmd ) input variant is more useful than the output one in my experience.

If you run set -x you'll clearly see what's really happening.
Code:
$ set -x
$ cat <( echo wibble ) <( echo wobble )
+ cat /dev/fd/63 /dev/fd/62
++ echo wibble
++ echo wobble
wibble
wobble
$
However, one problem with real world use of these is that you can't detect an error condition from the substituted process very easily, so best used sparingly, or where you know the process can't, or at least is unlikely to, fail.

Last edited by GazL; 04-27-2021 at 08:32 AM.
 
1 members found this post helpful.
Old 04-27-2021, 08:28 AM   #11
GazL
LQ Veteran
 
Registered: May 2008
Posts: 5,923

Rep: Reputation: 3899Reputation: 3899Reputation: 3899Reputation: 3899Reputation: 3899Reputation: 3899Reputation: 3899Reputation: 3899Reputation: 3899Reputation: 3899Reputation: 3899
Quote:
Originally Posted by igadoter View Post
Possible applications.
Here's an example usage from a snippet of my 'slacklist' utility where I use these substitutions when merging data into a table.
Code:
  list_available >&$available
  list_installed >&$installed

  sort -u -k 1,1 <( cut -f1 -d ' ' /dev/fd/$installed ) \
                 <( cut -f1 -d ' ' /dev/fd/$available ) \
    | join -a 1 -e 'not-installed' -o 1.1,2.2 - /dev/fd/$installed \
    | join -a 1 -e 'not-available' -o 1.1,1.2,2.2,2.3 - /dev/fd/$available \
    >&$table

Last edited by GazL; 04-27-2021 at 08:37 AM.
 
1 members found this post helpful.
Old 04-27-2021, 08:38 AM   #12
igadoter
Senior Member
 
Registered: Sep 2006
Location: wroclaw, poland
Distribution: many, primary Slackware
Posts: 2,248

Original Poster
Blog Entries: 1

Rep: Reputation: Disabled
I have combination with here document
Code:
% echo World > file
% cat <(cat <<EOF
> Hello
> EOF
> ) file
Hello
World
 
Old 04-27-2021, 08:58 AM   #13
GazL
LQ Veteran
 
Registered: May 2008
Posts: 5,923

Rep: Reputation: 3899Reputation: 3899Reputation: 3899Reputation: 3899Reputation: 3899Reputation: 3899Reputation: 3899Reputation: 3899Reputation: 3899Reputation: 3899Reputation: 3899
You know bash has a dedicated substitution for reading from a file?
Code:
$ echo world >/tmp/file
$ cat <<EOF
> hello $(< /tmp/file)
> EOF
hello world
$

Last edited by GazL; 04-27-2021 at 09:00 AM. Reason: clarification
 
1 members found this post helpful.
Old 04-27-2021, 09:04 AM   #14
shruggy
Senior Member
 
Registered: Mar 2020
Posts: 2,321

Rep: Reputation: Disabled
Quote:
Originally Posted by GazL View Post
The <( cmd ) input variant is more useful than the output one in my experience.
Here are a couple of examples of how I used the output variant when doing Kakoune golf:
Code:
#!/bin/bash
# https://www.vimgolf.com/challenges/4d1a71c0b8cb34093200010b
# Remember FizzBuzz?
# Output FizzBuzz to 100. Start with nothing.
kak \
 -f '33o<ret>Fizz<ret><esc> QABuzz<esc>5kQ19q%<a-s>"#P<a-k>z<ret>i <esc>' \
  >(cat)
Code:
#!/bin/bash
# https://www.vimgolf.com/challenges/5c4d042acfafb4000c9f06c2
# One to Ten
# Generate the sequence of numbers from 1 to 10, one number per line.
# Inspired by this Reddit thread: https://redd.it/ak4it2
kak -f '9<a-o>9C"#P' >(cat)
 
Old 04-27-2021, 09:24 AM   #15
igadoter
Senior Member
 
Registered: Sep 2006
Location: wroclaw, poland
Distribution: many, primary Slackware
Posts: 2,248

Original Poster
Blog Entries: 1

Rep: Reputation: Disabled
Quote:
Originally Posted by GazL View Post
You know bash has a dedicated substitution for reading from a file?
Code:
$ echo world >/tmp/file
$ cat <<EOF
> hello $(< /tmp/file)
> EOF
hello world
$
Don't get it
Code:
$(< /tmp/file)
is command output substitution - but where's command ? < /tmp/file is just redirection.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[bash] why is Process Substitution <() so much faster ?! hashbang#! Programming 4 11-26-2010 04:19 AM
[SOLVED] Bash: functional difference between process substitution and here string with $( ) ? catkin Programming 2 03-15-2010 02:08 AM
Problems with Process Substitution and scp - Solaris to Linux sdduuuude Linux - Newbie 3 09-28-2007 10:47 AM
Bash Process Substitution joshholt Programming 4 10-11-2005 03:15 AM
process substitution jk3us Linux - Software 1 12-02-2004 04:34 PM

LinuxQuestions.org > Forums > Non-*NIX Forums > Programming

All times are GMT -5. The time now is 11:43 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration