[SOLVED] BASH: Odd behavior

gearge · 09-05-2011, 03:31 PM

Hi guys,

Here's what I'm trying to accomplish:
Write a quick one-liner for BASH, that will attempt to find the desired binary/executable under paths listed in $PATH environment variable.

I need this, because "whereis" and "which" commands do not support wildcards and there are many handy commands/utilities with names derived from other commands/utilities (e.g. "grep", "egrep", "diff" "sdiff").

My approach was to extract paths from $PATH and transform them in a way appropriate for passing to a for loop:

Quote:

[root@centos-pc1 ~]# echo $PATH | sed s/:/\\n/g | grep -v root | cat -A
/usr/kerberos/sbin$
/usr/kerberos/bin$
/usr/local/sbin$
/usr/local/bin$
/sbin$
/bin$
/usr/sbin$
/usr/bin$

Then run find with the desired expression once for every path in the list (a textbook for loop):

Quote:

[root@centos-pc1 ~]# for x in `echo $PATH | sed s/:/\\n/g | grep -v root`; do find $x -name "*diff"; done
[root@centos-pc1 ~]#

stderr output, not suppressed by a pipe:

Quote:

[root@centos-pc1 ~]# for x in `echo $PATH | sed s/:/\\n/g`; do find $x -name "*diff"; done
find: /usr/kerberos/sbinn/usr/kerberos/binn/usr/local/sbinn/usr/local/binn/sbinn/binn/usr/sbinn/usr/binn/root: No such file or directory
[root@centos-pc1 ~]#

"sbinn", "binn"... Can you explain how or why these extra n-s appeared at the end of each path?

Here's another way of receiving an analogues set of paths:

Quote:

[root@centos-pc1 ~]# find / -maxdepth 3 -type d | grep bin | grep -v var | cat -A
/sbin$
/bin$
/usr/local/sbin$
/usr/local/bin$
/usr/sbin$
/usr/bin$
/usr/kerberos/sbin$
/usr/kerberos/bin$

When used with a for loop, the approach works like a charm:

Quote:

[root@centos-pc1 ~]# for x in `find / -maxdepth 3 -type d | grep bin | grep -v var`; do find $x -name "*diff"; done
/usr/sbin/clockdiff
/usr/bin/gendiff
/usr/bin/sdiff
/usr/bin/bzdiff
/usr/bin/vimdiff
/usr/bin/zdiff
/usr/bin/diff
/usr/bin/ptardiff
/usr/bin/sediff

Though, this solution is rugged, clearly buggy and does not fully solve the initial problem.

Thanks in advance.

unSpawn · 09-05-2011, 03:44 PM

Three backslashes instead of two should do.

eSelix · 09-05-2011, 04:11 PM

Some offtopic. For people looking for such commands they do not known name, but known similar names or tags there is a command "apropos". For example:

Code:

$ apropos -s 1 grep
bzegrep (1)          - search possibly bzip2 compressed files for a regular expression
bzfgrep (1)          - search possibly bzip2 compressed files for a regular expression
bzgrep (1)           - search possibly bzip2 compressed files for a regular expression
egrep (1)            - print lines matching a pattern
fgrep (1)            - print lines matching a pattern
grep (1)             - print lines matching a pattern
pgrep (1)            - look up or signal processes based on name and other attributes
rgrep (1)            - print lines matching a pattern
xzegrep (1)          - search compressed files for a regular expression
xzfgrep (1)          - search compressed files for a regular expression
xzgrep (1)           - search compressed files for a regular expression
zegrep (1)           - search possibly compressed files for a regular expression
zfgrep (1)           - search possibly compressed files for a regular expression
zgrep (1)            - search possibly compressed files for a regular expression
zipgrep (1)          - search files in a ZIP archive for lines matching a pattern

$ apropos -s 1 diff
sg (1)               - execute command as different group ID
vimdiff (1)          - edytuj dwie, trzy lub cztery wersje pliku w Vimie i zobacz różnice
kdiff3 (1)           - compares two or three input files or directories
bzdiff (1)           - compare bzip2 compressed files
diff (1)             - compare files line by line
diff3 (1)            - compare three files line by line
kompare (1)          - a KDE GUI for viewing differences between files
xzcmp (1)            - compare compressed files
ndiff (1)            - Utility to compare the results of Nmap scans
patch (1)            - apply a diff file to an original
ptardiff (1)         - program that diffs an extracted archive against an unextracted one
sdiff (1)            - side-by-side merge of file differences
xzdiff (1)           - compare compressed files
zdiff (1)            - compare compressed files

It has even more interesting options.

ta0kira · 09-05-2011, 05:52 PM

Code:

eval /bin/ls -d1 "{$( echo "$PATH" | tr ':' ',' )}/$THE_PROGRAM" 2> /dev/null

Note that this will list the paths if $THE_PROGRAM is empty, which hopefully isn't that big of a deal. It also won't work if $PATH only has one directory, but hey, you asked for one line. It works with all bash wildcards and expansions, e.g. THE_PROGRAM='*diff'.
Kevin Barry

grail · 09-05-2011, 09:43 PM

How about simple substitution:

Code:

find ${PATH//:/ } -type f -name '*diff'

ta0kira · 09-05-2011, 10:32 PM

Quote:

Originally Posted by grail

How about simple substitution:

Code:

find ${PATH//:/ } -type f -name '*diff'

Might want to add -maxdepth 1 to that in case . is in the path.
Kevin Barry

gearge · 09-06-2011, 12:27 PM

Quote:

Three backslashes instead of two should do.

unSpawn, three backslashes instead of two indeed worked. Wicked... :-)

[root@centos-pc1 ~]# for x in `echo $PATH | sed s/:/\\\n/g`; do find $x -name "*diff"; done 2>/dev/null
/usr/sbin/clockdiff
/usr/bin/tiffdiff
/usr/bin/interdiff
/usr/bin/ptardiff
/usr/bin/pdiff
/usr/bin/unwrapdiff
/usr/bin/rcsdiff
/usr/bin/diff
/usr/bin/splitdiff
/usr/bin/pamtohdiff
/usr/bin/dehtmldiff
/usr/bin/combinediff
/usr/bin/sdiff
/usr/bin/zdiff
/usr/bin/vimdiff
/usr/bin/editdiff
/usr/bin/recountdiff
/usr/bin/filterdiff
/usr/bin/ibtopodiff
/usr/bin/sgmldiff
/usr/bin/grepdiff
/usr/bin/gendiff
/usr/bin/lsdiff
/usr/bin/rediff
/usr/bin/fixcvsdiff
/usr/bin/bzdiff
/usr/bin/flipdiff
/usr/bin/espdiff
/usr/bin/sediff
[root@centos-pc1 ~]#

Quote:

Some offtopic. For people looking for such commands they do not known name, but known similar names or tags there is a command "apropos".

eSelix, "apropos" and "whatis" turned out to be amazingly useful tools. But instead of searching in $PATH, they query the man pages, extracting paths with a similar trick:
(A line from “cat /usr/bin/apropos” output)

manpath=`man --path | tr : '\040'`

[root@centos-pc1 ~]# man --path | tr : '\040'
/usr/kerberos/man /usr/local/share/man /usr/share/man/en /usr/share/man
[root@centos-pc1 ~]#

Quote:

Originally Posted by grail

How about simple substitution:

Code:

find ${PATH//:/ } -type f -name '*diff'

[root@centos-pc1 ~]# find ${PATH//:/ } -type f -name '*diff' 2>/dev/null
/usr/sbin/clockdiff
/usr/bin/tiffdiff
/usr/bin/interdiff
/usr/bin/ptardiff
/usr/bin/pdiff
/usr/bin/unwrapdiff
/usr/bin/rcsdiff
/usr/bin/diff
/usr/bin/splitdiff
/usr/bin/pamtohdiff
/usr/bin/dehtmldiff
/usr/bin/sdiff
/usr/bin/zdiff
/usr/bin/editdiff
/usr/bin/recountdiff
/usr/bin/filterdiff
/usr/bin/ibtopodiff
/usr/bin/sgmldiff
/usr/bin/gendiff
/usr/bin/rediff
/usr/bin/fixcvsdiff
/usr/bin/bzdiff
/usr/bin/espdiff
/usr/bin/sediff

I just realized, that one can directly provide several paths separated by spaces to “find” utility.

[root@centos-pc1 ~]# echo $PATH | sed s/:/\ /g
/usr/kerberos/sbin /usr/kerberos/bin /usr/local/sbin /usr/local/bin /sbin /bin /usr/sbin /usr/bin /root/bin
[root@centos-pc1 ~]# x=`echo $PATH | sed s/:/\ /g`; find $x -type f -name '*diff' 2>/dev/null
/usr/sbin/clockdiff
/usr/bin/tiffdiff
/usr/bin/interdiff
/usr/bin/ptardiff
/usr/bin/pdiff
/usr/bin/unwrapdiff
/usr/bin/rcsdiff
/usr/bin/diff
/usr/bin/splitdiff
/usr/bin/pamtohdiff
/usr/bin/dehtmldiff
/usr/bin/sdiff
/usr/bin/zdiff
/usr/bin/editdiff
/usr/bin/recountdiff
/usr/bin/filterdiff
/usr/bin/ibtopodiff
/usr/bin/sgmldiff
/usr/bin/gendiff
/usr/bin/rediff
/usr/bin/fixcvsdiff
/usr/bin/bzdiff
/usr/bin/espdiff
/usr/bin/sediff
[root@centos-pc1 ~]#

This is great. Now, can you please explain the regular expression used in “${PATH//:/ }” variable?

Thanks a bunch grail.

Quote:

Originally Posted by ta0kira

Might want to add -maxdepth 1 to that in case . is in the path.
Kevin Barry

ta0kira, thanks. But I would prefer not to keep the current path in the $PATH variable - it’s a clear security hazard.

Thank you all very much.

Diantre · 09-06-2011, 08:19 PM

[deleted]

grail · 09-06-2011, 11:38 PM

Just for clarification, it is parameter substitution. The reason I make the difference is because it uses globbing and not regex.

gearge · 09-07-2011, 10:47 AM

Quote:

That regular expression replaces the colons in $PATH with spaces. You can try it and see:

Yeah Diantre, it was clear what the purpose of ${PATH//:/ } was - to replace all occurrences of “:” with “ “, in value of variable $PATH. Though, the syntax (or how it was doing this) was unclear, so I enquired about it. The closest thing it reminded me of was a regular expressions... a rather odd one indeed. Well, turns out it’s not a regexp after all.

Quote:

Just for clarification, it is parameter substitution. The reason I make the difference is because it uses globbing and not regex.

grail, thanks, this is exactly what I was interested in. I will look closer at parameter substitution and globbing.

David the H. · 09-07-2011, 06:02 PM

Much better than using "three slashes", just quote the whole expression (this should always be done with sed and awk):

Code:

while read x ; do find "$x" -name "*diff" ; done < <( echo "$PATH" | sed 's/:/\n/g' )

But tr is the tool to use when substituting single characters, not sed.

As shown though, parameter substitution and globbing are powerful tools that can often replace external tools like these:

http://mywiki.wooledge.org/BashFAQ/073
http://mywiki.wooledge.org/glob

Notice also the other changes I made...

Always quote variables, to avoid word splitting:
http://mywiki.wooledge.org/Arguments
http://mywiki.wooledge.org/WordSplitting
http://mywiki.wooledge.org/Quotes

pathnames should generally be processed with while+read loops, not for, also because of word splitting:
http://mywiki.wooledge.org/DontReadLinesWithFor
http://mywiki.wooledge.org/BashFAQ/001

Process substitution is the recommended way to feed such a loop.
http://mywiki.wooledge.org/ProcessSubstitution

In addition, backticks are deprecated, replaced with the much easier to use and read $(..):
http://mywiki.wooledge.org/BashFAQ/082

gearge · 09-08-2011, 10:19 AM

Quote:

Originally Posted by David the H.

Much better than using "three slashes", just quote the whole expression (this should always be done with sed and awk):

Code:

while read x ; do find "$x" -name "*diff" ; done < <( echo "$PATH" | sed 's/:/\n/g' )

But tr is the tool to use when substituting single characters, not sed.

As shown though, parameter substitution and globbing are powerful tools that can often replace external tools like these:

http://mywiki.wooledge.org/BashFAQ/073
http://mywiki.wooledge.org/glob

Notice also the other changes I made...

Always quote variables, to avoid word splitting:
http://mywiki.wooledge.org/Arguments
http://mywiki.wooledge.org/WordSplitting
http://mywiki.wooledge.org/Quotes

pathnames should generally be processed with while+read loops, not for, also because of word splitting:
http://mywiki.wooledge.org/DontReadLinesWithFor
http://mywiki.wooledge.org/BashFAQ/001

Process substitution is the recommended way to feed such a loop.
http://mywiki.wooledge.org/ProcessSubstitution

In addition, backticks are deprecated, replaced with the much easier to use and read $(..):
http://mywiki.wooledge.org/BashFAQ/082

David, thank you very much for pointing to these very useful resources, sharing tips and insight on the subject.