[SOLVED] [ vs [[: filename expands between [[ ... ]] when it is not supposed to

andrew.comly · 05-31-2017, 12:46 AM

BACKGROUND
I am reading Advanced Bash Scripting by Mendel Cooper. In this eBook there is a Chapter 7 Tests, and between "Example 7-2. Equivalence of test" and "Example 7-3. Arithmetic Tests using (( ))" there is an enclosed section "The [[ ]] construct is the more versatile Bash version of [ ]."

TEXTBOOK
In this section it states: No filename expansion or word splitting takes place between [[ and ]], but there is parameter expansion and command substitution.

CONTRADICTION
However, the following example contradicts this.

Code:

$ foo=[a-z]* name=lhunath
$ [[ ${name} = "$foo" ]] && echo OK || echo NO
--> NO
$ [[ ${name} = $foo ]] && echo OK || NO
--> OK

PREMISES
Clearly the variable "foo", which equals "[a-z]*", contains a wildcard. And wildcards are used for filename expansion.

QUESTION
Can someone explain the contradiction between the former statement "No filename expansion ... takes place between [[ and ]]" and latter example where filename expansion does in fact take place between "[[" and "]]"?

REFLECTION
Looking back, I can see how the two statements in the text "No filename expansion or word splitting takes place between [[ and ]]" and "The [[ ]] construct is the more versatile Bash version of [ ]." aren't exactly parallel, so I'll understand if the author was understaffed (as many volunteers are) and had to hurry through this part to get to something more important.

pan64 · 05-31-2017, 12:55 AM

no, you missed a very importand point:

Code:

foo=[a-z]*

here will occur filename expansion, the value of foo is not the string [a-z]*.

(Sorry it is not true. The real explanation is in my next post)

astrogeek · 05-31-2017, 01:03 AM

Ah! But clearly "foo" does not equal "[a-z]*" within the [[..]]!

Code:

foo=[a-z]*
echo ${foo}

What does that do on your system?

pan64 · 05-31-2017, 01:15 AM

additionally you can use set -xv which will switch on debugging and you will see what's really happening

Code:

user@host:/tmp$ foo=[a-z]* name=lhunath
foo=[a-z]* name=lhunath
+ foo='[a-z]*'
+ name=lhunath
user@host:/tmp$ [[ ${name} = $foo ]] && echo OK || echo NO
[[ ${name} = $foo ]] && echo OK || echo NO
+ [[ lhunath = [a-z]* ]]
+ echo OK
OK
usr@host:/tmp$ [[ ${name} = "$foo" ]] && echo OK || echo NO
[[ ${name} = "$foo" ]] && echo OK || echo NO
+ [[ lhunath = \[\a\-\z\]\* ]]
+ echo NO
NO

Need to correct myself, that was not true. see man bash:

Code:

       [[ expression ]]
              ...
              Word  splitting  and  pathname expansion are not performed on the words between the [[ and ]]; tilde expansion, parameter and variable expansion, arithmetic expansion, command
              substitution, process substitution, and quote removal are performed

When the == and != operators are used, the string to the right of the operator is considered a pattern
and matched according to the rules described below under Pattern Matching, as if the extglob shell option were enabled.
The = operator is equivalent to ==. If the shell option nocasematch is enabled, the match is performed without regard to the case of alphabetic characters.
The return value is 0 if the string matches (==) or does not match (!=) the pattern, and 1 otherwise.
Any part of the pattern may be quoted to force the quoted portion to be matched as a string.

by the way you missed an echo before NO

rknichols · 05-31-2017, 09:06 AM

Quote:

Originally Posted by pan64

no, you missed a very importand point:

Code:

foo=[a-z]*

here will occur filename expansion, the value of foo is not the string [a-z]*.

Hmmm ...

Code:

$ set -x
$ foo=[a-z]*
+ foo='[a-z]*'
$ echo "$foo"
+ echo '[a-z]*'
[a-z]*
$ echo expands to: $foo
+ echo expands to: bin Desktop doc Documents Downloads Externals Financial icons mail Music Pictures Public soffice src Templates tmp Videos
expands to: bin Desktop doc Documents Downloads Externals Financial icons mail Music Pictures Public soffice src Templates tmp Videos

Indeed, variable foo is the literal string '[a-z]*'.

To the OP: There is no contradiction. When unquoted within the "[[ ... ]]' test, variable foo is interpreted as a pattern which means "begins with any single character 'a' through 'z' then followed by any number of any characters". Clearly, the string "lhunath" statsfies that condition.

pan64 · 05-31-2017, 11:30 AM

Quote:

Originally Posted by rknichols

Hmmm ...

Code:

$ set -x
$ foo=[a-z]*
+ foo='[a-z]*'
$ echo "$foo"
+ echo '[a-z]*'
[a-z]*
$ echo expands to: $foo
+ echo expands to: bin Desktop doc Documents Downloads Externals Financial icons mail Music Pictures Public soffice src Templates tmp Videos
expands to: bin Desktop doc Documents Downloads Externals Financial icons mail Music Pictures Public soffice src Templates tmp Videos

Indeed, variable foo is the literal string '[a-z]*'.

yes, I tried to correct it in the next post.

NevemTeve · 05-31-2017, 12:33 PM

Perhaps it depends on the content of the current directory: if it is empty, then [a-z]* won't be resolved to filenames. Also it might depend on the shell. Or the shell's current settings.

Edit: Nope, I misunderstood it.

MadeInGermany · 06-02-2017, 09:18 AM

Post#5 is right. The = operator in [[ ]] treats the right side as a pattern, just like in a case-esac.
For this reason I use == within [[ ]] (pattern!) and = within [ ] (simple equivalence), while of course the latter does the (argument-)expansions on both sides.

andrew.comly · 06-03-2017, 11:32 AM

VARIABLES
When I echo with quotes (literal value), I see rknichols's point that when setting the value of foo, the value is literally [a-z]*. When I echo w/o quotes (non-literal value/file expansion), unquoted variables containing wildcards expand.

QUOTES

Code:

set -x
$ echo "${foo} ${name}"
+ echo '[a-z]* lhunath'
[a-z]* lhunath

Thus the literal value is indeed [a-z]*.

NO QUOTES:

Code:

$ echo ${foo}; echo; echo ${name}
+ echo Desktop Documents Downloads ... Pictures Public 
Desktop Documents Downloads ... Pictures Public
+ echo

+ echo lhunath
lhunath

Above we see that without quotes, the variable expands.

CONDITIONAL STATEMENT

Code:

$ [[ ${name} == ${foo} ]] && echo Y || echo N
+ [[ lhunath == [a-z]* ]]
+ echo Y
Y
$ [ ${name} = ${foo} ] && echo Y || echo N
+ '[' lhunath = Desktop Documents Downloads ... Pictures Public ']'
bash: [: too many arguments
+ echo N
N

NOW I kind of feel that I am right back where I started from, facing the contradiction of 1) "No filename expansion ... takes place between [[ and ]]", and 2) the latter example.

Then it suddenly hits me, "[a-z]*" is a pattern, and when it expands within the "[[...]]", the variable is expanding, and no file expansion anywhere; The key process involved here is variable expansion!! Thus there is absolutely no contradiction. This problem stems from my miscategorization of [a-z]*. Thanks a lot rknichols and MadeInGermany!!

It's the damnest thing, but I always forget to utilize "set -x". Thanks Pan64 and rknichols for your kind reminders. Sincere regrets!

And yes, I forgot the echo before the last NO, thanks for the reminder pan64!

andrew.comly · 06-03-2017, 11:33 AM

QUESTION:
I still have to wonder, can anyone raise an example of

file expansion
word splitting?

I just can't find any examples on the net. What exactly is a file expansion??

rknichols · 06-03-2017, 12:55 PM

You have an example of file expansion (pathname expansion) above when you typed "echo ${foo}" and saw that "[a-z]*" was expanded to "Desktop Documents Downloads ... Pictures Public".

For an example of word splitting,

Code:

Dirs="Desktop Documents Downloads"
ls -ld $Dirs

The expansion of $Dirs is word-split into separate arguments to ls. Compare that to what happens for

Code:

ls -ld "$Dirs"

where the quotes prevent the word splitting, and the ls command doesn't find anything with the 29-character name, "Desktop Documents Downloads".

pan64 · 06-03-2017, 01:28 PM

filename expansion is for example: ls -ld D*
(try with and without " )

MadeInGermany · 06-04-2017, 10:53 AM

File name expansion and word splitting occur in a for loop (after the in) and in command arguments.
Commands are for example ls, cp, echo, printf, test.
Also [ is a command, indeed it is an alias of test - in [ mode it expects an additional last argument ].
file name expansion means the given word (argument) is taken as a pattern, and matches are done with file names in the current directory. The matching file names replace the given word. It usually becomes longer and can become many words --> "expansion".

A word splitting occurs when an unquoted variable is substituted by its value and if the value contains spaces.
Example

Code:

var="two words"
printf "%s\n" $var

BTW during file name expansion, if a file name contains spaces it is not split into words.
Example

Code:

touch "oneword" "two words"
var="two*"
printf "%s\n" $var

And what happens if there were

Code:

var="two *"

? Variable substitution and word splitting happen first!