bash: case and command substitution--what gives?

David the H. · 08-21-2009, 10:02 AM

I've been trying to embed a simple filtering statement into another command, but I keep getting a syntax error.

I start with something like this to grab files with a specified extension $ext:

Code:

for i; do
  case $i in
     *$ext) echo $i;;
  esac
done

This works perfectly on its own. But then when I try to embed it to say, set a variable:

Code:

files=$(
for i; do
  case $i in
     *$ext) echo $i;;
  esac
done
)

It always gives me "syntax error near unexpected token `;;'"

However, I can change the case structure to something else, like:

Code:

files=$(
for i; do
  if [[ "$i" = *$ext ]]; then
     echo $i
  fi
done
)

...with no errors.

So why doesn't case work here? What am I overlooking?

ilikejam · 08-21-2009, 10:20 AM

Hi.

The ')' in the case statement is matching the '(' from files=$(

Try using files=`blah blah blah` instead of files=$(blah blah blah)

Dave

David the H. · 08-21-2009, 11:14 AM

Well, whaddaya know. That works. I hadn't thought about trying the backticks.

But I had considered the idea that the ')' was causing it. I decided that that couldn't be it, however, because I can surround it with parentheses to create a subshell.

Code:

(for i; do
  case $i in
     *$ext) echo $i;;
  esac
done)

This gives no errors. It's only when I enclose it in $() that it breaks. Odd.

jlliagre · 08-21-2009, 11:58 AM

Here is the way your original code should be written:

Code:

files=$(
for i; do
  case $i in
     (*$ext) echo $i;;
  esac
done
)

David the H. · 08-21-2009, 08:41 PM

What, you can do that? I've never seen anything anywhere that said you could completely enclose the matching conditions in a case construct.

In any case, could this be a bug? I'd certainly consider it one. I don't think it should be necessary to materially alter the syntax of the commands inside of $(), particularly if it's not well-documented.

Edit: Did a little google searching, now that I know what's going on. Apparently it's a long-standing bug that's difficult to handle at the compiler level. And that appears to be the reason the fully-enclosed tests were introduced.

It would be nice if it were better-documented though. At least the ABSG could've mentioned it.

Edit two: This page has a chart that lists which shells can and can't handle various "unbalanced" parentheses statements inside $().

jlliagre · 08-22-2009, 02:12 AM

I wouldn't tell it's a shell bug as it was identified when the $() syntax was introduced by the korn shell a couple of decade ago, just a well known incompatibility between an old syntax and a newer one with a documented workaround.

I have been using the extra left parenthesis in all my case/esac statements for ages. It looks to me neater and more convenient than the original unbalanced design anyway. Despite being accepted by the standard, the newer shells accepting unbalanced parenthesis looks an ugly hack to me.

It is worth noticing the backquote syntax was considered as archaic in 1992 in the ksh documentation but is still widely used nowadays.

David the H. · 09-01-2009, 11:57 AM

Got a follow-up to this topic.

Debian has just included Bash v.4 in it's repositories (at least in unstable), and now the problem is gone.

They've fixed this little error.

I did come across some new problems, however. One of my scripts uses an embedded here document, and it started spitting out an error after the upgrade. I was using something like this:

Code:

output=$(cat <<-FOOBAR

Sample text.
Sample text.
Sample text.

FOOBAR)

echo "$output"

This worked fine in bash 3, but breaks in bash 4. Apparently it now chokes if there's anything else on the same line as the here statement's terminating string. Even comments mess it up. Moving the ")" to the next line lets it run.

Unfortunately though, there also seems to be a new, different bug. Any and all trailing newlines, in any kind of text output, now appear to be removed when inside $(). They not only disappear in the above code, but even when using something like echo.

Code:

#!/bin/bash

output="$(
echo "Sample text"
echo "Sample text"
echo "Sample text"
echo
echo -e "\n\n\n"
)"
echo "$output"
echo "======="

### The output ###

$ ./test.sh
Sample text
Sample text
Sample text
=======
$

Just about every test I've tried with text inside $() has shown the same effect. All trailing newlines disappear. Spaces and tabs do show up though.

I suppose I'll have to switch to using functions for generating large, formatted blocks of text instead.

All in all, though, bash 4 does seem to have some neat new features. I'm looking forward to playing with them.

jlliagre · 09-02-2009, 07:59 PM

These aren't two new bugs but actually two bug fixes.

Both removing trailing newlines and having the here document alone in its line are requirement to comply with the POSIX standard.

Quoted from http://www.opengroup.org/onlinepubs/...cu_chap02.html

The shell shall expand the command substitution by executing command in a subshell environment (see Shell Execution Environment) and replacing the command substitution (the text of command plus the enclosing "$()" or backquotes) with the standard output of the command, removing sequences of one or more <newline>s at the end of the substitution.

...

The here-document shall be treated as a single word that begins after the next <newline> and continues until there is a line containing only the delimiter and a <newline>, with no <blank>s in between.

David the H. · 09-03-2009, 02:03 AM

Thank you for that information. To be fair, I didn't say that the first one was a bug, only that the behavior was different than before.

The removing all trailing newlines thing is a bit annoying though. It's my opinion that command substitution should, you know, substitute, the complete output of the embedded commands, no matter what they contain. Editing shouldn't be part of the process, even if it's just a few extra newlines. It should be up to the scripter to ensure that the substitution produces the desired output.

I was used to depending on the former behavior. Now I'm going to have to rewrite a couple of my old scripts.

jlliagre · 09-03-2009, 02:08 AM

Yes, I agree that can be annoying. Actually, I'm missing what the rationale might be about removing trailing newlines.