[SOLVED] Dash shell script variable expansion problems ;/

Kaurin · 02-01-2013, 03:41 AM

This work is being done on a test virtualbox machine

In my /root dir, i have created the following:
"/root/foo"
"/root/bar"
"/root/i have multiple words"

Here is the (relevant)code I currently have

Code:

if [ ! -z "$BACKUP_EXCLUDE_LIST" ]
then
  TEMPIFS=$IFS
  IFS=:
  for dir in $BACKUP_EXCLUDE_LIST
  do
    if [ -e "$3/$dir" ] # $3 is the backup source
    then
        BACKUP_EXCLUDE_PARAMS="$BACKUP_EXCLUDE_PARAMS --exclude='$dir'"
    fi    
  done
  IFS=$TEMPIFS
fi


tar $BACKUP_EXCLUDE_PARAMS -cpzf  $BACKUP_PATH/$BACKUP_BASENAME.tar.gz -C $BACKUP_SOURCE_DIR $BACKUP_SOURCE_TARGET

This is what happens when I run my script with sh -x

Code:

+ IFS=:
+ [ -e /root/foo ]
+ BACKUP_EXCLUDE_PARAMS= --exclude='foo'
+ [ -e /root/bar ]
+ BACKUP_EXCLUDE_PARAMS= --exclude='foo' --exclude='bar'
+ [ -e /root/i have multiple words ]
+ BACKUP_EXCLUDE_PARAMS= --exclude='foo' --exclude='bar' --exclude='i have multiple words'
+ IFS= 	

# So far so good

+ tar --exclude='foo' --exclude='bar' --exclude='i have multiple words' -cpzf /backup/root/daily/root_20130131.071056.tar.gz -C / root
tar: have: Cannot stat: No such file or directory
tar: multiple: Cannot stat: No such file or directory
tar: words': Cannot stat: No such file or directory
tar: Exiting with failure status due to previous errors

# WHY? :(

The Check completes sucessfully, but the `--exclude='i have multiple words'` does not work.

Mind you that it DOES work when i type it in my shell, manually:

Code:

tar --exclude='i have multiple words' -cf /somefile.tar.gz /root

I know that this would work in bash when using arrays, but i want this to be POSIX.

Is there a solution to this in dash?

David the H. · 02-01-2013, 07:20 AM

To start with, "$BACKUP_EXCLUDE_LIST" contains what? I assume it's a colon-separated list of directory names?

Trying to store dynamic option lists in variables is quite difficult without arrays. You have to rely on shell word-splitting, but just as important, nearly all characters, such as quote-marks, will always be treated literally and have no special meaning to the shell later on, because they are processed before variable expansion. Only IFS word-splitting and globbing expansion happen after variable expansion.

I'm trying to put a command in a variable, but the complex cases always fail!
http://mywiki.wooledge.org/BashFAQ/050

I believe then that your --exclude='$dir'" is likely the main culprit. Remove the internal single quotes and see what happens. (Edit: Still won't work with multi-word inputs. See my next post.)

PS: I would also like to add that slightly cleaner formatting would help readability. Reduce your variable names to lowercase, put the then/do keywords on the same line as the test, and use more vertical whitespace.

And always be sure to quote all of your variables, unless they need to be word-split.

David the H. · 02-01-2013, 07:27 AM

I've stripped your script down to the essentials, and I think we need to do something like this. Since the directory names can be multi-word, we can't rely on default word-splitting. We need to keep a non-standard separator between the options:

Code:

#!/bin/dash

dirs="foo:bar:baz:foo bar baz"
IFS=:

for dir in $dirs; do

    BACKUP_EXCLUDE_PARAMS="$BACKUP_EXCLUDE_PARAMS:--exclude=$dir"

done

printf '%s\n' $BACKUP_EXCLUDE_PARAMS

I kept IFS set at colon, and used that to delimit the parameter list. Your only worry then would be if any directory name could include a colon in its name (in which case, just use a different delimiter).

Edit: One more idea for you, but perhaps not quite as useful. If you can spare the positional parameters, you can use those as a kind of array:

Code:

#!/bin/dash

param1=$1	#save the existing parameters to
param2=$2	#new variables for later use
param3=$3
oldifs=$IFS

dirs="foo:bar:baz:foo bar baz"

IFS=:
set -- $( printf -- '--exclude=%s:' $dirs )
IFS=$oldifs

printf '%s\n' "$@"

It does have the advantage of keeping the use of IFS word-splitting to an absolute minimum, however.

Kaurin · 02-01-2013, 08:59 AM

Thank you for steering me in the right direction! That wiki is a great resource, and has saved me a bunch of times in the past.

I have read up on parameter expansion, and now see my error. Most notably: I should never write complex scripts in dash...

To answer your question:

Quote:

To start with, "$BACKUP_EXCLUDE_LIST" contains what? I assume it's a colon-separated list of directory names?

Yes. I have a config file which gets included in the script. This is an example of it:

Code:

do_backup $BACKUP_DIR/root root /root "foo:bar:i have multiple words"

Your first solution does not work (I've tried not using quotes on '$dir' before).

This is what I get when I run it without quotes:

Code:

+ tar --exclude=foo --exclude=bar --exclude=i have multiple words -cpzf /backup/root/daily/root_20130201.093734.tar.gz -C / root
tar: have: Cannot stat: No such file or directory
tar: multiple: Cannot stat: No such file or directory
tar: words: Cannot stat: No such file or directory
tar: Exiting with failure status due to previous errors

This is the reason I included the quotes in the first place, but then parameter expansion is in the way.

I can't use positional parameters, because of the fore-mentioned include file... Or maybe I can but don't see how (without the config file looking fugly)

I have also tried doing this:

Code:

$BACKUP_EXCLUDE_LIST="foo,bar,'i have multiple words'"
tar --exclude={,$BACKUP_EXCLUDE_LIST} -cpzf $BACKUP_PATH/$BACKUP_BASENAME.tar.gz -C $BACKUP_SOURCE_DIR $BACKUP_SOURCE_TARGET

And still no luck.

What would you reccomend at this point?
Should I continue trying to find a solution for this script to work in dash, or should I just rewrite the script (optimizing along the way) for bash?

David the H. · 02-01-2013, 10:33 AM

brace expansion isn't posix, so dash doesn't understand it. But even in bash, braces expand before variables do, and so you can't use them to set brace values.

See here for a rundown of things you can't use in dash.
http://mywiki.wooledge.org/Bashism

I think the real secret is to try to think like the shell. All the command processing has one final purpose, to generate a list of tokens, individual, indivisible strings that act as arguments for the final command. Quoting is just one of the steps involved in this, so don't think that it's all you have to do.

In this case you have three steps to accomplish. First you have to break the original string up into individual directory names, second, you have to prepend a string to each one, and third, you have to insert them into the final command string, and do all of this in a way that keeps them as individual tokens.

Again this would be trivial with bash's arrays and advanced parameter substitution. But it is sometimes good to flex your posix muscles too.

The first bit of code I gave in post #3 does all three steps. It uses the same IFS setting to split both the original string and the modified one. All you need to do is flesh it out. I did what I could with the above, and it appears to work in my tests.

printf is a very good way to see what the individual tokens are, so I inserted one in front of the tar command that will print each token out on a separate line, bracketed.

Code:

#!/bin/dash

dirs="foo:bar:i have multiple words"	#you would use $4 here
BACKUP_PATH=/root
BACKUP_BASENAME=root
BACKUP_SOURCE_DIR=sourcedir
BACKUP_SOURCE_TARGET=target

TEMPIFS=$IFS

# set IFS to colon for the next two actions
# whitespace in parameters will be ignored
# also temporarily disable glob expansion
IFS=:
set -f

# loop through the $dirs variable, splitting it by IFS
# for each valid entry, add it to the PARAMS variable,
# again separated by a colon

for dir in $dirs; do

	if [ -e "$dir" ]; then
        BACKUP_EXCLUDE_PARAMS="$BACKUP_EXCLUDE_PARAMS:--exclude=$dir"
	fi

done

# run the tar command with all the necessary arguments.
# don't quote the PARAMS argument; it needs to be word-split.
# (printf used for testing)

printf '[%s]\n' \
  tar    $BACKUP_EXCLUDE_PARAMS \
  -cpzf "$BACKUP_PATH/$BACKUP_BASENAME.tar.gz" \
  -C    "$BACKUP_SOURCE_DIR" \
        "$BACKUP_SOURCE_TARGET"

# reset IFS and globbing to default
IFS=$TEMPIFS
set +f

(I had to remove the $3 for testing, and I split the final command into multiple lines for readability.)

With appropriately-name files in $PWD, this is what it prints out:

Code:

$ ./script.sh
[tar]
[]
[--exclude=foo]
[--exclude=bar]
[--exclude=i have multiple words]
[-cpzf]
[/root/root.tar.gz]
[-C]
[sourcedir]
[target]

It produces a superfluous empty argument, since the PARAMS variable is empty on the first run of the loop, but that shouldn't affect the actual execution. It can be eliminated with 'tar ${BACKUP_EXCLUDE_PARAMS#:} ....' if you want.

Just remove the printf when you're satisfied that it's working.

Kaurin · 02-01-2013, 01:08 PM

Holy mother of shells

Thank you so much for taking the time to better explain the inner workings of shell logic, as well as supplying the links for further education!

In my last post, I must have done something wrong. I have restored my script version from the first post, and carefully worked from there, following your advice/instructions.

The script works now!

I just have one thing to add

Using TEMPIFS borked up my script later on. I switched to just setting the IFS, and then unsetting it instead of using temp variables. The rest of the script works fine now.

Also, I have taken my time CAREFULLY going through the entire script and used quotes in the places they should be used.... I cringed a lot because I had a lot of variables holding path names that were not quoted!

Marking this thread as solved!

David the H. · 02-01-2013, 01:35 PM

Good for you! You're well on your way to being an expert scripter!

As a final comment, using IFS for anything should be kept to a bare minimum, and even more rarely as a global setting. It's very easy to get confused about what the current setting is, and its non-transparent behavior can make it hard to diagnose where the problem lies. Whenever possible set up such uses to run in subshells, or in functions with a local IFS setting, or something like that, so that it remains limited in scope.

You should only rarely really need it anyway, especially in bash or other modern shells with advanced string manipulation features built in. It's only in limited posix shells like this where you still have to depend on it with any frequency.

(One exception is its use in conjunction with the read built-in, where you will use it often, but that's ok, as it remains restricted to that command. See section 7 of the above link.)

Unsetting it like you did is just fine, btw, as the shell still acts as if the defaults were in place.

Kaurin · 02-01-2013, 01:44 PM

Yes, I probably will do that

Btw, I forgot to paste the code:

Code:

BACKUP_EXCLUDE_LIST="foo:bar:i have multiple words"

# Grouping our parameters
if [ ! -z "$BACKUP_EXCLUDE_LIST" ]
then
  IFS=:         # Here we set our temp $IFS
  set -f        # Disable globbing
  for dir in $BACKUP_EXCLUDE_LIST
  do
    if [ -e "$3/$dir" ]  # $3 is the directory that contains the directories defined in $BACKUP_EXCLUDE_LIST
    then
      BACKUP_EXCLUDE_PARAMS="$BACKUP_EXCLUDE_PARAMS:--exclude=$dir"
    fi    
  done
fi

# We are ready to tar

tar $BACKUP_EXCLUDE_PARAMS \
  -cpzf  "$BACKUP_PATH/$BACKUP_BASENAME.tar.gz" \
  -C "$BACKUP_SOURCE_DIR" \
  "$BACKUP_SOURCE_TARGET"
unset IFS       # our custom IFS has done it's job. Let's unset it!
set +f          # Globbing is back on

Following your last advice, the last piece of code should look like this:

Code:

BACKUP_EXCLUDE_LIST="foo:bar:i have multiple words"

$(
# Grouping our parameters
if [ ! -z "$BACKUP_EXCLUDE_LIST" ]
then
  IFS=:         # Here we set our temp $IFS
  set -f        # Disable globbing
  for dir in $BACKUP_EXCLUDE_LIST
  do
    if [ -e "$3/$dir" ]  # $3 is the directory that contains the directories defined in $BACKUP_EXCLUDE_LIST
    then
      BACKUP_EXCLUDE_PARAMS="$BACKUP_EXCLUDE_PARAMS:--exclude=$dir"
    fi    
  done
fi

# We are ready to tar

tar $BACKUP_EXCLUDE_PARAMS \
  -cpzf  "$BACKUP_PATH/$BACKUP_BASENAME.tar.gz" \
  -C "$BACKUP_SOURCE_DIR" \
  "$BACKUP_SOURCE_TARGET"
unset IFS       # our custom IFS has done it's job. Let's unset it!
set +f          # Globbing is back on
)

Sorry for the lack of indentation there, I'm typing on my laptop now, and I hate this keyboard

If I understand correctly, the unset in this last example would not even be necessary, because the custom IFS, and the set commands are only valid for that subshell?

David the H. · 02-01-2013, 02:18 PM

Almost.

The $(..) brackets are a substitution pattern, expanding just like a variable. In bash you can use (..) instead, which evaluates but doesn't expand. POSIX doesn't have that, but you can prefix the brackets with the ":" (true) command to neutralize the expansion.

In this case it may not matter much, since the commands above produce no direct output. But if it did produce output, it would be parsed as if it were a command and most likely cause an error. e.g.:

Code:

$ $( echo foobar )       #the brackets expand into the replacement text, and the shell attempts to parse it
bash: foobar: command not found

$ : $( echo foobar )     #expands, but true nullifies it.
< no output >

$ ( echo foobar )        #the command operates normally inside the subshell
foobar

Also, there's no need to unset any environment settings when you use a subshell. All changes are lost anyway when it closes.