Dash shell script variable expansion problems ;/
This work is being done on a test virtualbox machine
In my /root dir, i have created the following: "/root/foo" "/root/bar" "/root/i have multiple words" Here is the (relevant)code I currently have Code:
if [ ! -z "$BACKUP_EXCLUDE_LIST" ] Code:
+ IFS=: Mind you that it DOES work when i type it in my shell, manually: Code:
tar --exclude='i have multiple words' -cf /somefile.tar.gz /root Is there a solution to this in dash? |
To start with, "$BACKUP_EXCLUDE_LIST" contains what? I assume it's a colon-separated list of directory names?
Trying to store dynamic option lists in variables is quite difficult without arrays. You have to rely on shell word-splitting, but just as important, nearly all characters, such as quote-marks, will always be treated literally and have no special meaning to the shell later on, because they are processed before variable expansion. Only IFS word-splitting and globbing expansion happen after variable expansion. I'm trying to put a command in a variable, but the complex cases always fail! http://mywiki.wooledge.org/BashFAQ/050 I believe then that your --exclude='$dir'" is likely the main culprit. Remove the internal single quotes and see what happens. (Edit: Still won't work with multi-word inputs. See my next post.) PS: I would also like to add that slightly cleaner formatting would help readability. Reduce your variable names to lowercase, put the then/do keywords on the same line as the test, and use more vertical whitespace. And always be sure to quote all of your variables, unless they need to be word-split. |
I've stripped your script down to the essentials, and I think we need to do something like this. Since the directory names can be multi-word, we can't rely on default word-splitting. We need to keep a non-standard separator between the options:
Code:
#!/bin/dash Edit: One more idea for you, but perhaps not quite as useful. If you can spare the positional parameters, you can use those as a kind of array: Code:
#!/bin/dash |
Thank you for steering me in the right direction! That wiki is a great resource, and has saved me a bunch of times in the past.
I have read up on parameter expansion, and now see my error. Most notably: I should never write complex scripts in dash... To answer your question: Quote:
Code:
do_backup $BACKUP_DIR/root root /root "foo:bar:i have multiple words" This is what I get when I run it without quotes: Code:
+ tar --exclude=foo --exclude=bar --exclude=i have multiple words -cpzf /backup/root/daily/root_20130201.093734.tar.gz -C / root I can't use positional parameters, because of the fore-mentioned include file... Or maybe I can but don't see how (without the config file looking fugly) I have also tried doing this: Code:
$BACKUP_EXCLUDE_LIST="foo,bar,'i have multiple words'" And still no luck. What would you reccomend at this point? Should I continue trying to find a solution for this script to work in dash, or should I just rewrite the script (optimizing along the way) for bash? |
brace expansion isn't posix, so dash doesn't understand it. But even in bash, braces expand before variables do, and so you can't use them to set brace values.
See here for a rundown of things you can't use in dash. http://mywiki.wooledge.org/Bashism I think the real secret is to try to think like the shell. All the command processing has one final purpose, to generate a list of tokens, individual, indivisible strings that act as arguments for the final command. Quoting is just one of the steps involved in this, so don't think that it's all you have to do. In this case you have three steps to accomplish. First you have to break the original string up into individual directory names, second, you have to prepend a string to each one, and third, you have to insert them into the final command string, and do all of this in a way that keeps them as individual tokens. Again this would be trivial with bash's arrays and advanced parameter substitution. But it is sometimes good to flex your posix muscles too. The first bit of code I gave in post #3 does all three steps. It uses the same IFS setting to split both the original string and the modified one. All you need to do is flesh it out. I did what I could with the above, and it appears to work in my tests. printf is a very good way to see what the individual tokens are, so I inserted one in front of the tar command that will print each token out on a separate line, bracketed. Code:
#!/bin/dash With appropriately-name files in $PWD, this is what it prints out: Code:
$ ./script.sh Just remove the printf when you're satisfied that it's working. |
Holy mother of shells
Thank you so much for taking the time to better explain the inner workings of shell logic, as well as supplying the links for further education! In my last post, I must have done something wrong. I have restored my script version from the first post, and carefully worked from there, following your advice/instructions. The script works now! I just have one thing to add Using TEMPIFS borked up my script later on. I switched to just setting the IFS, and then unsetting it instead of using temp variables. The rest of the script works fine now. Also, I have taken my time CAREFULLY going through the entire script and used quotes in the places they should be used.... I cringed a lot because I had a lot of variables holding path names that were not quoted! Marking this thread as solved! |
Good for you! You're well on your way to being an expert scripter! :)
As a final comment, using IFS for anything should be kept to a bare minimum, and even more rarely as a global setting. It's very easy to get confused about what the current setting is, and its non-transparent behavior can make it hard to diagnose where the problem lies. Whenever possible set up such uses to run in subshells, or in functions with a local IFS setting, or something like that, so that it remains limited in scope. You should only rarely really need it anyway, especially in bash or other modern shells with advanced string manipulation features built in. It's only in limited posix shells like this where you still have to depend on it with any frequency. (One exception is its use in conjunction with the read built-in, where you will use it often, but that's ok, as it remains restricted to that command. See section 7 of the above link.) Unsetting it like you did is just fine, btw, as the shell still acts as if the defaults were in place. |
Yes, I probably will do that
Btw, I forgot to paste the code: Code:
BACKUP_EXCLUDE_LIST="foo:bar:i have multiple words" Code:
BACKUP_EXCLUDE_LIST="foo:bar:i have multiple words" If I understand correctly, the unset in this last example would not even be necessary, because the custom IFS, and the set commands are only valid for that subshell? |
Almost.
The $(..) brackets are a substitution pattern, expanding just like a variable. In bash you can use (..) instead, which evaluates but doesn't expand. POSIX doesn't have that, but you can prefix the brackets with the ":" (true) command to neutralize the expansion. In this case it may not matter much, since the commands above produce no direct output. But if it did produce output, it would be parsed as if it were a command and most likely cause an error. e.g.: Code:
$ $( echo foobar ) #the brackets expand into the replacement text, and the shell attempts to parse it |
All times are GMT -5. The time now is 08:31 PM. |