Bash script passing file names with spaces into another program
 

I'm trying to write a bash script where I read a file, get a load of file names and properties and pass those into another program.

If I echo the tar command, and copy and paste it, it works fine and looks something like this:

tar czf archive.tar.gz "My File With Spaces1.bin" "My File With Spaces2.bin" "My File With Spaces3.bin"

If I just run it, it craps out as though it is trying to run:

tar czf archive.tar.gz \"My File With Spaces1.bin\" \"My File With Spaces2.bin\" \"My File With Spaces3.bin\"

I have tried a whole manner of different ways to get this working properly, such as removing the escapes when building the file list, escaping the escape, escaping the escapes and the quotes. I think I am missing something really obvious but I just can't put my finger on it.

Any ideas?

The problem is that the escaped quotes are interpreted correctly by echo, whereas the tar command take them literally. You should force the shell to parse the command line again after the parameter substitution, that is when the $FILE_STRING parameter is evaluated you end up with:
as you already noticed. Using eval in front of the command line, forces the shell to re-interpret the line itself and at that point the escaped quotes are true double quotes:
that brings to:
However, there are some caveats about the usage of eval. In some situations it might be dangerous as explained here. To avoid problems I suggest an alternative: since the file names are already saved into a file, why not use the --files-from (or -T) option of tar? Example:
this reads the file names from the standard input (coming from the cut command that extracts the first colon separated field) and it doesn't complain about blank spaces. Hope this helps.

Excellent, that appears to have solved it, thank you. Been using Linux for more than 10 years and this is the first bash script I have written. Who'd have thought?

It's never too late! :)

If I understand your code correctly, here's what you need to do:
If you want to preserve positional arguments (eg. $@) you can wrap that into a function (or use bash arrays). Point is that you should avoid building command line since there are tons of situations where your code may fail (from the top of my head, if file name contains quote character, the whole thing won't work properly).

Uggh. I really, really, really recommend NOT using eval for anything like this. If you ever find yourself turning to eval, then stop right there and re-evaluate your entire code flow first. 95% of the time you're just doing something wrong.

Nor is it generally necessary to embed literal quote marks in variables unless you have to pass actual command syntax to another program. And you'll never need to do so just to protect spaces or other characters from later shell parsing.

To tell the truth, I haven't used tar much, but I don't think its syntax requires literal quote marks. I believe it just needs a list of files to archive, correct?

So the big problem here is that you are trying to improperly store a list of file names in a single scalar variable, and including embedded quotemarks in a vain and fruitless attempt to protect the spaces from later word-splitting.


mina86 has already shown how to do to it properly, although I think you're better advised to just use an array instead of the positional parameters, which is a trick that's only really necessary for shells that don't have array support.

Here's some required reading for you concerning this subject.

Begin with these three links. It's vital in scripting to understand how the shell handles arguments and whitespace:
http://mywiki.wooledge.org/Arguments
http://mywiki.wooledge.org/WordSplitting
http://mywiki.wooledge.org/Quotes

Remember, the entire purpose of the shell's parsing system is to reformat what you give it into a set of words, or more specifically tokens, that will be passed as arguments to the command to be executed. Don't think about quotes as "space protectors", but as just one (important) step in this tokenizing process.


How can I use array variables?
http://mywiki.wooledge.org/BashFAQ/005

How can I find and deal with file names containing newlines, spaces or both?
http://mywiki.wooledge.org/BashFAQ/020

Eval command and security issues
http://mywiki.wooledge.org/BashFAQ/048

I'm trying to put a command in a variable, but the complex cases always fail!
http://mywiki.wooledge.org/BashFAQ/050

The last one in particular is very germane to this situation.


PS: Since environment variables are generally all upper-case, it's recommended practice to keep your own user variables in lower-case or mixed-case to help differentiate them. See Scripting With Style.

Thanks for the extra input. Sorry for my late response, but I thought it was closed and done with.

FYI - The array didn't work right. I still had the same issue as before. The method described by mina is working fine though I think.

I don't think there are any double quotes in the filenames, but I suppose I need to make doubly sure. I'll take a look at those links in the meantime though.

I was wondering whether Python might be better to implement this. Traditionally, I have only every worked with C, C++ and Java, with a bit of PHP. I was thinking I should really attempt to make full use of the tools that have been in front of me for the past few years.

Also, I should add that the commands above are not being passed into tar. They are being passed into another command currently in development by someone else, but the principal is the same.

My rule of thumb when writing shell scripts is that once I feel the need to use arrays or anything more complicated I switch to Perl or Python, so yes, Python is not out of a question in my opinion. Than again, I dunno if your script not much more than what has been revealed in this thread, than you can just stick to my code (also, I don't use arrays in bash at all, but David's code should work just as well).

It might help to give a bit more detail as to what went wrong when you tried it. Did you use the code exactly as posted? Were you careful to properly quote all expansions as demonstrated in the links I gave? Did you get any errors or other output that could help the diagnosis?

Another important point, are you using #!/bin/bash at the top of your script? Because if you're using #!/bin/sh, then you don't actually have a bash script, but a POSIX shell script (with bash-specific syntax improperly included), and your system is trying to execute it with whatever shell is configured as it's POSIX standard shell. This may not be bash, but dash or some other shell that doesn't have array support built in. Always use the explicit path of the interpreter you want to use in your shebang.


@mina86: Arrays certainly simplify a lot of things like this that are otherwise very difficult to do in the shell. Trying to safely code something like the above in a POSIX shell can be extremely frustrating, especially if the positional parameters are unavailable, but it becomes almost trivial in bash or another array-supporting shell.

I highly recommend you take the time to familiarize yourself with at least the basics of shell arrays. It should be trivial to pick up if you're already familiar with arrays in other languages, and you might gain a whole new appreciation for what bash can do for you. :)

I do not consider shell to be a serious programming language, and bash extensions do not change that, so once my code becomes non-trivial I simply switch to more powerful language. And the same goes for Perl and Python – even though they are way more “serious” than shell, I still would not choose them for very complicated projects for witch I prefer languages w/o dynamic typing (for one).

You mean your input-file is sg like this:
If so, you can try this: