LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices


Reply
  Search this Thread
Old 03-25-2013, 10:07 AM   #1
yilez
Member
 
Registered: Apr 2004
Distribution: Slackware
Posts: 127

Rep: Reputation: Disabled
Bash script passing file names with spaces into another program


I'm trying to write a bash script where I read a file, get a load of file names and properties and pass those into another program.

Code:
FILE_NAME=$1
FILE_STRING=''

while IFS=$':' read -r -a FILE_ATTRS
do
    FILE_STRING="$FILE_STRING \"${FILE_ATTRS[0]}\""
done < "$FILE_NAME"

tar czf $FILE_NAME.tar.gz $FILE_STRING
If I echo the tar command, and copy and paste it, it works fine and looks something like this:

tar czf archive.tar.gz "My File With Spaces1.bin" "My File With Spaces2.bin" "My File With Spaces3.bin"

If I just run it, it craps out as though it is trying to run:

tar czf archive.tar.gz \"My File With Spaces1.bin\" \"My File With Spaces2.bin\" \"My File With Spaces3.bin\"

I have tried a whole manner of different ways to get this working properly, such as removing the escapes when building the file list, escaping the escape, escaping the escapes and the quotes. I think I am missing something really obvious but I just can't put my finger on it.

Any ideas?
 
Old 03-25-2013, 11:15 AM   #2
colucix
LQ Guru
 
Registered: Sep 2003
Location: Bologna
Distribution: CentOS 6.5 OpenSuSE 12.3
Posts: 10,509

Rep: Reputation: 1978Reputation: 1978Reputation: 1978Reputation: 1978Reputation: 1978Reputation: 1978Reputation: 1978Reputation: 1978Reputation: 1978Reputation: 1978Reputation: 1978
The problem is that the escaped quotes are interpreted correctly by echo, whereas the tar command take them literally. You should force the shell to parse the command line again after the parameter substitution, that is when the $FILE_STRING parameter is evaluated you end up with:
Code:
tar czf archive.tar.gz \"My File With Spaces1.bin\" \"My File With Spaces2.bin\" \"My File With Spaces3.bin\"
as you already noticed. Using eval in front of the command line, forces the shell to re-interpret the line itself and at that point the escaped quotes are true double quotes:
Code:
eval tar czf $FILE_NAME.tar.gz $FILE_STRING
that brings to:
Code:
tar czf archive.tar.gz "My File With Spaces1.bin" "My File With Spaces2.bin" "My File With Spaces3.bin"
However, there are some caveats about the usage of eval. In some situations it might be dangerous as explained here. To avoid problems I suggest an alternative: since the file names are already saved into a file, why not use the --files-from (or -T) option of tar? Example:
Code:
cut -d: -f1 $FILE_NAME | tar zcf $FILE_NAME.tar.gz -T-
this reads the file names from the standard input (coming from the cut command that extracts the first colon separated field) and it doesn't complain about blank spaces. Hope this helps.

Last edited by colucix; 03-25-2013 at 11:17 AM.
 
1 members found this post helpful.
Old 03-25-2013, 11:56 AM   #3
yilez
Member
 
Registered: Apr 2004
Distribution: Slackware
Posts: 127

Original Poster
Rep: Reputation: Disabled
Excellent, that appears to have solved it, thank you. Been using Linux for more than 10 years and this is the first bash script I have written. Who'd have thought?
 
Old 03-25-2013, 12:03 PM   #4
colucix
LQ Guru
 
Registered: Sep 2003
Location: Bologna
Distribution: CentOS 6.5 OpenSuSE 12.3
Posts: 10,509

Rep: Reputation: 1978Reputation: 1978Reputation: 1978Reputation: 1978Reputation: 1978Reputation: 1978Reputation: 1978Reputation: 1978Reputation: 1978Reputation: 1978Reputation: 1978
It's never too late!
 
Old 03-25-2013, 01:59 PM   #5
mina86
Member
 
Registered: Aug 2008
Distribution: Slackware
Posts: 493

Rep: Reputation: 210Reputation: 210Reputation: 210
If I understand your code correctly, here's what you need to do:
Code:
FILE_NAME=$1
FILE_STRING=''

set --
while IFS=: read -r name rest; do
    set -- "$@" "$name"
done <$FILE_NAME

tar czf "$FILE_NAME.tar.gz" "$@"
If you want to preserve positional arguments (eg. $@) you can wrap that into a function (or use bash arrays). Point is that you should avoid building command line since there are tons of situations where your code may fail (from the top of my head, if file name contains quote character, the whole thing won't work properly).

Last edited by mina86; 03-25-2013 at 02:02 PM.
 
1 members found this post helpful.
Old 03-26-2013, 04:40 PM   #6
David the H.
Bash Guru
 
Registered: Jun 2004
Location: Osaka, Japan
Distribution: Debian + kde 4 / 5
Posts: 6,837

Rep: Reputation: 1984Reputation: 1984Reputation: 1984Reputation: 1984Reputation: 1984Reputation: 1984Reputation: 1984Reputation: 1984Reputation: 1984Reputation: 1984Reputation: 1984
Uggh. I really, really, really recommend NOT using eval for anything like this. If you ever find yourself turning to eval, then stop right there and re-evaluate your entire code flow first. 95% of the time you're just doing something wrong.

Nor is it generally necessary to embed literal quote marks in variables unless you have to pass actual command syntax to another program. And you'll never need to do so just to protect spaces or other characters from later shell parsing.

To tell the truth, I haven't used tar much, but I don't think its syntax requires literal quote marks. I believe it just needs a list of files to archive, correct?

So the big problem here is that you are trying to improperly store a list of file names in a single scalar variable, and including embedded quotemarks in a vain and fruitless attempt to protect the spaces from later word-splitting.


mina86 has already shown how to do to it properly, although I think you're better advised to just use an array instead of the positional parameters, which is a trick that's only really necessary for shells that don't have array support.

Code:
FILE_NAME=$1
FILE_STRING=''

while IFS=: read -r name rest; do
     names+=( "$name" )
done <"$FILE_NAME"

tar czf "$FILE_NAME.tar.gz" "${names[@]}"
Here's some required reading for you concerning this subject.

Begin with these three links. It's vital in scripting to understand how the shell handles arguments and whitespace:
http://mywiki.wooledge.org/Arguments
http://mywiki.wooledge.org/WordSplitting
http://mywiki.wooledge.org/Quotes

Remember, the entire purpose of the shell's parsing system is to reformat what you give it into a set of words, or more specifically tokens, that will be passed as arguments to the command to be executed. Don't think about quotes as "space protectors", but as just one (important) step in this tokenizing process.


How can I use array variables?
http://mywiki.wooledge.org/BashFAQ/005

How can I find and deal with file names containing newlines, spaces or both?
http://mywiki.wooledge.org/BashFAQ/020

Eval command and security issues
http://mywiki.wooledge.org/BashFAQ/048

I'm trying to put a command in a variable, but the complex cases always fail!
http://mywiki.wooledge.org/BashFAQ/050

The last one in particular is very germane to this situation.


PS: Since environment variables are generally all upper-case, it's recommended practice to keep your own user variables in lower-case or mixed-case to help differentiate them. See Scripting With Style.
 
Old 04-03-2013, 03:48 AM   #7
yilez
Member
 
Registered: Apr 2004
Distribution: Slackware
Posts: 127

Original Poster
Rep: Reputation: Disabled
Thanks for the extra input. Sorry for my late response, but I thought it was closed and done with.

FYI - The array didn't work right. I still had the same issue as before. The method described by mina is working fine though I think.

I don't think there are any double quotes in the filenames, but I suppose I need to make doubly sure. I'll take a look at those links in the meantime though.

I was wondering whether Python might be better to implement this. Traditionally, I have only every worked with C, C++ and Java, with a bit of PHP. I was thinking I should really attempt to make full use of the tools that have been in front of me for the past few years.

Also, I should add that the commands above are not being passed into tar. They are being passed into another command currently in development by someone else, but the principal is the same.
 
Old 04-03-2013, 07:38 AM   #8
mina86
Member
 
Registered: Aug 2008
Distribution: Slackware
Posts: 493

Rep: Reputation: 210Reputation: 210Reputation: 210
Quote:
Originally Posted by yilez View Post
I was wondering whether Python might be better to implement this.
My rule of thumb when writing shell scripts is that once I feel the need to use arrays or anything more complicated I switch to Perl or Python, so yes, Python is not out of a question in my opinion. Than again, I dunno if your script not much more than what has been revealed in this thread, than you can just stick to my code (also, I don't use arrays in bash at all, but David's code should work just as well).
 
Old 04-03-2013, 05:56 PM   #9
David the H.
Bash Guru
 
Registered: Jun 2004
Location: Osaka, Japan
Distribution: Debian + kde 4 / 5
Posts: 6,837

Rep: Reputation: 1984Reputation: 1984Reputation: 1984Reputation: 1984Reputation: 1984Reputation: 1984Reputation: 1984Reputation: 1984Reputation: 1984Reputation: 1984Reputation: 1984
Quote:
Originally Posted by yilez View Post
FYI - The array didn't work right. I still had the same issue as before. The method described by mina is working fine though I think.
It might help to give a bit more detail as to what went wrong when you tried it. Did you use the code exactly as posted? Were you careful to properly quote all expansions as demonstrated in the links I gave? Did you get any errors or other output that could help the diagnosis?

Another important point, are you using #!/bin/bash at the top of your script? Because if you're using #!/bin/sh, then you don't actually have a bash script, but a POSIX shell script (with bash-specific syntax improperly included), and your system is trying to execute it with whatever shell is configured as it's POSIX standard shell. This may not be bash, but dash or some other shell that doesn't have array support built in. Always use the explicit path of the interpreter you want to use in your shebang.


@mina86: Arrays certainly simplify a lot of things like this that are otherwise very difficult to do in the shell. Trying to safely code something like the above in a POSIX shell can be extremely frustrating, especially if the positional parameters are unavailable, but it becomes almost trivial in bash or another array-supporting shell.

I highly recommend you take the time to familiarize yourself with at least the basics of shell arrays. It should be trivial to pick up if you're already familiar with arrays in other languages, and you might gain a whole new appreciation for what bash can do for you.
 
Old 04-04-2013, 05:26 AM   #10
mina86
Member
 
Registered: Aug 2008
Distribution: Slackware
Posts: 493

Rep: Reputation: 210Reputation: 210Reputation: 210
Quote:
Originally Posted by David the H. View Post
I highly recommend you take the time to familiarize yourself with at least the basics of shell arrays. It should be trivial to pick up if you're already familiar with arrays in other languages, and you might gain a whole new appreciation for what bash can do for you.
I do not consider shell to be a serious programming language, and bash extensions do not change that, so once my code becomes non-trivial I simply switch to more powerful language. And the same goes for Perl and Python – even though they are way more “serious” than shell, I still would not choose them for very complicated projects for witch I prefer languages w/o dynamic typing (for one).
 
Old 04-04-2013, 06:45 AM   #11
NevemTeve
Senior Member
 
Registered: Oct 2011
Location: Budapest
Distribution: Debian/GNU/Linux, AIX
Posts: 3,607

Rep: Reputation: 1105Reputation: 1105Reputation: 1105Reputation: 1105Reputation: 1105Reputation: 1105Reputation: 1105Reputation: 1105Reputation: 1105
You mean your input-file is sg like this:
Code:
some file:some other file:lastfile
If so, you can try this:
Code:
#!/bin/sh

FILE_NAME="$1"

tr ':' '\n' <"$FILE_NAME" >"$FILE_NAME".tmp

tar -T "$FILE_NAME".tmp -czvf "$FILE_NAME".tar.gz
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] Passing variables to a program in a bash script danbopes Programming 6 12-30-2010 08:50 AM
[SOLVED] Bash file names with spaces alex1986 Programming 5 07-26-2010 10:47 PM
bash: Passing arrays with spaces jakeo25 Programming 8 06-05-2009 09:00 AM
bash - replace all spaces in file, folder names babag Programming 24 04-20-2008 01:17 AM
Bash crashes ? File names with () and spaces Danodare Slackware 1 02-27-2004 03:50 PM

LinuxQuestions.org > Forums > Non-*NIX Forums > Programming

All times are GMT -5. The time now is 12:27 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration