LinuxQuestions.org
Review your favorite Linux distribution.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices

Reply
 
Search this Thread
Old 02-15-2010, 04:20 PM   #1
jkv
LQ Newbie
 
Registered: Oct 2009
Posts: 4

Rep: Reputation: 0
[Bash] How to expand path variable that contains spaces and wildcards


Hi all

First post here, I'm quite new with bash and now having bit of a trouble with path expansion of strings that contain some whitespace and wildcards

First my script sources a configuration file that contains array assignments

Code:
...
BACKUP_TARGET_FILES[2]=/boot/config-*                        #  no problems
BACKUP_TARGET_FILES[3]="/root/random dir with space/file*"   #  this is the problem
...
then later in the script I want to expand BACKUP_TARGET_FILES elements as below

Code:
IFS_DEFAULT="$IFS"
shopt -s nullglob
shopt -s dotglob
IFS=

for pattern in "${BACKUP_TARGET_FILES[@]}" ; do
    for file in $pattern ; do
        [ -f "$file" ] && BACKUP_TARGETS+=( "$file" )
    done
done

IFS="$IFS_DEFAULT"
shopt -u nullglob
shopt -u dotglob
this code seems to work but I'm not quite satisfied with it. I'd like to get rid those IFS changes, but haven't found out a solution as of yet.

Problem with default IFS seems to be that with it neither $pattern or "$pattern" work; it either interprets pattern as multiple words (because of spaces) and so expands to wrong paths or it ignores * because it's within quotes.

Any suggestions as to how I could do this better ? Thanks!
 
Old 02-15-2010, 05:16 PM   #2
allanf
Member
 
Registered: Sep 2008
Location: MN
Distribution: Gentoo, Fedora, Suse, Slackware, Debian, CentOS
Posts: 97
Blog Entries: 1

Rep: Reputation: 19
When you use the
Code:
     "${BACKUP_TARGET_FILES[@]}"
You expanded all the array values into a new single string by enclosing the reference within the "quotes".

Can you use
Code:
BACKUP_TARGET_FILES=(
                       ...
                       /boot/config-*
                       /root/random\ dir\ with\ space/file*
                       ...
                    )
Rather than the separate index usage assignments.

Code:
    for value in ${BACKUP_TARGET_FILES[@]}; do
       echo "value is:" ${value}
     done
or

Code:
  
     for sindex in ${!BACKUP_TARGET_FILES[@]}; do
        echo "value is:" ${BACKUP_TARGET_FILESs[${index}]}
     done

Last edited by allanf; 02-15-2010 at 05:18 PM. Reason: The use of "\" is a single character quote. So it can be used to quote the spaces.
 
Old 02-15-2010, 06:29 PM   #3
jkv
LQ Newbie
 
Registered: Oct 2009
Posts: 4

Original Poster
Rep: Reputation: 0
Quote:
When you use the
Code:
"${BACKUP_TARGET_FILES[@]}"

You expanded all the array values into a new single string by enclosing the reference within the "quotes".
Hm.. that contradicts a bit what I have learned earlier. Isn't ${BACKUP_TARGET_FILES[@]} handled a bit different within double quotes?

from bash man pages under "Special Parameters"
Quote:
@
Expands to the positional parameters, starting from one. When the expansion occurs within double quotes, each parameter expands to a separate word. That is, "$@" is equivalent to "$1" "$2" ... If the double-quoted expansion occurs within a word, the expansion of the first parameter is joined with the beginning part of the original word, and the expansion of the last parameter is joined with the last part of the original word. When there are no positional parameters, "$@" and $@ expand to nothing (i.e., they are removed).
and under "Arrays"
Quote:
Any element of an array may be referenced using ${name[subscript]}. The braces are required to avoid conflicts with pathname expansion. If subscript is @ or *, the word expands to all members of name. These subscripts differ only when the word appears within double quotes. If the word is double-quoted, ${name[*]} expands to a single word with the value of each array member separated by the first character of the IFS special variable, and ${name[@]} expands each element of name to a separate word. When there are no array members, ${name[@]} expands to nothing. If the double-quoted expansion occurs within a word, the expansion of the first parameter is joined with the beginning part of the original word, and the expansion of the last parameter is joined with the last part of the original word. This is analogous to the expansion of the special parameters * and @ (see Special Parameters above).
Am I reading it incorrectly or is there some more special cases mentioned somewhere? It's a long document

Quote:
Can you use
Code:

BACKUP_TARGET_FILES=(
...
/boot/config-*
/root/random\ dir\ with\ space/file*
...
)
Problem with this is that it does pathname expansion (*) too early. I source it at time when not all necessary mounts may not be done (/boot for example)... that's done a bit later in the script.
Code:
...
source /etc/gentoo-backup.conf
...
# -
# Do mounts (entries must exist in fstab)
# -
for dir in "${BACKUP_MOUNTS[@]}" ; do
    if [ ! "$(grep "$dir" /etc/mtab)" ] ; then
        echo "Mounting $dir"
        mount "$dir" || exit 1
    fi
done
Also I already tried to use "\" earlier with some code changes in the script and thought it worked, though I could be wrong... backtracked soon after that and then tried something else.
But I'd prefer to let user do simple double quoting in config file and then do all the hard work in the script.
 
Old 02-15-2010, 07:32 PM   #4
allanf
Member
 
Registered: Sep 2008
Location: MN
Distribution: Gentoo, Fedora, Suse, Slackware, Debian, CentOS
Posts: 97
Blog Entries: 1

Rep: Reputation: 19
Yes you are correct on the "${arrayvar[@]}" which is like the "$@" for the arguments. I did not read the '@' correctly.

When having delayed expansions it is also hard. If the stuff is fixed at the time you could try:

Code:
my_array=(
            ...
            $(cd /path/desired/ && ls wild-*)
            ...
         )
This will be in the other path, but will not include the path.



The use of spaces is really discourage (even in Windows) by any one that writes scripts (perl, python, batch, etc) bit windows does not allow the use of a '"' (quote in a name as they have to quote the name when working with them). Unix and Linux only that two characters that can not be in a name but many characters that can problems to people writting scripts. The not allowed characters are '/' and the character having a binary value of 0 (nul). But it is unwise to use lots of characters such as ! $ , ; ' " ( ) [ ] \ ...

Just because a character can be used is not a GOOD reason to use them.
 
Old 02-15-2010, 11:05 PM   #5
catkin
LQ 5k Club
 
Registered: Dec 2008
Location: Tamil Nadu, India
Distribution: Servers: Debian Squeeze and Wheezy. Desktop: Slackware64 14.0. Netbook: Slackware 13.37
Posts: 8,551
Blog Entries: 28

Rep: Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176
Thumbs down

Here's prrof of concept code without IFS manipulation
Code:
#!/bin/bash
#shopt -s extglob

# For testing
(
    dir='/tmp/dir with space in name'
    mkdir "$dir"
    touch "$dir/foo"
    touch "$dir/bar"
    touch "$dir/file with space in name"
)

# Configuration
BACKUP_TARGET_FILE_PATTERNS[0]="/tmp/d*/b*"
BACKUP_TARGET_FILE_PATTERNS[1]="/tmp/d*/f*"

# Intialisation
i=-1

# Gopher it!
for pattern in "${BACKUP_TARGET_FILE_PATTERNS[@]}" ; do
    echo "DEBUG: \$pattern is '$pattern'"
    for file in $pattern ; do
        echo "DEBUG: \$file is '$file'"
        if [[ -f "$file" ]]; then
            echo "DEBUG: file exists, adding to list of files to backup"
            let i=i+1
            BACKUP_TARGET_FILES[$i]="$file"
        fi
    done
done
Here's the output
Code:
DEBUG: $pattern is '/tmp/d*/b*'
DEBUG: $file is '/tmp/dir with space in name/bar'
DEBUG: file exists, adding to list of files to backup
DEBUG: $pattern is '/tmp/d*/f*'
DEBUG: $file is '/tmp/dir with space in name/file with space in name'
DEBUG: file exists, adding to list of files to backup
DEBUG: $file is '/tmp/dir with space in name/foo'
DEBUG: file exists, adding to list of files to backup
 
Old 02-16-2010, 06:32 AM   #6
jkv
LQ Newbie
 
Registered: Oct 2009
Posts: 4

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by catkin View Post
Here's prrof of concept code without IFS manipulation
Code:
#!/bin/bash
#shopt -s extglob

# Configuration
BACKUP_TARGET_FILE_PATTERNS[0]="/tmp/d*/b*"
BACKUP_TARGET_FILE_PATTERNS[1]="/tmp/d*/f*"
...
# Gopher it!
for pattern in "${BACKUP_TARGET_FILE_PATTERNS[@]}" ; do
    echo "DEBUG: \$pattern is '$pattern'"
    for file in $pattern ; do
        echo "DEBUG: \$file is '$file'"
        if [[ -f "$file" ]]; then
            echo "DEBUG: file exists, adding to list of files to backup"
            let i=i+1
            BACKUP_TARGET_FILES[$i]="$file"
        fi
    done
done
Any ideas how to get this code working in case there are both space and wildcard in patterns? This is exactly the problem I'm having: space(s) + wildcard(s) together. So fex. adding
Code:
BACKUP_TARGET_FILE_PATTERNS[2]="/tmp/dir with space*/f*"  # space+wildcard pattern
then extra output is

Code:
DEBUG: $pattern is '/tmp/dir with space*/f*'
DEBUG: $file is '/tmp/dir'
DEBUG: $file is 'with'
DEBUG: $file is 'space*/f*'
if $pattern is quoted instead

Code:
...
    echo "DEBUG: \$pattern is '$pattern'"
    for file in "$pattern" ; do
        echo "DEBUG: \$file is '$file'"
...
it stops working because no pathname expansions are done.

Maybe IFS manipulation is quite good solution afterall and not worth fixing?
 
Old 02-16-2010, 07:26 AM   #7
catkin
LQ 5k Club
 
Registered: Dec 2008
Location: Tamil Nadu, India
Distribution: Servers: Debian Squeeze and Wheezy. Desktop: Slackware64 14.0. Netbook: Slackware 13.37
Posts: 8,551
Blog Entries: 28

Rep: Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176
Quote:
Originally Posted by jkv View Post
Any ideas how to get this code working in case there are both space and wildcard in patterns?

[snip]

Maybe IFS manipulation is quite good solution afterall and not worth fixing?
Sorry I did not understand the problem

Having thought about the actual problem and researched some alternatives, I think your present solution is probably optimal. You could make it a little neater by not storing and restoring IFS; simply unsetting IFS is functionally equivalent to having it set at the default value.
Code:
unset IFS
.
 
Old 02-16-2010, 09:26 AM   #8
tuxdev
Senior Member
 
Registered: Jul 2005
Distribution: Slackware
Posts: 2,014

Rep: Reputation: 115Reputation: 115
Code:
patterns=("/usr/*/foo*", "/var/r*/bar*"j, "/mnt/qux*")
for pattern in "${patterns[@]}" ; do
   while IFS="" read -r -d "" file ; do
      echo "$file"
   done < <(find / -path "$pattern" -print0)
done
Quote:
Just because a character can be used is not a GOOD reason to use them.
This is no justification to allow scripts to break. Really, ever since I've started over-quoting, I haven't had a spaces-in-filename or shell metacharacters problem. I still do IFS=$'\n' on the top of my scripts, but that's more insurance than anything else.
 
Old 02-16-2010, 09:46 AM   #9
catkin
LQ 5k Club
 
Registered: Dec 2008
Location: Tamil Nadu, India
Distribution: Servers: Debian Squeeze and Wheezy. Desktop: Slackware64 14.0. Netbook: Slackware 13.37
Posts: 8,551
Blog Entries: 28

Rep: Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176
Quote:
Originally Posted by tuxdev View Post
Code:
patterns=("/usr/*/foo*", "/var/r*/bar*"j, "/mnt/qux*")
for pattern in "${patterns[@]}" ; do
   while IFS="" read -r -d "" file ; do
      echo "$file"
   done < <(find / -path "$pattern" -print0)
done
Hello tuxdev

Nice, robust technique.

I tried the find / -path approach but discarded it for a couple of reasons. The most important was that find does not regard the "/" character as special in -path patterns so "/etc/*.cfg" would match "/etc/foo/my.cfg" which is counter-intuitive. The second was that it searches the whole file system hierarchy which may be resource intensive.

The technique could be modified if the search patterns did not have wildcarded directories in which case the search pattern could be parsed into directory and filename_pattern
Code:
find "$directory" -maxdepth 1 -type f -name "filename_pattern" -print0
This would avoid both the problems described above.

BTW, there's a typo in the done < <(find / -path "$pattern" -print0), which should be done < $(find / -path "$pattern" -print0)
 
Old 02-16-2010, 03:00 PM   #10
tuxdev
Senior Member
 
Registered: Jul 2005
Distribution: Slackware
Posts: 2,014

Rep: Reputation: 115Reputation: 115
Quote:
I tried the find / -path approach but discarded it for a couple of reasons. The most important was that find does not regard the "/" character as special in -path patterns so "/etc/*.cfg" would match "/etc/foo/my.cfg" which is counter-intuitive. The second was that it searches the whole file system hierarchy which may be resource intensive.
Yeah, that's definitely an issue if that's not what you want to do. I wasn't exactly sure what 'find' behaviour the OP wanted, so the actual command is more a placeholder than anything else. Same with searching / rather than something more specific.

Quote:
BTW, there's a typo in the done < <(find / -path "$pattern" -print0), which should be done < $(find / -path "$pattern" -print0)
No, this is actually correct. <() is similar to $(), but substitutes to a file with the output of the command rather than expanding to the output of the command itself. <() behaves almost identically to <<< "$()", except that the latter breaks when the output contains null characters. Try
Code:
echo $(echo foo)
echo <(echo foo)
On my bash, this produced
Code:
foo
/dev/fd/63

Last edited by tuxdev; 02-16-2010 at 03:01 PM.
 
1 members found this post helpful.
Old 02-16-2010, 10:30 PM   #11
catkin
LQ 5k Club
 
Registered: Dec 2008
Location: Tamil Nadu, India
Distribution: Servers: Debian Squeeze and Wheezy. Desktop: Slackware64 14.0. Netbook: Slackware 13.37
Posts: 8,551
Blog Entries: 28

Rep: Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176
Quote:
Originally Posted by tuxdev View Post
No, this is actually correct. <() is similar to $(), but substitutes to a file with the output of the command rather than expanding to the output of the command itself. <() behaves almost identically to <<< "$()", except that the latter breaks when the output contains null characters. Try
Code:
echo $(echo foo)
echo <(echo foo)
On my bash, this produced
Code:
foo
/dev/fd/63
Thank you for something new Found the details here.
 
Old 02-17-2010, 12:19 AM   #12
allanf
Member
 
Registered: Sep 2008
Location: MN
Distribution: Gentoo, Fedora, Suse, Slackware, Debian, CentOS
Posts: 97
Blog Entries: 1

Rep: Reputation: 19
Quote:
Originally Posted by tuxdev View Post
Code:
patterns=("/usr/*/foo*", "/var/r*/bar*"j, "/mnt/qux*")
for pattern in "${patterns[@]}" ; do
   while IFS="" read -r -d "" file ; do
      echo "$file"
   done < <(find / -path "$pattern" -print0)
done

This is no justification to allow scripts to break. Really, ever since I've started over-quoting, I haven't had a spaces-in-filename or shell metacharacters problem. I still do IFS=$'\n' on the top of my scripts, but that's more insurance than anything else.
Here is a bunch of files that can be used to test script handling of valid but hard to handle characters. Just think of the problems.
Code:
http://freeyourip.oiihob.org/76743c05daab6e517da22463deec3092/strange_names.tbz
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
passing variable (a path) with spaces aolong Linux - General 2 06-28-2009 05:50 PM
How to make shell variable don't expand immediately linuxgentoo Programming 1 06-13-2009 08:36 AM
bash script path issue - how to pass a path as a string to a variable PiNPOiNT Programming 5 04-17-2009 05:48 PM
Execute command with spaces from variable in bash script klo_2k Linux - Newbie 4 04-13-2008 02:59 AM
how to expand wildcards in a path with spaces? sts Linux - General 1 11-12-2005 01:50 PM


All times are GMT -5. The time now is 10:00 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration