list filenames with spaces in a shell script

xp_newbie · 03-15-2009, 04:51 PM

I am trying to iterate on all files in the current directory, using a bash script:

Code:

LSFILELIST=`ls -1`

for EACHFILE in "${LSFILELIST}"; do
  echo "${EACHFILE}"
done

The problem: ${EACHFILE} "spits" the entire list of files, thinking it is one long filename (with spaces).

Trying to troubleshoot the problem, I came to the conclusion that the difficulty in using the for loop in such manner lies in the fact that I can current use $LSFILELIST in either:

1) bare (no quotes) $LSFILELIST:

Quote:

==> $LSFILELIST="file name1 file name2 file name3"

2) quoted "${LSFILELIST}"

Quote:

==> $LSFILELIST="file name1
file name2
file name3"

Neither of the above satisfies bash's 'for'!

What I really need instead is some mechanism that will generate for me $LSFILELIST like this:

Quote:

$LSFILELIST="file name1"
"file name2"
"file name3"

However, I don't know to accomplish this.

Clues, tips, hints or ideas would be greatly appreciated.

Thanks!

billymayday · 03-15-2009, 05:03 PM

This will find them for you

Code:

find ./ -type f -name '* *'

Hko · 03-15-2009, 06:14 PM

Two way of looping over file names from "ls" command, handling files names with spaces correctly:

Pipe output of "ls" directly to loop. Disadvantage is that if some variable is assigned to or changed inside the loop, the value will be lost after the loop. (TESTVAR here used to show this effect).

Code:

#!/bin/bash

TESTVAR="Initial test value"

ls -1 | while read FILE; do
    echo "== FIRST LINE OF: $FILE"
    head -1 "$FILE"  # double quotes needed for filename with spaces!
    echo 
    TESTVAR="Changed!"
done

echo "Value of TESTVAR after te loop = \"$TESTVAR\""
exit 0

One way to solve the issue with changes to a variable inside the loop it to save the output of ls to a temp file.

Code:

#!/bin/bash

TESTVAR="Initial test value"

TMP=`mktemp`  # Create a temp file, path stored in var $TMP
ls -1 >"$TMP" # Write list of files to temp file

while read FILE; do
    echo "== FIRST LINE OF: $FILE"
    head -1 "$FILE"  # double quotes needed for filename with spaces!
    echo 
    TESTVAR="Changed!"
done <$TMP # Feed the loop from temp file

echo "Value of TESTVAR after te loop = \"$TESTVAR\""
rm $TMP # When done with the file list, clean up temp file
exit 0

Note: When using a file name from a variable ($FILE in the examples) in a command, you should double-quote it to have it work with file names with spaces in them.

xp_newbie · 03-15-2009, 06:35 PM

Hko, you seem to have found a solution to my problem although I may have not explained it very well: I don't need to read the files - I only need to have their filenames for various operations.

Thus, the following script is enough and seems to work:

Code:

#!/bin/bash

ls -1 | while read FILE; do
    echo "$FILE"  # double quotes needed for filename with spaces!
done

exit 0

However, I am not sure what the 'read' above does: Does it really open the file and attempts to read from it?

If so, then this may not be very efficient. I just want to be able to check whether the file exists in another directory - I don't need to read it or even open it.

Is there a way to accomplish the above without the 'read'?

Thank you!

Hko · 03-15-2009, 07:07 PM

Quote:

Originally Posted by xp_newbie

Code:

#!/bin/bash

ls -1 | while read FILE; do
    echo "$FILE"  # double quotes needed for filename with spaces!
done

exit 0

However, I am not sure what the 'read' above does: Does it really open the file and attempts to read from it?

No. it does not read or open the files.

while read SOME_VAR; do just loops over each line it get from the standard input ("stdin"), assigning the line read to $SOME_VAR each time.

If you use it this way:

Code:

while read SOME_VAR; do
    echo "==> $SOME_VAR"
done

This script will just wait for you to enter some lines until you press ctrl-d to signal "end of file (EOF)". The loop will echo each line you typed with "==>" before each line.

If you pipe the contents of a file with "cat" to the standard input of such a while-read-loop:

Code:

cat /path/to/some/file.txt | while read LINE; do
    echo "==> $LINE"
done

Then "cat" reads the file and feeds it to the standard input (stdin) of the while-read-loop. So, the loop runs for each line in the file.

What I did in the post above is feed the output of "ls -1" to the while-read-loop the same way.

You said you need the list of file names "for various operations". If you need to loop more than one time over the same list of files, I'd recommend using the second example I posted (storing the list in a temp-file).

It is probably possible to do without a read-loop and do it with a for-loop instead. But I don't know one from the top of my head. Would need to fiddle with file seperators and some extra comands I suppose. But IMO there's no need to explicitly using a for-loop, and it would be probably be less efficient than while-read especially when the list of files is long.

David the H. · 03-15-2009, 07:34 PM

I've always found spaces to be a big headache too, and things like quoting variables and whatnot often don't work very well. I've found that the least troublesome way to handle spaces is to change the internal field separator in bas so that it doesn't recognize them.

Simply add the following to your script, or run them in the terminal:

Code:

OLDIFS=$IFS

IFS="
"        #this is a line break

{the block of commands you want to run}

IFS=$OLDIFS

Now if everything goes as planned, spaces will be ignored in filenames.
Note that it isn't really necessary to back up and reset the IFS in a script, since variables set in scripts are only local in scope.

The default IFS is space, tab, and newline, BTW.

xp_newbie · 03-15-2009, 07:46 PM

Quote:

Originally Posted by Hko

If you need to loop more than one time over the same list of files, I'd recommend using the second example I posted (storing the list in a temp-file).

Thank you very much for your great help. No, I don't need to loop more than once, so your original solution is 100% PERFECT for me. Thank you!