Read list of files as arguments in a script

garr0n · 10-17-2007, 05:17 AM

This is a theoretical problem at this point for me, but nonetheless it's something that's been bothering me for awhile.

Say I have a list of files in a text file, one file per line. Some files have characters that need escaping (spaces, etc.). I want to pass each file as an argument to a program. How?

Code:

% cat filelist
./dir 1/file 1
./dir 2/file 2

% cat `cat filelist`
cat: ./dir: No such file or directory
cat: 1/file: No such file or directory
cat: 1: No such file or directory
cat: ./dir: No such file or directory
cat: 2/file: No such file or directory
cat: 2: No such file or directory

% cat "`cat filelist`"
cat: ./dir 1/file 1
./dir 2/file 2: No such file or directory

radoulov · 10-17-2007, 05:52 AM

Code:

bash 3.2.25(1)$ mkdir 'dir 1' 'dir 2'
bash 3.2.25(1)$ echo FileOne>'./dir 1/file 1'
bash 3.2.25(1)$ echo FileTwo>'./dir 2/file 2'
bash 3.2.25(1)$ echo './dir 1/file 1
> ./dir 2/file 2'>filelist
bash 3.2.25(1)$ cat `cat filelist`
cat: ./dir: No such file or directory
cat: 1/file: No such file or directory
cat: 1: No such file or directory
cat: ./dir: No such file or directory
cat: 2/file: No such file or directory
cat: 2: No such file or directory
bash 3.2.25(1)$ (IFS=$'\n';cat `cat filelist`)
FileOne
FileTwo

chrism01 · 10-17-2007, 06:55 AM

If you set the IFS to newline only in your script, you'll then be able to use the cat cmd to read in each line.
You'll still have to be careful to escape spaces during further processing.

bigearsbilly · 10-17-2007, 08:10 AM

try...

Code:

cat file_list | 
while read file ;do
   command "$file"
done

syg00 · 10-17-2007, 08:18 AM

Mmmm - both Chris and billy responded to a thread like this, and neither suggested perl as the solution.

I'm disappointed ...

garr0n · 10-17-2007, 12:58 PM

radoulov, chrism1: Great, thanks! Guess I need to read the bash manual a bit more.

bigearsbilly: That actually calls command once per argument, no? For 'cat' it has the same result, obviously. Still, it's always good to learn another trick that might help sometime down the road.

syg00: Yeah, "Learn perl" would be a great solution, if a little much in the short term.

bigearsbilly · 10-18-2007, 02:07 AM

hey, why not use perl?

(are we so predictable?)

ghostdog74 · 10-18-2007, 02:23 AM

Quote:

Originally Posted by bigearsbilly

try...

Code:

cat file_list | 
while read file ;do
   command "$file"
done

the cat can be skinned.

Code:

while read file ;do
   command "$file"
done < file_list

ghostdog74 · 10-18-2007, 02:28 AM

Quote:

Originally Posted by garr0n

bigearsbilly: That actually calls command once per argument, no? For 'cat' it has the same result, obviously. Still, it's always good to learn another trick that might help sometime down the road.

"cat"'s purpose is to concatenate files (although almost everyone i see use it to display files). use tools like awk (or even the while loop) if you want to process files.

Quote:

syg00: Yeah, "Learn perl" would be a great solution, if a little much in the short term.

before there's Perl, there's awk.

garr0n · 10-25-2007, 08:18 PM

I know cat is meant to concatinate files, it was just the simplest thing I could think of for an example.

Can this be done with awk? I've used awk a little but it was mostly just copying examples and replacing a few strings... I think awk is the next thing on my list of Unix tools to explore properly.

ghostdog74 · 10-25-2007, 10:19 PM

Quote:

Originally Posted by garr0n

Can this be done with awk?

of course. depending on what your program that takes in argument is

Code:

awk '{
  cmd="program \047"$0"\047" #single quote the file path with spaces
  system(cmd) #or cmd|getline
  close(cmd)
}' file

Quote:

I've used awk a little but it was mostly just copying examples and replacing a few strings...

its more than that. it loops over files so almost always, i don't have to use while/for loops. its grep/sed/cat/etc combined, so almost always, i only use awk for my job. Of course, this is just what i do. No need to disagree with me on this.

Quote:

I think awk is the next thing on my list of Unix tools to explore properly.

it should your next big thing