Shell: script on each line in a file

jimmyrp · 05-22-2011, 12:07 PM

So if I have a file with a list of files:

file.txt
file1.a
file2.b
file3.c

and I wanted to perform a script using a loop or recursively on each of those files, is there any way that I could do this?

I saw that xargs might be able to help me.

Here is sorta what I want to do:

Code:

#!/bin/bash

cat file.txt | xargs?? # Some commands to turn each line into a variable

for var in ${@}; do # Some way to get each of the files

# script on each file

done

Any suggestions for me?

jschiwal · 05-22-2011, 12:25 PM

xargs would be used to add arguments to a command.

You could do something like:
filelist=$(cat file.txt)

Then iterate over filelist[0] .. filelist[n]

You could also do something like:
for file in $(cat filelist); do
<something>
done

If there is a particular command yoiu want to do, then xargs is handy.
find dir/ -type f -perm /002 | xargs chmod o-w

If the filename may contain whitespace or shell metacharacters, then it is a good idea to convert newlines to nulls and use -0 with xargs:

find dir/ -type f -perm /002 -print0 | xargs -0 chmod o-w

cat files.txt | tr '\n' '\0' | xargs -0 chmod o-w

Find and xargs are often used together. xargs also has options to limit the number of arguments which could cause an out of memory error in the shell. Using wild cards on a very large directory could cause such an error, and so find/xargs often is used to handle such situations.

David the H. · 05-22-2011, 01:29 PM

There are probably a dozen ways to do this.

xargs is one, certainly. AIUI, its main function is to build batches of commands that run as efficiently and with as few process threads as possible. It's most often used in combination with find, but it will work here too. It even has a -a option to read directly from a file.

Code:

xargs -I @ -a filelist.txt command @

The -I option specifies a replacement character or string. In this case "@" will be replaced by the filename in the final command. Here's a decent howto for more:
http://www.cyberciti.biz/faq/linux-u...lists-utility/

Most people however just use a simple shell loop of some kind.

Code:

while read filename, do 
	command "$filename"
done <filelist.txt

#==========#
IFS=$'\n'
for filename in $(<filelist.txt); do
	command "$filename"
done
unset IFS
#==========#

mapfile -t -O 1 filearray <filelist.txt  #mapfile is only available from bash v.4

IFS=$'\n'
filearray=( $(<filelist.txt) )  #alternate that will work in previous bashes,
unset IFS                       #but array will be zero-indexed!

for i in ${!filearray[@]}; do
	command "${filearray[i]}"
done

for i in 2 4 6; do
	command "${filearray[i]}"
done

The while-read loop is usually better, as reads in a whole line at a time, whereas the for loop uses an embedded command substitution to insert the complete file contents, and then separates the resulting input into individual "word" elements, which usually means it breaks on spaces. This can be worked around however, by temporarily setting the IFS field separator to newline-only.

Note also how the built-in < file redirector can be used here instead of of the external cat command.

The last one is the most flexible. mapfile loads lines from a file into an array, which you can then call back by index number. ${!filearray[@]} gives you a list of all index numbers available, or you can specify individual numbers instead. (Arrays are generally zero-indexed, but using mapfile's -O 1 option shifts the index origin to 1 to make the index numbers match the original file's line numbers).

Speaking of command substitution, if your command can take a list of files as-is, then it's possible to insert the contents of the file into the command line with it.

Code:

command $(<filelist.txt)

One final note. If you have a series of filenames that are all the same, except for an incrementing number or letter or something, such as foo1.txt foo2.txt foo3.txt, then you can use brace expansion to generate the list to work on.

Code:

for file in foo{1..3}.txt; do
   command "$file"
done

I suggest you simply work your way through a few bash tutorials and references, and try things out. You'll understand how it all works in time.

Here are a few useful bash scripting references:
http://www.linuxcommand.org/index.php
http://tldp.org/LDP/Bash-Beginners-G...tml/index.html
http://www.tldp.org/LDP/abs/html/index.html
http://www.gnu.org/software/bash/manual/bashref.html
http://wiki.bash-hackers.org/start