LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices

Reply
 
Search this Thread
Old 05-22-2011, 12:07 PM   #1
jimmyrp
LQ Newbie
 
Registered: May 2011
Posts: 9

Rep: Reputation: 0
Shell: script on each line in a file


So if I have a file with a list of files:

file.txt
file1.a
file2.b
file3.c

and I wanted to perform a script using a loop or recursively on each of those files, is there any way that I could do this?

I saw that xargs might be able to help me.

Here is sorta what I want to do:

Code:
#!/bin/bash

cat file.txt | xargs?? # Some commands to turn each line into a variable

for var in ${@}; do # Some way to get each of the files

# script on each file

done
Any suggestions for me?
 
Old 05-22-2011, 12:25 PM   #2
jschiwal
Guru
 
Registered: Aug 2001
Location: Fargo, ND
Distribution: SuSE AMD64
Posts: 15,733

Rep: Reputation: 654Reputation: 654Reputation: 654Reputation: 654Reputation: 654Reputation: 654
xargs would be used to add arguments to a command.

You could do something like:
filelist=$(cat file.txt)

Then iterate over filelist[0] .. filelist[n]

You could also do something like:
for file in $(cat filelist); do
<something>
done

If there is a particular command yoiu want to do, then xargs is handy.
find dir/ -type f -perm /002 | xargs chmod o-w

If the filename may contain whitespace or shell metacharacters, then it is a good idea to convert newlines to nulls and use -0 with xargs:

find dir/ -type f -perm /002 -print0 | xargs -0 chmod o-w

cat files.txt | tr '\n' '\0' | xargs -0 chmod o-w

Find and xargs are often used together. xargs also has options to limit the number of arguments which could cause an out of memory error in the shell. Using wild cards on a very large directory could cause such an error, and so find/xargs often is used to handle such situations.
 
Old 05-22-2011, 01:29 PM   #3
David the H.
Bash Guru
 
Registered: Jun 2004
Location: Osaka, Japan
Distribution: Debian sid + kde 3.5 & 4.4
Posts: 6,823

Rep: Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947
There are probably a dozen ways to do this.

xargs is one, certainly. AIUI, its main function is to build batches of commands that run as efficiently and with as few process threads as possible. It's most often used in combination with find, but it will work here too. It even has a -a option to read directly from a file.
Code:
xargs -I @ -a filelist.txt command @
The -I option specifies a replacement character or string. In this case "@" will be replaced by the filename in the final command. Here's a decent howto for more:
http://www.cyberciti.biz/faq/linux-u...lists-utility/


Most people however just use a simple shell loop of some kind.
Code:
while read filename, do 
	command "$filename"
done <filelist.txt

#==========#
IFS=$'\n'
for filename in $(<filelist.txt); do
	command "$filename"
done
unset IFS
#==========#

mapfile -t -O 1 filearray <filelist.txt  #mapfile is only available from bash v.4

IFS=$'\n'
filearray=( $(<filelist.txt) )  #alternate that will work in previous bashes,
unset IFS                       #but array will be zero-indexed!

for i in ${!filearray[@]}; do
	command "${filearray[i]}"
done

for i in 2 4 6; do
	command "${filearray[i]}"
done
The while-read loop is usually better, as reads in a whole line at a time, whereas the for loop uses an embedded command substitution to insert the complete file contents, and then separates the resulting input into individual "word" elements, which usually means it breaks on spaces. This can be worked around however, by temporarily setting the IFS field separator to newline-only.

Note also how the built-in < file redirector can be used here instead of of the external cat command.

The last one is the most flexible. mapfile loads lines from a file into an array, which you can then call back by index number. ${!filearray[@]} gives you a list of all index numbers available, or you can specify individual numbers instead. (Arrays are generally zero-indexed, but using mapfile's -O 1 option shifts the index origin to 1 to make the index numbers match the original file's line numbers).

Speaking of command substitution, if your command can take a list of files as-is, then it's possible to insert the contents of the file into the command line with it.
Code:
command $(<filelist.txt)
One final note. If you have a series of filenames that are all the same, except for an incrementing number or letter or something, such as foo1.txt foo2.txt foo3.txt, then you can use brace expansion to generate the list to work on.
Code:
for file in foo{1..3}.txt; do
   command "$file"
done


I suggest you simply work your way through a few bash tutorials and references, and try things out. You'll understand how it all works in time.

Here are a few useful bash scripting references:
http://www.linuxcommand.org/index.php
http://tldp.org/LDP/Bash-Beginners-G...tml/index.html
http://www.tldp.org/LDP/abs/html/index.html
http://www.gnu.org/software/bash/manual/bashref.html
http://wiki.bash-hackers.org/start
 
  


Reply

Tags
bash, shell script


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
bash shell script read file line by line. Darren[UoW] Programming 51 07-04-2013 08:16 PM
shell script replace a line from a file dthanoon Programming 11 03-05-2010 09:46 AM
how to replace line of file with another line using awk in shell script amit_pansuria Programming 3 03-29-2009 09:43 AM
shell script: insert line in a file noir911 Programming 6 02-04-2008 10:42 PM


All times are GMT -5. The time now is 04:41 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration