how to loop over text file lines within bash script for loop?

johnpaulodonnell · 01-24-2007, 08:43 AM

Hi.

I have a data file containing 4 rows of data say:

X1 Y1 Z1
X2 Y2 Z2
X3 Y3 Z3
X4 Y4 Z4

Associated with each individual row is a data file containing a subset of additional data for that row. These associated data files are single column files, but not all have the same number of columns. For example that data file associated with row 1 might be of form:

R1a
R1b
R1c
R1d

Is there a way that I can pass the data file above (R1a, etc) to a for loop within a bash script such that it loops over the lines of the data file and outputs the following:

X1 Y1 Z1 R1a
X1 Y1 Z1 R1b
X1 Y1 Z1 R1c
X1 Y1 Z1 R1d

- and so on for each record of the main data file.

I've seen examples with individual files of directories being passed into for loops, as in:

for i in *
do
........
done

But what is the equivalent code when you want to loop over text file lines?

Thanks.

colucix · 01-24-2007, 09:22 AM

There are many possible solutions. Here's mine, using gawk inside a while loop:

Code:

#!/bin/sh

i=1

while [ $i -le `wc -l filename.txt | gawk '{print $1}'` ] ; do

    line=`head -$i filename.txt | tail -1`
    
    gawk -v prefix="$line" '{print prefix,$0}' $i.txt >> output.txt

    i=`expr $i + 1`
    
done

where filename.txt is the datafile and I assume that each "single column" file is called 1.txt (associated with first row) 2.txt (associated with second row) and so on. If the real "single column" files have progressive numbers in their name, you can easily build the name inside the loop. Otherwise you can do a list of these files and parse this list with the head/tail construct above.

Hope this will help.

scriptjunkie1 · 04-30-2009, 07:28 PM

Set the internal file separator:

Code:

IFS=$'\n'; for line in `ls`; do echo $line; done

(this just gets each line for you easier than head/tail. the rest is the same)

ghostdog74 · 04-30-2009, 07:50 PM

this is an old thread.
first, no need to change IFS and using ls is "useless" .

Code:

for line in *

secondly, OP wants to loop over the lines in the files (based on his post's last sentence), not filenames.
lastly, the tool to use could be simple paste/join.

H_TeXMeX_H · 05-01-2009, 06:11 AM

Why not just use paste ?

Code:

paste file1 file2 > file 3

PMP · 05-01-2009, 08:27 AM

Try Something like this

exec 3>&0
exec 0<$filename
while read eachline
do
echo $eachline
done
exec 0>&3

Cheers

downeyt · 03-20-2011, 03:48 PM

Better late than never

I was looking for the same answer and saw this post.
Eventually, I came up with this:

for line in `cat fileName.txt`
do
---
done

akamikeym · 03-20-2011, 07:38 PM

My favorite:

Code:

while read line; do
    echo "$line"
done < "$filename"

mrm5102 · 04-05-2011, 09:18 AM

Nice work