Read multiple files line by line

idaham · 11-04-2009, 11:14 AM

Hi!

I've just started writing a script to read multiple files and save the data in different variables. More precisely, I have two different files containing one column of data each (both files equally many rows):

Code:

file1:    file2:
0         3
1         5
2         20
3         3
.         .
.         .

I want to read line by line in a for or while loop and save the corresponding row value in two different variables (that I will later use as input into another function). It would work something like, in "semi-code" =):

Code:

for i=1 to "number of rows in files"
   variable1 = i:throw in file1
   variable2 = i:th row in file2
   function(variable1, variable2)
end

I've managed to do what I want when just reading one file:

Code:

#!/bin/bash
cat file1.dat | while read line 
do 
   variable1="$line"
   function($variable1)
done

Any ideas how to transform this to reading multiple files at the same time? Thanks!
Ida

catkin · 11-04-2009, 11:45 AM

It could be done by setting up the other files on file descriptors (other than 0, 1 and 2 which are already in use) and using them as inputs to reads within the loop reading file1.dat. There are some examples at The Linux Documentation Project. It would require input redirect rather than the cat idiom in your existing code, for example

Code:

read <&3

bsat · 11-05-2009, 01:00 AM

Hi if you are aware of awk, it can be done a little more easily. I am not sure what you want to do in function1, hope this helps.

put this code into a file say "awk1"
FILENAME==file1{
array1[i++] = $1
}
FILENAME==file2{
array2[j++]=$1
}
END{
#Do what ever you want to do with the two sets of arrays
}

run the script using the following way

awk -f awk1 file1 file2

evo2 · 11-05-2009, 01:13 AM

You can use paste

Code:

#!/bin/bash
paste file1 file2 | (
while read col1 col2 ; do
echo "Data: ${col1}, ${col2}"
done
)

Evo2.

idaham · 11-05-2009, 03:33 AM

Thank you all for your great and quick tips!

I decided to follow the exec approach... I've now managed to read my files into arrays:

Code:

i=0
exec 3<"$file"
exec 4<"$file2"
while read value1 <&3
do
	read value2 <&4
	array1[$i]=$value1
	array2[$i]=$value2
	i=$(($i +1))
done

I now want to be able to actually use the values stored in the arrays like:

Code:

for j=0 to size of array-1
    start = array1[j]
    end = array1[j+1]
    some more stuff...
done

but I haven't been able to do this yet. I'm having problems actually retrieving the array data. I've tried with (and failed):

Code:

for ((j=0;j<=$(($i-1));j+=1))
do
    start=array1[$j]
    end=array1[$(($j+1))]
    echo $start - $end
done

Any more suggestions? thanks guys!
Ida

Disillusionist · 11-05-2009, 04:00 AM

Quote:

Originally Posted by idaham

I've tried with (and failed):

Code:

for ((j=0;j<=$(($i-1));j+=1))
do
    start=array1[$j]
    end=array1[$(($j+1))]
    echo $start - $end
done

Any more suggestions? thanks guys!
Ida

Try:

Code:

for ((j=0;j<=$(($i-1));j+=1))
do
    start=${array1[$j]}
    end=${array1[$j+1]}
    echo $start - $end
done

idaham · 11-05-2009, 04:19 AM

Thanks Disillusionist!
Such a simple solution that did just what I wanted! =)
/Ida

Quote:

Originally Posted by Disillusionist

Try:

Code:

for ((j=0;j<=$(($i-1));j+=1))
do
    start=${array1[$j]}
    end=${array1[$j+1]}
    echo $start - $end
done