How to read in line-by-line from a file and store them in several variables

ncjlee · 09-28-2008, 05:32 PM

Hi all,

I need to read line-by-line from multiple files of the same format and store each line to a variable so that I can do arithmatic with those multiple variables in bash. For example,
Let's say I have two text files of this format
1 2
3 4
5 6 7 and

9 10
11 12
13 14 15

I want to read each line from the two files and take average of each line so that resulting numbers will be

5 6
7 8
9 10 11.

How would I do this most effectively? I'm thinking using arrays will somehow be helpful but haven't been able to successfully implement it.

Thanks for any comments.

jan61 · 09-28-2008, 07:20 PM

Moin,

uhm, this question is not so easy to answer to

- I would prefer to implement it using a "real" programming language like perl, but you asked for a bash solution. It took me a while to get a running script - and it's not easy to understand! You have been warned ;-)

I assume, that the number of files, the number of lines within each file and the number of fields within each line are not pref-defined, but all files have the same structure (all files have exactly n lines and the line with number x has exactly m fields in each file)? I don't check that in my script.

So let's look at my bash script. The problem is, that you need a 3 dimension array (file, line, fields), but the bash offers only 1 dimension arrays. So I produced different arrays with dynamically created names. My way to do this is eval. I inserted some output on STDERR to show, how the script works. Feel free to redirect it to /dev/null, if you only want to see the result.

Code:

jan@jack:~/tmp/multi_files> cat data1
1 2
3 4
5 6 7
jan@jack:~/tmp/multi_files> cat data2
9 10
11 12
13 14 15
jan@jack:~/tmp/multi_files> cat data3
18 20
21 22
23 24 25
jan@jack:~/tmp/multi_files> cat average.sh
#! /bin/bash

file_no=0

for file_name in $*; do
  line_no=0
  echo $file_name "-->" $file_no >&2
  while read line; do
    echo -e "\tline_no=$line_no line=($line)" >&2
    eval "declare -a f_arr_${file_no}_${line_no}"
    eval "f_arr_${file_no}_${line_no}=($line)"
    eval echo '${f_arr_'${file_no}'_'${line_no}'[*]}' >&2
    line_no=$((line_no + 1))
  done <$file_name
  file_no=$((file_no + 1))
done

line_no=$((line_no - 1))
file_no=$((file_no - 1))
echo >&2

for l_no in `seq 0 $line_no`; do
  num_fields=`eval echo '${#f_arr_0_'${l_no}'[*]}'`
  echo l_no=$l_no num_fields=$num_fields >&2
  output=
  for (( i=0; i < $num_fields; i++ )); do
    sum=0
    for f_no in `seq 0 $file_no`; do
      eval echo l_no=$l_no f_no=$f_no i=$i '${f_arr_'${f_no}'_'${l_no}'['$i']}' >&2
      val=`eval echo '${f_arr_'${f_no}'_'${l_no}'['$i']}'`
      sum=$((sum + $val))
    done
    avg=$((sum / ($file_no + 1)))
    echo avg = $sum / $((file_no + 1)) >&2
    output="$output $avg"
  done
  echo $output
done

exit 0
jan@jack:~/tmp/multi_files> ./average.sh data* 2>/dev/null
9 10
11 12
13 14 15

I didn't comment anything - it's too late here in germany and I have to go to work in some hours. If you don't understand some parts of my script - ask. I'll answer later this day.

Jan