BASH-Constructing array from large numerical matrix .txt

hippotonic · 12-11-2009, 04:06 AM

Okay, so very to bash scripting but comfortable using MATLAB. I have searched multiple forums and websites (with lots of experimenting) but can't quite find a working solution to my query, so I hope someone can point me in the right direction.

I have a txt file containing a nx3 matrix of data:

11 21 33
4 56 90
4 45 45
2 1 90
.......etc., Where n=1......thousands

I want to read each line in bash, declaring each element as a separate variable.

e.g. For n=1: x=11, y=21, z=33

These variables are then used for a specific task which I will loop, changing the x,y,z variables at each iteration until the end of the file.

Current progress (limited):

ARRAY='cat artest.txt'
.... I get the whole matrix in the first element of the array, I can count the elements and also arrange into -L3 using xargs but cant access separate elements. What I think I need is to declare each element as an array and then loop updating..??

Matlab Example:

I include this to clarify what I'm aiming for:

P=load artest.txt
X=P(:,1); Y=(P(:,2); Z=(P(:,3));

for i=1:length(X)
xtask=(X(i));
ytask=(Y(i));
ztask=(Z(i));
%.........>>> these values are then used for a specific program
end

Bit of a rant. Thanks for any help you can provide.

ghostdog74 · 12-11-2009, 04:20 AM

you mentioned in the thousands, so you might as well use something better, such as Perl, or Python ( and use their matrix/math libraries if possible). If you are restricted to only shell+shell tools, try using awk.

Code:

awk '{
 print NR, $1,$2,$3
}' matrixfile

NR means the record number, $1 to $3 are your columns, ie, x, y, z in your example. please read more on awk (see my sig) to know how you can use it for your needs

catkin · 12-11-2009, 04:28 AM

Bash doesn't have two-dimensional arrays but you can simulate in this case by using index sets 11,12,13; 21,22,23; 31,32,33 etc.

Not tested, just for the ideas ...

Code:

#!/bin/bash

i=1
while read line
do
    buf=( $line )
    array[${i}1]=${buf[0]}
    array[${i}2]=${buf[1]}
    array[${i}3]=${buf[2]}
    let i=i+1
done <<< "$( cat artest.txt )"

konsolebox · 12-11-2009, 04:33 AM

do you mean something like this?

Code:

{
    N=0
    while read X Y Z; do
        (( ++N ))
        < do something >
    done
} < file.txt

catkin · 12-11-2009, 04:45 AM

Quote:

Originally Posted by ghostdog74

you mentioned in the thousands, so you might as well use something better, such as Perl, or Python ( and use their matrix/math libraries if possible). If you are restricted to only shell+shell tools, try using awk.

Good point -- bash is not quick when handling a lot of data.

hippotonic · 12-11-2009, 07:13 AM

Thanks for all of the above. The solution I'm using at the moment (as it seems to work) goes along these lines:

myfile=arexample.txt
exec 4<$myfile

linecount=0

while read x y z<&4; do
linecount+=1
read x y z <&4
echo $x $y $z <------------Reassurance it is working

...
.. other processes inserted here calling on $x $y $z values
...

done

exec 4<&-

konsolebox · 12-11-2009, 10:37 PM

yup exec does things good as well esp. when reading input from coprocesses. you can also use 'read -u 4' to make that a bit cleaner. :-)

Quote:

Originally Posted by ghostdog74

..., try using awk.

Quote:

Originally Posted by catkin

Good point -- bash is not quick when handling a lot of data.

I agree as well. Awk is powerful when it comes to text and it can also do some math like creating hashes. Even a script compiler can be cleanly made with it. Base from the benchmarks I made with my scripts, it's really fast.

chrism01 · 12-13-2009, 06:17 PM

The Perl Data Lang extension (PDL) is designed for exactly this sort of thing and will be much faster than bash.
http://pdl.perl.org/

ghostdog74 · 12-13-2009, 07:24 PM

Quote:

Originally Posted by chrism01

The Perl Data Lang extension (PDL) is designed for exactly this sort of thing and will be much faster than bash.
http://pdl.perl.org/

true, and for Python, Numpy.