LinuxQuestions.org
LinuxAnswers - the LQ Linux tutorial section.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices

Reply
 
Search this Thread
Old 12-11-2009, 04:06 AM   #1
hippotonic
LQ Newbie
 
Registered: Dec 2009
Posts: 7

Rep: Reputation: 0
BASH-Constructing array from large numerical matrix .txt


Okay, so very to bash scripting but comfortable using MATLAB. I have searched multiple forums and websites (with lots of experimenting) but can't quite find a working solution to my query, so I hope someone can point me in the right direction.

I have a txt file containing a nx3 matrix of data:

11 21 33
4 56 90
4 45 45
2 1 90
.......etc., Where n=1......thousands

I want to read each line in bash, declaring each element as a separate variable.

e.g. For n=1: x=11, y=21, z=33

These variables are then used for a specific task which I will loop, changing the x,y,z variables at each iteration until the end of the file.

Current progress (limited):

ARRAY='cat artest.txt'
.... I get the whole matrix in the first element of the array, I can count the elements and also arrange into -L3 using xargs but cant access separate elements. What I think I need is to declare each element as an array and then loop updating..??

Matlab Example:

I include this to clarify what I'm aiming for:

P=load artest.txt
X=P(:,1); Y=(P(:,2); Z=(P(:,3));

for i=1:length(X)
xtask=(X(i));
ytask=(Y(i));
ztask=(Z(i));
%.........>>> these values are then used for a specific program
end

Bit of a rant. Thanks for any help you can provide.
 
Old 12-11-2009, 04:20 AM   #2
ghostdog74
Senior Member
 
Registered: Aug 2006
Posts: 2,695
Blog Entries: 5

Rep: Reputation: 240Reputation: 240Reputation: 240
you mentioned in the thousands, so you might as well use something better, such as Perl, or Python ( and use their matrix/math libraries if possible). If you are restricted to only shell+shell tools, try using awk.

Code:
awk '{
 print NR, $1,$2,$3
}' matrixfile
NR means the record number, $1 to $3 are your columns, ie, x, y, z in your example. please read more on awk (see my sig) to know how you can use it for your needs
 
Old 12-11-2009, 04:28 AM   #3
catkin
LQ 5k Club
 
Registered: Dec 2008
Location: Tamil Nadu, India
Distribution: Servers: Debian Squeeze and Wheezy. Desktop: Slackware64 14.0. Netbook: Slackware 13.37
Posts: 8,528
Blog Entries: 27

Rep: Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176
Bash doesn't have two-dimensional arrays but you can simulate in this case by using index sets 11,12,13; 21,22,23; 31,32,33 etc.

Not tested, just for the ideas ...
Code:
#!/bin/bash

i=1
while read line
do
    buf=( $line )
    array[${i}1]=${buf[0]}
    array[${i}2]=${buf[1]}
    array[${i}3]=${buf[2]}
    let i=i+1
done <<< "$( cat artest.txt )"

Last edited by catkin; 12-11-2009 at 04:41 AM. Reason: Half the code jumped out of the code box -- all on its own :-)
 
Old 12-11-2009, 04:33 AM   #4
konsolebox
Senior Member
 
Registered: Oct 2005
Distribution: Gentoo, Slackware, LFS
Posts: 2,245
Blog Entries: 15

Rep: Reputation: 233Reputation: 233Reputation: 233
do you mean something like this?
Code:
{
    N=0
    while read X Y Z; do
        (( ++N ))
        < do something >
    done
} < file.txt
 
Old 12-11-2009, 04:45 AM   #5
catkin
LQ 5k Club
 
Registered: Dec 2008
Location: Tamil Nadu, India
Distribution: Servers: Debian Squeeze and Wheezy. Desktop: Slackware64 14.0. Netbook: Slackware 13.37
Posts: 8,528
Blog Entries: 27

Rep: Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176
Quote:
Originally Posted by ghostdog74 View Post
you mentioned in the thousands, so you might as well use something better, such as Perl, or Python ( and use their matrix/math libraries if possible). If you are restricted to only shell+shell tools, try using awk.
Good point -- bash is not quick when handling a lot of data.
 
Old 12-11-2009, 07:13 AM   #6
hippotonic
LQ Newbie
 
Registered: Dec 2009
Posts: 7

Original Poster
Rep: Reputation: 0
Working solution

Thanks for all of the above. The solution I'm using at the moment (as it seems to work) goes along these lines:

myfile=arexample.txt
exec 4<$myfile

linecount=0

while read x y z<&4; do
linecount+=1
read x y z <&4
echo $x $y $z <------------Reassurance it is working

...
.. other processes inserted here calling on $x $y $z values
...

done

exec 4<&-
 
Old 12-11-2009, 10:37 PM   #7
konsolebox
Senior Member
 
Registered: Oct 2005
Distribution: Gentoo, Slackware, LFS
Posts: 2,245
Blog Entries: 15

Rep: Reputation: 233Reputation: 233Reputation: 233
yup exec does things good as well esp. when reading input from coprocesses. you can also use 'read -u 4' to make that a bit cleaner. :-)

Quote:
Originally Posted by ghostdog74 View Post
..., try using awk.
Quote:
Originally Posted by catkin View Post
Good point -- bash is not quick when handling a lot of data.
I agree as well. Awk is powerful when it comes to text and it can also do some math like creating hashes. Even a script compiler can be cleanly made with it. Base from the benchmarks I made with my scripts, it's really fast.
 
Old 12-13-2009, 06:17 PM   #8
chrism01
Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.5, Centos 5.10
Posts: 16,226

Rep: Reputation: 2022Reputation: 2022Reputation: 2022Reputation: 2022Reputation: 2022Reputation: 2022Reputation: 2022Reputation: 2022Reputation: 2022Reputation: 2022Reputation: 2022
The Perl Data Lang extension (PDL) is designed for exactly this sort of thing and will be much faster than bash.
http://pdl.perl.org/
 
Old 12-13-2009, 07:24 PM   #9
ghostdog74
Senior Member
 
Registered: Aug 2006
Posts: 2,695
Blog Entries: 5

Rep: Reputation: 240Reputation: 240Reputation: 240
Quote:
Originally Posted by chrism01 View Post
The Perl Data Lang extension (PDL) is designed for exactly this sort of thing and will be much faster than bash.
http://pdl.perl.org/
true, and for Python, Numpy.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
How to input values from *.txt to an array in C++ HBMSGUY Programming 6 11-18-2008 09:33 PM
An array matrix problem in a class Asuralm Programming 4 12-06-2007 09:09 AM
C++: Saving numerical data from a file to an array geodpsa Programming 5 12-13-2005 02:31 PM
double matrix array in c alaios Programming 3 09-15-2005 11:34 AM


All times are GMT -5. The time now is 06:40 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration