Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question?
If it is not in the man pages or the how-to's this is the place! |
Notices |
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
Are you new to LinuxQuestions.org? Visit the following links:
Site Howto |
Site FAQ |
Sitemap |
Register Now
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
|
 |
12-11-2009, 04:06 AM
|
#1
|
LQ Newbie
Registered: Dec 2009
Posts: 7
Rep:
|
BASH-Constructing array from large numerical matrix .txt
Okay, so very to bash scripting but comfortable using MATLAB. I have searched multiple forums and websites (with lots of experimenting) but can't quite find a working solution to my query, so I hope someone can point me in the right direction.
I have a txt file containing a nx3 matrix of data:
11 21 33
4 56 90
4 45 45
2 1 90
.......etc., Where n=1......thousands
I want to read each line in bash, declaring each element as a separate variable.
e.g. For n=1: x=11, y=21, z=33
These variables are then used for a specific task which I will loop, changing the x,y,z variables at each iteration until the end of the file.
Current progress (limited):
ARRAY='cat artest.txt'
.... I get the whole matrix in the first element of the array, I can count the elements and also arrange into -L3 using xargs but cant access separate elements. What I think I need is to declare each element as an array and then loop updating..??
Matlab Example:
I include this to clarify what I'm aiming for:
P=load artest.txt
X=P(:,1); Y=(P(:,2); Z=(P(:,3));
for i=1:length(X)
xtask=(X(i));
ytask=(Y(i));
ztask=(Z(i));
%.........>>> these values are then used for a specific program
end
Bit of a rant. Thanks for any help you can provide.
|
|
|
12-11-2009, 04:20 AM
|
#2
|
Senior Member
Registered: Aug 2006
Posts: 2,697
|
you mentioned in the thousands, so you might as well use something better, such as Perl, or Python ( and use their matrix/math libraries if possible). If you are restricted to only shell+shell tools, try using awk.
Code:
awk '{
print NR, $1,$2,$3
}' matrixfile
NR means the record number, $1 to $3 are your columns, ie, x, y, z in your example. please read more on awk (see my sig) to know how you can use it for your needs
|
|
|
12-11-2009, 04:28 AM
|
#3
|
LQ 5k Club
Registered: Dec 2008
Location: Tamil Nadu, India
Distribution: Debian
Posts: 8,578
|
Bash doesn't have two-dimensional arrays but you can simulate in this case by using index sets 11,12,13; 21,22,23; 31,32,33 etc.
Not tested, just for the ideas ...
Code:
#!/bin/bash
i=1
while read line
do
buf=( $line )
array[${i}1]=${buf[0]}
array[${i}2]=${buf[1]}
array[${i}3]=${buf[2]}
let i=i+1
done <<< "$( cat artest.txt )"
Last edited by catkin; 12-11-2009 at 04:41 AM.
Reason: Half the code jumped out of the code box -- all on its own :-)
|
|
|
12-11-2009, 04:33 AM
|
#4
|
Senior Member
Registered: Oct 2005
Distribution: Gentoo, Slackware, LFS
Posts: 2,248
|
do you mean something like this?
Code:
{
N=0
while read X Y Z; do
(( ++N ))
< do something >
done
} < file.txt
|
|
|
12-11-2009, 04:45 AM
|
#5
|
LQ 5k Club
Registered: Dec 2008
Location: Tamil Nadu, India
Distribution: Debian
Posts: 8,578
|
Quote:
Originally Posted by ghostdog74
you mentioned in the thousands, so you might as well use something better, such as Perl, or Python ( and use their matrix/math libraries if possible). If you are restricted to only shell+shell tools, try using awk.
|
Good point -- bash is not quick when handling a lot of data.
|
|
|
12-11-2009, 07:13 AM
|
#6
|
LQ Newbie
Registered: Dec 2009
Posts: 7
Original Poster
Rep:
|
Working solution
Thanks for all of the above. The solution I'm using at the moment (as it seems to work) goes along these lines:
myfile=arexample.txt
exec 4<$myfile
linecount=0
while read x y z<&4; do
linecount+=1
read x y z <&4
echo $x $y $z <------------Reassurance it is working
...
.. other processes inserted here calling on $x $y $z values
...
done
exec 4<&-
|
|
|
12-11-2009, 10:37 PM
|
#7
|
Senior Member
Registered: Oct 2005
Distribution: Gentoo, Slackware, LFS
Posts: 2,248
|
yup exec does things good as well esp. when reading input from coprocesses. you can also use 'read -u 4' to make that a bit cleaner. :-)
Quote:
Originally Posted by ghostdog74
..., try using awk.
|
Quote:
Originally Posted by catkin
Good point -- bash is not quick when handling a lot of data.
|
I agree as well. Awk is powerful when it comes to text and it can also do some math like creating hashes. Even a script compiler can be cleanly made with it. Base from the benchmarks I made with my scripts, it's really fast.
|
|
|
12-13-2009, 06:17 PM
|
#8
|
LQ Guru
Registered: Aug 2004
Location: Sydney
Distribution: Rocky 9.x
Posts: 18,443
|
The Perl Data Lang extension (PDL) is designed for exactly this sort of thing and will be much faster than bash.
http://pdl.perl.org/
|
|
|
12-13-2009, 07:24 PM
|
#9
|
Senior Member
Registered: Aug 2006
Posts: 2,697
|
Quote:
Originally Posted by chrism01
The Perl Data Lang extension (PDL) is designed for exactly this sort of thing and will be much faster than bash.
http://pdl.perl.org/
|
true, and for Python, Numpy.
|
|
|
All times are GMT -5. The time now is 05:54 PM.
|
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.
|
Latest Threads
LQ News
|
|