Latest LQ Deal: Complete CCNA, CCNP & Red Hat Certification Training Bundle
Go Back > Forums > Linux Forums > Linux - Newbie
User Name
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!


  Search this Thread
Old 05-26-2011, 05:14 AM   #1
LQ Newbie
Registered: Jan 2010
Posts: 21

Rep: Reputation: 0
awk: how to read dan manipulate data in multi files from a file

Dear All,

I wonder capability of awk to manipulate data in consecutive multi files by read one batch file.
for example I have files: data1.dat, data2.dat,data3.dat and listfile.txt

cat data1.dat
23 34 54
43 45 76

cat data2.dat
76 64 92
43 76 89

cat data3.dat
65 83 92
37 94 90

cat listfile.txt

how to make awk script which can manipulate/manage data in data1,data2,data3 by read listfile.txt . for exampale we want to sum up all column 1 of data1.dat,data2.dat and data3.dat

thank you in advance
Old 05-26-2011, 05:31 AM   #2
Senior Member
Registered: Dec 2003
Location: Trondheim, Norway
Distribution: Debian and Ubuntu
Posts: 1,295

Rep: Reputation: 335Reputation: 335Reputation: 335Reputation: 335

You can use many files with awk, and the shell can use cat to get the file list. For example:

awk ... $(cat listfile.txt)
Old 05-26-2011, 05:44 AM   #3
LQ Veteran
Registered: Nov 2005
Location: London
Distribution: Slackware64-current
Posts: 5,825
Blog Entries: 1

Rep: Reputation: 1221Reputation: 1221Reputation: 1221Reputation: 1221Reputation: 1221Reputation: 1221Reputation: 1221Reputation: 1221Reputation: 1221
Or even without creating a separate file:

awk .... $(ls -1 /path/to/data*)
Please note that in "ls -1" it's number 1, not lower case L.
Old 05-26-2011, 06:52 AM   #4
LQ Guru
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 9,564

Rep: Reputation: 2901Reputation: 2901Reputation: 2901Reputation: 2901Reputation: 2901Reputation: 2901Reputation: 2901Reputation: 2901Reputation: 2901Reputation: 2901Reputation: 2901
Assuming that the files in listfile.txt are not all in the same location:
awk '{f = $0;while(getline < f)sum+=$1}END{print sum}' listfile.txt
Old 05-27-2011, 02:52 AM   #5
LQ Newbie
Registered: Jan 2010
Posts: 21

Original Poster
Rep: Reputation: 0
thank you for your nice explanation

sorry for ask more question, for example in each data file (data1.dat,data2.dat,data3.dat),there are not only one list of data they consist several list of data separated by some sentence. It look like:

cat data1.dat
temperature pressure stress
23 34 54
43 45 76
type charge
34 54
45 76

data2.dat and data3.dat are the same format.

If we only one to take data of type and charge, how to extract it. thank you very much.


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off

Similar Threads
Thread Thread Starter Forum Replies Last Post
want to open & manipulate column data from excel file. rinu budhbhatti Member Intro 1 02-01-2011 09:50 AM
Get data from multi lined text file using awk, sed or perl - grep & cut not upto par cam34 Programming 4 07-02-2010 04:10 AM
AWK/Perl for extracting data from txt file to numerous other files briana.paige Linux - Newbie 2 05-05-2009 10:53 AM
manipulate dcop data for File commandS? carl0ski Programming 2 07-02-2005 09:46 PM > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 09:02 AM.

Main Menu
Write for LQ is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration