awk: how to read dan manipulate data in multi files from a file

mauludi · 05-26-2011, 04:14 AM

Dear All,

I wonder capability of awk to manipulate data in consecutive multi files by read one batch file.
for example I have files: data1.dat, data2.dat,data3.dat and listfile.txt

cat data1.dat
23 34 54
43 45 76

cat data2.dat
76 64 92
43 76 89

cat data3.dat
65 83 92
37 94 90

cat listfile.txt
data1.dat
data2.dat
data3.dat

how to make awk script which can manipulate/manage data in data1,data2,data3 by read listfile.txt . for exampale we want to sum up all column 1 of data1.dat,data2.dat and data3.dat

thank you in advance

Guttorm · 05-26-2011, 04:31 AM

Hi

You can use many files with awk, and the shell can use cat to get the file list. For example:

Code:

awk ... $(cat listfile.txt)

sycamorex · 05-26-2011, 04:44 AM

Or even without creating a separate file:

Code:

awk .... $(ls -1 /path/to/data*)

Please note that in "ls -1" it's number 1, not lower case L.

grail · 05-26-2011, 05:52 AM

Assuming that the files in listfile.txt are not all in the same location:

Code:

awk '{f = $0;while(getline < f)sum+=$1}END{print sum}' listfile.txt

mauludi · 05-27-2011, 01:52 AM

thank you for your nice explanation

sorry for ask more question, for example in each data file (data1.dat,data2.dat,data3.dat),there are not only one list of data they consist several list of data separated by some sentence. It look like:

cat data1.dat
temperature pressure stress
23 34 54
43 45 76
type charge
34 54
45 76

data2.dat and data3.dat are the same format.

If we only one to take data of type and charge, how to extract it. thank you very much.