Reading files in shell scripting

barunparichha · 10-25-2010, 06:34 AM

Hi,
I am trying to read the fields of a file and manipulate them, record by record.

Lets say using awk :
awk -F":" `{print $1 $2 $3 $4 $5}' TrackMsgFile.0806`

This prints my fields on screen.But I dont want to print these fields while reading the records instead store them in some variable and manipulate them as per my logic.

Does "awk" or some other shell command provides something for this ?

GrapefruiTgirl · 10-25-2010, 06:36 AM

Assign the results to a shell variable:

Code:

var=$(awk -F":" '{print $1 $2 $3 $4 $5}' TrackMsgFile.0806)

echo "$var"

GrapefruiTgirl · 10-25-2010, 06:40 AM

If you want each field as a separate var:

Code:

awk -F":" '{print $1,$2,$3,$4,$5}' TrackMsgFile.0806 | while read one two three four five; do
    echo "Here's your vars: $one $two $three $four $five"
done

Note the commas separating the output, so the variables aren't all jumbled together in one.

ghostdog74 · 10-25-2010, 07:01 AM

Quote:

Originally Posted by GrapefruiTgirl

If you want each field as a separate var:

Code:

awk -F":" '{print $1,$2,$3,$4,$5}' TrackMsgFile.0806 | while read one two three four five; do
    echo "Here's your vars: $one $two $three $four $five"
done

Note the commas separating the output, so the variables aren't all jumbled together in one.

but the whole operation of awk+while is still iterating the file and processing them as it iterates. Therefore, the while loop really is redundant.

ghostdog74 · 10-25-2010, 07:03 AM

@OP, you might want to define clearly what you want to do. If you want to store your lines in some variable for LATER use, use and array

Code:

awk '{array[++d]=$0}
{ 
 ....
 # some other processing here.
} 
END{
  # here , you process your array
  for(i=1;i<=d;i++){
    print "do something with " array[i]
  }
}'

fbobraga · 10-25-2010, 07:10 AM

Quote:

Originally Posted by barunparichha

Does "awk" or some other shell command provides something for this ?

Or you can use a awk program to manipulate fields:

Code:

awk -F":" `{<do some stuff with $1-$5> ;print $1 $2 $3 $4 $5}' TrackMsgFile.0806`

there is a way of put the awk program externally, on a text file (prog.awk in the following example), with the -f switch:

Code:

awk -F":" -f prog.awk TrackMsgFile.0806`

see http://en.wikipedia.org/wiki/Awk

barunparichha · 10-25-2010, 08:26 AM

But lets say one record exists like
1::3::5

Here, I have omitted 2, 4 for explanation.
awk -F":" '{print $1,$2,$3,$4,$5}' TrackMsgFile.0806 | while read one two three four five;

3 will be stored for variable two, 5 for three.

Reason:
awk -F":" '{print $1,$2,$3,$4,$5}' TrackMsgFile.0806 , prints
1 3 5 (as 2nd, 4th filed are blank returns space character)

1 3 5 |while read one two three four five, takes

3,5 for variables two and three respectively.

=>
Is there a mechanism to avoid this ?

Likewise if it will show, wrong result for records like
1:2 3: 4:5 (the second filed contains "2 3") as we are separatinf fields using ":" as field separator)

GrapefruiTgirl · 10-25-2010, 08:38 AM

Code:

awk -F":" '{for (x=1;x<=5;x++){if($x==" "){$x="*"}};print $1,$2,$3,$4,$5}' TrackMsgFile.0806 | while read one two three four five; do
    echo "Here's your vars: $one $two $three $four $five"
done

Technically I'd have thought a space character would count as a character to be read into the appropriate variable, but for whatever reason (maybe multiple spaces count as one?) that doesn't happen.

There are mechanisms however, yes. For example, the above will make an "empty" field (a field that is a single space) return a "*" instead. So, you check your returned variables to see if it's a "*" and if so, then you know it was empty. Maybe a "*" is not the best choice of character to use - a better idea would be to return the word "EMPTY" instead. That would be sensible.

barunparichha · 10-25-2010, 08:53 AM

Assuming that, we dont use "*" in any of our records, the above code will work.

But it will still fail to parse records embedded with space.
1:2 3: 4:5 (the second field contains "2 3")

awk -F":" '{for (x=1;x<=5;x++){if($x==" "){$x="*"}};print $1,$2,$3,$4,$5}' TrackMsgFile.0806 | while read one two three four five; do
echo "Here's your vars: $one $two $three $four $five"
done

awk -F":" '{for (x=1;x<=5;x++){if($x==" "){$x="*"}};print $1,$2,$3,$4,$5}' TrackMsgFile.0806, returns
1 2 3 4 5
awk -F":" '{for (x=1;x<=5;x++){if($x==" "){$x="*"}};print $1,$2,$3,$4,$5}' TrackMsgFile.0806 | while read one two three four five
is same as
1 2 3 4 5 | while read one two three four five

But since field separator for read command is space " ", the above command takes:
two = 2
three =3

So, still we need some modifications.

GrapefruiTgirl · 10-25-2010, 09:12 AM

I don't really understand all of what you've written in the above post, especially the part about embedded whitespace; embedded whitespace should be irrelevant going into the awk, because the field separator is ":", not " ".

In the future, please use some formatting in your posts, and code tags ( http://www.phpbb.com/community/faq.php?mode=bbcode#f2r1 ) to show us code and output. Also it would help to show us exactly what your input file contains -- a few example lines would be good.

Now.. If I understand at all post #9, try this:

Code:

IFS=":"
awk -F":" '{OFS=":";for(x=1;x<=5;x++){if($x==" " || $x==""){$x="*"}};print $1,$2,$3,$4,$5}' TrackMsgFile.0806 | while read one two three four five; do
 echo "Here's your vars: '$one' '$two' '$three' '$four' '$five'"
done

If the problem you're describing still persists with that modification, you'll have to please re-explain what the problem is.
Note for demonstration purposes, I put single quotes around the outputted variables, to that any whitespace will be obvious.

barunparichha · 10-25-2010, 09:33 AM

I regret , if u r confused with post #9. There is nothing wrong with awk, but with the code "while read one two three four five". Please consider following example for clarification.
Let's say the input file TrackMsgFile contains a record like:

1:2 3: 4:5

Note, here the second field contains "2 3".
Now if u run the command

Code:

  awk -F":" '{print $1,$2,$3,$4,$5}' TrackMsgFile.0806 | while read one two three four five; do
    echo "two: $two"
done

Result:
two: 2

GrapefruiTgirl · 10-25-2010, 09:35 AM

OK, the use of $IFS in post #10 should fix that. Did you try it? Please tell me if that addresses this issue.

barunparichha · 10-25-2010, 09:58 AM

For record :
1:2 3: 4:5

Code:

 awk -F":" '{IFS=":"; print $1,$2,$3,$4,$5}' TrackMsgFile.0806 | while read one two three four five; do
    echo "two: $two"
 done

shows:
two: 2

Hence incorrect.

GrapefruiTgirl · 10-25-2010, 10:02 AM

Re-read the code in post #10 carefully- what you just executed is not according to that code. IFS is a bash thing, not for use inside the AWK. OFS is for use in AWK.

And, notice I set IFS=":" before running the code.

barunparichha · 10-25-2010, 10:19 AM

Consider the input file TestFile contains four records:
TestFile:
1::2:3:4:5
4:5:6:7:8
7:8:9:10
7: 8 9:1:2:3

Code:

awk -F":" '{OFS=":"; print $1, $2, $3, $4, $5}'' TestFile 

Returns :
1::2:3:4
4:5:6:7:8
7:8:9:10:
7: 8 9:1:2:3
::::

Therefore,

Code:

awk -F":" '{OFS=":";print $1,$2,$3,$4,$5}' TestFile | while read one two three four five; do echo "two : $two"; done

Returns
two :
two :
two :
two : 8
two :

So using "OFS" and "IFS" will not help. Using array with awk, might help.