LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices

Reply
 
Search this Thread
Old 10-25-2010, 06:34 AM   #1
barunparichha
Member
 
Registered: Jun 2006
Location: Bangalore,india
Distribution: Linux(Redhat,fedora,suse,ubantu), Solaris (s8/s9/s10/nevada/open-solaris)
Posts: 283

Rep: Reputation: 32
Reading files in shell scripting


Hi,
I am trying to read the fields of a file and manipulate them, record by record.

Lets say using awk :
awk -F":" `{print $1 $2 $3 $4 $5}' TrackMsgFile.0806`

This prints my fields on screen.But I dont want to print these fields while reading the records instead store them in some variable and manipulate them as per my logic.

Does "awk" or some other shell command provides something for this ?
 
Old 10-25-2010, 06:36 AM   #2
GrapefruiTgirl
Guru
 
Registered: Dec 2006
Location: underground
Distribution: Slackware64
Posts: 7,594

Rep: Reputation: 550Reputation: 550Reputation: 550Reputation: 550Reputation: 550Reputation: 550
Assign the results to a shell variable:
Code:
var=$(awk -F":" '{print $1 $2 $3 $4 $5}' TrackMsgFile.0806)

echo "$var"
 
Old 10-25-2010, 06:40 AM   #3
GrapefruiTgirl
Guru
 
Registered: Dec 2006
Location: underground
Distribution: Slackware64
Posts: 7,594

Rep: Reputation: 550Reputation: 550Reputation: 550Reputation: 550Reputation: 550Reputation: 550
If you want each field as a separate var:
Code:
awk -F":" '{print $1,$2,$3,$4,$5}' TrackMsgFile.0806 | while read one two three four five; do
    echo "Here's your vars: $one $two $three $four $five"
done
Note the commas separating the output, so the variables aren't all jumbled together in one.

Last edited by GrapefruiTgirl; 10-25-2010 at 06:41 AM.
 
Old 10-25-2010, 07:01 AM   #4
ghostdog74
Senior Member
 
Registered: Aug 2006
Posts: 2,696
Blog Entries: 5

Rep: Reputation: 241Reputation: 241Reputation: 241
Quote:
Originally Posted by GrapefruiTgirl View Post
If you want each field as a separate var:
Code:
awk -F":" '{print $1,$2,$3,$4,$5}' TrackMsgFile.0806 | while read one two three four five; do
    echo "Here's your vars: $one $two $three $four $five"
done
Note the commas separating the output, so the variables aren't all jumbled together in one.
but the whole operation of awk+while is still iterating the file and processing them as it iterates. Therefore, the while loop really is redundant.
 
Old 10-25-2010, 07:03 AM   #5
ghostdog74
Senior Member
 
Registered: Aug 2006
Posts: 2,696
Blog Entries: 5

Rep: Reputation: 241Reputation: 241Reputation: 241
@OP, you might want to define clearly what you want to do. If you want to store your lines in some variable for LATER use, use and array

Code:
awk '{array[++d]=$0}
{ 
 ....
 # some other processing here.
} 
END{
  # here , you process your array
  for(i=1;i<=d;i++){
    print "do something with " array[i]
  }
}'
 
Old 10-25-2010, 07:10 AM   #6
fbobraga
Member
 
Registered: Jul 2010
Location: São Paulo - Brasil
Distribution: Debian 7 / Crunchbang 11
Posts: 229

Rep: Reputation: 41
Quote:
Originally Posted by barunparichha View Post
Does "awk" or some other shell command provides something for this ?
Or you can use a awk program to manipulate fields:

Code:
awk -F":" `{<do some stuff with $1-$5> ;print $1 $2 $3 $4 $5}' TrackMsgFile.0806`
there is a way of put the awk program externally, on a text file (prog.awk in the following example), with the -f switch:
Code:
awk -F":" -f prog.awk TrackMsgFile.0806`


see http://en.wikipedia.org/wiki/Awk
 
Old 10-25-2010, 08:26 AM   #7
barunparichha
Member
 
Registered: Jun 2006
Location: Bangalore,india
Distribution: Linux(Redhat,fedora,suse,ubantu), Solaris (s8/s9/s10/nevada/open-solaris)
Posts: 283

Original Poster
Rep: Reputation: 32
But lets say one record exists like
1::3::5

Here, I have omitted 2, 4 for explanation.
awk -F":" '{print $1,$2,$3,$4,$5}' TrackMsgFile.0806 | while read one two three four five;

3 will be stored for variable two, 5 for three.

Reason:
awk -F":" '{print $1,$2,$3,$4,$5}' TrackMsgFile.0806 , prints
1 3 5 (as 2nd, 4th filed are blank returns space character)

1 3 5 |while read one two three four five, takes

3,5 for variables two and three respectively.

=>
Is there a mechanism to avoid this ?


Likewise if it will show, wrong result for records like
1:2 3: 4:5 (the second filed contains "2 3") as we are separatinf fields using ":" as field separator)

Last edited by barunparichha; 10-25-2010 at 08:33 AM.
 
Old 10-25-2010, 08:38 AM   #8
GrapefruiTgirl
Guru
 
Registered: Dec 2006
Location: underground
Distribution: Slackware64
Posts: 7,594

Rep: Reputation: 550Reputation: 550Reputation: 550Reputation: 550Reputation: 550Reputation: 550
Code:
awk -F":" '{for (x=1;x<=5;x++){if($x==" "){$x="*"}};print $1,$2,$3,$4,$5}' TrackMsgFile.0806 | while read one two three four five; do
    echo "Here's your vars: $one $two $three $four $five"
done
Technically I'd have thought a space character would count as a character to be read into the appropriate variable, but for whatever reason (maybe multiple spaces count as one?) that doesn't happen.

There are mechanisms however, yes. For example, the above will make an "empty" field (a field that is a single space) return a "*" instead. So, you check your returned variables to see if it's a "*" and if so, then you know it was empty. Maybe a "*" is not the best choice of character to use - a better idea would be to return the word "EMPTY" instead. That would be sensible.
 
Old 10-25-2010, 08:53 AM   #9
barunparichha
Member
 
Registered: Jun 2006
Location: Bangalore,india
Distribution: Linux(Redhat,fedora,suse,ubantu), Solaris (s8/s9/s10/nevada/open-solaris)
Posts: 283

Original Poster
Rep: Reputation: 32
Assuming that, we dont use "*" in any of our records, the above code will work.

But it will still fail to parse records embedded with space.
1:2 3: 4:5 (the second field contains "2 3")

awk -F":" '{for (x=1;x<=5;x++){if($x==" "){$x="*"}};print $1,$2,$3,$4,$5}' TrackMsgFile.0806 | while read one two three four five; do
echo "Here's your vars: $one $two $three $four $five"
done


awk -F":" '{for (x=1;x<=5;x++){if($x==" "){$x="*"}};print $1,$2,$3,$4,$5}' TrackMsgFile.0806, returns
1 2 3 4 5
awk -F":" '{for (x=1;x<=5;x++){if($x==" "){$x="*"}};print $1,$2,$3,$4,$5}' TrackMsgFile.0806 | while read one two three four five
is same as
1 2 3 4 5 | while read one two three four five

But since field separator for read command is space " ", the above command takes:
two = 2
three =3




So, still we need some modifications.

Last edited by barunparichha; 10-25-2010 at 08:55 AM.
 
Old 10-25-2010, 09:12 AM   #10
GrapefruiTgirl
Guru
 
Registered: Dec 2006
Location: underground
Distribution: Slackware64
Posts: 7,594

Rep: Reputation: 550Reputation: 550Reputation: 550Reputation: 550Reputation: 550Reputation: 550
I don't really understand all of what you've written in the above post, especially the part about embedded whitespace; embedded whitespace should be irrelevant going into the awk, because the field separator is ":", not " ".

In the future, please use some formatting in your posts, and code tags ( http://www.phpbb.com/community/faq.php?mode=bbcode#f2r1 ) to show us code and output. Also it would help to show us exactly what your input file contains -- a few example lines would be good.

Now.. If I understand at all post #9, try this:
Code:
IFS=":"
awk -F":" '{OFS=":";for(x=1;x<=5;x++){if($x==" " || $x==""){$x="*"}};print $1,$2,$3,$4,$5}' TrackMsgFile.0806 | while read one two three four five; do
 echo "Here's your vars: '$one' '$two' '$three' '$four' '$five'"
done
If the problem you're describing still persists with that modification, you'll have to please re-explain what the problem is.
Note for demonstration purposes, I put single quotes around the outputted variables, to that any whitespace will be obvious.

Last edited by GrapefruiTgirl; 10-26-2010 at 05:07 AM. Reason: formatting; added check if $x==""
 
Old 10-25-2010, 09:33 AM   #11
barunparichha
Member
 
Registered: Jun 2006
Location: Bangalore,india
Distribution: Linux(Redhat,fedora,suse,ubantu), Solaris (s8/s9/s10/nevada/open-solaris)
Posts: 283

Original Poster
Rep: Reputation: 32
I regret , if u r confused with post #9. There is nothing wrong with awk, but with the code "while read one two three four five". Please consider following example for clarification.
Let's say the input file TrackMsgFile contains a record like:

1:2 3: 4:5

Note, here the second field contains "2 3".
Now if u run the command

Code:
  awk -F":" '{print $1,$2,$3,$4,$5}' TrackMsgFile.0806 | while read one two three four five; do
    echo "two: $two"
done
Result:
two: 2
 
Old 10-25-2010, 09:35 AM   #12
GrapefruiTgirl
Guru
 
Registered: Dec 2006
Location: underground
Distribution: Slackware64
Posts: 7,594

Rep: Reputation: 550Reputation: 550Reputation: 550Reputation: 550Reputation: 550Reputation: 550
OK, the use of $IFS in post #10 should fix that. Did you try it? Please tell me if that addresses this issue.
 
Old 10-25-2010, 09:58 AM   #13
barunparichha
Member
 
Registered: Jun 2006
Location: Bangalore,india
Distribution: Linux(Redhat,fedora,suse,ubantu), Solaris (s8/s9/s10/nevada/open-solaris)
Posts: 283

Original Poster
Rep: Reputation: 32
For record :
1:2 3: 4:5

Code:
 awk -F":" '{IFS=":"; print $1,$2,$3,$4,$5}' TrackMsgFile.0806 | while read one two three four five; do
    echo "two: $two"
 done
shows:
two: 2

Hence incorrect.
 
Old 10-25-2010, 10:02 AM   #14
GrapefruiTgirl
Guru
 
Registered: Dec 2006
Location: underground
Distribution: Slackware64
Posts: 7,594

Rep: Reputation: 550Reputation: 550Reputation: 550Reputation: 550Reputation: 550Reputation: 550
Re-read the code in post #10 carefully- what you just executed is not according to that code. IFS is a bash thing, not for use inside the AWK. OFS is for use in AWK.

And, notice I set IFS=":" before running the code.

Last edited by GrapefruiTgirl; 10-25-2010 at 10:03 AM.
 
Old 10-25-2010, 10:19 AM   #15
barunparichha
Member
 
Registered: Jun 2006
Location: Bangalore,india
Distribution: Linux(Redhat,fedora,suse,ubantu), Solaris (s8/s9/s10/nevada/open-solaris)
Posts: 283

Original Poster
Rep: Reputation: 32
Consider the input file TestFile contains four records:
TestFile:
1::2:3:4:5
4:5:6:7:8
7:8:9:10
7: 8 9:1:2:3

Code:
awk -F":" '{OFS=":"; print $1, $2, $3, $4, $5}'' TestFile 

Returns :
1::2:3:4
4:5:6:7:8
7:8:9:10:
7: 8 9:1:2:3
::::
Therefore,

Code:
awk -F":" '{OFS=":";print $1,$2,$3,$4,$5}' TestFile | while read one two three four five; do echo "two : $two"; done

Returns
two :
two :
two :
two : 8
two :
So using "OFS" and "IFS" will not help. Using array with awk, might help.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] read from keyboard while reading from file in SHELL SCRIPTING m3ll0 Programming 11 10-30-2010 08:30 AM
shell-scripting.pdf files garethsays Linux - Newbie 2 01-30-2009 06:51 AM
Shell scripting,;problem reading from database and exporting to html sunksullen Programming 7 05-18-2007 02:16 PM
Combine files using shell scripting ccray Linux - Newbie 14 01-12-2006 11:14 AM
Reading text from a file using shell scripting. mrobertson Programming 11 06-29-2005 12:12 PM


All times are GMT -5. The time now is 11:57 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration