LinuxQuestions.org
Go Job Hunting at the LQ Job Marketplace
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices

Reply
 
LinkBack Search this Thread
Old 07-01-2009, 11:37 AM   #1
emmalg
Member
 
Registered: Jun 2009
Location: Spain
Distribution: Various, Ubuntu, Fedora, Open Solaris, Solaris, RHEL, CentOS
Posts: 64

Rep: Reputation: 16
variable substitution in awk statement


Hi All

I have an odd problem using awk (on Ubuntu 9.04 which is apparently mawk an improved awk).

The following statement works and prints out the line when 3 is in column two of file.txt:

Code:
out=`awk '$2==3 {print $0}' old2new.txt`
Where old2new.txt is like this (single space separated):
new_val old_val
1 0
2 1
3 2
4 3 ....

However, what I want to do is:
Code:
out=`awk '$2==$input_var {print $0}' old2new.txt`
but the variable isn't substituted.

The full usage of this I want is:

Code:
new_val=`awk '$2==$old_val {print $0}' file.txt | cut -d" " -f1`
Do the quotes or awk itself stop the variable substitution occuring? How could I make this work properly?

Many thanks in advance for the advice.

Emma
 
Old 07-01-2009, 01:10 PM   #2
catkin
LQ 5k Club
 
Registered: Dec 2008
Location: Tamil Nadu, India
Distribution: Servers: Debian Squeeze and Wheezy. Desktop: Slackware64 14.0. Netbook: Slackware 13.37
Posts: 8,516
Blog Entries: 27

Rep: Reputation: 1174Reputation: 1174Reputation: 1174Reputation: 1174Reputation: 1174Reputation: 1174Reputation: 1174Reputation: 1174Reputation: 1174
Hello Emma :-)

Which variable isn't substituted from where?

Assuming (always dangerous!) that $input_var has a value in the shell you are issuing the command from then you want
Code:
out=`awk '$2=='$input_var' {print $0}' old2new.txt`
You are right in suspecting a quoting problem. When the shell you are issuning the command from constructs the command line in back quotes, it leaves everything between single quotes exactly as it is, including $input_var.

In the form suggested above, the shell you are issuing the command from finds $2== in single quotes so leaves it exactly as it is then finds $input_var outside single quotes so substitutes its value.

Welcome to the subtle world of shell quoting!

You can make life a little easier for yourself by using $( <your command> ) rather than `<your command>`

Best

Charles
 
Old 07-01-2009, 01:12 PM   #3
David the H.
Bash Guru
 
Registered: Jun 2004
Location: Osaka, Japan
Distribution: Debian sid + kde 3.5 & 4.4
Posts: 6,823

Rep: Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946
Are you talking about using a bash variable inside of awk?

The single quotes around the awk expression keep the variable from expanding. Everything within them will be passed to awk as-is. To pass the contents of a bash variable to awk you need to "unquote" it.

(Think of quoting in bash as a toggle switch rather than a container. They turn special character interpretation on and off.)

Assuming that $old_val is the bash variable you want to use:

Code:
new_val=$(awk '$2=='$old_val' {print $0}' file.txt | cut -d" " -f1)
This should work, but I haven't tested it.

BTW, You can also set an awk variable to the value of a bash variable with the command option -v before entering the main expressions (you do know that awk has it's own variable system, don't you? ).

Code:
awk -v awkvar="$bashvar" '{awk expression}'
edit: what catkin said.

Last edited by David the H.; 07-01-2009 at 01:25 PM. Reason: further edit: re-worded for clarity
 
Old 07-02-2009, 03:38 AM   #4
emmalg
Member
 
Registered: Jun 2009
Location: Spain
Distribution: Various, Ubuntu, Fedora, Open Solaris, Solaris, RHEL, CentOS
Posts: 64

Original Poster
Rep: Reputation: 16
Thank you very much! I was sure there was some subtlety I was missing and I have to be honest I find the awk man page rather impenetrable but think awk is brilliant.

I also wasn't aware of the separate variables, as I've never used awk for anything terribly complicated before! In fact, I've only ever used it to extract columns of data from large files.

It's for a rather huge shell script I've been working on which splits a mysql database based on certain parameters and basically I need to use a look up table to make sure that each of the secondary keys in the new databases are correctly referenced.

Why shell?! I was given a huge perl script which didn't work and thought shell would be the easiest way for me to make sure I could re-write it correctly as I can test each command - that and I was too lazy to do it in C!

Thanks again for your help, have a great day!
 
Old 07-02-2009, 03:39 AM   #5
emmalg
Member
 
Registered: Jun 2009
Location: Spain
Distribution: Various, Ubuntu, Fedora, Open Solaris, Solaris, RHEL, CentOS
Posts: 64

Original Poster
Rep: Reputation: 16
Thumbs up

And I've just realised how silly I was being printing the whole line and cutting!

Edit...

I didn't get on with the variable substitution, after setting b_fId=3, I tried:
Code:
$ n_fId=$(awk -v old_val="$b_fId" '$2==$old_val {print $0}' old2new_fid.txt)
$ echo $n_fId
I think I'll have to find a bit of time to read up on how that works properly!

But this worked perfectly:
Code:
$ n_fId=$(awk '$2=='$b_fId' {print $1}' old2new_fid.txt)
$ echo $n_fId
3

Last edited by emmalg; 07-02-2009 at 04:09 AM. Reason: Addition
 
Old 07-02-2009, 04:00 AM   #6
H_TeXMeX_H
Guru
 
Registered: Oct 2005
Location: $RANDOM
Distribution: slackware64
Posts: 12,928
Blog Entries: 2

Rep: Reputation: 1266Reputation: 1266Reputation: 1266Reputation: 1266Reputation: 1266Reputation: 1266Reputation: 1266Reputation: 1266Reputation: 1266
Happy birthday David the H. just noticed it.

Anyway, you can also use environment variables in awk by using:

Code:
awk '{ print ENVIRON["PATH"] }'
prints the $PATH. But the above ways are usually better than this method.
 
Old 07-02-2009, 04:16 AM   #7
emmalg
Member
 
Registered: Jun 2009
Location: Spain
Distribution: Various, Ubuntu, Fedora, Open Solaris, Solaris, RHEL, CentOS
Posts: 64

Original Poster
Rep: Reputation: 16
I can see a love hate relationship developing between me and awk. I think I'll be buying a book about it one of these days because it is so useful but so difficult! I think I've seen a 2" thick one!

Thanks for the birthday wishes, I didn't know it showed it anywhere! 29 yesterday - I hate the 9s more than the 0s!

Edit -

I just realised what you meant, I registered on 1st June!

Last edited by emmalg; 07-02-2009 at 04:36 AM.
 
Old 07-02-2009, 04:32 AM   #8
H_TeXMeX_H
Guru
 
Registered: Oct 2005
Location: $RANDOM
Distribution: slackware64
Posts: 12,928
Blog Entries: 2

Rep: Reputation: 1266Reputation: 1266Reputation: 1266Reputation: 1266Reputation: 1266Reputation: 1266Reputation: 1266Reputation: 1266Reputation: 1266
Oh, so it's your birthday too ? What a coincidence. Happy Birthday !
 
Old 07-02-2009, 07:35 AM   #9
catkin
LQ 5k Club
 
Registered: Dec 2008
Location: Tamil Nadu, India
Distribution: Servers: Debian Squeeze and Wheezy. Desktop: Slackware64 14.0. Netbook: Slackware 13.37
Posts: 8,516
Blog Entries: 27

Rep: Reputation: 1174Reputation: 1174Reputation: 1174Reputation: 1174Reputation: 1174Reputation: 1174Reputation: 1174Reputation: 1174Reputation: 1174
Quote:
Originally Posted by emmalg View Post
I think I'll be buying a book about it one of these days because it is so useful but so difficult!
Despite its age, the best book for me was "The AWK Programming Language" by Aho, Kernighan and Weinberger known as the little grey book. In some ways it was even a good primer because of the authors' clarity. It has the same feel as K+R's The C Programming Language -- elegantly minimal but complete. If nothing else I grinned every time I saw the authors' names on the cover; you don't have to have a weird name to be a genius computing pioneer but it helps!
 
Old 07-02-2009, 07:53 AM   #10
H_TeXMeX_H
Guru
 
Registered: Oct 2005
Location: $RANDOM
Distribution: slackware64
Posts: 12,928
Blog Entries: 2

Rep: Reputation: 1266Reputation: 1266Reputation: 1266Reputation: 1266Reputation: 1266Reputation: 1266Reputation: 1266Reputation: 1266Reputation: 1266
There's a good awk tutorial here, along with others:
http://www.grymoire.com/Unix/
 
Old 07-02-2009, 08:16 AM   #11
David the H.
Bash Guru
 
Registered: Jun 2004
Location: Osaka, Japan
Distribution: Debian sid + kde 3.5 & 4.4
Posts: 6,823

Rep: Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946
Quote:
Originally Posted by H_TeXMeX_H View Post
Happy birthday David the H. just noticed it.
Aww, shucks. Someone noticed. I'm touched (and not just in the head).

I recommend the grymoire too. In fact, I only just finished reading through it myself, so I'm kind of an awk newbie too.

Do beware though that the articles are pretty old and there are a lot of typos. They're also mostly focused on old unix versions of the commands; some of the stuff won't apply perfectly to the gnu ones we use. They make good primers though, if you can avoid the hazards.
 
Old 07-02-2009, 08:37 AM   #12
emmalg
Member
 
Registered: Jun 2009
Location: Spain
Distribution: Various, Ubuntu, Fedora, Open Solaris, Solaris, RHEL, CentOS
Posts: 64

Original Poster
Rep: Reputation: 16
Brilliant, I really appreciate the recommendation too.

I have an update, I couldn't get the lines I said did work to work in the script, only on the command line.

Now I have something that works in the script but couldn't have done it without you all!

The solution is:
Code:
n_fId=$(awk '{if ($2==old_val) print $1}' old_val=$b_fId old2new_fid.txt)
Thank you all very much!

Now I get the birthday thing - I'm a bit slow to catch up sometimes! Happy birthday!! July babies are best

Last edited by emmalg; 07-02-2009 at 08:40 AM. Reason: addition
 
Old 07-02-2009, 08:39 AM   #13
David the H.
Bash Guru
 
Registered: Jun 2004
Location: Osaka, Japan
Distribution: Debian sid + kde 3.5 & 4.4
Posts: 6,823

Rep: Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946Reputation: 1946
Quote:
Originally Posted by emmalg View Post
I didn't get on with the variable substitution, after setting b_fId=3, I tried:
Code:
$ n_fId=$(awk -v old_val="$b_fId" '$2==$old_val {print $0}' old2new_fid.txt)
$ echo $n_fId
Try removing the $ from the variable ($2==old_val). I'm still not clear on why (like I said, I'm still new myself), but apparently it isn't needed when calling up the value of variables inside the awk expression. At least not when matching.

Edit, from your last post, it looks like you've figured it out yourself.

Edit2: Oh yes, I remember now. From the grymoire:
Quote:
One more point about the use of a dollar sign. In scripting languages like Perl and the various shells, a dollar sign means the word following is the name of the variable. Awk is different. The dollar sign means that we are refering to a field or column in the current line. When switching between Perl and AWK you must remener that "$" has a different meaning. So the following piece of code prints two "fields" to standard out. The first field printed is the number "5", the second is the fifth field (or column) on the input line.

BEGIN { x=5 }
{ print x, $x}

Last edited by David the H.; 07-02-2009 at 08:48 AM.
 
  


Reply

Tags
awk, bash, shell, substitution, variable


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off
Trackbacks are Off
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
bash asterisk variable substitution bt101 Linux - Software 3 02-08-2009 11:37 PM
Text substitution and processing with sed and awk shanecraddock@gmail.com Linux - Newbie 1 12-18-2008 11:34 AM
Trying to combine lookahead and variable substitution djeepp Programming 1 09-03-2008 04:32 PM
variable substitution in sed gaynut Programming 1 07-14-2008 07:38 AM
Bash variable substitution daYz Programming 3 04-14-2006 01:16 PM


All times are GMT -5. The time now is 05:27 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration