[SOLVED] variable substitution in awk statement

emmalg · 07-01-2009, 12:37 PM

Hi All

I have an odd problem using awk (on Ubuntu 9.04 which is apparently mawk an improved awk).

The following statement works and prints out the line when 3 is in column two of file.txt:

Code:

out=`awk '$2==3 {print $0}' old2new.txt`

Where old2new.txt is like this (single space separated):
new_val old_val
1 0
2 1
3 2
4 3 ....

However, what I want to do is:

Code:

out=`awk '$2==$input_var {print $0}' old2new.txt`

but the variable isn't substituted.

The full usage of this I want is:

Code:

new_val=`awk '$2==$old_val {print $0}' file.txt | cut -d" " -f1`

Do the quotes or awk itself stop the variable substitution occuring? How could I make this work properly?

Many thanks in advance for the advice.

Emma

catkin · 07-01-2009, 02:10 PM

Hello Emma :-)

Which variable isn't substituted from where?

Assuming (always dangerous!) that $input_var has a value in the shell you are issuing the command from then you want

Code:

out=`awk '$2=='$input_var' {print $0}' old2new.txt`

You are right in suspecting a quoting problem. When the shell you are issuning the command from constructs the command line in back quotes, it leaves everything between single quotes exactly as it is, including $input_var.

In the form suggested above, the shell you are issuing the command from finds $2== in single quotes so leaves it exactly as it is then finds $input_var outside single quotes so substitutes its value.

Welcome to the subtle world of shell quoting!

You can make life a little easier for yourself by using $( <your command> ) rather than `<your command>`

Best

Charles

David the H. · 07-01-2009, 02:12 PM

Are you talking about using a bash variable inside of awk?

The single quotes around the awk expression keep the variable from expanding. Everything within them will be passed to awk as-is. To pass the contents of a bash variable to awk you need to "unquote" it.

(Think of quoting in bash as a toggle switch rather than a container. They turn special character interpretation on and off.)

Assuming that $old_val is the bash variable you want to use:

Code:

new_val=$(awk '$2=='$old_val' {print $0}' file.txt | cut -d" " -f1)

This should work, but I haven't tested it.

BTW, You can also set an awk variable to the value of a bash variable with the command option -v before entering the main expressions (you do know that awk has it's own variable system, don't you?

).

Code:

awk -v awkvar="$bashvar" '{awk expression}'

edit: what catkin said.

emmalg · 07-02-2009, 04:38 AM

Thank you very much! I was sure there was some subtlety I was missing and I have to be honest I find the awk man page rather impenetrable but think awk is brilliant.

I also wasn't aware of the separate variables, as I've never used awk for anything terribly complicated before! In fact, I've only ever used it to extract columns of data from large files.

It's for a rather huge shell script I've been working on which splits a mysql database based on certain parameters and basically I need to use a look up table to make sure that each of the secondary keys in the new databases are correctly referenced.

Why shell?! I was given a huge perl script which didn't work and thought shell would be the easiest way for me to make sure I could re-write it correctly as I can test each command - that and I was too lazy to do it in C!

Thanks again for your help, have a great day!

emmalg · 07-02-2009, 04:39 AM

And I've just realised how silly I was being printing the whole line and cutting!

Edit...

I didn't get on with the variable substitution, after setting b_fId=3, I tried:

Code:

$ n_fId=$(awk -v old_val="$b_fId" '$2==$old_val {print $0}' old2new_fid.txt)
$ echo $n_fId

I think I'll have to find a bit of time to read up on how that works properly!

But this worked perfectly:

Code:

$ n_fId=$(awk '$2=='$b_fId' {print $1}' old2new_fid.txt)
$ echo $n_fId
3

H_TeXMeX_H · 07-02-2009, 05:00 AM

Happy birthday David the H. just noticed it.

Anyway, you can also use environment variables in awk by using:

Code:

awk '{ print ENVIRON["PATH"] }'

prints the $PATH. But the above ways are usually better than this method.

emmalg · 07-02-2009, 05:16 AM

I can see a love hate relationship developing between me and awk. I think I'll be buying a book about it one of these days because it is so useful but so difficult! I think I've seen a 2" thick one!

Thanks for the birthday wishes, I didn't know it showed it anywhere! 29 yesterday - I hate the 9s more than the 0s!

Edit -

I just realised what you meant, I registered on 1st June!

H_TeXMeX_H · 07-02-2009, 05:32 AM

Oh, so it's your birthday too ? What a coincidence. Happy Birthday !

catkin · 07-02-2009, 08:35 AM

Quote:

Originally Posted by emmalg

I think I'll be buying a book about it one of these days because it is so useful but so difficult!

Despite its age, the best book for me was "The AWK Programming Language" by Aho, Kernighan and Weinberger known as the little grey book. In some ways it was even a good primer because of the authors' clarity. It has the same feel as K+R's The C Programming Language -- elegantly minimal but complete. If nothing else I grinned every time I saw the authors' names on the cover; you don't have to have a weird name to be a genius computing pioneer but it helps!

H_TeXMeX_H · 07-02-2009, 08:53 AM

There's a good awk tutorial here, along with others:
http://www.grymoire.com/Unix/

David the H. · 07-02-2009, 09:16 AM

Quote:

Originally Posted by H_TeXMeX_H

Happy birthday David the H. just noticed it.

Aww, shucks. Someone noticed. I'm touched (and not just in the head).

I recommend the grymoire too. In fact, I only just finished reading through it myself, so I'm kind of an awk newbie too.

Do beware though that the articles are pretty old and there are a lot of typos. They're also mostly focused on old unix versions of the commands; some of the stuff won't apply perfectly to the gnu ones we use. They make good primers though, if you can avoid the hazards.

emmalg · 07-02-2009, 09:37 AM

Brilliant, I really appreciate the recommendation too.

I have an update, I couldn't get the lines I said did work to work in the script, only on the command line.

Now I have something that works in the script but couldn't have done it without you all!

The solution is:

Code:

n_fId=$(awk '{if ($2==old_val) print $1}' old_val=$b_fId old2new_fid.txt)

Thank you all very much!

Now I get the birthday thing - I'm a bit slow to catch up sometimes! Happy birthday!! July babies are best

David the H. · 07-02-2009, 09:39 AM

Quote:

Originally Posted by emmalg

I didn't get on with the variable substitution, after setting b_fId=3, I tried:

Code:

$ n_fId=$(awk -v old_val="$b_fId" '$2==$old_val {print $0}' old2new_fid.txt)
$ echo $n_fId

Try removing the $ from the variable ($2==old_val). I'm still not clear on why (like I said, I'm still new myself), but apparently it isn't needed when calling up the value of variables inside the awk expression. At least not when matching.

Edit, from your last post, it looks like you've figured it out yourself.

Edit2: Oh yes, I remember now. From the grymoire:

Quote:

One more point about the use of a dollar sign. In scripting languages like Perl and the various shells, a dollar sign means the word following is the name of the variable. Awk is different. The dollar sign means that we are refering to a field or column in the current line. When switching between Perl and AWK you must remener that "$" has a different meaning. So the following piece of code prints two "fields" to standard out. The first field printed is the number "5", the second is the fifth field (or column) on the input line.

BEGIN { x=5 }
{ print x, $x}