[SOLVED] Another newb Awk Q

Linux_Kidd · 04-18-2012, 12:34 PM

does regex "^" (begins with directive) still apply inside when ~ is being used, or is there a replacement for ~ that means "begins with"?

if ( NF == 0 || $0 ~ /^(xxx)|(yyy)|(zzz)|(aaa)|(^)/ ) {}

druuna · 04-18-2012, 12:54 PM

Hi,

Quote:

Originally Posted by Linux_Kidd

does regex "^" (begins with directive) still apply inside when ~ is being used, or is there a replacement for ~ that means "begins with"?

Yes, it does.

Quote:

if ( NF == 0 || $0 ~ /^(xxx)|(yyy)|(zzz)|(aaa)|(^)/ ) {}

I'm not sure about this command though. Have a look at this:

Code:

if ( NF == 0 || $0 ~ /^(xxx|yyy|zzz|aaa)/ ) { }

I removed the (^) part (why is it there?) and rewrote the way words are OR-ed (blue part).

Hope this helps.

Linux_Kidd · 04-18-2012, 12:59 PM

i'll try that.
i believe my regex was taking (^) as a literal, need to verify

druuna · 04-18-2012, 01:05 PM

Hi,

Quote:

Originally Posted by Linux_Kidd

i believe my regex was taking (^) as a literal, need to verify

If that is true then you need to escape it, otherwise it will be seen as at the beginning of the string and consequently all lines in the file are shown.

With looking for a literal ^ also included, the command becomes:

Code:

if ( NF == 0 || $0 ~ /^(xxx|yyy|zzz|aaa|\^)/ ) { }

Hope this helps.

Linux_Kidd · 04-18-2012, 01:37 PM

how to get remainder of fields into a variable?

for (i=7; i<=NF; i++) {
last = $last + $i }
print $1,$2,last

thnx

druuna · 04-18-2012, 02:01 PM

Hi,

Have a look at this example:

Code:

for ( i=1 ; i<=NF ; i++ ) { printf("%s ", $i) } { print "no more fields" }

Example run:

Code:

$ cat infile
1
1 2
1 2 3
1 2 3 4
1 2 3 4 5
$ awk '{ for ( i=1 ; i<=NF ; i++ ) { printf("%s ", $i) } { print "no more fields" }}' infile 
1 no more fields
1 2 no more fields
1 2 3 no more fields
1 2 3 4 no more fields
1 2 3 4 5 no more fields

Info about printf: Examples Using printf (gnu.org)

Hope this helps.

grail · 04-18-2012, 02:14 PM

If I am following the last example I assume you wish to concatenate all remaining fields into the variable last?
If so then simply place the FS between items, assuming you still wish to use it, as + is for arithmetic:

Code:

for (i=7; i<=NF; i++)
    last = last FS $i

print $1,$2,last

Also note that last is a string variable and not a field identifier so the $ is not needed.

If I am on the wrong path you can ignore me

Linux_Kidd · 04-18-2012, 02:17 PM

ok, my skills withing awk script of very rusty....

from within awk script, strip out the zero from field #1 when printing $1,$2,$3,$4,$5

0ABCD XXXX YYYY DDDD FFFF

grail · 04-18-2012, 02:32 PM

I am not sure I understand the difficulty?

Code:

awk 'sub(/0/,"",$1)' file

Linux_Kidd · 04-18-2012, 02:50 PM

i do something like this, was looking to make it a tad more elegant

Code:

#!/bin/bash
awk '
BEGIN {
OFS=",";
}
{
        if ( NF == 0 || $1 ~ /^(aaa|bbb|-|+|=|0$|1\/|\/\/|TPP|PASSWORD|DYNAMIC|1E)/ ) {}
        else {
        last = $6$7$8$9$10$11$12$13
        print $1,$2,$3,$4,$5,last;}
} ' | sed 's/^0\(.*\)/\1/'

i execute like this
./script.awk < infile > outfile

i also need to add regex for mm/dd/yy in the If statement, but having a hard time as \d{2}/\d{2}/\d{2} fails
should i use [:digit:]\{2\}\/[:digit:]\{2\}\/[:digit:]\{2\}

grail · 04-18-2012, 03:08 PM

Well I'll answer the last part first, unless using version 4 or greater of gawk you must use --re-interval as a switch to use {}.

Obviously the sed is a complete waste as shown previously to remove a zero.

As the truth of the if does nothing I would suggest simply negating the current test.

Code:

if ( NF && $1 !~ /^(aaa|bbb|-|+|=|0$|1\/|\/\/|TPP|PASSWORD|DYNAMIC|1E)/ )

And as for the concatenation, is the fifth field unique or the first of its kind? if so, you could do something like:

Code:

#if first of its kind
last = gensub(".*"$5,"","1"); gsub(/ /,"",last)

#if unique you can do the above all together
last = gensub(".*"$5"| ","","g")

Linux_Kidd · 04-18-2012, 03:59 PM

Quote:

Originally Posted by grail

If I am following the last example I assume you wish to concatenate all remaining fields into the variable last?
If so then simply place the FS between items, assuming you still wish to use it, as + is for arithmetic:

Code:

for (i=7; i<=NF; i++)
    last = last FS $i

print $1,$2,last

Also note that last is a string variable and not a field identifier so the $ is not needed.

If I am on the wrong path you can ignore me

all helps a lot. i dont need FS in between, i need a space.

druuna · 04-19-2012, 12:29 AM

Hi,

Quote:

Originally Posted by Linux_Kidd

all helps a lot.

You're welcome.

Quote:

i dont need FS in between, i need a space.

The default FS is a space. Using FS instead of a real space make the script more flexible. If you ever need to change the FS to something other then a space, you only need to change it once and do not have to search the complete script for relevant spaces that need changing.

When you create a small awk script (or one-liner) this isn't too relevant, but it is good practise nonetheless.

Hope this clears things up.

Linux_Kidd · 04-19-2012, 04:54 AM

Quote:

Originally Posted by druuna

Hi,
You're welcome.

The default FS is a space. Using FS instead of a real space make the script more flexible. If you ever need to change the FS to something other then a space, you only need to change it once and do not have to search the complete script for relevant spaces that need changing.

When you create a small awk script (or one-liner) this isn't too relevant, but it is good practise nonetheless.

Hope this clears things up.

my bad, i thought i defined FS, but it was OFS. i will try with FS in there and test. too many things going on right now.

this small script processes a bunch of txt files, spits it out pipe delimited for Excel. script took 3sec to process the files. the files were previously processed manually by a person, that person estimated he spent approx 40 man hrs to process the files !!!

grail · 04-19-2012, 06:50 AM

You don't happen to work in a bank by chance? I had a very similar experience where a process could only be run over the weekend as it took almost 40hrs to complete a single run. After I had been
there 3 months we were able to run it adhoc whenever we liked as it took about 3 minutes