Awk to Count Multiple patterns in a huge file

reach.sree@gmail.com · 06-07-2012, 05:50 PM

danielbmartin · 06-07-2012, 06:29 PM

Quote:

Originally Posted by reach.sree@gmail.com

Code:

countmap=`grep $SITEID $FILENAME | grep MAP | wc -l`
countmodal=`grep $SITEID $FILENAME | grep MODAL | wc -l`
countlink=`grep $SITEID $FILENAME | grep LINK | wc -l`
echo $SITEID\|$countmap\|$countmodal\|$countlink\|

How many different sites are found in your input file? Your example shows only two, but are there actually 20? If 20, are you making 60 passes through the file? (20 to count MAPs, 20 to count MODALs, and 20 to count LINKs?)

Daniel B. Martin

reach.sree@gmail.com · 06-07-2012, 06:32 PM

I have 10000 sites in my files. And yes grep is a very inefficient way of doing this

bigearsbilly · 06-07-2012, 07:13 PM

Code:

#!/usr/bin/perl 
use strict;

my %sites;
while(<>) {
	chomp;
	my (undef, $site, $thing) = split '\|';
	# print qq(undef, $site, $thing\n);
	$sites{$site}->{$thing}++;

}
$\ = "\n";

print "SiteName MAP MODAL LINK";
print "-" x 40;
while (my ($k, $v) = each (%sites)) {
        print "$k|$v->{MAP}|$v->{MODAL}|$v->{LINK}";
}

danielbmartin · 06-07-2012, 08:15 PM

Quote:

Originally Posted by reach.sree@gmail.com

I have 10000 sites in my files. And yes grep is a very inefficient way of doing this

Code:

Input file ...
|site01|MAP
|site02|MAP
|site01|MODAL
|site02|MAP
|site02|MODAL
|site02|LINK
|site01|LINK
|site11|MAP
|site12|MAP
|site11|MODAL
|site12|MAP
|site12|MODAL
|site12|LINK
|site11|LINK
|site04|MAP
|site05|MAP
|site06|MODAL
|site07|MAP
|site08|MODAL
|site08|LINK
|site08|LINK
|site12|MODAL
|site12|LINK
|site12|MODAL
|site12|LINK
|site12|MODAL
|site12|LINK

Code:

Run this ...
sort $InFile | uniq --count

Code:

Get this ...
      1 |site01|LINK
      1 |site01|MAP
      1 |site01|MODAL
      1 |site02|LINK
      2 |site02|MAP
      1 |site02|MODAL
      1 |site04|MAP
      1 |site05|MAP
      1 |site06|MODAL
      1 |site07|MAP
      2 |site08|LINK
      1 |site08|MODAL
      1 |site11|LINK
      1 |site11|MAP
      1 |site11|MODAL
      4 |site12|LINK
      2 |site12|MAP
      4 |site12|MODAL

Try this code on your big real-world input file. Let us know how long it took to execute.

Daniel B. Martin

reach.sree@gmail.com · 06-08-2012, 11:38 AM

Thanks a lot guys. I went with

awk -F"|" '{c[$2,$3]++;b[$2]=$2 FS 0+c[$2,"MAP"] FS 0+c[$2,"MODAL"] FS 0+c[$2,"LINK"] FS} END {for ( i in b) { print b[i]}}' filename

and then did a sort filename

It took me about 40 sec for the 430K file.

Thanks once again for your input.

danielbmartin · 06-08-2012, 01:04 PM

Quote:

Originally Posted by reach.sree@gmail.com

It took me about 40 sec for the 430K file.

I have no objection to your choosing the awk instead of my proposed solution. You benefit by having code provided, promptly and at no cost. You may repay the LQ community by publishing feedback.

Tell us what worked and what didn't. Tell us which code ran fast and which ran even faster. Tell us which solution you chose and why.

Daniel B. Martin