add a count to sort -u

casperdaghost · 05-31-2012, 10:39 AM

I run this over a log file

Code:

casper@casperbox ~ $ more badthings20120531 | perl -nle 'print /^<br\s\/>(\w+)\s/' | sort -u

casper2011
casper2023
casper2030
casper2031
casper2032
casper2040
casper2041
casper2042

I sort uniqued them out -
Would there be a way to put a wc -l number near each instance to show how many times it showed up.

like it would be great to not only indicate what servers showed up in log file, but how many times they showed up in the log file.

pan64 · 05-31-2012, 12:02 PM

I would solve it in perl, without more, sort and others.

Code:

#!/usr/bin/perl -w
use strict;

die if ! defined $ARGV[0];
my $file = $ARGV[0];

my %sum;

open (F, $file) or die "Error: cannot open file: $file, $!";
while (<F>) {
    next if $_ !~ /^<br\s\/>(\w+)\s/;
    $sum{$1}++;
}
close F or die "Error: cannot read file: $file, $!";

foreach (keys %sum) {
    print "$_, $sum{$_}\n";
}

you should give the filename as argument to it

__________________________________
Happy with solution ... mark as SOLVED
If someone helps you, or you approve of what's posted, click the "Add to Reputation" button, on the left of the post.

grail · 05-31-2012, 12:17 PM

I could probably whip up something in awk or ruby but would need to see the format.

casperdaghost · 06-01-2012, 10:27 AM

i use this alot

Code:

 netstat -an | grep ESTABLISHED | awk '{print $5}' | awk -F: '{print $1}' | sort | uniq -c | awk '{ printf("%s\t%s\t",$2,$1) ; for (i = 0; i < $1; i++) {printf("*")}; print "" }'

i am having trouble applying it to this issue - I don't understand the "
awk '{ printf("%s\t%s\t",$2,$1)"

Code:

walt@walt-nasdaq:~$ cat /home/walt/noo | sed -e 's/pats/casper/g'
<br />casper0053 19:40:08 init: process \'/etc/rc.d/rc.start_java 2>&1 | logger\' (pid 532) exited. Scheduling for restart.
<br />casper0053 19:40:08 init: starting pid 1339, tty \'/dev/null\': \'/etc/rc.d/rc.start_java 2>&1 | logger\'
<br />casper0038 19:40:09 kernel: e1000e: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
<br />casper0054 19:40:09 init: process \'/etc/rc.d/rc.start_java 2>&1 | logger\' (pid 890) exited. Scheduling for restart.
<br />casper0054 netstat -an | grep ESTABLISHED | awk '{print $5}' | awk -F: '{print $1}' | sort | uniq -c | awk '{ printf("%s\t%s\t",$2,$1) ; for (i = 0; i < $1; i++) {printf("*")}; print "" }'
 19:40:09 init: starting pid 1606, tty \'/dev/null\': \'/etc/rc.d/rc.start_java 2>&1 | logger\'
<br />casper0048 19:40:10 kernel: e1000e: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
<br />casper0055 19:40:11 init: process \'/etc/rc.d/rc.start_java 2>&1 | logger\' (pid 533) exited. Scheduling for restart.
<br />casper0055 19:40:11 init: starting pid 815, tty \'/dev/null\': \'/etc/rc.d/rc.start_java 2>&1 | logger\'
<br />casper0052 19:40:12 kernel: e1000e: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
<br />casper0059 19:40:12 init: process \'/etc/rc.d/rc.start_java 2>&1 | logger\' (pid 10090) exited. Scheduling for restart.
<br />casper0059 19:40:12 init: starting pid 10773, tty \'/dev/null\': \'/etc/rc.d/rc.start_java 2>&1 | logger\'
<br />casper0063 19:40:14 init: process \'/etc/rc.d/rc.start_java 2>&1 | logger\' (pid 23802) exited. Scheduling for restart.
<br />casper0063 19:40:14 init: starting pid 24329, tty \'/dev/null\': \'/etc/rc.d/rc.start_java 2>&1 | logger\'
<br />casper0053 19:40:14 kernel: e1000e: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
<br />casper0068 19:40:15 init: process \'/etc/rc.d/rc.start_java 2>&1 | logger\' (pid 3046) exited. Scheduling for restart.
<br />casper0068 19:40:15 init: starting pid 3583, tty \'/dev/null\': \'/etc/rc.d/rc.start_java 2>&1 | logger\'
<br />casper0054 19:40:16 kernel: e1000e: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
<br />casper0055 19:40:16 kernel: e1000e: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
<br />casper0070 19:40:17 init: process \'/etc/rc.d/rc.start_java 2>&1 | logger\' (pid 532) exited. Scheduling for restart.
<br />casper0070 19:40:17 init: starting pid 802, tty \'/dev/null\': \'/etc/rc.d/rc.start_java 2>&1 | logger\'

schneidz · 06-01-2012, 10:57 AM

does

Code:

sort | uniq -c

help ?

casperdaghost · 06-01-2012, 11:05 AM

oh snap schneidz - that does help - BIGTIME EASY

how can I sort by number and then just take the top ten avec bash?

schneidz · 06-01-2012, 12:36 PM

Quote:

Originally Posted by casperdaghost

oh snap schneidz - that does help - BIGTIME EASY

how can I sort by number and then just take the top ten avec bash?

Code:

sort | uniq -c | sort -n -r | head

worx, but theres gotta' be a more efficient way.

grail · 06-01-2012, 12:55 PM

Well firstly I would say that grep and multiple awks are never the way to go:

Code:

netstat -an | awk 'BEGIN{FS="[ \t:]+";OFS="*"}/ESTAB/{a[$(NF-2)]++}END{asorti(a,b);while(++i < length(b)){$0="";$(a[b[i]]+1)="";print b[i]"\t"a[b[i]]"\t"$0}}'

I am not sure how the rest relates to your question though?

pan64 · 06-02-2012, 03:31 AM

what about the perl script I gave you? This should work and also easily can be extended to sort and display the first ten:

Code:

#!/usr/bin/perl -w
use strict;

die if ! defined $ARGV[0];
my $file = $ARGV[0];

my %sum;

open (F, $file) or die "Error: cannot open file: $file, $!";
while (<F>) {
    next if $_ !~ /^<br\s\/>(\w+)\s/;
    $sum{$1}++;
}
close F or die "Error: cannot read file: $file, $!";

my $i = 0;
foreach ( sort { $sum{$b} <=> $sum{$a} } keys %sum )
{
        print "$_, $sum{$_}\n";
        last if (++$i > 4);   # we print only the first 5 lines
}

You can use it as:
perlscript.pl filename > result