[SOLVED] Shell script to get the most occurring entry from a list

Buddhike G · 09-10-2010, 01:06 AM

Hi,

i'm want to write a shell script to get the most occurring entry from a ip list which i'm having. the list is as follows

192.168.1.3
192.168.1.3
192.168.1.3
71.180.121.171
118.232.208.122
10.255.255.51
192.168.1.7
192.168.1.3
192.168.1.3
173.206.108.92
94.97.66.182
192.168.1.3
192.168.1.3
192.168.1.3
192.168.1.3
79.108.212.157
178.248.57.91
192.168.1.3
192.168.1.3
192.168.1.3
10.255.255.51
192.168.1.7
10.255.255.51
192.168.1.3
192.168.1.3
192.168.1.3
118.166.69.6
91.132.207.30
192.168.1.3
123.247.142.40
92.17.63.154
192.168.1.3
221.210.87.59
192.168.1.3
192.168.1.3
192.168.1.3
94.66.147.153
192.168.1.7
192.168.1.3
24.118.125.129
109.82.121.164
192.168.1.3
124.164.249.85
192.168.1.3
192.168.1.3
94.66.220.228
192.168.1.3
192.168.1.3
192.168.1.3
109.182.8.133
216.8.164.128
192.168.1.3
219.252.177.207
192.168.1.7
78.53.157.50
10.255.255.51
192.168.1.7
94.171.27.230
192.168.1.3
192.168.1.3
192.168.1.3
192.168.1.3
192.168.1.3
192.168.1.3
97.106.142.8
192.168.1.3
221.224.18.253
192.168.1.3
192.168.1.3
10.255.255.51
192.168.1.3
122.177.101.254
123.153.134.156
192.168.1.3
75.139.206.231
192.168.1.3
118.170.8.253
192.168.1.3
192.168.1.3
174.23.136.227
192.168.10.102
192.168.1.3
192.168.1.3
192.168.1.3
219.90.188.99
192.168.1.3
192.168.1.3
10.255.255.51
192.168.1.3
10.255.255.51
192.168.1.3
192.168.1.7
83.152.237.68
2.88.78.70
192.168.1.3
192.168.1.3
192.168.1.3
192.168.1.3
87.123.77.203
76.126.124.22
192.168.1.3
192.168.1.3

it would be a great help if anyone can help me out with writing that.

Thanx in advance.
BG

grail · 09-10-2010, 01:12 AM

So what have you tried? It looks reasonably trivial and could be done with a myriad of applications or languages.
You may wish to provide details on what you would like to use and where you are getting stuck?

dugan · 09-10-2010, 01:16 AM

Declare an associative array. Also known as a dictionary, map or hash table, depending on the language. Then loop through the list of addresses. For each one, if it's not in the associative array, then add it and associate a count of zero with it. If it's already there, then increment its associated count. Then go through the associative array and find the address with the largest count.

Bash 4.0 and later has associative arrays.
http://stackoverflow.com/questions/6...-shell-scripts

So do most programming languages and/or their standard libraries.

ghostdog74 · 09-10-2010, 01:20 AM

if you have Python

Code:

from collections import  defaultdict
d=defaultdict(int)
for line in open("file"):
    d[line.rstrip()]+=1
m=max(d.values())
for i,j in d.iteritems():
    if m==j:
        print i,j

output

Code:

$ python test.py
192.168.1.3 55

Guttorm · 09-10-2010, 01:33 AM

dugan · 09-10-2010, 01:39 AM

Quote:

Originally Posted by Guttorm

cat list |sort |uniq -c |sort |tail -n 1

Brilliant

druuna · 09-10-2010, 01:42 AM

Hi,

And an awk solution:

Code:

#!/bin/bash

awk ' { array[$1]++; }
END { for (count in array)
{ if ( max < array[count] )
    { max = array[count];
      maxcount = count; }
}
print maxcount, array[maxcount];
}' infile

Hope this helps.

crts · 09-10-2010, 01:57 AM

Another awk:

Code:

awk '(++count[$1]>count[res]){res=$1}END{print count[res]" "res}' file

crts · 09-10-2010, 01:59 AM

Quote:

Originally Posted by dugan

Brilliant

It looks very elegant, however, it does not give the correct result.

Code:

$ cat file |sort|uniq -c|sort|tail
      1 91.132.207.30
      1 92.17.63.154
      1 94.171.27.230
      1 94.66.147.153
      1 94.66.220.228
      1 94.97.66.182
      1 97.106.142.8
     55 192.168.1.3
      6 192.168.1.7
      7 10.255.255.51

ghostdog74 · 09-10-2010, 02:05 AM

Quote:

Originally Posted by crts

It looks very elegant, however, it does not give the correct result.

Code:

$ cat file |sort|uniq -c|sort|tail
      1 91.132.207.30
      1 92.17.63.154
      1 94.171.27.230
      1 94.66.147.153
      1 94.66.220.228
      1 94.97.66.182
      1 97.106.142.8
     55 192.168.1.3
      6 192.168.1.7
      7 10.255.255.51

that's because he did not use -n. Also, no need for cat.

grail · 09-10-2010, 02:10 AM

@crts - I came up with a similar solution to your but not sure how your test works?

Quote:

++count[$1]>res

The count value will be going up by 1 but res is an IP address (except for first round)???

Mine was:

Code:

awk '++sum[$0] > sum[f]{f = $0}END{print f, sum[f]}' file

So we get the same answer

but would like to know what I missing about yours?

Edit: worked it out ... yours only returns the last entry in the file

crts · 09-10-2010, 02:37 AM

Quote:

Originally Posted by grail

@crts - I came up with a similar solution to your but not sure how your test works?

Hi,

good catch! What I was actually thinking of was to store the biggest occurrence just as you did. I previously had another solution where res
did hold the actual biggest occurrence.

Code:

sort file|uniq -c|awk '($1 > res){res=$1}END{print res}'

Somehow I mixed it up.

Buddhike G · 09-10-2010, 07:44 AM

Hi,

Thank you so much for quick and comprehensive reply, And hats off for you all.

cheers !!
BG

grail · 09-10-2010, 08:20 AM

Hey BG, see my signature for marking the thread as SOLVED