[SOLVED] Awk or grep to print only seen once lines

danielbmartin · 10-22-2015, 08:03 AM

Quote:

Originally Posted by magische_vogel

Is it possible to improve this script to read from pipe instead of files.

I suppose you could use a temporary file (a work file).

With this InFile ...

Code:

41.200.103.2
41.200.103.2
41.200.103.220
41.201.103.2
41.201.103.2
41.201.103.2
41.201.103.220
41.201.103.2

... this code ...

Code:

cat $InFile                     \
|tee $Work1                     \
|awk '{if (2==++a[$0]) print}'  \
|grep -vxf - $Work1             \
>$OutFile

... produced this OutFile ...

Code:

41.200.103.220
41.201.103.220

Daniel B. Martin

Sefyir · 10-22-2015, 09:40 AM

Quote:

Originally Posted by pan64

I do not really understand why do you use 3 different commands if one would be enough:

Code:

sort -u

obviously it is not an answer to the original post, just a comment.

It's really only two commands (since it uses cat)

That being said, sort -u did not work for me with this output (bolded being parts that are supposed to be omitted)

Code:

105.101.173.112
105.97.167.43
105.99.68.250
41.110.184.109
41.200.103.2

rknichols · 10-22-2015, 10:10 AM

Quote:

Originally Posted by magische_vogel

Your gawk script is not working well with this input (The original order is not kept):

41.200.103.2
105.97.167.43
105.97.167.43
41.110.184.109
41.110.184.109
105.99.68.250
105.99.68.250
41.110.184.109
41.200.103.220
41.110.184.109
41.110.184.109
41.110.184.109
41.110.184.109
105.99.68.250
41.110.184.109
105.99.68.250
105.101.173.112

Ahh yes, it's the old string sort problem

. The array bb is indexed by the original line number, but array indices are strings and "17" sorts as less than "9".

If you happen to be running version 4 of gawk there is a trivial fix to add a third argument "@ind_num_asc" to the asorti() call, but for earlier versions you have to make sure the index is a string that will sort properly.

Code:

{ if($0 in aa) aa[$0] = 0; else aa[$0] = FNR }
END {
    for(x in aa) if(aa[x] > 0) bb[sprintf("%15s", aa[x])] = x
    n_unique = asorti(bb, cc)
    for(n = 1; n <= n_unique; ++n) print bb[cc[n]]
}

Padding that index to a 15-character string makes it work (at least until you have 10^15 lines in the input

).

grail · 10-22-2015, 10:19 AM

Code:

ruby -ane 'h||={}; h[$F[0]] = h[$F[0]].to_i + 1;END{h.each{|k,v| puts k if v == 1}}'

pan64 · 10-24-2015, 10:43 AM

Code:

awk '{ ++aa[$0]; x[i++]=$0 } END { for(j=0; j<i; j++) if(aa[x[j]] == 1) print x[j]}'