sort & uniq

tostay2003 · 06-27-2008, 08:02 AM

Hi there,

I have a file as below

K1|K2|dataA1|K3|dataA2|K4|dataA3
K1|K2|dataB1|K3|dataB2|K4|dataB3

what I need to do is sort on the keys K1,K2,K3,K4 and remove any duplicates on them (only one should remain, could be any thing). How can I do that using sort and uniq.

matthewg42 · 06-27-2008, 08:09 AM

You might like to use the -u option to sort. No need to use uniq (which operates per line, not per sort key).

AnanthaP · 06-27-2008, 08:15 AM

Use awk or perl and

(1) Read the input and build a HASH array with hash value as k1+k2+k3+k4 and two data values having the count and any one data set (say the first).
(2) On end of file, scan the hash array and dump out the elements where the count > 1.

End

Hko · 06-28-2008, 06:14 PM

Code:

sort -u -t\| -k1,1 /tmp/file.txt