[SOLVED] Is there a way to unique by column/field?

mitter1989 · 10-19-2013, 04:57 AM

Hello folk,

Is there a command in CentOS to unique by column/field?

My file looks like below :
[root@mitter log]# cat test,txt
test test2
test3 test4
test2 test1
test2 test4

and desired output :

test test2
test2 test1
test3 test4

druuna · 10-19-2013, 05:26 AM

When looking at your example: Nope.

There's no way to know which of these 2 need to be removed:

Code:

test2 test1
test2 test4

You can get rid of the first one or the last one, but both are unique.

Here's a way to only show the first occurrence:

Code:

awk '!($1 in array) { array[$1]; print}' <( sort test.txt )

This only shows the last one:

Code:

awk '!($1 in array) { array[$1]; print}' <( sort -r data.file ) | sort

mitter1989 · 10-19-2013, 05:59 AM

Hello Druuna,

Thanks!

ref : http://stackoverflow.com/questions/1...915773#1915773

sort -u -t, -k1,1 file