[SOLVED] How can I ignore lines with partial similarity and retain only one.

btacuso · 03-13-2010, 05:57 AM

Hi,

I need only one of the lines having the same ending. It does'nt matter which lines to discard as long long as one with the same ending is retained. Of course I have to retain any unique lines too. Thanks again.

myfile:

aaaa bb 12345
ccc aaa 1234
abc de zzzzz
b cc dd
abab ba ba 12345
xx xx 12345
fg hi zzzzz
a 12345

result:

aaaa bb 12345
ccc aaa 1234
b cc dd
abc de zzzzz

yucefrizk · 03-13-2010, 07:14 AM

Maybe on google you will find more tips but here is a quick script that you can run: input is "myfile" and output is "result"

#!/bin/bash

distinct=`awk '{print $NF}' myfile | sort -u`
for i in $distinct
do
grep " $i"$ myfile | head -1 >> result
done

btacuso · 03-13-2010, 02:28 PM

Wow !!! it worked.

Thanks a lot Yuce...

btacuso · 03-13-2010, 02:34 PM

Quote:

Originally Posted by btacuso

Hi,

I need only one of the lines having the same ending. It does'nt matter which lines to discard as long long as one with the same ending is retained. Of course I have to retain any unique lines too. Thanks again.

myfile:

aaaa bb 12345
ccc aaa 1234
abc de zzzzz
b cc dd
abab ba ba 12345
xx xx 12345
fg hi zzzzz
a 12345

result:

aaaa bb 12345
ccc aaa 1234
b cc dd
abc de zzzzz

It worked. Thanks a lot.