LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Linux - Newbie (https://www.linuxquestions.org/questions/linux-newbie-8/)
-   -   How can I ignore lines with partial similarity and retain only one. (https://www.linuxquestions.org/questions/linux-newbie-8/how-can-i-ignore-lines-with-partial-similarity-and-retain-only-one-795116/)

btacuso 03-13-2010 05:57 AM

How can I ignore lines with partial similarity and retain only one.
 
Hi,

I need only one of the lines having the same ending. It does'nt matter which lines to discard as long long as one with the same ending is retained. Of course I have to retain any unique lines too. Thanks again.

myfile:

aaaa bb 12345
ccc aaa 1234
abc de zzzzz
b cc dd
abab ba ba 12345
xx xx 12345
fg hi zzzzz
a 12345


result:

aaaa bb 12345
ccc aaa 1234
b cc dd
abc de zzzzz

yucefrizk 03-13-2010 07:14 AM

Maybe on google you will find more tips but here is a quick script that you can run: input is "myfile" and output is "result"

#!/bin/bash

distinct=`awk '{print $NF}' myfile | sort -u`
for i in $distinct
do
grep " $i"$ myfile | head -1 >> result
done

btacuso 03-13-2010 02:28 PM

Wow !!! it worked.

Thanks a lot Yuce...

btacuso 03-13-2010 02:34 PM

Quote:

Originally Posted by btacuso (Post 3896645)
Hi,

I need only one of the lines having the same ending. It does'nt matter which lines to discard as long long as one with the same ending is retained. Of course I have to retain any unique lines too. Thanks again.

myfile:

aaaa bb 12345
ccc aaa 1234
abc de zzzzz
b cc dd
abab ba ba 12345
xx xx 12345
fg hi zzzzz
a 12345


result:

aaaa bb 12345
ccc aaa 1234
b cc dd
abc de zzzzz

It worked. Thanks a lot.


All times are GMT -5. The time now is 11:08 PM.