LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 03-13-2010, 06:57 AM   #1
btacuso
Member
 
Registered: May 2009
Posts: 32

Rep: Reputation: 15
How can I ignore lines with partial similarity and retain only one.


Hi,

I need only one of the lines having the same ending. It does'nt matter which lines to discard as long long as one with the same ending is retained. Of course I have to retain any unique lines too. Thanks again.

myfile:

aaaa bb 12345
ccc aaa 1234
abc de zzzzz
b cc dd
abab ba ba 12345
xx xx 12345
fg hi zzzzz
a 12345


result:

aaaa bb 12345
ccc aaa 1234
b cc dd
abc de zzzzz
 
Old 03-13-2010, 08:14 AM   #2
yucefrizk
Member
 
Registered: Nov 2009
Posts: 44

Rep: Reputation: 18
Maybe on google you will find more tips but here is a quick script that you can run: input is "myfile" and output is "result"

#!/bin/bash

distinct=`awk '{print $NF}' myfile | sort -u`
for i in $distinct
do
grep " $i"$ myfile | head -1 >> result
done
 
1 members found this post helpful.
Old 03-13-2010, 03:28 PM   #3
btacuso
Member
 
Registered: May 2009
Posts: 32

Original Poster
Rep: Reputation: 15
Wow !!! it worked.

Thanks a lot Yuce...
 
Old 03-13-2010, 03:34 PM   #4
btacuso
Member
 
Registered: May 2009
Posts: 32

Original Poster
Rep: Reputation: 15
Quote:
Originally Posted by btacuso View Post
Hi,

I need only one of the lines having the same ending. It does'nt matter which lines to discard as long long as one with the same ending is retained. Of course I have to retain any unique lines too. Thanks again.

myfile:

aaaa bb 12345
ccc aaa 1234
abc de zzzzz
b cc dd
abab ba ba 12345
xx xx 12345
fg hi zzzzz
a 12345


result:

aaaa bb 12345
ccc aaa 1234
b cc dd
abc de zzzzz
It worked. Thanks a lot.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
how can I "cat" or "grep" a file to ignore lines starting with "#" ??? callagga Linux - Newbie 7 08-16-2013 07:58 AM
rpm similarity to windows? Chenchu Linux - Newbie 6 11-26-2009 04:10 PM
How would I ignore other lines in a file with non unique words? btacuso Linux - Newbie 1 05-24-2009 09:20 AM
Make sort ignore comment lines CrendKing Linux - General 3 04-17-2009 11:44 PM
image similarity test vaginismus Programming 1 06-18-2005 03:13 AM


All times are GMT -5. The time now is 07:35 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration