LinuxQuestions.org
Latest LQ Deal: Linux Power User Bundle
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 05-08-2010, 12:08 PM   #1
wapitismith
LQ Newbie
 
Registered: May 2009
Posts: 3

Rep: Reputation: 0
Using sed to remove lines with duplicate ID's, but different endings...


I have a file that contains lines representing the nodes of a polyline but I only need the first point in each segment. With the following text:

0,"013A",0.57,260739.891,4379258.87
0,"013A",0.57,260737.674,4379258.94
0,"013A",0.57,260684.628,4379258.35
1,"013A",0.545,260769.915,4379257.84
1,"013A",0.545,260739.891,4379258.87
2,"013A",1.059,259567.126,4379293.16
2,"013A",1.059,259562.637,4379302.59
2,"013A",1.059,259534.423,4379337.52
2,"013A",1.059,259460.853,4379414.3
3,"013A",1.036,259574.096,4379278.51
3,"013A",1.036,259567.126,4379293.16
4,"013A",1,259580.147,4379253.83
4,"013A",1,259574.415,4379277.84
4,"013A",1,259574.096,4379278.51
5,"013A",0.98,259581.802,4379185.53
5,"013A",0.98,259580.147,4379253.83
I would like to have this as output:

0,"013A",0.57,260737.674,4379258.94
1,"013A",0.545,260769.915,4379257.84
2,"013A",1.059,259567.126,4379293.16
3,"013A",1.036,259574.096,4379278.51
4,"013A",1,259580.147,4379253.83
5,"013A",0.98,259581.802,4379185.53
I've tried combinations of uniq and awk, and sed, but I am stumped. I'm sure I'm too close to the problem and can't find the simple solution.

The problem with uniq is that the last two colums will differ. I don't care about the x/y for any points following the first one.

Any assistance would be greatly appreciated.

~wapitismith~
 
Old 05-08-2010, 12:33 PM   #2
colucix
LQ Guru
 
Registered: Sep 2003
Location: Bologna
Distribution: CentOS 6.5 OpenSuSE 12.3
Posts: 10,509

Rep: Reputation: 1976Reputation: 1976Reputation: 1976Reputation: 1976Reputation: 1976Reputation: 1976Reputation: 1976Reputation: 1976Reputation: 1976Reputation: 1976Reputation: 1976
What about this?
Code:
awk -F, '{ if ( ! ( $1$2$3 in _ )) _[$1$2$3] = $0 } END { for ( i in _ ) print _[i] }' file | sort
 
Old 05-08-2010, 12:36 PM   #3
grail
LQ Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 9,252

Rep: Reputation: 2685Reputation: 2685Reputation: 2685Reputation: 2685Reputation: 2685Reputation: 2685Reputation: 2685Reputation: 2685Reputation: 2685Reputation: 2685Reputation: 2685
Or maybe:
Code:
awk -F, '!_[$1]++' file
 
Old 05-08-2010, 12:40 PM   #4
colucix
LQ Guru
 
Registered: Sep 2003
Location: Bologna
Distribution: CentOS 6.5 OpenSuSE 12.3
Posts: 10,509

Rep: Reputation: 1976Reputation: 1976Reputation: 1976Reputation: 1976Reputation: 1976Reputation: 1976Reputation: 1976Reputation: 1976Reputation: 1976Reputation: 1976Reputation: 1976
Great, grail!
 
Old 05-08-2010, 01:30 PM   #5
wapitismith
LQ Newbie
 
Registered: May 2009
Posts: 3

Original Poster
Rep: Reputation: 0
Incredilby easy! I knew I was missing something simple. very nice responses!!!

Thanks!
 
  


Reply

Tags
awk, duplicate, lines, sed


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
remove duplicate lines from shell script himu3118 Programming 16 04-21-2012 08:37 AM
script to remove the lines which are having the duplicate value in 2 fields ajcapri Linux - Newbie 10 11-29-2009 10:38 PM
[SOLVED] uniq -u : does not seem to remove duplicate lines boxb29 Linux - General 7 08-15-2009 07:34 PM
sed to remove specific lines in a file tekmann33 Linux - Newbie 3 05-21-2009 04:41 PM
Removing duplicate lines with sed tireseas Programming 10 01-12-2005 04:27 AM


All times are GMT -5. The time now is 02:47 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration