LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices

Reply
 
Search this Thread
Old 01-27-2013, 07:55 AM   #1
raosr020
LQ Newbie
 
Registered: Nov 2012
Posts: 16

Rep: Reputation: Disabled
Arrow Removing a set of Duplicate lines from a file


Hi,

How do i remove a set of duplicate lines from a file.

My file contains the lines:
Code:
abc
def
ghi

abc
def
ghi

jkl
mno
pqr

jkl
mno
pqr

stu
vwx
yza
My output should look like:
Code:
abc
def
ghi

jkl
mno
pqr

stu
vwx
yza
Please let me know if this is not clear..

Thanks in advance.

-SREE

Last edited by raosr020; 01-27-2013 at 08:01 AM.
 
Old 01-27-2013, 09:07 AM   #2
Habitual
Senior Member
 
Registered: Jan 2011
Distribution: Slack14_64_Multilib
Posts: 3,072
Blog Entries: 4

Rep: Reputation: 741Reputation: 741Reputation: 741Reputation: 741Reputation: 741Reputation: 741Reputation: 741
Code:
uniq file
 
Old 01-27-2013, 09:08 AM   #3
buttugly
Member
 
Registered: Feb 2006
Location: Texas of course
Distribution: Slackware-current
Posts: 37

Rep: Reputation: 10
http://www.cyberciti.biz/faq/unix-li...plicate-lines/

http://www.unix.com/shell-programmin...l-sed-awk.html

Last edited by buttugly; 01-27-2013 at 09:12 AM. Reason: stumbled upon another link
 
Old 01-29-2013, 10:22 AM   #4
dru8274
Member
 
Registered: Oct 2011
Location: New Zealand
Distribution: Debian
Posts: 105

Rep: Reputation: 36
Code:
sort -u data.dat
Happy with ur solution... then tick "yes" and mark as Solved!
 
Old 01-29-2013, 11:09 AM   #5
colucix
Moderator
 
Registered: Sep 2003
Location: Bologna
Distribution: CentOS 6.5 OpenSuSE 12.3
Posts: 10,458

Rep: Reputation: 1941Reputation: 1941Reputation: 1941Reputation: 1941Reputation: 1941Reputation: 1941Reputation: 1941Reputation: 1941Reputation: 1941Reputation: 1941Reputation: 1941
Code:
awk 'BEGIN {RS = "\n\n"} !_[$0]++ {if (NR == 1) printf "%s", $0; else printf "\n\n%s", $0}' file
This assumes the sets of lines are separated by an empty line, as in your example. HTH.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
delete duplicate lines from a file which has one field different jkeertir Linux - Newbie 2 02-15-2011 12:45 AM
Delete Duplicate Lines in a file, leaving only the unique lines left xmrkite Linux - Software 6 01-14-2010 06:18 PM
How to compare two lines and delete the duplicate line from a file? Shobhna Linux - Newbie 10 12-05-2008 01:08 PM
Finding duplicate lines in a file MikeyCarter Linux - Software 3 10-05-2008 05:28 PM
Removing duplicate lines with sed tireseas Programming 10 01-12-2005 03:27 AM


All times are GMT -5. The time now is 04:11 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration