LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices

Reply
 
Search this Thread
Old 10-26-2007, 01:52 AM   #1
bharatbsharma
Member
 
Registered: Oct 2007
Posts: 33

Rep: Reputation: 15
how do u delete duplicate lines


I have i/p file

AAAA
AAAA
BBBB
BBBB
CCC
CCC

i want o/p file like this

AAAA
BBBB
CCC
 
Old 10-26-2007, 01:59 AM   #2
jschiwal
Guru
 
Registered: Aug 2001
Location: Fargo, ND
Distribution: SuSE AMD64
Posts: 15,733

Rep: Reputation: 654Reputation: 654Reputation: 654Reputation: 654Reputation: 654Reputation: 654
Use the uniq command.
 
Old 10-26-2007, 02:22 AM   #3
ghostdog74
Senior Member
 
Registered: Aug 2006
Posts: 2,695
Blog Entries: 5

Rep: Reputation: 241Reputation: 241Reputation: 241
and also sort, if duplicates are all over the place
 
Old 10-26-2007, 03:57 AM   #4
jschiwal
Guru
 
Registered: Aug 2001
Location: Fargo, ND
Distribution: SuSE AMD64
Posts: 15,733

Rep: Reputation: 654Reputation: 654Reputation: 654Reputation: 654Reputation: 654Reputation: 654
Quote:
Originally Posted by ghostdog74 View Post
and also sort, if duplicates are all over the place
Yes, if the sorted result is what you want. Otherwise you may need read each line, and check subsequent lines for duplicates. If you use sort, use the "-u" option to eliminate duplicates.

Last edited by jschiwal; 10-26-2007 at 03:59 AM.
 
Old 10-29-2007, 06:04 PM   #5
sundialsvcs
Guru
 
Registered: Feb 2004
Location: SE Tennessee, USA
Distribution: Gentoo, LFS
Posts: 5,283

Rep: Reputation: 1089Reputation: 1089Reputation: 1089Reputation: 1089Reputation: 1089Reputation: 1089Reputation: 1089Reputation: 1089
Sorting, in a situation like this, is crucial.

When you sort an arbitrarily-large file by any key, all of the occurrences of any key-value will become adjacent, and if there are any gaps in the key-sequence you automatically know that none of the keys in that gap exist anywhere.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Delete first and last lines of a file ChainsawPenguin Programming 5 09-28-2007 07:28 AM
awk/gawk/sed - read lines from file1, comment out or delete matching lines in file2 rascal84 Linux - General 1 05-24-2006 09:19 AM
checking for duplicate lines in text files (vb.net) mrobertson Programming 11 08-01-2005 12:40 PM
Removing duplicate lines with sed tireseas Programming 10 01-12-2005 03:27 AM
Delete duplicate hardware listings, and another... Chakthi Linux - Hardware 4 12-04-2002 08:19 PM


All times are GMT -5. The time now is 07:41 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration