LinuxQuestions.org
LinuxAnswers - the LQ Linux tutorial section.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices

Reply
 
Search this Thread
Old 01-14-2010, 03:13 PM   #1
xmrkite
Member
 
Registered: Oct 2006
Location: California, USA
Distribution: Mint 16, Lubuntu 14.04, Mythbuntu 14.04, Kubuntu 13.10, Xubuntu 10.04
Posts: 540

Rep: Reputation: 30
Delete Duplicate Lines in a file, leaving only the unique lines left


Hello. i've seen several simple commands used to delete duplicate lines from a file, but I've yet to see a command that can leave a file with only the unique lines.

So, for example, here is the file:

1
2
3
1
2
3
4
5
6

I want to have it so that after the command is run on the file, it looks like this:

4
5
6

Any ideas?

-Thanks for the help.
 
Old 01-14-2010, 03:16 PM   #2
GrapefruiTgirl
Guru
 
Registered: Dec 2006
Location: underground
Distribution: Slackware64
Posts: 7,594

Rep: Reputation: 550Reputation: 550Reputation: 550Reputation: 550Reputation: 550Reputation: 550
try the `uniq` command

Sasha
 
Old 01-14-2010, 03:22 PM   #3
xmrkite
Member
 
Registered: Oct 2006
Location: California, USA
Distribution: Mint 16, Lubuntu 14.04, Mythbuntu 14.04, Kubuntu 13.10, Xubuntu 10.04
Posts: 540

Original Poster
Rep: Reputation: 30
how does that delete the lines that are 1 2 or 3? I think using unique i can get a file that reads:

1
2
3
4
5
6

but not one that reads only

4
5
6

Am i wrong?
--Thanks
 
Old 01-14-2010, 03:25 PM   #4
GrapefruiTgirl
Guru
 
Registered: Dec 2006
Location: underground
Distribution: Slackware64
Posts: 7,594

Rep: Reputation: 550Reputation: 550Reputation: 550Reputation: 550Reputation: 550Reputation: 550
Check the man page; observe the -u option, to print only unique lines.

Sasha
 
Old 01-14-2010, 03:28 PM   #5
xmrkite
Member
 
Registered: Oct 2006
Location: California, USA
Distribution: Mint 16, Lubuntu 14.04, Mythbuntu 14.04, Kubuntu 13.10, Xubuntu 10.04
Posts: 540

Original Poster
Rep: Reputation: 30
Perfect.

Here is the command i use: cat filename | sort | uniq -u

The issue i had is that uniq can do it, but only if the file is first sorted.

-Thanks for the quick help.
 
Old 01-14-2010, 05:59 PM   #6
chrism01
Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.5, Centos 5.10
Posts: 16,242

Rep: Reputation: 2024Reputation: 2024Reputation: 2024Reputation: 2024Reputation: 2024Reputation: 2024Reputation: 2024Reputation: 2024Reputation: 2024Reputation: 2024Reputation: 2024
UUOC (useless use of cat), use

sort file|uniq -u
 
Old 01-14-2010, 06:18 PM   #7
xmrkite
Member
 
Registered: Oct 2006
Location: California, USA
Distribution: Mint 16, Lubuntu 14.04, Mythbuntu 14.04, Kubuntu 13.10, Xubuntu 10.04
Posts: 540

Original Poster
Rep: Reputation: 30
Even Better. Thanks.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
How would I ignore other lines in a file with non unique words? btacuso Linux - Newbie 1 05-24-2009 08:20 AM
How to compare two lines and delete the duplicate line from a file? Shobhna Linux - Newbie 10 12-05-2008 01:08 PM
Finding duplicate lines in a file MikeyCarter Linux - Software 3 10-05-2008 05:28 PM
how do u delete duplicate lines bharatbsharma Programming 4 10-29-2007 06:04 PM
awk/gawk/sed - read lines from file1, comment out or delete matching lines in file2 rascal84 Linux - General 1 05-24-2006 09:19 AM


All times are GMT -5. The time now is 03:47 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration