Delete Duplicate Lines in a file, leaving only the unique lines left

xmrkite · 01-14-2010, 03:13 PM

Hello. i've seen several simple commands used to delete duplicate lines from a file, but I've yet to see a command that can leave a file with only the unique lines.

So, for example, here is the file:

1
2
3
1
2
3
4
5
6

I want to have it so that after the command is run on the file, it looks like this:

4
5
6

Any ideas?

-Thanks for the help.

GrapefruiTgirl · 01-14-2010, 03:16 PM

try the `uniq` command

Sasha

xmrkite · 01-14-2010, 03:22 PM

how does that delete the lines that are 1 2 or 3? I think using unique i can get a file that reads:

1
2
3
4
5
6

but not one that reads only

4
5
6

Am i wrong?
--Thanks

GrapefruiTgirl · 01-14-2010, 03:25 PM

Check the man page; observe the -u option, to print only unique lines.

Sasha

xmrkite · 01-14-2010, 03:28 PM

Perfect.

Here is the command i use: cat filename | sort | uniq -u

The issue i had is that uniq can do it, but only if the file is first sorted.

-Thanks for the quick help.

chrism01 · 01-14-2010, 05:59 PM

UUOC (useless use of cat), use

sort file|uniq -u

xmrkite · 01-14-2010, 06:18 PM

Even Better. Thanks.