LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices



Reply
 
Search this Thread
Old 01-12-2004, 02:46 PM   #1
benjithegreat98
Senior Member
 
Registered: Dec 2003
Location: Shelbyville, TN, USA
Distribution: Fedora Core, CentOS
Posts: 1,019

Rep: Reputation: 45
remove identical lines in a file


If I have a file called test.tmp that contains

Kyle's mom is a dirty bitch.
Kyle's mom is not a dirty bitch.
Kyle's mom is a dirty bitch.

Is there a single command I can use to display all the lines with out repeating duplicates.

example:
desiredCommand /home/benji/test.tmp
and the output is
Kyle's mom is a dirty bitch
Kyle's mom is not a dirty bitch.

I can accomplish this with a script I made but when there is 1000 lines in the file, it take a few seconds to finish. I was hoping there would be a quick command to do it.

Also if the command would just plain out remove the lines from the file and I could output it myself that would be great too.

Thanks all.
 
Old 01-12-2004, 02:53 PM   #2
slakmagik
Senior Member
 
Registered: Feb 2003
Distribution: Slackware
Posts: 4,113

Rep: Reputation: Disabled
Not a single command that I know of but you can look into 'sort', 'uniq', and 'sed'. Which you may have already done. It would have helped to post the script you're not satisfied with. And you maybe could have posted different examples.
 
Old 01-12-2004, 03:25 PM   #3
benjithegreat98
Senior Member
 
Registered: Dec 2003
Location: Shelbyville, TN, USA
Distribution: Fedora Core, CentOS
Posts: 1,019

Original Poster
Rep: Reputation: 45
Ok, that helped! I can use a combo of sort and uniq for what I'm doing. It will be _much_ faster. Thanks.
 
Old 04-21-2004, 07:50 PM   #4
linuxfond
Member
 
Registered: Jan 2003
Location: Belgium
Distribution: Mandrake 9.2
Posts: 475

Rep: Reputation: 30
What is sort and uniq? Are these commands?
I got: command not found.
 
Old 04-24-2004, 07:12 AM   #5
slakmagik
Senior Member
 
Registered: Feb 2003
Distribution: Slackware
Posts: 4,113

Rep: Reputation: Disabled
Yeah. Something must be messed up with your path or your install or your permissions - those are pretty fundamental commands. Part of coreutils and installed in /usr/bin on my box - probably there or possibly /bin on any distro.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
How to remove the first part of lines? daYz Linux - Software 3 11-26-2005 07:09 PM
Remove odd lines from a text file Mr. Gone Programming 2 09-19-2005 12:16 PM
How to remove lines from a file doza Linux - General 2 04-27-2005 12:59 PM
How do i remove blank lines from a file? kakho Programming 1 04-15-2004 04:57 AM
[bash] remove lines from a file Drimo Programming 3 03-20-2004 12:16 PM


All times are GMT -5. The time now is 09:36 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration