LinuxQuestions.org
LinuxAnswers - the LQ Linux tutorial section.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices

Reply
 
Search this Thread
Old 07-04-2007, 10:55 AM   #1
sph90457
Member
 
Registered: Oct 2003
Location: UK
Posts: 52

Rep: Reputation: 15
Getting only unique lines using grep..


Guys,

I have a file that is text, and comma delimited and is over 65536 lines. I want to run a grep command or awk or something that will look down the file and check the second column which is a specific date and just return the dates. Then I would like to have one date displayed for each no matter how many times it's in the file :-

e.g

4,070327,9,41,20,-1,-10,-250,001
4,070328,9,41,20,-1,-101,-800,001
4,070328,9,41,20,-1,-102,-800,001
7,070327,9,41,20,-1,-102,-800,001
7,070329,9,41,20,-1,-103,-800,001
4,070329,9,41,20,-1,-104,-800,001
7,070327,9,41,20,-1,-104,-800,001
4,070328,9,41,20,-1,-106,-800,001
7,070326,41,20,-1,-106,-800,001
4,070324,41,20,-1,-112,-800,001
4,070227,20,-1,-118,-400,001
4,070327,29,41,20,-1,-12,-200,001
4,070328,29,41,20,-1,-102,-800,001
7,070327,88,41,20,-1,-102,-800,001
7,070329,99,41,20,-1,-103,-800,001
4,070329,93,41,20,-1,-104,-800,001
7,070327,93,41,20,-1,-104,-800,001
4,070328,93,41,20,-1,-106,-800,001
7,070326,42,20,-1,-106,-800,001
4,070324,42,20,-1,-112,-800,001
4,070227,22,-1,-118,-400,001
4,070327,91,41,20,-1,-12,-200,001
etc...

So I would like to see 070227, 070326, 070327,070328, 070329 displayed like this :-

070227
070326
070327
070328
070329

Any help would be much appreciated.....

cheers....
 
Old 07-04-2007, 11:34 AM   #2
SlacUser
Member
 
Registered: Apr 2007
Location: México
Distribution: Slackware, Fedora
Posts: 61

Rep: Reputation: 15
Read the man page for grep

You can count how many times a pattern repeats with the -c flag, instead of displaying all lines that matches.

Be more specific in what do you want to find in your file.
 
Old 07-04-2007, 11:40 AM   #3
nx5000
Senior Member
 
Registered: Sep 2005
Location: Out
Posts: 3,307

Rep: Reputation: 52
You can extract the second field, sort it numerically and remove duplicates:
Code:
 cat file | cut -d',' -f 2 | sort -n | uniq
 
Old 07-05-2007, 01:40 AM   #4
sph90457
Member
 
Registered: Oct 2003
Location: UK
Posts: 52

Original Poster
Rep: Reputation: 15
That works a treat....

Thanks for taking the time to help me, this will be very useful....

cheers....
 
Old 07-05-2007, 06:10 AM   #5
timmeke
Senior Member
 
Registered: Nov 2005
Location: Belgium
Distribution: Red Hat, Fedora
Posts: 1,515

Rep: Reputation: 61
Code:
cat file | cut -d',' -f 2 | sort -n | uniq
can be simplified to
Code:
cut -d',' -f 2 file | sort -nu
 
Old 07-05-2007, 06:24 AM   #6
nx5000
Senior Member
 
Registered: Sep 2005
Location: Out
Posts: 3,307

Rep: Reputation: 52
Good spot, I didn't know the u option of sort.
 
Old 07-05-2007, 08:31 AM   #7
timmeke
Senior Member
 
Registered: Nov 2005
Location: Belgium
Distribution: Red Hat, Fedora
Posts: 1,515

Rep: Reputation: 61
Thanks, nx5000.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
grep lines before and after magnum81 Programming 3 07-29-2011 04:52 PM
Removing partially unique lines lluciano Programming 9 01-11-2006 11:20 AM
How to grep similar lines in bash? bruno buys Linux - Software 2 12-02-2005 11:56 PM
Unique lines based on specific fields. carl.waldbieser Programming 6 08-21-2005 02:26 PM
Grep lines gsibble Linux - General 1 04-12-2004 01:30 AM


All times are GMT -5. The time now is 09:59 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration