LinuxQuestions.org
View the Most Wanted LQ Wiki articles.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices

Reply
 
Search this Thread
Old 10-03-2009, 05:45 AM   #1
snowman81
Member
 
Registered: Aug 2006
Location: Michigan
Distribution: Ubuntu
Posts: 225

Rep: Reputation: 30
Use uniq on first part of file but print whole line.


I have a script that does an md5 hash on a bunch of files in 2 folders and prints them to a text file. I know I can run uniq to print out the well, unique, ones but the problem is the lines are from different folders. As an example:

Code:
 d41d8cd98f00b204e9800998ecf8427e  /home/jason/Desktop/folder1/testin3
and

Code:
 d41d8cd98f00b204e9800998ecf8427e  /home/jason/Desktop/folder2/testin2
I am going to see if I can use awk to only run uniq on the first part but actually print out the whole line but I wanted to see if anyone had a better idea? Thanks.
 
Old 10-03-2009, 05:54 AM   #2
catkin
LQ 5k Club
 
Registered: Dec 2008
Location: Tamil Nadu, India
Distribution: Servers: Debian Squeeze and Wheezy. Desktop: Slackware64 14.0. Netbook: Slackware 13.37
Posts: 8,546
Blog Entries: 28

Rep: Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176
What do you want to do? Do you want to display all lines having the same md5 hash?
 
Old 10-03-2009, 05:54 AM   #3
konsolebox
Senior Member
 
Registered: Oct 2005
Distribution: Gentoo, Slackware, LFS
Posts: 2,245
Blog Entries: 15

Rep: Reputation: 233Reputation: 233Reputation: 233
won't 'uniq -w 32' do the trick?
 
Old 10-03-2009, 06:04 AM   #4
jschiwal
Guru
 
Registered: Aug 2001
Location: Fargo, ND
Distribution: SuSE AMD64
Posts: 15,733

Rep: Reputation: 654Reputation: 654Reputation: 654Reputation: 654Reputation: 654Reputation: 654
If you want to find files with duplicates, you can use
"sort list | uniq -w32 -D" to print the file and it's duplicates.

You can pipe the output through the cut command to remove the md5sum field from the final list.

sort, uniq, cut and around 100 other very useful commands are supplied by the coreutils package. I would recommend scanning through the info manual for coreutils. I even downloaded the source to create a print worthy pdf from the .texi source, and printed it for a 3-ring binder hard copy.

Last edited by jschiwal; 10-03-2009 at 06:06 AM.
 
Old 10-03-2009, 06:22 AM   #5
snowman81
Member
 
Registered: Aug 2006
Location: Michigan
Distribution: Ubuntu
Posts: 225

Original Poster
Rep: Reputation: 30
Yup, you're both right. I'm looking at the man page and didn't even see the -w option. I feel sheepish now
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Perl: Match part of a line and replace with another line from the same file briana.paige Linux - Newbie 8 06-27-2009 06:35 AM
LXer: Tutorial: Sharing a Samba File and Print Server Across Subnets, Part 2 LXer Syndicated Linux News 0 01-18-2008 07:00 AM
LXer: Tutorial: Sharing a Samba File and Print Server Across Subnets, Part 1 LXer Syndicated Linux News 0 01-11-2008 01:20 AM
script to get a part of line from a file mystical dervish Programming 7 05-11-2007 02:57 AM
Regex Question: Only print part of line that matches TheMeteorPolice Programming 5 01-12-2006 01:21 PM


All times are GMT -5. The time now is 09:16 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration