Share your knowledge at the LQ Wiki.
Go Back > Forums > Non-*NIX Forums > Programming
User Name
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.


  Search this Thread
Old 10-03-2009, 05:45 AM   #1
Registered: Aug 2006
Location: Michigan
Distribution: Ubuntu
Posts: 240

Rep: Reputation: 30
Use uniq on first part of file but print whole line.

I have a script that does an md5 hash on a bunch of files in 2 folders and prints them to a text file. I know I can run uniq to print out the well, unique, ones but the problem is the lines are from different folders. As an example:

 d41d8cd98f00b204e9800998ecf8427e  /home/jason/Desktop/folder1/testin3

 d41d8cd98f00b204e9800998ecf8427e  /home/jason/Desktop/folder2/testin2
I am going to see if I can use awk to only run uniq on the first part but actually print out the whole line but I wanted to see if anyone had a better idea? Thanks.
Old 10-03-2009, 05:54 AM   #2
LQ 5k Club
Registered: Dec 2008
Location: Tamil Nadu, India
Distribution: Debian
Posts: 8,576
Blog Entries: 31

Rep: Reputation: 1195Reputation: 1195Reputation: 1195Reputation: 1195Reputation: 1195Reputation: 1195Reputation: 1195Reputation: 1195Reputation: 1195
What do you want to do? Do you want to display all lines having the same md5 hash?
Old 10-03-2009, 05:54 AM   #3
Senior Member
Registered: Oct 2005
Distribution: Gentoo, Slackware, LFS
Posts: 2,248
Blog Entries: 8

Rep: Reputation: 235Reputation: 235Reputation: 235
won't 'uniq -w 32' do the trick?
Old 10-03-2009, 06:04 AM   #4
LQ Guru
Registered: Aug 2001
Location: Fargo, ND
Distribution: SuSE AMD64
Posts: 15,733

Rep: Reputation: 671Reputation: 671Reputation: 671Reputation: 671Reputation: 671Reputation: 671
If you want to find files with duplicates, you can use
"sort list | uniq -w32 -D" to print the file and it's duplicates.

You can pipe the output through the cut command to remove the md5sum field from the final list.

sort, uniq, cut and around 100 other very useful commands are supplied by the coreutils package. I would recommend scanning through the info manual for coreutils. I even downloaded the source to create a print worthy pdf from the .texi source, and printed it for a 3-ring binder hard copy.

Last edited by jschiwal; 10-03-2009 at 06:06 AM.
Old 10-03-2009, 06:22 AM   #5
Registered: Aug 2006
Location: Michigan
Distribution: Ubuntu
Posts: 240

Original Poster
Rep: Reputation: 30
Yup, you're both right. I'm looking at the man page and didn't even see the -w option. I feel sheepish now


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off

Similar Threads
Thread Thread Starter Forum Replies Last Post
Perl: Match part of a line and replace with another line from the same file briana.paige Linux - Newbie 8 06-27-2009 06:35 AM
LXer: Tutorial: Sharing a Samba File and Print Server Across Subnets, Part 2 LXer Syndicated Linux News 0 01-18-2008 07:00 AM
LXer: Tutorial: Sharing a Samba File and Print Server Across Subnets, Part 1 LXer Syndicated Linux News 0 01-11-2008 01:20 AM
script to get a part of line from a file mystical dervish Programming 7 05-11-2007 02:57 AM
Regex Question: Only print part of line that matches TheMeteorPolice Programming 5 01-12-2006 01:21 PM > Forums > Non-*NIX Forums > Programming

All times are GMT -5. The time now is 03:29 AM.

Main Menu
Write for LQ is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration