LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Programming (http://www.linuxquestions.org/questions/programming-9/)
-   -   Use uniq on first part of file but print whole line. (http://www.linuxquestions.org/questions/programming-9/use-uniq-on-first-part-of-file-but-print-whole-line-759391/)

snowman81 10-03-2009 05:45 AM

Use uniq on first part of file but print whole line.
 
I have a script that does an md5 hash on a bunch of files in 2 folders and prints them to a text file. I know I can run uniq to print out the well, unique, ones but the problem is the lines are from different folders. As an example:

Code:

d41d8cd98f00b204e9800998ecf8427e  /home/jason/Desktop/folder1/testin3
and

Code:

d41d8cd98f00b204e9800998ecf8427e  /home/jason/Desktop/folder2/testin2
I am going to see if I can use awk to only run uniq on the first part but actually print out the whole line but I wanted to see if anyone had a better idea? Thanks.

catkin 10-03-2009 05:54 AM

What do you want to do? Do you want to display all lines having the same md5 hash?

konsolebox 10-03-2009 05:54 AM

won't 'uniq -w 32' do the trick?

jschiwal 10-03-2009 06:04 AM

If you want to find files with duplicates, you can use
"sort list | uniq -w32 -D" to print the file and it's duplicates.

You can pipe the output through the cut command to remove the md5sum field from the final list.

sort, uniq, cut and around 100 other very useful commands are supplied by the coreutils package. I would recommend scanning through the info manual for coreutils. I even downloaded the source to create a print worthy pdf from the .texi source, and printed it for a 3-ring binder hard copy.

snowman81 10-03-2009 06:22 AM

Yup, you're both right. I'm looking at the man page and didn't even see the -w option. I feel sheepish now :)


All times are GMT -5. The time now is 06:15 AM.