LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 02-21-2011, 11:17 AM   #1
blue frog
LQ Newbie
 
Registered: Feb 2011
Posts: 1

Rep: Reputation: 0
detect duplicate files using md5sum


Hi guys,
What i am trying is to check the file duplication in a folder and remove a file if it is a duplicate of another file ie the contents are duplicate; but names may be same.
Basically i am using md5sum to calculate the md5sum values of each file and redirecting to a file. And i am thinking of comparing the md5sum values.But i am finding it hard to decide how to complete the code after redirecting the output of calculation of md5sum to a file. Any help will be appreciated .
 
Old 02-21-2011, 11:44 AM   #2
corp769
LQ Guru
 
Registered: Apr 2005
Location: /dev/null
Posts: 5,818

Rep: Reputation: 1007Reputation: 1007Reputation: 1007Reputation: 1007Reputation: 1007Reputation: 1007Reputation: 1007Reputation: 1007
Ok, so how are you trying to do this? In a script? In what distro (good to know)? You need to provide us with enough information to help you. Read the following link to know how to ask a question in these forums... http://www.linuxquestions.org/linux/...Ask_a_Question

As far as your problem, I would throw suggestions out, but I don't even know how you are trying to do it. Reply back and I will help out. Thanks.

Josh
 
Old 02-21-2011, 11:48 AM   #3
silvyus_06
Member
 
Registered: Oct 2010
Distribution: Ubuntu 10.04 , Linux Mint Debian Edition , Microsoft Windows 7
Posts: 390

Rep: Reputation: 50
if there are lots of big files e.g. file>1GB then you have some problems..

though if they are small files perhaps this code might help you. it first checks the exact size and if the sizes match then it continues to check md5sums ... ... prob is it only finds them,,

Code:
find -not -empty -type f -printf "%s\n" | sort -rn | uniq -d | xargs -I{} -n1 find -type f -size {}c -print0 | xargs -0 md5sum | sort | uniq -w32 --all-repeated=separate
also this is a script check it out

http://www.unix.com/shell-programmin...od-enough.html this guy already did the script

EDIT: and also if you are interested in more there are more results in the google search i made http://www.google.com/search?ie=UTF-...ased+on+md5sum

Last edited by silvyus_06; 02-21-2011 at 11:50 AM.
 
Old 02-21-2011, 12:25 PM   #4
David the H.
Bash Guru
 
Registered: Jun 2004
Location: Osaka, Japan
Distribution: Arch + Xfce
Posts: 6,852

Rep: Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037
Check out fdupes.

http://www.ubuntugeek.com/how-to-fin...in-ubuntu.html
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
does tar or bzip2 squash duplicate or near-duplicate files? garydale Linux - Software 6 11-19-2009 04:43 PM
md5sum on a directory with lots of files. solarmax Linux - Software 5 08-11-2005 05:22 PM
how to repair suse md5sum files...? enigmatech Linux - Newbie 3 08-05-2005 11:29 AM
md5sum to check files? serendipitysdc Fedora - Installation 2 06-22-2004 08:12 PM
how to detect duplicate IP gsbarry Linux - Networking 2 02-21-2003 04:28 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 07:36 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration