-   Linux - General (
-   -   Script to find duplicate files within one or more directories (

peter88 12-06-2006 01:50 AM

Script to find duplicate files within one or more directories
Hi, has anyone got a script which does more or less the following please:

I have 2 directories with say about 1500 photos in each dir. Now, I know that some of the photos are the same even if they have different timestamps and names. To avoid laboriously doing a visual comparison, what I would like to do is run a scrit (bash or other) which will list the files which are similar. It could be that within each dir there are duplicates with differing names or/and that there are duplicates between the 2 dirs. Actually in theory one should be able to pass $1 $2 $3... as directories to compare, it need not be limited to one or two directories. Also, here I am talking of image files, however I woudl like the program to be generic to be able to compare any type of file in the directories being compared and I would guess that if using a md5 hash signature then the type of file is immaterial (please tell me if I'm talking nonse, I won't be offended :) ).

I was thinking a script could do this by creating a md5 or other hash checksum of the files in the directories and then compare each file to the stored checksums to create a list of files which have the same md5 value and hence should be identical. Perhaps someone knows existing functions with the unix/linux suite of tools such as the various shells or awk, perl, php, python etc. which I am not aware of..

If someone knows a progrom or a script I could run under w32 (WXP say) then that would be useful too as I can perform the task on either system and then move the files across if necessary. In anycase as I use both environments it would be useful to know how to do it in both.

Any advise appreciated.


psisquare 12-06-2006 07:33 PM

Here's a quick hack with bash and awk:

md5sum dir1/* dir2/* | sort | awk '{ if(lastmd5==$1) print lastfile, $2; lastmd5=$1; lastfile=$2; }'
If you need something that also runs on Windows, you can do essentially the same with Ruby, Python or Perl - whichever language you know best.

Also, note that there's a slight twist to using hashes for this: If the images were not just copied between the directories, but re-encoded, they can have different md5sums even though they look the same. If that's the case, you could try GQview, which has a "find duplicates" feature (that I've never tested).

psisquare 12-06-2006 08:10 PM

Oow well, I'm trying to learn Ruby anyway and this looks like a nice exercise. :)


require 'digest/md5'

list = {}

Dir.foreach(ARGV[0]) do |file|
  path = File.join(ARGV[0], file)
  next if not File.file? path
  list[Digest::MD5.digest(] = path

Dir.foreach(ARGV[1]) do |file|
  path = File.join(ARGV[1], file)
  next if not File.file? path
  if otherpath=list[Digest::MD5.digest(]
      puts path + "\t" + otherpath

frob23 12-06-2006 08:36 PM

Here is a little bit more complicated script. But what it returns in exchange for complexity is that it's less intense than the quick hack above. Note: I wrote this on FreeBSD so things might be a little bit off. You may want to check that the -ls option to find returns the size in the 7th spot and the filename in the 11th. Modify those values if they don't.

This program checks all subdirectories below the points you request. It also only sums the files which have matching sizes (which will very likely reduce the load tremendously as you're not hashing every file).



#md5prog="md5 -r"

docompare() {
        ${md5prog} ${*} | sort | awk 'BEGIN{
{if (prevsum==$1) {
        printf "%s ",previous;
else {
        if (dup==1) {
                printf "%s\n\n",previous;

mainloop() {
        if [ ${new} -eq ${old} ]; then
                count=`echo "${count}+1" | bc`;
                files="${files} ${2}";
                if [ ${count} -gt 1 ]; then
                        docompare ${files};

if [ x"${1}" = "x" ]; then
        echo "Usage: `basename ${0}` {path1} [path2 ...]";
        exit 1;

find ${*} -ls | awk '{print $7 " " $11}' | sort -n | while read LINE
        mainloop $LINE

Edit: A couple lines above might look odd to the more experienced Bash programmers because this script is /bin/sh compatible. And I couldn't use some of the neat little tricks you might know about.

psisquare 12-07-2006 10:29 AM

Fearing that my clumsy and inefficient hack might give Ruby a bad reputation, I'll give you a more elegant version along the lines of my bash hack:

require 'digest/md5'; last=[]
  map{|f| [Digest::MD5.digest(, f]}.sort_by{|p| p[0]}.
  each{|p| puts last[1]+"\t"+p[1] if p[0]==last[0]; last=p}

After reading frob23's post, I also came up with the following one. It's the fastest solution I've tested so far, also comparing sizes first, does a recursive search of an arbitrary number of directories and adds some error checking:

require 'digest/md5'
(puts "Usage: #{$0} {dir1} [dir2 ...]"; exit) if ARGV.empty?
sizes={}; prev=[]; dup=false

Dir[File.join("{#{ARGV.join(',')}}","**","*")].select{|f| File.file?(f)}.
        each{|f| if sizes[size=File.size(f)] then sizes[size].push f else sizes[size] = [f] end}

sizes.each do |size,files|
        next if files.length==1{|f| [Digest::MD5.digest(, f]}.sort_by{|p| p[0]}.each do |p|
                if p[0]==prev[0]
                        prev[1] += "\t" + p[1]
                        dup = true
                        puts prev[1] if dup
                        dup = false
                        prev = p

peter88 12-10-2006 02:37 AM

Thanks a lot for your suggestions everyone, I will try them :)

FYI: for those interested in a possible W32 solution, here is a nice small program I found:

FINDDUPE: Duplicate file detector and eliminator:

Tortanick 12-10-2006 05:17 AM


Originally Posted by frob23
Edit: A couple lines above might look odd to the more experienced Bash programmers because this script is /bin/sh compatible. And I couldn't use some of the neat little tricks you might know about.

Just out of curiosity, why not just say #! /bin/bash rather than /bin/sh and then use the tricks?

All times are GMT -5. The time now is 08:53 AM.