App to find and remove duplicate images?

Zaraphrax · 12-13-2010, 06:17 PM

Hi all,

I am looking for a Linux app that can find and remove duplicate images (with different filenames if that's at all possible).

Someone suggested fdupes but I'm interested in alternatives.

Thanks.

GlennsPref · 12-13-2010, 07:40 PM

Hi, I use fslint, but it works by name and size,

Hope this helps, Regards Glenn

Zaraphrax · 12-13-2010, 09:03 PM

Quote:

Originally Posted by GlennsPref

Hi, I use fslint, but it works by name and size,

Hope this helps, Regards Glenn

That looks like it'll do the job for some of the duplicates. Thanks! Does anyone have a tool that works via EXIF data?

Cheers.

devnull10 · 12-14-2010, 06:20 AM

Any reason why you can't use the find and diff commands?
Two nested loops using find (restrict to jpg files) then just do a diff on the two files in the inner-most loop. Make sure that they aren't the same file of course.
Whilst it may not be the most efficient algorithm, it uses tools already available to you.
Something like:

The other option may be that you create a hash of each file and then compare the hashes. I guess it depends on how big your files are and how many there are as to which you choose to do.

catkin · 12-14-2010, 06:28 AM

szboardstretcher described a versatile script to do things like that in this LQ post.

jschiwal · 12-14-2010, 06:34 AM

If the files themselves are identical, you could create an md5sum list, sort the list and use "uniq -w32 -D" to list the originals and duplicates.

find pictures/ -type f -iname "*.jpg" -exec md5sum '{}' \; >md5sumlist
sort md5sumlist >md5sumlist.s
uniq -w32 -D md5sumlist.s