Duplicates of photos

hkjz · 04-11-2019, 04:12 PM

Dear All,

lets say that there is directory with 5 000.jpg files.
Many of those files are duplicates of each other (photos), but each file have unique name.

Any ideas how this problem can be resolved without sorting this manually?

Thank you!

hydrurga · 04-11-2019, 06:07 PM

Have you tried something like fslint to find the duplicates?

syg00 · 04-11-2019, 07:53 PM

No doubt a continuation of your previous thread. There are several tools and innumerable threads about similar. If they are all in the one directory you are fortunate and can allow the tool to simply delete dups without regard to which to keep (i.e. which is the "original").

RandomTroll · 04-11-2019, 08:57 PM

I'd create an md5sum for each, make a database associating each md5sum with file name, search on duplicate md5sums.

Thymox · 04-12-2019, 01:08 PM

Quote:

Originally Posted by RandomTroll

I'd create an md5sum for each, make a database associating each md5sum with file name, search on duplicate md5sums.

Which is pretty much exactly what "fdupes" does, so no need to reinvent the wheel.

RandomTroll · 04-12-2019, 11:36 PM

Quote:

Originally Posted by Thymox

Which is pretty much exactly what "fdupes" does, so no need to reinvent the wheel.

Groovy! Never heard of 'fdupes', not in Slackware.

LQ496873 · 04-13-2019, 03:14 AM

I had over 25 000 images with a decent number of duplicates. I think it was fslint I used to sort through them. It worked really well. Just had a quick look at the folder... I guess I removed about 3000 duplicates in about an hour.

pan64 · 04-13-2019, 03:48 AM

not to speak about the tool named dupfinder (http://dupfinder.com/)

hkjz · 04-14-2019, 12:16 PM

Code:

fdupes

is what i used, it worked,
though i had to manually choice which file suppose to be saved from the list - for every single duplicates

1640 times i hit '1' and then 'enter', it was quite a thing hehe

Thymox · 04-16-2019, 02:08 PM

Quote:

Originally Posted by hkjz

Code:

fdupes

is what i used, it worked,
though i had to manually choice which file suppose to be saved from the list - for every single duplicates

1640 times i hit '1' and then 'enter', it was quite a thing hehe

From the man page of fdupes:

Code:

       -d --delete
              prompt user for files to  preserve,  deleting  all  others  (see
              CAVEATS below)

       -N --noprompt
              when  used  together  with  --delete, preserve the first file in
              each set of duplicates and delete the others  without  prompting
              the user

So this would have solved that:

Code:

$ fdupes -rdN ./

hkjz · 04-16-2019, 02:17 PM

beauty