I have two directories, I want to know which files in the second directory also appear in the first and delete the duplicate in the second directory. Filenames might be different (so that rules out diff).

My problem is that various programs (such as fdupes and freedup) are very capable of finding duplicate files but randomly delete (or link) files from the first or the second directory.

Here an example with fdupes:


As you can see, the file in the third pair is removed from dir1 instead of from dir2. My aim is to have only files deleted from the dir1. I know that fdupes can't do this, as I emailed with the author.

By grepping the output on "dir2" and then removing all those entries by eg. feeding them to xargs.

You could use `find' to locate the files in the two directories, with -exec md5sum '{}' to produce an md5sum listing of each file.
Then you can use `uniq' to show the duplicates.

find /dir1 /dir2 -type f -exec md5sum '{}' \; | sort | uniq -w32 -D >duplicates

You could use grep to just return files from dir2/ and delete these files.

# make sure duplicates aren't in same directory. The following should be empty. If not, remove the entries from "dupicates" file
grep /dir2 duplicates | sort -w32 -D

#delete duplicates in /dir2
grep /dir2 duplicates | cut -d' ' -f3 | tr '\n' '\0' | xargs -0 rm -v