Look for duplicates in folder tree A and folder tree B, then delete the duplicates only from A.

grumpyskeptic · 10-14-2018, 03:21 AM

I would like to find identical files which are in folder or directory tree A and in folder or directory tree B. (The tree might consist of just one folder). Then I want to delete the duplicated files from tree A only, and not delete anything from tree B.

What is the best way to do this please?

The FS program lists duplicates, but there is no method that I am aware of for getting it to highlight the duplicate files in tree A only. You would have to go through the list manually and manually select those files in tree A, which would be too big a job to do if a large number of files are involved.

Thanks.

Edit: I should have typed FSlint, not FS.

fatmac · 10-14-2018, 03:35 AM

Maybe something like this will work, (not tested).

Code:

ls -a A > A.lst;ls -a B > B.lst; cmp A.lst B.lst > duplist

Plus,maybe.

Code:

grep A

syg00 · 10-14-2018, 03:57 AM

I know not of this "FS" program of which you speak, but I refuse to allow dup finders to automatically delete files. It will delete things you don't want it to. Maybe not this time, but it will happen.

fdupes and fslint are probably the best known CLI ones, but I use duff as it allows me to put decent regex into th mix. But I always generate a list I can script later into a "rm" loop.
Much safer, and a trivial to knock up.

l0f4r0 · 10-15-2018, 02:08 AM

Quote:

Originally Posted by fatmac

Maybe something like this will work, (not tested).

Code:

ls -a A > A.lst;ls -a B > B.lst; cmp A.lst B.lst > duplist

Plus,maybe.

Code:

grep A

I think it could be dangerous to implement such an algo based on filenames only...except if this is OP's requirement (but the OP didn't define what he called "duplicates/identical files" though...is it same filenames and/or same contents?).
Maybe a safer approach would be to go with hashes? Otherwise with a specialized tool but I have no experience with any...

descendant_command · 10-15-2018, 02:35 AM

fdupes is my preferred tool.
It can delete directly or produce a list or further inspection/processing.

Pay attention to syg00's warning too.

grumpyskeptic · 10-27-2018, 03:26 AM

Do any Linux file managers include being able to find duplicate files? If they do, then using them may be the simplest option.

Thanks.

MadeInGermany · 10-27-2018, 08:51 PM

With find and a while loop.
By cd'ing into one of the folders one works with relative file names and can simply prepend the other folder name.

Code:

A=A
B=B
(
cd "$A" &&
find . -type f
) |
while IFS= read f
do
  if [ -f "$B/$f" ] && cmp "$f" "$B/$f"
  then
    echo rm "$f"
  fi
done

BW-userx · 10-27-2018, 10:23 PM

rdfind??

I'm going to check out MadeInGermany script, that looks interesting.