Dupe File Finder

jdandcocacola@hotmail.co · 02-24-2010, 11:49 AM

Hi all,

Was wondering could I get some help, have a piece of code:

OUTF=rem-duplicates.sh;
echo "#! /bin/sh" > $OUTF;
find "$@" -type f -exec md5sum {} \; |
sort --key=1,32 | uniq -w 32 -d --all-repeated=separate |
sed -r 's/^[0-9a-f]*( )*//;s/([^a-zA-Z0-9./_-])/\\\1/g;s/(.+)/#rm \1/' >> $OUTF;
chmod a+x $OUTF; ls -l $OUTF

This finds duplicate files but how would I go about deleting the files found?

jdandcocacola
jdandcocacola is online now Report Post Edit/Delete Message

rweaver · 02-24-2010, 02:14 PM

I take it you didn't write this code heh.

Assuming it finds what you ~WANT~ it to find (which looking at it I suspect it's going to find a whole lot more than you want) deleting it would be as easy as changing the code to this (keep in mind, I would never execute this script or the resultant simple bash script to remove the files-- the results imo are not worthwhile or useful.)--

Code:

chmod o+x find-dupes.sh && ./find-dupes.sh /path/to/dir/to/check/for/dupes

Code:

OUTF=rem-duplicates.sh;
echo "#! /bin/sh" > $OUTF;
find "$@" -type f -exec md5sum {} \; |
sort --key=1,32 | uniq -w 32 -d --all-repeated=separate |
sed -r 's/^[0-9a-f]*( )*//;s/([^a-zA-Z0-9./_-])/\\\1/g;s/(.+)/rm \1/' >> $OUTF;
chmod o+x $OUTF; ls -l $OUTF

then run...

Code:

./rem-duplicates.sh

However, I DON'T recommend running this and before you ever run a script like this ensure you have good backups. Seriously.

I don't think this is going to do what you want... you can cat the rem-duplicates.sh file out to see what it is going to do exactly...

jdandcocacola@hotmail.co · 02-25-2010, 04:30 AM

Basically what I am trying to achieve is:

To create a script that is able to find duplicate files within a filesystem from a given startpoint.

So when I use my code it pulls up all the duplicate files into rem-duplicates.sh

So from rem-duplicates is it possible to delete the duplicated files leaving only the original copy?

Many Thanks,

jdandcocacola