Hey,

Does anyone know of an app or maybe a shell trick that would let me remove duplicates in a directory? I moved my mail server to another computer and due to a stupid typo on my part I now have 2 of every email in 6 mailboxes which is leading to some curious phone calls from people

It's not a life threatening situation but if anyone has any suggestions I'd love to hear them

TIA

Not sure of the precise details of your situation. Are these files with same name in different dirs or files with different names but the same contents? Etc. Anyway - simplest thing for some situations might be http://www.homepages.lu/pu/fdups.html

This is a script I have been working on that seems to work with image files. It's not a perfect script and pobably not the best way written, but you maybe able to mess around with it a bit to work with email messages. I have made it by default move files to a duplicate directory rather than deleting them, just so you can check before finally deleting them

########################################################
#!/bin/sh

OUTPUT=duplicates
DUPES=rem.sh

# remove any previous files
rm -rf $OUTPUT
rm -rf $DUPES

find . "$@" -type f -print0 |
xargs -0 -n1 md5sum |
sort --key=1,32 | uniq -w 32 -d --all-repeated=separate | \
sed -e '$!N; /^\(.*\)\n\1$/!P; D' >> $OUTPUT

cat > $DUPES<<'EOF'
#!/bin/sh
ME=`whoami`
DUPE_DIR=~/$ME/duplicates
if [ ! -d $DUPE_DIR ]; then
mkdir -p $DUPE_DIR
fi
FOUND='
EOF

cat $OUTPUT | sed -n '/^$/{p;h;};/./{x;/./p;}' >> $DUPES
cat >> $DUPES <<'EOF'
'
for x in $FOUND; do
mv $x $DUPE_DIR
done
EOF

sh rem.sh
########################################################

Just test this first in a test directory before actually using it, not sure how good it will work for you.

sorry double post