Help answer threads with 0 replies.
Go Back > Forums > Linux Forums > Linux - General
User Name
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.


  Search this Thread
Old 02-24-2010, 11:49 AM   #1
LQ Newbie
Registered: Feb 2010
Location: ireland
Posts: 2

Rep: Reputation: 0
Question Dupe File Finder

Hi all,

Was wondering could I get some help, have a piece of code:;
echo "#! /bin/sh" > $OUTF;
find "$@" -type f -exec md5sum {} \; |
sort --key=1,32 | uniq -w 32 -d --all-repeated=separate |
sed -r 's/^[0-9a-f]*( )*//;s/([^a-zA-Z0-9./_-])/\\\1/g;s/(.+)/#rm \1/' >> $OUTF;
chmod a+x $OUTF; ls -l $OUTF

This finds duplicate files but how would I go about deleting the files found?

jdandcocacola is online now Report Post Edit/Delete Message
Old 02-24-2010, 02:14 PM   #2
Senior Member
Registered: Dec 2008
Location: Louisville, OH
Distribution: Debian, CentOS, Slackware, RHEL, Gentoo
Posts: 1,833

Rep: Reputation: 164Reputation: 164
I take it you didn't write this code heh.

Assuming it finds what you ~WANT~ it to find (which looking at it I suspect it's going to find a whole lot more than you want) deleting it would be as easy as changing the code to this (keep in mind, I would never execute this script or the resultant simple bash script to remove the files-- the results imo are not worthwhile or useful.)--

chmod o+x && ./ /path/to/dir/to/check/for/dupes
echo "#! /bin/sh" > $OUTF;
find "$@" -type f -exec md5sum {} \; |
sort --key=1,32 | uniq -w 32 -d --all-repeated=separate |
sed -r 's/^[0-9a-f]*( )*//;s/([^a-zA-Z0-9./_-])/\\\1/g;s/(.+)/rm \1/' >> $OUTF;
chmod o+x $OUTF; ls -l $OUTF
then run...
However, I DON'T recommend running this and before you ever run a script like this ensure you have good backups. Seriously.

I don't think this is going to do what you want... you can cat the file out to see what it is going to do exactly...

Last edited by rweaver; 02-24-2010 at 02:17 PM.
Old 02-25-2010, 04:30 AM   #3
LQ Newbie
Registered: Feb 2010
Location: ireland
Posts: 2

Original Poster
Rep: Reputation: 0

Basically what I am trying to achieve is:

To create a script that is able to find duplicate files within a filesystem from a given startpoint.

So when I use my code it pulls up all the duplicate files into

So from rem-duplicates is it possible to delete the duplicated files leaving only the original copy?

Many Thanks,



Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Similar Threads
Thread Thread Starter Forum Replies Last Post
Subtle dynamic leading to dupe threads? Simon Bridge LQ Suggestions & Feedback 7 07-27-2009 02:12 AM
remove dupe wireless interface amp_man Ubuntu 2 03-31-2005 02:03 PM
Need a dupe-killer, please dhave Linux - Software 1 01-17-2005 12:49 PM
Problem installing JAVAUNIX. needed for Duplicate File Finder bglnelissen Linux - Software 1 12-22-2004 04:53 PM
mysql dupe entry error, how to avoid? hydro Linux - Software 0 09-20-2003 12:08 PM > Forums > Linux Forums > Linux - General

All times are GMT -5. The time now is 10:28 AM.

Main Menu
Write for LQ is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration