LinuxQuestions.org
Help answer threads with 0 replies.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices


Reply
  Search this Thread
Old 02-24-2010, 11:49 AM   #1
jdandcocacola@hotmail.co
LQ Newbie
 
Registered: Feb 2010
Location: ireland
Posts: 2

Rep: Reputation: 0
Question Dupe File Finder


Hi all,

Was wondering could I get some help, have a piece of code:

OUTF=rem-duplicates.sh;
echo "#! /bin/sh" > $OUTF;
find "$@" -type f -exec md5sum {} \; |
sort --key=1,32 | uniq -w 32 -d --all-repeated=separate |
sed -r 's/^[0-9a-f]*( )*//;s/([^a-zA-Z0-9./_-])/\\\1/g;s/(.+)/#rm \1/' >> $OUTF;
chmod a+x $OUTF; ls -l $OUTF


This finds duplicate files but how would I go about deleting the files found?

jdandcocacola
jdandcocacola is online now Report Post Edit/Delete Message
 
Old 02-24-2010, 02:14 PM   #2
rweaver
Senior Member
 
Registered: Dec 2008
Location: Louisville, OH
Distribution: Debian, CentOS, Slackware, RHEL, Gentoo
Posts: 1,833

Rep: Reputation: 164Reputation: 164
I take it you didn't write this code heh.

Assuming it finds what you ~WANT~ it to find (which looking at it I suspect it's going to find a whole lot more than you want) deleting it would be as easy as changing the code to this (keep in mind, I would never execute this script or the resultant simple bash script to remove the files-- the results imo are not worthwhile or useful.)--

Code:
chmod o+x find-dupes.sh && ./find-dupes.sh /path/to/dir/to/check/for/dupes
Code:
OUTF=rem-duplicates.sh;
echo "#! /bin/sh" > $OUTF;
find "$@" -type f -exec md5sum {} \; |
sort --key=1,32 | uniq -w 32 -d --all-repeated=separate |
sed -r 's/^[0-9a-f]*( )*//;s/([^a-zA-Z0-9./_-])/\\\1/g;s/(.+)/rm \1/' >> $OUTF;
chmod o+x $OUTF; ls -l $OUTF
then run...
Code:
./rem-duplicates.sh
However, I DON'T recommend running this and before you ever run a script like this ensure you have good backups. Seriously.

I don't think this is going to do what you want... you can cat the rem-duplicates.sh file out to see what it is going to do exactly...

Last edited by rweaver; 02-24-2010 at 02:17 PM.
 
Old 02-25-2010, 04:30 AM   #3
jdandcocacola@hotmail.co
LQ Newbie
 
Registered: Feb 2010
Location: ireland
Posts: 2

Original Poster
Rep: Reputation: 0
Unhappy

Basically what I am trying to achieve is:

To create a script that is able to find duplicate files within a filesystem from a given startpoint.

So when I use my code it pulls up all the duplicate files into rem-duplicates.sh

So from rem-duplicates is it possible to delete the duplicated files leaving only the original copy?

Many Thanks,

jdandcocacola
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Subtle dynamic leading to dupe threads? Simon Bridge LQ Suggestions & Feedback 7 07-27-2009 02:12 AM
remove dupe wireless interface amp_man Ubuntu 2 03-31-2005 02:03 PM
Need a dupe-killer, please dhave Linux - Software 1 01-17-2005 12:49 PM
Problem installing JAVAUNIX. needed for Duplicate File Finder bglnelissen Linux - Software 1 12-22-2004 04:53 PM
mysql dupe entry error, how to avoid? hydro Linux - Software 0 09-20-2003 12:08 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - General

All times are GMT -5. The time now is 10:28 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration