Register a domain and help support LQ
Go Back > Forums > Non-*NIX Forums > Programming
User Name
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.


  Search this Thread
Old 05-25-2010, 08:47 PM   #1
LQ Newbie
Registered: May 2010
Location: Utah
Distribution: Debian 5
Posts: 1

Rep: Reputation: 0
Script to remove duplicate jpg files

Thanks y'all for the great script and explanation. This helped a lot in my own project. I thought I'd share the efforts.

The project is this: I've got lots of duplicate JPGs from all the family members who've named the same photo with different names. Since md5sum generates a "fingerprint" based on the file contents, not the name, I want to use the md5sum of each jpg to uniquely name each photo and also remove exact duplicates.

It has the following flaws:
0) it doesn't handle certain non-alphanumerics
1) it keeps both photo-shopped and unaltered photos (different md5s)
2) it (currently) doesn't preserve descriptive filenames.

(For me, removal of duplicates is more important than keeping the filenames. I may change that to concatenate the md5 and the filename.)

Please note that the commented "rename" command should be used to strip non-aphanumerics from the file names, and the script should be launched with the commented "find" command.

### prepare files for renaming by removing alphanumerics
# rename -vf 's/[^a-zA-Z0-9]//g' *.jpg   

# then launch this script find . -type f -maxdepth 1 -name "*jpg" -exec ./ {} \;

if [ "$1x" != "x" ] ; then
  new_filename=`(/usr/bin/md5sum "$filename") | cut -f 1 -d ' '`
  echo "mv ${filename} $new_filename"
  mv $filename $jpg_filename

Last edited by pixellany; 05-26-2010 at 09:21 AM.
Old 05-26-2010, 09:22 AM   #2
LQ Veteran
Registered: Nov 2005
Location: Annapolis, MD
Distribution: Arch/XFCE
Posts: 17,802

Rep: Reputation: 735Reputation: 735Reputation: 735Reputation: 735Reputation: 735Reputation: 735Reputation: 735
Welcome to LQ!!

Please don't jump into 6-year old threads---I moved this into its own thread


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off

Similar Threads
Thread Thread Starter Forum Replies Last Post
Script to copy specific directory based on variable to folder with that variable name fluxburn Programming 7 01-07-2010 08:59 PM
Problem with bash script - variable name within variable name steven.c.banks Linux - Newbie 3 03-10-2009 04:08 AM
Calling perl script and passing variable from php script hosea Programming 5 10-21-2008 09:01 AM
Help: removing a variable substring from a string variable in sh script gnparsons Programming 2 06-04-2008 06:21 PM
setting a variable variable in a script... this works, but could it be more elegant? pwc101 Programming 3 08-18-2006 12:23 PM

All times are GMT -5. The time now is 03:28 AM.

Main Menu
Write for LQ is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration