LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Blogs > jere21
User Name
Password

Notices

Rate this Entry

Sorting pictures, removing duplicates

Posted 11-20-2010 at 08:37 AM by jere21
Updated 02-03-2012 at 03:49 PM by jere21

EDIT 2012-02-03:
In Debian I just discovered "duff". A tool to find duplicate files
---


All files with their md5sum
Code:
find . -type f -exec md5sum '{}' ';' > pics_complete.txt
Lists with only md5sum, once with duplicates removed
Code:
cat pics_complete.txt | sed -r "s|^(.*) .*|\1|"| sort > pics_sorted.txt
cat pics_complete.txt | sed -r "s|^(.*) .*|\1|"| sort -u > pics_sorted-uniq.txt
md5sum of duplicates
Code:
diff pics_sorted.txt pics_sorted-uniq.txt | grep \< | sed "s|< ||" > pics.duplicates.txt
List of all duplicate files (sorted by their md5sum)
Code:
grep -e "$(cat pics.duplicates.txt)" pics_complete.txt | sort
Posted in Uncategorized
Views 3355 Comments 2
« Prev     Main     Next »
Total Comments 2

Comments

  1. Old Comment
    nice tip... I never would have thought this would be a problem... but then, I treat my pictures much like my music files and put them all in one directory... If I have duplicate scenes, it's because they are cropped differently or re-sized.

    What would be nice is an optical recognition program that would have a tolerance to avoid the pitfalls of image compression.
    Posted 11-22-2010 at 03:02 PM by lumak lumak is offline
  2. Old Comment
    Quote:
    Originally Posted by lumak View Comment
    What would be nice is an optical recognition program that would have a tolerance to avoid the pitfalls of image compression.
    I use md5sum for comparing the pictures, so I can only tell which files are 100% identical. What you suggest would require an application that detects the contents of pictures. I assume that would need a lot computing power.
    Posted 11-22-2010 at 03:45 PM by jere21 jere21 is offline
 

  



All times are GMT -5. The time now is 03:28 PM.

Main Menu

My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration