LinuxQuestions.org
LinuxAnswers - the LQ Linux tutorial section.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices

Reply
 
LinkBack Search this Thread
Old 09-27-2004, 09:02 AM   #1
ryedunn
Member
 
Registered: Jul 2003
Location: Chicago
Distribution: Fedora, ubuntu
Posts: 458

Rep: Reputation: 30
searching for multiple files


What command can I run in order to search for duplicate files within a single directory and the sub directories?

Thank you,
R
 
Old 09-27-2004, 12:27 PM   #2
meblost
Member
 
Registered: May 2004
Location: At Keyboard
Distribution: Mandrake 10.0, SuSE 9.0
Posts: 114

Rep: Reputation: 15
How about

$> find . | awk -F / '{print $NF}' | sort | uniq -d

This will tell you which files have the same filename. This version won't tell you where the file is though.
 
Old 09-27-2004, 12:28 PM   #3
chrism01
Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.5, Centos 5.10
Posts: 16,093

Rep: Reputation: 1995Reputation: 1995Reputation: 1995Reputation: 1995Reputation: 1995Reputation: 1995Reputation: 1995Reputation: 1995Reputation: 1995Reputation: 1995Reputation: 1995
If you know the name of the file(s) you could use:
Code:
for file in `cat filelist.dat`
do
    find <topdir> -name $file -print  2>/dev/null
done
if you have the names in a file.
 
Old 09-27-2004, 12:32 PM   #4
meblost
Member
 
Registered: May 2004
Location: At Keyboard
Distribution: Mandrake 10.0, SuSE 9.0
Posts: 114

Rep: Reputation: 15
This will tell you where the file with the same name is located, but it will take longer


for file in $(find . | awk -F / '{print $NF}' | sort | uniq -d)
do
find . -name $file
done
 
Old 09-27-2004, 03:21 PM   #5
ryedunn
Member
 
Registered: Jul 2003
Location: Chicago
Distribution: Fedora, ubuntu
Posts: 458

Original Poster
Rep: Reputation: 30
for future refrence

To find duplicate files (by content, not name) in a directory tree at /home/user:

find /home/user -type f -exec md5sum {} \; | sort >dups

dups will hold the md5 checksums and names of all the files in the directory tree, sorted by checksum.
All files that have the same contents will be next to each other in dups, since they will have the same checksum.

To compare files in two or more directories, just include all the directories in the find command:

find /usr/src/linux-2.4.20-18.7 /usr/src/linux-2.4.20-19.7 -type f -exec md5sum {} \; | sort >dups

To remove the non-duplicates from a file (or output stream), you can use the uniq command. For example, to see only the lines from dups that have duplicates (matching only the checksums, which are the first 32 characters of each line):

uniq -D -w 32 dups | less

To learn more about sort and uniq, and other text utilities, type:

info textutils
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off
Trackbacks are Off
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
searching for files deept Programming 1 08-15-2005 11:54 AM
help searching files jorasmi Slackware 3 02-02-2005 04:06 AM
Searching files on Linux dasoslukos Linux - Software 2 10-24-2004 11:12 PM
Alsa searching for multiple cards gbell72 Linux - General 1 12-20-2003 06:07 PM
Searching for files in KDE 3.1? bolinux Linux - Newbie 1 10-10-2003 11:40 PM


All times are GMT -5. The time now is 03:39 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration