Identify Duplicate Files by File Name

rickh · 06-05-2006, 11:57 PM

I have used 'mp3report' to make a text file listing all my MP3s suitable for import into OO-Calc. It looks like the excerpt shown below. In spite of the appearance of this section, the file is not sorted by Artist/Title. (Fields 2 and 3, if you will.) Obviously, if the actual files were in one directory, there would be no duplicates, but they're in about 30 different directories.

Code:

0001|April Stevens|Teach Me Tiger|01|2.19 MB|128 kbps|44.1 kHz|02:23
0002|Billy Joe Shaver|I'm Going Crazy in 3-4 Time|01|3.23 MB|128 kbps|44.1 kHz|03:31
0003|Billy Joe Shaver|Old Chunk of Coal|01|2.34 MB|128 kbps|44.1 kHz|02:33
0004|Billy Joe Shaver|Serious Souls|01|2.00 MB|128 kbps|44.1 kHz|02:10
0005|Blue Velvet Band|Hitch Hiker|01|3.32 MB|160 kbps|44.1 kHz|02:53
0006|Blue Velvet Band|Ramblin' Man|01|3.25 MB|160 kbps|44.1 kHz|02:50
0007|Blue Velvet Band|Sittin' on Top of the World|01|3.89 MB|160 kbps|44.1 kHz|03:23
0008|Blue Velvet Band|Somebody Else You've Known|01|2.83 MB|160 kbps|44.1 kHz|02:28
0009|Blue Velvet Band|Sweet Moments|01|2.86 MB|160 kbps|44.1 kHz|02:29
0010|Blue Velvet Band|The Knight Upon the Road|01|4.21 MB|160 kbps|44.1 kHz|03:40
0011|Blue Velvet Band|Weary Blues From Waitin'|01|3.51 MB|160 kbps|44.1 kHz|03:03
0012|Blue Velvet Band|You'll Find Her Name Written There|01|3.16 MB|160 kbps|44.1 kHz|02:45
0013|Bonnie Raitt|Let me In|01|3.36 MB|128 kbps|44.1 kHz|03:40
0014|Burl Ives|Time|01|2.70 MB|128 kbps|44.1 kHz|02:57
.
.
.

I would like to have a program or script that will scan the entire file and identify any Artist/Title duplicates.

If necessary, I could of course create another file that contains only that (Artist/Title) data... but if I could scan the file as is, that would be even better.

I've looked thru google and these forums, but suggestions for identifying duplicate files I've seen are based on file size or some sort of hashing scheme rather than file names. I could easily have the same song twice with different file sizes so that won't work.

Up to now, I've moved such a file to Windows, and used MS Access to identify the duplicates. For obvious reasons, I'd like to stop doing that. Suggestions would be appreciated.

CroMagnon · 06-06-2006, 01:12 AM

I'm pretty sure you're going to need human intervention to positively ID all the dupes, but if the artist and titles are identical for many of them, you could do this:

Code:

cut -d '|' -f 2,3 mp3listfile | sort > list1.txt
cut -d '|' -f 2,3 mp3listfile | sort | uniq > list2.txt

then "diff list1.txt list2.txt" would show you the duplicates (if there were three of a given mp3, you would see two lines).

bigearsbilly · 06-06-2006, 04:06 AM

I've got a perl script at home that positively
identifies duplicate files in directory trees using checksums.
it will work for this too.

I'll dig it out later if you like.

rickh · 06-06-2006, 06:52 AM

Quote:

There are not many. Maybe 2 or 3 at the most out of 5000+ files. This technique may work tho ... I'm studying 'man cut' now.

Quote:

identifies duplicate files in directory trees using checksums.

This will definitely not work for reasons described in the original post.

bigearsbilly · 06-06-2006, 07:01 AM

something like this, you can easy do
this will find duplicate lines

Code:

sort | uniq -d

you'll have to strip some fields out you don't want

rickh · 06-06-2006, 07:13 AM

Hmmm! ...and, according to the man page, sort has a --unique option. May even be able to skip the '| uniq'

I'll have to rewrite the file to include only Artist/Title, but it looks like I can use some the 'cut' command described above to do that.

Thanks.

bigearsbilly · 06-06-2006, 07:15 AM

no, you are trying to find the duplicates aren't you?
sort -u will remove the duplicates.

something like this:

Code:

 cut -f2,3 -d\| files.lis | sort| uniq -d  > dupes
 grep -f dupes files.lis

rickh · 06-06-2006, 07:25 AM

All right! Those commands, exactly as you gave them, work perfectly.

Code:

debian:~$ grep -f dupes mp3.txt
3612|June Carter Cash|Meeting in the Air|18|1.88 MB|128 kbps|44.1 kHz|02:02
3769|June Carter Cash|Meeting in the Air|19|1.88 MB|128 kbps|44.1 kHz|02:03
4684|Cluster Pluckers|Keep on the Sunny Side|25|2.64 MB|128 kbps|44.1 kHz|02:52
4981|Cluster Pluckers|Keep on the Sunny Side|27|2.65 MB|128 kbps|44.1 kHz|02:53
debian:~$

Thanks again.

archtoad6 · 06-21-2006, 03:04 PM

They sure do.

Here is a possibly simpler way, no stripping fields, no intermediate files:

Code:

sort -t\| -k2,3 files.lis  | uniq -Dt\| -f1 -W2  | less

And here is my test corpus:

Code:

0001|April Stevens|Teach Me Tiger|01|2.19 MB|128 kbps|44.1 kHz|02:23
0002|Billy Joe Shaver|I'm Going Crazy in 3-4 Time|01|3.23 MB|128 kbps|44.1 kHz|03:31
0003|Billy Joe Shaver|Old Chunk of Coal|01|2.34 MB|128 kbps|44.1 kHz|02:33
0004|Billy Joe Shaver|Serious Souls|01|2.00 MB|128 kbps|44.1 kHz|02:10
0013|Bonnie Raitt|Let me In|01|3.36 MB|128 kbps|44.1 kHz|03:40
0014|Burl Ives|Time|01|2.70 MB|128 kbps|44.1 kHz|02:57
3612|June Carter Cash|Meeting in the Air|18|1.88 MB|128 kbps|44.1 kHz|02:02
3769|June Carter Cash|Meeting in the Air|19|1.88 MB|128 kbps|44.1 kHz|02:03
xxxx|June Carter Cash|Keep on the Sunny Side|25|2.64 MB|128 kbps|44.1 kHz|02:52
4684|Cluster Pluckers|Keep on the Sunny Side|25|2.64 MB|128 kbps|44.1 kHz|02:52
4981|Cluster Pluckers|Keep on the Sunny Side|27|2.65 MB|128 kbps|44.1 kHz|02:53
0011|Blue Velvet Band|Weary Blues From Waitin'|01|3.51 MB|160 kbps|44.1 kHz|03:03
0012|Blue Velvet Band|You'll Find Her Name Written There|01|3.16 MB|160 kbps|44.1 kHz|02:45

Notice the made up line "xxxx" to ensure that 2 artists doing the same title isn't called a duplicate.