I want to list directories recursively but I having trouble with the different file types.
I want the list to group the the extensions together and I'm getting frustrated.
This does what I want, except for grouping them:
Everything I've tried so far makes a mess.
Which option will do this?

Not all files have extensions and some have several - assuming you mean a bunch of characters after a dot?

Music directory with mp3, flac and jpg files
I'm trying to get a straight listing so I can actually get rid of duplicates.

None of the programs like fslint, fdupes or shredder [I]can even find duplicates[I] and there's a lot of them - i guess there must be slight differences in the file names (which I can't see) that confuse these programs. (Or are they neurotic about all the details and files must be exact dupes to succeed?)

Looking at about 5000 files here in about 150 directories, and I'm sick of dealing with duplicates.
I need a plain list so I can manually do this without having to go into every freakin' directory one-by-one.

Have you tried sorting the output?

Edit -- Nah. That didn't work...well it does show you duplicates, but doesn't tell you where they are.

As per "ls --help", the -X option will sort by extension.

However, if you use -R to list subdirectories recursively, that will output each directory path, then the sorted files within the directory.

One workaround is to use "find" to handle the recursion and pipe the results through "ls -1X" to sort by extension.

1 members found this post helpful.

@boughtonp - tell me more please

the man page for find is even scarier

If you're trying to detect duplicate filenames, you might find this more useful than sorting by extension:

Then you can check for sequential lines with the same prefix \n[^\t]+\t to locate the duplicates

(Obviously if your filenames might have tabs in, use a different delimiter.)

[chuckle]

It doesn't much make any difference to UNIX/Linux but some people still expect there to be a "bunch of characters after a dot".[1]

I wonder how they deal with ".tar.bz2"? "Two dots? Argh!!"

But I digress...


[1] -- Except graphical file managers that still want to associate file extensions to applications.

Graphical file managers look at the mime-type of a file - this can occasionally include looking at what follows a dot in the name.

I think that finding all the files in your music directory trees and obtaining their checksums would be a much better means of finding true duplicates.
That'll allow you to find duplicates that, for whatever reason, have different filenames but are actually the same file. You should be able to visually examine the sorted list of checksums+filenames so see the duplicates. If there are a ton of duplicates, you can extract the checksums, sort them and run them through uniq(1) to show the duplicates and use each record of that list to grep the original file containing checksums and filepaths.

If you obtain a list of filenames, you'd still need to obtain information about each file and, say, keep the one with the highest bit rate. If you have a mixture of MP3, Ogg, and FLAC, the inspection gets a [ahem] tad more complicated. Personally, I'd keep anything in FLAC format. Period. Then compare any duplicate MP3s and Oggs to see what bit rate they were encoded with and keep the one with the higher bit rate. If you have an MP3 and an Ogg of the same song and the bit rates are close, maybe give each a listen and see which one sounds "better" to your ears. If they're the same bit rate, I'd keep the Ogg. I once found I had a few duplicates that both sounded "OK" but I ran them through "baudline". That showed pretty clearly that one of them had had a low-pass filter applied while it was encoded. That's the one I tossed.

Personally, I don't get too worked up over a few duplicates. I know that some of the 23,000+ (and growing) music files I have on disk are duplicates but I'm not going to quibble if a song appears in an artist's album-specific subdirectory and also in a greatest hits subdirectory. Sure I'd save a little space by replacing all but one of those with a symbolic link but I'd have to weigh whether it's worth the time tracking them down.

Good luck...

A lot of the time they seem to simply throw up their hands and list darned near everything under /usr/bin as a candidate for opening the file. Fortunately for the OP, music files with no file extensions are rare. I was the one that brought up file managers, not the OP so let's reset our focus. :^D