list directories where all foles are dumpicated (rmlint)

funkytwig · 03-31-2019, 03:34 PM

Hi, I want to scan a filesystem and list any directories which have been duplicated, I want to ignore date/time and ownership. Bu duplicated I mean dave the same files in them.

rmlint looks promising but not sure how to get it to just show a list of duplicated directories. fdfine and fdupes also seem promising but can't quite figure it out.

This is because some users have uploaded directory structures to multiple places but may of done it at different times from different users.

dugan · 03-31-2019, 04:29 PM

Quote:

Originally Posted by funkytwig

This is because some users have uploaded directory structures to multiple places but may of done it at different times from different users.

First idea I have:

Just list all the directories, and then sort the list by the base directory (the rightmost one) and its parent. Then you do the rest of the checking manually/visually.

Here's a quick Python implementation.

Code:

import os
def sortKey(root):
    ds = root.split('/')
    if len(ds) > 1:
        return '/'.join(ds[-2:])
    return ds[-1]


for root in reversed(sorted((root[0] for root in os.walk('.')), key=sortKey)):
    print(root)

berndbausch · 03-31-2019, 06:28 PM

Quote:

Originally Posted by funkytwig

Bu duplicated I mean dave the same files in them.

Do you mean files with the same name, or the same content, or really identical, i.e. same inode number?

syg00 · 03-31-2019, 10:47 PM

Quote:

Originally Posted by funkytwig

rmlint looks promising but not sure how to get it to just show a list of duplicated directories.

Have you read the doco ?. I found a section on just this in the tutorial listed on the github homepage.

rmlint is on my list of things to check-out - looks a great addition to the toolset.

funkytwig · 04-01-2019, 04:29 AM

Quote:

Originally Posted by berndbausch

Do you mean files with the same name, or the same content, or really identical, i.e. same inode number?

Just the same name and content. Can get a list of duplicated but creating a report of duplicated folders is more tricky. sure I can work it out but will take a while and wondering if someone else has done this.

Seems like a fairly standard usecase.

Ben

funkytwig · 04-01-2019, 04:30 AM

Quote:

Originally Posted by syg00

Have you read the doco ?. I found a section on just this in the tutorial listed on the github homepage.

rmlint is on my list of things to check-out - looks a great addition to the toolset.

Tried to -D flag but the output was not that useful.

Want something that just tells be directory x is duplicated in directory x (and z)...

Ben

ondoho · 04-01-2019, 02:15 PM

any program will never do what you wish but only what was implemented.

that said you could write yourself a wrapper around the output that formats the output in a desirable manner.