I'm writing a little app that goes through a directory and reports disk usage based on user, folder, file type, age, etc.

A typical root of a search will contain 100K files or so, so it's very expensive to do the search. I cache all the data between requests, so all subsequent requests are super fast, but the problem is how to update the cache, preferably without searching through the entire file structure again, when a new file is added or a file is modified.

Does there exist some way of seeing what new files were added? I imagine it would be somewhat low level - something like how df can report disk usage in an instant (granted, I know df doesn't care about individual files, so I can't use the same mechanism... just wondering if something similar exists).

Any ideas other than adding monitoring tools which is almost definitely not an option.

Have a look at inotify - this might be worth a read.

Very interesting. Thanks!