How to monitor filesystem during compilation?
 

I am working for a quite big embedded software product whose dependencies are not very well known. Especially, all public headers are collected to one flat folder and it is very hard to tell who is using what.

I am hoping to get some information on the dependencies by monitoring the filesystem during a build. This is basically the same idea than in the commercial Electric Insight tool. I just thought it could be done for free but it seems to be not totally straightforward.

The build is based on recursive makeing of the subsystems. What I would like to do would be to monitor the entire source tree (or at least the folder for public headers) and record the files accessed during compilation of each sybsystem (sybsystem "A" uses these, etc...).

I noticed it is easy to implement the monitoring using inotify or pyinotify as I did in my try. This allowed me to find there are lots of files which are not used at all.

My problem really is how to convey the subsystem information to an inotify event. Now, I just get to knew which files are accessed but do not know which subsystem needs them. The easy way would be to revert to sequential make but I would like to get ideas how to solve the problem with make -j (parallel jobs). I would like to integrate the dependency data generation to the normal build process.

Thanks!
-Janne

What's wrong with gcc -M?

I did not probably share enough of the product. The product has code in two languages: our proprietary language and C. Our proprietary language is translated into C but there is the challenge that all headers are defined in our proprietary language. The tools for translating the headers into C work in such a manner that they either generate one big header file (for native C components) or open the definitions directly into translated code (for components written in our proprietary language). There are some features in the tools that should allow generating the dependency information but it seems they are buggy and they assume certain structure of programs which makes them often fail. Using gcc -M would just generate dependency information to these generated and disposable headers.

I was thinking monitoring the file system (in similar manner than Electric Insight) could provide us a shortcut way because fixing the proprietary tools for dependency generation seems to be a big pain. But maybe we need to fix the tools...

The problem with dependency generators is that they only tell you which headers are included, not which are needed.
If I included bitmaplib.h because I thought I needed a function from it, but I no longer use it, it will likely stay there til the end of time. Unused variables generate a warning, but unused libraries don't -- seems like a bug to me.

Unless your filesystem is set to noatime, you can check whether any file has been opened for read by comparing the access time output of stat before and after compiling. File system event monitoring that I've used is driven by changes to the filesystem and isn't triggered by reading a file, so wouldn't help you.