I was thinking of putting some grep commands in to filter the tree output to try and weed out more stuff that is likely to be unimportant. Maybe adding some grep -v after the tree command to suppress lines that we needn't worry about. E.g.:
But part of me wonders if defining these various filters might just be a really long, arduous, thankless process. Perhaps I should be looking into samhain or aide instead? Does no one ever need this kind of scrutiny for installers?

I have noticed that a huge number of the differences in snap1 and snap2 are caused by changes in /var/lib/lxcfs. If I put a grep filter to exclude /var/lib/lxcfs, then my output is suddenly manageable.

But this raises the question "shouldn't I be watching that folder too?" If the point of this audit is security, then anyone aware of the process might hide nefarious activity in one of these folders that I'm ignoring. Is it safe to ignore these folders?

Using screen to manage your workflow is always a good idea, not really specific to this problem, so add it to your toolbox! I prefer Tmux, but similar idea. If you have questions about usage of those it might be better to open a separate thread for that.

I think you asked and answered your own question there!

Your original question, "technique to observer *all* changes made to file system when installing something", would require that you monitor everything. There are some practical limitations you may want to apply to that such as /proc and /sys, but if you truly want to see *all* changes, and maybe even accesses as mentioned previously, then you are going to have to look at *all* possibilities.

For inotify based methods, you are looking at a stream of dynamically reported filesystem events, so you need confidence that it is reporting all events and missing none. My own expereince using that approach has resulted in low confidence that it will report everything.

For a diff based approach such as using tree, you must produce before and after snapshot lists for comparison. The lists must be as comprehensive as you want your audit to be.

Only you can decide how comprehensive you want that to be.

I'm not so sure, but it's hard to say because I don't know what these files are. Been looking into it a bit but lxcfs is a whole other rabbit hole. Hackers fiddling around in there doesn't seem as dangerous as one of them modifying /bin/ssh or /bin/bash, but I don't really know what a hacker might accomplish in there.

Well this introduces another question I guess: "Was I asking the right question?"
It occurs to me that hackers might hide stuff in these busy folders and no one would ever notice. Kinda like Han Solo parking the Millenium Falcon amid the garbage to escape from the Empire.

I really appreciate this valuable insight. I'm using both tree and inotify because inotify reports access actions also. I wonder if this race condition problem is solvable? Has anyone told the devs?

Well therein lies the conundrum. I suppose my decision will be based on what it is feasible to inspect. If I were to get a federal grant for a $1M I might hire some folks to help dig through it all.

This is a pretty sobering realization. The more I look into security, the harder it seems.

If I'm absolutely suspicious of this application, I would not even bother to install it. Period.
If I'm just curious about it and as long as it can be installed by a regular user, I would give it a go.

As for Lxcfs, start here and form your own conclusions, I have not explored it much.

The Han Solo tactic is a favorite of those who have gone over to the dark side!

It is mentioned in the man pages and other documentation, as noted earlier...

That has been in there for as long as I have known of inotify, so apparently no one has found a way to fix it yet.

Yes, there is no plug-n-play for security. As often stated in various ways, security is a process and an attitude, not a program.

I would not say I'm completely suspicious about it. There are a lot of reputable companies that use it to install their sdk/library, including Rackspace, Sendgrid, others. Let's just say I've seen some really poor conduct among library/plugin writers for various CMSes. There is a way to install it locally. It just want to know what it does.

And let's assume that this thread isn't so much about Composer as it is about creating some tools to calm my suspicions.