Hi,

I'm trying to find a way to measure disk usage on lxc containers (I happen to be using proxmox, but that's less important in this context) using prometheus. I've already installed node_exporter, cadvisor and grafana.

I'm not sure how I can go about doing that if I've got tens of containers. I'm trying to using a solution at the host-level, without needing to install anything on each container.

Any suggestions?

In the simplest, most ordinary case, the files seen by a container occupant are "host files," and the omniscient host can see them even when the container occupant cannot – or sees them in an entirely different way.

Some "highly scalable" container environments, such as those hosted by (say ...) RackSpace, do employ more exotic technologies to provide for that rapid scale-up and to avoid creating "hot spots," but the essential concepts are the same. Unlike a virtual machine, you don't have to install instrumentation inside the container's environment, since that environment is a highly restricted subset of the host's. (At the end of the day, the container occupant's software is being executed directly by the host, without virtualization.)

1 members found this post helpful.

It's actually slightly more complicated than that. And that might be because of how proxmox implements LXCs. So this might be relevant.

When using openvz, you could simply cd into the root directory from the hypervisor. I understand that the same thing applies to dockers. When they adopted lxc, they placed LVM in between, as it were, so it's looks much more like a normal virtual machine. So this might be something related to the way proxmox uses it rather than the lxc per se.
For instance if I write lvs, I get something like that:
All are VM except 200 which is actually an lxc container with Centos 7.4 (the host is Debian). These logical volumes are not meant to be mounted on the host, even if they're containers - or so it seems. And some vm are even proxmox.

In LXD containers, you can use the ZFS storage driver
and you have quite some good visibility per container, per image, per snapshot.

You would simply run

and get the result. I believe that the issue you are getting, it related to the storage driver of choice.

By storage driver you mean lvm in this case?

Can anyone explain exactly how this implementation of prommox is achieved? How come lvm resides 'in-between', as it were? And why it is different from other standard container implementation?

Each Linux container implementation uses the same Linux kernel security features (namespaces, cgroups, etc) to separate processes.
However, each implementation may choose different ways to deal with the storage aspect and the way to store for the files of each container.
They use the terminology "storage backend" or "storage driver".

Depending on your choice of "storage backend" when you configure the Linux container installation, you can get features like "copy-on-write".
That is, if you create ten containers of Ubuntu Linux 16.04, then the initial storage hit is about 360MB (size of uncompressed single Ubuntu Linux 16.04 cloud image).
Any additional files on top of the common cloud image are counted as extra space.

I do not know how it works with proxmox over the LVM storage backend/driver.
As far as I know, LVM can do copy-on-write (therefore you need to take that in count when estimating the container size), but cannot do quotas.

Only LVM thin, to be more exact, can do copy-on-write. That is to say, it expands dynamically as the data is being written. LVM also know snapshots, which also uses the principle of copy-on-write, does't it?

So the question remains - if anyone could answer it - how would I be able to monitor disk usage for containers that use LVM?

You would need to have a look at the management tools for your specific Linux Containers implementation. The way those containers are stored, is specific to that implementation.

From the output you showed earlier, you cannot solve the problem with only LVM commands. The volumes do not appear to have information as to which container they correspond to.

Identifying whose lvm storage that is is a non-issue. Proxmox tells you that either in the gui or in the config files. The problem is that it's as if they were using the same technology as the one used in virtual machines. I mean, yes, they both use lvm, but one uses kvm/qemu, the other uses lxc. So, returning to the initial issue, I couldn't tell what the disk usage is from the host point of view. I'd need to connect to the containers and install some kind of agent there.