Docker with 2 containers growing out exceedingly

lpallard · 01-28-2022, 07:53 AM

So I installed Docker on a Rocky Linux VM with a 10GB hard drive (runs on Proxmox) for two small docker applications (rsshub & tinytinyrss). The applications were not taking much more than 2.5GB of hard drive space at the moment of installation, and the VM had plenty of space left in /

Fast forward today, I keep getting notifications from this VM that the hard drive is nearly full (the VM is configured to send an email notification via cron if a storage layer reaches 80% or more)... df -H reveals:

Code:

Filesystem                                       Size  Used Avail Use% Mounted on
devtmpfs                                         490M     0  490M   0% /dev
tmpfs                                            509M     0  509M   0% /dev/shm
tmpfs                                            509M   52M  457M  11% /run
tmpfs                                            509M     0  509M   0% /sys/fs/cgroup
/dev/mapper/rl_vm--template-root                 8.6G  6.9G  1.8G  80% /
/dev/vda1                                        1.1G  341M  724M  32% /boot
overlay                                          8.6G  6.9G  1.8G  80% /var/lib/docker/overlay2/0770d9a34e34bd2b7d6af52755cd3b1d9a6c3f36b7b2c2964dd56ecb15d28b5c/merged
overlay                                          8.6G  6.9G  1.8G  80% /var/lib/docker/overlay2/62cd457a4cb51385cfc66c8296559abfa75bb5e5d66335ddf6d99843033278ce/merged
overlay                                          8.6G  6.9G  1.8G  80% /var/lib/docker/overlay2/3321797212b44e1d23ec2d438dfc5997145e28fa6051c7ae71d2b261a79d7ae1/merged
overlay                                          8.6G  6.9G  1.8G  80% /var/lib/docker/overlay2/06b6a4f4cfbb0bc7a80bb1e103e320b10cf90fbdb6e2b580afa71ff7304d8718/merged
tmpfs                                            102M     0  102M   0% /run/user/1000

Examining / I believe the issue is inside "/var/lib/docker/overlay2/" but to be transparent, googling this issue sent me in so many different directions that I am not sure. I know very little about docker, and I believe it uses virtual FS's to perform storage, and some users have pointed out that even if overlay2 seems to be the culprit, in reality it could be some other files occupying all the space somewhere else within the docker virtual FS... Could be bull**** too...

Anyways, /var/lib/docker is indeed 5.8GB

Code:

du -hs /var/lib/docker
5.8G	/var/lib/docker

Looking at docker's FS usage stats:

Code:

docker system df
TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          4         4         2.657GB   0B (0%)
Containers      4         4         174.4MB   0B (0%)
Local Volumes   2         2         146.1kB   0B (0%)
Build Cache     0         0         0B        0B

Looking at docker's container sizes:

Code:

docker ps --size
CONTAINER ID   IMAGE                                   COMMAND                  CREATED        STATUS        PORTS                                                           NAMES                  SIZE
f260f3a94b0f   diygod/rsshub                           "dumb-init -- npm ru…"   2 months ago   Up 13 hours   0.0.0.0:1200->1200/tcp, :::1200->1200/tcp                       rsshub_rsshub_1        174MB (virtual 408MB)
d2e03bd28b6e   redis:alpine                            "docker-entrypoint.s…"   2 months ago   Up 13 hours   6379/tcp                                                        rsshub_redis_1         0B (virtual 32.4MB)
064474c5af54   browserless/chrome:1.43-chrome-stable   "./start.sh"             2 months ago   Up 13 hours   3000/tcp                                                        rsshub_browserless_1   11B (virtual 2.14GB)
3ebad0761681   portainer/portainer-ce                  "/portainer"             2 months ago   Up 13 hours   8000/tcp, 9443/tcp, 0.0.0.0:9000->9000/tcp, :::9000->9000/tcp   portainer              0B (virtual 252MB)

Finally trying to find the largest folders within /var/lib/docker:

Code:

du -a /var/lib/docker | sort -n -r | head -n 20
6054432	/var/lib/docker
5864496	/var/lib/docker/overlay2
2181636	/var/lib/docker/overlay2/62cd457a4cb51385cfc66c8296559abfa75bb5e5d66335ddf6d99843033278ce
2181624	/var/lib/docker/overlay2/62cd457a4cb51385cfc66c8296559abfa75bb5e5d66335ddf6d99843033278ce/merged
1755720	/var/lib/docker/overlay2/62cd457a4cb51385cfc66c8296559abfa75bb5e5d66335ddf6d99843033278ce/merged/usr
1349052	/var/lib/docker/overlay2/de6aaef229cdce3712f113911ef72859e10980217a24901976f87f87eb80fb7c
1349044	/var/lib/docker/overlay2/de6aaef229cdce3712f113911ef72859e10980217a24901976f87f87eb80fb7c/diff
1300180	/var/lib/docker/overlay2/de6aaef229cdce3712f113911ef72859e10980217a24901976f87f87eb80fb7c/diff/usr
957564	/var/lib/docker/overlay2/62cd457a4cb51385cfc66c8296559abfa75bb5e5d66335ddf6d99843033278ce/merged/usr/lib
863272	/var/lib/docker/overlay2/de6aaef229cdce3712f113911ef72859e10980217a24901976f87f87eb80fb7c/diff/usr/lib
609260	/var/lib/docker/overlay2/06b6a4f4cfbb0bc7a80bb1e103e320b10cf90fbdb6e2b580afa71ff7304d8718
480908	/var/lib/docker/overlay2/62cd457a4cb51385cfc66c8296559abfa75bb5e5d66335ddf6d99843033278ce/merged/usr/lib/x86_64-linux-gnu
446508	/var/lib/docker/overlay2/de6aaef229cdce3712f113911ef72859e10980217a24901976f87f87eb80fb7c/diff/usr/lib/x86_64-linux-gnu
437464	/var/lib/docker/overlay2/06b6a4f4cfbb0bc7a80bb1e103e320b10cf90fbdb6e2b580afa71ff7304d8718/merged
415556	/var/lib/docker/overlay2/9127947de0ee3a35e55a3139facf3b3d1f8c248adcfc1ec31cfdbfcfd129ecb4
415548	/var/lib/docker/overlay2/9127947de0ee3a35e55a3139facf3b3d1f8c248adcfc1ec31cfdbfcfd129ecb4/diff
335492	/var/lib/docker/overlay2/9127947de0ee3a35e55a3139facf3b3d1f8c248adcfc1ec31cfdbfcfd129ecb4/diff/usr
335284	/var/lib/docker/overlay2/9127947de0ee3a35e55a3139facf3b3d1f8c248adcfc1ec31cfdbfcfd129ecb4/diff/usr/src/app
335284	/var/lib/docker/overlay2/9127947de0ee3a35e55a3139facf3b3d1f8c248adcfc1ec31cfdbfcfd129ecb4/diff/usr/src
335284	/var/lib/docker/overlay2/62cd457a4cb51385cfc66c8296559abfa75bb5e5d66335ddf6d99843033278ce/merged/usr/src/app

I thought the logs and json files were the issue but looking at the total space used by each of these file types (find . -name "*.log" | xargs du -sch):
-logs < 519MB
-JSON < 7.5MB
-DEB < 139MB
....

1. What is causing the filesystem to grow out of control?
2. How to clean up Docker to free up HDD space?

I hope some can help me with this.

Thanks

pan64 · 01-28-2022, 08:06 AM

As far as I know docker and overlayfs is ok, it is the app inside the docker container (most probably the cache of chrome or something similar).

lpallard · 01-28-2022, 11:05 AM

You may be onto something... I have created 2 more env variables for rsshub as per their wiki (https://docs.rsshub.app/en/install/#docker-image) in the hope the cache will clear itself at some point but I am not expecting anything at this moment...

For the chrome browserless docker app, I also believe it may be the root cause... I however cant find how to clear cache, or limit its growth...

If anybody has info please share.

pan64 · 01-29-2022, 08:26 AM

theoretical you may go inside the docker image and look around, as if it was a real VM. But that require some knowledge about them.
you may get some ideas (for example) here: https://phoenixnap.com/kb/how-to-ssh...cker-container

divyashree · 02-07-2022, 09:52 AM

There might be unused/dangling objects.

Have you tried to prune unused objects of docker ?

check this official document : https://docs.docker.com/config/pruning/

This will help you cleanup if any unused image/container/network/volumes present in overlay directory.

sundialsvcs · 02-09-2022, 11:23 AM

"Unused objects" won't account for this. Something inside the containerized environment must be the culprit in this case.

lpallard · 03-03-2022, 01:38 PM

Quote:

Originally Posted by sundialsvcs

"Unused objects" won't account for this. Something inside the containerized environment must be the culprit in this case.

Fully agree with you. I tried pruning the system, and it deleted a few unused images and cleared up 233MB of space, barely doing a dent in the problem.

"docker image ls" gives:

Code:

REPOSITORY               TAG                  IMAGE ID       CREATED         SIZE
diygod/rsshub            latest               f40e0167073d   5 weeks ago     234MB
portainer/portainer-ce   latest               ad0ecf974589   3 months ago    252MB
redis                    alpine               5c08f13a2b92   3 months ago    32.4MB
browserless/chrome       1.43-chrome-stable   ec7bb30f39c5   15 months ago   2.14GB

clearly the problem is "ec7bb30f39c5".

ssh into the container, I could see that /usr is taking 1.7GB, with /usr/src and /usr/lib taking most of that space...

Code:

0	/bin
0	/boot
0	/dev
0	/lib
0	/lib32
0	/lib64
0	/libx32
0	/media
0	/mnt
0	/proc
0	/root
0	/sbin
0	/srv
0	/sys
4.0K	/home
8.0K	/run
2.2M	/etc
26M	/var
86M	/tmp
228M	/opt
1.7G	/usr

Am I wrong to say that there's little to do with this unless screwing around in the container trying to save space, when perhaps its the container that is NOT built intelligently (to clean itself up)?

sundialsvcs · 03-03-2022, 05:42 PM

These days, I've encountered a lot of software (specifically including "Chrome") that is extremely wasteful of disk space probably because the authors thereof figure that it doesn't matter. It is entirely correct to presume that the "stock" container definition also did not bother. Really, the only thing that you can do at this point is to enter the container and have a look around ... look at the configuration files, zero-in on exactly what is in (say ...) "/opt," and then exactly why.

Incidentally – this has become one of the specific reasons why I have come to dislike "Docker." In my view, it leaves you too far disconnected from what is actually going on. "The price of convenience" may turn out to be too much. However, that's just me . . .

pan64 · 03-04-2022, 12:16 AM

Quote:

Originally Posted by lpallard

Am I wrong to say that there's little to do with this unless screwing around in the container trying to save space, when perhaps its the container that is NOT built intelligently (to clean itself up)?

Yes, you are wrong. First of all you can drop the whole container and start over with a new one.
Next you can use ncdu or other software to see what is in /usr and clean that manually (if possible).
This is not the container, it cannot clean itself (as a VM or any other OS cannot clean the storage), but the software running inside (=chrome) which eats up your disk.
Docker (and other container solutions) will encapsulate the running process by having its own environment and virtually locking out anything else. Docker can also control the used resources like RAM or CPU, see for example here: https://docs.docker.com/engine/refer...-per-container