What metrics is oom-killer using to determine memory usage in Cgroup
I am trying to find a metrics that represents memory usage logged in syslog when container reaches the threshold and gets killed.
This is the message I refer to: Nov 6 10:16:24 pool-a53hsbota-7h3co kernel: [2111341.288726] memory: usage 524288kB, limit 524288kB, failcnt 118 Nov 6 10:16:24 pool-a53hsbota-7h3co kernel: [2111341.289672] memory+swap: usage 524288kB, limit 9007199254740988kB, failcnt 0 Nov 6 10:16:24 pool-a53hsbota-7h3co kernel: [2111341.298582] kmem: usage 5800kB, limit 9007199254740988kB, failcnt 0 I tried to collect different matrices using Prometheus and compare their values to the value in the log, but I couldn't find a metrics or combination of matrices which represents the same value as the one logged at that point of time. I tried: - /sys/fs/cgroup/memory/kubepods/burstable/<pod>/<container>/memory.stat - ps command All what I am trying to do is to show in Grafana using proper metrics that memory usage for the container grew and when it reached the limit, the container was killed. Your help here is highly appreciated. |
Does it use cgroup info?
Have a look at this nice oom killer article: https://www.baeldung.com/linux/memor...ent-oom-killer |
If using croup2, add this to your reading list. Yes I know it says facebook, but those folks did all the work for PSI then released it for public consumption.
Go get a beverage of choice before starting. |
Quote:
Basically I would like to shows on some chart, that used memory was rising and just before the invocation of OOM the consumed memory was close to the limit. I cannot find a single metrics which would show it. |
Quote:
Thank you for sharing |
Quote:
After answering last time, I remember reading up on some specific cgroup "issues" with OOM killer, but most of it was how to solve it by splitting cgroup memory so it doesn't kill a full "container" when it reaches the limit, but rather killing the high memory consumer inside the container before OOM killer kills the container. |
Quote:
|
All times are GMT -5. The time now is 04:54 PM. |