So, I'm running a docker container that's getting killed.
Memory cgroup out of memory: Kill process 1014588 (my-process) score 1974 or sacrifice child
The pid doesn't really help since the instance will be restarted. I'm not sure what to make of the score 1974
portion. Is that some kind of rating? Is that the number of bytes it needs to drop to?
Could the kill be issued because of other things on the system that are squeezing this container, or can this only this container is topped out?
And the sacrifice child part, I think that would be in regards to some kind of subprocess?
The memory subsystem of the cgroups feature isolates the memory behavior of a group of processes (tasks) from the rest of the system. It reports on memory resources used by the processes in a cgroup, and sets limits on memory used by those processes.
What Are cgroups? A control group (cgroup) is a Linux kernel feature that limits, accounts for, and isolates the resource usage (CPU, memory, disk I/O, network, and so on) of a collection of processes.
Cgroups allow you to allocate resources — such as CPU time, system memory, network bandwidth, or combinations of these resources — among user-defined groups of tasks (processes) running on a system.
A cgroup has a set 21GB memory limit. The process inside the cgroup has already used some of the memory quota with its page cache, and then the process starts requesting anonymous memory. The cgroup's RSS (Resident Set Size, as reported by the cgroup's memory.
I believe this is actually answered here
If you check the Linux kernel code here. You'll see:
/*
* If any of p's children has a different mm and is eligible for kill,
* the one with the highest oom_badness() score is sacrificed for its
* parent. This attempts to lose the minimal amount of work done while
* still freeing memory.
*/
mm means 'Memory Management'.
The only difference here is that this kill is getting triggered by cgroups because you have probably run into memory limits.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With