Menu
This answer describes the out of memory killer on Linux.
Is there a way to get notified when the OOM killer has killed something? Email is preferable. One might do this with a watchdog, but I wonder what the easiest and most robust way is.
510
The solution that the linux kernel employs is to invoke the OOM Killer to review all running processes and kill one or more of them in order to free up system memory and keep the system running. The OOM Killer will only get invoked when the system is critically low on memory.
Killing a Process When one or more processes are selected, then OOM-Killer calls the oom_kill_task() function. This function is responsible to send the terminate/kill signal to the process.
You can use the OOM Killer logs both to identify which hosts in your system have run out of memory and to get detailed information on how much memory different processes were using at the time of the error. You can find an annotated example in the git commit that added this logging information to the Linux kernel.
The "logwatch" program will periodically mail you all log lines it's configured to be interested in.
It can be configured to email you about an OOM (unless it runs out of memory itself).
Email notifications aren't a good substitute for proper monitoring from an external system as a machine which has no free memory will not normally be able to send email.
Also, if your system is important, then email is probably too slow to get a response from Ops (especially out of hours)
130
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
