Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to send alerts based on Kubernetes / Docker events?

Is it possible to somehow send alerts (to email / slack) based on events that occur within a Kubernetes cluster?

In particular, it would be useful to get an alert if a pod has restarted unexpectedly or if a pod cannot start. Similarly it would be useful to know if a pod's CPU usage was over a certain threshold and get an alert.

We have Heapster (with InfluxDB / Grafana backend) installed. While this gives useful data, it unfortunately does not provide us with alerting.

like image 300
DrGecko Avatar asked Dec 07 '15 16:12

DrGecko


People also ask

How do I get events in Kubernetes?

Accessing Kubernetes Events Running the kubectl describe command on specific cluster resources will list the events for that resource. A more generic way of doing this is by running the kubectl get events command, which lists the specific resources' events or the entire cluster.

How do you check event logs in Kubernetes pod?

Access event logs Developers, application and infrastructure operators can use the kubectl describe command against specific resources, or use the more generic kubectl get event command to list events for a specific resource, or for the entire cluster.

How do Kubernetes events work?

A Kubernetes event is an object that shows what's happening inside a cluster, node, pod, or container. These objects are usually generated in response to changes that occur inside your K8s system. The Kubernetes API Server enables all core components to create these events.


2 Answers

Both sysdig and Datadog provide this functionality as well.

like image 58
Michael Hausenblas Avatar answered Nov 10 '22 15:11

Michael Hausenblas


You can check and probably replace InfluxDB with http://prometheus.io (I am not sure about your cases but generally it is possible) which has built-in metrics support.

like image 28
Vyacheslav Enis Avatar answered Nov 10 '22 14:11

Vyacheslav Enis