Menu
We are running a Kubernetes Cluster in AWS and we are collecting the metrics in DataDog using the dd-agent DaemonSet.
We have a Pod being displayed in our metrics tagged as "no_pod" and it is using a lot of resources, Memory/CPU/NetworkTx/NetworkRX.
Is there any explanation to what this pod is, how I can find it, kill it, restart it etc?
I have found the dd-agent source code which seems to define the "no_pod" label but I can't make much sense of why it is there, where it is coming from and how I can find it through kubectl etc.
614
Datadog integrates with each part of your Kubernetes cluster to provide you with a complete picture of health and performance: The Datadog Agent’s Kubernetes integration collects metrics, events, and logs from your cluster components, workload pods, and other Kubernetes objects
I don't know if it's possible to check from datadog that no pods has kube_service running. You can look in to other datadog kube metrics like kubernetes.replicas.available / total to alert if no of available - total < 0. Same can be done or for daemonset pods also there is a specific metric exposed.
As long as your cluster has sufficient resources, Pending pods usually transition to Running status on their own as the Kubernetes scheduler assigns them to suitable nodes. However, in some scenarios, Pending pods will fail to get scheduled until you fix the underlying problem.
When the Datadog Agent detects those containers running anywhere in the cluster, it will attempt to apply a standard configuration template to the containerized application and begin collecting monitoring data.
After speaking to the support team at DataDog, I managed to find out the following information relating to what the no_pod pods were.
Our Kubernetes check is getting the list of containers from the Kubernetes API, which exposes aggregated data. In the metric explorer configuration here, you can see a couple of containers named /docker and / that are getting picked up along with the other containers. Metrics with pod_name:no_pod that come from container_name:/ and container_name:/docker are just metrics aggregated across multiple containers. (So it makes sense that these are the highest values in your graphs.) If you don't want your graphs to show these aggregated container metrics though, you can clone the dashboard and then exclude these pods from the query. To do so, on the cloned dashboard, just edit the query in the JSON tab, and in the tag scope, add !pod_name:no_pod.
So it appears that these pods are the docker and root level containers running outside of the cluster and will always display unless you want to filter them out specifically which I now do.
Many thanks to the support guys at DataDog for looking into the issue for me and giving me a great explanation as to what the pods were and essentially confirming that I can just safely filter these out and not worry about them.
81
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
