Why is my DataDog instance reporting a Kubernetes "no_pod"?

Question

We are running a Kubernetes Cluster in AWS and we are collecting the metrics in DataDog using the dd-agent DaemonSet.

We have a Pod being displayed in our metrics tagged as "no_pod" and it is using a lot of resources, Memory/CPU/NetworkTx/NetworkRX.

Is there any explanation to what this pod is, how I can find it, kill it, restart it etc?

I have found the dd-agent source code which seems to define the "no_pod" label but I can't make much sense of why it is there, where it is coming from and how I can find it through kubectl etc.

Answer 1

After speaking to the support team at DataDog, I managed to find out the following information relating to what the no_pod pods were.

Our Kubernetes check is getting the list of containers from the Kubernetes API, which exposes aggregated data. In the metric explorer configuration here, you can see a couple of containers named /docker and / that are getting picked up along with the other containers. Metrics with pod_name:no_pod that come from container_name:/ and container_name:/docker are just metrics aggregated across multiple containers. (So it makes sense that these are the highest values in your graphs.) If you don't want your graphs to show these aggregated container metrics though, you can clone the dashboard and then exclude these pods from the query. To do so, on the cloned dashboard, just edit the query in the JSON tab, and in the tag scope, add !pod_name:no_pod.

So it appears that these pods are the docker and root level containers running outside of the cluster and will always display unless you want to filter them out specifically which I now do.

Many thanks to the support guys at DataDog for looking into the issue for me and giving me a great explanation as to what the pods were and essentially confirming that I can just safely filter these out and not worry about them.