Sending data from one pod to another pod running specifically on the same host (DaemonSet)

Question

I have an agent (datadog agent but could be something else) running on all the nodes of my cluster, deployed through a DaemonSet. This agent is collecting diverse metrics about the host: cpu and memory usage, IO, which containers are running.

It can also collect custom metrics, by listening on a specific port 1234.

How can I send data from a pod to the instance of the agent running on the same node than the pod? If I use a Kubernetes service the calls to send the metric will be load balanced across all my agents and I'll lose the correlation between the pod emitting the metric and the host it's running on.

Answer 1

I use the exact same setup, dd-agent running as a DaemonSet in my kubernetes cluster. Using the same port mapping you commented here, you can just send metrics to the hostname of the node an application is running on.

You can add the node name to the pods environment using the downward api in your pod spec:

env:
- name: NODE_NAME
  valueFrom:
    fieldRef:
      fieldPath: spec.nodeName 

Then, you can just open an UDP connection to ${NODE_NAME}:8125 to connect to the datadog agent.

Answer 2

Some preliminary Google searching lands me on https://github.com/kubernetes/kubernetes/pull/42717 by way of https://github.com/kubernetes/kubernetes/issues/24657. It looks like the pull request was merged in time to be in Kubernetes 1.7. This should mean that you can use the Downward API to expose status.hostIP as an environment variable (https://kubernetes.io/docs/tasks/inject-data-application/environment-variable-expose-pod-information/) or a file in a volume (https://kubernetes.io/docs/tasks/inject-data-application/downward-api-volume-expose-pod-information/). Your application would then need to read the environment variable or file to get the value of the actual host IP address.