Datadog: API Key invalid dropping transaction when installing Datadog agent

Question

I'm trying to install Datadog agent for a Kubernetes cluster using Helm.

This is the helm command I'm using for it:

helm repo add datadog https://helm.datadoghq.com

helm repo update

helm upgrade --install datadog datadog/datadog \
  --namespace monitoring \
  --create-namespace \
  --atomic \
  --set datadog.apiKey=<MY-DATADOG-API-KEY> \
  --set targetSystem=linux \
  --values values.yaml

Values file:

datadog:
  kubelet:
    host:
      valueFrom:
        fieldRef:
          fieldPath: spec.nodeName
    hostCAPath: /etc/kubernetes/certs/kubeletserver.crt
    tlsVerify: false # Required as of Agent 7.35. See Notes.

However, when I run the liveness probe error with error 500 which shows the error below:

CLUSTER | ERROR | (pkg/forwarder/transaction/transaction.go:344 in internalProcess) | API Key invalid, dropping transaction for https://orchestrator.datadoghq.com/api/v1/orchestrator.

Answer 1

Here's how I solved it:

The issue had to do with the Datadog Destination Site. The Destination site for my metrics, traces, and logs is supposed to be datadoghq.eu. This is set using the variable DD_SITE, and it defaults to datadoghq.com if it is not set.

To check what your Datadog Destination Site just look at the URL of your Datadog dashboard:

• For US it will be - https://app.datadoghq.com/
• For EU it will be - https://app.datadoghq.eu/

To set this in your helm chart simply do either of the following:

helm repo add datadog https://helm.datadoghq.com

helm repo update

helm upgrade --install datadog datadog/datadog \
  --namespace monitoring \
  --create-namespace \
  --atomic \
  --set datadog.apiKey=<MY-DATADOG-API-KEY> \
  --set targetSystem=linux \
  --set datadog.site=datadoghq.eu \
  --values values.yaml

OR set it in your values file:

datadog:
  site: datadoghq.eu
  kubelet:
    host:
      valueFrom:
        fieldRef:
          fieldPath: spec.nodeName
    hostCAPath: /etc/kubernetes/certs/kubeletserver.crt
    tlsVerify: false # Required as of Agent 7.35. See Notes.

References:

1. Datadog Agent Forwarder fails liveness probe when new spot instance joins cluster, causing multiple restarts #1697

2. DD_SITE Set to us3.datadoghq.com, but process-agent and security-agent Still Try to Connect to non us3 endpoints #9180