Error in setting Prometheus datasource in Grafana on Kubernetes

Question

I am setting up Grafana on Kubernetes with some custom dashboards and Prometheus as a default data source.

All services are running behind Traefik. Prometheus is running at https://myapp.dev/svc/prometheus/ and Grafana is running at https://myapp.dev/svc/grafana/

I can log in to both grafana(also able to see dashboards that were added to config) and prometheus, however when I test the Prometheus data source, I see the following error in grafana logs:

lvl=eror msg="Data proxy error" logger=data-proxy-log userId=1 orgId=1 uname=xxx path=/api/datasources/proxy/1/api/v1/query remote_addr=192.168.64.1 referer=https://myapp.dev/svc/grafana/datasources/edit/1/ error="http: proxy error: dial tcp: lookup prometheus on 10.96.0.10:53: no such host"

Grafana deployment

kind: Deployment
apiVersion: apps/v1
metadata:
  namespace: default
  name: grafana-deployment
  labels:
    app: grafana
spec:
  replicas: 1
  selector:
    matchLabels:
      app: grafana
  template:
    metadata:
      labels:
        app: grafana
    spec:
      containers:
      - name: grafana
        image: grafana/grafana:latest
        ports:
        - containerPort: 3000
        volumeMounts:
          - mountPath: /etc/grafana/provisioning/datasources/
            name: datasource-volume
          - mountPath: /etc/grafana/provisioning/dashboards/
            name: dashboard-volume
          - mountPath: /var/lib/grafana
            name: grafana-storage
      volumes:
        - name: grafana-storage
          emptyDir: {}
        - configMap:
            defaultMode: 420
            name: grafana-datasource-cm
          name: datasource-volume
        - configMap:
            defaultMode: 420
            name: grafana-dashboard-cm
          name: dashboard-volume

Configmap

kind: ConfigMap
apiVersion: v1
metadata:
  name: grafana-datasource-cm
data:
  datasource.yml: |-
    apiVersion: 1
    datasources:
    - name: Prometheus
      type: prometheus
      orgId: 1
      access: proxy
      url: http://prometheus:9090/svc/prometheus/
      basicAuth: false

I am not sure on how to fix this error and test Prometheus datasource.

Answer 1

Sounds like it cannot find the prometheus service in the cluster. Maybe:

• It's in a different namespace❓
• It's defined but with a different name❓

If it's a different namespace you can try changing your config with the full FQDN of the service. prometheus.<namespace>.svc.cluster.local

kind: ConfigMap
apiVersion: v1
metadata:
  name: grafana-datasource-cm
data:
  datasource.yml: |-
    apiVersion: 1
    datasources:
    - name: Prometheus
      type: prometheus
      orgId: 1
      access: proxy
      url: http://prometheus.<namespace>.svc.cluster.local:9090/svc/prometheus/
      basicAuth: false

See if you can get to the service from your grafana pod 🔧:

$ kubectl exec -it <grafana-pod> sh
# curl http://prometheus.<namespace>.svc.cluster.local:9090

✌️☮️