Grafana HTTP Error Bad Gateway and Templating init failed errors

Question

Use helm installed Prometheus and Grafana on minikube at local.

$ helm install stable/prometheus
$ helm install stable/grafana

Prometheus server, alertmanager grafana can run after set port-forward:

$ export POD_NAME=$(kubectl get pods --namespace default -l "app=prometheus,component=server" -o jsonpath="{.items[0].metadata.name}")
$ kubectl --namespace default port-forward $POD_NAME 9090

$ export POD_NAME=$(kubectl get pods --namespace default -l "app=prometheus,component=alertmanager" -o jsonpath="{.items[0].metadata.name}")
$ kubectl --namespace default port-forward $POD_NAME 9093

$ export POD_NAME=$(kubectl get pods --namespace default -l "app=excited-crocodile-grafana,component=grafana" -o jsonpath="{.items[0].metadata.name}")
$ kubectl --namespace default port-forward $POD_NAME 3000

Add Data Source from grafana, got HTTP Error Bad Gateway error:

Import dashboard 315 from:

https://grafana.com/dashboards/315

Then check Kubernetes cluster monitoring (via Prometheus), got Templating init failed error:

Why?

Answer 1

In the HTTP settings of Grafana you set Access to Proxy, which means that Grafana wants to access Prometheus. Since Kubernetes uses an overlay network, it is a different IP.

There are two ways of solving this:

1. Set Access to Direct, so the browser directly connects to Prometheus.
2. Use the Kubernetes-internal IP or domain name. I don't know about the Prometheus Helm-chart, but assuming there is a Service named prometheus, something like http://prometheus:9090 should work.