stable/prometheus-operator - adding persistent grafana dashboards

Question

I am trying to add a new dashboard to the below helm chart

https://github.com/helm/charts/tree/master/stable/prometheus-operator

The documentation is not very clear.

I have added a config map to the name space like the below -

apiVersion: v1
kind: ConfigMap
metadata:
  name: sample-grafana-dashboard
  namespace: monitoring
  labels:
     grafana_dashboard: "1"
data:
  etcd-dashboard.json: |-
{JSON}

According to the documentation, this should just be "picked" up and added, but its not. https://github.com/helm/charts/tree/master/stable/grafana#configuration

The sidecar option in my values.yaml looks like -

grafana:
  enabled: true

  ## Deploy default dashboards.
  ##
  defaultDashboardsEnabled: true

  adminPassword: password

  ingress:
    ## If true, Grafana Ingress will be created
    ##
    enabled: false

    ## Annotations for Grafana Ingress
    ##
    annotations: {}
      # kubernetes.io/ingress.class: nginx
      # kubernetes.io/tls-acme: "true"

    ## Labels to be added to the Ingress
    ##
    labels: {}

    ## Hostnames.
    ## Must be provided if Ingress is enable.
    ##
    # hosts:
    #   - grafana.domain.com
    hosts: []

    ## Path for grafana ingress
    path: /

    ## TLS configuration for grafana Ingress
    ## Secret must be manually created in the namespace
    ##
    tls: []
    # - secretName: grafana-general-tls
    #   hosts:
    #   - grafana.example.com
  #dashboardsConfigMaps:
    #sidecarProvider: sample-grafana-dashboard
  sidecar:
    dashboards:
      enabled: true
      label: grafana_dashboard

I have also tried adding this to the value.yml

dashboardsConfigMaps:
   - sample-grafana-dashboard

Which, doesn't work.

Does anyone have any experience with adding your own dashboards to this helm chart as I really am at my wits end.

Answer 1

To sum up: For sidecar you need only one option set to true - grafana.sidecar.dashboards.enabled

1. Install prometheus-operator witch sidecard enabled:

helm install stable/prometheus-operator --name prometheus-operator --set grafana.sidecar.dashboards.enabled=true --namespace monitoring

2. Add new dashboard, for example MongoDB_Overview:

wget https://raw.githubusercontent.com/percona/grafana-dashboards/master/dashboards/MongoDB_Overview.json
kubectl -n monitoring create cm grafana-mongodb-overview --from-file=MongoDB_Overview.json

3. Now the tricky part, you have to set a correct label for your configmap, by default grafana.sidecar.dashboards.label is set tografana_dashboard, so:

kubectl -n monitoring label cm grafana-mongodb-overview grafana_dashboard=mongodb-overview

Now you should find your newly added dashboard in grafana, moreover every confimap with label grafana_dashboard will be processed as dashboard.

The dashboard is persisted and safe, stored in configmap.

UPDATE:

January 2021:

Prometheus operator chart was migrated from stable repo to Prometheus Community Kubernetes Helm Charts and helm v3 was released so:

1. Create namespace:

kubectl create namespace monitoring

2. Install prometheus-operator from helm chart:

helm install prometheus-operator prometheus-community/kube-prometheus-stack --namespace monitoring

3. Add Mongodb dashboard as an example

wget https://raw.githubusercontent.com/percona/grafana-dashboards/master/dashboards/MongoDB_Overview.json
kubectl -n monitoring create cm grafana-mongodb-overview --from-file=MongoDB_Overview.json

4. Lastly, label the dashboard:

kubectl -n monitoring label cm grafana-mongodb-overview grafana_dashboard=mongodb-overview

Answer 2

You have to:

• define you dashboard json as a configmap (as you have done, but see below for an easier way)
• define a provider: to tell where to load the dashboard
• map the two together

from values.yml:

  dashboardsConfigMaps:
    application: application
  dashboardProviders:
    dashboardproviders.yaml:
      apiVersion: 1
      providers:
        - name: application
          orgId: 1
          folder: "Application Metrics"
          type: file
          disableDeletion: true
          editable: false
          options:
            path: /var/lib/grafana/dashboards/application

Now the application config map should create files in this directory in the pod, and as has been discussed the sidecar should load them into an Application Metrics folder, seen in the GUI.

That probably answers your issue as written, but as long as your dashboards aren't too big using kustonmise mean you can have the json on disk without needing to include the json in another file thus:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
# May choose to enable this if need to refer to configmaps outside of kustomize
generatorOptions:
  disableNameSuffixHash: true

namespace: monitoring

configMapGenerator:
- name: application
  files:
    - grafana-dashboards/application/api01.json
    - grafana-dashboards/application/api02.json

For completeness sake you can also load dashboards from url or from the Grafana site, although I don't believe mixing method in the same folder works.

So:

  dashboards:
    kafka:
      kafka01:
        url: https://raw.githubusercontent.com/kudobuilder/operators/master/repository/kafka/docs/latest/resources/grafana-dashboard.json
        folder: "KUDO Kafka"
        datasource: Prometheus
    nginx:
      nginx1:
        gnetId: 9614
        datasource: Prometheus
  dashboardProviders:
    dashboardproviders.yaml:
      apiVersion: 1
      providers:
        - name: kafka
          orgId: 1
          folder: "KUDO Kafka"
          type: file
          disableDeletion: true
          editable: false
          options:
            path: /var/lib/grafana/dashboards/kafka
        - name: nginx
          orgId: 1
          folder: Nginx
          type: file
          disableDeletion: true
          editable: false
          options:
            path: /var/lib/grafana/dashboards/nginx

Creates two new folders containing a dashboard each, from external sources, or maybe you point this at your git repo you de-couple your dashboard commits from your deployment.