Persistence in Grafana

Question

We have setup a kubernetes cluster on GCP with Istio (installed using Helm and Grafana is enabled) and it has Workload Identity enabled.

We are able to see Istio related dashboards in Grafana but when we create any new dashboard, folder, notification channel, etc. it doesn't remain persistent. When we restart the pod, all over custom creations get wiped out.

Can anyone suggest how to make them persistent? Do we need to modify grafana.persist option (if yes, from where can we do) or do we need to use PVC or something?

Thank you.

Answer 1

you can persist storage by mounting point at /var/lib/grafana.

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: grafana
  name: grafana
spec:
  replicas: 1
  selector:
    matchLabels:
      app: grafana
  template:
    metadata:
      labels:
        app: grafana
    spec:
      containers:
      - image: grafana/grafana:5.4.3
        name: grafana
        ports:
        - containerPort: 3000
          name: http

        volumeMounts:
          - name: grafana-storage
            mountPath: /var/lib/grafana
      volumes:
        - name: grafana-storage
          persistentVolumeClaim:
            claimName: grafana-storage
      securityContext:
        runAsNonRoot: true
        runAsUser: 65534
        fsGroup: 472

or else if you have any existingClaim you can use it.

grafana:
  persistence:
    enabled: true
    storageClassName: prometheus
    existingClaim: prometheus-prometheus-operator-prometheus-db-prometheus-prometheus-operator-prometheus-0