Delete Kubernetes secret on Helm delete

Question

I am creating some secrets when helm install is executed via pre-install hooks.

Everything works great. However when helm delete is performed the secrets created are not deleted. This is because any resource installed using pre-install is considered to be self managed. So I read this could be done using post-delete hooks.

So questions are:

1. How do I delete secrets in post delete?

2. If we remove pre-install hooks then then delete works just fine. But then how to guarantee that secrets are created before the pods are even created when we perform helm install?

Answer 1

Tiller creates resources in a specific order (find it in the source code here: https://github.com/kubernetes/helm/blob/master/pkg/tiller/kind_sorter.go#L26)

So for this specific user case there is no need for hooks or any other mechanism, just include your secret and your pods and magic will happen ;)

That said, there is still the issue with pre-installed objects. The documentation states that this is the desired behaviour:

Practically speaking, this means that if you create resources in a hook, you cannot rely upon helm delete to remove the resources. To destroy such resources, you need to either write code to perform this operation in a pre-delete or post-delete hook or add "helm.sh/hook-delete-policy" annotation to the hook template file.

The only solution is to add a job to the chart, with the post-delete hook, that deletes those resources.

Answer 2

You can use any image with kubectl installed to delete this with pre-delete hook, like @ignacio-millán mentioned. For this, you will also need a role to run this job privileged so it can delete secrets.

Or you can just use the K8s REST API and curl it like this:

apiVersion: batch/v1
kind: Job
metadata:
  labels:
  annotations:
    "helm.sh/hook": pre-delete # << run before delete (we still need the role)
...
spec:
  template:
    spec:
      serviceAccountName: your-privileged-serviceaccount
      automountServiceAccountToken: true 
    # this will mount var/run/secrets/kubernetes.io/serviceaccount/token
    containers:
        - name: pre-delete
          image: "appropriate/curl" # alpine + curl (3 MB)
          env:
            - name: NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          command: 
            - "/bin/sh"
            - "-ec"
            - |
              curl -s -X DELETE -k https://kubernetes.default.svc/api/v1/namespaces/${NAMESPACE}/secrets/your-secret-name-nere \
                -H "Authorization: Bearer $( cat /var/run/secrets/kubernetes.io/serviceaccount/token )" \
                -H "Content-Type: application/json" \
                -H "Accept: application/json" \
                -d "{ \"kind\": \"Secret\", \"apiVersion\": \"v1\", \"metadata\": { \"name\": \"your-secret-name-nere\", \"namespace\": \"${NAMESPACE}\" } }"  > /dev/null
{{- end }}

We need to "RoleBind" a ServiceAccount with a Role (or clusterRole). Use helm.sh/hook-weight for ordering all this. the role should have something like this:

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role # Roles works only in namespace vs ClusterRole works in all the cluster (warning!)
metadata:
   namespace: {{ .Release.Namespace }}
  annotations:
    "helm.sh/hook": pre-install
    "helm.sh/hook-weight": "20"
    "helm.sh/hook-delete-policy": before-hook-creation
  labels:
     ....
rules:
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get", "watch", "list", "create", "update"]

Note we forward the curl output to null with > /dev/null, we don't want this in the logs. ;) comment this and "helm.sh/hook-delete-policy" for debugging.

There is a working in progress here with something like this applied here:

https://github.com/Flag5/consul-helm/tree/tls-encryption/templates see tls-*.yaml files.

hope it helps :)