How perform a rolling update with Vault HA in Kubernetes?

Question

I am running a HashiCorp Vault stateless set in K8s with 3 pods on three nodes.

After deployment I manually unseal Vault. Vault then stays unsealed all the time.

The problem is when one of the nodes is restarted, the Vault pod restarts in unsealed mode. Is there any way to automatically unseal the Vault node by server-server-communication with one of the already unsealed pods?

I do not want to have to unseal the Vault pods manually when my Kubernetes environment gets updated and all nodes get restarted (rolling update - one by one).

I also do not want to store the unseal keys in a K8s secret or even a file as this makes encryption of my secrets useless.

This is my yaml:

apiVersion: v1
kind: Service
metadata:
  name: vault
spec:
  clusterIP: None
  ports:
    - name: http
      port: 8200
    - name: server
      port: 8201
  selector:
    xyz.service: vault
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: vault
  labels:
    xyz.service: vault
spec:
  serviceName: "vault"
  selector:
    matchLabels:
      xyz.service: vault
  replicas: 3
  template:
    metadata:
      labels:
        xyz.service: vault
    spec:
      imagePullSecrets:
      - name: reg-dhc-xyzoms-pull-secret
      securityContext:
        runAsUser: 100
        fsGroup: 1000
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: xyz.service
                    operator: In
                    values:
                      - vault
              topologyKey: kubernetes.io/hostname
      containers:
        - name: vault
          image: vault:0.11.0
          resources:
            requests:
              memory: "100Mi"
          env:
          - name: SKIP_SETCAP
            value: dontcare
          - name: POD_IP
            valueFrom:
              fieldRef:
                fieldPath: "status.podIP"
          - name: "VAULT_CLUSTER_ADDR"
            value: "https://$(POD_IP):8201"
          ports:
            - name: http
              containerPort: 8200
              protocol: "TCP"
            - name: server
              containerPort: 8201
              protocol: "TCP"

Answer 1

After some more digging I learned, what I want is impossible. Whenever a Vault instance is restarted it will be unsealed first and there is no way to unseal it automatically using Vault-own techniques.

You can find a lot of "vault-unsealer" implementations in GitHub and Docker store which try to fill this gap by regularly checking the Vault pods state and unsealing it if necessary.

It is suggested to use an K8s readinessprobe to avoid that services access a sealed Vault pod.

As there is no official "vault-unsealer" image, the public implementations must be used with caution. I ended up writing my own "vault-unsealer" to avoid security flaws and licensing problems.

My solution is a sidecar-container with each Vault pod. The unseal keys first have to be entered once manually with kubectl exec ... at one sidecar. The sidecars regularly check all Vault pods and communicate the unseal keys to the other sidecar if sealed. If a sidecar receives unseal keys, they are stored in memory and are used to unseal its own Vault instance.

1. kubect aply -f vault.yaml -> vault-0 starting
2. kubectl exec vault-0 -c sidecar ... to enter unseal keys -> vault-0 sidecar unseals vault-0 and is ready
3. vault-1 starting
4. vault-0 sidecar detects vault-1 unsealed and calls the vault-1 sidecar to transmit the unseal keys. -> vault-1 sidecar unseals vault-0 and is ready
5. and so on...