Adding a volume to a Kubernetes StatefulSet using kubectl patch

Question

Problem summary:

I am following the Kubernetes guide to set up a sample Cassandra cluster. The cluster is up and running, and I would like to add a second volume to each node in order to try enable backups for Cassandra that would be stored on a separate volume.

My attempt to a solution:

I tried editing my cassandra-statefulset.yaml file by adding a new volumeMounts and volumeClaimTemplates entry, and reapplying it, but got the following error message:

$ kubectl apply -f cassandra-statefulset.yaml 
storageclass.storage.k8s.io/fast unchanged
The StatefulSet "cassandra" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden

I then tried to enable rolling updates and patch my configuration following the documentation here: https://kubernetes.io/docs/tasks/run-application/update-api-object-kubectl-patch/

$ kubectl patch statefulset cassandra -p '{"spec":{"updateStrategy":{"type":"RollingUpdate"}}}'
statefulset.apps/cassandra patched (no change)

My cassandra-backup-patch.yaml:

spec:
  template:
    spec:
      containers:
        volumeMounts:
        - name: cassandra-backup
          mountPath: /cassandra_backup
  volumeClaimTemplates:
  - metadata:
      name: cassandra-backup
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: fast
      resources:
        requests:
          storage: 1Gi

However this resulted in the following error:

$ kubectl patch statefulset cassandra --patch "$(cat cassandra-backup-patch.yaml)"
The request is invalid: patch: Invalid value: "map[spec:map[template:map[spec:map[containers:map[volumeMounts:[map[mountPath:/cassandra_backup name:cassandra-backup]]]]] volumeClaimTemplates:[map[metadata:map[name:cassandra-backup] spec:map[accessModes:[ReadWriteOnce] resources:map[requests:map[storage:1Gi]] storageClassName:fast]]]]]": cannot restore slice from map

Could anyone please point me to the correct way of adding an additional volume for each node or explain why the patch does not work? This is my first time using Kubernetes so my approach may be completely wrong. Any comment or help is very welcome, thanks in advance.

Answer 1

The answer is in your first log:

The StatefulSet "cassandra" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy'

You can't change some fields in a statefulset after creation. You will likely need to delete and recreate the statefulset to add a new volumeClaimTemplate.

edit: It can many times be useful to leave your pods running even when you delete the statefulset. To accomplish this use the --cascade=false flag on the delete operation.

kubectl delete statefulset <name> --cascade=false

Then your workload will stay running while you recreate your statefulset with the updated VPC.