Permission denied when chown on elasticsearch data directory in kubernetes statefulset

Question

Hoping someone can help me resolve what appears to be a permissions error. I'm trying to start a 3-node elasticsearch cluster using the official elasticsearch docker image. When the container was started I was getting an "access denied" error from elasticsearch on /usr/share/elasticsearch/data/nodes so I tried adding a command to make elasticsearch the owner of /usr/share/elasticsearch/data...but I get these errors when I include the chown command:

chown: cannot read directory '/usr/share/elasticsearch/data/lost+found': Permission denied
chown: changing ownership of '/usr/share/elasticsearch/data': Operation not permitted

Here is my statefulset yaml file:

apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: esnode
spec:
  serviceName: elasticsearch-transport
  replicas: 3
  template:
    metadata:
      labels:
        app: evo-pro-cluster
    spec:
      initContainers:
      - name: init-sysctl
        image: busybox
        imagePullPolicy: IfNotPresent
        command: ["sysctl", "-w", "vm.max_map_count=262144"]
        securityContext:
          privileged: true
      containers:
      - name: elasticsearch
        securityContext:
          privileged: true
          capabilities:
            add:
            - IPC_LOCK
            - SYS_RESOURCE
        command: ["/bin/sh"]
        args: ["-c", "chown -R elasticsearch:elasticsearch /usr/share/elasticsearch/data"]
        image: docker.elastic.co/elasticsearch/elasticsearch:5.6.1
        imagePullPolicy: Always
        env:
        - name: "ES_JAVA_OPTS"
          value: "-Xms6g -Xmx6g"
        ports:
        - containerPort: 9200
          name: http
          protocol: TCP
        - containerPort: 9300
          name: transport
          protocol: TCP
        volumeMounts:
        - name: storage
          mountPath: /usr/share/elasticsearch/data
        - name: config
          mountPath: /usr/share/elasticsearch/config/elasticsearch.yml
          subPath: elasticsearch.yml
      volumes:
        - name: config
          configMap:
            name: elasticsearch-config
  volumeClaimTemplates:
  - metadata:
      name: storage
      annotations:
        storageClassName: standard
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 110Gi

Answer 1

This particular docker image expects the data directory to be writable by uid 2000. You can tell Kubernetes to chown (sort of) the mount point for your pod by adding .spec.securityContext.fsGroup:

apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: esnode
spec:
  ...
  securityContext:
    fsGroup: 2000

(end of course you can get rid of the chown hack or the initContainer)

fsGroup: integer: A special supplemental group that applies to all containers in a pod. Some volume types allow the Kubelet to change the ownership of that volume to be owned by the pod: 1. The owning GID will be the FSGroup 2. The setgid bit is set (new files created in the volume will be owned by FSGroup) 3. The permission bits are OR'd with rw-rw---- If unset, the Kubelet will not modify the ownership and permissions of any volume.