k8s: configmap mounted inside symbolic link to "..data" directory

Question

Here my volumeMount:

volumeMounts:
- name: interpreter-spec-volume
  mountPath: /zeppelin/k8s/interpreter

Here my volumes:

volumes:
- name: interpreter-spec-volume
  configMap:
    name: zeppelin-files
    items:
      - key: interpreter-spec.yaml
        path: interpreter-spec.yaml

Problem arises how volume is mounted. My volumeMount is mounted as:

kubectl exec -ti zeppelin-759db57cb6-xw42b -- ls -la /zeppelin/k8s/interpreter
total 0
drwxrwxrwx. 3 root root 88 Jul  7 13:18 .
drwxr-xr-x. 3 root root 53 Jun  8 12:12 ..
drwxr-xr-x. 2 root root 35 Jul  7 13:18 ..2020_07_07_13_18_32.149716995
lrwxrwxrwx. 1 root root 31 Jul  7 13:18 ..data -> ..2020_07_07_13_18_32.149716995
lrwxrwxrwx. 1 root root 28 Jul  7 13:18 interpreter-spec.yaml -> ..data/interpreter-spec.yaml

Why it's mounting ..data directory to itself?

Answer 1

What can I say - this is almost not documented expected behavior. This is due to how secrets and configmaps are mounted into the running container.

When you mount a secret or configmap as volume, the path at which Kubernetes will mount it will contain the root level items symlinking the same names into a ..data directory, which is symlink to real mountpoint.

For example,

kubectl exec -ti zeppelin-759db57cb6-xw42b -- ls -la /zeppelin/k8s/interpreter
total 0
drwxrwxrwx. 3 root root 88 Jul  7 13:18 .
drwxr-xr-x. 3 root root 53 Jun  8 12:12 ..
drwxr-xr-x. 2 root root 35 Jul  7 13:18 ..2020_07_07_13_18_32.149716995
lrwxrwxrwx. 1 root root 31 Jul  7 13:18 ..data -> ..2020_07_07_13_18_32.149716995
lrwxrwxrwx. 1 root root 28 Jul  7 13:18 interpreter-spec.yaml -> ..data/interpreter-spec.yaml

The real mountpoint (..2020_07_07_13_18_32.149716995 in the example above) will change each time the secret or configmap(in your case) is updated, so the real path of your interpreter-spec.yaml will change after each update.

What you can do is use subpath option in volumeMounts. By design, a container using secrets and configmaps as a subPath volume mount will not receive updates. You can leverage on this feature to singularly mount files. You will need to change the pod spec each time you add/remove any file to/from the secret / configmap, and a deployment rollout will be required to apply changes after each secret / configmap update.

volumeMounts:
- name: interpreter-spec-volume
  mountPath: /zeppelin/k8s/interpreter
  subPath: interpreter-spec.yaml

volumes:
- name: interpreter-spec-volume
  configMap:
    name: zeppelin-files

I would also like to mention Kubernetes config map symlinks (..data/) : is there a way to avoid them? question here, where you may find additional info.