Kubernetes ConfigMap volume doesn't create file in container

Question

k8s 1.2 deployed locally, single-node docker

Am I doing something wrong? Is this working for everyone else or is something broken in my k8s deployment?

Following the example in the ConfigMaps guide, /etc/config/special.how should be created below but is not:

[root@totoro brs-kubernetes]# kubectl create -f example.yaml 
configmap "special-config" created
pod "dapi-test-pod" created
[root@totoro brs-kubernetes]# kubectl exec -it dapi-test-pod -- sh
/ # cd /etc/config/
/etc/config # ls
/etc/config # ls -alh
total 4
drwxrwxrwt    2 root     root          40 Mar 23 18:47 .
drwxr-xr-x    7 root     root        4.0K Mar 23 18:47 ..
/etc/config # 

example.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: special-config
  namespace: default
data:
  special.how: very
  special.type: charm
---
apiVersion: v1
kind: Pod
metadata:
  name: dapi-test-pod
spec:
  containers:
    - name: test-container
      image: gcr.io/google_containers/busybox
      command: ["sleep", "100"]
      volumeMounts:
      - name: config-volume
        mountPath: /etc/config
  volumes:
    - name: config-volume
      configMap:
        name: special-config
        items:
        - key: special.how
          path: how.file
  restartPolicy: Never

Summary of conformance test failures follows (asked to run by jayunit100). Full run in this gist.

Summarizing 7 Failures:

[Fail] ConfigMap [It] updates should be reflected in volume [Conformance] 
/home/schou/dev/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/configmap.go:262

[Fail] Downward API volume [It] should provide podname only [Conformance] 
/home/schou/dev/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/util.go:1637

[Fail] Downward API volume [It] should update labels on modification [Conformance] 
/home/schou/dev/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/downwardapi_volume.go:82

[Fail] ConfigMap [It] should be consumable from pods in volume with mappings [Conformance] 
/home/schou/dev/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/util.go:1637

[Fail] Networking [It] should function for intra-pod communication [Conformance] 
/home/schou/dev/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/networking.go:121

[Fail] Downward API volume [It] should update annotations on modification [Conformance] 
/home/schou/dev/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/downwardapi_volume.go:119

[Fail] ConfigMap [It] should be consumable from pods in volume [Conformance] 
/home/schou/dev/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/util.go:1637

Ran 93 of 265 Specs in 2875.468 seconds
FAIL! -- 86 Passed | 7 Failed | 0 Pending | 172 Skipped --- FAIL: TestE2E (2875.48s)
FAIL

Output of findmnt:

[schou@totoro single-node]$ findmnt
TARGET                                SOURCE     FSTYPE  OPTIONS
/                                     /dev/mapper/fedora-root
│                                                ext4    rw,relatime,data=ordere
├─/sys                                sysfs      sysfs   rw,nosuid,nodev,noexec,
│ ├─/sys/kernel/security              securityfs securit rw,nosuid,nodev,noexec,
│ ├─/sys/fs/cgroup                    tmpfs      tmpfs   ro,nosuid,nodev,noexec,
│ │ ├─/sys/fs/cgroup/systemd          cgroup     cgroup  rw,nosuid,nodev,noexec,
│ │ ├─/sys/fs/cgroup/cpuset           cgroup     cgroup  rw,nosuid,nodev,noexec,
│ │ ├─/sys/fs/cgroup/net_cls,net_prio cgroup     cgroup  rw,nosuid,nodev,noexec,
│ │ ├─/sys/fs/cgroup/memory           cgroup     cgroup  rw,nosuid,nodev,noexec,
│ │ ├─/sys/fs/cgroup/hugetlb          cgroup     cgroup  rw,nosuid,nodev,noexec,
│ │ ├─/sys/fs/cgroup/cpu,cpuacct      cgroup     cgroup  rw,nosuid,nodev,noexec,
│ │ ├─/sys/fs/cgroup/perf_event       cgroup     cgroup  rw,nosuid,nodev,noexec,
│ │ ├─/sys/fs/cgroup/pids             cgroup     cgroup  rw,nosuid,nodev,noexec,
│ │ ├─/sys/fs/cgroup/blkio            cgroup     cgroup  rw,nosuid,nodev,noexec,
│ │ ├─/sys/fs/cgroup/freezer          cgroup     cgroup  rw,nosuid,nodev,noexec,
│ │ └─/sys/fs/cgroup/devices          cgroup     cgroup  rw,nosuid,nodev,noexec,
│ ├─/sys/fs/pstore                    pstore     pstore  rw,nosuid,nodev,noexec,
│ ├─/sys/firmware/efi/efivars         efivarfs   efivarf rw,nosuid,nodev,noexec,
│ ├─/sys/kernel/debug                 debugfs    debugfs rw,relatime
│ ├─/sys/kernel/config                configfs   configf rw,relatime
│ └─/sys/fs/fuse/connections          fusectl    fusectl rw,relatime
├─/proc                               proc       proc    rw,nosuid,nodev,noexec,
│ ├─/proc/sys/fs/binfmt_misc          systemd-1  autofs  rw,relatime,fd=32,pgrp=
│ └─/proc/fs/nfsd                     nfsd       nfsd    rw,relatime
├─/dev                                devtmpfs   devtmpf rw,nosuid,size=8175208k
│ ├─/dev/shm                          tmpfs      tmpfs   rw,nosuid,nodev
│ ├─/dev/pts                          devpts     devpts  rw,nosuid,noexec,relati
│ ├─/dev/mqueue                       mqueue     mqueue  rw,relatime
│ └─/dev/hugepages                    hugetlbfs  hugetlb rw,relatime
├─/run                                tmpfs      tmpfs   rw,nosuid,nodev,mode=75
│ ├─/run/user/42                      tmpfs      tmpfs   rw,nosuid,nodev,relatim
│ │ └─/run/user/42/gvfs               gvfsd-fuse fuse.gv rw,nosuid,nodev,relatim
│ └─/run/user/1000                    tmpfs      tmpfs   rw,nosuid,nodev,relatim
│   └─/run/user/1000/gvfs             gvfsd-fuse fuse.gv rw,nosuid,nodev,relatim
├─/tmp                                tmpfs      tmpfs   rw
├─/boot                               /dev/sda2  ext4    rw,relatime,data=ordere
│ └─/boot/efi                         /dev/sda1  vfat    rw,relatime,fmask=0077,
├─/var/lib/nfs/rpc_pipefs             sunrpc     rpc_pip rw,relatime
├─/var/lib/kubelet/pods/fd20f710-fb82-11e5-ab9f-0862662cf845/volumes/kubernetes.io~secret/default-token-qggyv
│                                     tmpfs      tmpfs   rw,relatime
├─/var/lib/kubelet/pods/2f652e15-fb83-11e5-ab9f-0862662cf845/volumes/kubernetes.io~configmap/config-volume
│                                     tmpfs      tmpfs   rw,relatime
└─/var/lib/kubelet/pods/2f652e15-fb83-11e5-ab9f-0862662cf845/volumes/kubernetes.io~secret/default-token-6bzfe
                                      tmpfs      tmpfs   rw,relatime
[schou@totoro single-node]$ 

Answer 1

Thanks to @Paul Morie for helping me diagnose and fix this (from github issue):

bingo, the mount propagation mode of /var/lib/kubelet is private. try changing the mount flag for the kubelet dir to -v /var/lib/kubelet:/var/lib/kubelet:rw,shared

I also had to change MountFlags=slave to MountFlags=shared in my docker systemd file.