dynamic envoy configuration from k8s configmap

Question

I tried to configure envoy in my kubernetes cluster by following this example: https://www.envoyproxy.io/docs/envoy/latest/start/quick-start/configuration-dynamic-filesystem

My static envoy config:

    node:
      cluster: test-cluster
      id: test-id

    dynamic_resources:
      cds_config:
        path: /var/lib/envoy/cds.yaml
      lds_config:
        path: /var/lib/envoy/lds.yaml

    admin:
      access_log_path: "/dev/null"
      address:
        socket_address:
          address: 0.0.0.0
          port_value: 19000

The dynamic config from configmap is mounted to and contains the files .

I used a configmap to mount the config files (cds.yaml and lds.yaml) into to envoy pod (to /var/lib/envoy/) but unfortunately the envoy configuration doesn't change when I change the config in the configmap. The mounted config files are updated as expected.

I can see from the logs, that envoy watches the config files:

[2021-03-01 17:50:21.063][1][debug][file] [source/common/filesystem/inotify/watcher_impl.cc:47] added watch for directory: '/var/lib/envoy' file: 'cds.yaml' fd: 1
[2021-03-01 17:50:21.063][1][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:140] maybe finish initialize state: 1
[2021-03-01 17:50:21.063][1][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:149] maybe finish initialize primary init clusters empty: true
[2021-03-01 17:50:21.063][1][info][config] [source/server/configuration_impl.cc:95] loading 0 listener(s)
[2021-03-01 17:50:21.063][1][info][config] [source/server/configuration_impl.cc:107] loading stats configuration
[2021-03-01 17:50:21.063][1][debug][file] [source/common/filesystem/inotify/watcher_impl.cc:47] added watch for directory: '/var/lib/envoy' file: 'lds.yaml' fd: 1

and once I update the configmap I also get the logs that something changed:

[2021-03-01 17:51:50.881][1][debug][file] [source/common/filesystem/inotify/watcher_impl.cc:72] notification: fd: 1 mask: 80 file: ..data
[2021-03-01 17:51:50.881][1][debug][file] [source/common/filesystem/inotify/watcher_impl.cc:72] notification: fd: 1 mask: 80 file: ..data

but envoy doesn't reload the config.

It seems that kubernetes updates the config files by changing a directory and envoy doesn't recognise that the config files are changed.

Is there an easy way to fix that? I don't want to run and xDS server for my tests but hot config reload would be great for my testing 😇

Thanks!

Answer 1

It can now be solved by using watched_directory in path_config_source. See the documentation of watched_directory for using xDS with ConfigMap