configuration management in Docker Containers

Question

Currently I have configurations files stored in GitHub. I have a single json file with format as below { DEV: { key1 : val1, key2 : val2 }, PROD: { key1 : val1, key2 : val2 } }.

My build system clones the git repo, builds the projects and creates a Docker image and stores in private Docker registry. I have the jar files and configuration files copied into Docker image. Whenever I spin up a container I inject an environment variable (ENV=DEV/PROD) which my code uses to read configs based on environment.

I have couple of questions here:

• What is the best way to maintain environment specific configuration?

• Configuration files may contain sensitive data such as api keys and secrets how can I encrypt and store and decrypt when I build Docker image?

• If I want to change some configuration I need to trigger build because my configuration file is placed inside Docker image. Can I place config files outside Docker container as a volume so that I can replace config file and restart container so that code reads updated configs? If I want to place outside of Docker container can I still use any cluster management tools for container orchestration/management (Kubernetes/ECS)?
• What is the way to make apps running in Docker containers read updated configs by just restarting the container instead of building a new docker image and deploying a new container?

Answer 1

My answer to most of your questions would be "Vault by Hashicorp"

You could spin up a Vault cluster in your environment and have your container connect to, and retrieve secrets from Vault at boot-up and runtime. The most trivial way of implementing the secret retrieval is to have the secrets read in to the container at boot up time and pushed in to the container's ENV.

If you go one step further, you can use a language level vault client library to retrieve and refresh secrets as your application is running, removing the necessity to restart your container to retrieve new secrets.

How does this fit in to what you asked?

• What is the best way to maintain environment specific configuration?

I would say Vault. You can write secrets out to a file or folder using Puppet with Hiera if you wanted, but that means writing secrets to disk which is not ideal. TMK K8s stores secrets unencrypted at rest, which is also not ideal. I would prefer Vault here. Each environment has its own Vault cluster, and there's your 'environment specific configuration'.

• Configuration files may contain sensitive data such as api keys and secrets how can I encrypt and store and decrypt when I build Docker image?

See above. Vault stores secrets encrypted at rest. Use your Vault token (which you can mount in to your container through a volume if you want) to access the secrets and decrypt them. You can then push them to your ENV before starting your containers main process, or do a tighter integration and make Vault part of your application's logic.

• If I want to change some configuration I need to trigger build because my configuration file is placed inside Docker image. Can I place config files outside Docker container as a volume so that I can replace config file and restart container so that code reads updated configs? If I want to place outside of Docker container can I still use any cluster management tools for container orchestration/management (Kubernetes/ECS)?

The answer to this is highly dependent on what the configuration is containing. If it's specific to the code revision, I'd highly recommend bundling it with your code in order to prevent any mismatches in code and config.

If it's effectively independent of code (e.g: tweaking values or secrets), I'd throw them in Vault - why put them in a file when you can store them in an encrypted secret management system?

However if your heart is set on files, you can easily mount them in from a known volume to a known path, and all the scheduling systems (including Nomad now) allow volume mounting with deployed containers.

• What is the way to make apps running in Docker containers read updated configs by just restarting the container instead of building a new docker image and deploying a new container?

With Vault: envconsul is your friend if you want the secrets only changed at boot up time.

See above for my opinion on run-time changes to config values.

The tl;dr of the actual execution of this is to create a 'startup' script which is the entrypoint for your container. The startup script will retrieve the secrets from whatever management system you choose (volume mounts, Vault, a file, whatever), and ready them for your application to read.

The startup script then starts your main app's run process.

Hope this helps!

PS: I don't work for Hashicorp, but I do like their software for working with containers. A lot of what you can get by buying in to Hashicorp's disparate software you can also get from K8s as a single solution. Single apps (Hashicorp) allow you to pick and mix the bits of the solution you want, but also requires a higher maintenance overhead. K8s is a pretty heavy lock-in, but does most of the things you'll likely want without needing to learn anything else.

Good luck