Kustomize overlays when using a shared ConfigMap

Question

I have an environment made of pods that address their target environment based on an environment variable called CONF_ENV that could be test, stage or prod.

The application running inside the Pod has the same source code across environments, the configuration file is picked according to the CONF_ENV environment variable.

I'v encapsulated this CONF_ENV in *.properties files just because I may have to add more environment variables later, but I make sure that each property file contains the expected CONF_ENV e.g.:

• test.properites has CONF_ENV=test,
• prod.properties has CONF_ENV=prod, and so on...

I struggle to make this work with Kustomize overlays, because I want to define a ConfigMap as a shared resource across all the pods within the same overlay e.g. test (each pod in their own directory, along other stuff when needed).

So the idea is:

• base/ (shared) with the definition of the Namespace, the ConfigMap (and potentially other shared resources
• base/pod1/ with the definition of pod1 picking from the shared ConfigMap (this defaults to test, but in principle it could be different)

Then the overlays:

• overlay/test that patches the base with CONF_ENV=test (e.g. for overlay/test/pod1/ and so on)
• overlay/prod/ that patches the base with CONF_ENV=prod (e.g. for overlay/prod/pod1/ and so on)

Each directory with their own kustomize.yaml.

The above doesn't work because when going into e.g. overlay/test/pod1/ and I invoke the command kubectl kustomize . to check the output YAML, then I get all sorts of errors depending on how I defined the lists for the YAML keys bases: or resources:.

I am trying to share the ConfigMap across the entire CONF_ENV environment in an attempt to minimize the boilerplate YAML by leveraging the patching-pattern with Kustomize.

The Kubernetes / Kustomize YAML directory structure works like this:

├── base
│   ├── configuration.yaml # I am trying to share this!
│   ├── kustomization.yaml
│   ├── my_namespace.yaml # I am trying to share this!
│   ├── my-scheduleset-etl-misc
│   │   ├── kustomization.yaml
│   │   └── my_scheduleset_etl_misc.yaml
│   ├── my-scheduleset-etl-reporting
│   │   ├── kustomization.yaml
│   │   └── my_scheduleset_etl_reporting.yaml
│   └── test.properties # I am trying to share this!
└── overlay
    └── test
        ├── kustomization.yaml # here I want tell "go and pick up the shared resources in the base dir"
        ├── my-scheduleset-etl-misc
        │   ├── kustomization.yaml
        │   └── test.properties # I've tried to share this one level above, but also to add this inside the "leaf" level for a given pod
        └── my-scheduleset-etl-reporting
            └── kustomization.yaml

The command kubectl with Kustomize:

• sometimes complains that the shared namespace does not exist:

error: merging from generator &{0xc001d99530 {  map[] map[]} {{ my-schedule-set-props merge {[CONF_ENV=test] [] [] } <nil>}}}: 
id resid.ResId{Gvk:resid.Gvk{Group:"", Version:"v1", Kind:"ConfigMap", isClusterScoped:false}, Name:"my-schedule-set-props", Namespace:""} 
does not exist; cannot merge or replace

• sometimes doesn't allow to have shared resources inside an overlay:

error: loading KV pairs: env source files: [../test.properties]: 
security; file '/my/path/to/yaml/overlay/test/test.properties' 
is not in or below '/my/path/to/yaml/overlay/test/my-scheduleset-etl-misc'

• sometimes doesn't allow cycles when I am trying to have multiple bases - the shared resources and the original pod definition:

error: accumulating resources: accumulation err='accumulating resources from '../': 
'/my/path/to/yaml/overlay/test' must resolve to a file': 
cycle detected: candidate root '/my/path/to/yaml/overlay/test' 
contains visited root '/my/path/to/yaml/overlay/test/my-scheduleset-etl-misc'

The overlay kustomization.yaml files inside the pod dirs have:

bases:
  - ../ # tried with/without this to share the ConfigMap
  - ../../../base/my-scheduleset-etl-misc/

The kustomization.yaml at the root of the overlay has:

bases:
  - ../../base

The kustomization.yaml at the base dir contains this configuration for the ConfigMap:

# https://gist.github.com/hermanbanken/3d0f232ffd86236c9f1f198c9452aad9
configMapGenerator:
  - name: my-schedule-set-props
    namespace: my-ss-schedules
    envs:
      - test.properties

vars:
  - name: CONF_ENV
    objref:
      kind: ConfigMap
      name: my-schedule-set-props
      apiVersion: v1
    fieldref:
      fieldpath: data.CONF_ENV

configurations:
  - configuration.yaml

With configuration.yaml containing:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

varReference:
- path: spec/confEnv/value
  kind: Pod

How do I do this?

How do I make sure that I minimise the amount of YAML by sharing all the ConfigMap stuff and the Pods definitions as much as I can?

Answer 1

If I understand your goal correctly, I think you may be grossly over-complicating things. I think you want a common properties file defined in your base, but you want to override specific properties in your overlays. Here's one way of doing that.

In base, I have:

$ cd base
$ tree
.
├── example.properties
├── kustomization.yaml
└── pod1
    ├── kustomization.yaml
    └── pod.yaml

Where example.properties contains:

SOME_OTHER_VAR=somevalue
CONF_ENV=test

And kustomization.yaml contains:

resources:
  - pod1

configMapGenerator:
  - name: example-props
    envs:
      - example.properties

I have two overlays defined, test and prod:

$ cd ../overlays
$ tree
.
├── prod
│   ├── example.properties
│   └── kustomization.yaml
└── test
    └── kustomization.yaml

test/kustomization.yaml looks like this:

resources:
- ../../base

It's just importing the base without any changes, since the value of CONF_ENV from the base directory is test.

prod/kustomization.yaml looks like this:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../base

configMapGenerator:
  - name: example-props
    behavior: merge
    envs:
      - example.properties

And prod/example.properties looks like:

CONF_ENV=prod

If I run kustomize build overlays/test, I get as output:

apiVersion: v1
data:
  CONF_ENV: test
  SOME_OTHER_VAR: somevalue
kind: ConfigMap
metadata:
  name: example-props-7245222b9b
---
apiVersion: v1
kind: Pod
metadata:
  name: example
spec:
  containers:
  - command:
    - sleep
    - 1800
    envFrom:
    - configMapRef:
        name: example-props-7245222b9b
    image: docker.io/alpine
    name: alpine

If I run kustomize build overlays/prod, I get:

apiVersion: v1
data:
  CONF_ENV: prod
  SOME_OTHER_VAR: somevalue
kind: ConfigMap
metadata:
  name: example-props-h4b5tc869g
---
apiVersion: v1
kind: Pod
metadata:
  name: example
spec:
  containers:
  - command:
    - sleep
    - 1800
    envFrom:
    - configMapRef:
        name: example-props-h4b5tc869g
    image: docker.io/alpine
    name: alpine

That is, everything looks as you would expect given the configuration in base, but we have provided a new value for CONF_ENV.

You can find all these files here.