Make large static data files available to kubernetes pods

Question

I have a few quite large UTF-8 data files that pods need to load into memory on start up - from a couple of hundred KBs to around 50 MB.

The project (including helm chart) is open source but some of these files are not - otherwise I would probably just include them in the images. My initial thinking was to create configmaps but my understanding is that 50 MB is more than configmaps were intended for, so that might end up being a problem in some circumstances. I think secrets would also be overkill - they aren't secret, they just shouldn't be put on the open internet.

For performance reasons I'd rather have a copy in memory in each pod rather than going for a shared cache but I might be wrong on that. At the very least that will likely add more complexity than it's worth.

Are configmaps the way to go?

Answer 1

From my point of view, the best solution would be using init container to download the files from a secured storage (as it was mentioned by Andrew Savinykh in the comments), to the pod's volume and then use it in the pod's main container.

Please see the example:

apiVersion: v1
kind: Pod
metadata:
  name: init-demo
spec:
  containers:
  - name: nginx
    image: nginx
    ports:
    - containerPort: 80
    volumeMounts:
    - name: workdir
      mountPath: /usr/share/nginx/html
  # These containers are run during pod initialization
  initContainers:
  - name: install
    image: busybox
    command:
    - wget
    - "-O"
    - "/work-dir/index.html"
    - http://kubernetes.io
    volumeMounts:
    - name: workdir
      mountPath: "/work-dir"
  dnsPolicy: Default
  volumes:
  - name: workdir
    emptyDir: {}