How to pass extra configuration to RabbitMQ with Helm?

Question

I'm using this chart: https://github.com/helm/charts/tree/master/stable/rabbitmq to deploy a cluster of 3 RabbitMQ nodes on Kubernetes. My intention is to have all the queues mirrored within 2 nodes in the cluster.

Here's the command I use to run Helm: helm install --name rabbitmq-local -f rabbitmq-values.yaml stable/rabbitmq

And here's the content of rabbitmq-values.yaml:

persistence:
  enabled: true

resources:
  requests:
    memory: 256Mi
    cpu: 100m

replicas: 3

rabbitmq:
  extraConfiguration: |-
    {
      "policies": [
        {
          "name": "queue-mirroring-exactly-two",
          "pattern": "^ha\.",
          "vhost": "/",
          "definition": {
            "ha-mode": "exactly",
            "ha-params": 2
          }
        }
      ]
    }

However, the nodes fail to start due to some parsing errors, and they stay in crash loop. Here's the output of kubectl logs rabbitmq-local-0:

BOOT FAILED
===========

Config file generation failed:
=CRASH REPORT==== 23-Jul-2019::15:32:52.880991 ===
  crasher:
    initial call: lager_handler_watcher:init/1
    pid: <0.95.0>
    registered_name: []
    exception exit: noproc
      in function  gen:do_for_proc/2 (gen.erl, line 228)
      in call from gen_event:rpc/2 (gen_event.erl, line 239)
      in call from lager_handler_watcher:install_handler2/3 (src/lager_handler_watcher.erl, line 117)
      in call from lager_handler_watcher:init/1 (src/lager_handler_watcher.erl, line 51)
      in call from gen_server:init_it/2 (gen_server.erl, line 374)
      in call from gen_server:init_it/6 (gen_server.erl, line 342)
    ancestors: [lager_handler_watcher_sup,lager_sup,<0.87.0>]
    message_queue_len: 0
    messages: []
    links: [<0.90.0>]
    dictionary: []
    trap_exit: false
    status: running
    heap_size: 610
    stack_size: 27
    reductions: 228
  neighbours:

15:32:53.679 [error] Syntax error in /opt/bitnami/rabbitmq/etc/rabbitmq/rabbitmq.conf after line 14 column 1, parsing incomplete
=SUPERVISOR REPORT==== 23-Jul-2019::15:32:53.681369 ===
    supervisor: {local,gr_counter_sup}
    errorContext: child_terminated
    reason: killed
    offender: [{pid,<0.97.0>},
               {id,gr_lager_default_tracer_counters},
               {mfargs,{gr_counter,start_link,
                                   [gr_lager_default_tracer_counters]}},
               {restart_type,transient},
               {shutdown,brutal_kill},
               {child_type,worker}]
=SUPERVISOR REPORT==== 23-Jul-2019::15:32:53.681514 ===
    supervisor: {local,gr_param_sup}
    errorContext: child_terminated
    reason: killed
    offender: [{pid,<0.96.0>},
               {id,gr_lager_default_tracer_params},
               {mfargs,{gr_param,start_link,[gr_lager_default_tracer_params]}},
               {restart_type,transient},
               {shutdown,brutal_kill},
               {child_type,worker}]

If I remove the rabbitmq.extraConfiguration part, the nodes start properly, so it must be something wrong with the way I'm typing in the policy. Any idea what I'm doing wrong?

Thank you.

Answer 1

According to https://github.com/helm/charts/tree/master/stable/rabbitmq#load-definitions, it is possible to link a JSON configuration as extraConfiguration. So we ended up with this setup that works:

rabbitmq-values.yaml:

rabbitmq:
  loadDefinition:
    enabled: true
    secretName: rabbitmq-load-definition
  extraConfiguration:
    management.load_definitions = /app/load_definition.json

rabbitmq-secret.yaml:

apiVersion: v1
kind: Secret
metadata:
  name: rabbitmq-load-definition
type: Opaque
stringData:
  load_definition.json: |-
    {
      "vhosts": [
        {
          "name": "/"
        }
      ],
      "policies": [
        {
          "name": "queue-mirroring-exactly-two",
          "pattern": "^ha\.",
          "vhost": "/",
          "definition": {
            "ha-mode": "exactly",
            "ha-params": 2
          }
        }
      ]
    }

The secret must be loaded into Kubernetes before the Helm chart is played, which goes something like this: kubectl apply -f ./rabbitmq-secret.yaml.

Answer 2

You can use config default of HelmChart

If needed, you can use extraSecrets to let the chart create the secret for you. This way, you don't need to manually create it before deploying a release. For example :

extraSecrets:
  load-definition:
    load_definition.json: |
      {
        "vhosts": [
          {
            "name": "/"
          }
        ]
      }
rabbitmq:
  loadDefinition:
    enabled: true
    secretName: load-definition
  extraConfiguration: |
    management.load_definitions = /app/load_definition.json

https://github.com/helm/charts/tree/master/stable/rabbitmq