Changing hostname breaks Rabbitmq when running on Kubernetes

Question

I'm trying to run Rabbitmq using Kubernetes on AWS. I'm using the official Rabbitmq docker container. Each time the pod restarts the rabbitmq container gets a new hostname. I've setup a service (of type LoadBalancer) for the pod with a resolvable DNS name.

But when I use an EBS to make the rabbit config/messsage/queues persistent between restarts it breaks with:

exception exit: {{failed_to_cluster_with,
                     ['rabbitmq@rabbitmq-deployment-2901855891-nord3'],
                     "Mnesia could not connect to any nodes."},
                 {rabbit,start,[normal,[]]}}
  in function  application_master:init/4 (application_master.erl, line 134)

rabbitmq-deployment-2901855891-nord3 is the previous hostname rabbitmq container. It is almost like Mnesia saved the old hostname :-/

The container's info looks like this:

              Starting broker...
=INFO REPORT==== 25-Apr-2016::12:42:42 ===
node           : rabbitmq@rabbitmq-deployment-2770204827-cboj8
home dir       : /var/lib/rabbitmq
config file(s) : /etc/rabbitmq/rabbitmq.config
cookie hash    : XXXXXXXXXXXXXXXX
log            : tty
sasl log       : tty
database dir   : /var/lib/rabbitmq/mnesia/rabbitmq

I'm only able to set the first part of the node name to rabbitmq using the RABBITMQ_NODENAME environment variable.

Setting RABBITMQ_NODENAME to a resolvable DNS name breaks with:

Can't set short node name!\nPlease check your configuration\n"

Setting RABBITMQ_USE_LONGNAME to true breaks with:

Can't set long node name!\nPlease check your configuration\n"

Update:

• Setting RABBITMQ_NODENAME to rabbitmq@localhost works but that negates any possibility to cluster instances.

            Starting broker...
  =INFO REPORT==== 26-Apr-2016::11:53:19 ===
  node           : rabbitmq@localhost
  home dir       : /var/lib/rabbitmq
  config file(s) : /etc/rabbitmq/rabbitmq.config
  cookie hash    : 9WtXr5XgK4KXE/soTc6Lag==
  log            : tty
  sasl log       : tty
  database dir   : /var/lib/rabbitmq/mnesia/rabbitmq@localhost
  

• Setting RABBITMQ_NODENAME to the service name, in this case rabbitmq-service like so rabbitmq@rabbitmq-service also works since kubernetes service names are internally resolvable via DNS.

            Starting broker...
  =INFO REPORT==== 26-Apr-2016::11:53:19 ===
  node           : rabbitmq@rabbitmq-service
  home dir       : /var/lib/rabbitmq
  config file(s) : /etc/rabbitmq/rabbitmq.config
  cookie hash    : 9WtXr5XgK4KXE/soTc6Lag==
  log            : tty
  sasl log       : tty
  database dir   : /var/lib/rabbitmq/mnesia/rabbitmq@rabbitmq-service
  

Is this the right way though? Will I still be able to cluster multiple instances if the node names are the same?

Answer 1

In addition to the first reply by @ant31:

Kubernetes now allows to setup a hostname, e.g. in yaml:

template:
  metadata:
    annotations:
      "pod.beta.kubernetes.io/hostname": rabbit-rc1

See https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/dns # A Records and hostname Based on Pod Annotations - A Beta Feature in Kubernetes v1.2

It seems that the whole configuration alive multiple restarts or re-schedules. I've not setup a cluster however I'm going to follow the tutorial for mongodb, see https://www.mongodb.com/blog/post/running-mongodb-as-a-microservice-with-docker-and-kubernetes

The approach will be probably almost same from kubernetes point of view.

Answer 2

The idea is to use a different 'service' and 'deployment' for each of the node you want to create.

As you said, you have to create a custom NODENAME for each i.e:

RABBITMQ_NODENAME=rabbit@rabbitmq-1

Also rabbitmq-1,rabbitmq-2,rabbitmq-3 have to be resolved from each nodes. For that you can use kubedns. The /etc/resolv.conf will look like:

search rmq.svc.cluster.local 

and /etc/hosts must contains:

127.0.0.1 rabbitmq-1  # or rabbitmq-2 on node 2...

The services are here to create a stable network identity for each nodes

rabbitmq-1.svc.cluster.local
rabbitmq-2.svc.cluster.local
rabbitmq-3.svc.cluster.local

The different deployments resources will allow you to mount a different volume on each node.

I'm working on a deployment tool to simplify those actions: I've done a demo on how I scale and deploy rabbitmq from 1 to 3 nodes on kubernetes: https://asciinema.org/a/2ktj7kr2d2m3w25xrpz7mjkbu?speed=1.5

More generally, the complexity your facing to deploy a clustered application is addressed in the 'petset proposal': https://github.com/kubernetes/kubernetes/pull/18016