RabbitMQ cluster - best practice for updating a node in a load balanced cluster?

Question

Summary: What's the best practice for updating a node in a load balanced cluster?

We use RabbitMQ Cluster behind a ha proxy load balancer to support easy clustering for our clients, as suggested in the RabbitMQ docs

Though the docs suggest this, they don't describe the best way to remove a node from the cluster for upgrades, and put it back in.

Here's the process I think we should use:

1. remove node from cluster by running rabbitmqctl stop_app on the node itself, and wait for it to shutdown
2. put node in maint mode in haproxy
3. perform maint work
4. join node back to cluster, confirm it rejoins and sync.
5. remove node from maint mode in haproxy

but I've had it suggested that we should remove it from ha proxy first, basically swapping steps 1 and 2 above

Here's the process suggested by another team member:

1. put node in maint mode in haproxy
2. remove node from cluster by running rabbitmqctl stop_app on the node itself, and wait for it to shutdown
3. perform maint work
4. join node back to cluster, confirm it rejoins and sync.
5. remove node from maint mode in haproxy

Which is the best way to do this?

Answer 1

For me, the obvious way would be to tell you haproxy that you want to stop sending requests to a server, and then stop the server itself, instead of the other way around.

I'm curious as to why you would want to stop the server first, and then put it into maint ? If you do it like this, some requests will go to your node before it is known that it is gone. I believe you can have haproxy set up to re-send those missed calls; so best case you have some requests that are a bit slower, worst case you have some missed requests.

There is no specific downside I can see with setting it in maint mode first so I would not consider the other option personally.