RabbitMQ "Zombie" consumers

Question

I'm using spot instances on EC2 to consume from a RabbitMQ queue. Each job takes a few seconds, and requires a manual 'ack'.

Problems arise when an instance is terminated. There's no time for a clean shutdown of the AMQP consumers, and so I'm left with the RabbitMQ server still thinking the dead consumers still exists. It doesn't re-deliver their jobs, and it still routes new jobs to them.

How can I get RabbitMQ to automatically timeout the zombie consumers and remove them from the worker pool?

Answer 1

I'd recommend setting the Heartbeat property on the RabbitMQ connection. If the heartbeat stops, RabbitMQ will eventually cut the connection and put the unacked messages back into the queue.

Answer 2

I was in the same place as you and resolved it by setting the connection heartbeat property on the consumer side (Node.Js using node_amqp module). Connection is closed and removed, along with the "zombie" consumer. Unacked jobs go back to Ready.