Celery configure separate connection for producer and consumer

Question

We have an application setup on heroku, which uses celery to run background jobs. The celery app uses RabbitMQ as the broker. We used heroku’s RabbitMQ Bigwig add-on as AMQP message broker. This add-on specifies two separate url one optimized for producer and other optimized for consumer. Also, as per RabbitMQ documentation it is recommended to use separate connections for producer and consumer.

Celery documentation does not provide a ways to specify connections separately to producer and consumer. Is there a way to specify two different broker urls in celery?

Answer 1

Unfortunately, there isn't a clean way to do that. You can provide a custom broker connection explicitly on task.apply_async, but that means giving up on the connection pool feature. It might work for you.

from kombu import BrokerConnection
conn = BrokerConnection(hostname="producerbroker")

mytask.apply_async(args, kwargs, connection=conn)

The most straightforward solution is probably to have different config files for producer and worker.