Celery connection drop with AWS ELB and RabbitMQ

Question

In our environment, we use RabbitMQ and Celery on AWS to run tasks in parallel over many nodes.

Recently we turned RabbitMQ into a cluster of 3 nodes, configured a ha policy and added an AWS elastic load balancer (ELB) for port 5672 to all 3 nodes. Our Celery workers and client code all use the ELB DNS as the broker URL.

We have noticed since that change, that waiting for async tasks to finish will throw an exception IOError: Socket closed.

The ELB will shutdown all idle connections after 60 seconds. We have tasks that take few hours to complete.

Setting BROKER_HEARTBEAT to a value lower than 60 solved connection drops on the workers end. But we can't seem to find any setting that will keep the client connection alive.

Is this the correct approach to wait for long running tasks with Celery?

One workaround we haven't tested yet, is to recall the AsyncResult.wait() method until it ends successfully. So for example:

async_result = task.delay(params)

while True:
    try:
        async_result.wait()
        break
    except IOError:
        pass

We use:

1. RabbitMQ 3.6.5
2. Celery 3.1.20
3. Celery backend is pyamqp
4. Celery results backend is rpc

Answer 1

I believe what you need to do is extend the timeout on the AWS ELB. What's happening is the connection is being closed before the task is complete. You can accomplish this by issuing the following command

elb-modify-lb-attributes myTestELB --connection-settings "idletimeout=3600" --headers

This would give you an hour to complete the task. See https://aws.amazon.com/blogs/aws/elb-idle-timeout-control/ for more info on this.

If an hour is not enough then you're going to have to disable connection pooling. Add these two settings to your celery config

BROKER_POOL_LIMIT = None
BROKER_TRANSPORT_OPTIONS = {'confirm_publish': True}

Second will have a performance hit since it adds some overhead. Since you have long running tasks this may not be an issue. The second setting may not be necessary but I would recommend it given that you're behind a load balancer. This setting will make sure messages are received and not lost in the process.

Another option is breaking your long task into smaller tasks too! This may mean more code but it may be worth it in the long run.